mjhjmtu

I ran a job that exceeded the memory, and began to write into the swap. The swap is 25 GB, and the RAM is 1TB.

I have stopped the job when the swap was full until 11GB, so 50% of it was still empty.

It did not trigger any OOM kill jobs, so everything is going as normal as before it happened.

I have only 8GB swap used now (3GB is cleaned after I have stopped the job), and it is slowly decreasing. But when I check vmstat, the si and so are both 0, so there is nothing going in or out of swap? How can this be possible?

free -lm
 total used free shared buff/cache available
Mem: 1031757 475637 49100 63 507019 553720
Low: 1031757 982657 49100
High: 0 0 0
Swap: 25767 8272 17495

I have a free space of 40GB in RAM, so should I expect that my running jobs will be killed by OOM not now but later on? 8GB used in swap is smaller than 40GB free space in RAM, so it looks fine, but I am not sure if still tells that OOM won't be triggered..

Couple months ago, when the OOM was triggered, it killed all my jobs and used swap was 25GB (all full) and then 10 minutes later 0GB.
However in this case, it takes a day to clean 2-3 GB from swap.
Is it bad news for my running jobs?

Do you think my running jobs are in danger of being killed, not now but later on, when free swapspace reaches 0 GB as it would trigger OOM to kill?
If that is the case, how can I prevent it from happening?

I would appreciate any help.

vmstat without parameters shows the average values since reboot. As the swap in / swap out is shown as blocks/second, it's not strange that those are showing 0, if you have a reasonable uptime.

Whatever is now in swap is memory that is in use, but not used since the time it was swapped out due to the overloading of the memory by your process. This is actually a good thing, as many processes have memory that is e.g. only used during startup. Having this swapped out means that you have more free RAM available for processes and buffers/cache.

The reason that in the earlier case, when you did get OOM, afterwards all swapspace was free again shortly after the event is probably because the OOM-causing process had used all the space and after it was stopped all the swapspace was free again.

The only time when you will get OOM is when there is no free swap space, and there is no RAM free (taking into account the buffers/cache, i.e. the "available" column of the free command).

For the rest you can usually trust Linux's memory management to do the right thing. Only if you have really special requirements due to the sort of load / application you're running can it be necessary to start tuning the memory management.