mjhjmtu

I've recently examined a RHEL7.2 that hanged almost totally just because it have written to a CIFS filesystem. With the default settings of dirty_ratio = 30 and cifs being cached (for both reading and writing), these dirty pages were mostly cifs ones.

Under memory pressure, when system reclaimed most of the read cache, system stubbornly tried to flush&reclaim the dirty (write) cache. So the situation was a huge CPU iowait accompanied with an excellent local disk I/O completion time, a lot of processes in D uninterruptible wait and a totally unresponsive system. OOM killer never engaged, because there was free memory that system wasn't giving out. (I think there is also a bug with CIFS, that crawled the flushing to incredibly slow speeds. But nevermind that here.)

I was flabbergasted to find out that kernel treated flushing pages to some slow remote CIFS box in exactly the same way as to super-fast local SSD drive. It's just insensible to have a single dirty_ratio bag, it quickly leads to the situation where 30% of RAM contains dirty data from the slowest devices. What a waste of money.

The situation is reproducible; setting dirty_ratio = 1 solves the problem completely. But why do I need to sacrifice the cache of local disks just because I use a cifs mount?

Other than completely disabling caching of some devices, or setting vm.dirty_ratio to a very low value, are there any ways to "whitelist" the fast devices to have more write cache? Or to have the slow devices (or remote "devices" like //cifs/paths) use less write cache?

Q: Are there any ways to "whitelist" the fast devices to have more write cache? Or to have the slow devices (or remote "devices" like //cifs/paths) use less write cache?

It is possible to set limits manually. The bdi ("backing device") object has a sysfs attribute max_ratio. min_ratio can also be used to make some sort of reservation, but I don't know exactly what that means.

https://www.kernel.org/doc/Documentation/ABI/testing/sysfs-class-bdi

Currently, these limits will only start to take effect in once the global writeback cache has filled up to dirty_ratio + dirty_background_ratio / 2.

https://lore.kernel.org/lkml/20131029203050.GE9568@quack.suse.cz/

Q: I was flabbergasted to find out that kernel treated flushing pages to some slow remote CIFS box in exactly the same way as to super-fast local SSD drive.

Actually, the sysfs-class-bdi document claims:

Under normal circumstances each device is given a part of the
total write-back cache that relates to its current average
writeout speed in relation to the other devices.

Q: Under memory pressure, when system reclaimed most of the read cache, system stubbornly tried to flush&reclaim the dirty (write) cache. So the situation was a huge CPU iowait accompanied with an excellent local disk I/O completion time, a lot of processes in D uninterruptible wait and a totally unresponsive system. OOM killer never engaged, because there was free memory that system wasn't giving out. (I think there is also a bug with CIFS, that crawled the flushing to incredibly slow speeds. But nevermind that here.)

Here is one possible explanation:

"memory pressure" can actually decrease the limit calculated by dirty_ratio. dirty_ratio is actually a percentage of "available" memory, which means free memory + page cache. So if "available" memory goes down? It must put pressure on the size of the writeback cache, in some way. I'm not certain what happens, if the writeback cache you need to shrink is for a very slow device, but it sounds like a difficult case to handle.

See Writeback cache (`dirty`) seems to be limited to even less than dirty_background_ratio. What is it being limited by? How is this limit calculated?

I've seen some kernel code that handles a related possibility, where one BDI is completely non-responsive. It tries to make sure other BDI's can still make forward progress, but... they would be much more limited in how much writeback cache they can use. Performance would likely be degraded, even if there is only one writer.

After experimenting, I've found out that dirty_ratio "bag" is quite properly balanced. The processes that make pages dirty are somehow constrained. One cp process can easily take almost all the write cache possible, but even if you run 10 competing processes bursting, they rarely reach the write cache cap (dirty_ratio) at all.

Therefore, I attribute all the trouble to that CIFS-related bug I've mentioned. If more processes want to write to fast local disk, kernel would have used less for CIFS. Here, more processes wanted to just use memory and kernel couldn't flush&reclaim a large CIFS write cache due to said bug. Probably the 30% dirty_ratio wouldn't be a problem if not the bug.

I think you can set the proportion of the dirty ratio per device via

echo RATIO > /sys/class/bdi/MAJOR:MINOR/max_ratio