mjhjmtu

The pernicious USB-stick stall problem - LWN.net, 2013.

Artem S. Tashkinov recently encountered a problem that will be familiar to at least some LWN readers. Plug a slow storage device (a USB stick, say, or a media player) into a Linux machine and write a lot of data to it. The entire system proceeds to just hang, possibly for minutes.

This time around, though, Artem made an interesting observation: the system would stall when running with a 64-bit kernel, but no such problem was experienced when using a 32-bit kernel on the same hardware.

On 64-bit x86, the writeback cache was allowed to grow to 20% of system RAM by default. Linus suggested to effectively limit it to ~180MB on all platforms, mimicking a limitation of the 32-bit x86 code. However current Linux (v4.18) does not do this. Compare Linus's suggested patch, to the current function in Linux 4.18.

But I also read an another article, describing some code which was merged in 2011 (Linux 3.2):

No-I/O dirty throttling - LWN.net, 2011

That is where Fengguang's patch set comes in. He is attempting to create a control loop capable of determining how many pages each process should be allowed to dirty at any given time. Processes exceeding their limit are simply put to sleep for a while to allow the writeback system to catch up with them.

[...]

The goal of the system is to keep the number of dirty pages at the setpoint; if things get out of line, increasing amounts of force will be applied to bring things back to where they should be.

[...]

This ratio cannot really be calculated, though, without taking the backing device (BDI) into account. A process may be dirtying pages stored on a given BDI, and the system may have a surfeit of dirty pages at the moment, but the wisdom of throttling that process depends also on how many dirty pages exist for that BDI. If a given BDI is swamped with dirty pages, it may make sense to throttle a dirtying process even if the system as a whole is doing OK. On the other hand, a BDI with few dirty pages can clear its backlog quickly, so it can probably afford to have a few more, even if the system is somewhat more dirty than one might like. So the patch set tweaks the calculated pos_ratio for a specific BDI using a complicated formula looking at how far that specific BDI is from its own setpoint and its observed bandwidth. The end result is a modified pos_ratio describing whether the system should be dirtying more or fewer pages backed by the given BDI, and by how much.

So Linux had some per-device control over the dirty page cache, since at least 2011 and Linux 3.2. Why did we still have the problem in 2013, that the USB-stick stall affects the whole system?

The "USB-stick stall" article provides no evidence for its claim. I think it is very misleading, and maybe just wrong.

Artem did not report the entire system hanging while it flushes cached writes to a USB stick. His report only complained that sync could take up to "dozens of minutes".

In a followup, Artem reported "the server almost stalls and other IO requests take a lot more time to complete even though mysqldump is run with ionice -c3". But this was not the USB-stick problem. It happened after creating a 10GB file on an internal disk.