mjhjmtu

On my system with 4.9.86 kernel I have noticed weird behaviour with my disk (HDD with 5400 rpm), the %util goes 100% for quite some time constantly (for 5 minutes or so), I do see the avrg-rq size is 8K when this happens. avgqu-sz and await is also very high, causing many processes going into D state (including jdb2 thread) . I have also noticed KBDirty going high this time (658 MB in this case which is usually in few KBs otherwise), Am I hitting disk saturation?

SAR Memory Usage:======================================
Linux 4.9.86     01/07/19        _x86_64_        (32 CPU)
11:29:20    kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit  kbactive   kbinact   kbdirty
11:29:21     80270488  52009236     39.32    354368  17373312  15789156      7.92  10257860  15388656    658488
Average:     80270488  52009236     39.32    354368  17373312  15789156      7.92  10257860  15388656    658488

SAR IO Usage:======================================
Linux 4.9.86     01/07/19        _x86_64_        (32 CPU)

11:29:22          tps      rtps      wtps   bread/s   bwrtn/s
11:29:23       351.00      0.00    351.00      0.00   2808.00
Average:       351.00      0.00    351.00      0.00   2808.00

SAR Device IO activity:======================================
Linux 4.9.86     01/07/19        _x86_64_        (32 CPU)

11:29:23          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
11:29:24        loop5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
11:29:24          sda    285.00      0.00   2280.00      8.00    143.51    510.94      3.51    100.00
11:29:24        vault      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util

Average:        loop5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
Average:          sda    285.00      0.00   2280.00      8.00    143.51    510.94      3.51    100.00
Average:        vault      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

SAR Queue and Load avg:======================================
Linux 4.9.86     01/07/19        _x86_64_        (32 CPU)

11:29:25      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15   blocked
11:29:26            0      1043      3.39      2.30      2.15         2
Average:            0      1043      3.39      2.30      2.15         2

the file system mounted as ext3 with ext4 driver, data=ordered,barrier=0 setting with journaling enabled.

Raid configuration:

  Model:SAS2008 Firmware Version: 9.00.00.00 RAID Level:RAID1
 

This looks like what you'd expect to see if process were to send a lot of non-sequential (random) small writes. You'd have a relatively small average request size (8, which probably means 8×512 byte sectors = 4K, so the minimum with normal writes). Having more dirty buffers also is consistent, it means the writes have been passed off to the kernel & the kernel is working on writing them to disk. 285 tps is pretty darn nice performance for a magnetic disk.

You need to investigate what programs are writing to disk and see if they're exhibiting abnormal behavior. Or if the programs can be configured to spread write better (e.g., if it's a database, the dirty page writeback speed is often configurable).

ext3 isn't really recommended for, well, anything. ext4 is a conservative choice for a replacement (but with barrier=0, you're clearly not concerned about that); XFS is another good choice (and still very reliable). But I doubt that'll really help here. SSDs, though, certainly will give far higher IOPS.