mjhjmtu

I have an x86_64 Ubuntu 17.10 install (stock 4.13 kernel) with an SSD and three 1TB WD HDDs which each have a 750GB partition that's used in a 1.45TB RAID5 array. The SSD has my / on it, and the RAID array has LVM defined which I use for /home.

Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : active raid5 sdc1[3] sdd1[1] sdb1[0]
 1572601856 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/3] [UUU]
 [====>................] resync = 21.3% (168261416/786300928) finish=64.7min speed=159157K/sec
 bitmap: 6/6 pages [24KB], 65536KB chunk

It's worked fine, until about Christmas-time, since when I've repeatedly turned my computer on and found:

[ 2.334153] md/raid:md0: not clean -- starting background reconstruction
[ 2.334164] md/raid:md0: device sdc1 operational as raid disk 2
[ 2.334165] md/raid:md0: device sdd1 operational as raid disk 1
[ 2.334165] md/raid:md0: device sdb1 operational as raid disk 0
[ 2.334333] md/raid:md0: raid level 5 active with 3 out of 3 devices, algorithm 2
[ 2.334479] md0: bitmap file is out of date (39126 < 39127) -- forcing full recovery
[ 2.334493] md0: bitmap file is out of date, doing full recovery
[ 2.422418] md0: detected capacity change from 0 to 1610344300544
[ 2.422606] md: resync of RAID array md0
...
[ 9.537010] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null)

So to be clear, this is the bitmap itself is out of date, and thus a full (slow) resync takes place. The filesystem itself comes up clean. I assume it's a timing problem on shutdown and LVM is being unmounted but the RAID not halted before poweroff? I can't see any odd behaviour when I turn the machine off. The syslogs show some things shutting down and that's it.

If I perform a halt instead of a poweroff this drastically reduces the chances of this happening, but it's still happened this morning, hence finally writing about it after being out of ideas for three months.

Detail of the RAID array:

/dev/md0:
 Version : 1.2
 Creation Time : Fri Sep 11 17:49:27 2015
 Raid Level : raid5
 Array Size : 1572601856 (1499.75 GiB 1610.34 GB)
 Used Dev Size : 786300928 (749.88 GiB 805.17 GB)
 Raid Devices : 3
 Total Devices : 3
 Persistence : Superblock is persistent

 Intent Bitmap : Internal

 Update Time : Mon Apr 2 08:38:28 2018
 State : active, resyncing 
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
 Spare Devices : 0

 Layout : left-symmetric
 Chunk Size : 512K

 Resync Status : 52% complete

 Name : underlay:0 (local to host underlay)
 UUID : 520c8995:8d934562:0e2f5b8e:d460bfed
 Events : 40381

 Number Major Minor RaidDevice State
 0 8 17 0 active sync /dev/sdb1
 1 8 49 1 active sync /dev/sdd1
 3 8 33 2 active sync /dev/sdc1

I don't even know how to investigate this further. I've set GRUB to disable splash screens so I can watch dmesg on screen and see nothing interesting. Sometimes I've had services fail to exit and systemd has waited 90s before killing them. I've not been able to work out which they are and whether they'd be the ones that cause a safe unmount but unsafe RAID (turn off? disable? unmount?). I don't even really understand how the kernel normally turns off RAIDs to see what it's doing wrong here.

Secondly, any tips on a RAID resync not totally destroying the interactivity of my desktop would be appreciated. IO throttling via /proc/sys/dev/raid/speed_limit_max doesn't actually work in the way I hoped, my computer just syncs at full tilt for e.g. 10s then waits for 3s so it syncs slower and is still annoying to use for two hours.

The problem turned out to be a network mount in my fstab that was sometimes hanging on shutdown. I'm not sure why as the network mount wasn't on a mountpoint inside the RAID filesystem, they both mounted on / which is my SSD.

I only really spotted it as migrating to 18.04 didn't fix it and I had delays on startup which turned out to be related to the netmount.