mjhjmtu

I'm trying to replace a failed disk in a RAID1 btrfs filesystem.

I can still mount the partition rw (after about 5 minute delay and lots of I/O kernel errors).

I started replace with -r in an attempt to have the failed disk not impact the speed of the operation:

 -r
 only read from <srcdev> if no other zero-defect mirror exists.
 (enable this if your drive has lots of read errors, the access
 would be very slow)

Still, I'm getting really poor performance. The partition is 3.6TiB, and in 9.25 hours I got:

3.8% done, 0 write errs, 0 uncorr. read errs

At this rate, it will take over 10 days to complete!!!

Due to circumstances beyond my control, this is too long to wait.

I'm seeing kernel errors regarding the failed disk quite commonly, averaging every 5 minutes or so:

Jan 26 09:31:53 tara kernel: print_req_error: I/O error, dev sdc, sector 68044920
Jan 26 09:31:53 tara kernel: BTRFS warning (device dm-3): lost page write due to IO error on /dev/mapper/vg4TBd2-ark
Jan 26 09:31:53 tara kernel: BTRFS error (device dm-3): bdev /dev/mapper/vg4TBd2-ark errs: wr 8396, rd 3024, flush 58, corrupt 0, gen 3
Jan 26 09:31:53 tara kernel: BTRFS error (device dm-3): error writing primary super block to device 2
Jan 26 09:32:32 tara kernel: sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 26 09:32:32 tara kernel: sd 2:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
Jan 26 09:32:32 tara kernel: sd 2:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
Jan 26 09:32:32 tara kernel: sd 2:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 02 eb 9e 23 00 00 04 00
Jan 26 09:32:32 tara kernel: print_req_error: critical medium error, dev sdc, sector 391967000

I'm guessing that the errors are due to btrfs trying to write accounting data to the disk (even though is is completely idle).

Even mounted ro, btrfs may try to write to a disk. Mount option -onologreplay:

 Warning
 currently, the tree log is replayed even with a read-only
 mount! To disable that behaviour, mount also with nologreplay.

How can I speed up the process?

This article says that a replace will continue after reboot.

I'm thinking:

1. Cancel the current replace
   


2. Remove the failed disk


3. 
   mount -o degraded,rw
   
   
   • Hope there's no power outage given the gotcha of this one-time-only mount option)
   
   

At this point in time, I propose to simultaneously:

1. Allow replace to continue without the failed disk present (a recent scrub showed that the good disk has all the data)


2. Convert the data to single to allow mounting again rw in case of power outage during the process

Is this a sound plan to have the replace complete earlier?

My calculations say 6.5 hours (not 10 days) would be feasible given disk I/O speeds.

This answer mentions writes to the failed disk causing the replace to grind to a halt.

It suggests to dmsetup to setup a COW device on top of the failed disk such that any writes succeed.

Caution: In this case the filesystem was enclosed within a dmcrypt device. See my comment regarding the "gotcha" and potential data loss if this is not the case.

Given that the replace was crawling, I did the following:

1. Ensured that the degraded filesystem was noauto in /etc/fstab
   


2. Rebooted the machine (which took about 20 minutes due to I/O hangs)


3. 
   

   Disabled the LVM VG containing the btrfs fs on the failed drive:

   
   
   

   sudo vgchange -an <failed-vg>
   

   
   


4. 
   

   Disabled the failed device:

   
   
   

   echo 1 | sudo tee /sys/block/sdb/device/delete
   

   
   



5. Mounted the filesystem -o ro,degraded (degraded can only be used once)



6. 
   

   Checked replace status and saw it was suspended:

   
   
   

   Started on 26.Jan 00:36:12, suspended on 26.Jan 10:13:30 at 4.1%, 0 write errs, 0 
   

   
   


7. 
   

   Mounted -o remount,rw and saw the replace continue:

   
   
   

   kernel: BTRFS info (device dm-5): continuing dev_replace from <missing disk> (devid 2) to target /dev/mapper/vg6TBd1-ark @4%
   

   
   

As I'm writing this:

• 
  replace status shows a healthy 0.1% progress every 30 seconds or so


• 
  iostat -d 1 -m <target-dev> shows about 145MB/s (Seagate advertises 160MB/s)

Update:

After completion, I noticed that btrfs device usage /mountpoint was showing some Data,DUP and Metadata,single, rather than only RAID1, so I rebalanced:

btrfs balance start -dconvert=raid1,soft -mconvert=raid1,soft /mountpoint

Also, consider resizeing if both devices now contain slack:

btrfs filesystem resize max /mountpoint

I would also recommend that you scrub as I had 262016 correctable csum errors seemingly related to the interrupted replace.