mjhjmtu

I ended up with my 3-disk mdadm array not being able to assemble because each of the disks had corrupted superblocks.

I tried to force reassembling the array using

mdadm --create

This resulted in the ability to assemble the raid, but now there is no (ext4) filesystem. Other indicators seem to show that the data is still there, but I need to fix the filesystem.

I tried to fix the filesystem by seeing if I can restore the superblock:

mkfs.ext4 -n /dev/md0
fsck.ext4 -b <tried_all_of_the_blocks> /dev/md0

But I get

fsck.ext4: Filesystem has unexpected block size while trying to open /dev/md0
The superblock could not be read or does not describe a valid ext2/ext4/ext4 filesystem...

So I take it that there is no filesystem and therefore, no valid superblocks from which to restore the filesystem.

I have two questions:

1. Can I safely use mkfs.ext4 on the raid to regenerate the filesystem, without losing the data that appears to still be in the array?

2. Can I fix the superblock of the array using a backup from one of the individual disks?

Obviously, I do not understand many things, so I appreciate your kind response.

Here is the mdadm --examine output:

 root@server:~# mdadm --examine /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/md0
/dev/sdb1:
 Magic : a92b4efc
 Version : 1.2
 Feature Map : 0x1
 Array UUID : aa70a8ec:192f1719:23bc5df4:1ddac384
 Name : server:0 (local to host server)
 Creation Time : Sat Oct 28 00:21:46 2017
 Raid Level : raid5
 Raid Devices : 3

 Avail Dev Size : 4294703103 (2047.87 GiB 2198.89 GB)
 Array Size : 4294702080 (4095.75 GiB 4397.77 GB)
 Used Dev Size : 4294702080 (2047.87 GiB 2198.89 GB)
 Data Offset : 262144 sectors
 Super Offset : 8 sectors
 Unused Space : before=262056 sectors, after=1023 sectors
 State : clean
 Device UUID : bf5a4ff5:e4e3659e:99caca7c:333475f3

Internal Bitmap : 8 sectors from superblock
 Update Time : Sat Oct 28 05:48:33 2017
 Bad Block Log : 512 entries available at offset 72 sectors
 Checksum : 71521ea5 - correct
 Events : 3358

 Layout : left-symmetric
 Chunk Size : 512K

 Device Role : Active device 0
 Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
 Magic : a92b4efc
 Version : 1.2
 Feature Map : 0x1
 Array UUID : aa70a8ec:192f1719:23bc5df4:1ddac384
 Name : server:0 (local to host server)
 Creation Time : Sat Oct 28 00:21:46 2017
 Raid Level : raid5
 Raid Devices : 3

 Avail Dev Size : 4294703103 (2047.87 GiB 2198.89 GB)
 Array Size : 4294702080 (4095.75 GiB 4397.77 GB)
 Used Dev Size : 4294702080 (2047.87 GiB 2198.89 GB)
 Data Offset : 262144 sectors
 Super Offset : 8 sectors
 Unused Space : before=262056 sectors, after=1023 sectors
 State : clean
 Device UUID : 91528c6d:77861852:a1a4f630:9d8eb8ab

Internal Bitmap : 8 sectors from superblock
 Update Time : Sat Oct 28 05:48:33 2017
 Bad Block Log : 512 entries available at offset 72 sectors
 Checksum : 9b0ed7c - correct
 Events : 3358

 Layout : left-symmetric
 Chunk Size : 512K

 Device Role : Active device 1
 Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
 Magic : a92b4efc
 Version : 1.2
 Feature Map : 0x1
 Array UUID : aa70a8ec:192f1719:23bc5df4:1ddac384
 Name : server:0 (local to host server)
 Creation Time : Sat Oct 28 00:21:46 2017
 Raid Level : raid5
 Raid Devices : 3

 Avail Dev Size : 4294703103 (2047.87 GiB 2198.89 GB)
 Array Size : 4294702080 (4095.75 GiB 4397.77 GB)
 Used Dev Size : 4294702080 (2047.87 GiB 2198.89 GB)
 Data Offset : 262144 sectors
 Super Offset : 8 sectors
 Unused Space : before=262056 sectors, after=1023 sectors
 State : clean
 Device UUID : bdc61c9f:321a7ca6:2ed914d0:d10b96a4

Internal Bitmap : 8 sectors from superblock
 Update Time : Sat Oct 28 05:48:33 2017
 Bad Block Log : 512 entries available at offset 72 sectors
 Checksum : 27a0a727 - correct
 Events : 3358

 Layout : left-symmetric
 Chunk Size : 512K

 Device Role : Active device 2
 Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm: No md superblock detected on /dev/md0.

This is all on Ubuntu Server 16.04, with mdadm version 1.2

Between the original failure and your recovery efforts, it sounds like your array is pretty well wrecked. If you're very lucky, mdadm --create put the disks back in their original order with the original layout. In that case, you can point some data-recovery software such as foremost at the array and pull some of the files out, or you can send the disks off to a data recovery company and hope they'll do a better job at it than you can.