Today I noticed that there are bunch of messages complaining about the RAID array (it's a software RAID10), so I started looking into it but need help because I'm unsure if I interpret the status output correctly (I've kinda forgotten the actual RAID set-up because the machine is at a remote location and I configured it about a year or two ago)... if I remember correctly the system was suppose to have 8x 2TB disks, but that's about all I can remember.

System mail:

 N 14 root@edmedia.loca Wed May 25 21:30 32/1059 Fail event on /dev/md/0:EDMedia
 N 15 root@edmedia.loca Thu May 26 06:25 30/1025 DegradedArray event on /dev/md/0:EDMedia
 N 16 root@edmedia.loca Thu May 26 06:25 30/1025 SparesMissing event on /dev/md/0:EDMedia

The bit that's specifically confusing me, now that I'm looking at the outputs, is this:

Number Major Minor RaidDevice State
 0 0 0 0 removed

Does it mean that a disk has been removed (or that it dropped from the array)? Should I try re-adding '/dev/sda1' to it? And is there any way I can tell that '/dev/sda1' was part of '/dev/md0' without adding a partitioned disk in-use by something, only to make things worse?

Status outputs:

'mdadm -D /dev/md0' output:

/dev/md0:
 Version : 1.2
 Creation Time : Mon Feb 8 23:15:33 2016
 Raid Level : raid10
 Array Size : 2197509120 (2095.71 GiB 2250.25 GB)
 Used Dev Size : 1465006080 (1397.14 GiB 1500.17 GB)
 Raid Devices : 3
 Total Devices : 2
 Persistence : Superblock is persistent

 Intent Bitmap : Internal

 Update Time : Thu Sep 1 19:54:05 2016
 State : clean, degraded
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
 Spare Devices : 0

 Layout : near=2
 Chunk Size : 512K

 Name : EDMEDIA:0
 UUID : 6ebf98c8:d52a13f0:7ab1bffb:4dbe22b6
 Events : 4963861

 Number Major Minor RaidDevice State
 0 0 0 0 removed
 1 8 17 1 active sync /dev/sdb1
 2 8 33 2 active sync /dev/sdc1

'lsblk' output:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 1.4T 0 disk
└─sda1 8:1 0 1.4T 0 part
sdb 8:16 0 1.4T 0 disk
└─sdb1 8:17 0 1.4T 0 part
 └─md0 9:0 0 2T 0 raid10
 ├─md0p1 259:0 0 1.5M 0 md
 ├─md0p2 259:1 0 244.5M 0 md /boot
 └─md0p3 259:2 0 2T 0 md
 ├─EDMedia--vg-root 253:0 0 2T 0 lvm /
 └─EDMedia--vg-swap_1 253:1 0 16G 0 lvm [SWAP]
sdc 8:32 0 1.4T 0 disk
└─sdc1 8:33 0 1.4T 0 part
 └─md0 9:0 0 2T 0 raid10
 ├─md0p1 259:0 0 1.5M 0 md
 ├─md0p2 259:1 0 244.5M 0 md /boot
 └─md0p3 259:2 0 2T 0 md
 ├─EDMedia--vg-root 253:0 0 2T 0 lvm /
 └─EDMedia--vg-swap_1 253:1 0 16G 0 lvm [SWAP]
sdd 8:48 0 1.4T 0 disk
└─sdd1 8:49 0 1.4T 0 part
sdj 8:144 0 298.1G 0 disk
└─sdj1 8:145 0 298.1G 0 part
sr0 11:0 1 1024M 0 rom

'df' output:

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/dm-0 2146148144 1235118212 801988884 61% /
udev 10240 0 10240 0% /dev
tmpfs 1637644 17124 1620520 2% /run
tmpfs 4094104 0 4094104 0% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 4094104 0 4094104 0% /sys/fs/cgroup
/dev/md0p2 242446 34463 195465 15% /boot

'watch -n1 cat /proc/mdstat' output:

Every 1.0s: cat /proc/mdstat Thu Sep 1 21:26:22 2016

Personalities : [raid10]
md0 : active raid10 sdb1[1] sdc1[2]
 2197509120 blocks super 1.2 512K chunks 2 near-copies [3/2] [_UU]
 bitmap: 16/17 pages [64KB], 65536KB chunk

unused devices: <none>

Seems that your raid10 array was configured to have 2 active drives plus one spare. The spare is missing.

This can have several reasons:

1. Maybe you removed the spare disk from the server


2. Maybe one drive died and the existing hot spare became active now after a rebuild.


3. Maybe the hot spare died before it could be ever used.


4. Maybe one drive (or cable) "was" broken at one time in past and has been automatically removed from the array.

You may check if your server has one broken disk which you don't even see anymore in lsblk output. Could also be that one of your other drives (sda1 or sdd1) was part of your array in past but is broken now. (It can't be sdj1 because it's too small ).

Remove all broken drives from the server.

To avoid the warnings re-add a hot spare drive (maybe one of the unused, non-broken ones) or configure your array to not have a hot-spare anymore.
Be aware that in case 4 the probability that the same drive will fail again is high.

BTW to see what exactly happened in past you could grep the old logfiles for relevant messages.

Inspected the system logs as rudimeier suggested and found out there had been a power outage event back at May, after which the RAID array errors started popping up. Since this is a software RAID10 (1+0), I'm thankful only the spare disk flew out of the array instead of the whole array irreversibly crashing. After doing a couple HDD tests with the trusty old Hiren's boot CD and just for variety - Partition Wizard bootable... all suspicious disks checked out with no errors/issues.

I erased (with Partition Wizard bootable, so that the disk would be unformatted and unpartitioned) and then re-added the spare using:

mdadm --add /dev/md0 /dev/sda1