I found a question about mdadm spare disks which almost answers my question, but it isn't clear to me what is happening.We have a RAID5 set up with 4 disks - and all are labeled in normal operation as active/sync:    Update Time : Sun Sep 29 03:44:01 2013          State : clean  Active Devices : 4Working Devices : 4 Failed Devices : 0  Spare Devices : 0...Number   Major   Minor   RaidDevice State   0     202       32        0      active sync   /dev/sdc   1     202       48        1      active sync   /dev/sdd   2     202       64        2      active sync   /dev/sde    4     202       80        3      active sync   /dev/sdfBut then when one of the disks failed, the RAID stopped working:    Update Time : Sun Sep 29 01:00:01 2013          State : clean, FAILED  Active Devices : 2Working Devices : 3 Failed Devices : 1  Spare Devices : 1...Number   Major   Minor   RaidDevice State   0     202       32        0      active sync   /dev/sdc   1     202       48        1      active sync   /dev/sdd   2       0        0        2      removed   3       0        0        3      removed   2     202       64        -      faulty spare   /dev/sde   4     202       80        -      spare   /dev/sdfWhat is really going on here??The fix was to reinstall the RAID - luckily I can do that. Next time it'll probably have some serious data on it. I need to understand this so I can have a RAID that won't fail because of a single disk failure.I realized I didn't list what I expected vs. what happened.I expect that a RAID5 with 3 good disks and 1 bad will operate in a degraded mode - 3 active/sync and 1 faulty.What happened was a spare was created out of thin air and declared faulty - then a new spare was also created out of thin air and declared sound - after which the RAID was declared inoperative.This is the output from blkid:$ blkid/dev/xvda1: LABEL=/ UUID=4797c72d-85bd-421a-9c01-52243aa28f6c TYPE=ext4 /dev/xvdc: UUID=feb2c515-6003-478b-beb0-089fed71b33f TYPE=ext3 /dev/xvdd: UUID=feb2c515-6003-478b-beb0-089fed71b33f SEC_TYPE=ext2 TYPE=ext3 /dev/xvde: UUID=feb2c515-6003-478b-beb0-089fed71b33f SEC_TYPE=ext2 TYPE=ext3 /dev/xvdf: UUID=feb2c515-6003-478b-beb0-089fed71b33f SEC_TYPE=ext2 TYPE=ext3 The TYPE and SEC_TYPE are interesting as the RAID has XFS, not ext3....Logs for a mount attempted on this disk - which resulted in the end result listed earlier, as every other mount did - has these log entries:Oct  2 15:08:51 it kernel: [1686185.573233] md/raid:md0: device xvdc operational as raid disk 0Oct  2 15:08:51 it kernel: [1686185.580020] md/raid:md0: device xvde operational as raid disk 2Oct  2 15:08:51 it kernel: [1686185.588307] md/raid:md0: device xvdd operational as raid disk 1Oct  2 15:08:51 it kernel: [1686185.595745] md/raid:md0: allocated 4312kBOct  2 15:08:51 it kernel: [1686185.600729] md/raid:md0: raid level 5 active with 3 out of 4 devices, algorithm 2Oct  2 15:08:51 it kernel: [1686185.608928] md0: detected capacity change from 0 to 2705221484544Oct  2 15:08:51 it kernel: [1686185.615772] md: recovery of RAID array md0Oct  2 15:08:51 it kernel: [1686185.621150] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.Oct  2 15:08:51 it kernel: [1686185.627626] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.Oct  2 15:08:51 it kernel: [1686185.634024]  md0: unknown partition tableOct  2 15:08:51 it kernel: [1686185.645882] md: using 128k window, over a total of 880605952k.Oct  2 15:22:25 it kernel: [1686999.697076] XFS (md0): Mounting FilesystemOct  2 15:22:26 it kernel: [1686999.889961] XFS (md0): Ending clean mountOct  2 15:24:19 it kernel: [1687112.817845] end_request: I/O error, dev xvde, sector 881423360Oct  2 15:24:19 it kernel: [1687112.820517] raid5_end_read_request: 1 callbacks suppressedOct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423360 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: Disk failure on xvde, disabling device.Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: Operation continuing on 2 devices.Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423368 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423376 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423384 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423392 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423400 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423408 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423416 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423424 on xvde).Oct  2 15:24:19 it kernel: [1687112.821837] md/raid:md0: read error not correctable (sector 881423432 on xvde).Oct  2 15:24:19 it kernel: [1687113.432129] md: md0: recovery done.Oct  2 15:24:19 it kernel: [1687113.685151] Buffer I/O error on device md0, logical block 96Oct  2 15:24:19 it kernel: [1687113.691386] Buffer I/O error on device md0, logical block 96Oct  2 15:24:19 it kernel: [1687113.697529] Buffer I/O error on device md0, logical block 64Oct  2 15:24:20 it kernel: [1687113.703589] Buffer I/O error on device md0, logical block 64Oct  2 15:25:51 it kernel: [1687205.682022] Buffer I/O error on device md0, logical block 96Oct  2 15:25:51 it kernel: [1687205.688477] Buffer I/O error on device md0, logical block 96Oct  2 15:25:51 it kernel: [1687205.694591] Buffer I/O error on device md0, logical block 64Oct  2 15:25:52 it kernel: [1687205.700728] Buffer I/O error on device md0, logical block 64Oct  2 15:25:52 it kernel: [1687205.748751] XFS (md0): last sector read failedI don't see xvdf listed there...
RAID 5 with 4 disks fails to operate with one failed disk?
linux;raid;mdadm