You replaced the failed drive, started the rebuild, and somewhere around 60% a second drive dropped out. Now the array is offline and the controller is reporting two failed members in a set that tolerates one.
This is a bad afternoon. It’s also more recoverable than the controller’s error message implies — provided you stop here rather than trying the next obvious thing.
If this is a Synology or QNAP box rather than a server with a RAID card, the NAS version of this article is more specific to your situation.
Why rebuilds fail so often
A RAID 5 rebuild reconstructs the missing drive by reading every sector of every surviving drive and computing parity. On a degraded array there’s no redundancy left, so any read error during that process is unrecoverable — and the controller responds by marking that drive failed too.
Two things make this likely rather than unlucky:
Unrecoverable read errors are a statistical certainty at scale. Consumer drives are specified around one unrecoverable read error per 10^14 bits, roughly one per 12 TB read. A rebuild across several multi-terabyte drives reads a comparable amount of data. You are, in a real sense, rolling dice with poor odds.
Your drives are the same age from the same batch. They were bought together, racked together, and have run identical hours at identical temperature. When one reaches end of life, the others are near it. Correlated failure is the normal case, not the exception.
Add the fact that a rebuild is the heaviest sustained load an array ever sees, and second-drive failures during rebuild stop being surprising.
Stop doing these immediately
Don’t force drives back online repeatedly. Most controllers let you force a failed member online. Each attempt can write new configuration metadata, and repeated forcing across different members leaves inconsistent generation counters that make reconstructing the true array state much harder.
Don’t run another rebuild. If the first one killed a drive, the second one will finish the job — and a rebuild writes across members, potentially over data that was still recoverable.
Don’t initialize, clear the configuration, or create a new array. This is the one that ends cases. Controller metadata is what records stripe size, drive order, and parity rotation.
Don’t run CHKDSK or fsck on the array volume. It’s not a file system problem yet, and repairing in place on degraded storage is how recoverable cases stop being recoverable.
Don’t swap drives between bays to test. Order is everything.
Free evaluation. Firm quote before any work begins.
Do this instead
- Power the array down. Every hour it stays up is more stress on marginal drives.
- Label the drives by bay before removing them. Number them, photograph the serials in order. This is the single most useful thing you can do.
- Record the controller details — make and model, RAID level, stripe size if you know it, and what the event log said before things went wrong.
- Write down the sequence of events. Which drive failed first, what was attempted, in what order. It genuinely changes the approach.
Why this is usually recoverable
Here’s the part the controller doesn’t tell you: a drive marked “failed” is rarely dead.
RAID controllers are conservative by design. A drive that takes too long on one read gets dropped from the array to protect data integrity. That drive typically still contains 99.99% readable data — it just has a handful of bad sectors and a controller that lost patience with it.
So a “two drive failure” is usually one genuinely failed drive plus one drive with a small number of unreadable sectors. That’s a much better position than it sounds.
The approach is to image every member individually, handling bad sectors carefully, then determine the array parameters — stripe size, drive order, parity rotation, offset — and assemble the array virtually from the images. Nothing is ever assembled on the original drives. Where a sector is unreadable on one image, parity from the others often reconstructs it exactly, which is the whole point of RAID and it still works during recovery even when it failed during the rebuild.
When it gets hard
Multiple drives with real mechanical failures. If two members are clicking, that’s cleanroom work on both and I’ll refer it out.
A rebuild that ran a long way before failing. Partial rebuilds write reconstructed data across members. Recoverable, but messier.
Arrays where the configuration was cleared. Parameters have to be derived analytically from the data itself. Possible, slower, more expensive.
Large enterprise configurations. Beyond a handful of drives I’ll tell you honestly that a larger lab is a better fit.
The bottom line
A failed rebuild usually means one dead drive and one drive the controller gave up on, not two dead drives. Power down, label the bays, and resist every prompt offering to force, rebuild, or reinitialize. Almost all of the unrecoverable RAID cases I hear about were made unrecoverable by the third or fourth recovery attempt, not by the original failure.
Array down after a failed rebuild? Start a case — tell me the controller model, the RAID level, how many drives, and exactly what’s been attempted since it dropped. That last part matters most.