Monday to Saturday, 8:00 – 19:00Free collection across Malta and GozoMaltiWhatsApp+356 7933 2877

NAS and RAID

RAID 5 rebuild failed: what happened and what to do now

A RAID 5 is built to survive the loss of one disk. The danger comes afterwards: the rebuild reads every sector of every remaining disk, and that is exactly when a second weakness shows up. If your rebuild stopped part way, the data is almost always still there, as long as nobody makes the next move in a hurry.

Published 3 min readBy Mirush Murataj

Cover of the RecData Malta guide: RAID 5 rebuild failed: what happened and what to do now
A failed rebuild rarely means two dead disks: usually one stale disk and one with unreadable sectors.

In short

  • A failed rebuild rarely means two dead disks: usually one stale disk and one with unreadable sectors.
  • Do not restart the rebuild, recreate the array or swap more disks.
  • In the lab, every disk is cloned and the array is rebuilt virtually, leaving out the out-of-date member.

Why a RAID 5 rebuild fails

When one disk of a RAID 5 fails, the controller or NAS keeps working in degraded mode: each missing block is recalculated on the fly from the other disks and the parity. When you insert a replacement, the rebuild recalculates the whole missing disk. To do that, it must read every sector of every remaining disk, from the first to the last.

That is where things go wrong. Disks in the same array usually come from the same batch, have the same age and have done the same work. If one has failed, the others are often close behind. Consumer drives are specified for one unrecoverable read error every 10^14 bits read, roughly every 12.5 TB. Reading four 8 TB disks in full is well within that range: one weak sector is enough for many controllers to stop the rebuild and take the whole volume offline.

The three most common scenarios

  • A second disk with bad sectors: the rebuild hits an unreadable area and stops. The disk is not dead, just partly unreadable.
  • A disk that dropped out long ago: the array had been degraded for weeks without anybody noticing. The disk that "just failed" is actually the second one, and the first holds old data.
  • The wrong disk pulled: a healthy disk is removed instead of the failed one, and the array has lost two members at once.

What not to do

  • Do not restart the rebuild. Each attempt reads the weak disk again and can turn a few bad sectors into a head crash.
  • Do not force a disk back online if you do not know when it left the array. Mixing an old member with current ones corrupts the file system silently.
  • Do not recreate or initialise the array in the controller or the NAS interface, even with "the same settings".
  • Do not run CHKDSK, fsck or a "repair" function on the degraded volume.
  • Label each disk with its slot number before you take anything out.

How we recover a failed RAID 5

1. Cloning every disk

Each disk is imaged sector by sector on hardware that handles weak areas gently, reading the easy zones first and coming back to the difficult ones. A disk with a mechanical fault is repaired in the cleanroom first. From then on, your original disks are no longer touched.

2. Finding the parameters

On the clones, we work out the disk order, the stripe size (often 64 KB on hardware controllers, 64 or 512 KB with Linux mdadm), the parity rotation (left-symmetric, right-asymmetric and so on) and the data offset. On software RAID, the superblocks give most of this directly, including an event counter that shows which disk dropped out first.

3. Virtual reconstruction

The array is assembled in software, leaving out the stale member. Where the second disk has unreadable sectors, the missing blocks are recalculated from the parity of the other disks: as long as two disks are not unreadable at the same place, nothing is lost.

4. File system and files

The volume is mounted read-only, the file system (NTFS, ext4, Btrfs, XFS, VMFS) is checked, then the files are extracted. You receive the list of recovered files before you pay.

Preventing the next one

  • Use RAID 6 or RAID 10 for arrays of more than four or five large disks.
  • Turn on email alerts so a degraded array never goes unnoticed.
  • Run scheduled data scrubbing so weak sectors are found while the array is still healthy.
  • Keep a real backup: RAID protects against a disk failure, not against a failed rebuild, deletion or ransomware.

For the general principles, see The complete guide to RAID. For NAS units in particular, see Synology volume crashed or storage pool degraded and QNAP storage pool inactive: causes and recovery.

Frequently asked questions

Can a RAID 5 with two failed disks be recovered?

Very often, yes. Two disks are rarely unreadable at the same place: the stale disk and the one with bad sectors complement each other once they are cloned.

Should I replace the second failed disk and rebuild again?

No. Without a full copy of the weak disk, a second rebuild can finish it off. Switch the array off and have the disks cloned first.

How long does a lab recovery take?

From a few days to about two weeks, depending on the number and size of the disks and their condition. The Standard (€49) and Emergency (€99, diagnosis within 24 to 48 hours) levels are available for urgent cases.

Is your data important?

Do not try anything that could make the fault worse. The diagnosis is free and you get a fixed quote before any work starts.

Request a free diagnosis RAID 5 data recovery

Your device has stopped responding? Switch it off and talk to us.

WhatsAppFree diagnosis