RAID 6 LVM recovery from interrupted reconstruction

[rikjwells]

I have a pretty huge (16x1TB disks) RAID6 LVM array on a Red Hat box which has really thrown a spanner in its works and I'm in a right fix: I was part-way through rebuilding the array with one replacement disk when a second died and a third seemed to have fallen from the array. I was replacing a disk which had been showing signs through SMART of probable failure (uncorrectable errors) and I noticed the second disk was just starting to give similar but slightly less numerous errors.

Now my logic based on a gradually growing understanding of the workings of the LVM is that I stand a chance of resurrecting things as the data held on the machine has been static for a while (a few days) before my attempted drive replacement and none of the disks have actually FAILED. I have replaced the unit I removed so the physical devices are as they were before the exercise. I clearly need to carefully convince the configuration that it's "how it was" and there are a few meta-data files in the /etc/lvm/archive I believe I can use to re-"set the scene". Presently the machine fails to boot completely and it's patiently waiting in single-user mode for my next visit. Thankfully the whole office is on holiday until 10th January although it will be needed from then.

I intend investigating as much as possible prior to any action to avoid sudden and catastrophic complete loss of data on the array but would VERY much like to have any guidance.


Many thanks in advance,
Rik

[Rubberman]

Unfortunately, a lot of people think that RAIDs don't need to be backed up... In reality, they need to be backed up regularly just like normal discs/arrays (LVM). Their main benefit to the user is higher availability as they will continue to work even if one disc fails. Caveate User! Most arrays have discs from the same manufacturing batch, which means that when one fails, more are likely to follow suit. Result? Unrecoverable systems/data. Sorry, but I don't have a solution for you, other than to try to save your data to an external array/device/system(s). Not so helpfull, I know, but what is, is. So, wishing you a prosperous New Year, I remain unhelpfully yours...

[rikjwells]

I appreciate your comments and regrettably acknowledge the vulnerability of even RAID 6 and consequential need for Proper Backups. However, I am trying to delve into the bowels of the existing system and need some guidance as to how best attempt recovery of what I believe is still available, albeit a tricky operation ... Logically I believe I can recover the data without the need to resort to forensics or a specialist recovery service.

Alas this is clearly a significant Weak Link in the existing configuration and it has snapped in a particularly inconvenient manner but thankfully at a "quiet time"!


Cheers for now,
Rik.

[rikjwells]

I managed to piece the array back together and will write up a full account of the steps presently.

BIG PHEW.