RAID5 woes, time to send to a commercial data recovery expert?
Greetings, enlightened reader...
I work in an academic research lab. We recently discovered that the 5-disk RAID5 array in our main workstation was not being monitored by the cron job that was supposed to be monitoring it. We discovered it when the second drive failed in some way and the whole filesystem disappeared.
No problem, I thought. We've got everything backed up on a second external RAID. I replaced the two failed drives, and a third that smartctl revealed was having a few read errors, and built a new md0 in the workstation.
I realized that I could copy data onto the new array even while it was in the process of re-calculating parity bits. I started doing that, then realized that the script that had been supposedly backing up from the main array to the external backup ALSO had not been working, for the last seven months. I immediately stopped the rebuild, about 10-15% of the way through I'd guess, and cursed my fate and stupidity.
I'm currently backing up all eight involved drives (the original 5, plus the 3 new ones that may now carry some useful information) using dd conv=noerror. Once I'm done with that, I'm sort of at a loss. I've been reading voraciously over the past week to try to figure out the best course of action. It seems like there might be some way that I might be able to get back at least some of the data that I care about. It also seems like the more I play with things, the harder it will be to recover anything. Paying for a professional recovery ($5000-$10000) is a colossal strain on the lab's budget; but is it the only way to go? Or are there some other resources I'm unaware of, or things that I could try without risking making things worse?
Many many thanks for any advice anybody can give.