recover data from ext3 partition

[wolfgangsz]

Hi all,

here is the situation:
A HP Netserver LP2000r, with original SCSI controller and HP NetRaid-2M controller, 3x 36GB Ultra3 HDD in RAID5, Debian (sarge/etch), has crashed after 992 days without reboot. From all that I can see, a hardware failure, most likely with the memory. The HP Diagnostic tools cannot find any problem, but everytime I boot into Knoppix, I get between 2minutes and 2 hours of runtime, and then either a kernel oops or just a complete and sudden halt.
Well, the box has earned its money. However, there is some data on the drives that I need to recover (yes, I have beaten myself up properly about not backing up that data, don't even go there !). There are three partitions: sda1 is /, sda2 is swap and sda3 is a LVM volume with 3 logical volumes on it. As far as I can tell, the hardware defect must have been creeping in and has made a total mess of the inodes in all these partitions.
After booting into Knoppix, I can restore the volumes using pvscan, vgscan, lvscan, vgcfgrestore and vgchange. If I try and mount them: mayhem. So I try and check them, using fsck.ext3. All sorts of interesting nonsense. such as a completely empty inode 11 (the first inode) and then obviously from there on all else is pointless.
I tried using debugfs, but the information on what to do with it is somewhat spurious.

So now here is my question: Is there anybody out there who could help me with this? Any pointers would be highly appreciated.

P.S.: Tomorrow I will go and get myself a 16GB Flash Drive and then hopefully I will be able to dump the partitions one by one onto that drive and transfer the images onto a different computer for analysis and data recovery. But the help will still be highly appreciated.

[oz]

The best tools that I've seen for partition/data recovery are TestDisk and PhotoRec:

TestDisk - CGSecurity

PhotoRec - CGSecurity

You can find them both on the Parted Magic LiveCD:

Parted Magic News

[Lakshmipathi]

To add to Ozar,You should try Carlo wood's ext3grep. check this HOWTO undelete removed files and directories on an ext3 file system. and also check ext3grep | Google Groups . giis tool won't be helpful.
also drop a mail to ext3user mailing list.

[Rubberman]

Murphy laughs. No backups? No problem - no data...

R - Randomly
A - Accessible
I - (or) Inaccessible
D - Data

One thing that many years of work with RAID arrays has shown me, is that data that has not been backed up, will still not be available when the system fails. For me, the best backup system is to use 1.5TB sata-3 drives like tape. Fast, reliable, cheap, and easy to resstore. I'd rather use LVM to join multiple drives into one file system image, then use large/cheap esata drives for incremental backups. When one of the drives fails, you can restore the missing data without too much gnashing of teeth (other than having the system unavailable for a couple of hours). To avoid that, you keep a mirror image going, on another system.

[Rubberman]

One last comment about this "high availability" subject. There are some commercial (expensive) RAID devices that do handle single drive failures well, and allow hot removal of failed drives, their replacement, and online restoration of the data. They aren't cheap, and are typically NAS or SAN devices with an appropriately high price point such that most SMB sites are not likely to make the significant investment required. Most of the installations of this type with which I am personally familiar are enterprise systems where downtime is going to cost over a million USD per hour. Big commercial web sites, major manufacturers, and safety critical installations are examples of the types of sites that need such reliability.