Backup a hard-drive through the CLI.

[BravoMike]

Dont quite know if this is the right area to post this in but here it goes.

Lately my external 500gb hard drive has been having some read write errors on certain files. If this happens when I'm using Windows then I have to close the program trying to access it and unmount and remount the hard drive to get it working again.

If this happens in Linux I just get a read-write error which is fine because its just a case of clicking "ok".

I just bough a new external hard drive and I am looking for a command which will copy all the files on one hard drive to other while outputting a list of all files which failed to copy (and obviously save me from clicking OK each time).

Would the following command do what I want?

cp -r FIRST_DRIVE NEW_DRIVE 2>>failed_files

Where FIRST_DRIVE is the root of the old drive and NEW_DRIVE is (unsurprisingly) the new hard drives root directory.

[wempy]

GNU's ddrescue at gnu.org (google ddrescue) may be a better tool for this. If you have a knoppix cd then a simple procedure to clone your failing drive onto the new one might be (example from ddrescue info pages):

1. install both drives in the machine (preferably both as the master drive on different channels of the HDD controller)
2. boot knoppix to a command line (knoppix 2 - at the isolinux prompt)
3. e2fsck /dev/hda(n) - fsck the failing partition(s) first using the correct fsck for your filesystem types.
4. run ddrescue "ddrescue [OPTIONS] INFILE OUTFILE [LOGFILE]"
both INFILE and OUTFILE can be device nodes rather than files

Example 1: Rescue a whole disc with two ext2 partitions in /dev/hda to
/dev/hdb
Note: you do not need to partition /dev/hdb beforehand.

ddrescue -n /dev/hda /dev/hdb logfile
ddrescue -dr3 /dev/hda /dev/hdb logfile
fdisk /dev/hdb
e2fsck -v -f /dev/hdb1
e2fsck -v -f /dev/hdb2

Your distribution may already have ddrescue installed - try info ddrescue at a prompt. There is another tool called dd_rescue which does a similiar job, though I have found that the GNU ddrescue to be more capable, and recovers from errors better.

[BravoMike]

I did a bit of research into ddrescue and it seemed the best tool for the job. I found it in a great set of applications built into a stripped down version of linux called System Restore CD, dont let the name fool you. I installed it on my flash drive.

So I booted into SysRes and got started.

First thing to do was mount the drives

root@sysresccd /root % cd /dev       
root@sysresccd /dev % ls -l sd*
brw-r----- 1 root disk 8,  0 2009-07-02 00:28 sda
brw-r----- 1 root disk 8,  1 2009-07-02 00:28 sda1
brw-r----- 1 root disk 8,  2 2009-07-02 00:28 sda2
brw-r----- 1 root disk 8,  5 2009-07-02 00:28 sda5
brw-r----- 1 root disk 8,  6 2009-07-02 00:28 sda6
brw-r----- 1 root disk 8,  7 2009-07-02 00:28 sda7
brw-r----- 1 root disk 8, 16 2009-07-02 00:28 sdb
brw-r----- 1 root disk 8, 17 2009-07-02 00:28 sdb1
brw-r----- 1 root disk 8, 32 2009-07-02 00:31 sdc
brw-r----- 1 root disk 8, 33 2009-07-02 00:31 sdc1

root@sysresccd /dev % mount /dev/sdc1 /media/BrokenHD
root@sysresccd /dev % mount /dev/sdd1 /media/NewHD

Then check that they had mounted ok by checking the contents of each.

root@sysresccd /dev % ls -l /media/BrokenHD 
total 128
dr-x------ 1 root root  4096 2009-04-04 23:14 Apps
dr-x------ 1 root root     0 2009-04-15 11:29 BK
dr-x------ 1 root root     0 2009-05-31 14:40 Conor
dr-x------ 1 root root  4096 2009-03-06 15:01 Docs
dr-x------ 1 root root  4096 2009-01-02 14:45 Flash Drive backup
dr-x------ 1 root root 32768 2009-06-17 01:13 Images
dr-x------ 1 root root  8192 2009-04-04 11:08 Isos
dr-x------ 1 root root     0 2008-07-23 20:38 Music
dr-x------ 1 root root     0 2009-05-30 12:55 RECYCLER
dr-x------ 1 root root  4096 2009-04-23 19:35 Roms
dr-x------ 1 root root  4096 2009-05-31 14:34 System Volume Information
dr-x------ 1 root root 32768 2009-06-26 10:21 To Be Sorted
dr-x------ 1 root root  4096 2009-01-22 22:25 Uni
dr-x------ 1 root root 28672 2009-05-23 15:21 Videos
dr-x------ 1 root root  4096 2009-05-23 01:44 Windows

root@sysresccd /dev % ls -l /media/NewHD 
total 0

All working, great.

Tried to run ntfsck on the hard drive and got this error

root@sysresccd /dev % ntfsck /dev/sdc1
Unsupported: replay_log()
Unsupported: check_volume()
Checking 44288 MFT records.
Unsupported cases found.

I've never used ntfsck before, dont know if I used it right but I assumed it was an issue with the HD.

Nevertheless I plowed on to using ddrescue.

root@sysresccd /dev % ddrescue -n /dev/sdc1 /dev/sdd1 log.txt


Press Ctrl-C to interrupt
rescued:   12000 MB,  errsize:       903 MB,  current rate:   0
   ipos:   11396 MB,   errors:       2096,    average rate:   204 kb/s
   opos:   11396 MB
^Cpying data...
Interrupted by user

One thing to note is that I forgot to copy the results from this ()so the values above are roughly what I remembered

That took just over 6 hours!

When I checked the contents of the new hard drive after cancelling an error message saying some data was corrupted but it did contain one folder from the original hard drive.

I realised I didnt take a note of the error message so I booted into my linux partition (openSuse 11.1 + gnome) and checked again which advised the whole hard drive was corrupted and to run a chkdsk in windows and then reboot twice.

I ran chkdsk in windows and it repaired so many errors and orphaned files that if I were to paste the output here I would break the internet so heres the end of it.

 488384000 KB total disk space.
 460757680 KB in 33001 files.
     20996 KB in 4497 indexes.
         0 KB in bad sectors.
    117812 KB in use by the system.
     65536 KB occupied by the log file.
  27487512 KB available on disk.

      4096 bytes in each allocation unit.
 122096000 total allocation units on disk.
   6871878 allocation units available on disk.

I checked the drive and it all looked ok at first, but all of the files which were corrupted originally were there, but didnt hold any data.

Do you think that running

ddrescue -dr3 /dev/sdc1 /dev/sdd1 log.txt

will be able to recover more of my data?

Also my new HD now shows as having a max size of 500gb instead of 1TB. I dont quite know why....

[BravoMike]

Ok finally backed up my fscking hard drive. I basically ran the command I mentioned at the top

cp -rv FIRST_DRIVE NEW_DRIVE 2>>failed_files.txt

I knew the files that were failing to copy would never be copied, since DD Rescue didnt work so I was happy with the files not copying but just being listed.

I ran the command, it took a few hours (dont know exactly how long) and by the end I had the contents of the first HD copied to the 2nd (less approx 40gb).

I know 40gb sounds like a lot but there was a few 4gb isos in there as well as DVD rips

Also wanted to say thanks to wempy because DD Rescue seems like a great tool. As does the System Restore cd software.

Cheers