Backup a hard-drive through the CLI.

Printable View

07-01-2009
BravoMike

Backup a hard-drive through the CLI.

Dont quite know if this is the right area to post this in but here it goes.

Lately my external 500gb hard drive has been having some read write errors on certain files. If this happens when I'm using Windows then I have to close the program trying to access it and unmount and remount the hard drive to get it working again.

If this happens in Linux I just get a read-write error which is fine because its just a case of clicking "ok".

I just bough a new external hard drive and I am looking for a command which will copy all the files on one hard drive to other while outputting a list of all files which failed to copy (and obviously save me from clicking OK each time).

Would the following command do what I want?

Code:

cp -r FIRST_DRIVE NEW_DRIVE 2>>failed_files

Where FIRST_DRIVE is the root of the old drive and NEW_DRIVE is (unsurprisingly) the new hard drives root directory.
07-01-2009
wempy

GNU's ddrescue at gnu.org (google ddrescue) may be a better tool for this. If you have a knoppix cd then a simple procedure to clone your failing drive onto the new one might be (example from ddrescue info pages):

1. install both drives in the machine (preferably both as the master drive on different channels of the HDD controller)
2. boot knoppix to a command line (knoppix 2 - at the isolinux prompt)
3. e2fsck /dev/hda(n) - fsck the failing partition(s) first using the correct fsck for your filesystem types.
4. run ddrescue "ddrescue [OPTIONS] INFILE OUTFILE [LOGFILE]"
both INFILE and OUTFILE can be device nodes rather than files

Example 1: Rescue a whole disc with two ext2 partitions in /dev/hda to
/dev/hdb
Note: you do not need to partition /dev/hdb beforehand.

ddrescue -n /dev/hda /dev/hdb logfile
ddrescue -dr3 /dev/hda /dev/hdb logfile
fdisk /dev/hdb
e2fsck -v -f /dev/hdb1
e2fsck -v -f /dev/hdb2

Your distribution may already have ddrescue installed - try info ddrescue at a prompt. There is another tool called dd_rescue which does a similiar job, though I have found that the GNU ddrescue to be more capable, and recovers from errors better.
07-02-2009
BravoMike

I did a bit of research into ddrescue and it seemed the best tool for the job. I found it in a great set of applications built into a stripped down version of linux called System Restore CD, dont let the name fool you. I installed it on my flash drive.

So I booted into SysRes and got started.

First thing to do was mount the drives

Code:

root@sysresccd /root % cd /dev root@sysresccd /dev % ls -l sd* brw-r----- 1 root disk 8, 0 2009-07-02 00:28 sda brw-r----- 1 root disk 8, 1 2009-07-02 00:28 sda1 brw-r----- 1 root disk 8, 2 2009-07-02 00:28 sda2 brw-r----- 1 root disk 8, 5 2009-07-02 00:28 sda5 brw-r----- 1 root disk 8, 6 2009-07-02 00:28 sda6 brw-r----- 1 root disk 8, 7 2009-07-02 00:28 sda7 brw-r----- 1 root disk 8, 16 2009-07-02 00:28 sdb brw-r----- 1 root disk 8, 17 2009-07-02 00:28 sdb1 brw-r----- 1 root disk 8, 32 2009-07-02 00:31 sdc brw-r----- 1 root disk 8, 33 2009-07-02 00:31 sdc1 root@sysresccd /dev % mount /dev/sdc1 /media/BrokenHD root@sysresccd /dev % mount /dev/sdd1 /media/NewHD

Then check that they had mounted ok by checking the contents of each.

Code:

root@sysresccd /dev % ls -l /media/BrokenHD total 128 dr-x------ 1 root root 4096 2009-04-04 23:14 Apps dr-x------ 1 root root 0 2009-04-15 11:29 BK dr-x------ 1 root root 0 2009-05-31 14:40 Conor dr-x------ 1 root root 4096 2009-03-06 15:01 Docs dr-x------ 1 root root 4096 2009-01-02 14:45 Flash Drive backup dr-x------ 1 root root 32768 2009-06-17 01:13 Images dr-x------ 1 root root 8192 2009-04-04 11:08 Isos dr-x------ 1 root root 0 2008-07-23 20:38 Music dr-x------ 1 root root 0 2009-05-30 12:55 RECYCLER dr-x------ 1 root root 4096 2009-04-23 19:35 Roms dr-x------ 1 root root 4096 2009-05-31 14:34 System Volume Information dr-x------ 1 root root 32768 2009-06-26 10:21 To Be Sorted dr-x------ 1 root root 4096 2009-01-22 22:25 Uni dr-x------ 1 root root 28672 2009-05-23 15:21 Videos dr-x------ 1 root root 4096 2009-05-23 01:44 Windows root@sysresccd /dev % ls -l /media/NewHD total 0

All working, great.

Tried to run ntfsck on the hard drive and got this error

Code:

root@sysresccd /dev % ntfsck /dev/sdc1 Unsupported: replay_log() Unsupported: check_volume() Checking 44288 MFT records. Unsupported cases found.

I've never used ntfsck before, dont know if I used it right but I assumed it was an issue with the HD.

Nevertheless I plowed on to using ddrescue.

Code:

root@sysresccd /dev % ddrescue -n /dev/sdc1 /dev/sdd1 log.txt Press Ctrl-C to interrupt rescued: 12000 MB, errsize: 903 MB, current rate: 0 ipos: 11396 MB, errors: 2096, average rate: 204 kb/s opos: 11396 MB ^Cpying data... Interrupted by user

One thing to note is that I forgot to copy the results from this (:oops:)so the values above are roughly what I remembered :rolleyes:

That took just over 6 hours!

When I checked the contents of the new hard drive after cancelling an error message saying some data was corrupted but it did contain one folder from the original hard drive.

I realised I didnt take a note of the error message so I booted into my linux partition (openSuse 11.1 + gnome) and checked again which advised the whole hard drive was corrupted and to run a chkdsk in windows and then reboot twice.

I ran chkdsk in windows and it repaired so many errors and orphaned files that if I were to paste the output here I would break the internet so heres the end of it.

Code:

488384000 KB total disk space. 460757680 KB in 33001 files. 20996 KB in 4497 indexes. 0 KB in bad sectors. 117812 KB in use by the system. 65536 KB occupied by the log file. 27487512 KB available on disk. 4096 bytes in each allocation unit. 122096000 total allocation units on disk. 6871878 allocation units available on disk.

I checked the drive and it all looked ok at first, but all of the files which were corrupted originally were there, but didnt hold any data.

Do you think that running

Code:

ddrescue -dr3 /dev/sdc1 /dev/sdd1 log.txt

will be able to recover more of my data?

Also my new HD now shows as having a max size of 500gb instead of 1TB. I dont quite know why....
07-13-2009
BravoMike

Ok finally backed up my fscking hard drive. I basically ran the command I mentioned at the top

Code:

cp -rv FIRST_DRIVE NEW_DRIVE 2>>failed_files.txt

I knew the files that were failing to copy would never be copied, since DD Rescue didnt work so I was happy with the files not copying but just being listed.

I ran the command, it took a few hours (dont know exactly how long) and by the end I had the contents of the first HD copied to the 2nd (less approx 40gb).

I know 40gb sounds like a lot but there was a few 4gb isos in there as well as DVD rips :rolleyes:

Also wanted to say thanks to wempy because DD Rescue seems like a great tool. As does the System Restore cd software.

Cheers :D