Backup Solutions? CP too slow.

[x0054]

I am looking for a simple and fast way to copy all data from one drive to another. This should be simple.... Right So say I have /hd1 and /hd2 mounted. To copy all filed from /hd1 to /hd2 I use the following command:

cp /hd1/* /hd2/ -pr

And it does exactly what I need, the only problem is it takes for ever to finish (about 9hours for 50GB). I have 1GB of ram and 4x450Mhz CPUs. The actual copy rate is 7 to 10 MB/s (timed for a single file). However the disk is full of thousands of really tiny files, and that's what bogs down the copy. When copying the files the cpu raves up all the way up to full power and stays like that.

Are there any other ways to do the same thing. I looked into some backup packages but they are too complex for my needs, I basically need a CP command, but one that works fast..

Any Ideas,

- Bogdan

[puntmuts]

Software RAID may be a solution

[x0054]

Well actually I already have a Hard Ware RAID Setup This would be a backup to my backup /hd2 in that example would be my IDE backup drive that is dismounted and shutdown in between backups to help preserve it. It is refreshed only once a week just in case the rest of my drives all fail in the same time. Sounds over protective, ain't it

- Bogdan

[puntmuts]

You could tar and gzip it first before copying. That way you will have less data to copy to the other disc. Don't know what's slowing down in your setup.

[x0054]

well, gzing 50GB will take quite a bit of time as well, in the end I doubt that it would make it any faster. The problem looks to me in the preparing for copy and post copy things that cp does. It just take to dam long to do what id does. Alternatively is there a way to clone the actual row data. So instead of coping the files through the fs layer I would just clone the bits from one partition on to another?

- Bogdan

[x0054]

I tried dd as well, with no luck The fastest I get is 3M/s. This is strange because both disks can read and write at least at 15MB/s, actually faster, but the fastest I was able to benchmark it at was 15. When using dd the cpu loads up to almost 70% slowing down the rest of the server. Now, should a simple clone be so CPU intensive? One disk is SCSI and the other is IDE, both about same speed. Anyway, this really bugs me. Any ideas?

- Bogdan

[jaboua]

Backup? Maybe not what you're after, but you can take a 100% copy of the harddrive/partition with DD

[sxeraverx]

You could try (when in /hd1)

tar -c *|tar -xpC /hd2

That would tar and untar on the fly, resulting in what I think would be the same as moving one large file. Not sure about that, though, and now that I'm writing it, it's not making anymore sense to me what I was thinking. Better post anyway, just in case. Sorry for the rant.

[scm]

I tried dd as well, with no luck The fastest I get is 3M/s. This is strange because both disks can read and write at least at 15MB/s, actually faster, but the fastest I was able to benchmark it at was 15. When using dd the cpu loads up to almost 70% slowing down the rest of the server. Now, should a simple clone be so CPU intensive? One disk is SCSI and the other is IDE, both about same speed. Anyway, this really bugs me. Any ideas?

- Bogdan

What block size did you use? If you do it in 512 byte blocks (the default, I believe) you'll fall foul of disk latency. Try bs=1024K for starters.

[kkubasik]

you could try

cat /dev/hda > /mnt/backup


or something of the like, i use

cat /dev/cdrom > /home/qub333/cd.iso

to make iso's all the time, faster than cp'ing the files, youll have to fiddle w/ how to pipe it to a writable source, but see what works.