Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 12
I am looking for a simple and fast way to copy all data from one drive to another. This should be simple.... Right So say I have /hd1 and /hd2 ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jul 2004
    Posts
    85

    Backup Solutions? CP too slow.


    I am looking for a simple and fast way to copy all data from one drive to another. This should be simple.... Right So say I have /hd1 and /hd2 mounted. To copy all filed from /hd1 to /hd2 I use the following command:
    Code:
    cp /hd1/* /hd2/ -pr
    And it does exactly what I need, the only problem is it takes for ever to finish (about 9hours for 50GB). I have 1GB of ram and 4x450Mhz CPUs. The actual copy rate is 7 to 10 MB/s (timed for a single file). However the disk is full of thousands of really tiny files, and that's what bogs down the copy. When copying the files the cpu raves up all the way up to full power and stays like that.

    Are there any other ways to do the same thing. I looked into some backup packages but they are too complex for my needs, I basically need a CP command, but one that works fast..

    Any Ideas,

    - Bogdan

  2. #2
    Linux Enthusiast puntmuts's Avatar
    Join Date
    Dec 2004
    Location
    Republic Banana
    Posts
    562
    Software RAID may be a solution
    I\'m so tired .....
    #200472

  3. #3
    Just Joined!
    Join Date
    Jul 2004
    Posts
    85
    Well actually I already have a Hard Ware RAID Setup This would be a backup to my backup /hd2 in that example would be my IDE backup drive that is dismounted and shutdown in between backups to help preserve it. It is refreshed only once a week just in case the rest of my drives all fail in the same time. Sounds over protective, ain't it

    - Bogdan

  4. #4
    Linux Enthusiast puntmuts's Avatar
    Join Date
    Dec 2004
    Location
    Republic Banana
    Posts
    562
    You could tar and gzip it first before copying. That way you will have less data to copy to the other disc. Don't know what's slowing down in your setup.
    I\'m so tired .....
    #200472

  5. #5
    Just Joined!
    Join Date
    Jul 2004
    Posts
    85
    well, gzing 50GB will take quite a bit of time as well, in the end I doubt that it would make it any faster. The problem looks to me in the preparing for copy and post copy things that cp does. It just take to dam long to do what id does. Alternatively is there a way to clone the actual row data. So instead of coping the files through the fs layer I would just clone the bits from one partition on to another?

    - Bogdan

  6. #6
    Just Joined!
    Join Date
    Jul 2004
    Posts
    85
    I tried dd as well, with no luck The fastest I get is 3M/s. This is strange because both disks can read and write at least at 15MB/s, actually faster, but the fastest I was able to benchmark it at was 15. When using dd the cpu loads up to almost 70% slowing down the rest of the server. Now, should a simple clone be so CPU intensive? One disk is SCSI and the other is IDE, both about same speed. Anyway, this really bugs me. Any ideas?

    - Bogdan

  7. #7
    Linux Engineer
    Join Date
    Mar 2005
    Posts
    1,431
    Backup? Maybe not what you're after, but you can take a 100% copy of the harddrive/partition with DD

  8. #8
    Linux Newbie
    Join Date
    Apr 2005
    Location
    Charlottesville, VA
    Posts
    175
    You could try (when in /hd1)
    Code:
    tar -c *|tar -xpC /hd2
    That would tar and untar on the fly, resulting in what I think would be the same as moving one large file. Not sure about that, though, and now that I'm writing it, it's not making anymore sense to me what I was thinking. Better post anyway, just in case. Sorry for the rant.
    ---sxeraverx---
    Linux without a C Compiler is like eating Spaghetti with your mouth sewn shut. It just doesn't make sense.

  9. #9
    scm
    scm is offline
    Linux Engineer
    Join Date
    Feb 2005
    Posts
    1,044
    Quote Originally Posted by x0054
    I tried dd as well, with no luck The fastest I get is 3M/s. This is strange because both disks can read and write at least at 15MB/s, actually faster, but the fastest I was able to benchmark it at was 15. When using dd the cpu loads up to almost 70% slowing down the rest of the server. Now, should a simple clone be so CPU intensive? One disk is SCSI and the other is IDE, both about same speed. Anyway, this really bugs me. Any ideas?

    - Bogdan
    What block size did you use? If you do it in 512 byte blocks (the default, I believe) you'll fall foul of disk latency. Try bs=1024K for starters.

  10. #10
    Linux Guru kkubasik's Avatar
    Join Date
    Mar 2004
    Location
    Lat: 39:03:51N Lon: 77:14:37W
    Posts
    2,396
    you could try

    cat /dev/hda > /mnt/backup


    or something of the like, i use

    cat /dev/cdrom > /home/qub333/cd.iso

    to make iso's all the time, faster than cp'ing the files, youll have to fiddle w/ how to pipe it to a writable source, but see what works.
    Avoid the Gates of Hell. Use Linux
    A Penny for your Thoughts

    Formerly Known as qub333

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •