Find the answer to your Linux question:
Results 1 to 4 of 4
Not sure if this is the right place to post... I need to move (copy) a large quantity of files from one disc to another, the size of the data ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Linux Enthusiast
    Join Date
    Aug 2006
    Location
    Portsmouth, UK
    Posts
    539

    Copying a large quantity of files from one disc to another (1.5Million)


    Not sure if this is the right place to post...

    I need to move (copy) a large quantity of files from one disc to another, the size of the data isn't a problem (36Gb) but the fact there are about 1.5million files to handle!

    I tried to do a straight copy using cp and a list file names generated by one of our databases but it only managed to copy 51,000 in almost 3Hours.

    I did a test using tar, which managed to archive 17Gb on a similar filesytem in about 34 minutes (no compression).

    The question is, given the number of files how long approximately will it take to untar the 1.5million files as the killer is going to be file creation time.

    FYI the destination disc is a 300Gb SCSI U320 10K

    Any help in getting a ball pack time frame would be greatly appreciated.

  2. #2
    Super Moderator devils casper's Avatar
    Join Date
    Jun 2006
    Location
    Chandigarh, India
    Posts
    24,729
    hi matonb !!

    try PartImage... actually its used to create image of whole OS .. and its compression speed is very good too... create an image of data.....

    <=== { casper } ===>
    It is amazing what you can accomplish if you do not care who gets the credit.
    New Users: Read This First

  3. #3
    Linux Guru bigtomrodney's Avatar
    Join Date
    Nov 2004
    Location
    Ireland
    Posts
    6,133
    Untarring will be a similar time to tarring. The best step is as you said not to compress, but disk imaging could be the solution. One of the largest overheads in transfers like this is the number of files not their size so a straight image if at all possible could work out nicely.

  4. $spacer_open
    $spacer_close
  5. #4
    Linux Enthusiast
    Join Date
    Aug 2006
    Location
    Portsmouth, UK
    Posts
    539
    Thanks for the suggestions guys,

    We've opted to use rsync as a test showed that it didn't impact on performance as far as our users were concerned.

    As you pointed out bigtomrodney, that time in this case is taken creating files not copying the data.

    I don't believe that untarring would be as quick as tarring the files simply due to the quantity of files (but I've not tested that )

    Nice thing with the rsync option is that we can stop and start it when we like...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •