Find the answer to your Linux question:
Results 1 to 4 of 4
I've got a folder with a very large number of files and subdirectories (it's full of maildir folders for my company actually) which has files that rank in the millions. ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Linux Newbie humbletech99's Avatar
    Join Date
    Nov 2005
    Posts
    225

    Syncing very large number of files to another server


    I've got a folder with a very large number of files and subdirectories (it's full of maildir folders for my company actually) which has files that rank in the millions. I need to replicate this to another server as a backup.

    The number of files is not the only consideration, the volume is quite significant, even over gigabit.

    I've tried using rsync to try to avoid re-copying a large volume of data by only taking the differences across to the backup server, but the extremely large number of files tends to hurt rsync and take forever while it's building up file lists.

    Does anybody have a better idea or another tool for replicating a directory structure with both large volume and a large number of files?
    The Human Equation:

    value(geeks) > value(mundanes)

  2. #2
    Linux Engineer khafa's Avatar
    Join Date
    Apr 2008
    Location
    Tokyo, Japan
    Posts
    858
    the problem with rsync is that it needs to scan.
    have you tried the cpio command(it does not scan)?

    if cpio can't solve the problem i would write a small script that divides the files and directories into many smaller directories and then use a script to rsync them one at a time.
    Linux and me it's a love story

  3. #3
    Linux Engineer Thrillhouse's Avatar
    Join Date
    Jun 2006
    Location
    Arlington, VA, USA
    Posts
    1,377
    I suppose you could try rdist although I believe rdist is intended to distribute to more than 2 servers.

    If this is an important enough job, it may be worth investing in a NAS.

  4. $spacer_open
    $spacer_close
  5. #4
    Linux Newbie humbletech99's Avatar
    Join Date
    Nov 2005
    Posts
    225
    I'll check out rdist.

    I'm not sure what the point of the NAS is considering it would suffer from the same problem as rsync to a remote server...
    The Human Equation:

    value(geeks) > value(mundanes)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •