Syncing very large number of files to another server

[humbletech99]

I've got a folder with a very large number of files and subdirectories (it's full of maildir folders for my company actually) which has files that rank in the millions. I need to replicate this to another server as a backup.

The number of files is not the only consideration, the volume is quite significant, even over gigabit.

I've tried using rsync to try to avoid re-copying a large volume of data by only taking the differences across to the backup server, but the extremely large number of files tends to hurt rsync and take forever while it's building up file lists.

Does anybody have a better idea or another tool for replicating a directory structure with both large volume and a large number of files?

[khafa]

the problem with rsync is that it needs to scan.
have you tried the cpio command(it does not scan)?

if cpio can't solve the problem i would write a small script that divides the files and directories into many smaller directories and then use a script to rsync them one at a time.

[Thrillhouse]

I suppose you could try rdist although I believe rdist is intended to distribute to more than 2 servers.

If this is an important enough job, it may be worth investing in a NAS.

[humbletech99]

I'll check out rdist.

I'm not sure what the point of the NAS is considering it would suffer from the same problem as rsync to a remote server...