Rsync File List Generation time is huge

[smandiga]

Hi,

We currently have setup RSync to backup the file system of our production server to a backup server. RSync takes really long and please see below the log details. As you notice here, the file list generation time is really huge and the transfer time is small. Is there any way that we can decrease the file list generation time.

We are using: rsync -avz -e 'ssh -c arcfour' source destination command line options.


Number of files: 21589453
Number of files transferred: 36715
Total file size: 550.92G bytes
Total transferred file size: 1.01G bytes
Literal data: 994.76M bytes
Matched data: 18.06M bytes
File list size: 899066429
File list generation time: 19921.798 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.01M
Total bytes received: 1.35G

sent 1.01M bytes received 1.35G bytes 42.20K bytes/sec
total size is 550.92G speedup is 408.81

[Irithori]

lsyncd or (if blocklevel is acceptable: ) DRBD might be suitable for you.

Other than that, setting noatime as a mount option helps a bit, but in general many little files are a pita.
You could try and kill the problem with hardware (lots of 15k spindles in raid10 or multiple SSDs) or divide the data.