Find the answer to your Linux question:
Results 1 to 2 of 2
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1

    Post Rsync File List Generation time is huge


    We currently have setup RSync to backup the file system of our production server to a backup server. RSync takes really long and please see below the log details. As you notice here, the file list generation time is really huge and the transfer time is small. Is there any way that we can decrease the file list generation time.

    We are using: rsync -avz -e 'ssh -c arcfour' source destination command line options.

    Number of files: 21589453
    Number of files transferred: 36715
    Total file size: 550.92G bytes
    Total transferred file size: 1.01G bytes
    Literal data: 994.76M bytes
    Matched data: 18.06M bytes
    File list size: 899066429
    File list generation time: 19921.798 seconds
    File list transfer time: 0.000 seconds
    Total bytes sent: 1.01M
    Total bytes received: 1.35G

    sent 1.01M bytes received 1.35G bytes 42.20K bytes/sec
    total size is 550.92G speedup is 408.81

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    lsyncd or (if blocklevel is acceptable: ) DRBD might be suitable for you.

    Other than that, setting noatime as a mount option helps a bit, but in general many little files are a pita.
    You could try and kill the problem with hardware (lots of 15k spindles in raid10 or multiple SSDs) or divide the data.
    You must always face the curtain with a bow.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts