Results 1 to 2 of 2
Enjoy an ad free experience by logging in. Not a member yet? Register.
- Join Date
- Feb 2012
Overcoming lftp mirror starvation
I'm new to this forum so I'm sorry if this is the wrong place to post this question. Please redirect me if needed.
I have recently set up lftp to mirror directories between two of my servers. It works by having an FTP server running on one server and lftp running in a loop from the client to mirror the directories.
I'm running the lftp mirror option with --continue and --parallel=5 which works really well, except I discovered an unexpected starvation issue with this setup. The lftp process sometimes runs into very big files which take hours to download, and during this time other files are added to the directory that need to be mirrored. Although not all the 5 threads are transferring files at the moment, the mirror operation won't detect the new files until the current operation has completed and lftp is run again. Therefore one big file can delay by hours the transfer of many small files.
The only solution I have in mind is to detect changes from a different process, and when such changes are detected stop the lftp session and run it again. This is, however, far from optimal in resource usage.
Is there any solution I'm missing? Perhaps there's a parameter for lftp to rescan files for the mirror queue while running?
I'd welcome any new ideas.
Thanks in advance!
Hmm, I would suggest to use lsyncd instead of ftp and lftp.
a) lsyncd reacts to changes to the filesystem, not on a fixed time schedule:
The ftp/lftp approach means constant load,
while lsyncd only triggers actions if there is actually something to sync.
b) The protocol is rsync not ftp.
I generally dislike ftp for its use of two connections/ports.
Using the rsync protocoll enables you to resume on big files and/or transmit only changed blocks.You must always face the curtain with a bow.