Overcoming lftp mirror starvation
I'm new to this forum so I'm sorry if this is the wrong place to post this question. Please redirect me if needed.
I have recently set up lftp to mirror directories between two of my servers. It works by having an FTP server running on one server and lftp running in a loop from the client to mirror the directories.
I'm running the lftp mirror option with --continue and --parallel=5 which works really well, except I discovered an unexpected starvation issue with this setup. The lftp process sometimes runs into very big files which take hours to download, and during this time other files are added to the directory that need to be mirrored. Although not all the 5 threads are transferring files at the moment, the mirror operation won't detect the new files until the current operation has completed and lftp is run again. Therefore one big file can delay by hours the transfer of many small files.
The only solution I have in mind is to detect changes from a different process, and when such changes are detected stop the lftp session and run it again. This is, however, far from optimal in resource usage.
Is there any solution I'm missing? Perhaps there's a parameter for lftp to rescan files for the mirror queue while running?
I'd welcome any new ideas.
Thanks in advance!