Results 1 to 9 of 9
Enjoy an ad free experience by logging in. Not a member yet? Register.
- Join Date
- Nov 2008
QUICKLY copying files between two servers
I have two servers in the same rack. Nightly, I want to copy the contents of directory /xyz and its subdirectories from Server-1 to Server-2. The contents of /xzy is approximately 100GB. Both rsync and scp are slow.
What's the fastest method of copying /xzy from Server-1 to Server-2?
Try a nfs ... at least in my experience scp reoughly twice as fast as ftp.
rsync is usually as fast as it gets.
Can you give a bit more info?
- is it everyday 100Gbyte additional data?
- or 100GByte in total?
- how many files are we talking?
- how are the servers connected? crossover, same switch, or stacked switches?
- What is the theoretical maximum network throughput between them?
- What is the theoretical maximum read IO of server-1 and the theoretical maximum write IO of server-2?
- Is server-1 and/or server-2 already under high load at the time the copy process starts?You must always face the curtain with a bow.
if the files themselves are not constantly changing rsync would be your best option, perhaps you are not using the proper options when setting it up, or there maybe pieces of the puzzle that i am missing here. of course if its something like tar archives, it won't really help as it will have to copy over the entire file all over again.
scp will always take longer, because of the overhead involved with encryption.
i've seen a lot of vendors offer up this 'bit-by-bit' backup or something along those lines. what it does is looks at the file and modifies only the portions of it that have changed. haven't really seen anything along the lines of free though that will do it.linux user # 503963
But it's still a network copy.
At my home with a 100Mb lan, I can only get something like 2MB/s with nfs.
But this means just what it means , maybe others can get more.
As the OP is absent, and we don't really know what he wants, just between us, I'm now using subversion to share some things between machines. I'm very happy with it, it was a all new world for me. And Tortoise for windows allows some of my "non-linux " co-workers to use it too.
Not saying svn is fast. But after the initial load, it only gets changes and additions.
is there any chance of gigbit ethernet?linux user # 503963