QUICKLY copying files between two servers

[rsync]

Scenario:

I have two servers in the same rack. Nightly, I want to copy the contents of directory /xyz and its subdirectories from Server-1 to Server-2. The contents of /xzy is approximately 100GB. Both rsync and scp are slow.

Question:

What's the fastest method of copying /xzy from Server-1 to Server-2?

[shtromm]

Try a nfs ... at least in my experience scp reoughly twice as fast as ftp.

[Irithori]

rsync is usually as fast as it gets.

Can you give a bit more info?
- is it everyday 100Gbyte additional data?
- or 100GByte in total?
- how many files are we talking?
- how are the servers connected? crossover, same switch, or stacked switches?
- What is the theoretical maximum network throughput between them?
- What is the theoretical maximum read IO of server-1 and the theoretical maximum write IO of server-2?
- Is server-1 and/or server-2 already under high load at the time the copy process starts?

[ptkobe]

Try a nfs ... at least in my experience scp reoughly twice as fast as ftp.

nfs doesn't copy files, it just allows access over the network, mounting and using them as if they were local, but they are not.

Probably the OP knows that, but if the idea is somehow a backup, nfs won't do it.

[scathefire]

if the files themselves are not constantly changing rsync would be your best option, perhaps you are not using the proper options when setting it up, or there maybe pieces of the puzzle that i am missing here. of course if its something like tar archives, it won't really help as it will have to copy over the entire file all over again.

scp will always take longer, because of the overhead involved with encryption.

i've seen a lot of vendors offer up this 'bit-by-bit' backup or something along those lines. what it does is looks at the file and modifies only the portions of it that have changed. haven't really seen anything along the lines of free though that will do it.

[shtromm]

nfs doesn't copy files, it just allows access over the network, mounting and using them as if they were local, but they are not.

Yes, but than, having access to the remote FS locally, a simple "cp -r" would do.

[shtromm]

scp will always take longer, because of the overhead involved with encryption.

I thought this too, until I run some tests. It seems that with modern machines the overhead does not matter anymore.

[ptkobe]

Yes, but than, having access to the remote FS locally, a simple "cp -r" would do.

Thanks, shtromm. Very true, right.

But it's still a network copy.
At my home with a 100Mb lan, I can only get something like 2MB/s with nfs.
But this means just what it means , maybe others can get more.

As the OP is absent, and we don't really know what he wants, just between us, I'm now using subversion to share some things between machines. I'm very happy with it, it was a all new world for me. And Tortoise for windows allows some of my "non-linux " co-workers to use it too.

Not saying svn is fast. But after the initial load, it only gets changes and additions.

Best regards
Luis

[scathefire]

is there any chance of gigbit ethernet?