Find the answer to your Linux question:
Results 1 to 9 of 9
Scenario: I have two servers in the same rack. Nightly, I want to copy the contents of directory /xyz and its subdirectories from Server-1 to Server-2. The contents of /xzy ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Nov 2008
    Posts
    3

    QUICKLY copying files between two servers


    Scenario:

    I have two servers in the same rack. Nightly, I want to copy the contents of directory /xyz and its subdirectories from Server-1 to Server-2. The contents of /xzy is approximately 100GB. Both rsync and scp are slow.

    Question:

    What's the fastest method of copying /xzy from Server-1 to Server-2?

  2. #2
    Just Joined! shtromm's Avatar
    Join Date
    Jun 2010
    Posts
    31
    Try a nfs ... at least in my experience scp reoughly twice as fast as ftp.

  3. #3
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,356
    rsync is usually as fast as it gets.

    Can you give a bit more info?
    - is it everyday 100Gbyte additional data?
    - or 100GByte in total?
    - how many files are we talking?
    - how are the servers connected? crossover, same switch, or stacked switches?
    - What is the theoretical maximum network throughput between them?
    - What is the theoretical maximum read IO of server-1 and the theoretical maximum write IO of server-2?
    - Is server-1 and/or server-2 already under high load at the time the copy process starts?
    You must always face the curtain with a bow.

  4. #4
    Linux User ptkobe's Avatar
    Join Date
    Feb 2008
    Location
    Torres Vedras, PT
    Posts
    276
    Quote Originally Posted by shtromm View Post
    Try a nfs ... at least in my experience scp reoughly twice as fast as ftp.
    nfs doesn't copy files, it just allows access over the network, mounting and using them as if they were local, but they are not.

    Probably the OP knows that, but if the idea is somehow a backup, nfs won't do it.

  5. #5
    Linux Enthusiast scathefire's Avatar
    Join Date
    Jan 2010
    Location
    Western Kentucky
    Posts
    626
    if the files themselves are not constantly changing rsync would be your best option, perhaps you are not using the proper options when setting it up, or there maybe pieces of the puzzle that i am missing here. of course if its something like tar archives, it won't really help as it will have to copy over the entire file all over again.

    scp will always take longer, because of the overhead involved with encryption.

    i've seen a lot of vendors offer up this 'bit-by-bit' backup or something along those lines. what it does is looks at the file and modifies only the portions of it that have changed. haven't really seen anything along the lines of free though that will do it.
    linux user # 503963

  6. #6
    Just Joined! shtromm's Avatar
    Join Date
    Jun 2010
    Posts
    31
    Quote Originally Posted by ptkobe View Post
    nfs doesn't copy files, it just allows access over the network, mounting and using them as if they were local, but they are not.
    Yes, but than, having access to the remote FS locally, a simple "cp -r" would do.

  7. #7
    Just Joined! shtromm's Avatar
    Join Date
    Jun 2010
    Posts
    31
    Quote Originally Posted by scathefire View Post
    scp will always take longer, because of the overhead involved with encryption.
    I thought this too, until I run some tests. It seems that with modern machines the overhead does not matter anymore.

  8. #8
    Linux User ptkobe's Avatar
    Join Date
    Feb 2008
    Location
    Torres Vedras, PT
    Posts
    276
    Quote Originally Posted by shtromm View Post
    Yes, but than, having access to the remote FS locally, a simple "cp -r" would do.
    Thanks, shtromm. Very true, right.

    But it's still a network copy.
    At my home with a 100Mb lan, I can only get something like 2MB/s with nfs.
    But this means just what it means , maybe others can get more.

    As the OP is absent, and we don't really know what he wants, just between us, I'm now using subversion to share some things between machines. I'm very happy with it, it was a all new world for me. And Tortoise for windows allows some of my "non-linux " co-workers to use it too.

    Not saying svn is fast. But after the initial load, it only gets changes and additions.

    Best regards
    Luis

  9. #9
    Linux Enthusiast scathefire's Avatar
    Join Date
    Jan 2010
    Location
    Western Kentucky
    Posts
    626
    is there any chance of gigbit ethernet?
    linux user # 503963

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •