Find the answer to your Linux question:
Results 1 to 9 of 9
I have a big file on server 1, each night i copy it on server 2 (lan). This operation is very slow. Can I transfer only differences of file?...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Jul 2009
    Posts
    1

    How to syncronize 2 big files


    I have a big file on server 1, each night i copy it on server 2 (lan).

    This operation is very slow.

    Can I transfer only differences of file?

  2. #2
    Super Moderator Roxoff's Avatar
    Join Date
    Aug 2005
    Location
    Nottingham, England
    Posts
    3,931
    The tool you require is called rsync. If this file is in a live filesystem and subject to frequent updates, then you may want to consider using LVM, which will allow you to take a 'live' snapshot of the filesystem while you sync the file.
    Linux user #126863 - see http://linuxcounter.net/

  3. #3
    Just Joined!
    Join Date
    Jul 2009
    Posts
    58
    Depending on the type of file that is being synced, you might want to add "-z" to compress the stream. If it's something that doesn't compress well than skip it as it will just add to the overhead.

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined!
    Join Date
    Jul 2009
    Location
    Brazil
    Posts
    6
    I think rsync is better.

    you can use:

    # rsync -alvgoupz /src 10.10.10.10:/dst

    Will be copied only files modified and sent with compression.

    You can see the man, for more details.


    Leandro.

  6. #5
    Just Joined!
    Join Date
    Sep 2007
    Location
    Lafayette, IN
    Posts
    83
    rsync, I believe, will copy the whole file if it has changed. Since we're talking about one large file, rsync won't help in this case. What might work better is a version control system like git or Subversion. Thoughts?

  7. #6
    Linux Newbie
    Join Date
    Sep 2004
    Location
    UK
    Posts
    161
    CVS, subversion probably store the whole file when dealing with binary files rather than diffs.

    Have a look at xdelta.
    In a world without walls and fences, who needs Windows and Gates?

  8. #7
    Linux Guru
    Join Date
    Nov 2007
    Posts
    1,763
    From the rsync man page:

    Rsync is a fast and extraordinarily versatile file copying tool. It
    can copy locally, to/from another host over any remote shell, or
    to/from a remote rsync daemon. It offers a large number of options
    that control every aspect of its behavior and permit very flexible
    specification of the set of files to be copied. It is famous for its
    delta-transfer algorithm, which reduces the amount of data sent over
    the network by sending only the differences between the source files
    and the existing files in the destination.
    Rsync is widely used for
    backups and mirroring and as an improved copy command for everyday use.

  9. #8
    Just Joined!
    Join Date
    Jul 2009
    Posts
    58
    HROAdmin26 is right, rsync will only transfer what was changed:

    Some data:

    [primary] # dd if=/dev/zero of=/tmp/testFile count=0 seek=1024
    [primary] # ls /tmp/testFile
    -rw-r--r-- 1 root root 524290 Jul 23 15:31 testFile

    [backup] # rsync -alvz ...
    sent 63143 bytes received 81876 bytes 41434.00 bytes/sec
    total size is 1139575829 speedup is 7858.11

    [primary] # echo "0" >> /tmp/testFile
    [backup] # rsync -alvz ...
    sent 4416 bytes received 78886 bytes 55534.67 bytes/sec
    total size is 1139575831 speedup is 13680.05

    4416 bytes transferred instead of 63143+ on a 524290+ size file.

  10. #9
    Just Joined!
    Join Date
    Sep 2007
    Location
    Lafayette, IN
    Posts
    83
    HRO, I guess I missed that in the man page. Thanks for pointing it out.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •