How to syncronize 2 big files

[xandre]

I have a big file on server 1, each night i copy it on server 2 (lan).

This operation is very slow.

Can I transfer only differences of file?

[Roxoff]

The tool you require is called rsync. If this file is in a live filesystem and subject to frequent updates, then you may want to consider using LVM, which will allow you to take a 'live' snapshot of the filesystem while you sync the file.

[aram535]

Depending on the type of file that is being synced, you might want to add "-z" to compress the stream. If it's something that doesn't compress well than skip it as it will just add to the overhead.

[lcoronato]

I think rsync is better.

you can use:

# rsync -alvgoupz /src 10.10.10.10:/dst

Will be copied only files modified and sent with compression.

You can see the man, for more details.


Leandro.

[Ben Cotton]

rsync, I believe, will copy the whole file if it has changed. Since we're talking about one large file, rsync won't help in this case. What might work better is a version control system like git or Subversion. Thoughts?

[blinky]

CVS, subversion probably store the whole file when dealing with binary files rather than diffs.

Have a look at xdelta.

[HROAdmin26]

From the rsync man page:

Rsync is a fast and extraordinarily versatile file copying tool. It
can copy locally, to/from another host over any remote shell, or
to/from a remote rsync daemon. It offers a large number of options
that control every aspect of its behavior and permit very flexible
specification of the set of files to be copied. It is famous for its
delta-transfer algorithm, which reduces the amount of data sent over
the network by sending only the differences between the source files
and the existing files in the destination. Rsync is widely used for
backups and mirroring and as an improved copy command for everyday use.

[aram535]

HROAdmin26 is right, rsync will only transfer what was changed:

Some data:

[primary] # dd if=/dev/zero of=/tmp/testFile count=0 seek=1024
[primary] # ls /tmp/testFile
-rw-r--r-- 1 root root 524290 Jul 23 15:31 testFile

[backup] # rsync -alvz ...
sent 63143 bytes received 81876 bytes 41434.00 bytes/sec
total size is 1139575829 speedup is 7858.11

[primary] # echo "0" >> /tmp/testFile
[backup] # rsync -alvz ...
sent 4416 bytes received 78886 bytes 55534.67 bytes/sec
total size is 1139575831 speedup is 13680.05

4416 bytes transferred instead of 63143+ on a 524290+ size file.

[Ben Cotton]

HRO, I guess I missed that in the man page. Thanks for pointing it out.