rdiff-backup in-use log files

[flashmantom]

Hi there,

Probably an easy one but can't seem to find it in any searches. Basically I want to back up my log files to another server via rdiff-backup. works really well for all the old rotated log files, but the current files won't sync, I'm guessing because they're in use. Duh. I'd rather not disable logging or shut off services, so is there any way to copy the current apache and mail logs while they're still in use? The term "Shadow copy" comes to mind but I think that was a windows thing.

[Irithori]

In unix, the approach to files is more optimistic, ie: you will never get a "file is in use by another process" error.
Consequently, there is no need for a "shadow copy", at least for this purpose.


On the other hand: you need to make sure or be sure, that the file you access is consistent.
As you can imagine, a logfile is always open and in use.

You can still watch it via e.g. "tail",
you can still copy it
or even write to it.

But the copy will probably miss some lines.

The standard approach towards (apache) logfiles is
1) to use s/th like cronolog in your apache conf for logrotating purposes
2) and then rsyslog to follow the logfiles and send them to a central rsyslog server

[flashmantom]

With regards to file consistency, I figured with rdiff backup, each file may not be 100% at any one time, but each time it's synced the previous missed lines would be caught up. So it would only ever be a few lines behind.

You say though that linux should have no problems copying files while in use, but that's exactly the problem I'm having. All the old rotated logs are coming through fine, but the current log files aren't coming through the rdiff.

Pretty basic command I'm using. Maybe there's some magic parameter I'm missing?

/usr/bin/rdiff-backup -v 2 SOURCESERVER::/var/log /DESTDIR/settings/logs

[Irithori]

Probably because rdiff-backup realizes that the file changed while copying and therefore discards it.
Perfectly reasonable logic for a backup application.

The easy approach is to restrict the backup to rotated files.

The slightly more sophisticated is the one I outlined above:
Utilize rsyslog, which is a syslog implementation with quite a lot of usefull features like templates, RELP, multiple input and output modules, etc