remotely read file from last EOF

[fredo2906]

Hi Geeks,

I would like to get some idea on "how to read remote servers log files starting from the last EOF".

This, the best way possible because the file can be really big (couple of GB).
So recording the last line treated and re-scan the whole file may not be enough efficient.

The language of choice is Bash or Perl scripting.

Please let me know your thoughts.
Thanks a lot.
Fred.

[Eddy_Em]

I don't think bash will help you.

In C I would do simple: save last seekpos to other file.
For example [ http: //code.google .com/p/small-util/source/browse/parsesquid.c]here I used a file with marks (time / seek position) to make squid logs parcing more quickly.

In your case it's quite simpler: you need to save only one seek mark.

[fredo2906]

I don't think bash will help you.

In C I would do simple: save last seekpos to other file.
For example [ http: //code.google .com/p/small-util/source/browse/parsesquid.c]here I used a file with marks (time / seek position) to make squid logs parcing more quickly.

In your case it's quite simpler: you need to save only one seek mark.

I definitely understand that, but my files are stored remotely on linux servers, so getting the last EOF is a bit tricky I think.

[Eddy_Em]

You can use FUSE-sshfs to mount remote filesystem or directly run you scripts there.

If you have access to files only by http, use wget or curl (they allow to continue uploading).

[Kloschüssel]

You could definitly log into the remote server via ssh, reverse the file (tail -r) and scp the result to your local computer and while it downloads and grows in size you can examine it with tail.

[mactruck]

I have logs similar to that and when I need to read them I use the tail function.

tail -1000 <filename> | more

This will display that last 1000 lines of that log. You could also direct it to a tmp file.