Technique for parsing a log file

[BecTech]

the more research I do the more confusing it gets..

I have a log file on several linux clients that lists results for the diagnostics that run on machines they are connected to daily (not always daily but close enough).
The log file is appended to so I only want to tail it for a certain amount of days from the current date every time I run it.


and example of the data format using fake data:


Diag sw begin.
Mon Oct 15 08:00:00 EDT 2007
PASS: (a) Bootup Test
PASS: (b) server start
FAIL: (g) telnet server
.
.
.
Diag sw end.
Mon Oct 15 08:10:00 EDT 2007


Then that same format repeats through the file..
the data I'm concerned with is always 'sandwiched' between the "Diag sw begin." and "Diag sw end." lines so I thought that may be good starting point.

I was trying to just use awk to pull the information..
For instance, if I want to find out what failed I was trying

awk '/begin|EDT|FAIL|end/'

to get just those lines but I can't figure out how to just pull the lines for a certain time period... (i.e. current date-7 days)

If I can get the format right for one log file I wanted to go the extra step and maybe use a shell script to ftp the log files from all 4 machines and put them into a tmp directory so I can then compile all that data into one output file..

so my questions:

1. how would I parse the data for a certain time period ? is awk the best way to go ?

2. if I can get the output from each machine right, what is the best way to compile the data into one result file.. not only grouping them together but also automating the ftp...

my goal is to have a shell script that I can run from a command line whenever I want to see the diagnostics results for all the machines at once.. (rather than telnet'ing to each one separately and tail'ing the log file).

Any help pointing me in the right direction would be greatly appreciated..

Thanks,
Bec

[vigour]

1. how would I parse the data for a certain time period ? is awk the best way to go ?

On your place, I would have used unix_timestamp for the data. Thus I will be able to grap and easily identify the data with perl. Conversion from unix_timespamp to human readable is pretty easy.

2. if I can get the output from each machine right, what is the best way to compile the data into one result file.. not only grouping them together but also automating the ftp...

If you use Perl, logging trough an FTP account, extracting the files and putting them all together is no so difficult. I think Net::FTP was the module.

Cheers,
Ventsi

[BecTech]

well, I figured out the multiple connection issues, but I still need a way to get specific information from the large log files..

the log is tool generated, I don't believe it is easily modified... since there was already a timestamp in the file I'm hoping to figure out a way to only get the last part if the file info.

I can use awk to get the lines I need but I need to figure our a way to get something like the last 30 days of the log file, not the whole log file...

(i'll just use rexec and the .netrc file to connect to the machines..)

[Franklin52]

Try something like:

sed -n '/Mon Oct 15 08:00:00 EDT 2007/,$p' file

This should give all lines from the line with the given date till the end of the file.

Regards