Perl:Get log files content on specific time range

[charith]

Hi All,
I need to extract logs on specific time range from several log files. I used following in my perl script but it gives logs only on specify times (not time range) also it not retrieving logs from all log files (need to specify log file name as CDSServer.log can't use CDSServer.* course I have several log files like CDSServer.log CDSServer.log01 CDSServer.log02 ...etc )


My perl script awk:

my $OldTime= 'Sep 10, 2012 5:20:41';
my $NewTime='Sep 10, 2012 5:49:40';

my $test2 = qx{ssh -o stricthostkeychecking=no $WLS "awk '/$OldTime/,/$NewTime/' $WLSP/logs/CDSServer.* "};

My log file format:

####<Sep 10, 2012 5:44:41 PM IST> <Info> Kernel>> <>  <Connection for pool  closed.> 
####<Sep 10, 2012 5:44:41 PM IST> <Info> <JDBC> <cdsserver> <CDSServer11> <[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'> 
####<Sep 10, 2012 5:44:41 PM IST> <Info> <JDBC> <12d58f5205084394:4db5aec7:1393b6c3ce7:-8000-00000000
####<Sep 10, 2012 5:44:41 PM IST> <Info> <JDBC> <cdsserver> <CDSServer11> <[ACTIVE] ExecuteThread: '

[atreyu]

Hi,

I'd make two suggestions. The first one is to use the Date::Parse Perl module. It is hopefully already packaged for your distro. This module will allow you to easily convert date/time strings to seconds since the epoch (which is an easy way to do date/time math). It will give you the equivalent output to this GNU date command:

date +%s -d "Sep 10, 2012 5:20:41 PM"

The second suggestion would be to put the script on the server and pass to it 3 arguments:

1. the start date/time range
2. the end date/time range
3. the log file to parse

then you'd do something like this to call it:

ssh server /tmp/parse-log.pl 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:44:42 PM' /path/to/CDSserver.log

and here is the parse-log.pl script:

#!/usr/bin/perl
use strict;
use warnings;
use Date::Parse;

# get command line arguments (3)
die "
  Usage: $0 '<START_TIME>' '<STOP_TIME>' <LOG_FILE>
   E.g.: $0 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:49:40 PM' CDSServer.log\n"
  unless($#ARGV == 2);

my $startTime = $ARGV[0];
my $stopTime  = $ARGV[1];
my $log = $ARGV[2];

# make sure the log file exists
die "$log: No such file\n" unless(-f$log);

# convert date/time strings to seconds since epoch
my $start_sec = str2time($startTime);
my $stop_sec = str2time($stopTime);

print "Start time: $startTime ($start_sec)\n";
print "Stop time: $stopTime ($stop_sec)\n";

open(LOG,'<',$log) or die "can't read '$log': $!\n";
while(<LOG>){
  chomp;
  my $line = $_; # save original line
  s/[ \t]+/ /;   # replace contiguous white spaces w/single space
  if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
    my $timedate = $1;

    # convert date/time string in log entro to epoch seconds
    my $seconds = str2time($timedate);

    # print line if it falls into the range
    print $line,"\n" if(($seconds >= $start_sec)&&($seconds <= $stop_sec));
  }
}
close(LOG);

[charith]

Hi atreyu,

Thank you very much for your great clean reply.

Your script working fine and it gives lines those match given times but what i need is get all lines whatever between that time range. I'm sorry for my bad log file format i attached correct log file below.[Errors not begin with time]

####<Aug 25, 2012 11:11:42 PM IST> <Warning> <JDBC> <cdsserver> f-tuning)'> <<WLS Kernel>> <> <> <1345916502340> <B
####<Aug 25, 2012 11:11:42 PM IST> <Info> <JDBC> <cdsserver> <CDSServer2> <[ACTIVE] ExecuteThread: '20' for queue: '
java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:443)
	at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:670)
	at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:230)
###<Aug 25, 2012 11:11:47 PM IST> <Warning> <JDBC> <cdsserver> <CDSServer2> <[ACTIVE] ExecuteThrea
####<Aug 25, 2012 11:11:47 PM IST> <Info> <JDBC> <cdsserver> <CDSServer2> <[ACT

Above java.sql.SQLRecoverableException: IO Error: part should be retrieve.

[atreyu]

ah, okay. yeah, that changes things, but not by too much. basically, you can just set a marker once the start time string is matched, then set a stop marker once the end time string is matched, and save everything in between to an array. then print the array once you're done looping thru the file.

#!/usr/bin/perl
use strict;
use warnings;
use Date::Parse;

# get command line arguments (3)
die "
  Usage: $0 '<START_TIME>' '<STOP_TIME>' <LOG_FILE>
   E.g.: $0 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:49:40 PM' CDSServer.log\n"
  unless($#ARGV == 2);

my $startTime = $ARGV[0];
my $stopTime  = $ARGV[1];
my $log = $ARGV[2];

# make sure the log file exists
die "$log: No such file\n" unless(-f$log);

# convert date/time strings to seconds since epoch
my $start_sec = str2time($startTime);
my $stop_sec = str2time($stopTime);

# make sure we got nothing but digits in the variables
die "Failed to convert $startTime to seconds\n" unless($start_sec =~ /^[0-9]*$/);
die "Failed to convert $stopTime to seconds\n" unless($stop_sec =~ /^[0-9]*$/);

print "Start time: $startTime ($start_sec)\n";
print "Stop time: $stopTime ($stop_sec)\n";

my @lines;
my $stop;

open(LOG,'<',$log) or die "can't read '$log': $!\n";
while(<LOG>){
  chomp;
  my $line = $_; # save original line
  s/[ \t]+/ /;   # replace contiguous white spaces w/single space
  if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
    my $timedate = $1;

    # convert date/time string in log entro to epoch seconds
    my $seconds = str2time($timedate);

    # save line w/time string to array if it falls into the range
    if($seconds >= $start_sec){
      push(@lines,$line) unless($stop);
    }elsif($seconds >= $stop_sec){
      push(@lines,$line) unless($stop);
      $stop = 1;
    }
  }else{
    # save line w/o time string to array if it falls into the range
    push(@lines,$line) if(($#lines>=0)&&!($stop));
  }
}
close(LOG);

# print the saved lines
print "$_\n" for(@lines);

[charith]

Hi atreyu,
It's working fine thank you very much.

Did small change:

# save line w/time string to array if it falls into the range
    if($seconds >= $start_sec){

as

if (($seconds >= $start_sec)&&($seconds <= $stop_sec)){

otherwise it will print all from $start_sec

I like your regex (/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/) can you please give some guide lines(some links) to learn those. Thanks again