Find the answer to your Linux question:
Results 1 to 5 of 5
Hi All, I need to extract logs on specific time range from several log files . I used following in my perl script but it gives logs only on specify ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Nov 2010
    Posts
    26

    Perl:Get log files content on specific time range


    Hi All,
    I need to extract logs on specific time range from several log files. I used following in my perl script but it gives logs only on specify times (not time range) also it not retrieving logs from all log files (need to specify log file name as CDSServer.log can't use CDSServer.* course I have several log files like CDSServer.log CDSServer.log01 CDSServer.log02 ...etc )


    My perl script awk:
    Code:
    my $OldTime= 'Sep 10, 2012 5:20:41';
    my $NewTime='Sep 10, 2012 5:49:40';
    
    my $test2 = qx{ssh -o stricthostkeychecking=no $WLS "awk '/$OldTime/,/$NewTime/' $WLSP/logs/CDSServer.* "};
    My log file format:
    Code:
    ####<Sep 10, 2012 5:44:41 PM IST> <Info> Kernel>> <>  <Connection for pool  closed.> 
    ####<Sep 10, 2012 5:44:41 PM IST> <Info> <JDBC> <cdsserver> <CDSServer11> <[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'> 
    ####<Sep 10, 2012 5:44:41 PM IST> <Info> <JDBC> <12d58f5205084394:4db5aec7:1393b6c3ce7:-8000-00000000
    ####<Sep 10, 2012 5:44:41 PM IST> <Info> <JDBC> <cdsserver> <CDSServer11> <[ACTIVE] ExecuteThread: '

  2. #2
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    Hi,

    I'd make two suggestions. The first one is to use the Date::Parse Perl module. It is hopefully already packaged for your distro. This module will allow you to easily convert date/time strings to seconds since the epoch (which is an easy way to do date/time math). It will give you the equivalent output to this GNU date command:

    Code:
    date +%s -d "Sep 10, 2012 5:20:41 PM"
    The second suggestion would be to put the script on the server and pass to it 3 arguments:

    1. the start date/time range
    2. the end date/time range
    3. the log file to parse

    then you'd do something like this to call it:

    Code:
    ssh server /tmp/parse-log.pl 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:44:42 PM' /path/to/CDSserver.log
    and here is the parse-log.pl script:

    Code:
    #!/usr/bin/perl
    use strict;
    use warnings;
    use Date::Parse;
    
    # get command line arguments (3)
    die "
      Usage: $0 '<START_TIME>' '<STOP_TIME>' <LOG_FILE>
       E.g.: $0 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:49:40 PM' CDSServer.log\n"
      unless($#ARGV == 2);
    
    my $startTime = $ARGV[0];
    my $stopTime  = $ARGV[1];
    my $log = $ARGV[2];
    
    # make sure the log file exists
    die "$log: No such file\n" unless(-f$log);
    
    # convert date/time strings to seconds since epoch
    my $start_sec = str2time($startTime);
    my $stop_sec = str2time($stopTime);
    
    print "Start time: $startTime ($start_sec)\n";
    print "Stop time: $stopTime ($stop_sec)\n";
    
    open(LOG,'<',$log) or die "can't read '$log': $!\n";
    while(<LOG>){
      chomp;
      my $line = $_; # save original line
      s/[ \t]+/ /;   # replace contiguous white spaces w/single space
      if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
        my $timedate = $1;
    
        # convert date/time string in log entro to epoch seconds
        my $seconds = str2time($timedate);
    
        # print line if it falls into the range
        print $line,"\n" if(($seconds >= $start_sec)&&($seconds <= $stop_sec));
      }
    }
    close(LOG);

  3. #3
    Just Joined!
    Join Date
    Nov 2010
    Posts
    26
    Hi atreyu,

    Thank you very much for your great clean reply.

    Your script working fine and it gives lines those match given times but what i need is get all lines whatever between that time range. I'm sorry for my bad log file format i attached correct log file below.[Errors not begin with time]

    Code:
    ####<Aug 25, 2012 11:11:42 PM IST> <Warning> <JDBC> <cdsserver> f-tuning)'> <<WLS Kernel>> <> <> <1345916502340> <B
    ####<Aug 25, 2012 11:11:42 PM IST> <Info> <JDBC> <cdsserver> <CDSServer2> <[ACTIVE] ExecuteThread: '20' for queue: '
    java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connection
    	at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:443)
    	at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:670)
    	at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:230)
    ###<Aug 25, 2012 11:11:47 PM IST> <Warning> <JDBC> <cdsserver> <CDSServer2> <[ACTIVE] ExecuteThrea
    ####<Aug 25, 2012 11:11:47 PM IST> <Info> <JDBC> <cdsserver> <CDSServer2> <[ACT
    Above java.sql.SQLRecoverableException: IO Error: part should be retrieve.

  4. $spacer_open
    $spacer_close
  5. #4
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    ah, okay. yeah, that changes things, but not by too much. basically, you can just set a marker once the start time string is matched, then set a stop marker once the end time string is matched, and save everything in between to an array. then print the array once you're done looping thru the file.

    Code:
    #!/usr/bin/perl
    use strict;
    use warnings;
    use Date::Parse;
    
    # get command line arguments (3)
    die "
      Usage: $0 '<START_TIME>' '<STOP_TIME>' <LOG_FILE>
       E.g.: $0 'Sep 10, 2012 5:20:41 PM' 'Sep 10, 2012 5:49:40 PM' CDSServer.log\n"
      unless($#ARGV == 2);
    
    my $startTime = $ARGV[0];
    my $stopTime  = $ARGV[1];
    my $log = $ARGV[2];
    
    # make sure the log file exists
    die "$log: No such file\n" unless(-f$log);
    
    # convert date/time strings to seconds since epoch
    my $start_sec = str2time($startTime);
    my $stop_sec = str2time($stopTime);
    
    # make sure we got nothing but digits in the variables
    die "Failed to convert $startTime to seconds\n" unless($start_sec =~ /^[0-9]*$/);
    die "Failed to convert $stopTime to seconds\n" unless($stop_sec =~ /^[0-9]*$/);
    
    print "Start time: $startTime ($start_sec)\n";
    print "Stop time: $stopTime ($stop_sec)\n";
    
    my @lines;
    my $stop;
    
    open(LOG,'<',$log) or die "can't read '$log': $!\n";
    while(<LOG>){
      chomp;
      my $line = $_; # save original line
      s/[ \t]+/ /;   # replace contiguous white spaces w/single space
      if(/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/){
        my $timedate = $1;
    
        # convert date/time string in log entro to epoch seconds
        my $seconds = str2time($timedate);
    
        # save line w/time string to array if it falls into the range
        if($seconds >= $start_sec){
          push(@lines,$line) unless($stop);
        }elsif($seconds >= $stop_sec){
          push(@lines,$line) unless($stop);
          $stop = 1;
        }
      }else{
        # save line w/o time string to array if it falls into the range
        push(@lines,$line) if(($#lines>=0)&&!($stop));
      }
    }
    close(LOG);
    
    # print the saved lines
    print "$_\n" for(@lines);

  6. #5
    Just Joined!
    Join Date
    Nov 2010
    Posts
    26
    Hi atreyu,
    It's working fine thank you very much.

    Did small change:
    Code:
    # save line w/time string to array if it falls into the range
        if($seconds >= $start_sec){
    as
    Code:
    if (($seconds >= $start_sec)&&($seconds <= $stop_sec)){
    otherwise it will print all from $start_sec

    I like your regex (/^####<([a-zA-Z]{3} [0-9]{1,2}, [0-9]{4} [0-9]{1,2}:[0-9]{2}:[0-9]{2} [AP]M) [A-Z]{3}>/) can you please give some guide lines(some links) to learn those. Thanks again

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •