Log File Analysis

[elluk]

Hi All,

I'm after some advice.

I have a log file with the following format (yeah I know, it's IIS! some things are out of my control!).

########################
2007-11-27 01:03:37 10.102.103.22 POST /page/placeselect_put.aspx - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 200 0 0
2007-11-27 03:39:38 10.102.103.22 GET /scripts/validation.js - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 304 0 0
2007-11-27 05:49:38 10.102.103.22 POST /page/joblist_put.aspx - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 200 0 0
########################

I wanted to parse the entire log to calculate number requests per page. (Active content only). Easy.
But I need to do it over a specific time range. The best I came up with was this:

cat logfile.log|grep -i ".aspx"|grep '(03:[3-5][0-9]|04:[0-2][0-9]|04:30)'|gawk "{print $5}"|sort|uniq -c|sort|nc printer 9100

Is there a better way to process time ranges using bash?
I'd like to stick to POSIX-BASH styleee.

Ultimatley, I need to work out how may request per page, per minute! Any ideas?

Would I really need to resort to perl?

Can you guys recommend a good log parser?

Thanks everyone.

[wje_lf]

Ok, here's the deal.

Your question has been sitting there for five hours. Nobody's touched it.

There's an old rule in Usenet (back in the days before web browsers, when newsgroups were primarily used as forums for discussion, not media for swapping music and other products) that if someone poses a question, it could just languish for days and days.

But if someone posts information that might be wrong, someone else will jump in immediately and correct that possible misinformation.

So I'm going to jump in and give you my opinion: Your best bet is to bite the bullet and use Perl (or python or ruby or FORTRAN or something).

Ok. Let's sit back and see if someone corrects me and recommends a good log parser.

That's the best I can do, friend.

[ghostdog74]

Hi All,

I'm after some advice.

I have a log file with the following format (yeah I know, it's IIS! some things are out of my control!).

########################
2007-11-27 01:03:37 10.102.103.22 POST /page/placeselect_put.aspx - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 200 0 0
2007-11-27 03:39:38 10.102.103.22 GET /scripts/validation.js - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 304 0 0
2007-11-27 05:49:38 10.102.103.22 POST /page/joblist_put.aspx - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 200 0 0
########################

I wanted to parse the entire log to calculate number requests per page. (Active content only). Easy.
But I need to do it over a specific time range. The best I came up with was this:

cat logfile.log|grep -i ".aspx"|grep '(03:[3-5][0-9]|04:[0-2][0-9]|04:30)'|gawk "{print $5}"|sort|uniq -c|sort|nc printer 9100

Is there a better way to process time ranges using bash?
I'd like to stick to POSIX-BASH styleee.

Ultimatley, I need to work out how may request per page, per minute! Any ideas?

here's an awk script, just an example...

awk 'BEGIN {
   print "Enter date from (YYYY MM DD): "
   getline dfrom < "-"
   print "Enter date to(YYYY MM DD): "
   getline dto <"-"
   print "Enter time from (hh mm ss): "
   getline tfrom < "-"
   print "Enter time to (hh mm ss): "
   getline tto <"-"
   df = dfrom " " tfrom
   dt = dto " " tto
   datetimefrom = mktime(df)
   datetimeto = mktime(dt)
}
{
   gsub(/-/," ",$1)
   gsub(/:/ ," ",$2)  
   dat = $1 " " $2
   thedate = mktime(dat)
   if (( thedate >= datetimefrom)  &&  ( thedate <= datetimeto )) {
     print 
   }
}
' file

output interaction:

# ./test.sh
Enter date from (YYYY MM DD):
2007 11 27
Enter date to(YYYY MM DD):
2007 11 27
Enter time from (hh mm ss):
00 00 00
Enter time to (hh mm ss):
03 40 00
2007 11 27 01 03 37 10.102.103.22 POST /page/placeselect_put.aspx - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 200 0 0
2007 11 27 03 39 38 10.102.103.22 GET /scripts/validation.js - 81 - 10.102.109.12 Mozilla/4.0+(compatible;+MSIE+5.5;+Windows+NT+4.0) 304 0 0
it530192:~/yhlee/test #

Would I really need to resort to perl?

No need.

Can you guys recommend a good log parser?

you can use the microsoft logparser, its IIS remember?
example reference