Commands for logfile handling

[nl405575]

Hello everybody,

I'm new...

I've got this log file and I need to get all sorts of information from it...

24 - [02/Sep/2010:00:01:16 +0200] - 10.1.53.62 - 200
23 - [02/Sep/2010:00:01:26 +0200] - 10.1.53.62 - 200
19 - [02/Sep/2010:00:01:56 +0200] - 10.1.53.62 - 200
19 - [02/Sep/2010:00:01:58 +0200] - 10.1.53.62 - 200
25 - [02/Sep/2010:00:02:37 +0200] - 10.1.53.62 - 200
24 - [02/Sep/2010:00:04:04 +0200] - 10.1.53.62 - 200
24 - [02/Sep/2010:00:04:49 +0200] - 10.1.53.62 - 200
28 - [02/Sep/2010:00:05:13 +0200] - 10.1.53.62 - 200
24 - [02/Sep/2010:00:05:50 +0200] - 10.1.53.62 - 200
20 - [02/Sep/2010:00:05:57 +0200] - 10.1.53.62 - 200
21 - [02/Sep/2010:00:06:12 +0200] - 10.1.53.62 - 200
20 - [02/Sep/2010:00:06:21 +0200] - 10.1.53.62 - 200
24 - [02/Sep/2010:00:07:02 +0200] - 10.1.53.62 - 200
...
and it goes on and on... but you get the point

structure of each line is

number_of_milisecs - [date] - ip_address - http_status

I'm having trouble with Linux commands for:

1. How many requests came altogether
2. How many requests with status 200
3. The slowest 10 requests
4. How many requests with status 200 were handled in between 1 and 2 seconds

Can somebody tell me where is the best place to look for the answers for these sorts of questions?

Thanks so much!

[Nagarjuna]

Hello, nl405575. Welcome to Linux-Forums.

1. How many requests came altogether:

cat log | wc -l

2. How many requests with status 200:

cat log | cut -d ' ' -f8 | wc -l

3. The slowest 10 requests:

cat log | cut -d: -f3,4 | cut -c1-5 | sort -r | tail -10 | nl

4. How many requests with status 200 were handled in between 1 and 2 seconds:

cat log | grep ":00:00:" || cat log | grep ":00:01:"

All in one script:

#!/bin/bash

echo "Number of requests:"
cat log | wc -l
echo

echo "Number of requests with status of 200:"
cat log | cut -d ' ' -f8 | wc -l
echo

echo "Top 10 slowest requests:"
cat log | cut -d: -f3,4 | cut -c1-5 | sort -r | tail -10 | nl
echo

echo "Status 200 requests handled between 1 and 2 seconds:"
cat log | grep ":00:00:" || cat log | grep ":00:01:"
echo

You will of course need to replace the word 'log' with the location of the above log file.

I hope this helps. Let me know if you have any questions or run into any problems with the script.

[nl405575]

Thanks man, I do have some questions...

structure of each line is

number_of_milisecs - [date] - ip_address - http_status

2. How many requests with status 200:
Code:

cat log | cut -d ' ' -f8 | wc -l

Isn't this just going to count the lines without checking the status? What if the status if 404, this command doesn't check the status.

3. The slowest 10 requests:
Code:

cat log | cut -d: -f3,4 | cut -c1-5 | sort -r | tail -10 | nl

Isn't this command supposed to check the miliseconds? What does the cut -c1-5 do?

4. How many requests with status 200 were handled in between 1 and 2 seconds:
Code:

cat log | grep ":00:00:" || cat log | grep ":00:01:"

This is supposed to check the miliseconds again, not time, right? How many lines with status 200 is between 1000 and 2000 miliseconds...

Thanks for your help man, I really appreciate it!

[Nagarjuna]

Hey, bud. Sorry for the mistakes, and thanks for pointing them out. I'm still learning the ropes here, so bare with me.

Hopefully the below will fix these commands. I'm not at home where I can experiment at the moment, but I'll be sure to test 'em out when I can to be sure they work.

Okay, for the 2nd command:

cat log | cut -d ' ' -f8 | grep "200" | wc -l

The above will cut out the status column and will only count the number "200". Does this achieve what your looking for?

cat log | cut -d ' ' -f1 | sort -r | tail -10 | nl

This should cut the first field of text, which is the miliseconds right? It may cut the second field though, I'll need to check this when I get home. It should then sort it from highest-to-lowest, cut the top 10 and number the lines.

The last I will need to experiment with when I get home. I will need to somehow cut the status and the miliseconds out seperately and parse them. It'll probably be a multiline script.

I'm sure someone with more experience could find better ways of doing these, but the above should hopefully at least get the job done. I really need to start learning fancy text manipulation languages like awk and sed after I get BASH comfortably under my belt..

I hope this helps. I'll be back later to check my work and finish that last command.

[nl405575]

Hey man, nice to see somebody helping a noob!

Okay, for the 2nd command:

Code:

cat log | cut -d ' ' -f8 | grep "200" | wc -l

This works perfect...

However... the third command

cat log | cut -d ' ' -f1 | sort -r | tail -10 | nl

is getting the first field which is miliseconds but is sorting by string value and not by integer value and therefore doesn't do the job... you see? so for example 100 < 21 ... get it?

So this is the line of code we should use...

cat log | cut -d ' ' -f1 | sort -r -n | tail -10 | nl

You have to add -n to sort it numerically... And the last question is still unknown

[nl405575]

and here's the last one... got it!

#!/bin/bash
    clear
    a=`cat localhost.txt | awk '$1 < 2000' | wc -l`
    b=`cat localhost.txt | awk '$1 < 1000' | wc -l`
    echo A is $a
echo B is $b
echo $(($a - $b))

Anywayz my man, thanks for helping me out, never an easy job to start on something, I know people think this is for school and stuff like that so they won't answer but it isn't... so I'm deeply grateful for time you put in solving my questions...

That would be all, thanks again

Nick

[Nagarjuna]

Awesome! I'm glad you got it..

Thanks again for helping me clear up some confusion.

Cheers.