Grep Issue

[dbo3587]

Hey Forum,

I have a greping issue i was hoping to get help with. I am running a large series of test scripts and the output from these scripts gets put into a very large log file. I would like to grep through this file to pull out the results of only the tests that failed.

This is an example of how the log file looks:

==============================================
Test case: test_1.txt
.
.
.
The settings of the test case
.
.
.
Errors:
verdicts....blah
verdicts....blah
verdicts....blah
verdicts....blah

==============================================
Test case: test_2.txt
.
.
.
The settings of the test case
.
.
.

RESULT: SUCCESS

==============================================


...And so on for several hundred tests...I only really need to see the information about the failed test cases like below:

Test case: test_1.txt
.
.
.
The settings of the test case
.
.
.
Errors:
verdicts....blah
verdicts....blah
verdicts....blah
verdicts....blah


My thinking is that it would be easiest to grep from ==== to ==== and if there is the word "Errors:" between them...but i cant seem to figure out how to do this? Is there an easy way to do this?

Thanks.

[atreyu]

You can do this w/awk. I'd first get the names of all the tests, then loop thru all of them, searching for each tests results, using awk's built-in range function: the range starts with the test name and ends with a line that starts with ==. here's an example:

#!/bin/bash
tests=$(cat /tmp/log.txt|awk '/^Test case:/{print $3}')
for test in $tests; do
  unset err
  printf "\nTest $test ... "
  err=$(cat /tmp/log.txt|awk "/^Test case: $test/,/^==/")
  printf "$err\n"|grep -qi error
  if [ $? -ne 0 ]; then
    echo OK
  else
    echo ERRORS!
    printf "$err\n"
  fi
done

[drl]

Hi.

Here are two solutions using an augmented data set, data1. The first is awk, but uses the string as the Record Separator (RS). The second uses a flexible grep designed for this kind of search, cgrep. After listing the context of execution and data file, the results of the runs are displayed.

#!/usr/bin/env bash

# @(#) s1	Demonstrate bracketed extraction, matching.
# cgrep-home2 http://sourceforge.net/projects/cgrep/

# Section 1, setup, pre-solution, $Revision: 1.2 $".
# Infrastructure details, environment, debug commands for forum posts. 
# Uncomment export command to run script as external user.
# export PATH="/usr/local/bin:/usr/bin:/bin" HOME=""
set +o nounset
pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
pl() { pe;pe "-----" ;pe "$*"; }
edges() { _n="$1" _f="$2";head -n $_n $_f; pe "  ---";tail -n $_n $_f ; }
db() { : ; }
db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
C=$HOME/bin/context && [ -f $C ] && $C awk cgrep
set -o nounset
pe

FILE=${1-data1}

# Display a sample data file, with head & tail as a last resort.
db " Section 1: display of input data."
pe " || start sample $FILE"
cat $FILE
pe " || end"

# Section 2, solution.
pl " Solution with awk:"
db " Section 2: solution."
awk '
BEGIN	{ RS = ORS = "==============================================" ; print "\n" }
/Errors:/
END	{ ORS = "\n" ; print ORS }
' $FILE

pl " Solution with cgrep:"
delimiter="=============================================="
cgrep -D +I2 -+w "$delimiter" Errors: $FILE
pe "$delimiter"

exit 0

producing:

% ./s1

Environment: LC_ALL = C, LANG = C
(Versions displayed with local utility "version")
OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
GNU bash 3.2.39
GNU Awk 3.1.5
ATT cgrep 8.15

 db,  Section 1: display of input data.
 || start sample data1
==============================================
Test case: test_1.txt
The settings of the test case
Errors:
verdicts....blah
verdicts....blah
verdicts....blah
verdicts....blah

==============================================
Test case: test_2.txt
The settings of the test case
RESULT: SUCCESS

==============================================
Test case test_3.txt
junk
Errors:
stuff

==============================================
Test case test_4.txt
junk
RESULT: SUCCESS

==============================================
 || end

-----
 Solution with awk:
 db,  Section 2: solution.

==============================================
Test case: test_1.txt
The settings of the test case
Errors:
verdicts....blah
verdicts....blah
verdicts....blah
verdicts....blah

==============================================
Test case test_3.txt
junk
Errors:
stuff

==============================================


-----
 Solution with cgrep:
==============================================
Test case: test_1.txt
The settings of the test case
Errors:
verdicts....blah
verdicts....blah
verdicts....blah
verdicts....blah

==============================================
Test case test_3.txt
junk
Errors:
stuff

==============================================

See the URL for cgrep, man awk or man gawk for details ... cheers, drl

[dbo3587]

So for both of these scripts, what would the file type be? .sh, .bat? And is this outputting the results to the same file or to the screen?

[atreyu]

Well, the type of the file is ASCII text. The file extension is whatever you want to put on the end of it. It gets executed (which is what I think you really mean) by how it is called.

If you call a program like this:

sh ~/myprog.pl

Then the sh/bash shell will be invoked to run ~/myprog.pl, even though it is probably a perl script (and the failure would be epic).

If your program is executable and you call it like this:

~/myprog.pl

Then it would look for the command line interpreter (usually the first line in the code, and starts with a #!) and use that to invoke it. So in the case of my code, /bin/bash would be called to invoke the script. And in drl's (much smarter) case, the env program would be used to find bash in the user's PATH and use that to invoke the script.

So, in short, use ".sh".

[dbo3587]

Ok, thats how i thought is should be trying to execute it. I took the code that you wrote and modified it to fit my needs, i saved it as Errors.sh. When i try to execute using "sh Errors.sh" i get an error

'rrors.sh: line 3: syntax error near unexpected token `do
'rrors.sh: line 3: `for test in $tests; do

And i am not sure what it is caused by... Also what would you say would be the easiest method for outputting these errors to their own file?

Thanks a bunch~

[atreyu]

It might be that some of your bash syntax is incorrect - not quite sure based upon the error you posted.

can you post the whole text of your modified code - enclosed in CODE brackets?

[dbo3587]

#!/bin/bash
tests=$(cat ./_CDW_test_1.txt|awk "/^Test case:/{print $3}")
for test in $tests; do
  unset err
  printf "\nTest $test ... "
  err=$(cat ./_CDW_test_1.txt|awk "/^Test case: $test/,/^==/")
  printf "$err\n"|grep -qi error
  if [ $? -ne 0 ]; then
    echo OK
  else
    echo ERRORS!
    printf "$err\n"
  fi
done

[atreyu]

Your script works for me.

Do you maybe have DOS/Windows carriage returns in the log file?

do this to your text file:

cat -v _CDW_test_1.txt

If you see a bunch of ^M characters at the end of each line, that's your problem.

To fix it, either edit/create the file in Linux, or use dos2unix to convert it, e.g.:

dos2unix _CDW_test_1.txt

You may need to install the dos2unix package first...

[dbo3587]

Hey thanks for the help!

You were right about the issue but the ^M characters were actually in the script file itself.;

Thanks again, this made my life easier by cutting down a 1000 page error file into about 10 pages!