Find the answer to your Linux question:
Results 1 to 10 of 10
Hey Forum, I have a greping issue i was hoping to get help with. I am running a large series of test scripts and the output from these scripts gets ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Nov 2011
    Posts
    12

    Grep Issue


    Hey Forum,

    I have a greping issue i was hoping to get help with. I am running a large series of test scripts and the output from these scripts gets put into a very large log file. I would like to grep through this file to pull out the results of only the tests that failed.

    This is an example of how the log file looks:

    ==============================================
    Test case: test_1.txt
    .
    .
    .
    The settings of the test case
    .
    .
    .
    Errors:
    verdicts....blah
    verdicts....blah
    verdicts....blah
    verdicts....blah

    ==============================================
    Test case: test_2.txt
    .
    .
    .
    The settings of the test case
    .
    .
    .

    RESULT: SUCCESS

    ==============================================


    ...And so on for several hundred tests...I only really need to see the information about the failed test cases like below:

    Test case: test_1.txt
    .
    .
    .
    The settings of the test case
    .
    .
    .
    Errors:
    verdicts....blah
    verdicts....blah
    verdicts....blah
    verdicts....blah


    My thinking is that it would be easiest to grep from ==== to ==== and if there is the word "Errors:" between them...but i cant seem to figure out how to do this? Is there an easy way to do this?

    Thanks.

  2. #2
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    You can do this w/awk. I'd first get the names of all the tests, then loop thru all of them, searching for each tests results, using awk's built-in range function: the range starts with the test name and ends with a line that starts with ==. here's an example:

    Code:
    #!/bin/bash
    tests=$(cat /tmp/log.txt|awk '/^Test case:/{print $3}')
    for test in $tests; do
      unset err
      printf "\nTest $test ... "
      err=$(cat /tmp/log.txt|awk "/^Test case: $test/,/^==/")
      printf "$err\n"|grep -qi error
      if [ $? -ne 0 ]; then
        echo OK
      else
        echo ERRORS!
        printf "$err\n"
      fi
    done

  3. #3
    drl
    drl is offline
    Linux Engineer drl's Avatar
    Join Date
    Apr 2006
    Location
    Saint Paul, MN, USA / CentOS, Debian, Slackware, {Free, Open, Net}BSD, Solaris
    Posts
    1,305
    Hi.

    Here are two solutions using an augmented data set, data1. The first is awk, but uses the string as the Record Separator (RS). The second uses a flexible grep designed for this kind of search, cgrep. After listing the context of execution and data file, the results of the runs are displayed.
    Code:
    #!/usr/bin/env bash
    
    # @(#) s1	Demonstrate bracketed extraction, matching.
    # cgrep-home2 http://sourceforge.net/projects/cgrep/
    
    # Section 1, setup, pre-solution, $Revision: 1.2 $".
    # Infrastructure details, environment, debug commands for forum posts. 
    # Uncomment export command to run script as external user.
    # export PATH="/usr/local/bin:/usr/bin:/bin" HOME=""
    set +o nounset
    pe() { for _i;do printf "%s" "$_i";done; printf "\n"; }
    pl() { pe;pe "-----" ;pe "$*"; }
    edges() { _n="$1" _f="$2";head -n $_n $_f; pe "  ---";tail -n $_n $_f ; }
    db() { : ; }
    db() { ( printf " db, ";for _i;do printf "%s" "$_i";done;printf "\n" ) >&2 ; }
    C=$HOME/bin/context && [ -f $C ] && $C awk cgrep
    set -o nounset
    pe
    
    FILE=${1-data1}
    
    # Display a sample data file, with head & tail as a last resort.
    db " Section 1: display of input data."
    pe " || start sample $FILE"
    cat $FILE
    pe " || end"
    
    # Section 2, solution.
    pl " Solution with awk:"
    db " Section 2: solution."
    awk '
    BEGIN	{ RS = ORS = "==============================================" ; print "\n" }
    /Errors:/
    END	{ ORS = "\n" ; print ORS }
    ' $FILE
    
    pl " Solution with cgrep:"
    delimiter="=============================================="
    cgrep -D +I2 -+w "$delimiter" Errors: $FILE
    pe "$delimiter"
    
    exit 0
    producing:
    Code:
    % ./s1
    
    Environment: LC_ALL = C, LANG = C
    (Versions displayed with local utility "version")
    OS, ker|rel, machine: Linux, 2.6.26-2-amd64, x86_64
    Distribution        : Debian GNU/Linux 5.0.8 (lenny) 
    GNU bash 3.2.39
    GNU Awk 3.1.5
    ATT cgrep 8.15
    
     db,  Section 1: display of input data.
     || start sample data1
    ==============================================
    Test case: test_1.txt
    The settings of the test case
    Errors:
    verdicts....blah
    verdicts....blah
    verdicts....blah
    verdicts....blah
    
    ==============================================
    Test case: test_2.txt
    The settings of the test case
    RESULT: SUCCESS
    
    ==============================================
    Test case test_3.txt
    junk
    Errors:
    stuff
    
    ==============================================
    Test case test_4.txt
    junk
    RESULT: SUCCESS
    
    ==============================================
     || end
    
    -----
     Solution with awk:
     db,  Section 2: solution.
    
    ==============================================
    Test case: test_1.txt
    The settings of the test case
    Errors:
    verdicts....blah
    verdicts....blah
    verdicts....blah
    verdicts....blah
    
    ==============================================
    Test case test_3.txt
    junk
    Errors:
    stuff
    
    ==============================================
    
    
    -----
     Solution with cgrep:
    ==============================================
    Test case: test_1.txt
    The settings of the test case
    Errors:
    verdicts....blah
    verdicts....blah
    verdicts....blah
    verdicts....blah
    
    ==============================================
    Test case test_3.txt
    junk
    Errors:
    stuff
    
    ==============================================
    See the URL for cgrep, man awk or man gawk for details ... cheers, drl
    Welcome - get the most out of the forum by reading forum basics and guidelines: click here.
    90% of questions can be answered by using man pages, Quick Search, Advanced Search, Google search, Wikipedia.
    We look forward to helping you with the challenge of the other 10%.
    ( Mn, 2.6.n, AMD-64 3000+, ASUS A8V Deluxe, 1 GB, SATA + IDE, Matrox G400 AGP )

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined!
    Join Date
    Nov 2011
    Posts
    12
    So for both of these scripts, what would the file type be? .sh, .bat? And is this outputting the results to the same file or to the screen?

  6. #5
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    Well, the type of the file is ASCII text. The file extension is whatever you want to put on the end of it. It gets executed (which is what I think you really mean) by how it is called.

    If you call a program like this:
    Code:
    sh ~/myprog.pl
    Then the sh/bash shell will be invoked to run ~/myprog.pl, even though it is probably a perl script (and the failure would be epic).

    If your program is executable and you call it like this:
    Code:
    ~/myprog.pl
    Then it would look for the command line interpreter (usually the first line in the code, and starts with a #!) and use that to invoke it. So in the case of my code, /bin/bash would be called to invoke the script. And in drl's (much smarter) case, the env program would be used to find bash in the user's PATH and use that to invoke the script.

    So, in short, use ".sh".

  7. #6
    Just Joined!
    Join Date
    Nov 2011
    Posts
    12
    Ok, thats how i thought is should be trying to execute it. I took the code that you wrote and modified it to fit my needs, i saved it as Errors.sh. When i try to execute using "sh Errors.sh" i get an error

    Code:
    'rrors.sh: line 3: syntax error near unexpected token `do
    'rrors.sh: line 3: `for test in $tests; do
    And i am not sure what it is caused by... Also what would you say would be the easiest method for outputting these errors to their own file?

    Thanks a bunch~

  8. #7
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    It might be that some of your bash syntax is incorrect - not quite sure based upon the error you posted.

    can you post the whole text of your modified code - enclosed in CODE brackets?

  9. #8
    Just Joined!
    Join Date
    Nov 2011
    Posts
    12
    Code:
    #!/bin/bash
    tests=$(cat ./_CDW_test_1.txt|awk "/^Test case:/{print $3}")
    for test in $tests; do
      unset err
      printf "\nTest $test ... "
      err=$(cat ./_CDW_test_1.txt|awk "/^Test case: $test/,/^==/")
      printf "$err\n"|grep -qi error
      if [ $? -ne 0 ]; then
        echo OK
      else
        echo ERRORS!
        printf "$err\n"
      fi
    done

  10. #9
    Trusted Penguin
    Join Date
    May 2011
    Posts
    4,353
    Your script works for me.

    Do you maybe have DOS/Windows carriage returns in the log file?

    do this to your text file:

    Code:
    cat -v _CDW_test_1.txt
    If you see a bunch of ^M characters at the end of each line, that's your problem.

    To fix it, either edit/create the file in Linux, or use dos2unix to convert it, e.g.:
    Code:
    dos2unix _CDW_test_1.txt
    You may need to install the dos2unix package first...

  11. #10
    Just Joined!
    Join Date
    Nov 2011
    Posts
    12
    Hey thanks for the help!

    You were right about the issue but the ^M characters were actually in the script file itself.;

    Thanks again, this made my life easier by cutting down a 1000 page error file into about 10 pages!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •