Awk and loads of files

[noadankes]

Hi, i have an awk srcipt that runs over a batch of log files. The script is running fine which is good!. However when i run the awk command i have to specify a number of files to be processed - for example i run

awk -f myawkfile.awk /adirectory/file1 /adirectory2/file2 etc etc

There are about 20 files that have to be added to the list of files to process and i have to enter these when i want to run the command. I have put them all into a shell script so i don't have to type them in every time but when i want to add or remove one of these files i have to alter the script.

Is there any way to either:
1. Tell awk to use a wild card and recursivly search from a directory to find the files. For example process all files that are *.log in any sub directories.
2. Tell awk to match files based on a regex for the file path.
3. Get a shell script to get the list of these files and then add them to the awk command.

Thanks

[ghostdog74]

usually, you can just do this:

awk -f myawk.awk /path/*.log

[Franklin52]

You can use find, try this:

find /yourDirname -name *log -exec "awk -f myawkfile.awk" {} \:

Regards

[Franklin52]

usually, you can just do this:

awk -f myawk.awk /path/*.log

This doesn't work recursively.

Regards

[anomie]

You can use find, try this:

find /yourDirname -name *log -exec "awk -f myawkfile.awk" {} \:

Regards

You're going to want to single-quote the -name parameter value, otherwise you risk expanding it on the command line when it's issued - i.e. '*log' instead of *log.

Also, that should be a semicolon at the end (not a colon).

[Franklin52]

You're going to want to single-quote the -name parameter value, otherwise you risk expanding it on the command line when it's issued - i.e. '*log' instead of *log.

Also, that should be a semicolon at the end (not a colon).

Good comments, thanks for the attentiveness.

Regards

[ghostdog74]

This doesn't work recursively.

Regards

how is this not so?

#more test1.log
this is first line of test1.log
second line of test1.log
last line of test1.log
#more test2.log
this is first line of test2.log
second line of test2.log
last line of test2.log
#more test3.log
this is first line of test3.log
second line of test3.log
last line of test3.log
#ls test[123]
test1:
test1.log

test2:
test2.log

test3:
test3.log
#awk '{print "Now in " FILENAME;print}' test[123]/*.log
Now in test1/test1.log
this is first line of test1.log
Now in test1/test1.log
second line of test1.log
Now in test1/test1.log
last line of test1.log
Now in test2/test2.log
this is first line of test2.log
Now in test2/test2.log
second line of test2.log
Now in test2/test2.log
last line of test2.log
Now in test3/test3.log
this is first line of test3.log
Now in test3/test3.log
second line of test3.log
Now in test3/test3.log
last line of test3.log

seeing that OP has around 20 plus files to process, it might be faster to just use shell's expansion (assuming that OP knows where these log files are created.) rather than find which may take longer if the files are not in the directories and it so happens there are many directories to traverse.

[Franklin52]

how is this not so?

That means that it only works in the current directory not in the subdirectories of the mentioned path.

Regards

[noadankes]

Its been a help.

I am interested in knowing more about the 'expansion' mentioned. I have previously tried this and it has not worked for me.

If i have these directories that contain my filed

server-15/de/logfile.txt
server-15/uk/logfile.txt
server-16/de/logfile.txt
server-16/uk/logfile.txt

How can i use expansion in this? I thought that i would be able to somthing like this
server-[15|16]/[uk|de]/logfile.txt
But then i get problems and i think its because of the pipes?

[anomie]

I think you want something like this:

$ ls server-{15,16}/{uk,de}/logfile.txt

(Not that you want to use 'ls', but that's an example using expansion.)

I haven't tested that - you may need to tweak it.