work on memory

[chesterman]

hi folks,

some days ago i had to write a script to parse a giant log file and write some entries in other file. the script was working fine (was just an egrep and a echo plus some more little things) but was taking hours to parse each log file. a friend of mine did the same procedure in java and the parse ran superfast. the logic was the same, and analizing, we concluded that java ran faster cause worked with the informantion in memory to write to the disc after the procedure, when the bash script was writing to the disc after every cicle.

is there a way, in bash, to work in memory, doing some kind of mmap, and write in the disc after a while?

forgive my poor english and tks in advance

[Rubberman]

You could write to an array in a loop, incrementing the index after each loop. Then when done, iterate the loop and echo (with concat) to the output file. However, that is likely just as slow as your current method. The major time factor here is the open, move to file end, then close the file. The writing is the least significant factor here, as you have learned. Shell scripts using egrep/sed/awk/echo et al are great for small jobs, but for bigger ones, the file system overhead of opening, reading, executing so many on-disk commands, even concidering that most of them are in cache most of the time, can be excessive. For processing large log files, use more appropriate tools, such as Java, C/C++, Perl, Python, TCL, etc.