Writing to log file affecting performance

[pv2010]

Hi,

I have 3 c files(one of them forks out 5 instances) all writing to one log file. Now to avoid the confusion of opening and closing in each application or instance and running into a situation of not having closed a file I decided to open and close the log file inside the log function for each write.

So what I do currently is fopen, flock, fwrite, unlock, fclose for each write. All the log messages from all the files get written fine and there are no errors but I see a performance hit. The applications talk to each other using SHM(shared memory). So when I try to set a timer and check number of messages lets say I get X messages. Each time I remove or add a log call the number of messages changes. When it is a 1 sec or 5 sec timer it doesnt make a very big diff..few hundreds but when I check it over a longer period..every log call added decreases 1000 messages in count. So I want to know what is an efficient way of implementing the custom log across the application.

Thanks,

[Irithori]

Why reinvent the wheel?
Logging to syslog might be less painful.

[gogalthorp]

Opening files is expensive. Open shared then lock , write and unlock as needed.

[pv2010]

The thing with syslog is I want the file custom to my application ONLY and not in one central place where I find system logs as well as logs from many other apps.

Can you please elaborate on sharing the file,lock, write ,unlock? how should I go about that.. do you mean open the file in one main place..how do I then share it and ensure it is closed as I have to account for each instance of the application having to end abruptly and not returning back to main and doing a file close

[gogalthorp]

Each thread process opens the file for itself and keeps it open until it exits. It is called file sharing. This will save all the overhead of opening and closing files for each operation. Pass the file handle to the reporting function. When locking the file be sure to test that the lock succeeds or wait until it does to prevent multiple process from writing at the same time.

[Irithori]

ok, so you log yourself. No problem

Just for sake of completeness:
It is easily possible to tell syslogd to write to a different location/file for each process that connects.

[pv2010]

I just tried implementing the changes to open and close in each appl and just lock,write unlock in the function but that also has the same effect. I have used such a log function before but never seen a performance problem. Ofcourse not in SHM IPC. But if it works with socket programs and other stuff I dont know why this should be any different.

Irithori, I did not know syslog can be configured to drop the log in your directory. I just read briefly about it and thought it would not give a clean log for debugging in the development process. I have been working on the custom log for sometime now. So instead of just going ahead and removing it I am going to just try to study a little bit more why this causes a problem and if it still doesnt work, I will be using syslog.

P.S: I tested for 5 secs ..twice with the calls and twice removing 4 of them ...first set was 4000-4500 messages and second was 9000 messages