Find the answer to your Linux question:
Results 1 to 7 of 7
Hi, I have 3 c files(one of them forks out 5 instances) all writing to one log file. Now to avoid the confusion of opening and closing in each application ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Apr 2010
    Posts
    18

    Writing to log file affecting performance


    Hi,

    I have 3 c files(one of them forks out 5 instances) all writing to one log file. Now to avoid the confusion of opening and closing in each application or instance and running into a situation of not having closed a file I decided to open and close the log file inside the log function for each write.

    So what I do currently is fopen, flock, fwrite, unlock, fclose for each write. All the log messages from all the files get written fine and there are no errors but I see a performance hit. The applications talk to each other using SHM(shared memory). So when I try to set a timer and check number of messages lets say I get X messages. Each time I remove or add a log call the number of messages changes. When it is a 1 sec or 5 sec timer it doesnt make a very big diff..few hundreds but when I check it over a longer period..every log call added decreases 1000 messages in count. So I want to know what is an efficient way of implementing the custom log across the application.

    Thanks,

  2. #2
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,410
    Why reinvent the wheel?
    Logging to syslog might be less painful.
    You must always face the curtain with a bow.

  3. #3
    Linux Guru gogalthorp's Avatar
    Join Date
    Oct 2006
    Location
    West (by God) Virginia
    Posts
    3,104
    Opening files is expensive. Open shared then lock , write and unlock as needed.

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined!
    Join Date
    Apr 2010
    Posts
    18
    The thing with syslog is I want the file custom to my application ONLY and not in one central place where I find system logs as well as logs from many other apps.

    Can you please elaborate on sharing the file,lock, write ,unlock? how should I go about that.. do you mean open the file in one main place..how do I then share it and ensure it is closed as I have to account for each instance of the application having to end abruptly and not returning back to main and doing a file close

  6. #5
    Linux Guru gogalthorp's Avatar
    Join Date
    Oct 2006
    Location
    West (by God) Virginia
    Posts
    3,104
    Each thread process opens the file for itself and keeps it open until it exits. It is called file sharing. This will save all the overhead of opening and closing files for each operation. Pass the file handle to the reporting function. When locking the file be sure to test that the lock succeeds or wait until it does to prevent multiple process from writing at the same time.

  7. #6
    Trusted Penguin Irithori's Avatar
    Join Date
    May 2009
    Location
    Munich
    Posts
    3,410
    ok, so you log yourself. No problem

    Just for sake of completeness:
    It is easily possible to tell syslogd to write to a different location/file for each process that connects.
    You must always face the curtain with a bow.

  8. #7
    Just Joined!
    Join Date
    Apr 2010
    Posts
    18
    I just tried implementing the changes to open and close in each appl and just lock,write unlock in the function but that also has the same effect. I have used such a log function before but never seen a performance problem. Ofcourse not in SHM IPC. But if it works with socket programs and other stuff I dont know why this should be any different.

    Irithori, I did not know syslog can be configured to drop the log in your directory. I just read briefly about it and thought it would not give a clean log for debugging in the development process. I have been working on the custom log for sometime now. So instead of just going ahead and removing it I am going to just try to study a little bit more why this causes a problem and if it still doesnt work, I will be using syslog.

    P.S: I tested for 5 secs ..twice with the calls and twice removing 4 of them ...first set was 4000-4500 messages and second was 9000 messages

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •