best methodology to debug a Linux daemon written in C++

[aryan_]

Hi All,

I have a bunch of C++ application which runs as daemons in Linux.

Furthermore, since these daemons rely on each other, they use IPC (inter-process communication) to communicate with each other.

These daemons are compiled with -g -O0 compiler flags, and started in right order using a shell script.

These daemons seems to have buggy behaviour and "sometimes" crash.

Another reason for it to crash might be corrupted input data that it processes.

To find out about the real cause, I tried both gdb and Valgrind to debug it.

I used --leak-check=yes and --log-file=foo.txt options with Valgrind.

However, Valgrind did not report any invalid write/read or uninitialized variable errors which may have caused the crash. FAQ on Valgrind's official website says that this is the nature of Valgrind that you can not change such that it can not replicate the native execution environment.

What is the best debugger options to use with Valgrind or gdb to debug a Linux daemon ?

Should I attach Valgrind or gdb to already running process ? or Is it ok to start the daemon with Valgrind in shell script ?

Thanks.

[Rubberman]

I normally attach gdb to the process id of the daemon in question. I also write my servers/daemons with variable trace levels so I can at run time increase the diagnostic output to help narrow the possible locations of the bug, or to identify what may be causing it (malformed data, etc). Interestingly enough, the error trapping and tracing code took up a considerable portion of the code base, though when you have servers which need to run 24x365 and downtime costs the user about $10M USD per hour, this is not an optional exercise.

[aryan_]

I also write my servers/daemons with variable trace levels so I can at run time increase the diagnostic output to help narrow the possible locations of the bug, or to identify what may be causing it (malformed data, etc).

I also wrote my daemons with variable trace levels, which can be configured at run time via a conf file. I use syslog() facility to output diagnostic messages.

Is syslog() mechanism the only/best way of outputing diagnostic messages ?

When you call syslog() system function in your application, you send the message to syslog daemon, and I am not sure how fast syslog daemon handles these messages.

[Rubberman]

There are other logging mechanisms as well, such as log4c (similar in function to log4j, the java logger). I haven't had much chance to see how they perform on Linux systems as yet. Most of my work in the past was on big iron Unix systems (HP, DEC, Sun) running enterprise manufacturing execution systems software for 1000's of users. We used log4c as well as our own trace logger. The main impact on performance of any of the trace loggers was due to the volume of messages being logged, so normal trace levels were set to "error messages only". Really gnarly problems could cause the admins to ratchet the level up to "trace all function entries/exits", which naturally caused the server so set to really slow down, but still didn't affect the system as a whole unless other processes were sending trace messages to the same log server. Since our log servers were single-threaded, this could be a problem if the admins didn't assign a private log server to the process being diagnosed at such a verbose level. On that subject, I'm not certain whether the syslog() facility is multi-threaded or not. It could become a cause of resource contention I would suspect, but don't know for sure.