Debugging Multithreaded Application

[Kaasa]

Hi everybody,
i am new to multi-threaded programming in c, i need to debug a application which is using many threads. i am using eclipse with GDB debugger but it is not possible to debug it since it has a lot of different threads and i can not not debug it as wre debug in case of single thread(main) . So i would like to know, is there a way to debug such an application, if there is some tutorial that would be nice to know.??
Thanks in advance.
regards.

[Kloschüssel]

Debugging a event based multi threaded application is always hard. You can't simply step through to find out the cause of a problem. Mostly you'll have to reconstruct the callstack and set breakpoints at potential candidates. Anyway, this works only where the code is wrong. In case you have race conditions or missing locks, you won't become happy with that either and you will have to use your brain. A good start is always a well designed, reentrant and well locked logging system. Even if it is slow, it makes sure that log messages pop up as they come in. If there you see two or more log lines alter from time to time where they should be always in the same order, it most probably is a race condition.

[Kaasa]

Actually my application does not have any error. But some how the behavior is not according to expectations, since it has alot of code in different threads i dont know where to put breakpoints. is there a tool to debug the whole application and see what is happening inside.

[Kloschüssel]

Actually my application does not have any error. But some how the behavior is not according to expectations [...]

Either a software behaves like it should or not, in which case it is erroneous.

"Well, it is not exactly what I wanted, but most of the times it does what I planned it to do. My software has no bugs, only features!"

Unfortunatly I cannot give you a hint of what could be wrong/going on without having a closer look. From my experience I can tell you that around 90% of all errors in multithreaded software it is a missing/misplaced synchronization or race condition.

[Kaasa]

From my experience I can tell you that around 90% of all errors in multithreaded software it is a missing/misplaced synchronization or race condition.

Do you know how to check these synchronization and race condition.?

Thanks

[Kloschüssel]

You can't by means of an automated debugger or tool. There are kind of software that analyze code semantically (see: checkstyle) and can also detect potential synchronization problems and plenty other stuff. Unfortunatly they all cook with water only and I - even lacking the formal proof - believe this kind of problem is NP-hard, thus, unsolvable in rational time by nature.

If you write data the best approach would be to identify the different processes and their data sources which they read / write from / to. Print a graph of everything and take a closer look at the parts where there are multiple processes writing to one single datasource. That would be a approach to find a producer/consumer problem.

If you do not write data, you may have calculations that should be synchronized (bank transaction problems). There a good approach is to find two exact positions A and B in your code where a calculation must go through A before it goes through B in order to function properly. At this point you may build in a logging facility and run load tests if this is always true. If the software behaves correctly it should always show in the logfile:

start A
end A
start B
end B
start B
end B
start A
end A
start B
end B
[..]

If you see something like:

start A
end A
start B
start A
start B
end B
[..]

You have one (or two) problem(s) to solve.

[lemons]

(gdb) start
...
(gdb) set scheduler-locking on
(gdb) step
(gdb) step
(gdb) step
...
[New Thread ...]
(gdb) thread 2
(gdb) step
(gdb) step
(gdb) step
...
(gdb) thread 1

GDB is a pain when debugging more than one thread, but luckily, using set scheduler-locking on you can stop it from letting other threads run, so keep running the main thread until it reaches a mutex or whatever, then use thread to switch to another thread until it calls 'the death event', and then switch back.

You do need to be careful though, because you can get yourself into deadlocks by using this.

[Kloschüssel]

If you prevent other threads from executing you tend to not raise any race conditions.

[markcole]

I too have found that logging is the best way to find and resolve these situations. There is no easy solution other than to just log and run it over and over to check that the execution paths are in the right order and that the data is being read/written at the correct time.

[naman_jain]

If i put a breakpoint in a code which is being parallely used by several threads , how will it break?