Capturing an oops from my buggy kernel module

Hey guys, this is my first post so I'll try to make it a good one!

My setup: VMWare workstation running Ubuntu (Windows host).

I'm working on a kernel module that hooks into netfilter and modifies the behavior of TCP acks (pretty vague huh?!). It works fine for one or two TCP flows but when I have ten or more flows active at one time I get a crash (or a panic, I don't have my mind wrapped around the terminology yet). The crash looks like this: everything freezes and in the right hand corner i see "Numlock on/Numlock off" flashing. The whole thing is pretty much dead

At first I tried debugging using printk. For most bugs I've seen thus far printk seems to work fine, but this one is a bit more tricky. I figured my two options here are:

1.) Capture an oops message
2.) Remote debug using kgdb

I'd like to try the OOPS message first. I realize that because the system dies I need to use a serial console to capture the message. I setup a serial console between my windows host and the Ubuntu Virtual machine.

I generated the panic again from the serial console using wget, but I did not see any OOPS displayed in the console. My question is: How can I get the OOPS to appear on the serial console? Do I have to make some changes to how OOPS messages get logged?

Thanks in advance for any help you can provide!
-Tim

Chances are the system hung before you output OOPS. In any case, the terminal output is delayed, so by the time it was output from the kernel ring buffer your system is fubar.

Anyway, from what you are saying, it would seem to me that you have a race condition, and some sort of deadlock is happening in the kernel. Since a great deal of the system behavior depends upon the TCP/IP components, mucking with them is fraught with danger, Will Robinson! :-) So, without seeing exactly what you are doing, this is difficult to advise you on.

If Rubberman is correct (and I think he is) then using kgdb won't be of much value. Use of debuggers on multi-threaded apps or in this case the kernel, usually do not yield the same results when you single step through them by way of a debugger. This is one of the reasons why Linus Torvalds has been resistant to adding the necessary hooks to implement a full scale debugging capability within the kernel. It is much more effective to massage the problem and observe the outcome and then read the code.

One way I usually use when I can't see the output from an Oops is to go to the area of code I suspect and put in a conditional test and then a "panic()" call with whatever detailed information I may need. It's labourious but I can at least narrow the search down to the actual line of code that is causing the problem and then I can have a hard look at what's happening in the specific area of the problem. I've said it before, have the kernel source code handy in order to be able to look through the kernel's API calls so that you can really understand what's happening.


Good Luck!!

Cheers!!

And by the way, this was a good first post! Let us know what you find when you resolve this.

Thanks for the help guys (BTW, I'm in Chicago too Rubberman)! I thought that this may be the case, so it's back to printk! I guess Linus would be proud...

A race condition sounds like a good possibility. My shared circular buffer does not have any locks around it, so that may be the first thing to try out. Regardless, I'm glad I don't have to muck around with debuggers and OOPses any longer, getting that stuff setup is pretty frustrating.

I'll let you know what I find!

My time spent with it was long and difficult . . I switched to sun's Virtual Box MUCH better and does everything I need it to do . .Oh yeah from root type in VBOX_USB=USBFS VirtualBox And then install windblows and windblows crap . . It took me several days to find the above information in one of the forums and it works . . Be carefull as to what you make avaliable to windows So you do not shut off the web connection

Quote:

---

Originally Posted by lynne14

My time spent with it was long and difficult . . I switched to sun's Virtual Box MUCH better and does everything I need it to do . .Oh yeah from root type in VBOX_USB=USBFS VirtualBox And then install windblows and windblows crap . . It took me several days to find the above information in one of the forums and it works . . Be carefull as to what you make avaliable to windows So you do not shut off the web connection

---

Please keep on topic lynne14. This thread is about issues the poster is having with his kernel-level network code, not the virtual machine software he is using... :-(

It was on topic Virtual Box is much better and EASIER to run and will do the job inside of Ubuntu ! ! I know I did the bleeping VMware stuff and got completely away from it . . maybe it is why Virtual Box was made . . Does what the VMware was to do and faster and easier I do not know about better as I never got VMware to work so I could LOAD windblows

Rubberman, does the phrase "Without a clue" come to mind with you as well?

tmerrifi,

You are implementing a circular buffer, have you looked at the various data management tools available to you in the kernel? There is a highly optimized doubly linked list that is available. Have a look at the file list.h for the implementation. I've been using it a lot in my current driver but only as a stack for some strange stack oriented memory management system. I mention this only as a "By the Way" comment. I prefer to re-use as much code as possible since then I know what works and what may need to be examined more closely.

Cheers!!

Thanks for the recommendation bloggins...I'll take a look as soon as I can.

Also, doing some debugging tonight...I'm starting to see that my problem may be caused by kmalloc. Currently, I'm allowing blocking by using the GFP_KERNEL flag. However, when I make the switch to GFP_ATOMIC I get no crash (so far atleast).

I'm confused because my code is a module that uses the netfilter hooks, shouldn't that be allowed to block? It's not holding on to a lock and it's obviously not an interrupt handler or anything. Any thoughts?