High load and high cpu kernel usage

[enid]

Hello to all,

On one Debian GNU/Linux 4.0 server , running different servers like dns/bind, sendmail, apache etc, I'm having high load and with top command there is not anything abnormal, but with htop I can see that kernel cpu usage is getting around 100% for all the cores (showing the bars in red) and also the total load average of the server is getting above 100

The nr of processes and RAM usage seems ok.

Where can I look for any problem related with this?

Thanks,
Enid

[Mudgen]

Are you running a graphical desktop on the console? I.E. runlevel 5? That's a gratuitous kernel hog on a server, and almost never really necessary. Try starting in runlevel 3 and I bet kernel cpu goes down to what you'd expect.

[jippie]

df -hP | column -t
(filesystems full? As a result of 15yrs UNIX experience, this is usually my default first check on problems with a server...)

log files? => /var/log/...
(sendmail will probably be telling you that is is waiting for the high load to fade, probalby other processes are complaining about something?)

dmesg | tail
(any errors?)

swapon -s
(swap space nearly exhausted?)

I'm not sure how a overheated CPU would look like in the log files, but once CPU throttling kicks in, you may just about see these symptoms. I know for a fact that not all motherboard report high-temperatures into logfiles. Check your fans and filters for dust.

You give is little to go on ... so I'm curious what it turnes out to be.

[enid]

@greyhairweenie
No it is running as default runlevel 2, and not any graphical desktop.

@jippie
The filesystems isn't full as df -hP | columnt -t shows the partitions with a lot off free space
The log files doesn't tell any specific problems, also dmesg | tail.

Only 500MB from 4GB were used for swap.

Yesterday the server went "Kernel panic" and the only thing to do was a force reboot, in dmesg I'm getting a lot of this msgs:
pop3[21475]: segfault at 0000000000000004 rip 000000000044333a rsp 00007fff7ffe3a10 error 4 (what can they be?)

Also I suspect that the high load have to do with the I/O operations with the HDD's (configured as Raid5), after the reboot until now it seems ok, but I'm monitoring it continually.

Thanks,
Enid

[jippie]

Yesterday the server went "Kernel panic" and the only thing to do was a force reboot, in dmesg I'm getting a lot of this msgs:
pop3[21475]: segfault at 0000000000000004 rip 000000000044333a rsp 00007fff7ffe3a10 error 4 (what can they be?)

I'd definately investigate the segfaults. The number between square brackets is the PID that crashed, the name of the process is pop3.

Also I suspect that the high load have to do with the I/O operations with the HDD's (configured as Raid5)

Raid5 shouldn't be the problem. Check the percentage of iowait in 'top', this is the percentage of cpu time that your server is waiting for disks to respond.


I think you use your time best investigating the segfaults. dmesg output should pretty much be silent after reboot. When you run into bad performance again: check dmesg again for the segfaults; check the sound/speed of the fans (hi-temp).
Try to stop the service that starts pop3d an see if it stabilizes the system.

[enid]

I did upgrade of dovecot because of the segfaults (with some difficulties because the server is in production) with the tar.gz version 2.0.3.

I upgraded the kernel also from the last vanilla kernel (2.6.35.5) , compiled and installed it.
Now the load has decreased significantly, also upgrading dovecot made the receiving of messages significantly faster.

Although the disk i/o problem seems not to have gone completely, I plan to upgrade RAM as well and see the differences.

Thanks,
Enid