crazy load average

[melonheadjr44]

Hi guys,

I need help with one of my servers.

The load average goes nuts every so often.

top - 10:19:02 up 1 day, 23:29, 2 users, load average: 55.33, 53.05, 41.77

13887 mysql 16 0 118m 25m 1576 S 99.9 1.3 15:23.78 mysqld
17758 nobody 16 0 16388 8532 1104 R 80.8 0.4 0:34.29 httpd
17659 nobody 17 0 15292 6868 556 S 73.5 0.3 0:25.75 httpd
17978 nobody 17 0 16124 8588 960 R 70.6 0.4 0:29.57 httpd
17784 nobody 16 0 15608 8216 1008 S 68.5 0.4 0:18.50 httpd
17981 nobody 16 0 1204m 807m 612 R 68.5 39.9 3:00.73 httpd
17688 nobody 16 0 1204m 38m 584 R 66.4 1.9 5:26.19 httpd
17785 nobody 15 0 15796 8216 916 R 66.0 0.4 0:54.86 httpd
17560 nobody 16 0 2383m 91m 428 R 64.5 4.5 6:00.01 httpd
17895 nobody 15 0 15668 7828 1140 S 56.0 0.4 0:22.14 httpd

and then goes down back to below 1.00

There is ALOT of httpd processes.

In MySQL i checked the processlist, and when its below 1.00, theres about 20-30 connections.

When its over 20.0 theres over 119 connections.

I put the max connections to 200.

So my question is, how do I know if the problem is with MySQL or PHP, or anything else?

[Rubberman]

What is the system usage? Is this a web server, connected to the internet or internal network?

[melonheadjr44]

It's a web server with at least 1000 hits a day.

Ive been able to determine what the problem is ( not whats causing it ) but I dont know how to fix it.

Some HTTPD proccess are reaching 1.8 GIGS

22574 nobody 18 0 2392m 1.6g 1140 D 4.0 80.9 1:01.13 httpd

So I keep killing them.

Ive set the max size every where. Example in httpd.conf

LimitRequestBody,

in my.cnf and in php.ini

None seem to do anything.

Any ideas how to kill a proccess thats taking to much memory, and what could be causing it? And if it is a php script that we have, how do I determine which one?

[Rubberman]

Have you tried setting the ulimits for the system account that is running the threads? Anyway, without knowing more about your system, and what it is doing, there isn't much I can suggest. This behavior can well be "normal" for the mix of operations it is supporting.

[melonheadjr44]

How would i go about to changing the ulimit of "nobody" the task thats going over 1gig of memory?

[Rubberman]

Well, if you can change the ulimit for those processes and that "user", it is only treating the symptoms, not the cause. First, try to find out what they are doing that is causing this. The problem may be a coding error, or it may be an indication that someone is trying to pwn your system, probing it for buffer overflow conditions, etc.