LAMP server very slow

[littlebigman]

Hello

Still investigating why this two-Apache + one-MySQL setup is so slow after a few hundred simultaneours users are pounding on it.

For the experts out there, what would you say about this output?

Apache #1 (includes load balancer to forward connections to Apache #2)

top - 19:35:21 up 26 days,  2:21,  3 users,  load average: 165.76, 163.97, 200.46
Tasks: 829 total, 155 running, 673 sleeping,   1 stopped,   0 zombie
Cpu(s): 36.3%us, 10.2%sy,  0.0%ni, 47.2%id,  5.1%wa,  0.1%hi,  1.0%si,  0.0%st
Mem:   2070172k total,  1978392k used,    91780k free,    48016k buffers
Swap:   998376k total,   244336k used,   754040k free,   518996k cached

Apache #2

top - 19:33:46 up 21 days,  3:23,  3 users,  load average: 511.90, 468.51, 351.50
Tasks: 1325 total, 105 running, 1219 sleeping,   1 stopped,   0 zombie
Cpu(s): 28.5%us, 65.1%sy,  0.0%ni,  0.0%id,  0.0%wa,  1.1%hi,  5.4%si,  0.0%st
Mem:   2061660k total,  2012144k used,    49516k free,     2084k buffers
Swap:   922600k total,   430876k used,   491724k free,   104796k cached

MySQL server

top - 19:36:34 up 26 days,  3:19,  4 users,  load average: 5.32, 4.50, 3.53
Tasks: 116 total,   1 running, 115 sleeping,   0 stopped,   0 zombie
Cpu(s): 31.3%us, 11.5%sy,  0.0%ni, 55.6%id,  0.2%wa,  0.2%hi,  1.2%si,  0.0%st
Mem:   2070684k total,  1995300k used,    75384k free,    24036k buffers
Swap:   361448k total,        0k used,   361448k free,  1012184k cached

"ps -aux | grep apache | head -n 20" on Apache #1
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
root      3221  0.0  0.4 156564  9064 ?        Ss   19:16   0:01 /usr/sbin/apache2 -k start
www-data  7692  1.7  0.6 157200 12724 ?        S    19:58   0:12 /usr/sbin/apache2 -k start
www-data  7778  1.4  0.6 159508 13312 ?        S    19:58   0:09 /usr/sbin/apache2 -k start
www-data  7801  0.8  0.6 157336 14308 ?        S    19:58   0:05 /usr/sbin/apache2 -k start
www-data  7835  0.8  0.6 158836 13768 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7836  0.6  0.8 157264 17444 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7837  0.6  0.7 159096 14952 ?        S    20:00   0:03 /usr/sbin/apache2 -k start
www-data  7868  0.7  0.7 159080 15504 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7873  0.7  0.6 157456 13500 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7878  0.4  0.5 159076 11868 ?        S    20:00   0:02 /usr/sbin/apache2 -k start
www-data  7883  0.7  0.6 159096 13672 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7888  0.7  0.7 157460 15772 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7922  0.9  0.7 157444 15316 ?        S    20:00   0:05 /usr/sbin/apache2 -k start
www-data  7928  0.7  0.5 159244 12172 ?        S    20:00   0:04 /usr/sbin/apache2 -k start
www-data  7943  1.0  0.7 157584 15676 ?        S    20:00   0:06 /usr/sbin/apache2 -k start
www-data  7947  0.9  0.7 159252 15644 ?        S    20:00   0:05 /usr/sbin/apache2 -k start
www-data  7949  0.5  0.6 157180 14460 ?        S    20:00   0:03 /usr/sbin/apache2 -k start
www-data  7951  1.5  0.6 159544 14376 ?        S    20:00   0:09 /usr/sbin/apache2 -k start
www-data  7966  0.6  0.7 157444 16424 ?        S    20:00   0:03 /usr/sbin/apache2 -k start
www-data  7972  0.5  0.7 157316 15288 ?        S    20:00   0:03 /usr/sbin/apache2 -k start

What I notice:
1. Huge load average on those dual-core PC's
2. Apache #1 (also acting as load balancer) is 50% idle, while #2 is 0% idle
3. Apache #2 has 5% I/O wait, and spending 65% running kernel apps
4. Both Apache use swap

Thanks for any tip.

PS: Is there a way to just show user applications, and hide kernel apps like init, kthreadd, etc. when running "ps aux"?

[davidanand]

Hello,
Is it an plain red hat box or ...some other flavor I guess it is an Ubuntu
It seems that many connections in Apache if the load goes up you get temporary errors
try to reduce the load .... try killing the php apps
1. run an rkunter
2. compile Mod security and Mod evasive
3.make good firewall rules
4.imply APF or CSF firewall
5.check for the ips whick has max conn in server block that..
Get back with the results....

[littlebigman]

Thanks much for the help.

Is it an plain red hat box or ...some other flavor I guess it is an Ubuntu

Sorry, forgot to mention it's Debian.

It seems that many apps trys to restart your apache ...

Because of the long list of "/usr/sbin/apache2 -k start", while Apache2 should only have a couple of processes, and use threads to serve connections from within processes?

Do you think someone's intentionally trying to slow it down? FWIW, performance is only a problem in the evening, with users > 500; It's responsive below this.

5.check for the ips whick has max conn in server block that..

I didn't catch this :-/

Thank you.

[davidanand]

Hi,

I have edited the reply plz check ..
"It seems that many apps trys to restart your apache "
2. I asked you to check for ip's accessing more resources ...
such as check the command below ( I am not sure about Debian)

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

The result will be ips using the resources such as
30 198.162.10.10
30 is the no of connection from the server block the ip might be an DDOS .

[littlebigman]

I'll check it out, but I don't think it's a DoS. Outside the rush hour in the evenings, the site is pretty responsive. The hoster tells us that it's the NFS and disk access that are the bottleneck. I'll also check the Apache and PHP configuration.

BTW, is mod_proxy (or whatever Apache module is used to provide load balancing) on par with dedicated boxes?

Thanks.

[davidanand]

hi ,

check the following ->
mod_proxy.c
mod_proxy_connect.c
mod_proxy_ftp.c
mod_proxy_http.c
mod_proxy_ajp.c
mod_proxy_balancer.c
also
mod_backhand: A load balancing module for the Apache web server

[littlebigman]

Will do. Thanks.

[davidanand]

let me know the result after compiling the result....sill the lamp is sluggish
go for eAccelerator...that will help