Alright I'm dying here - Apache help? MySQL? PHP?

[SamuraiNigel]

I've only been a member here for a little while, but I think this will be my first post. I've searched the Internet for an answer to my problem, and I've searched around here, but I just can't find an answer. I should also mention that my knowledge of Linux, Apache, MySQL and PHP are pretty limited, so if you have any help to offer for my problem, please be patient with me!

I run a web forum (PHP/MySQL - phpBB) that has anywhere between 30 and 50 users connected at any given time of the day, sometimes up to 100 or so. Normally, the server runs pretty smoothly and even with 200 users, the server just chugs along happily handing out requests to anyone that comes along. I've been having a problem recently, however, with one user in particular who’s very presence seems to bring my server to a screeching halt. He’s not a malicious user, and in fact, he’s a long-time member, but he recently got a new job and when surfing from work, he somehow manages to break my site.

When he surfs the pages and makes his database requests for pages, searches, etc, it seems that he ends up opening dozens, sometimes thirty or forty simultaneous connections to the site. When I do a netstat –a, I get a stream of sockets open by him, all sitting in CLOSE_WAIT. It looks like this, essentially:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:smtp                  *:*                     LISTEN
tcp        0      0 *:https                 *:*                     LISTEN
tcp        0      0 *:20000                 *:*                     LISTEN
tcp        0      0 *:imaps                 *:*                     LISTEN
tcp        0      0 *:pop3s                 *:*                     LISTEN
tcp        0      0 *:mysql                 *:*                     LISTEN
tcp        0      0 *:pop3                  *:*                     LISTEN
tcp        0      0 *:imap                  *:*                     LISTEN
tcp        0      0 *:http                  *:*                     LISTEN
tcp        0      0 *:ftp                   *:*                     LISTEN
tcp        0      0 *:ssh                   *:*                     LISTEN
tcp        0      0 mywebsite.com:http   server4.samford.e:60650 TIME_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37633  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57270 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37634  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57269 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37635  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:35771 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:35770 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37646  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57272 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37640  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:35773 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:33959 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:33958 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37650  TIME_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37651  TIME_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37662  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:33961 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37656  TIME_WAIT
tcp        0      0 mywebsite.com:http   67.111.213.3.ptr.:50018 ESTABLISHED
tcp        0      0 mywebsite.com:http   mywebsite.com:37664  TIME_WAIT
tcp        0      0 mywebsite.com:http   mywebsite.com:37666  TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:61845 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   server4.samford.e:57025 TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57244 CLOSE_WAIT
tcp        0      0 mywebsite.com:http   server4.samford.e:58074 TIME_WAIT
tcp        0      0 mywebsite.com:http   server4.samford.e:58076 TIME_WAIT
tcp        0      0 mywebsite.com:http   server4.samford.e:58078 TIME_WAIT
tcp        0      0 mywebsite.com:http   server4.samford.e:56273 TIME_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57587 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57586 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57585 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57584 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57590 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57588 CLOSE_WAIT
tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57595 CLOSE_WAIT

etc...

He’s obviously the “bankon” connections. What exactly is going on here? Do I have a configuration problem or is his work doing something strange that purposefully brings servers to their knees? Any other information that I can provide for you to help diagnose this problem? I’m running out of options here, and I’d rather not block the IP address, but I’ve exhausted the limits of the settings I know how to tinker with, and my other users are starting to get a little irate.

Thanks in advance for any help you can provide!

- Nigel

[kode]

I've seen poorly configured proxies do things like that, and end up unintentionally DOSing servers in the process. If that ends up the case, all you can do is complain to their IT department, but that might get your friend in trouble if your site isn't work related...

[kkubasik]

another possibility is that he has enabled the lovely pipelining feature in firefox, but incorrectly, If i remember correctly, the max number of connections to one server should never be beyond 8(?) but many think that bumping that number much much higher will increase performance (when infact it severly limits it)

Without confronting him, you could just shorten your keepalive, or a better idea would be to ask him if hes using a proxie if not, inquire as to his browser/any recent changes to his browsing configuration.

I think the proxy is more likely (especialy if it is only when he logs on from work, as proxies at the office are nothing new) While it is more a problem with his proxy, it seems unlikely you could do much to change that. Best bet in my opinion is to kill idle connections faster.

[Giro]

Or just let Apache do it all http://httpd.apache.org/docs/mod/cor...paliverequests

[SamuraiNigel]

I've seen poorly configured proxies do things like that, and end up unintentionally DOSing servers in the process. If that ends up the case, all you can do is complain to their IT department, but that might get your friend in trouble if your site isn't work related...

That sounds like the most likely to me, because all of the configuration changes I've made haven't seem to have done anything. The important parts of my httpd.conf look like this:

MaxClients 179
StartServers 1
Port 80
Timeout 60
KeepAlive on
MaxKeepAliveRequests 5
KeepAliveTimeout 5
MinSpareServers 1
MaxSpareServers 10
MaxRequestsPerChild 10

My sysctl.conf looks like this:

net.ipv4.tcp_fin_timeout = 5
net.ipv4.tcp_keepalive_time = 5
net.ipv4.tcp_window_scaling = 0
net.ipv4.tcp_sack = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.ip_forward = 0
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 0

I really don't know what other settings are relevant here and I can't figure out anything else that would help me out. Kepalive on and off and with a wide range of settings hasn't helped, am I doing something wrong? And yes, I'm a relative n00b here, but I do know to stop and restart apache after I make changes.

Oh, and the user in question is using IE6.x, for reference.

[SamuraiNigel]

Anything?

Bump.

[kkubasik]

Have you asked the user if he is behing a proxy or has enabled pipelining type features in his browser?

[Giro]

And yes, I'm a relative n00b here, but I do know to stop and restart apache after I make changes.

This is why the changes arent taking affect, normally its either /etc/init.d/apache restart or /etc/init.d/httpd restart

[scm]

And yes, I'm a relative n00b here, but I do know to stop and restart apache after I make changes.

This is why the changes arent taking affect, normally its either /etc/init.d/apache restart or /etc/init.d/httpd restart

Eh? He's said he does know he has to restart apache after making changes![/b]

[SamuraiNigel]

Have you asked the user if he is behing a proxy or has enabled pipelining type features in his browser?

Yeah, sorry about that. He's almost certainly behind a proxy, as he's working at a large company who's certainly got thousands of internal users, but he's not 100% sure of this of course. He's not using any pipelining type features in his browser, and is definitely using IE6.x. He's unable to try a different browser either, as he's working on a managed company machine.

This is why the changes arent taking affect, normally its either /etc/init.d/apache restart or /etc/init.d/httpd restart

Does stopping and starting the server differ from doing the "restart" command?

Some random searching about the web over the last few days have uncovered that IE sends TCP requests differently than other browsers, namely it sends the actual request before it sends a SYN/ACK, hoping that there's an IIS server on the other end. Apparently, IE and IIS don't bother with SYN and ACK right away, so it sends the actual requests first, to make IE seem faster than other browsers on IIS servers.

Another blurb I found while looking up proxy related unintentional/intentionall DoS attacks was this:

The TCP specification does not specify clearly certain transitions. As an example, suppose an attacker sends a TCP segment with both the SYN and the FIN bit set. Depending on the TCP implementation, victim host processes the SYN flag first, generates a reply packet with the corresponding ACK flag set, and perform a state-transition to the state SYN_RCVD. Victim host then processes the FIN flag, performs a transition to the state CLOSE_WAIT, and sends the ACK packet back to attacker. The attacking host does not send any other packet to victim. TCP state machine in the victim for this connection is in CLOSE_WAIT state. The victim connection gets stuck in this state until the expiry of the keep-alive timer.

Could this be what's happening? If so, how would I prevent it?

Last, someone suggested that "squid" may be the solution I'm looking for, but I don't really understand how that would solve my problem here, and I'm not about to battle through a squid install and configuration for nothing.

So, back to square one, I guess.