Find the answer to your Linux question:
Page 1 of 3 1 2 3 LastLast
Results 1 to 10 of 22
I've only been a member here for a little while, but I think this will be my first post. I've searched the Internet for an answer to my problem, and ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    Mar 2005
    Posts
    13

    Alright I'm dying here - Apache help? MySQL? PHP?


    I've only been a member here for a little while, but I think this will be my first post. I've searched the Internet for an answer to my problem, and I've searched around here, but I just can't find an answer. I should also mention that my knowledge of Linux, Apache, MySQL and PHP are pretty limited, so if you have any help to offer for my problem, please be patient with me!

    I run a web forum (PHP/MySQL - phpBB) that has anywhere between 30 and 50 users connected at any given time of the day, sometimes up to 100 or so. Normally, the server runs pretty smoothly and even with 200 users, the server just chugs along happily handing out requests to anyone that comes along. I've been having a problem recently, however, with one user in particular whos very presence seems to bring my server to a screeching halt. Hes not a malicious user, and in fact, hes a long-time member, but he recently got a new job and when surfing from work, he somehow manages to break my site.

    When he surfs the pages and makes his database requests for pages, searches, etc, it seems that he ends up opening dozens, sometimes thirty or forty simultaneous connections to the site. When I do a netstat a, I get a stream of sockets open by him, all sitting in CLOSE_WAIT. It looks like this, essentially:

    Code:
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State
    tcp        0      0 *:smtp                  *:*                     LISTEN
    tcp        0      0 *:https                 *:*                     LISTEN
    tcp        0      0 *:20000                 *:*                     LISTEN
    tcp        0      0 *:imaps                 *:*                     LISTEN
    tcp        0      0 *:pop3s                 *:*                     LISTEN
    tcp        0      0 *:mysql                 *:*                     LISTEN
    tcp        0      0 *:pop3                  *:*                     LISTEN
    tcp        0      0 *:imap                  *:*                     LISTEN
    tcp        0      0 *:http                  *:*                     LISTEN
    tcp        0      0 *:ftp                   *:*                     LISTEN
    tcp        0      0 *:ssh                   *:*                     LISTEN
    tcp        0      0 mywebsite.com:http   server4.samford.e:60650 TIME_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37633  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57270 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37634  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57269 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37635  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:35771 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:35770 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37646  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57272 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37640  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:35773 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:33959 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:33958 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37650  TIME_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37651  TIME_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37662  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:33961 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37656  TIME_WAIT
    tcp        0      0 mywebsite.com:http   67.111.213.3.ptr.:50018 ESTABLISHED
    tcp        0      0 mywebsite.com:http   mywebsite.com:37664  TIME_WAIT
    tcp        0      0 mywebsite.com:http   mywebsite.com:37666  TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:61845 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   server4.samford.e:57025 TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57244 CLOSE_WAIT
    tcp        0      0 mywebsite.com:http   server4.samford.e:58074 TIME_WAIT
    tcp        0      0 mywebsite.com:http   server4.samford.e:58076 TIME_WAIT
    tcp        0      0 mywebsite.com:http   server4.samford.e:58078 TIME_WAIT
    tcp        0      0 mywebsite.com:http   server4.samford.e:56273 TIME_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57587 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57586 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57585 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57584 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57590 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57588 CLOSE_WAIT
    tcp        1  14480 mywebsite.com:http   dsds1bcx02.bankon:57595 CLOSE_WAIT
    etc...

    Hes obviously the bankon connections. What exactly is going on here? Do I have a configuration problem or is his work doing something strange that purposefully brings servers to their knees? Any other information that I can provide for you to help diagnose this problem? Im running out of options here, and Id rather not block the IP address, but Ive exhausted the limits of the settings I know how to tinker with, and my other users are starting to get a little irate.

    Thanks in advance for any help you can provide!

    - Nigel

  2. #2
    Linux Newbie
    Join Date
    Dec 2004
    Location
    Barrie, Ontario
    Posts
    219
    I've seen poorly configured proxies do things like that, and end up unintentionally DOSing servers in the process. If that ends up the case, all you can do is complain to their IT department, but that might get your friend in trouble if your site isn't work related...
    Blog - KB5UMQ - Linux User #272983
    3 Rules:
    1) "It doesn't work..." is simply not useful information.
    2) Don't cross post!
    3) If you are asking for help, start by telling us your distro/os and version.

  3. #3
    Linux Guru kkubasik's Avatar
    Join Date
    Mar 2004
    Location
    Lat: 39:03:51N Lon: 77:14:37W
    Posts
    2,396
    another possibility is that he has enabled the lovely pipelining feature in firefox, but incorrectly, If i remember correctly, the max number of connections to one server should never be beyond 8(?) but many think that bumping that number much much higher will increase performance (when infact it severly limits it)

    Without confronting him, you could just shorten your keepalive, or a better idea would be to ask him if hes using a proxie if not, inquire as to his browser/any recent changes to his browsing configuration.

    I think the proxy is more likely (especialy if it is only when he logs on from work, as proxies at the office are nothing new) While it is more a problem with his proxy, it seems unlikely you could do much to change that. Best bet in my opinion is to kill idle connections faster.
    Avoid the Gates of Hell. Use Linux
    A Penny for your Thoughts

    Formerly Known as qub333

  4. #4
    Linux Engineer Giro's Avatar
    Join Date
    Jul 2003
    Location
    England
    Posts
    1,219

  5. #5
    Just Joined!
    Join Date
    Mar 2005
    Posts
    13
    Quote Originally Posted by Kode
    I've seen poorly configured proxies do things like that, and end up unintentionally DOSing servers in the process. If that ends up the case, all you can do is complain to their IT department, but that might get your friend in trouble if your site isn't work related...
    That sounds like the most likely to me, because all of the configuration changes I've made haven't seem to have done anything. The important parts of my httpd.conf look like this:

    MaxClients 179
    StartServers 1
    Port 80
    Timeout 60
    KeepAlive on
    MaxKeepAliveRequests 5
    KeepAliveTimeout 5
    MinSpareServers 1
    MaxSpareServers 10
    MaxRequestsPerChild 10

    My sysctl.conf looks like this:

    net.ipv4.tcp_fin_timeout = 5
    net.ipv4.tcp_keepalive_time = 5
    net.ipv4.tcp_window_scaling = 0
    net.ipv4.tcp_sack = 0
    net.ipv4.tcp_timestamps = 0
    net.ipv4.ip_forward = 0
    net.ipv4.conf.all.rp_filter = 1
    kernel.sysrq = 0

    I really don't know what other settings are relevant here and I can't figure out anything else that would help me out. Kepalive on and off and with a wide range of settings hasn't helped, am I doing something wrong? And yes, I'm a relative n00b here, but I do know to stop and restart apache after I make changes.

    Oh, and the user in question is using IE6.x, for reference.

  6. #6
    Just Joined!
    Join Date
    Mar 2005
    Posts
    13
    Anything?

    Bump.

  7. #7
    Linux Guru kkubasik's Avatar
    Join Date
    Mar 2004
    Location
    Lat: 39:03:51N Lon: 77:14:37W
    Posts
    2,396
    Have you asked the user if he is behing a proxy or has enabled pipelining type features in his browser?
    Avoid the Gates of Hell. Use Linux
    A Penny for your Thoughts

    Formerly Known as qub333

  8. #8
    Linux Engineer Giro's Avatar
    Join Date
    Jul 2003
    Location
    England
    Posts
    1,219
    Quote Originally Posted by SamuraiNigel
    And yes, I'm a relative n00b here, but I do know to stop and restart apache after I make changes.
    This is why the changes arent taking affect, normally its either /etc/init.d/apache restart or /etc/init.d/httpd restart

  9. #9
    scm
    scm is offline
    Linux Engineer
    Join Date
    Feb 2005
    Posts
    1,044
    Quote Originally Posted by Giro
    Quote Originally Posted by SamuraiNigel
    And yes, I'm a relative n00b here, but I do know to stop and restart apache after I make changes.
    This is why the changes arent taking affect, normally its either /etc/init.d/apache restart or /etc/init.d/httpd restart
    Eh? He's said he does know he has to restart apache after making changes![/b]

  10. #10
    Just Joined!
    Join Date
    Mar 2005
    Posts
    13
    Quote Originally Posted by qub333
    Have you asked the user if he is behing a proxy or has enabled pipelining type features in his browser?
    Yeah, sorry about that. He's almost certainly behind a proxy, as he's working at a large company who's certainly got thousands of internal users, but he's not 100% sure of this of course. He's not using any pipelining type features in his browser, and is definitely using IE6.x. He's unable to try a different browser either, as he's working on a managed company machine.

    This is why the changes arent taking affect, normally its either /etc/init.d/apache restart or /etc/init.d/httpd restart
    Does stopping and starting the server differ from doing the "restart" command?

    Some random searching about the web over the last few days have uncovered that IE sends TCP requests differently than other browsers, namely it sends the actual request before it sends a SYN/ACK, hoping that there's an IIS server on the other end. Apparently, IE and IIS don't bother with SYN and ACK right away, so it sends the actual requests first, to make IE seem faster than other browsers on IIS servers.

    Another blurb I found while looking up proxy related unintentional/intentionall DoS attacks was this:

    The TCP specification does not specify clearly certain transitions. As an example, suppose an attacker sends a TCP segment with both the SYN and the FIN bit set. Depending on the TCP implementation, victim host processes the SYN flag first, generates a reply packet with the corresponding ACK flag set, and perform a state-transition to the state SYN_RCVD. Victim host then processes the FIN flag, performs a transition to the state CLOSE_WAIT, and sends the ACK packet back to attacker. The attacking host does not send any other packet to victim. TCP state machine in the victim for this connection is in CLOSE_WAIT state. The victim connection gets stuck in this state until the expiry of the keep-alive timer.
    Could this be what's happening? If so, how would I prevent it?

    Last, someone suggested that "squid" may be the solution I'm looking for, but I don't really understand how that would solve my problem here, and I'm not about to battle through a squid install and configuration for nothing.

    So, back to square one, I guess.

Page 1 of 3 1 2 3 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •