Serious speed issues on server

[TheMikeh]

Hi,

I have an RPS from a OVH hosting and I use it to run approx 20 websites that are my own personal sites or belong to friends/family, now I have WHM/Cpanel installed for ease of use and its running CentOs 5.5.

Recently (past 4-5 days) the server has run down to a crawl, its forever overloaded and I dont know why as user loads, services etc havnt changed (or that Im aware of). Anyhow ... looking at the 'top' I get


top - 12:47:01 up 22:20, 1 user, load average: 8.63, 7.70, 7.15
Tasks: 5732 total, 4 running, 5728 sleeping, 0 stopped, 0 zombie
Cpu(s): 42.3%us, 39.3%sy, 0.0%ni, 4.0%id, 12.5%wa, 0.2%hi, 1.7%si, 0.0%st
Mem: 1022672k total, 851896k used, 170776k free, 4344k buffers
Swap: 1952692k total, 497184k used, 1455508k free, 29184k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26513 root 20 0 2172 824 708 R 18.7 0.1 0:00.68 ps
22245 root 20 0 5540 4304 792 R 17.1 0.4 0:27.44 top
30655 squid 20 0 38220 22m 3084 S 12.9 2.2 2:26.11 squid
26546 root 20 0 2172 820 708 R 11.0 0.1 0:00.40 ps


If memory serves me correctly, before the CPu was never ever this high, also the number of sleeping processes seems like quite an alarming number as well, is this normal?

Considering the server has 4 intel atom cores and 1gb of memory I cannot see why it is struggling with a handful of small low volume websites.

Now Im being 100% honest here as Im a complete linux beginner, but where do I begin here? doing a ps faux gives row after row of data (thousands in fact) and my only access to the rig is via Putty/SSH ...

So any ideas what the next step I should take is? WHM is pretty much unusable taking over 3 minutes to load up then 40-100 secs per page change! so stuck to purely SSH control, which even then lags a lot.

Any log files worth checking? any whizzy commands to help locate telltale or rogue processes?

Help

Thanks a lot
Mike

[coopstah13]

You have 500mb of swap in use, that is a lot. Swapping will severely slow down any system. Open top and sort by memory usage and look at the processes that are using the most memory.

[TheMikeh]

Ive just done the TOP and pressed M to sort by memory, the largest is SQUID at 8% the rest above 1% seem to be SPAMD. Ive tried turning SQUID on and off (server squid stop, service squid start) and it the server load is still high regardless of the service status hence why i leave it running.

The server is a little bit more responsive atm, the TOP command is showing


top - 14:39:43 up 1 day, 12 min, 1 user, load average: 11.92, 12.43, 12.77
Tasks: 5500 total, 2 running, 5498 sleeping, 0 stopped, 0 zombie
Cpu(s): 25.7%us, 23.2%sy, 0.0%ni, 4.7%id, 43.0%wa, 0.6%hi, 2.8%si, 0.0%st
Mem: 1022672k total, 871420k used, 151252k free, 5504k buffers
Swap: 1952692k total, 581752k used, 1370940k free, 40896k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30655 squid 20 0 54360 36m 2732 D 20.3 3.7 16:03.25 squid
23126 root 20 0 37820 27m 2404 S 0.0 2.7 0:18.80 spamd
30492 root 20 0 36624 26m 2344 S 0.0 2.6 0:22.82 spamd
31168 root 39 19 67728 20m 7220 D 0.0 2.1 0:06.56 yum-updatesd-he
23216 named 20 0 75220 18m 1324 S 2.0 1.9 3:06.42 named
303 nobody 20 0 36332 9.9m 2244 S 0.0 1.0 0:01.60 httpd
3049 root 20 0 15120 9988 1680 S 0.6 1.0 0:00.58 tailwatchd
26783 nobody 20 0 36608 9872 2280 S 0.0 1.0 0:01.32 httpd
29765 nobody 20 0 36068 9408 2352 S 0.0 0.9 0:01.32 httpd
27940 nobody 20 0 35312 9372 2280 S 0.0 0.9 0:01.38 httpd
25303 nobody 20 0 34548 8936 2428 S 0.0 0.9 0:00.46 httpd
9791 nobody 20 0 37124 7740 2328 S 0.0 0.8 0:00.70 httpd
29726 nobody 20 0 36068 6780 2204 S 0.0 0.7 0:00.82 httpd
30079 root 20 0 31748 5656 2124 S 0.0 0.6 0:07.58 spamd
25403 mysql 20 0 102m 4892 2456 S 0.3 0.5 0:07.62 mysqld

[coopstah13]

run this command and post results

vmstat -n 5 10

[TheMikeh]

Just run that and got an output of

root@atom2 [/]# vmstat -n 5 10
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 3 487256 127484 7868 52068 64 25 97 42 52 7 20 21 15 43 0
1 1 486964 120840 7960 52600 166 0 218 221 1760 3111 20 20 33 27 0
0 6 487132 116400 8040 52820 338 166 378 303 2060 2739 18 18 21 43 0
1 12 487288 122060 8108 53080 31 38 59 138 2614 1743 7 9 1 83 0
2 13 486880 127144 8132 53372 152 20 166 110 2865 3449 24 27 0 49 0
2 9 486440 122092 8224 53656 271 42 312 177 1962 3832 29 31 0 40 0
3 9 486480 120756 8304 54056 222 96 254 419 1950 4332 35 35 4 27 0
1 7 486152 117368 8328 54428 147 0 177 102 1441 3170 22 23 5 50 0
1 6 486008 112804 8372 54660 200 42 265 179 1586 3412 19 19 20 41 0
1 7 485556 126152 7632 40296 157 33 206 146 2379 3611 21 22 10 47 0


Since I made the topic the load has died down a bit although still running a load average around 12 atm and feels sluggish in its response times.

[coopstah13]

Your problem is definitely swapping based on the output there. If you notice the 2nd to last column that is IOwait, that is a VERY high number, almost 50% of CPU time is spent waiting on IO. Your server needs more memory.

[coopstah13]

run this code

ps aux | tr -s ' ' | cut -f "4-6 11-" -d " " | awk '{if ($1 > 0) print}' | sort

it will print out a list of processes, it will show you which processes are using the most memory, the first column is the % of total memory

[TheMikeh]

Hi,

Thanks for that but the highest % is 3 the rest are all 0.1 ... The bit im struggling with was the server was running fine and as far as Im aware nothing has been installed to the rig that would cause the sudden increase in memory usage.

Thanks for the help so far.


PROBLEM SOLVED!! turned out to be a service called OCO which is a monitoring system by OVH, it kept spawning new process's and leaving them in a sleep state hence the 8000 sleeping processes. Contacted OVH and got them to reinstall OCO and now its purring along properly.

Thanks for all the help/advise, it was your input above that led me to finding the OCO instances.