high load average & apache cpu

[netsys]

Hi all,
I'm new and want a help, my problem is the high load average in my web server (chat) between 11pm and midnight.

Commands:

#uptime
10:43:04 up 76 days, 19:36, 2 users, load average: 254.49, 233.09, 196.61

#top
top - 10:43:42 up 76 days, 19:36, 2 users, load average: 243.21, 232.78, 197.86
Tasks: 1662 total, 6 running, 1656 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 8.8%sy, 0.0%ni, 85.3%id, 0.9%wa, 0.5%hi, 0.2%si, 0.0%st
Mem: 16620352k total, 6556200k used, 10064152k free, 182320k buffers
Swap: 4192924k total, 0k used, 4192924k free, 959768k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3155 www-data 20 0 349m 10m 5524 D 23 0.1 0:16.24 apache2
3610 www-data 20 0 349m 10m 5504 D 23 0.1 0:14.99 apache2
3868 www-data 20 0 349m 10m 5460 S 20 0.1 0:09.86 apache2
1607 root 20 0 3316 2000 800 R 15 0.0 0:00.09 top
3360 www-data 20 0 349m 10m 5372 D 13 0.1 0:13.69 apache2
3258 www-data 20 0 349m 11m 5772 D 12 0.1 0:10.80 apache2
4026 www-data 20 0 349m 10m 5448 D 12 0.1 0:17.26 apache2
4338 www-data 20 0 349m 11m 5924 D 12 0.1 0:08.67 apache2
2976 www-data 20 0 350m 11m 6072 D 10 0.1 0:14.92 apache2
3613 www-data 20 0 349m 10m 5356 D 10 0.1 0:23.06 apache2
3946 www-data 20 0 350m 11m 5968 D 10 0.1 0:16.96 apache2
4016 www-data 20 0 349m 9512 4172 D 10 0.1 0:12.14 apache2
2974 www-data 20 0 349m 10m 5804 D 8 0.1 0:14.25 apache2
3151 www-data 20 0 349m 10m 4956 S 8 0.1 0:23.74 apache2
3158 www-data 20 0 349m 9.9m 4580 R 8 0.1 0:20.47 apache2
3270 www-data 20 0 349m 9.9m 4632 D 8 0.1 0:15.51 apache2
3648 www-data 20 0 349m 11m 6116 D 8 0.1 0:12.69 apache2
2909 www-data 20 0 349m 10m 5560 D 7 0.1 0:13.04 apache2
2984 www-data 20 0 349m 11m 6176 D 7 0.1 0:12.66 apache2
3547 www-data 20 0 349m 10m 5280 D 7 0.1 0:12.93 apache2
3569 www-data 20 0 349m 11m 5940 D 7 0.1 0:18.98 apache2
4072 www-data 20 0 349m 10m 5776 D 7 0.1 0:22.67 apache2
4254 www-data 20 0 349m 11m 5964 D 7 0.1 0:15.71 apache2
2903 www-data 20 0 349m 9.8m 4592 D 5 0.1 0:14.12 apache2
3326 www-data 20 0 349m 10m 5560 D 5 0.1 0:08.22 apache2
2938 www-data 20 0 349m 10m 5296 D 3 0.1 0:15.71 apache2
2975 www-data 20 0 349m 10m 5400 R 3 0.1 0:18.22 apache2
3068 www-data 20 0 349m 11m 6008 R 3 0.1 0:09.98 apache2
3127 www-data 20 0 349m 10m 5156 D 3 0.1 0:16.32 apache2
3291 www-data 20 0 349m 10m 5508 D 3 0.1 0:12.81 apache2
3297 www-data 20 0 349m 10m 5388 D 3 0.1 0:21.41 apache2
3542 www-data 20 0 349m 10m 5664 D 3 0.1 0:30.68 apache2
3612 www-data 20 0 349m 11m 5804 D 3 0.1 0:12.70 apache2
3614 www-data 20 0 349m 10m 5180 D 3 0.1 0:11.38 apache2
3725 www-data 20 0 349m 9.9m 4616 D 3 0.1 0:16.29 apache2
3938 www-data 20 0 349m 10m 5600 D 3 0.1 0:11.76 apache2
2884 www-data 20 0 349m 10m 5516 D 2 0.1 0:17.94 apache2
2915 www-data 20 0 349m 11m 5992 D 2 0.1 0:14.36 apache2
2933 www-data 20 0 349m 10m 5296 D 2 0.1 0:16.54 apache2
2991 www-data 20 0 350m 10m 5388 D 2 0.1 0:11.71 apache2
3235 www-data 20 0 349m 10m 5564 D 2 0.1 0:11.79 apache2
3351 www-data 20 0 349m 9.8m 4632 D 2 0.1 0:16.09 apache2
3450 www-data 20 0 349m 10m 5336 D 2 0.1 0:13.59 apache2
3459 www-data 20 0 349m 10m 5664 S 2 0.1 0:11.04 apache2
3649 www-data 20 0 349m 11m 5904 D 2 0.1 0:13.84 apache2
3713 www-data 20 0 349m 10m 5368 D 2 0.1 0:11.54 apache2
3921 www-data 20 0 349m 9468 4160 S 2 0.1 0:12.11 apache2
4029 www-data 20 0 349m 10m 5616 R 2 0.1 0:15.65 apache2
4175 www-data 20 0 350m 10m 5428 S 2 0.1 0:18.37 apache2
4279 www-data 20 0 349m 11m 6284 R 2 0.1 0:14.14 apache2
5432 ircd 10 -10 64348 60m 1624 S 2 0.4 3432:01 ircd
1 root 20 0 1980 628 540 S 0 0.0 1:16.16 init
2 root 20 0 0 0 0 S 0 0.0 0:00.00 kthreadd
3 root RT 0 0 0 0 S 0 0.0 0:01.14 migration/0
4 root 20 0 0 0 0 S 0 0.0 0:22.70 ksoftirqd/0
5 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/0
6 root RT 0 0 0 0 S 0 0.0 5:29.75 migration/1
7 root 20 0 0 0 0 S 0 0.0 2:08.66 ksoftirqd/1
8 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
9 root RT 0 0 0 0 S 0 0.0 2:17.37 migration/2
10 root 20 0 0 0 0 S 0 0.0 6:12.33 ksoftirqd/2
11 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/2
12 root RT 0 0 0 0 S 0 0.0 2:18.08 migration/3
13 root 20 0 0 0 0 S 0 0.0 2:09.35 ksoftirqd/3

#apache2ctl status
Apache Server Status for localhost

Server Version: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny10 with Suhosin-Patch
Server Built: Dec 11 2010 18:58:55

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Current Time: Wednesday, 04-May-2011 10:44:27 CEST
Restart Time: Tuesday, 03-May-2011 18:22:55 CEST
Parent Server Generation: 3
Server uptime: 16 hours 21 minutes 32 seconds
Total accesses: 2340607 - Total Traffic: 2.9 GB
CPU Usage: u11634 s38800.8 cu3.54 cs0 - 85.6% CPU load
39.7 requests/sec - 51.1 kB/second - 1315 B/request
303 requests currently being processed, 1221 idle workers

W_____W____________W_WK__W__________WWW_W_W_____.. _._...____..W.
.__....__..__._W.._._..KW___._KW__W__...__....__._ _.._.____._.._
W.._._._W.K__W._.._._._...__._.___WW...__.._..W... ______._W._._.
_.__.._...___._W._...W____K_....____..W.._._._._._ ._._.___.____.
..._.___K__.__W__.._.___W..__..K_..___K_...._...__ ___...._____._
._____.K_W_.W.___.__.W__.._.______K___.W_.________ ._____W....__.
___W.__.__CW__._..._K__....__W.W..__.._.__K.___.W. .W..._._W_.__.
.._.___._..____W___W__..__W_...._W.___.K__..W_.__. ......__W__.K.
_____.W___K.____.W___..___._._K._W_.W..WK_W._..W__ __.__W...._..W
W__.__K__._._._W_._._..K._._W__WK._______.W..._W__ W_.._...___WW_
_..._._..W._.._W.._W.__._..__._..___.____.___._.K_ _W._._....__._
K___..K_..____.W___.__....WK_._._W__W_____._...W__ ..__W.__.W.._W
__W...__.__.R..._K._._._W__W.W.W....._._W_K___.._. _.K.W_..WK__.W
______..._._._W__._..W__...___.__..._.__.__K._.___ ._.___..__.___
.____.___..W._W._..._.___.W.___..K...___.W___.WW.. ____._W._._.W_
.W._.._._K..___.._C__W__.__._.__..._.____.___._... K.___..._..__.
_.____._W__...______.KR.__WC__.._W________.KR__.._ _._K._____._.W
____._K__W__.K._W_.___.W_._____W_._..____.____.___ C_.._.______.K
__.___K_.._._____._.____._.___W_K..___._....W_____ .__._._._._.W_
__...__.___.._._..__W.._..._C_.___...__.__W_______ _.__._._._.__.
_.__._...__..__._____._.__..._W___._.K_.___._.W__W ...____.___._.
W__._..K_._.._W__K____._____._.K...___.._.._.WW__. _.__.K___.__._
____.WW_.__.__K._.K.__..__.__W._K__C__W.__.WW.K_._ W___._.._W__..
_W________.____K..._W.__..W.._W__.K.._._.._W__.__. K__C___W___.K_
_._.._..KW___.__W__._....W_._._.W_____W_.._._.____ .K_.__.___..._
K._.____K_._.__.._.._W__.______..____....K._.__._. ..__.W__R___W.
.W___W_..___K.__..__.K__...__..___.____._._._..W.. .__._.._.W__W_
..__..__..___.___.._W..C.__._._W_._.K__W..._WK___W ________...._.
___._____._.W...__._..__._.._._W_._K_._W_.__W.WKW_ W._K_._W.___.K
____K_.....K.C_____._._._.W__W.__..._.____.__WK__. K_W__.__.K..__
__C_.___._.___.__..._.___._K__W__.__..W.__.__...__ ____.___WW_WW.
.W_W.._W._..__.__.__K___.___.._..____..W___W___.__ .W___.._____._
.___._W_._.._.W._.__________._._.___.._WW...._._.. _K_._.K...K___
.W_K_W__._.._W.W___.__.___._.W____W_.._.WW.C...... W.__W_.__._W_.
W...W.W__W_W._.._..____.__W__W__._K._W_.W_.._.WW_. WW__.W_.._....
__._C__.___K_._._..____._..W._K____._.__._.____.__ __.K___W___._.
._.._______.__W___..._W_.__._________W.__.._

Scoreboard Key:
"_" Waiting for Connection, "S" Starting up, "R" Reading Request,
"W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup,
"C" Closing connection, "L" Logging, "G" Gracefully finishing,
"I" Idle cleanup of worker, "." Open slot with no current process


# php5 -v
PHP 5.2.6-1+lenny10 with Suhosin-Patch 0.9.6.2 (cli) (built: Mar 19 2011 01:46:14)
Copyright (c) 1997-2008 The PHP Group
Zend Engine v2.2.0, Copyright (c) 1998-2008 Zend Technologies
with XCache v1.2.2, Copyright (c) 2005-2007, by mOo

Thanks

[netsys]

heeeeeeeeeeeeelp