Interpreting vmstat output

[kerm1t]

I'm trying to improve performance on one of my machines (a web server, with 4 CPU cores). Here's the output of vmstat 3:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  3  62284 1540400 234200 3283712    1    2     5   151   20   29 17  1 39 43  0
 3  3  62284 1535716 234200 3283712    0    0     0    44 1565  627  9  1 42 48  0
 1  3  62284 1540312 234200 3283712    0    0     0     0 1517  489 18  1 38 43  0
 0  3  62284 1539792 234200 3283712    0    0     0    85 1216  275  3  0 47 50  0
 1  3  62284 1539420 234200 3283712    0    0     0     0 1171  238  3  0 47 50  0
 1  3  62284 1540696 234200 3283712    0    0     0    36 2113 1197 32  1 27 39  0
 0  3  62284 1541680 234200 3283712    0    0     0   128 1365  348  9  0 43 47  0
 0  3  62284 1539308 234200 3283716    0    0     0    67 1420  555 14  1 43 42  0
 0  3  62284 1543268 234204 3283712    0    0     0   731 1914  898  7  1 44 49  0
 0  3  62284 1543640 234204 3283716    0    0     0     0 1151  163  1  0 49 50  0
 1  3  62284 1539316 234204 3283716    0    0     0    33 1536  696 12  1 39 48  0
 0  3  62284 1545020 234204 3283716    0    0     0     0 1161  163  6  0 46 48  0
 0  3  62284 1542468 234208 3283680    0    0     0   148 1395 1862 19 14 33 34  0
 0  3  62284 1542708 234208 3283684    0    0     0   405 1407  477  4  1 46 50  0
 2  3  62284 1543264 234208 3283688    0    0     0     0 1789  943 22  1 39 38  0
 0  3  62284 1541032 234208 3283688    0    0     0   141 1824  994 13  2 39 46  0
 1  3  62284 1533600 234208 3283688    0    0     0     0 1490  594 19  1 40 40  0

I'm curious about those 3 blocking processes. Disk IO is pretty low (the machine has 8 GB of RAM, so plenty of caching going on) and the CPU seems to be spending around half its time waiting.

Any suggestion on how to interpret this?

How would I go about tracking down those blocking processes?

[coopstah13]

is your web server running just apache or are you running some kind of application server? Additionally you can get disk information with iostat -d -x where you can see the actual IO and verify that isn't the problem.

[kerm1t]

Thanks for the reply. I tried iostat, but it didn't really shed any light. I'd previously run hdparm -tT, and the disk performance is great, so I'm still baffled as to the blocking processes

[coopstah13]

What are you using for a web server? Are you using a database in the back end?