Below is sample output of iostat from our Prod env. What I am seeing is that when avgrq_sz goes up out latency also goes significantly up even out r/s and w/s are not that big. So in below output first 2 lines have desired latency but then next 2 lines have high latency even though r/s and w/s is the same.

My questions:

1. What drives avgrq-sz? Why is this high in one scenario? I am assuming that's driving the latency high.
2. How to improve avgrq-sz so that latency remains as expected?

I have some more question at the bottom.


Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util

sdh 0.01 0.01 1.46 1.83 46.06 79.51 38.15 0.02 5.23 4.83 1.59
sdi 0.01 0.01 1.16 1.89 36.07 79.75 38.00 0.02 5.65 5.23 1.60
sdt 0.02 0.02 0.09 0.13 48.77 52.40 469.71 0.00 16.51 10.45 0.23
sdu 0.02 0.02 0.09 0.13 48.86 52.41 471.72 0.00 17.14 10.83 0.23



More question:

Definition of await is this "The average time (in milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them." Does this mean it is wait time + service time. So if I did await - svctime I get the real wat time?