Mysterious I/O

[russellrobinson]

Hi,

This is a real mystery to me...

Ubuntu 9.04, kernel 2.6.28-15-generic, x86_64.

I have two USB drives attached: call them sdb and sdc

The partitions on both are LVM and are encrypted using luks.

All file systems are ext4.

The computer has 4GB RAM.

I've started an "mv" of a bunch of files from an fs on sdb to and fs on sdc, around 120GB in total.

My root file system is on sda5 and is not encrypted. Swap is on sda6 and is not encrypted.

When the "mv" is running:

• other threads start writing to disk (according to iotop)
• sda5 (my root fs) is getting up to 200 tps and input of up to 2MB/s and output of up to 2MB/s (according to iostat)
• load avg (according to top) goes thru the roof (around 15, but have seen 25)

If I pause the "mv" (stop the process with with ctrl-Z), the system goes quiet, and the IO to sda5 stops. If un-pause the "mv", it all fires up again.

Any ideas?

regards,
RR

[russellrobinson]

Mystery solved...

Not a kernel problem.

The processes that were causing the problem were vmware-vmx.

I stop VMWare, problem goes away.

The question now is: WTF is vmware doing?

Will post on the VMWare website forums...

[russellrobinson]

Solution...

It appears the problem was not VMWare per se, but the swappiness settings of the kernel.

It also appears that iotop attributes IO to threads even though it's the kernel (kswapd or bdflush) that's perform the IO.

These settings in /etc/sysctl.conf have solved the problem:

vm.swappiness = 0
vm.overcommit_memory = 1
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_expire_centisecs = 1000
dev.rtc.max-user-freq = 1024

My conclusion is that VMWare threads were being targeted simply because they held a lot of memory.

The overall tuning including the above settings were obtained from here: fewt.com/2008/06/performance-tuning-vmware-server-on.html