Pls help: slow fsync?

[stwong]

Hi all,

I note that there is performance difference on 2 of our testing Dell
PowerEdge 1950 (8GB RAM each, SAS 6/iR controller, kernel
2.6.18-92.1.6.el5). The only difference is disk:
machine A: 2 x 146GB (Seagate ST3146855SS, 15000rpm, RAID 1)
machine B: 2 x 500GB (Seagate ST3500630NS, 7200rpm, RAID 1)


Both systems has drive write back cache enabled as shown in dmesg and LSIUTIL. Some IO performance test tools show results as expected (machine A is a bit faster than machine B due to faster hard disks):

- dd if=/dev/zero of=bigfile count=10240 bs=1M; sync
- iozone
- bonnie++

However, when I run load test with the code attached below that I
borrowed from the net, performance of machine B is almost 4 times of that of machine A:

----------------------------- cut here --------------------------------
#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>


#define NUM_BLKS 200
#define BLK_SIZE 4096 /* use either 20 or 4096 */


char buff[BLK_SIZE * NUM_BLKS];


int main(int argc, char * argv[])
{
int fd, k;


fd = open("tst.txt", O_WRONLY | O_CREAT, 0644);
for (k = 0; k < NUM_BLKS; ++k) {
write(fd, buff + (k * BLK_SIZE), BLK_SIZE);
fdatasync(fd);
}
close(fd);
return 0;



}

----------------------------- cut here --------------------------------

I've no idea why and thus run a simple systemap script against this program:


probe kernel.function("*@fs/buffer.c").return {
printf ("%s <- %s(%d)\n", thread_indent(-1), probefunc(),
gettimeofday_ms())
}

Found that on the machine A, number of calls in buffer.c is much greater (almost 4 times) than machine B:

alloc_buffer_head 130 556
alloc_page_buffers 130 556
__block_commit_write 4787 19489
__block_prepare_write 4792 19854
block_prepare_write 9583 39709
block_sync_page 4523 18731
__block_write_full_page 4783 19834
block_write_full_page 9565 39660
__bread 258 1083
__brelse 666 2959
create_empty_buffers 130 554
__do_fsync 4663 19308
do_fsync 9327 38616
__find_get_block 656 2813
__find_get_block_slow 138 697
generic_commit_write 4787 19842
__getblk 656 2804
init_page_buffers 0 2
inode_has_buffers 0 2
invalidate_inode_buffers 0 2
ll_rw_block 0 6
mark_buffer_dirty 4789 19852
recalc_bh_state 130 556
set_bh_page 130 556
submit_bh 4786 19838
sys_fdatasync 4663 19303
unlock_buffer 785 3356
unmap_underlying_metadata 129 552


Seems the performance difference is caused by the number of calls
requested - looks like a smaller write buffer cache is used on one machine
and thus needs more fsync to flush data to disk. However, I got stuck
here and have no idea about the actual reason behind. Would anyone
pls help? Sorry for the newbie question.

Thanks a lot.
/ST Wong

[stwong]

Some more findings on this issue:

I run performance test tool of BerkeleyDB on both systems. Performance differs a lot when running the 'sync-on-commit' test that calls fdatasync. Using top and ps, found that WCHAN of one system always shows log_commit_wait. While all filesystems are ext3 mounted with default option (data=ordered), I wonder why fdatasync doesn't skip metadata update while both systems run the same kernel (2.6.18-92.1.6.el5)...

Would anyone pls help? Thanks a lot.