Hello Everyone,

# uname -a
Linux WhatEverHostName 2.6.18-194.26.1.el5 #1 SMP Tue Nov 9 12:54:20 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
CentOS release 5.5 (Final)

# mysql --version
mysql Ver 14.12 Distrib 5.0.45, for unknown-linux-gnu (x86_64) using readline 5.0

We have --TWO-- separate systems each with the above characteristics. Each system houses a MySQL database which has grown to 18TB over the past 5 years.

Problem:

We only recently started to notice that the MySQL flush command takes longer to flush tables to disk as we add more data per week (ie, 200G .. 1000 files). As it stands, it now takes 3 hours to do a flush table at the end of the business week. A subsequent flush table takes merely seconds.

MySQL and OS logs contain nothing out of the ordinary.

Additionally, when I run the following command on each system, it takes one system about 5 minutes to return an answer and the other system never returns an answer so i kill the command after about 15 minutes:

# time lsof | grep -i mysql | wc -l
~3000

Note, one host has as uptime of 220 days and the other 40 days -- the former host takes longer to respond but each has same about of data.

Do you have any ideas that I can try to alleviate this production issue. Note,
Ive google both MySQL and Linux forums extensively but have not found an exact match to this situation.

Thank you very much.