Disk Full... But not...?

[KyleFarris]

I'm the administrator of an Amazon EC2 instance that my company is running a game server from. Recently we've been getting errors about how players couldn't access the game so I, of course, went to check out what was going on and discovered that our drive was full.

If you are unfamiliar with how an EC2 instance is setup... it runs Red Hat 4.0.1-5. It has two drives: one 9.9 GB drive, and one 147 GB drive. the 9.9 GB drive, which houses basically everything except the actual game files and game server, is full... (also httpd log files are sym-linked to the '/mnt/httpd_logs' directory).

Here's the output of df -h:

Filesystem             Size    Used  Avail    Use%  Mounted on
/dev/sda1              9.9G  9.4G     0      100%  /
/dev/sda2             147G  4.1G  136G      3%  /mnt

Okay, so, my main hard drive is full... big deal, we'll just find the offending file(s) (likely log files or something...) and we'll get rid of them, right?

So, I run du -sh * from the '/' directory to find out what's going on... Here are the results:

5.9M    bin
13M     boot
36K     dev
12M     etc
241M    home
109M    lib
16K     lost+found
4.0K    misc
4.0G    mnt
0       net
4.0K    opt
746M    proc
4.7M    root
17M     sbin
4.0K    selinux
4.0K    srv
0       sys
8.0K    tmp
814M    usr
64M     var

So... basically that's about 5.9GBs (confirmed by du -sh)... and that includes the /mnt directory which is actually another drive/partition all together (right?)...

I haven't found anything in the forums that helped with my particular problem. Most of the fixes had to do with n00bs not realizing that when they deleted something it went to the trash and not completely off the hard drive. But, seeing as we don't even have X installed, that really doesn't seem to be a likely culprit.

Also, this happened without any administrative interaction. It was working one day (people were happily playing our MMO) and then, all of a sudden, the drive was "full" .

Any tips or ideas would be greatly appreciated. Keep in mind, I only have SSH access to this system--no GUI.

Thanks in advance,
Kyle Farris

[HROAdmin26]

Google search / suggestion => Link

[meton_magis]

do a search for the fuser application. This is helpful for finding out if a file is in use before you delete it (or script it or whatever.) It wont help you once you run into this problem, but for future reference.

[korisk]

lsof util may be useful in that situation too

[KyleFarris]

Sorry that I haven't gotten back to you all sooner. Man... it just can't ever be easy can it? I'v actually been trying to get lsof to work since last night sometime (right after HROAdmin26) suggested that link. In fact, I actually had found the same article (or one similar at least) a while before I posted this but kind of put it off because I was getting a "lsof:command not found" message.

So, now, not only can I not find out where the space is being used up in the file system, I can't use the program to find the phantom used up space. Hopeless, haha...

So, here's the issue... I can't run lsof for some reason. It's installed. In fact, here's the output of yum list installed|grep lsof:

lsof.i386                      4.74-7                 installed

However, if I try to run lsof, I get:

bash: lsof: command not found

I tried yum remove lsof and then I reinstalled it, yum install lsof. Still didn't work.

So, I guess I'm temporarily changing the subject, but I need to in order to get the original problem fixed. Any ideas as to why lsof won't work?

Thanks for the great responses so far. I really do appreciate it!

-Kyle

[meton_magis]

lsof is put into /usr/sbin by default (at least on fedora) which is not in the path of regular users by default. try running /usr/sbin/lsof
or run lsof as root

(you can also do `su - -c "lsof"` the -c switch tells su to run the "lsof" and then exit immediately. Useful if you're wanting to follow the least permisive model.)


Also, the reason I suggested fuser, is because you can run it on a file. lsof (to my knowledge) just shows everything, to which you can use your stream editors to narrow down to what you want. fuser uses less typing, and is just a cleaner format IMO.

lsof is much more useful for finding files used by an app when you don't know where they are

[HROAdmin26]

Also, the reason I suggested fuser, is because you can run it on a file. lsof (to my knowledge) just shows everything

No

lsof /usr/local/firefox/firefox
COMMAND   PID    USER   FD   TYPE DEVICE SIZE  NODE NAME
firefox 13424 localuser  255r   REG    8,6 3951 43554 /usr/local/firefox/firefox

fuser /usr/local/firefox/firefox
/usr/local/firefox/firefox: 13424

[meton_magis]

I stand corrected, thank you.

I still believe fuser to be of cleaner format, but that may just be because i'm used to it.

[KyleFarris]

Oh, awesome, thanks meton_magis, I ran it from /usr/sbin and it worked fine! I thought I was going nuts. Okay... now that I got that working, what should I be looking for--I've never actually found myself in the need to use lsof so I really don't have any experience with it.

I looked through the results of lsof (no arguments) and this particular line seemed to trigger my wtf-o-meter seeing as it's "size" column registers in at about 8.5 GBs...:

httpd      1380       root    7w      REG        8,1 8526970880     525662 /var/log/httpd/access_log (deleted)

So, theoretically speaking, if this was the cause, what would I do to remedy the situation? Would I need to kill httpd and then research for the offending file?

Any help would be awesomely appreciated. If you guys aren't up the task (I would understand, you've helped me a lot already), I guess I could always search Google.

BTW, I don't think I have fuser on the server and I can't seem to find it in the yum repositories, so, lsof it is. Thanks though!

Thanks a bunch you guys,
Kyle

[KyleFarris]

Ok, so after I posted I realized I had nothing better to with my life so I just went ahead and Googled it... first result: http://www.opensource.apple.com/darw...f/00QUICKSTART. Pretty informative.

So, I did the lsof +L1 command and go this as a result:

COMMAND    PID   USER   FD   TYPE DEVICE       SIZE NLINK    NODE NAME
sendmail  1360   root    5wW  REG    8,1         33     0  525600 /var/run/sendmail.pid (deleted)
httpd     1380   root    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1380   root    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1380   root    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1754 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1754 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1754 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1780 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1780 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1780 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1850 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1850 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1850 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1851 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1851 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1851 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1912 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1912 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1912 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1916 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1916 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1916 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1926 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1926 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1926 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1947 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1947 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1947 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1961 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1961 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1961 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1998 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1998 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1998 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     1999 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1999 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     1999 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2000 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2000 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2000 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2001 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2001 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2001 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2002 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2002 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2002 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2003 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2003 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2003 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2004 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2004 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2004 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2005 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2005 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2005 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2006 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2006 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2006 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2007 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2007 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2007 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
httpd     2009 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2009 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
httpd     2009 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
mysqld   30123  mysql    6u   REG    8,1          0     0 1099875 /tmp/ibbDTTS2 (deleted)
mysqld   30123  mysql    7u   REG    8,1          0     0 1099876 /tmp/ibwUTFdN (deleted)
mysqld   30123  mysql    8u   REG    8,1          0     0 1099877 /tmp/ibEyo6Gx (deleted)
mysqld   30123  mysql   12u   REG    8,1          0     0 1099878 /tmp/ib2mtiei (deleted)

So, it seems my prediction was correct that it was httpd because it is "unlinked"... So, all I needed to do this whole time was to restart apache, which emptied the access_log file...

What a nightmare... for something so simple... thanks so much for your help--all of you! I really really appreciate it.

-Kyle