Find the answer to your Linux question:
Page 1 of 2 1 2 LastLast
Results 1 to 10 of 11
I'm the administrator of an Amazon EC2 instance that my company is running a game server from. Recently we've been getting errors about how players couldn't access the game so ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined! KyleFarris's Avatar
    Join Date
    Nov 2008
    Location
    Gainesville, FL
    Posts
    4

    Exclamation Disk Full... But not...?


    I'm the administrator of an Amazon EC2 instance that my company is running a game server from. Recently we've been getting errors about how players couldn't access the game so I, of course, went to check out what was going on and discovered that our drive was full.

    If you are unfamiliar with how an EC2 instance is setup... it runs Red Hat 4.0.1-5. It has two drives: one 9.9 GB drive, and one 147 GB drive. the 9.9 GB drive, which houses basically everything except the actual game files and game server, is full... (also httpd log files are sym-linked to the '/mnt/httpd_logs' directory).

    Here's the output of df -h:
    Code:
    Filesystem             Size    Used  Avail    Use%  Mounted on
    /dev/sda1              9.9G  9.4G     0      100%  /
    /dev/sda2             147G  4.1G  136G      3%  /mnt
    Okay, so, my main hard drive is full... big deal, we'll just find the offending file(s) (likely log files or something...) and we'll get rid of them, right?

    So, I run du -sh * from the '/' directory to find out what's going on... Here are the results:

    Code:
    5.9M    bin
    13M     boot
    36K     dev
    12M     etc
    241M    home
    109M    lib
    16K     lost+found
    4.0K    misc
    4.0G    mnt
    0       net
    4.0K    opt
    746M    proc
    4.7M    root
    17M     sbin
    4.0K    selinux
    4.0K    srv
    0       sys
    8.0K    tmp
    814M    usr
    64M     var
    So... basically that's about 5.9GBs (confirmed by du -sh)... and that includes the /mnt directory which is actually another drive/partition all together (right?)...

    I haven't found anything in the forums that helped with my particular problem. Most of the fixes had to do with n00bs not realizing that when they deleted something it went to the trash and not completely off the hard drive. But, seeing as we don't even have X installed, that really doesn't seem to be a likely culprit.

    Also, this happened without any administrative interaction. It was working one day (people were happily playing our MMO) and then, all of a sudden, the drive was "full" .

    Any tips or ideas would be greatly appreciated. Keep in mind, I only have SSH access to this system--no GUI.

    Thanks in advance,
    Kyle Farris

  2. #2
    Linux Guru
    Join Date
    Nov 2007
    Posts
    1,755
    Google search / suggestion => Link

  3. #3
    Linux Enthusiast meton_magis's Avatar
    Join Date
    Oct 2006
    Location
    arizona
    Posts
    699
    do a search for the fuser application. This is helpful for finding out if a file is in use before you delete it (or script it or whatever.) It wont help you once you run into this problem, but for future reference.
    New to the internet, technical forums, or the hacker / open source community??
    Read this to learn good posting habits http://www.catb.org/~esr/faqs/smart-questions.html

    RHCE for RHEL version 5
    RHCT for RHEL version 4

  4. #4
    Just Joined!
    Join Date
    Nov 2008
    Posts
    16
    lsof util may be useful in that situation too

  5. #5
    Just Joined! KyleFarris's Avatar
    Join Date
    Nov 2008
    Location
    Gainesville, FL
    Posts
    4

    Nothing is easy... ever...

    Sorry that I haven't gotten back to you all sooner. Man... it just can't ever be easy can it? I'v actually been trying to get lsof to work since last night sometime (right after HROAdmin26) suggested that link. In fact, I actually had found the same article (or one similar at least) a while before I posted this but kind of put it off because I was getting a "lsof:command not found" message.

    So, now, not only can I not find out where the space is being used up in the file system, I can't use the program to find the phantom used up space. Hopeless, haha...

    So, here's the issue... I can't run lsof for some reason. It's installed. In fact, here's the output of yum list installed|grep lsof:

    Code:
    lsof.i386                      4.74-7                 installed
    However, if I try to run lsof, I get:

    Code:
    bash: lsof: command not found
    I tried yum remove lsof and then I reinstalled it, yum install lsof. Still didn't work.

    So, I guess I'm temporarily changing the subject, but I need to in order to get the original problem fixed. Any ideas as to why lsof won't work?

    Thanks for the great responses so far. I really do appreciate it!

    -Kyle

  6. #6
    Linux Enthusiast meton_magis's Avatar
    Join Date
    Oct 2006
    Location
    arizona
    Posts
    699
    lsof is put into /usr/sbin by default (at least on fedora) which is not in the path of regular users by default. try running /usr/sbin/lsof
    or run lsof as root

    (you can also do `su - -c "lsof"` the -c switch tells su to run the "lsof" and then exit immediately. Useful if you're wanting to follow the least permisive model.)


    Also, the reason I suggested fuser, is because you can run it on a file. lsof (to my knowledge) just shows everything, to which you can use your stream editors to narrow down to what you want. fuser uses less typing, and is just a cleaner format IMO.

    lsof is much more useful for finding files used by an app when you don't know where they are
    Last edited by meton_magis; 11-10-2008 at 02:05 AM. Reason: added fuser info
    New to the internet, technical forums, or the hacker / open source community??
    Read this to learn good posting habits http://www.catb.org/~esr/faqs/smart-questions.html

    RHCE for RHEL version 5
    RHCT for RHEL version 4

  7. #7
    Linux Guru
    Join Date
    Nov 2007
    Posts
    1,755
    Also, the reason I suggested fuser, is because you can run it on a file. lsof (to my knowledge) just shows everything
    No

    Code:
    lsof /usr/local/firefox/firefox
    COMMAND   PID    USER   FD   TYPE DEVICE SIZE  NODE NAME
    firefox 13424 localuser  255r   REG    8,6 3951 43554 /usr/local/firefox/firefox
    Code:
    fuser /usr/local/firefox/firefox
    /usr/local/firefox/firefox: 13424

  8. #8
    Linux Enthusiast meton_magis's Avatar
    Join Date
    Oct 2006
    Location
    arizona
    Posts
    699
    I stand corrected, thank you.

    I still believe fuser to be of cleaner format, but that may just be because i'm used to it.
    New to the internet, technical forums, or the hacker / open source community??
    Read this to learn good posting habits http://www.catb.org/~esr/faqs/smart-questions.html

    RHCE for RHEL version 5
    RHCT for RHEL version 4

  9. #9
    Just Joined! KyleFarris's Avatar
    Join Date
    Nov 2008
    Location
    Gainesville, FL
    Posts
    4
    Oh, awesome, thanks meton_magis, I ran it from /usr/sbin and it worked fine! I thought I was going nuts. Okay... now that I got that working, what should I be looking for--I've never actually found myself in the need to use lsof so I really don't have any experience with it.

    I looked through the results of lsof (no arguments) and this particular line seemed to trigger my wtf-o-meter seeing as it's "size" column registers in at about 8.5 GBs...:
    Code:
    httpd      1380       root    7w      REG        8,1 8526970880     525662 /var/log/httpd/access_log (deleted)
    So, theoretically speaking, if this was the cause, what would I do to remedy the situation? Would I need to kill httpd and then research for the offending file?

    Any help would be awesomely appreciated. If you guys aren't up the task (I would understand, you've helped me a lot already), I guess I could always search Google.

    BTW, I don't think I have fuser on the server and I can't seem to find it in the yum repositories, so, lsof it is. Thanks though!

    Thanks a bunch you guys,
    Kyle

  10. #10
    Just Joined! KyleFarris's Avatar
    Join Date
    Nov 2008
    Location
    Gainesville, FL
    Posts
    4
    Ok, so after I posted I realized I had nothing better to with my life so I just went ahead and Googled it... first result: http://www.opensource.apple.com/darw...f/00QUICKSTART. Pretty informative.

    So, I did the lsof +L1 command and go this as a result:

    Code:
    COMMAND    PID   USER   FD   TYPE DEVICE       SIZE NLINK    NODE NAME
    sendmail  1360   root    5wW  REG    8,1         33     0  525600 /var/run/sendmail.pid (deleted)
    httpd     1380   root    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1380   root    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1380   root    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1754 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1754 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1754 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1780 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1780 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1780 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1850 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1850 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1850 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1851 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1851 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1851 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1912 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1912 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1912 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1916 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1916 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1916 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1926 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1926 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1926 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1947 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1947 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1947 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1961 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1961 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1961 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1998 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1998 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1998 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     1999 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1999 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     1999 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2000 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2000 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2000 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2001 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2001 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2001 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2002 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2002 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2002 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2003 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2003 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2003 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2004 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2004 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2004 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2005 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2005 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2005 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2006 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2006 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2006 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2007 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2007 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2007 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    httpd     2009 apache    2w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2009 apache    6w   REG    8,1     380928     0  525464 /var/log/httpd/error_log (deleted)
    httpd     2009 apache    7w   REG    8,1 8526970880     0  525662 /var/log/httpd/access_log (deleted)
    mysqld   30123  mysql    6u   REG    8,1          0     0 1099875 /tmp/ibbDTTS2 (deleted)
    mysqld   30123  mysql    7u   REG    8,1          0     0 1099876 /tmp/ibwUTFdN (deleted)
    mysqld   30123  mysql    8u   REG    8,1          0     0 1099877 /tmp/ibEyo6Gx (deleted)
    mysqld   30123  mysql   12u   REG    8,1          0     0 1099878 /tmp/ib2mtiei (deleted)
    So, it seems my prediction was correct that it was httpd because it is "unlinked"... So, all I needed to do this whole time was to restart apache, which emptied the access_log file...

    What a nightmare... for something so simple... thanks so much for your help--all of you! I really really appreciate it.

    -Kyle

Page 1 of 2 1 2 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •