OS partition suddenly full

[jas.]

i have a computer running xubuntu 9.04, which basically serves as a server, running headless. its main hard drive is split into a 20GB partition for the the various system directories and a 128GB partition for /home.

it's been going along fine for several years now; i think i've been running it since maybe 7.04? i don't think that much about it, run upgrades when canonical pushes them out, even for all the things i don't use.

anyway, i recently bought a usb hard drive. plugged it in, formatted it as ext3, mounted it in /usb, and ran rsync from /home.

now this could be coincidental, but i accidentally shut down my terminal session before it was finished, and it kicked back a bunch of errors and suddenly i had several rsync processes running that i couldn't kill, and it was a big mess.

anyway, now my system is complaining that the 20GB partition is nearly 100% full, and i can't figure out what's taking up the space.

$df -lh
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 19G 18G 0 100% /
tmpfs 502M 0 502M 0% /lib/init/rw
varrun 502M 352K 501M 1% /var/run
varlock 502M 0 502M 0% /var/lock
udev 502M 132K 502M 1% /dev
tmpfs 502M 0 502M 0% /dev/shm
lrm 502M 2.2M 500M 1% /lib/modules/2.6.28-18-generic/volatile
/dev/sda3 128G 96G 26G 79% /home
/dev/sdb1 917G 36G 835G 5% /usb

$du -sh

6.1M /bin
140M /boot
0 /cdrom
132K /dev
9.7M /etc
96G /home <--- other partition
4.0K /initrd
0 /initrd.img
0 /initrd.img.old
962M /lib
16K /lost+found
12K /media
4.0K /mnt
4.0K /opt
0 /proc
288K /root
7.2M /sbin
4.0K /selinux
4.0K /srv
0 /sys
20K /tmp
38G /usb <--- external drive
2.0G /usr
318M /var
0 /vmlinuz
0 /vmlinuz.old


Where is the 19GB that's taking up the OS partition?

I've been saying for a while now that xubuntu installed a lot of things that I don't need (cups support when i don't even have a printer, for instance), and I should really reinstall a much more basic version and build a lighter system. and i'm not too worried about doing that because everything important to me (pictures, music, movies) are stored in /home. but before i do that i would like to get /home backed up just in case, and also just figure out what's wrong with the system partition that it's so full all of a sudden.

anyone have any thoughts? i'd really appreciate it.

[tpl]

welcome to the forum

what I do to find the disk hogs is to get into the root directory,
then

"du | sort -nr >DUs &"

this gives a listing, in descending size order, of all the directories

[Rubberman]

Could be in log files, or temporary ones. See how much room /tmp and /var/log are taking up. Assuming you killed the rsync processes and daemons, clear out /tmp and look in the /var/log directory tree, clearing out any really big files you see there, or some of the older ones. Each log file switches when it reaches a certain size (normally), so you will see names like messages, messages.1, messages.2, etc. Also, if there is a directory /var/log/rsync or something like that, just clear it out altogether.

FWIW, on my CentOS 5 (RHEL 5) system, my root partition (not including /boot) is 26GB. Right now, /tmp is taking up 20MB and /var/log is taking up 47MB.

Also, check /opt and /usr/share. My /opt and /usr/share right now are each taking up 5.5GB.

If these 4 directory trees cannot account for the space being used, you will need to dig a bit further to find the "sink" for your disc space.

[jas.]

thanks for the welcome and advice, tpl and Rubberman.

still nothing turning up that makes up 19GB of stuff. /usr/share is only 2GB, /var/log is only 49MB. /opt is only 4K, and /tmp is 20K, and only has a couple of lock files in it.

interesting that your root partition is so large, Rubberman. maybe this is not such a big deal after all. i just don't remember it being like this before.

maybe i'll start another thread about my rsync troubles. i would like to get that figured out too...

[Rubberman]

Mine is so large because it is a development system as well as my personal server. I have just about every database known installed (Oracle, MySQL, Postgres, etc), audio and video processing and production tools, network analyzers, office tools, cross-platform gnu toolchains for a number of different processors, virtual machines, etc. My yum list of installed packages (not counting the ones I built from source) totals almost 4000...

[jas.]

so i guess my question is: if du summarizes the actual usage of each directory, and it doesn't add up to the amount df is saying, i'm not sure what my next steps are. should i be stressing about the stability of my system?

i read a couple of articles about why df and du output different values.

it was interesting, but didn't really address the problem. I reckon i could force an fsck on reboot and see if there are things there that shouldn't be...

[Rubberman]

First, as root (or sudo), go to / and run "du -sk *" to see which directory tree is taking the most space. Note that if you have another file system mounted, it will see that too. In any case, you should get some idea where the data is going. If it doesn't appear there, then you are correct in thinking that something has taken up file system space, but isn't accessible, in which case running fsck may fix the problem.

[jas.]

k will do. thanks again.

[jas.]

well, still stumped. i let my backup run (and it took like 30 hours), so at least my data is safe. then since it was the root directory, i forced fsck on reboot:

# cat checkroot
Log of fsck -C3 -f -a -t ext3 /dev/sda1
Sun Apr 11 07:42:25 2010

fsck 1.41.4 (27-Jan-2009)
/dev/sda1: 177066/2443200 files (2.0% non-contiguous), 4782625/4883752 blocks

Sun Apr 11 07:43:49 2010
----------------


i guess i just don't understand how to get at what's taking up all that space. du shows about 3.9 gb of stuff in root. df shows 19GB.

[Route]

A Full Root Partition | Linux Journal

might have some useful info