[methodical]

Hello Forum Members,
I'm relatively new to Linux and server administration... taking classes and teaching myself some things, but I have lightyears to go ... I have the following server:

Processor Info
Processor #1 Vendor: GenuineIntel
Processor #1 Name: Pentium III (Coppermine)
Processor #1 speed: 797.589 MHz
Processor #1 cache size: 256 KB

Memory Information
Memory: 512884k/523008k available (1114k kernel code, 9736k reserved, 781k data, 292k init, 0k highmem)

Physical Drives
hda: WDC WD400AB-00CMB0, ATA DISK drive
hdc: TOSHIBA CD-ROM XM-6702B, ATAPI CD/DVD-ROM drive
hda: 78165360 sectors (40021 MB) w/2048KiB Cache, CHS=4865/255/63, UDMA(66)

Current Memory Usage
total used free shared buffers cached
Mem: 513176 492644 20532 0 127200 199028
-/+ buffers/cache: 166416 346760
Swap: 1052216 10928 1041288
Total: 1565392 503572 1061820

Current Disk Usage
Filesystem Size Used Avail Use% Mounted on
/dev/hda6 1011M 207M 753M 22% /
/dev/hda1 45M 7.3M 35M 17% /boot
/dev/hda8 30G 953M 27G 4% /home
none 251M 0 250M 0% /dev/shm
/dev/hda7 1011M 33M 927M 4% /tmp
/dev/hda3 2.0G 1.6G 361M 82% /usr
/dev/hda2 2.0G 173M 1.6G 10% /var

My server Emailed me the following message:
Drive Warning: /hda3 (/usr) is 82% full

Presently, I am the only real user on my server. I have been practicing with adding domains, and have about 15 test accounts on the server.

Does this look suspicious to anyone? With a 40 GB Hard Drive, of which I have used almost 3 GB for the test accounts, I can't understand why the /usr directory is so full...

Could someone offer some insight on this?

Most gratefully,

methodical

[Dolda2000]

1.6 GB is rather typical for a /usr directory when you've installed enough programs. Do you have GNOME and KDE installed? In that case, that is why.
Why do you have seperate partitions for /usr and /var? Why not just use one root partition that includes them?

[methodical]

Hi Dolda2000 - thanks for the reply. As far as I know, I don't have KNOME or KDE installed. This server is leased from a company at a DC in Atlanta. It came pre-configured. And I haven't installed anything new since I've had it, and that's been almost 10 months. One day last week I just started getting the "Drive Warning" messages from the server monitor.

Using SSH, I looked around the /usr directory. In my exploration, I came across a file: quota.user*

I wanted to see what was in it, so:
[/usr]# cat quota.user* | more

This opened a file with this is the beginning:

*** CIS-B ENQ received ***
2@¦Ç: Ç: +


<(N

É[ê+

7
ä^LG


4¶

... although these symbols were actually smilie faces, hearts, diamonds and other garbage.

This file was so large, and most pages were blank, that I had to quit it after not reaching the end after a couple hundred spacebar strokes.

I still don't know what it is, or if I need it.
--------------------------------
Then I accessed the server via SFTP. Opening /usr brought me to what seemed to be a mirror of my root directory

./ bin/ etc/ lib/ mnt/ root/ tmp/
../ boot/ home/ lost+found/ opt/ sbin/ usr/
.autofsck dev/ initrd/ misc/ proc/ scripts/ var/

... and clicking again on /usr opened an identical "mirror" of this same root directory structure...

I clicked on /usr about ten times, and each time, a "mirror" of the root directory opened. I don't know enough about server administration yet to know if these results are normal or not. Here is an image of the sftp session:



posted on a test server, not my own.

As far as having /usr and /var, I couldn't answer that question, since, as I stated, I leased this server pre-configured (Also has CPanel installed).

[Dolda2000]

The quota.user file is the quota definition file, which is not human-readable. To see what quotas are actually defined, run this command:

repquota /usr

That server seems to have a relatively strange setup. Could you run the following commands and post back here what they return?

ls -l /
ls -l /usr
mount
uname -a

I know that the reason that you're getting repetative /usr directories is because /usr/usr is a symlink to itself, but since it seems to be a mirror of your root directory, it would seems that /usr would be a symlink to /, which would is, however, inconsistent with hda3 being mounted on /usr. On the other hand, that would be consistent with the GNU system.
The output of the commands I gave above will clear this out.

[methodical]

Hi Dolda2000 - thanks for the insight... here are the results of the commands you asked me to run (my servername is altered here for security purposes). The line following the server name says "unknown". I don't like that.

ls -l
total 177
drwxr-xr-x 20 root root 4096 Jun 16 00:58 ./
drwxr-xr-x 20 root root 4096 Jun 16 00:58 ../
-rw-r--r-- 1 root root 0 Jun 16 00:58 .autofsck
drwxr-xr-x 2 root root 4096 Mar 12 05:13 bin/
drwxr-xr-x 4 root root 1024 Jul 19 20:21 boot/
drwxr-xr-x 18 root root 86016 Jun 16 00:59 dev/
drwxr-xr-x 47 root root 8192 Jul 26 05:13 etc/
drwxr-xr-x 30 root root 4096 Jul 21 11:43 home/
drwxr-xr-x 2 root root 4096 Jun 21 2001 initrd/
drwxr-xr-x 6 root root 4096 Apr 3 05:13 lib/
drwxr-xr-x 2 root root 16384 Sep 1 2002 lost+found/
drwxr-xr-x 2 root root 4096 Aug 29 2001 misc/
drwxr-xr-x 4 root root 4096 Sep 2 2002 mnt/
drwxr-xr-x 2 root root 4096 Aug 23 1999 opt/
dr-xr-xr-x 92 root root 0 Jun 15 19:58 proc/
drwxr-x--- 12 root root 4096 Jul 26 05:13 root/
drwxr-xr-x 3 root root 4096 Jul 19 05:17 sbin/
drwxr-xr-x 2 root root 8192 Jul 25 10:44 scripts/
drwxrwxrwt 3 root root 4096 Jul 26 05:14 tmp/
drwxr-xr-x 20 root root 4096 Jul 19 20:22 usr/
drwxr-xr-x 22 root root 4096 Jul 24 15:38 var/

ls -l /usr
total 164
drwxr-xr-x 20 root root 4096 Jul 19 20:22 ./
drwxr-xr-x 20 root root 4096 Jun 16 00:58 ../
-rwxr--r-- 1 root root 12288 Jul 26 05:13 aquota.user*
drwxr-xr-x 2 root root 20480 Jul 24 23:57 bin/
drwxr-xr-x 2 root root 4096 Feb 6 1996 dict/
drwxr-xr-x 6 root root 4096 Jul 24 23:57 doc/
drwxr-xr-x 2 root root 4096 Feb 6 1996 etc/
drwxr-xr-x 3 root root 4096 Sep 2 2002 games/
drwxr-xr-x 62 root root 8192 Jul 19 05:17 include/
drwxr-xr-x 2 root root 4096 Oct 19 2002 info/
drwxr-xr-x 3 root root 4096 Sep 2 2002 java/
drwxr-xr-x 8 root root 4096 Mar 21 18:42 kerberos/
drwxr-xr-x 47 root root 16384 Jul 23 05:17 lib/
drwxr-xr-x 5 root root 4096 Mar 21 05:42 libexec/
drwxr-xr-x 19 root root 4096 Jan 24 2003 local/
drwxr-xr-x 2 root root 16384 Sep 1 2002 lost+found/
drwxr-xr-x 6 root root 4096 Sep 2 2002 man/
-rwxr--r-- 1 root root 1024352 Jul 19 20:22 quota.user*
drwxr-xr-x 2 root root 8192 Jul 24 23:57 sbin/
drwxr-xr-x 57 root root 4096 Apr 26 05:13 share/
drwxr-xr-x 5 root root 4096 Feb 23 17:35 src/
lrwxrwxrwx 1 root root 10 Sep 1 2002 tmp -> ../var/tmp/
lrwxrwxrwx 1 root root 4 Sep 2 2002 usr -> /usr/
drwxr-xr-x 7 root root 4096 Sep 1 2002 X11R6/

mount
/dev/hda6 on / type ext3 (rw)
none on /proc type proc (rw)
usbdevfs on /proc/bus/usb type usbdevfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hda8 on /home type ext3 (rw,usrquota)
none on /dev/shm type tmpfs (rw)
/dev/hda7 on /tmp type ext3 (rw)
/dev/hda3 on /usr type ext3 (rw,usrquota)
/dev/hda2 on /var type ext3 (rw,usrquota)

uname -a
Linux brother.hostserver.net 3.3.16-11-NOAPM #2 Thu Sep 5 21:23:18 EDT 2002 i686
unknown

Finally:

repquota /usr
*** Report for user quotas on device /dev/hda3
Block grace time: 7days; Inode grace time: 7days
Block limits File limits
User used soft hard grace used soft hard grace
----------------------------------------------------------------------
root -- 1407548 0 0 68415 0 0
lp -- 1976 0 0 5 0 0
games -- 4 0 0 2 0 0
nobody -- 51508 0 0 4524 0 0
mailnull -- 4 0 0 1 0 0
rpm -- 10300 0 0 78 0 0
apache -- 4 0 0 1 0 0
cpanel -- 34516 0 0 3938 0 0
mailman -- 23676 0 0 1630 0 0
lennymil -- 80 25600 25600 20 0 0
#103 -- 904 0 0 43 0 0
#500 -- 6160 0 0 215 0 0
#503 -- 2216 0 0 235 0 0
#1000 -- 10680 0 0 770 0 0
#1001 -- 1324 0 0 73 0 0
#1078 -- 5284 0 0 532 0 0
#32010 -- 4 256000 256000 1 0 0

Thanks for your generosity...

methodical

[Dolda2000]

Well, your /usr directory is different from your root directory, even if visually only slightly. I still don't understand why it contains a symlink to itself, though, but it doesn't hurt.
The uname output was very strange, though. It reports that you're using kernel 3.3.16, which doesn't exist. The very absolutely latest version is 2.6.0, and it's still in its beta stages. Would you mind asking those who installed the server about that? It doesn't make any sense.

Anyway, to find out what is taking so much, run the following command as a first step. It will take some time to execute.

cd /usr; du -s $(du -s * | sort -n | tee /dev/tty | tail -n 1 | awk '{print $2;}')/* | sort -n

That will give you a list of which directories take the most space, and also do the same thing for all subdirectories of the one taking the most space.

[methodical]

Hello Dolda2000,
Again, thanks for the generous sharing of your time and knowledge.

You wrote...The uname output was very strange, though. It reports that you're using kernel 3.3.16, which doesn't exist.

Actually, I changed those numbers before I posted the response because I didn't know what they were, and didn't want to accidentally compromise myself on a public forum. The numbers I replaced are: 2.4.18-10-NOAPM #2. Sorry for the unintended mystery.

The results of the command line you recommended I run:

[/usr]# du -s $(du -s * | sort -n | tee /dev/tty | tail -n 1 | awk '{print $2;}')/* | sort -n


0 tmp
0 usr
4 dict
4 etc
4 info
12 aquota.user
16 lost+found
24 quota.user
28 games
108 libexec
564 doc
3660 kerberos
13720 man
15632 sbin
22592 include
62208 X11R6
80908 bin
81736 java
215196 src
277664 share
309784 lib
472688 local
0 local/jdk
0 local/jre
0 local/ssl
4 local/doc
4 local/etc
4 local/games
4 local/sbin
4 local/src
52 local/share
100 local/bandmin
196 local/Zend
388 local/man
904 local/flash
1492 local/libexec
1932 local/lib
3040 local/include
3892 local/bin
27040 local/frontpage
161460 local/apache
272168 local/cpanel

I look forward to the day when I will be able to understand these commands as fluently as you do. Even if that day comes years from now.

Best wishes,
methodical

[Dolda2000]

Oh, 2.4.18? That explains it. =)

The space distributions was quite perplexing, I must say. How have they managed to install apache so that it takes over 100MiB?! And what is cpanel? Does anyone know? Is it the project on cpanel.net? If so, why does it take over 250 MiB?
Would you mind running that command in those directories as well to find out why they're taking so much? Ie. changing "cd /usr" into "cd /usr/local/apache" and "cd /usr/local/cpanel".

That command actually isn't as complex as it might seem. Do you know how pipelines work? Begin by looking at the part that looks

du -s * | sort -n | tee /dev/tty | tail -n 1 | awk '{print $2;}'

The du command looks through all files/directories given on the command line and calculates how much space each takes. The -s switch is for just display the "summary" for each file. Without it, du would print a line for every single file it counts, in all subdirectories.
That output is then feeded into into the sort command, which simply sorts the output. -n means that it does numerical sorting instead of alphabetical, which is the default.
Then it's fed into tee. tee just splits the output. It feeds it partly into /dev/tty, so that you can see it, and also further into the pipeline.
That is where tail awaits. tail -n 1 just outputs the last of its input. -n 1 means to take only the last line. The default is -n 10, which means to output the last 10 lines.
awk '{print $2;}' is much more complex, since awk is actually an entire programming language in itself. This awk command means to output only the second column of its input, which means to output only the directory name, and not the size of it.
So all in all, that means to check the disk usage (du), sort it (sort), display the sorted list (tee), take the line describing the last, ie. largest, directory (tail), and then output only the actual directory name.

Then, by enclosing that in $( ), it takes that directory and uses that as the argument to the outermost du command, concatenated with "/*", which in this case expands to local/*, ie. all the files/directories in /usr/local. Thus, it also calculates and sort the disk usage of all the subdirectories in the largest directory.

[methodical]

Hi Dolda2000 -

You wrote...And what is cpanel? Does anyone know? Is it the project on cpanel.net?

Yes. CPanel is a program that actually uses two separate control panels to help automate server administration and domain hosting account creation and management (WebHost Manager), and then to give hosting clients individual control over their hosting accounts (CPanel - allows them to configure POP Email accounts, FTP, sub-domains, etc.).

I purposely leased a server which included CPanel, because I needed to have automated access and control over various functions of the server so that I could use the server to learn Linux and Server Administration. I've setup about 10 test domains there for practice, and now actually have two real design/hosting clients.

I've taken Intro to Linux and Intro to Networks, and Intro to TCP/IP, and intro to JavaScript, and intro to Unix. These were two day, 16 hour crash courses, and I have to say most of what was taught zipped right over my head. Too fast.

If the instruction had been as clearly organized and presented as your explanation of the du -s * | sort -n | tee /dev/tty | tail -n 1 | awk '{print $2;}' command that you offered here, I'd be a lot further ahead than I am now. My kudos to you for your excellent communications skills with technical subject matter that can, for beginners, be overwhelming.

My introduction to pipelines was very limited - something to use to pipe file information to "more" so that I can read one page at a time: cat file | more.

My goal is to learn enough to be independent of CPanel.

Didn't mean to ramble on like that, but now you know about CPanel.

I ran the command on the two directories you asked about:

[/usr/local/apache]# du -s $(du -s * | sort -n | tee /dev/tty | tail -n 1 | awk '{print $2;}')/* | sort -n
0 etc
4 proxy
12 cgi-bin
100 man
476 include
888 bin
900 icons
1348 conf
2548 core
3920 libexec
9316 domlogs
18548 htdocs
123564 logs
0 logs/ssl_mutex.12816
0 logs/ssl_mutex.5322
0 logs/ssl_scache.dir
4 logs/httpd.pid
4 logs/suexec_log.offset
8 logs/ssl_scache.pag
532 logs/suexec_log
20740 logs/ssl_engine_log
22836 logs/access_log
79436 logs/error_log

[/usr/local/cpanel]# du -s $(du -s * | sort -n | tee /dev/tty | tail -n 1 | awk '{print $2;}')/* | sort -n
4 APACHE_CONFIG
4 ChangeLog.html
4 core
4 cpanelcc
4 cpanelccp
4 cpaneld.md5
4 cpanel.md5
4 cpanelphp
4 cpconf
4 ExampleModule.test.html
4 findlatest
4 inslibs
4 installtheme
4 ipaliases
4 LICENSE
4 liscerr
4 phpcpanel
4 SafeFile.pm
4 setupapache
4 startup
4 uninstall
4 version
4 vstartup
4 webmaild.md5
8 api.txt
8 conf
8 Cpanel.pm
8 dist
8 mail
8 newapi.txt
8 startstunnel
12 redhat
12 tickets
16 cpanel.lisc
16 CVS
16 var
28 apache
32 img-sys
32 tests
36 install
44 history
44 vsrvmgr
56 Makefile
64 cpanellogd
64 entropychat
72 lang
92 lib
108 ChangeLog.OLD
148 Cpanel
192 ChangeLog
200 share
232 logs
356 cpkeyclt
504 etc
648 java-sys
816 vsrvmgrqd
948 webmaild
1264 cpaneld
1412 addons
1432 cpanel-email
1548 vsrvmgrd
1588 root@
1664 perl
3304 cgi-sys
4496 cpanel
11904 cptmp
13976 bin
18324 whostmgr
55292 src
66052 base
84948 3rdparty
0 3rdparty/phpMyAdminRR
0 3rdparty/rmagic
4 3rdparty/cpanel.interchange.diff
4 3rdparty/Makefile
4 3rdparty/phpchat.mysql
8 3rdparty/examples
8 3rdparty/interface
16 3rdparty/contrib
28 3rdparty/xmb2phpbb.tar.gz
100 3rdparty/etc
144 3rdparty/guestbook.tar.gz
168 3rdparty/xmb2phpbb
248 3rdparty/xmb.tar.gz
316 3rdparty/html
324 3rdparty/rmagic-2.12.tar.gz
348 3rdparty/phpSysInfo
420 3rdparty/man
532 3rdparty/phpBB2.tar.gz
576 3rdparty/phpmychat.tar.gz
624 3rdparty/agora.tar.gz
808 3rdparty/doc
836 3rdparty/jar
1432 3rdparty/phpMyAdmin
1464 3rdparty/rmagic-2.12
1772 3rdparty/interchange.tar.gz
1780 3rdparty/interchange.old.tar.gz
2252 3rdparty/store
2408 3rdparty/chat
2840 3rdparty/sbin
3296 3rdparty/include
3648 3rdparty/share
6468 3rdparty/lib
10688 3rdparty/interchange
17876 3rdparty/bin
23504 3rdparty/mailman

It seems as though this warning message I am receiving might actually be an automated function of the CPanel. CPanel is updated on my server nightly. I believe there's a function there now that allows, by default, the server to send an Email message to warn about disk usage.

From my communication with you, I get the feeling that the server is not being used by a bunch of hidden users, thereby causing the disk usage warning.

My sincere gratitude for your willingness to help others,

methodical

[Dolda2000]

Maybe CPanel is actually supposed to take that much. It seems a bit strange, but nontheless it seems like it.

However, your apache logs are taking up more than 100 MiB. I suggest that you do something about that, either by clearing the logs if you don't find them valuable, or if you do find them valuable, you can at the very least compress them, if not transfer them to another computer.

To clear the logs, run these commands:

>/usr/local/apache/logs/ssl_engine_log
>/usr/local/apache/logs/access_log
>/usr/local/apache/logs/error_log

To clear the logs while retaining a compressed backup, run these commands:

cd /usr/local/apache/logs
for file in ssl_engine_log access_log error_log; do gzip -9 <$file >$file.gz; >$file; done

You might also want to set up log rotation to prevent this from happening again in the future. Chech out the logrotate command, if you have it installed. Run "man 8 logrotate" to get the manual for it.

If you want to get a more thorough explanation of pipelines, read this old post. It's not necessarily that well explained, so if you have more questions, just reply back again.