How to Find The Directory With The Biggest Number Of Files

[Lampdocs]

I've got a serious problem: on my server there are lots of small files and I am getting No space left on device error though the free space exists. The problem is the number of files that can be held by filesystem and I already have lots of files. My question is how to identify the folders with the biggest number of files in them? I think there should be a simple script for it or even a command...

[scm]

If you use find to locate all your directories and use ls -ld to display them you'll see which are the biggest from their size. Something like

ls -ld $(find / -type d 2>/dev/null)

should do it, providing you don't have so many directories that you blow your argument space, otherwise

find / -type d | xargs ls -ld

could be used. I'll leave you to work out how to sort the resultant list into biggest first or last, if you'd find that a convenient way to view them.

[Lampdocs]

Thank you for a quick reply! Here is what I get:
ls -ld $(find /var/www/html/ -type d 2>/dev/null)
-bash: /bin/ls: Argument list too long

Any ideas?

[Geeth]

Try this one

du -h | sort -nr

[Lampdocs]

Can't see anything after 3 hours of running the command du -h | sort -nr
...

[scm]

Thank you for a quick reply! Here is what I get:
ls -ld $(find /var/www/html/ -type d 2>/dev/null)
-bash: /bin/ls: Argument list too long

Any ideas?

Yeah, use the second command I posted if, as I said, the first one blows your environment space.

[drl]

Hi, Geeth.

Try this one

du -h | sort -nr

A few comments on your suggestion.

First, this seems to be reporting the wrong characteristic. The problem is with running out of inode space, but du reports disk space used by a directory. If you know what an inode is, then you can see the issue. If not, see inode - Wikipedia, the free encyclopedia , noting a specific reference to this kind of problem:

It is possible to "run out" of inodes. When this happens, you cannot add data to the device, even though there may be free space available.

See also a man page, where one would see that filesystems have a specific amount of storage allocated for inodes:

       -N number-of-inodes
              overrides the default calculation of the number of  inodes  that
              should  be  reserved  for  the filesystem (which is based on the
              number of blocks and the bytes-per-inode  ratio).  

-- excerpt from man mkfs.ext2

Secondly, the human-readable option on commands like du may produce strings that sort will not handle correctly. Here's the output from a script that examined one such directory in my system using the du-sort piepline:

% ./s1

(Versions displayed with local utility "version")
Linux 2.6.11-x1
GNU bash 2.05b.0
du (coreutils) 5.2.1
sort (coreutils) 5.2.1
grep (GNU grep) 2.5.1

  Excerpt from results:
12K     ./awk-gawk-nawk/printing-numbers
12K     ./awk
9.9M    ./user-problem
8.0K    ./zsh/positional-parameter-last
8.0K    ./zsh/parameter-colon-h-or-t

which at first glance seems OK. However, looking more closely one can see that the suffixes are not interpreted as one might like, producing the condition that 9.9 MB appears to be less than 12 KB.

So the OP was looking for a way to reduce the number of files by finding the directories that had the largest number of files, presumably to start pruning, backing-up-removing, etc., in those areas.

I think that the inode / data allocation is a very old design feature from the original UNIX systems, and I didn't know that Linux filesystems still used the strict allocation scheme. It simply may be easier to do that instead of having the driver trying to do the allocation on the fly. Most folks don't need to look very closely at that unless they know ahead of time that they will need something different, or until they run into trouble. I don't recall changing it on any system I have installed recently.

Best wishes ... cheers, drl

[shaurz]

Sorry to reply to an old thread, but I came across this when I was looking for the same thing. I gave up looking and solved the problem myself with this shell script (which I call "dirsizes"):

#!/bin/bash

if [ -n "$1" ]; then
    DIR="$1"
else
    DIR="`pwd`"
fi

find "$DIR" -type d | while read dir; do
    echo `ls -A "$dir" | wc -l` "$dir"
done | sort -nr

It takes the directory path as an argument, or uses the current directory if no arguments were given.

Hope this is useful to someone else who comes looking.