How many files can I have safely in a subdir?

[z0diac]

Ok, I'm running a vBulletin forum (3.8.4) and found that all user attachments go into 1 single directory for each user. For each attached file in the forum, there's 2 files on disk (*.attach and *.thumb), for pictures that is.

One user already has over 100,000 attachements, thus, over 200,000 files in his attach directory.

Someone recently told me to 'keep an eye on it' because certain setups can't hold more than X number of files in a single directory. Yet someone else said I could have over 1 trillion files in a single directory if the HDD was large enough...

Here's my setup:

linux version: 2.6.18-92.1.10.el5
php: 5.1.6
mySQL: 5.0.45
File system: ext3

vB support has told me any limitation there might be, will not be the result of vB, so now I'm looking at either Linux and the way it handles files, or the ext3 file system.

Does anyone know if I can just keep going with putting files into one directory? (there will be over 1million probably by year's end. Hopefully not more than 5 million ever).

And, will having so many files in a single directory cause any performance problems? (ie: slowdowns)

My only option is to hire a coder to somehow have it split the 1M+ files into several subdirs, say 50,000 per subdir. But even though it's messy, if it really doesn't make a difference in the end whether they're in 50 subdirs, or just 1 dir, then I won't bother (and can sigh a breath of relief)


Thanks in advance!!

[Rubberman]

The biggest issue here isn't the number of files, but the time it takes to traverse the directory. It certainly has a negative impact on the performance of the file system as a whole. I think that recoding the tool to put files in sub directories makes sense, but I would have to think that a date-based algorithm would be more appropriate. Just remember, every system resource has its limits. What they are depend on how the system is coded, how much memory you have, what file system is in use, etc.

[z0diac]

The biggest issue here isn't the number of files, but the time it takes to traverse the directory. It certainly has a negative impact on the performance of the file system as a whole. I think that recoding the tool to put files in sub directories makes sense, but I would have to think that a date-based algorithm would be more appropriate. Just remember, every system resource has its limits. What they are depend on how the system is coded, how much memory you have, what file system is in use, etc.

Thanks for the reply. yes another reply I got (on another site) mentioned it has to do with the # of inodes that were created when the drive was originally formatted. Doing a "df -i" shows I"ve only used 15M Inodes so far, and have 590M left, so I'm not worried about the total files per dir.

But yes, it was also mentioned that too many files in a subdir could slow thing down, although it was never explained why. If the directory structure says "this file starts at track x, sector y", it doesn't really matter if it's in a dir with 1M other files or not. But I don't know how ext3's structure of storing file locations works so I'm not sure.

Unfortunately, with vBulletin, *all* attachments for a user go into a single directory, so I'm resorting to creating multiple accounts to do my uploading - which is a real mess as it throws stats off, private messaging, posting/replying, etc, etc... having to login to multiple accounts to check things... etc...

I'll have to look for someone who can right a mod that will move files to a subdir then update the dbase locations to the files.

CHeers.