Results 1 to 2 of 2
hi, I wish to understand the way kernel works when a user/app tries to create a file in a directorty. The background - We have a java applicaiton which consumes ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
- 08-05-2010 #1
- Join Date
- Aug 2010
how soes kernel handle new file creation
I wish to understand the way kernel works when a user/app tries to create a file in a directorty.
The background - We have a java applicaiton which consumes messages over JMS, processes it and then writes the XML to an outbound queue+a local directory. Yesterday we obeserved unsual delays in writing to the directory. On 'ls|wc -l' we found >300,000 files in there. Did a quick strace on the process and found it full of mutex calls (More than 3/4 calls in the strace were mutex).
So i thought that new file creation is taking time becasue the system has to every time check certain things (e.g name of files to make sure that the new file with a specific name can be created) amongst 300,000 files and then create a file.
I cleared the directory and the applicaiton resumed to normal service levels.
1. Was my analysis correct (It seems cuz the app started working fine after a clear down)?
2. More imporatant, how does the kernel work when you try to creat a new file in directory.
3. Can the abnormal number of mutex calls be attributed to the high number of files in the directory?
- 08-05-2010 #2
- Join Date
- Aug 2010
Cannot answer all your question, but
1.I'd say that this is not stricly kernel dependant but filesystem dependant. Some filesystems handled beter lot of small files, other handle beter huge files, ... Personnaly I'll go for XFS wich is stable, fast, good for huge file and 2nd good for many small files. It's a good choice. It's included in many distro or can be easily (like in RedHat/CentOS). Ext is stable but not so good for performance. I always use XFS for file server and also for virtual machines (big files).
2.Why in your app, don't you divide your directory ? Some app (like Squid for instance) create a structure of 16 subdir each divided in 16 again, so you have in final 256 dir to write to and then much less file per dir.