no space left on device /data

[kueifen]

I have a failed job with the ERROR msg:

No space left on device /data.

The df -f indicates there are 13 TB space under /data:

12k :/data/production/dev4/run/excpr/us/20091014_1220 > df -h
Filesystem Size Used Avail Use% Mounted on
data 65T 53T 13T 82% /data

I also see from the df -i that inodes are depleted and none are
available within the file system to create any more files.

12k :/home/ma000004 > df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
data 4194304 4194304 0 100% /data

I googled to find out how those inodes are generated and
how many should be needed to complete our cycle of
production jobs. This is the first time we encountered this issue after 7 yrs.

Is there a formula available to help me diagnose/predict?

[hazel]

I don't think you can change the number of inodes on an existing filesystem. You can however choose the number you think you will need when making a new filesystem. Basically, each file needs an inode (plus a variable number of data blocks) so you are most likely to run into this problem if you are creating a large number of small files.

Is it possible to copy all the data from this partition to another temporary location? If so, you could remake the partition using mke2fs -N (number_of_inodes) and then copy the data back again.

[kueifen]

Will it be helpful to delete files in /data to free up inodes number under /data ? We tried to delete files that are not needed under /data , but we only see 2% ( df -i ) free of Inodes space but we do see 21TB space free on /data by issusing df -h .

LEHD12k-a:/home/ma000004 > df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
data 4194304 4083136 111168 98% /data


LEHD12k-a:/home/ma000004 > df -h
Filesystem Size Used Avail Use% Mounted on
data 65T 44T 21T 68% /data

So let's say that we cleared at least 10% of the data that was on the disk (as well as at least 100,000 links), but we have reduced the inodes from 100% of capacity to only 98%. I cleared about 7 terabyte in /data last night, and had a much smaller percent impact on the number of free inodes. If we keep removing data, it's looking like we'll be gaining inode space at a much slower rate; and I don't think we have a good sense of how much inode space we will need for a complete production run.

So, does anyone has any theories about what is going on here? I'm curious why increasing the inode space may be the answer, given that we have loaded the volume 100% full in the past and not had the inode issue.

Please advise !

[hazel]

There's no direct relationship between inode use and space use on a disk because the former depends on the number of files created and the latter on their total size. So, for example, if you get rid of one or two big files, that may make a great difference to your space usage but very little difference to your inode usage.

There are some processes - particularly some compilations - that create loads and loads of little files and then get rid of them again. This is the kind of thing that causes you to run out of inodes without necessarily using up much space. It happened to me once when I was compiling gcc under Gentoo. I had plenty of space on that partition, but not enough inodes to cope with all the intermediate files that the build created.

Was there any difference between the job you were running when the problem arose and the jobs you ran before?