Converting a directory to a readonly filesystem

[dmalcor]

I have a lot of image files that are all about the same size (and the users just keep creating more and more). We want to be able to take a directory that is reaching 60-80GB and convert it into a read only filesystem of its own. The problem is in getting the sizing right. Default settings make too many inodes and waste space. Analyzing the directory with find just seems to get too close and then some overhead of creating the FS makes it just not fit either on inodes or space.

I'm using RHEL4 and LVM2.

     1  #!/bin/bash
     2
     3  DIR=${1}
     4  NAME=${DIR##*/}
     5
     6  if [[ ! -d ${DIR} ]]; then
     7          echo "[${DIR}] is not a directory"
     8          exit 2
     9  fi
    10
    11  declare -i total=0 cnt=0
    12  while read blocks;do
    13          (( ++cnt ))
    14          (( total += blocks ))
    15  done < <( /usr/bin/find ${DIR} -printf "%k\n" )
    16
    17  echo "inodes=${cnt}, 1kBlocks=${total}"
    18
    19  set -x
    20  lvcreate --name ${NAME} --size ${total}k ReadOnly
    21  [[ $? -ne 0 ]] && exit 1
    22  mke2fs -m 0 -O sparse_super,filetype -N ${cnt} -L ${NAME} /dev/ReadOnly/${NAME}    23  [[ $? -ne 0 ]] && exit 1
    24  mkdir /testing/${NAME}
    25  mount /dev/ReadOnly/${NAME} /testing/${NAME}
    26  [[ $? -ne 0 ]] && exit 1
    27  df -P /testing/${NAME};df -Pi /testing/${NAME}
    28  cd ${DIR}
    29  tar cf - . | (cd /testing/${NAME};tar xf -)
    30  df -P /testing/${NAME};df -Pi /testing/${NAME}
    31  echo "umount /testing/${NAME};lvremove /dev/ReadOnly/${NAME}"

Here is a typical failed run:

# ./MakeImgVol.sh  /cacheimg/raidu0 
inodes=1122588, 1kBlocks=69711504
+ lvcreate --name raidu0 --size 69711504k ReadOnly
  Rounding up size to full physical extent 66.48 GB
  Logical volume "raidu0" created
+ [[ 0 -ne 0 ]]
+ mke2fs -m 0 -O sparse_super,filetype -N 1122588 -L raidu0 /dev/ReadOnly/raidu0
mke2fs 1.35 (28-Feb-2004)
Filesystem label=raidu0
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
1123584 inodes, 17428480 blocks
0 blocks (0.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=20971520
532 block groups
32768 blocks per group, 32768 fragments per group
2112 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424

Writing inode tables:   done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 37 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
+ [[ 0 -ne 0 ]]
+ mkdir /testing/raidu0
mkdir: cannot create directory `/testing/raidu0': File exists
+ mount /dev/ReadOnly/raidu0 /testing/raidu0
+ [[ 0 -ne 0 ]]
+ df -P /testing/raidu0
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-raidu0  69568904     53272  69515632       1% /testing/raidu0
+ df -Pi /testing/raidu0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-raidu0 1123584      11 1123573    1% /testing/raidu0
+ cd /cacheimg/raidu0
+ tar cf - .
+ cd /testing/raidu0
+ tar xf -
tar: ./pcl/v102/v10271900v/imge126a.tif: Wrote only 4608 of 10240 bytes
tar: Skipping to next header
tar: ./pcl/v102/v10271900v/imge1423.tif: Cannot write: No space left on device
tar: Skipping to next header
tar: ./pcl/v102/v10271900v/imge156d.tif: Cannot write: No space left on device
tar: Skipping to next header
... over 4,000 messages like this ...
tar: ./pcl/v988/v98816750/imgd1ca9.tif: Cannot open: No such file or directory
tar: ./pcl/v988/v98816750/imgd1d94.tif: Cannot open: No such file or directory
tar: Error exit delayed from previous errors
+ df -P /testing/raidu0
Filesystem         1024-blocks      Used Available Capacity Mounted on
/dev/mapper/ReadOnly-raidu0  69568904  69568904         0     100% /testing/raidu0
+ df -Pi /testing/raidu0
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/ReadOnly-raidu0 1123584 1118460    5124  100% /testing/raidu0
+ echo 'umount /testing/raidu0;lvremove /dev/ReadOnly/raidu0'
umount /testing/raidu0;lvremove /dev/ReadOnly/raidu0

In general it seems more art than algorithm, but think it is just that I don't understand all the fiddley bits of the filesystem. Any ideas?

:-Dan

[wje_lf]

I can't give a real answer to this question, but maybe I can give some helpful hints.

All this is assuming you're using an ext2 filesystem.

On the question of inodes, since your users are storing images, you could ask for double the number of inodes you think you need so that you never underestimate, and still the percentage of wasted space would be very, very low.

On the question of how much total space to ask for, larger files often have additional overhead associated with keeping track of where all the blocks are in the file. There are also other ext2 overhead considerations.

You can go two ways on the total space issue.

First, you can go the "art" way and get a feel for how much space is needed, or even write a script which takes a stab, detects failure, adds 10% to the space requested, takes another stab, rinse, repeat.

Second, you can go the "algorithm" way. I spent a couple of frustrating but joyful months learning all about the innards of the ext2 filesystem layout. Most of them are well-documented, but some I had to learn through careful observation. If you wish to go this route, read the ext2 article in wikipedia; that article isn't too useful as a definitive ext2 format guide, but it has links at the bottom, some of which promise more than they deliver. You will nevertheless find everything you need.

Going the "algorithm" way will afford enormous amounts of fun at company expense.

Consider carefully the "art" way before rejecting it.

[mrjohnson]

Okay, I've been scratching my head for a while.

Check this out:
Section 18.2. Ext2 Disk Data Structures

According to that, each inode is 128 bytes. Using the information there, you should also be able to figure out the number of block groups and blocks for each group required.

Though I'm curious to know what the old filesystem looks like. Can't you just take the usage information from that?

Keep me posted. I'd love to hear what you come up with.

[dmalcor]

Though I'm curious to know what the old filesystem looks like. Can't you just take the usage information from that?

I left off the details of the actual problem we are solving (always remember: "solutions cause problems"). The first problem is one of switching scanning volumes (as well as sizing the scanning volume correctly).

Previously we have attempted to guess at the right settings for a filesystem for people to scan images into. When that fills up we would make a new scanning volume and mark the old one read only. This is nothing but art and pain and brinkmanship. Instead of choosing the best day to switch scanning volumes we are forced to move on a particular day and we can never fill up the volumes people are scanning into. So we determined to make a large area and when it started to get full we would switch scanning directories and then build a "rightsized" filesystem for what had already been scanned.

Just a note on the file sizes. These are all images of standard pages (think captured fax images). So the size isn't too big and average file size is very consistent (I think 50-60K, but there are millions of them per volume).

Thanks for thinking with me...

:-Dan

[mrjohnson]

I see...

Well, to figure this out the "right" way it seems you'd have to first figure out how many block groups you have. Then add the fixed blocks for each group. Then the group descriptors and the inodes per group (round up to nearest block?).

I agree though with the other poster... that'll get complicated. Though untar'ing 60 gigs worth of files to just see it fail would suck, too. Having too many inodes isn't the end of the world like he says -- they're pretty small.

Creating a 1g file with dd in my home directory and running mke2fs on it with your options (without fooling with the inode creation) used 1284 blocks for formatting. Seems to me you could figure on using roughly 1300 blocks per gig for formatting with reasonably little waste...

[dmalcor]

Restricting the inodes started a while back when we realized there the standard settings created too many and they were keeping us from 100% usage on space (then of course we went the other way and hit 100% on inodes before filling up the space).

I seem to be guessing the space wrong (I end up at 100% and still have thousands of inodes let to go). I have tried using both "du" and "find" to add up the actual blocks in use. Maybe this is one of those times where a 16MB block size is better than a 4MB block size? This is an option that can be tweaked, but not a lot of guidance on when you might want to tweak it.

:-Dan