Which filesystem to use

[MarkTrent]

I've spent the last week trying things out and I'm at my wit's end (Since it takes 4 hours to perform the transfer each time), it seems I'll be installing Windows again tomorrow.

I've got a 2TB hard drive, I want to get the most storage from it as possible under Linux but with a good transfer speed. So far my results have been: (when formatting the drive, and then copying 1.3TB of 700mb files to it)


Filesystem:::::GB left

NTFS:::::583
ext2:::::554
ext4 (with journal removed and reserved space set to 0%):::::549
xfs:::::579
jfs (with log size set to 1mb) :::::578

so results are:

ext2: 29gb available less than NTFS
ext4 (without journal or system reserved blocks): 34gb available less than NTFS
xfs: 6gb available less than NTFS
jfs: 5gb available less than NTFS

...from 1.3tb of data copied to the drive, so more will be lost once I've filled the entire disk of 1.82TB.


I would use NTFS for my drive, but it only gives 15mb/s transfer speed compared to 80-115 for the other filesystems.

What filesystem am I supposed to use on Linux which gives me the same storage capacity of NTFS but a better transfer speed under Linux??? There must be one surely!


Note - Please don't tell me to run these commands:

tune2fs -O ^has_journal /dev/****
tune2fs -m 0 /dev/****

Read my post, I've already run these.

[lugoteehalt]

Why worry about little more than ?1% of the drive. Different tools tend to give different values for this sort of thing.

[MarkTrent]

I worry because NTFS manages not to lose this space somehow, whereas none of the Linux filesystem I've tried can match it.

Multiply it by the 5x 2TB hard drives I have and in the case of ext4 it adds up to 170GB, that's a hell of a lot of space to lose compared to NTFS for seemingly no reason.

[elija]

You pays your money and you takes your choice.

Most Linux file systems have advantages in recoverability, speed and fragmentation levels (I've never had to de-fragment a hard disk when running Linux for example) and it would seem from what you have tried that NTFS will get a little bit more on a drive. It might be that you could specify a more space efficient block size when formatting with one of the ext file systems.

Now you have to decide which is more important to you. The only person who can answer that is you. Or maybe your boss if it's a work thing.

[elija]

OK, when I run

dumpe2fs /dev/sda1 | grep -i "block size"

on my Linux Mint Debian Edition machine I get 4096 returned which means it is using 4K blocks. When I run the same command on my SL6 VM, it returns 1024 which shows 1K blocks.

It might be that formatting with 1K blocks would be more space efficient.

You will need to replace sda1 with the appropriate device for your system and run the command with root privileges

[ramin.honary]

It might be that formatting with 1K blocks would be more space efficient.

Correct me if I am wrong, but for filesystems with mostly large files, I think you would need to increase the block size to improve storage efficiency. If your the contents of your filesystem consists mostly of 1K-sized files (say 95% of all files are less than 1K), then decreasing the block size improves storage space. But for mostly large files, you want to increase the block size as much as possible, use XFS with 64K block sizes so you decrease the number of inodes used. My reasoning is this:

The filesystem needs to break each file up into N blocks. When storing the file, the directory entry for that file will require N inodes. For example a CD image, 700MB / 4096 = 179200 blocks per file, and also 179200 inodes per file. Decreasing the block size increases the number of inodes: 700MB / 1024 = 716800 inodes.

Each inode requires 256 bytes on the disk, so:

• block size=4096, N = inodes per 700MB file = 179200, 256*N = 43.75 MB of space used by inodes
• block size=1024, N = inodes per 700MB file = 716800, 256*N = 175 MB of space used by inodes

That's a 400% increase space wasted by inodes.

You can't increase the size of the inodes beyond 4096 bytes on ext4 systems. I would say, go with XFS and use 65536-byte block sizes. This should improve space-efficiency a good deal assuming you are storing many large files (like CD images). If you need to store lots of small files (many files smaller than 64K) on the same filesystem amongst all of those CD images, you won't see much of a decrease in wasted space. But I would say in that case, create a separate Ext4 partition for small files, keep only CD images on the XFS partition.

[elija]

I'm no file system expert but when I was reading through ext4 - Wikipedia, the free encyclopedia I had the impression that extents would handle a lot of the inefficiency but on a second read through it seems that the speed / fragmentation gains will be offset by a loss of space.