Thoughts on a USB filesystem.

[monday90]

Over recent months there has been a lot of fuss over the FAT filesystem, specifically the Microsoft patents related to it. This got me to thinking, Why is an archaic filesystem like FAT still in use? The answer to this question is I believe in removable storage.
Firstly removable storage by nature needs to be portable between machines and as such NTFS security isn't really required. This also applies to EXT2/3/4 /Reiser FS.

Secondly FAT is supported in one form or another across many operating systems. FAT itself was developed many moons ago and has slowly been enhanced by different parties to their own ends. The patent issue stems from a Microsoft development in the handling of filenames (perhaps over simplified but that's the gist of it).

FAT's handling of storage is pretty poor to say the least with files becoming fragmented easily. It also has certain size limitations. More so on Windows systems.

I guess what I'm getting round to is:

Is it worth developing a high performance file system for removable devices, particularly flash memory devices which only have a limited number of writes to a particular sector?

My thoughts on this are that any candidate filesystem should have the following features:

Support for very large volumes. I personally have a 750GB removable HDD I have friends with 1.5TB drives. To make the system future proof it should support as large size a volume as possible.

Excellent disk organisation. Arrangement of files be they large or small in such a way as to minimise fragmentation. Also consideration should be given to minimal writes, This would improve the lifespan of flash memory.

Low file security. By this I mean that the concept of an owner (NTFS or EXT/Reiser) is redundant when moving between systems. Having an executable flag would seem like a good idea though. Without an owner it is near impossible to assign user rights to a file as you have no kind of reference point to start with. For removable storage this is actually a good thing.

Support for encryption. If this could be done at file-level all the better. Encrypting files would give the security that making the device portable takes away.

Perhaps the most sensitive point, freely available open drivers for Linux, BSD, Unix, and Windows. Yes Windows. Lets be fair here Windows makes up the majority of desktops. Making the filesystem available to Windows ensures interoperability.

So what would we gain from this?

Well first of all manufacturers of devices such as moble phones, cameras, satnavs, etc.. would no longer have to use FAT and pay Microsoft for the privilege. Open drivers mean that they could distribute them with their product for free.

Second, we get a decent driver designed from the ground up for removable storage. It outperforms FAT and is patent free, it also extends the life of flash memory.

Third, with any luck we can head off some of these Microsoft+* deals where Microsoft starts to benefit from the adoption of Linux. Lets face it you buy a Tomtom satnav you're supporting Linux. Microsoft gets a slice of that revenue though.

Does anybody feel that this is a worthwhile project? Or for that matter have any other ideas to throw into the mix? It would be good to hear some feedback.

[elija]

There is a patch for the FAT Code in Linux

New Linux patch could circumvent Microsoft's FAT patents - Ars Technica

[monday90]

I am aware of the patch. Ultimately though I feel that we are "flogging a dead horse" . It only supports a 4GB file size and volumes of (depending on various tweaks) a maximum 8 TB. Say you wanted to put a DVD image on there, thats 4.4 GB straight away. What about a VM image? 10-40 GB?

While I fully respect those who worked to produce the patch, wouldn't now be the time to develop something capable of much more?

[MikeTbob]

I think the reality is, if Microsoft would support Linux/Unix file systems then we wouldn't have this problem. You and I could just use Reiser or any ext-3/4 file system on any device we wanted and still be able to access it from Windows or Linux. But we all know that isn't going to happen in Steve Balmers lifetime.

[monday90]

The beauty of this is that Microsoft wouldn't have to. We develop the drivers. It has been done already with the Ext drivers for Windows. A little polish and were there. The alternative is: as removable storage hits 8TB (about 3 years) users format thier drives NTFS as its easier than having multiple FAT partitions. NTFS has its drawbacks but is the only solution for Windows users. Linux users end up having to make multiple FAT partitions on devices. Linux users might try and use EXT2/3/4 but ultimately end up having to log on as root on the destination system in order to set permissions for the user of the drive. Thats not portability.

The leverage we have here is 3rd partys. Make it available for free and every camera manufacturer (SD cards) phone manufacturer (same) and satnav manufacturer will jump at it. Buy a camera, it still comes with a driver disk, this time that disk also has support for the new filesystem.

Manufacturers like it as it is guaranteed free of patents. Users like it as they can get more from thier removable storage.

Its a win win situation.

[HROAdmin26]

If you care about using it in Windows, use NTFS.

If you don't care about Windows, you can use any of the many existing filesystems that Linux supports.

It is not a big deal because there is already a solution being used on a daily basis. Filesystem development efforts are underway in EXT4 and BTRFS, among others. If it is feasible and/or needed enough, maybe a Windows driver will be written to support these at a later date.

There is an existing Windows EXT2/3 driver for anyone that wants to use EXT2/3 on their drives.

[reed9]

It is not a big deal because there is already a solution being used on a daily basis. Filesystem development efforts are underway in EXT4 and BTRFS, among others. If it is feasible and/or needed enough, maybe a Windows driver will be written to support these at a later date.

There is also NILFS.

[monday90]

For what its worth I don't use Windows. However most people do and need compatibility. A person may use Linux at home but be forced to use Windows at work. Currently FAT is the only filesystem supported accross multiple operating systems. FAT doesn't have much life left in it. Try and move a VMware disk image from a Windows system to a Linux system (virtualbox does all it can by supporting foreign formats). If your VMDK file is bigger than 4 GB you're **** out of luck. Either that or you format your USB stick NTFS. I don't want that, do you?

[HROAdmin26]

I don't know what you're talking about - why not format with NTFS? It's a lot better than FAT in pretty much all respects.

And again, there are several Linux filesystem alternatives that can be read with Windows.

I *do* use Windows and Linux on a daily basis and this is a non-issue.

There is also NILFS.

Great point for the OP's questions about SSD's/thumb drives.

* Although performance is noted to be good on SSD's, I wonder if it addresses "excessive writes" known to affect some types of solid state storage?

[monday90]

But this is again the point. The developers of the NTFS-3G driver have recently signed a patent deal with Microsoft link here. If we use this driver we are to all intents and purposes lining Microsofts pockets. We need something free of patents, like a new filesystem that out performs anything on offer. We could close ourselves off from the Windows crowd but I don't see that as helpful. If we want people to use Linux then surely the transition should be made as easy as possible without Microsoft taking a bean from it.




--Edited for spelling.