exFAT is REALLY slow wrt entry operations

[xorbe]

I discovered that even when the directory structure is cached in ram, exFAT is like 60x slower than NTFS, it's literally CPU bound just to traverse large source code directory structures. This is like when I figured out ext3/4 was so much faster than ReiserFS with loads of rpm files, it was the difference between instantaneous and 10 seconds just to iterate over the repo filenames. I think the data transfer rate (read/write) itself is fine. I can only guess that Windows is reparsing a lot of the exFAT table when looking up each entry. exFAT is about 80/second whereas NTFS is close to 5000. I get that NTFS (whatever Win11 uses these days) is their bread and butter filesystem, but that's obnoxiously slow for exFAT.

 

[R-T-B]

FAT is a horrible filesystem, and exFAT is just the 64-bit inode version of it so... it could just be it performs that bad due to the fact it's a shitty filesystem.

 

[xorbe]

It's a very simple filesystem with the name and filesize enumerated in the directory entries. I am guessing that Windows might be validating filesize against tracing the FAT table, or something like that, every time. There's no technical reason for it to be so slow, as far as I can tell. I should check Linux out of curiosity.

 

[R-T-B]

I suppose the slowness would be more on HDDs than flash given we now have good access times... yeah good point.

Also, being unjournaled filesystem checks are kinda poopy, but that's totally unrelated.

 

[qubit]

FAT is a horrible filesystem, and exFAT is just the 64-bit inode version of it so... it could just be it performs that bad due to the fact it's a shitty filesystem.

I agree it's likely just badly written. There's another downside to this: potentially poorer file integrity. Think about it, files are lost at risk when being written and the filenames / directory structure being updated, so one wants to get these operations over with as quickly as possible. Stretching it out like this is just asking for trouble.

 

[xorbe]

I used to use a FAT partition to dual-boot between Windows and Linux to share files. But FAT's file timestamp resolution is only 2 seconds (!), whereas exFAT is 10ms (wrt source code and object file time stamps). 10ms is still bad though (ext4 is 1ns, ntfs is 0.1ns).

Maybe you missed my point about exFAT being slow -- I pointed out it was still slow when no actual device access was occurring, so 100% cached in ram. It's whatever algorithm they are using is very unoptimized for traversing directory entries.

 

[R-T-B]

It's whatever algorithm they are using is very unoptimized for traversing directory entries.

I think that's because it's coding priority filesystem wise is very low. Microsoft has been kind of wishing FAT would just go away for years, doubtful they put good coders on it.