Setting 4k sector size on NVMe SSDs: does performance actually change?

[L'Eliminateur]

well that's not something that you can usually do.

also, i'm surprised at the shitty results of clear linux which is the most optimized linux out there, and conversely the big result on freebsd which is avery conservative AND BSD

 

[SchumannFrequency]

well that's not something that you can usually do.

also, i'm surprised at the shitty results of clear linux which is the most optimized linux out there, and conversely the big result on freebsd which is avery conservative AND BSD

CPU performance and IOPS are two different things. Clear Linux is best known for its CPU performance.

BSD has better performance than Linux (Ubuntu) in many domains:

Nginx Perfomace - FreeBSD vs CentOS (Under Heavy Load)

PostgreSQL benchmark on FreeBSD, CentOS, Ubuntu Debian and openSUSE | redByte blog

redbyte.eu

FreeBSD/Ubuntu Dual Boot Homelab in The Bedroom by the bed testbed

adventurist.me

FreeBSD's Bhyve Overview: Why it's better than other hypervisors. At least for our use-case.

Linux Distributions vs. BSDs With netperf & iperf3 Network Performance - Phoronix

www.phoronix.com

Scalable Event Multiplexing: epoll vs. kqueue

FreeBSD: A Faster Platform For Linux Gaming Than Linux? - Phoronix

www.phoronix.com

Linux – NetBSD web server performance – Valuable Tech Notes

Shell benchmarks comparing sh, bash, and ksh
by u/vogelke in freebsd

Benchmark comparison BSD and Linux

Performance of Linux vs. FreeBSD NFS clients

I am seeing a dramatic difference in performance between Linux and FreeBSD servers mounting the FreeNAS server I have setup. I have searched the forums and not found a discussion of this as a general problem, so wonder what in my setup might be causing this. The FreeNAS 8.3 server has a raidz2...

www.truenas.com

FreeBSD Linux Benchmark

FreeBSD and Linux Nginx benchmark

docs.pritunl.com

I find it very easy to switch operating systems. Especially if you start with something like Nobara Project, Mint, Artix Linux or MX Linux.

If you use the above systems for several years, the step to Void Linux, NetBSD, FreeBSD, NixOS, Clear Linux as a daily driver becomes relatively simple, you don't have to be gifted or anything like that.

I know many people who started using a 'more advanced Unix-like system' as a daily driver straight away and were successful at this. Obviously these are the people who have a certain level of talent and interest in this sort of thing, but they are quite numerous.

 

[Ferrum Master]

'more advanced Unix-like system' as a daily driver straight away and were successful at this. Obviously these are the people who have a certain level of talent and interest in this sort of thing, but they are quite numerous.

It ain't that bad, there are enough of Linux users, I just don't experiment much and don't switch to anything else than Fedora and plain Debian for core tasks out of habbit using it on RPI. We are actually pretty conservative and these experiments are like sports news about drag racing. Unless it hits second stable noone would touch it.

But... the point is... I was waiting you to say btw I use arch

 

[SchumannFrequency]

It ain't that bad, there are enough of Linux users, I just don't experiment much and don't switch to anything else than Fedora and plain Debian for core tasks out of habbit using it on RPI. We are actually pretty conservative and these experiments are like sports news about drag racing. Unless it hits second stable noone would touch it.

But... the point is... I was waiting you to say btw I use arch

I've used Arch Linux for quite some time, and I've also maintained and supported it on my dad's laptop for five years. Now I installed Void Linux on my dad's laptop because that laptop is getting old, and with Void it boots 17 seconds faster than with Arch, which is a noticeable difference.

On my desktop, I've been using FreeBSD for about four years, and I've never compiled anything, always just used pkg to install stuff. I have Alpine Linux and MX Linux (Fluxbox) installed in a VM but I hardly ever use them because FreeBSD has good support for Linux binaries and even for RPM packages. On the old netbook I have FreeBSD and use it as a music player. On a laptop I have MX Linux with XFCE.

I'm actually quite happy with this general setup. Related to this topic I can say that ZFS on FreeBSD 12.3 has quite a lot of IOPS. App launches are always very fast, especially with 0 A.D. which boots in a second, and GIMP in 4 seconds. Firefox and Chromium start in about 2 seconds (the first time I start it). For my old hardware that is very good.

 

[Ferrum Master]

For my old hardware that is very good.

Well that's the main point. The HW is the decisive factor often when picking distro branches. I you look up at the tree for the same driver it really acts different on each distros. For example like toothache named RTL8125 on BSD while it worked fine in Fedora/Debian it is kinda fixed but pfSense peps still are sore about it. Not that I can change what kind of NIC IC is soldered on the boards lately it is what it is. I had much better experience on Fedora than anything else, maybe Slackware being the second best and any Ubuntu flavor being the brown worst stability wise for me.

Boot times are really irrelevant factor for me. It just depends how you HW is configured and how much you have of it and how straight the hands were for the BIOS coder. I am really hoping BTRFS will take up pacing. But the recent kernel speedups for EXT4 are also very good and welcome as a healthy competition goes on. Filesystem 4K wise linux should loose to windows due to less aggressive CPU drivers and maybe even absence of them in general, it needs tweaks. The sad story, the boost algo drivers come very very late. And 4K benchies are very single CPU thread Turbo sensitive. On AMD the Turbo stucks on the wrong CCX as usual and on Intel it doesn't kick in at all, it is too lazy(conservative). There could be one more cheat tho. On intel you can properly disable all mitigations at grub and those really influence SSD 4K benchmarks. I've seen numbers degrading around 30% depending on the arch. On windows some are baked in kernel, you cannot turn them off anymore for recent actual Win 10/11 builds. Maybe we are digging the wrong way with acceleration for nvme performance at all. I am not sure the Benchmarks we see are made with mitigations on or off also. Knowing those are Linux peps, my vote goes on OFF.

Many thanks for crap BIOS makers, I currently at war with Gigabyte Refusing to fix bios for my Z590I-Vision D that constantly produces MCE error on each boot no matter what distro or kernel I use, a circus, on Windows you don't get the error, so no wonder no one complains. Maybe I will post my full outcome of support ticket to shame them in public, it takes two months already on QA and yesterday I was asked again how to reproduce the bug thus it looped(I said just install any Linux distro and see the error pop up or do journalctl -b). For example my tickets for Asus motherboard were answered and BIOS was fixed without much complain, the better - they fixed all my complaints within few weeks.

 

[R-T-B]

This OS talk is somewhat off-topic though. This thread is about the hardware-based benefits of 4k LBA sectors vs 512 LBA sectors.

 

[AsRock]

Tried it and made no difference what so ever

 

[R-T-B]

Tried it and made no difference what so ever

I have doubts of the performance benefits myself, but maybe some drives have some and some don't. At any rate the threads subject is still about the LBA size.

 

[Solid State Brain]

I have doubts of the performance benefits myself, but maybe some drives have some and some don't.

Or maybe there are other potential benefits like lower temperatures as another user suggested earlier.

I started this thread because SSD manufacturers generally suggest that the larger LBA size is the "best" option for performance, but there are no benchmarks or reports showing this clearly, in particular on Windows.

I still don't have other SSDs with configurable LBA size that I can play with, though.

 

[SchumannFrequency]

This topic has also been discussed in the FreeBSD forums related to ZFS: https://forums.freebsd.org/threads/best-ashift-for-samsung-mzvlb512hajq-pm981.75333/#post-529225

OTOH using 4k filesystem sectors on a 512n disk would only mean that the drive firmware has to translate that one request into 8 distinct requests to the storage. You needed to access all of those 512byte sectors anyway, so the performance impact is miniscule if there even is any.

So it is always the safer option to go for 4k sectors on the filesystem side, especially if your pools are usually "here to stay". I've used ashift=13 during pool creation for several years now and even pools that have migrated from spinning rust to SSD and now to NVMe I've never seen any performance issues.

Except that the disk we're talking about here is not spinning rust, but flash cells. It has a real block size, which is likely to be much larger than either 512 or 4K. There are interestingly complex performance implications of adjust the block size that the file system uses (ashift for ZFS), and bigger is not alway better. The biggest problem is that a larger ashift value may waste space. In my personal opinion, that is much less relevant than people typically think (since most space in a file system is typically used by a small fraction of large files), but depending on the usage and workload, the correct answer can matter significantly.

Is there a measurable performance improvement using ashift=13 vs ashift=12. Personally i don't think so.

I am sure NVMe have some flash translation. Do you think they use pages too?
Deeper Question. Why do we need software trim when the drives handle garbage collection at the controller level?

Then there is also someone who has done a benchmark:

4096 # sectorsize
4 kbytes: 17.6 usec/IO = 222.2 Mbytes/s
8 kbytes: 19.3 usec/IO = 405.6 Mbytes/s
16 kbytes: 23.6 usec/IO = 661.4 Mbytes/s
32 kbytes: 29.7 usec/IO = 1050.8 Mbytes/s
64 kbytes: 42.4 usec/IO = 1473.2 Mbytes/s
128 kbytes: 68.6 usec/IO = 1823.1 Mbytes/s
256 kbytes: 118.8 usec/IO = 2104.9 Mbytes/s
512 kbytes: 235.2 usec/IO = 2125.7 Mbytes/s
1024 kbytes: 468.8 usec/IO = 2133.2 Mbytes/s
2048 kbytes: 938.9 usec/IO = 2130.2 Mbytes/s
4096 kbytes: 1870.4 usec/IO = 2138.6 Mbytes/s
8192 kbytes: 3758.2 usec/IO = 2128.7 Mbytes/s

512 # sectorsize
4 kbytes: 17.5 usec/IO = 223.3 Mbytes/s
8 kbytes: 19.1 usec/IO = 409.5 Mbytes/s
16 kbytes: 23.8 usec/IO = 657.9 Mbytes/s
32 kbytes: 29.7 usec/IO = 1052.2 Mbytes/s
64 kbytes: 42.5 usec/IO = 1471.1 Mbytes/s
128 kbytes: 68.1 usec/IO = 1834.2 Mbytes/s
256 kbytes: 130.2 usec/IO = 1920.9 Mbytes/s
512 kbytes: 252.6 usec/IO = 1979.4 Mbytes/s
1024 kbytes: 497.4 usec/IO = 2010.6 Mbytes/s
2048 kbytes: 958.2 usec/IO = 2087.2 Mbytes/s
4096 kbytes: 1861.7 usec/IO = 2148.6 Mbytes/s
8192 kbytes: 3718.1 usec/IO = 2151.6 Mbytes/s

No increase in speed at any level.

ZFS - Best ashift for SAMSUNG MZVLB512HAJQ (PM981)

Hello, I'm running a server with 4xSAS raidz plus 1xNVMe SSD (SAMSUNG MZVLB512HAJQ). The NVMe drive is split in 3 parts: # zpool status pool: nvme state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM nvme ONLINE 0 0 0...

forums.freebsd.org

He achieves 1834.2 Mbytes/s for 128kbytes, but he achieves this for random writes. While this should be about the maximum sequential write speed according to Samsung.

This is striking because for 128kbytes there are huge differences between sequential and random writing:

I/Os Are Not Created Equal – Random I/O Versus Sequential I/O

Sequential I/Os always outperform Random I/Os. Accessing data randomly is slower & less efficient than accessing it sequentially. True for HDDs and SSDs.

condusiv.com

https://i0.wp.com/codecapsule.com/wp-content/uploads/2014/02/ssd-presentation-05.jpg?w=720&ssl=1

My guess is that windows 11 with ZFS will only get around 600 Mbytes/s with this drive for (128 kbytes) random writes or most likely even lower.

Linux will get around 1000 Mbytes/s for (128 kbytes) random writes on ZFS, or lower.

ZFS is mainly designed for features and it's the most robust form of data storage that exists. It is not designed for speed..
These are remarkable results.
We should not forget that NTFS is more than 40 years behind ZFS in terms of technological sophistication.

We can clearly see that the operating system is by far the single biggest determinant of write/read and IOPS performance.

 

[Calenhad]

One theoretical benefit of having a drive with a reported 4K "physical" sector size, is that a file system with 4K allocation unit size will perform one write or read operation, instead of 8, per 4K unit. But due to how modern ssd controllers work, this is most likely not a performance issue anyways. The worst case scenario is higher temps for some controllers. And depending on the NAND used, the actual read and (especially) write operations can not directly address 4K sized groups of cells anyway.

 

[Mussels]

I have doubts of the performance benefits myself, but maybe some drives have some and some don't. At any rate the threads subject is still about the LBA size.

I haven't been able to find any reports of this actually doing anything, other than OS's that were originally 4K unaware and potentially had misaligned partitions in the first place.

is that a file system with 4K allocation unit size will perform one write or read operation, instead of 8, per 4K unit.

they buffer these things, they're 4K natively with 512 emulated - not the other way around. The only time they'd do those extra writes is a really old OS, like formatting the drive in DOS and running XP on it.

Everything is buffered in the DRAM or HBM, so writes this small should never happen one at a time, regardless

 

[chrcoluk]

Its just a firmware change, in it makes the drive report itself as a native 4k instead of 512 emulated.

If you already using a filesystem with 4k blocks then changing this firmware setting wont affect performance.

Its main benefit is to filesystems like ZFS automatically configuring a 512byte ashift which it is prone to doing on 512e drives.

Personally I dont know why drive vendors are still producing 512e drives. Is there really that much demand for it?

4k formatted FS should reduce erase cycles vs 512byte, which is probably the bigger reason to use 4k not performance.

 

[Veseleil]

Personally I dont know why drive vendors are still producing 512e drives. Is there really that much demand for it?

Many companies still use Win XP, along with hospitals, civil services like police etc., and governments worldwide.

 

[R-T-B]

Many companies still use Win XP, along with hospitals, civil services like police etc., and governments worldwide.

This is probably a factor too. It's been surprisingly hard to kill XP, especially in small integrated machines like POS.

 

[Assimilator]

Smells like snake oil to me.

 

[Athlonite]

Many companies still use Win XP, along with hospitals, civil services like police etc., and governments worldwide.

And most likely on old hardware that at most may have an SATA SSD replacement for the original HDD but definitely not an NVMe SSD though

 

[Espionage724]

I have a Hynix P31 and by-default it's in 512-emulation mode. I didn't see it advertised anywhere for this drive, but it exposes a 4K sector LBA.

Switching to it on Windows exposed some kind of incompatibility with Steam that I ran into: https://steamcommunity.com/discussi...76804012/?tscn=1641033020#c320374734291541958

If I plan on a Windows install, I keep it in 512e mode, but if I do a Linux install, I do 4K since it seemingly works fine.

Has anyone tried installing iTunes with the install drive as 4K sectors (not 512e)? I suspect it causes issues with Bonjour's installation, causes dnssd.dll to not be found, and causes iTunes itself not to work.

 

[Mussels]

Has anyone tried installing iTunes with the install drive as 4K sectors (not 512e)? I suspect it causes issues with Bonjour's installation, causes dnssd.dll to not be found, and causes iTunes itself not to work.

This seems entirely impossible, your .dll is missing for another reason

 

[mechtech]

On Asus B550 options. Interesting no raid support on 4k

 

[Hardcore Games]

I use 4096 byte blocks with NTFS as this allows for more small files to be stored on a given volume. SSD and HD alike all are fine with 4K blocks,.

4K emerged as disks capacity grew in sectors per track and beyond the error correction possible. Switching to larger sectors allowed for fewer error recovery blocks needed and a capacity gain was achieved.

check disk allocation:
fsutil fsinfo sectorinfo d:

where d: is the disk under consideration, use PowerShell or command prompt