Larger cluster size?

[Bundy]

I have read scattered over the interwebz that setting a larger cluster size than the default NTFS 4096bytes can result in improved performance at the expense of storage efficiency. I have plenty of space on the main OS drive so am able to use a larger size but was wondering, has anyone else tried this?

For reference, my system

RAID 0, 2 x 320Gb
Stripe 128k
cluster 4k
average file size 1190k (this looks skewed due to some very large files, the mode will be less)

 

[hellrazor]

Yeah it will - but not by anything really noticeable (unless you bump it up to like 32k, but don't do that).

 

[Bundy]

Why not 32k? I have heaps of space. 429GB free of 596GB. Is loss of storage the only disadvantage?

 

[Hayder_Master]

interesting thread i know large cluster raise up the performance but don't know if there is problem with high cluster or not, am interesting to read the replays

 

[Batou1986]

To my knowledge larger clusters means more wasted space on your drive, a 8k file will take up a whole 32k cluster resulting in the drive having to seek more to load things that are less then 32k and result in more fragmentation.
The performance difference is close to nothing, the stripe size is what id be more concerned with optimizing.

 

[Mussels]

short version:


if you had a 16KB file and a 4KB cluster size, then it has to write four seperate clusters for the one file. Thats four chunks to fragment, four chunks to be read to get that one file, etc.
(This is where the performance boost comes in, its less likely to fragment, and easier to defrag too.)


The reverse is that a 4KB file on a 16KB cluster, takes up the whole 16KB - the leftover space is wasted.


it should also be noted that windows doesnt like to boot off drives that have larger than 4KB clusters, it breaks file and folder compression and windows doesnt like that for some reason.


IMO, OS drive at 4KB, all other drives at the maximum (64KB i think) - unless you know its going to be full of thousands of really small files. (seriosly, look at your files - how many <64KB files do you have except for the odd .txt or .nfo file?)

 

[Hayder_Master]

sorry to hack on your thread my friend but i was think about ask same thing if you don't mind

To my knowledge larger clusters means more wasted space on your drive, a 8k file will take up a whole 32k cluster resulting in the drive having to seek more to load things that are less then 32k and result in more fragmentation.
The performance difference is close to nothing, the stripe size is what id be more concerned with optimizing.

so if there is no notice in performance so it's not useful, other thing i see some gig's waste now isn't important for better performance cuz HDD's have large size now

 

[Mussels]

sorry to hack on your thread my friend but i was think about ask same thing if you don't mind




so if there is no notice in performance so it's not useful, other thing i see some gig's waste now isn't important for better performance cuz HDD's have large size now

please see my post above yours, in case you missed it.

it seems a lot of people are going by really outdated information, when tiny little files were more common - and back when people didnt run so many large HDD's.

if you can reduce the amount of clusters by a factor of 16 (4KB -> 64KB), thats 1/16th the amount of fragments for your HDD to seek to, that need defragging, etc.

edit: oh and dont forget seeks times, if the drive takes 10ms to seek a file, 16 times more chunks will take quite a bit longer, even if its defragged and they're all in physical order on the disk (which aint always that likely)

 

[Hayder_Master]

please see my post above yours, in case you missed it.

it seems a lot of people are going by really outdated information, when tiny little files were more common - and back when people didnt run so many large HDD's.

if you can reduce the amount of clusters by a factor of 16 (4KB -> 64KB), thats 1/16th the amount of fragments for your HDD to seek to, that need defragging, etc.

sorry mussels i was writing in same time with u and u finish your post before me, until i post i see your post sorry just i take more time than you when writing

 

[Mussels]

heres a good example from my games drive, with 64KB cluster size.

i get 1/16th the fragments (awesome for defragging/seek times) and not much in the way of wasted space. 7GB lost on a games drive (that is ALL games, which often have smaller dll files and txt files with them) - so this is pretty much worst case scenario here.

my storage drives have next to no wasted space at 64KB.

 

[Bundy]

it should also be noted that windows doesnt like to boot off drives that have larger than 4KB clusters

That does it for me I was just wondering how I might squeeze a bit more out of my HDD's. I can see a SSD living here one day, but in the meantime I must do with what I have.

 

[Bundy]

sorry to hack on your thread my friend but i was think about ask same thing if you don't mind

No worries mate

I thought for sure that the drive (RAID 0) would run better with a bigger cluster size, I'm a bit dissapointed that windows might stop me from trying it out.

 

[Mussels]

No worries mate

I thought for sure that the drive (RAID 0) would run better with a bigger cluster size, I'm a bit dissapointed that windows might stop me from trying it out.

partition? having a large C: partition is just silly in its own right, so why not partition the rest into D:, and use a larger cluster for that?

 

[Yukikaze]

(seriosly, look at your files - how many <64KB files do you have except for the odd .txt or .nfo file?)

If you look up studies about the distribution of file sizes, you might be surprised. There have been many studies on this topic.

If we go by "A Large-Scale Study of File-System Contents" by John R. Douceur and William J. Bolosky of Microsoft Research, then we will find out that on a Windows file system around 80% of the files are 16k in size or less. For a study done for UNIX files systems around 80% of the files are 32kb in size or less ("File Size Distribution on UNIX Systems—Then and Now" by Andrew S. Tanenbaum, Jorrit N. Herder and Herbert Bos of Vrije Universiteit in Amsterdam).

 

[Mussels]

If you look up studies about the distribution of file sizes, you might be surprised. There have been many studies on this topic.

If we go by "A Large-Scale Study of File-System Contents" by John R. Douceur and William J. Bolosky of Microsoft Research, then we will find out that on a Windows file system around 80% of the files are 16k in size or less. For a study done for UNIX files systems around 80% of the files are 32kb in size or less ("File Size Distribution on UNIX Systems—Then and Now" by Andrew S. Tanenbaum, Jorrit N. Herder and Herbert Bos of Vrije Universiteit in Amsterdam).

those studies include the OS, which if you'll notice i stated should be on a separate partition.


if you use seperate partitions and drives, your OS drive should be 4K, your games drive should be somewhere in the middle (8/16K) and any and all storage drives should be 64K.

 

[Yukikaze]

those studies include the OS, which if you'll notice i stated should be on a separate partition.


if you use seperate partitions and drives, your OS drive should be 4K, your games drive should be somewhere in the middle (8/16K) and any and all storage drives should be 64K.

Yes, they do include the OS. But how many of the small files on the HDD are located in the OS and how many are not? Unless you can show that the OS takes up a significant percentage of those small files, then what you saying is still hanging out there.

Note, that I am not saying that you are wrong. You might well be right, but unless you have something proving your statement, then it is just a shot in the dark.

 

[Mussels]

Yes, they do include the OS. But how many of the small files on the HDD are located in the OS and how many are not? Unless you can show that the OS takes up a significant percentage of those small files, then what you saying is still hanging out there.

Note, that I am not saying that you are wrong. You might well be right, but unless you have something proving your statement, then it is just a shot in the dark.

the OS certainly does include a large amount of small files. let me find something to screenshot.


Windows vs games drive.

despite the games drive having 310GB more data, the file count is only 30,000 higher.
or in other numbers - (roughly) 11.7x more data, for 1.3x higher file count.

Games are fairly common to see small files, yet the OS has 10x more small files per GB than a games only drive. storage/backup drives would differ even more.

didnt bother including a storage drive, should be clear to everyone that backups are often compressed (thus, single files) or its storing avi's/MKV's/ISO/exe etc which are all going to be larger, single files.

 

[Bundy]

partition? having a large C: partition is just silly in its own right, so why not partition the rest into D:, and use a larger cluster for that?

It's big by coincidence, I originally purchased them to run an OS and store lots of files but now I store externally so the drive is becoming empty. What is the disadvantage in having a large C: partition?

 

[Mussels]

It's big by coincidence, I originally purchased them to run an OS and store lots of files but now I store externally so the drive is becoming empty. What is the disadvantage in having a large C: partition?

backups, and reinstalling windows.

to make a working OS you need to reinstall the OS onto the C: drive, so the less data there is there the less you need to backup and restore.

i find a 20GB partition enough for XP, vista or 7 i use 60GB. (mostly to cover things like office/photoshop, which are getting bigger every year)


also, the OS drive fragments itself constantly - so its a good idea to lock that performance degradation onto a smaller partition, leaving the rest of your data in peak condition.

 

[Bundy]

Thanks! That clears that one up too then.