FB-DIMM advice and benchmarks ♪

[lemonadesoda]

This thread is for FB-DIMM discussion, for those TPU members using, or planning to use, FB-DIMM based DDR2. You are welcome to post benchmarks, advice, experience, experiments, etc. Really good facts/infos that are posted in this thread will be transcribed as a Q&A in this first post, for quick reference.

Q1. Should I use x4 or x8 DIMMs? What's the difference?

A1. I have found it very difficult to track down info on the net on this. The best source I have found is: http://download.micron.com/pdf/technotes/ddr2/tn4721.pdf , page 13 and 14. Use x8 (lower power) according to p14

x4 seems to come with advanced feature set AMB, but for most workstation users the features actually reduce performance... but possibly improve reliability in a server situation. If you need reliability at the sacrifice of performance, then x4 may give you additional options.

Q2. 1.8v or 1.55v

A2. FBDIMMs use a lot of power and get hot. The low-voltage 1.55V FBDIMMs will make a big difference over 1.8v in terms of power requirement and heat output. Most i5400 based MBs are compatible with 1.55v, but check first! Not every board is.

*update* it seems that the 1.5v FB-DIMMs have disappeared off the market. I had ordered some 3 months ago, they were "wait listed", now the supplier says they cannot source them at all. I think some fab in Asia has stopped making them. (Moved on to DDR3 I guess).

Q3. Single or dual rank?

A3. Dual rank is faster than single rank at the same speed. Bandwidth is higher because each rank can be accessed independently. See the benchmarks (post#7) to look at performance of various RAM configurations.

You can expect a 20-30% improvement in memory speeds using dual-rank over single rank RAM. That's very similar performance improvement like going from single to dual channel.

Q4. How to spread the FB-DIMMs? All in one channel (increasing rank), or one in each channel

A4. See reference 1 p110. Better to have one DIMM in each channel, allowing memory performance to scale from single, to dual, to quad channel. Even better still to have two DIMM in each channel (or single DIMM but dual rank in each channel), to have each channel operate at 81% efficiency instead of 59%. See ref 1 p 109

Useful Resources:
1./ http://www.ece.umd.edu/~blj/papers/thesis-MS-nasr--FBDIMM.pdf
2./ http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00913926/c00913926.pdf
3./ http://download.micron.com/pdf/technotes/ddr2/TN4716.pdf

 

[lemonadesoda]

Some benchmarks found on http://www.legitreviews.com/article/724/4/

... although the results seem very slow and somewhat inconsistent, e.g. Everest read and write performance of dual and quad channel. We know that read performance should not be lower than write performance on FB-DIMM, and there is no reason why write is scaling on quad channel whereas read isnt. Something is up.

 

[Deleted member 3]

Q3. Single or dual rank?

A3. I cant get to the bottom of this one. Anyone got any experience? It seems that dual rank has double the number of chips, and therefore uses more power... BUT it could be that the bandwidth is higher, since it is equivalent of having two DIMM sticks in one channel. ???

As you say below, more ranks per channel is better. So 4-rank modules are even better.

Q4. How to spread the FB-DIMMs? All in one channel, or one in each channel

A4. See reference 1 p110. Better to have one DIMM in each channel, allowing memory performance to scale from single, to dual, to tipple to quad channel. Even better still to have two DIMM in each channel, to have each channel operate at 81% efficiency instead of 59%.

Basically it's not DIMMs you want per channel, but ranks. ie many entry level boards have 6 banks, you can still run quad channel with 6 DIMMs cause of this. (ie 4x 512mb single rank, 2x 1GB dual rank)

 

[Deleted member 3]

Some benchmarks found on http://www.legitreviews.com/article/724/4/

... although the results seem very slow and somewhat inconsistent, e.g. Everest read and write performance of dual and quad channel. We know that read performance should not be lower than write performance on FB-DIMM, and there is no reason why write is scaling on quad channel whereas read isnt. Something is up.

Nothing is up, those numbers seem perfectly normal. Though they can get a bit higher.

 

[lemonadesoda]

Nothing is up, those numbers seem perfectly normal. Though they can get a bit higher.

Could be. You may well be getting similar results. But I dont think WRITE should ever be faster than READ on FB-DIMM. Technically, it shouldn't be, given how data is sent. So this may be a quirk of the Everest algorithms being used to test and calculate bandwidth.

Using Everest I have just discovered that READ performance goes DOWN when moving from single channel to dual channel FB-DIMM, whereas WRITE performance went up about 50% from single to dual channel.

Something is a little funky with the method everest is using, or it may be "correct" but just doesnt show FB-DIMM in it's true light. ALL OTHER benchmarks go UP moving from single to dual channel FB-DIMM, e.g. generic CPU crunch, cinebench, etc.

Quad channel will be tested once I get these apple heatspreaders modded. There is 2mm overlap between memory modules. The Apple obviously has DIMM slots a bit wider apart than ASUS.

 

[lemonadesoda]

Benchmark Test FB-DIMM, single RANK, Micron, 1.5v/1.8v, 1GB, DDR2-667, on Xeon chipset 5400B with 2xE5420 at stock (2.5Ghz).

Using EVEREST 5 - Memory Copy
1 channel = 2,912 Mb/s
2 channel = 3,596 Mb/s
4 channel = 3,996 Mb/s

Using EVEREST 5 - PhotoWorxx
1 channel = 10,583 score
2 channel = 13,778 score
4 channel = 20,016 score

Using Geekbench 2.1.2
1 channel = 6,708 score
2 channel = 7,162 score
4 channel = 7,323 score

 

[lemonadesoda]

OK, I've swapped my Apple single rank RAM out for Kingston dual-ranks. Here are the new stats:

Using EVEREST 5 - Memory Copy
1 channel = 2,912 Mb/s (single rank)
2 channel = 3,596 Mb/s (single rank)
4 channel = 3,996 Mb/s (single rank)
4 channel = 4,738 Mb/s (dual rank) = +20%

Using EVEREST 5 - PhotoWorxx
1 channel = 10,583 score (single rank)
2 channel = 13,778 score (single rank)
4 channel = 20,016 score (single rank)
4 channel = 25,980 score (dual rank) = +30%

Using Geekbench 2.1.2
1 channel = 6,708 score (single rank)
2 channel = 7,162 score (single rank)
4 channel = 7,323 score (single rank)
4 channel = 7,324 score (dual rank) = +0%

Using WinRAR 3.8 Benchmark
1 channel = 1276 kb/s (single rank)
2 channel = 1359 kb/s (single rank)
4 channel = 1686 kb/s (single rank)
4 channel = 1844 kb/s (dual rank) = +10%

There is a BIG improvement in memory intensive applications, e.g. PhotoWorxx, a test of processing large amounts of photo data requiring huge quantities simultaneous reads and writes. 30% improvement with dual rank! Clearly this is a memory bottlenecked process and the dual rank setup has significantly improved performance.

There is NO improvement in CPU intensive applications that dont rely on memory bandwidth.

You can therefore expect an application performance improvement somewhere between 0% and 30% depending on application.

I added the WinRAR benchmark which is a mix of CPU power and memory bandwidth. Perhaps this is a more typical scenario than synthetic benchmarks. Result 10% performance improvement.