12-core Xeon engineering workstation build

[PooPipeBoy]

For years now I've had some ideas brewing in my mind of what I'd like to use for an engineering workstation and now I finally have the parts at my disposal to make it happen. This is going to be a very interesting one.

NOTE: This build definitely is NOT how you SHOULD build a workstation. But instead for personal use it will be plenty powerful enough for my own purposes.

There's a couple of stipulations I had for this build:

1. CPU must be a Xeon on a relatively modern architecture (less than 10 years old) with lots of cores and cache.
2. ECC registered memory, something I've never personally used and would be great to have.
3. Quadro graphics card that is powerful enough to play a game or two.

For those who want to get straight into the specifications list, here you go:

Spoiler: Xeon Build Specifications

• Intel Xeon E5-2678 V3 X99 12-core Haswell processor
• Coolermaster Hyper T4 cooler
• Huananzhi X99-TF motherboard
• 4x8GB Samsung DDR3 ECC 1600MHz dual rank PC3L memory
• Nvidia Quadro K620 2GB graphics card
• Samsung 850 Evo 250GB SSD
• 1TB 7200rpm hard drive?
• Fractal Design Core 2300 case
• Zalman ZM-700SV power supply

I've already done some initial testing to make sure nothing is DOA. Everything is working and good to go!

More to come.

 

[PooPipeBoy]

The heart of this build will be the Xeon E5-2678 V3 processor. 12 cores on the Haswell architecture with 30MB of cache should be plenty if I want to run multiple workloads at the same time. Yeah the single core performance and gaming performance isn't brilliant, but it's not like I'm a heavy gamer and I've got my main Ryzen system for that stuff anyway.

I'm also keen to try out China's finest X99 motherboard, the Huananzhi X99-TF. I ordered the motherboard and processor as a bundle, totalling US$250. But it comes with a bunch of expansion features and has received lots of praise from reviewers for being a great budget motherboard. It also supports both DDR3 ECC memory (cheaper than DDR4 ECC), which is what I'm going for.

Speaking of memory, this was a great used deal. I got 32GB of DDR3 ECC registered 1600MHz Samsung used server memory for US$60. I opted for dual rank PC3L (1.35V instead of 1.5V per module) which is thankfully compatible with this motherboard.

For the graphics I already had the perfect candidate just sitting around and being used to test other systems. I think I paid US$30 about a year ago for this one, which was a steal considering you get 384 cuda cores on the Maxwell architecture and 2GB of DDR3 VRAM. Can't overclock this guy unfortunately (as with any Quadro) and you don't get GDDR5, but you do get CUDA version 5.0 and it performs similar to a GTX 650 in games. Should be great for 3D modelling and other workstation tasks.

 

[PooPipeBoy]

Reserved, most likely for a review of the motherboard (500th post, wooooot!)

 

[PooPipeBoy]

For the case I'm going with a brand new Fractal Design Core 2300 case for US$55 that can accept a full size ATX motherboard. Looks nice and unassuming, it's sturdy, and it has the provisions of top fan ventilation and two 5.25" drive bays.

Installing components on the board and just verifying thermal paste contact patch. The Hyper T4 I'm using has a smaller contact patch than the processor IHS, but I'm not too worried when the cores run very cool anyway and it's also easier to clean up the paste when it stays on top of the IHS.

Apparently the securing ring they've included for the heatsink is compatible with any AM4-style cooler. Works for me.

 

[oldwalltree]

Awesome!

 

[PooPipeBoy]

I pulled the processor out to check it out. Processor came pre-installed in the socket. Actually the first time I've ever even seen an X99 socket in person.

Final assembly of motherboard components. I actually quite like this orientation of the heatsink so I might leave it like this for now.

Test fitting the board and getting the standoffs in. Just enough room to fit the board in. One of these days I need a better tool to install standoffs because using pliers is so tedious and cumbersome, and it's super easy to scratch everything.

 

[storm-chaser]

For years now I've had some ideas brewing in my mind of what I'd like to use for an engineering workstation and now I finally have the parts at my disposal to make it happen. This is going to be a very interesting one.

NOTE: This build definitely is NOT how you SHOULD build a workstation. But instead for personal use it will be plenty powerful enough for my own purposes.

There's a couple of stipulations I had for this build:

1. CPU must be a Xeon on a relatively modern architecture (less than 10 years old) with lots of cores and cache.
2. ECC registered memory, something I've never personally used and would be great to have.
3. Quadro graphics card that is powerful enough to play a game or two.

For those who want to get straight into the specifications list, here you go:

Spoiler: Xeon Build Specifications

• Intel Xeon E5-2678 V3 X99 12-core Haswell processor
• Coolermaster Hyper T4 cooler
• Huananzhi X99-TF motherboard
• 4x8GB Samsung DDR3 ECC 1600MHz dual rank PC3L memory
• Nvidia Quadro K620 2GB graphics card
• Samsung 850 Evo 250GB SSD
• 1TB 7200rpm hard drive?
• Fractal Design Core 2300 case
• Zalman ZM-700SV power supply

I've already done some initial testing to make sure nothing is DOA. Everything is working and good to go!

View attachment 189423

More to come.

Good call on the CPU. Pretty good bang for the buck if you ask me. OEM Xeons are the way to go and typically . Usually running higher/very similar clocks using less voltage vs their retail counterparts.

Matter of fact I was eyeing that very CPU for my next workstation build.

I also have two of them in my C4130 GPU server and it's definitely a powerhouse of a chip.

 

[Isaac`]

Dam this would make a pretty nice compile machine
Certainley be faster then my 3200g

 

[storm-chaser]

The heart of this build will be the Xeon E5-2678 V3 processor. 12 cores on the Haswell architecture with 30MB of cache should be plenty if I want to run multiple workloads at the same time. Yeah the single core performance and gaming performance isn't brilliant, but it's not like I'm a heavy gamer and I've got my main Ryzen system for that stuff anyway.

View attachment 189424

I'm also keen to try out China's finest X99 motherboard, the Huananzhi X99-TF. I ordered the motherboard and processor as a bundle, totalling US$250. But it comes with a bunch of expansion features and has received lots of praise from reviewers for being a great budget motherboard. It also supports both DDR3 ECC memory (cheaper than DDR4 ECC), which is what I'm going for.

View attachment 189425

Speaking of memory, this was a great used deal. I got 32GB of DDR3 ECC registered 1600MHz Samsung used server memory for US$60. I opted for dual rank PC3L (1.35V instead of 1.5V per module) which is thankfully compatible with this motherboard.

View attachment 189426

For the graphics I already had the perfect candidate just sitting around and being used to test other systems. I think I paid US$30 about a year ago for this one, which was a steal considering you get 384 cuda cores on the Maxwell architecture and 2GB of DDR3 VRAM. Can't overclock this guy unfortunately (as with any Quadro) and you don't get GDDR5, but you do get CUDA version 5.0 and it performs similar to a GTX 650 in games. Should be great for 3D modelling and other workstation tasks.

View attachment 189427

I have a couple extra Quadro video cards in my stockpile here would you be interested? Both are better than this one. If you are interested I can round them up and get you the particulars.

 

[tabascosauz]

I pulled the processor out to check it out. Processor came pre-installed in the socket. Actually the first time I've ever even seen an X99 socket in person.

View attachment 189579

Call me eccentric, but something about the Haswell-E and Broadwell-E IHS just rubs me the right way, and no other CPU since then has done quite the same

 

[bobbybluz]

I have a couple extra Quadro video cards in my stockpile here would you be interested? Both are better than this one. If you are interested I can round them up and get you the particulars.

I may be interested in a Quadro for my E5 2683 V4 workstation.

 

[PooPipeBoy]

Good call on the CPU. Pretty good bang for the buck if you ask me. OEM Xeons are the way to go and typically . Usually running higher/very similar clocks using less voltage vs their retail counterparts.

Matter of fact I was eyeing that very CPU for my next workstation build.

I also have two of them in my C4130 GPU server and it's definitely a powerhouse of a chip.

I'm impressed by the 2678 V3 so far! While I was checking to make sure everything was working, it was idling around 61 watts and at full load was pulling 190 watts. It's not the power hog I was expecting it to be. Maximum temperatures on any of the cores was in the low 50C range. Multicore performance also seems to be not far off my Ryzen 5600X in Cinebench R15, maybe about 100cb lower.

I also tested my E5-2630L V3 eight core chip, and it's pretty good but it has around half the multicore performance and needs DDR4 memory.

Call me eccentric, but something about the Haswell-E and Broadwell-E IHS just rubs me the right way, and no other CPU since then has done quite the same

I know what you mean, it's like the chips haven't aged in their appearance at all. One thing I didn't expect is that these X99 chips have a very thick PCB on them (around 1.5 to 2.0mm) that's very different to any other CPU I've owned.

I have a couple extra Quadro video cards in my stockpile here would you be interested? Both are better than this one. If you are interested I can round them up and get you the particulars.

Nah that's alright, I'm good for now thanks. I should've pulled up on my spending a long time ago

 

[PooPipeBoy]

For the power supply I'm re-purposing my Zalman ZM700-SV 700W 80 Plus Silver unit that I just extracted from my main system. It's served me well for 7 years and it's a good candidate for a build that needs decent reliability. It's got a single 12V rail capable of 660 watts and the ripple is "up to 120mV". No idea on the actual performance because there's practically no reviews or documentation on this particular power supply. That's why it's a good idea to go with the more popular models.

The whole build is starting to get this nice white-on-black theme going on, looks quite nice so far.

 

[storm-chaser]

For the power supply I'm re-purposing my Zalman ZM700-SV 700W 80 Plus Silver unit that I just extracted from my main system. It's served me well for 7 years and it's a good candidate for a build that needs decent reliability. It's got a single 12V rail capable of 660 watts and the ripple is "up to 120mV". No idea on the actual performance because there's practically no reviews or documentation on this particular power supply. That's why it's a good idea to go with the more popular models.

View attachment 189592

The whole build is starting to get this nice white-on-black theme going on, looks quite nice so far.

View attachment 189593

Nice! It's like the inverse proportion to that of my

I have a couple extra Quadro video cards in my stockpile here would you be interested? Both are better than this one. If you are interested I can round them up and get you the particulars.

So I know I have another one around but for now its just the NVS 450. Also check specs and compare to make sure it's superior to your current GPU (obviously) lol

 

[PooPipeBoy]

So unfortunately I've come across potentially the first major disappointment with this build: The Huananzhi X99-TF motherboard may, in fact, not support error-correcting memory.

Okay I'll elaborate a bit. ECC memory is technically compatible with this board, in the sense that it will boot and work just fine. But the problem is that there seems to be no actual error-checking enabled behind the scenes. The DDR3 ECC registered memory will run just fine, but there's no actual ECC capabilities, therefore it just behaves like regular DDR3 memory.

Wuh, wuh, wuuuuuh....

I verified that all four DDR3 modules are definitely ECC and definitely registered using MemTest86. No error correcting though. I combed through every single BIOS setting for a good hour and couldn't find anything that would "enable" ECC. Resetting the bios, loading optimized defaults.....nothing.

Online information is conflicting to say the least:

Diego on 11 September 2020 02:09 says:

I have installed an E5-2650Lv3 in a Kllisre X99 LGA2011-3 with a Micron MTA9ASF1G72AZ–8GB (DDR4 ECC). I know is not an usual combination, but I was looking for a low power consumption with lots of cores. Everything is working ok (can install linux, runs memtest), but after testing it I have noticed that ECC, although detected, is not used. ECC support in BIOS is Enabled, lots of options about scrubbing, patrolling, but none about enabling usage of ECC.
Linux at boot says:
kernel: EDAC sbridge: CPU SrcID #0, Ha #0, Channel #0 has DIMMs, but ECC is disabled kernel: EDAC sbridge: Couldn't find mci handler kernel: EDAC sbridge: Failed to register device with error -19.
Memtest86+ also states that ECC is disabled, this is the easiest way to test, without booting Linux.
All the reviews I have found talk about the support of ECC memory, but I have never found more info regarding the actual usage of ECC. So I understand that support is only about the system using DIMMs with ECC capabilities, but not using the actual error correcction.
I have tested other DIMMs with ECC, and results are the same.
As far as I have discovered, something is missing in the motherboard or in the BIOS to enable ECC usage, or as some post in other forum says, there are missing traces in the motherboard that make ECC usage not possible.
You are by far the most expert person in chinese X99 boards. Have you ever run memtest to check if ECC is being used, as part of any of your reviews? What is your opinion about ECC usage? Maybe I could flash other BIOS that could add the support I'm missing? Nobody cares about using ECC?

Miyconst on 11 September 2020 09:18 says:

Diego, from your comment it's not clear which Kllisre X99 motherboard you talk about. I have validated Memtest86 on JingSha X99 D8 and Huananzhi X99-TF/F8/T8, these boards are surely using ECC, since I have some memory modules which are slightly faulty, and Memtest86 detects errors corrected by ECC.

So......does it support ECC? The answer seems to be a definite "yesn't".

I may need to run some memory tests to see if there are any ECC errors detected, and if there are no actual errors, it may be the case that ECC is actually working. Even though everything is currently reporting that it's not.

 

[Lorec]

I'm also keen to try out China's finest X99 motherboard, the Huananzhi X99-TF. I ordered the motherboard and processor as a bundle, totalling US$250. But it comes with a bunch of expansion features and has received lots of praise from reviewers for being a great budget motherboard. It also supports both DDR3 ECC memory (cheaper than DDR4 ECC), which is what I'm going for.
View attachment 189425

I love how those huananzhi board look nowadays! Ive been eyeing them for ages, still for my 12C xeon system (on x79) i preferred an asus board.
Looked and felt more reliable

 

[PooPipeBoy]

I love how those huananzhi board look nowadays! Ive been eyeing them for ages, still for my 12C xeon system (on x79) i preferred an asus board.
Looked and felt more reliable

For any serious build you're better off going with a real-deal motherboard, although my impressions of the Huananzhi board are that I've encountered surprisingly few problems so far. It fired up just fine on the first try and there's no weirdness going on. The RAM slots do feel quite cheap with how the mechanisms click into place, the matte black PCB stains easily, and the front panel pins aren't labelled at all. Manufacturer support for BIOS updates is non-existent. But those are mainly small niggling issues. You do get stuff like the on-PCB power button and code readout which are nice. VRM temperatures are in the mid-50C range and the two fans on there are quiet. It's strengths definitely outweigh the weaknesses and it seems good so far. Still yet to find out if ECC actually works or not.

It's been relatively budget-friendly, honestly. For the US$320 I've invested in the mobo/CPU/RAM combo for this rig, you could definitely get something else that would be much better for gaming. But in terms of the multi-core performance and the quantity of RAM, I'm not sure how you'd be able to beat this.

Right now it's been sitting there for an hour running MemTest86. No errors so far. I added an LED strip to see how that would look.

 

[PooPipeBoy]

Well I'm damn impressed right now. I had MemTest86 running for 7.5 hours during the night and it finished four passes without a single error reported. That's excellent for a Chinese motherboard and used Ebay memory. I was a bit peeved off that there's no error-checking capabilities, but with this result it might not matter anyway. No doubt now that the memory is reliable.

 

[PooPipeBoy]

Starting to roll some benchmarks. Apparently without the turbo boost hack you're still getting around 2.9GHz on all cores, not bad. I don't think I'll bother doing the turbo hack at this stage.

Multithreaded performance is getting within striking distance of the Ryzen 5600X:

Unigine Valley at 1080p Ultra with no anti-aliasing:

 

[storm-chaser]

Starting to roll some benchmarks. Apparently without the turbo boost hack you're still getting around 2.9GHz on all cores, not bad. I don't think I'll bother doing the turbo hack at this stage.

Yes 2.9GHz is the all core turbo frequency for that chip, effectively boosting your base clock substantially.

The 2680 v3 has identical turbo specs as well.

 

[PooPipeBoy]

Yes 2.9GHz is the all core turbo frequency for that chip, effectively boosting your base clock substantially.

The 2680 v3 has identical turbo specs as well.

I recall seeing in Memtest86 that the turbo multiplier is 32x on two cores. It had all the turbo specs listed out. I should bring it up again later to see what it said.

 

[bobbybluz]

Can you raise the BLCK on that mobo? If so, 103.5 has worked well for the E5 V3's and V4's I've tried in X99. The V3's max out at 2133 and the V4's at 2400 by default.

 

[storm-chaser]

I recall seeing in Memtest86 that the turbo multiplier is 32x on two cores. It had all the turbo specs listed out. I should bring it up again later to see what it said.

Here is handy chart that outlines the turbo operation of a 2678/2680 v3

 

[storm-chaser]

It also supports both DDR3 ECC memory (cheaper than DDR4 ECC), which is what I'm going for.

Does the board actually have provisions for both DDR3 and DDR4 (slots) on that motherboard?

 

[PooPipeBoy]

Here is handy chart that outlines the turbo operation of a 2678/2680 v3

View attachment 189771

33x on two cores, well there you go. I must be confusing it with the 32x multiplier that everyone runs these chips at on all cores with the turbo hack.

Does the board actually have provisions for both DDR3 and DDR4 (slots) on that motherboard?

Yeah the grey slots are DDR3 and the black slots are DDR4. The stickers show which Xeon chips have a DDR3 memory controller, although the E5-2678 V3 is the only affordable one. I believe the E5-2680 V3 (which is almost identical) doesn't have a DDR3 controller. Most of the Haswell-E chips out there are only DDR4 compatible, like my E5-2630L V3 that I also tried (which worked just fine on DDR4).

Depending on whether you go DDR3 or DDR4, you can get quad channel using either configuration.