trying the near impossible... most powerful GOD BOX in a miniITX case

[venturi]

I wasn't even sure where to post it as I have some questions that don't fall into any particular category.


I decided to ask the question: what is the most powerful build I can do and keep it tiny, and run of a normal household socket.

This was my first list, and I stuck to it:

Asus c621e sage
Dual Xeon 8180M
384GB of ram
4x Titan V including Titan V CEO ed 32GB
960pro nvme (OS)
10x SSD RAID (Apps and Backup 8x 2x )
1600W PSU (digital) noiseless
MS 2016 Data Center, Ubuntu
TT miniITX case


reverse engineering the design to fit in a small enough case proved a bit challenging:

The case had to be deconstructed and rebuilt better, reinforced and be given new hard points.
Wires and connectors had to be remade by hand to do the low profile turns.
Cable management had to be ZEN like.
It would have to run cool all by itself and be rock stable.

so after a week of getting parts in and building the toy (after the daytime job as to be able to afford this little hobby)

well, here is the mostly complete and fully functional final result:

It looks harmless enough, but it was a lot of work,

The drives had to be RAIDED and stacked in their own cooling tower

The 4x Titan V required decent power that had to be shared with the dual 8180M xeons

those required a little order and management

while I was building it, with all its guts (wires) hanging out, I ran a quick cinebench to see how it would do

score: 8857

temps seem reasonable under light load, temps rise about 10-15 'F for cpus under heavy load (up to about 40-41'C)
video cards can get as hot under load as 68'C

so it runs fast and stable, games look amazing and are smooth at 4k+, not to mention the little bit of work I use it for (deep learning, clinical decision support, isotopes, medical imaging, Lara Croft, etc)


SO I have some questions that answers seem to exist NOWHERE on the net.

Questions from bios

1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)

2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?

Question on Video cards.

1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card? I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.

I really need to tidy those wires up....


Anyhow, any help on the questions would be greatly appreciated.

Grazie


V

 

[Flyordie]

Just here to admire the hardware. lol. Not gonna lie, its a damn awesome build. :-o

 

[venturi]

Thank you, hopefully I'll get enough time off from work to actually play on it

 

[lexluthermiester]

So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.

 

[Mescalamba]

This would belong to XtremeSystems for others to admire, if it was still alive.

Quite insane machine.

Would probably try to watercool it with external big radiator. But ofc it would be bigger. Awesome nonetheless. And insanely expensive..

 

[Mussels]

And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?

 

[lexluthermiester]

How do the titans handle being so closely stacked together heat-wise?

While I can't speak for the OP, being in an open-air case like it is likely gives plenty of room to breath and thus heat may not be a problem. That's my hypothesis.

 

[venturi]

And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?

The titans, in the room at 21-22C ambient, run idle at about 29'C. Under load they can get as high as 68'C (on occasion even 71'C)

I removed the back plate to the titans and this allowed more room and heat dissipation. The back plate has a plastic layer under the metal plate and the whole thing is more of an insulator -so I got rid of it. Cards are very happy. So I don't really experience any thermal throttling, thermal throttling (t would occurs between 85'-91'C)

I have attached the temps on medium -light load

So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.

I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.

 

[steen]

First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.

 

[venturi]

First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.

I did not see a priority or sequence setting in the bios for the pcie slots.omly optometrist enable ir dissable.
Disabling prom but for all except the one I want yielded lots of beeping and no post, so I reverted back.

Unfortunately the microcode mitigation’s hurt performance, and I’d rather not have them if they are a negative on perf. I disabled the os mitigation’s by adding ‘3’ to the dword in the feature section in reg/meme I’m HKLM and that helped substantially on raid performance and video performance

 

[FordGT90Concept]

1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)

Might be ASUS branding for Intel Turbo Boost. Was it enabled by default? If yes, I assume they are one in the same.

2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?

You could find the most recent BIOS for the board that don't have the specter/meltdown fixes. The board is fairly old so most of the non-CPU microcode fixes should have been implemented in earlier BIOS revisions. The board is roughly a year old. The BIOS may have had the specter/meltdown fixes baked in from the first revision. I'd just leave well enough alone.

1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card? I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.

When a computer boots, it enumerates displays and connected monitors. The first card with a connected display gets flagged as primary until the driver loads. If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card. The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot. Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics). Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.

 

[FreedomEclipse]

yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good good friend of mine.

 

[venturi]

Might be ASUS branding for Intel Turbo Boost. Was it enabled by default? If yes, I assume they are one in the same.


When a computer boots, it enumerates displays and connected monitors. The first card with a connected display gets flagged as primary until the driver loads. If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card. The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot. Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics). Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.

The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.



Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like, does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.


I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...

yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good good friend of mine.

Ok, but there might be a slight handling charge to go with the shipping cost....

 

[FordGT90Concept]

The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.

I looked through all of the documentation I could find including manual and BIOS updates and there are no references to it.

Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like, does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.

That's pretty stupid. Your work around is probably the best you're going to get then, sadly.

I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...

That's terrible. Maybe got that response because it's technically a workstation/business board. Might be a separate department or tier 2 support that handles them.

 

[blobster21]

Congratulation for this godbox.

How much did you spend for the hardware ?

 

[XXL_AI]

Congratulation for this godbox.

How much did you spend for the hardware ?

looking at the specs, it must be around tesla model 3 level.

 

[venturi]

Congratulation for this godbox.

How much did you spend for the hardware ?

Cost: pound of flesh and a pint of blood

Over the years, I've paid for this hobby.

over the course of 20 years, I've built god boxes along the way. At least, in their own time they were consider unique.

19 years ago, I was the first to water-cool a dual cpu rig, chipset and video card.. I used a motorcycle oil cooler for the radiator.

I also was the first to overclock a duallie.

15 years ago I was one of the first to QUAD Sli

I was the first to overclock/watercool video cads from voodoo, specter, stb etc and the obsidian x-24 dual gpu.

I was the first to overclock the 3dlabs willdcat card and 17 years ago had a VR headset working with gyros off. The wildcat card was the first integrated dual GPU.

A few years ago I was the one that implemented 4x quad SLI on pascal as soon as nvidia tried to disable the 3 and 4 way sli on pascal - I have it fully functional and with nice proportional scaling.



here are some of the pictures from those years:

yes, even the chipset on this duallie

yes, that was really first qua slI

dual PSU, cable management, quad sli and all 8 scsi drives are raided, circa mid 2000s

I took 6 years off as I worked on macs, windows vista made me leave the MS world

P5

P3

P1 - latest

ending with where things are right now

I only build mutlti cpu/ multi gpu / with Raid rigs



well, that was 20 years, enjoy

 

[notb]

I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.

Any noise and heat concerns?
It's basically a server in an open case.

 

[Vario]

My budget, it would be near impossible.
With your budget, it is possible.
I am estimating this machine cost around $40,000.

 

[blobster21]

looking at the specs, it must be around tesla model 3 level.

Yes i think you're spot on.

 

[venturi]

Any noise and heat concerns?
It's basically a server in an open case.

Runs silent, runs cool, no heat or noise issues. please see earlier pic with temps.

Even the power supply fan seldom comes on, and that is only when the unit is maxed with all 4 gpus going.
Under load, the gpus do get louder, but its not bad. I have not heard the gpu blower fans max-up yet.

I also made a lot of sections on TIM, example, every gpu was dismantled and reassembled with better TIM and thermal pads that were much better than factory.

I made a lot of effort in selecting fans, fan blade, rpms, and positioning for cool and quiet.

 

[Vario]

What heatsinks and fans are you running on those 28 core monsters? Looks like Noiseblockers? I wonder if the CPU would cool better on other heatsink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.

 

[venturi]

What heat-sinks and fans are you running on those 28 core monsters? Looks like Noiseblockers? I wonder if the CPU would cool better on other heat-sink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.

Orientation is limited. The fans are in pull mode, not push. The air is pulled up and out the top. There wasn't enough room for the fans in push mode. However, I tested it with one less video card in place in push mode and found that pull mode is 1-2 'C cooler (in PULL mode). So the orientation and pull mode ended up being the most efficient.

My primary card is on the bottom, when all 4 gpus are going full the pull mode does not acquire a waft of warm air, instead, all 4 blower gpus push warm ait out the back pulling cool air in. The result is that the chipset and surrounding area are receiving cooler air. Again, tested both ways. But I will be exploring alternative alignments. In push mode (down) the 4th card was being warmed up by the 2 cpu coolers and it was fighting the natural rise of heat.

The fans are noise blocker, 1800rpm, PWM. The heatsinks are Noctua NH-D9 DX-3647 4U

my preferred cpu cooler/fan orientation is push mode from right to left - but the market is sparce and the noctua 92mm is only SILENT configuration i can find. There are Supermicro/Dynatron coolers that will fit but sound like a hair drier compared to the noctuas

please see temp chart under medium load in a 73'F room:

 

[notb]

Runs silent, runs cool, no heat or noise issues. please see earlier pic with temps.

Misunderstanding. I didn't mean temperature of the components. I really meant heat.
At full load you'll be near that 1600W the PSU delivers. Essentially, it's like a small heater, isn't it?

As for the noise: Titans are one of the loudest cards around. I had the "pleasure" of working with them lately. Just a single card in a Dell tower and I had to use headphones to concentrate on math. But good luck. ;-)

I wonder if the CPU would cool better on other heatsink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.

It's more efficient to push air into the heatsink than pull from. I bet that's what he's doing.
Similarly, it's better to get air through CPU first and onto GPUs. GPUs are usually operating far from thermal limits and they can take it.
That's also how servers work (disks -> CPUs -> GPUs).

 

[XXL_AI]

Titan V at full load can consume up to 300W when it peaks, 4 of them so it is 1200W for the GPU, each CPU is rated for 205W but we know intel always lies so count it as 250W, 500W for CPU, I didn't even count the ram sticks and ssds here and it is already reached to 1700W peak, you have 1600W, you'll have PSU issues if you want to run deep learning/machine learning applications at the long run.

but, since you are running windows and not using all the capability of the machine for running deep learning and only have presumably a single 4K screen you will never going to utilize all the gpus with gaming, even if you push all the limits of the hardware you have you'll stay under 1600W but not much, it'd better to have dual PSU or something at the realm of 2000W at least.

there are tons of tweaks on the nvidia hardware, so if we take those into the consideration, when you want to train a dataset you can halve the gpu power and save up to 700W from GPUs, since dataset training uses barely any cpu, we can add 100W, and another 200W for ram and storage, in the minimum consumption you'll again hit ~1000W it is a small heater and you'll need active airflow on the motherboard which I don't see on your photos. servergrade motherboards rely on active airflow and horizontal usage, you'll damage $30000 build no more than 2 years with active use.

if the room is 73F, how the hell SSD's running at 68F?