• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

trying the near impossible... most powerful GOD BOX in a miniITX case

Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
I wasn't even sure where to post it as I have some questions that don't fall into any particular category.


I decided to ask the question: what is the most powerful build I can do and keep it tiny, and run of a normal household socket.

This was my first list, and I stuck to it:

Asus c621e sage
Dual Xeon 8180M
384GB of ram
4x Titan V including Titan V CEO ed 32GB
960pro nvme (OS)
10x SSD RAID (Apps and Backup 8x 2x )
1600W PSU (digital) noiseless
MS 2016 Data Center, Ubuntu
TT miniITX case



reverse engineering the design to fit in a small enough case proved a bit challenging:

The case had to be deconstructed and rebuilt better, reinforced and be given new hard points.
Wires and connectors had to be remade by hand to do the low profile turns.
Cable management had to be ZEN like.
It would have to run cool all by itself and be rock stable.


so after a week of getting parts in and building the toy (after the daytime job as to be able to afford this little hobby)

well, here is the mostly complete and fully functional final result:

IMG_9013.jpg


It looks harmless enough, but it was a lot of work,

IMG_9019.JPG


The drives had to be RAIDED and stacked in their own cooling tower

IMG_9012.JPG


The 4x Titan V required decent power that had to be shared with the dual 8180M xeons

those required a little order and management

IMG_9023.JPG


IMG_9015.JPG


while I was building it, with all its guts (wires) hanging out, I ran a quick cinebench to see how it would do

score: 8857

IMG_8889.JPG


temps seem reasonable under light load, temps rise about 10-15 'F for cpus under heavy load (up to about 40-41'C)
video cards can get as hot under load as 68'C


IMG_9061.JPG





so it runs fast and stable, games look amazing and are smooth at 4k+, not to mention the little bit of work I use it for (deep learning, clinical decision support, isotopes, medical imaging, Lara Croft, etc)


SO I have some questions that answers seem to exist NOWHERE on the net.

Questions from bios

1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)

2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?

Question on Video cards.

1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card? I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.


IMG_9068.JPG



I really need to tidy those wires up....


Anyhow, any help on the questions would be greatly appreciated.

Grazie


V
 

Attachments

  • IMG_9006.JPG
    IMG_9006.JPG
    664.2 KB · Views: 605
  • IMG_9071.JPG
    IMG_9071.JPG
    457.4 KB · Views: 430
Last edited:
Joined
Oct 26, 2008
Messages
2,242 (0.40/day)
System Name Budget AMD System
Processor Threadripper 1900X @ 4.1Ghz (100x41 @ 1.3250V)
Motherboard Gigabyte X399 Aorus Gaming 7
Cooling EKWB X399 Monoblock
Memory 4x8GB GSkill TridentZ RGB 14-14-14-32 CR1 @ 3266
Video Card(s) XFX Radeon RX Vega₆⁴ Liquid @ 1,800Mhz Core, 1025Mhz HBM2
Storage 1x ADATA SX8200 NVMe, 1x Segate 2.5" FireCuda 2TB SATA, 1x 500GB HGST SATA
Display(s) Vizio 22" 1080p 60hz TV (Samsung Panel)
Case Corsair 570X
Audio Device(s) Onboard
Power Supply Seasonic X Series 850W KM3
Software Windows 10 Pro x64
Just here to admire the hardware. lol. Not gonna lie, its a damn awesome build. :-o
 
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
Thank you, hopefully I'll get enough time off from work to actually play on it ;)
 
Joined
Jul 5, 2013
Messages
25,559 (6.52/day)
So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.
 
Joined
Jul 24, 2009
Messages
1,002 (0.19/day)
This would belong to XtremeSystems for others to admire, if it was still alive.

Quite insane machine.

Would probably try to watercool it with external big radiator. But ofc it would be bigger. Awesome nonetheless. And insanely expensive..
 

Mussels

Freshwater Moderator
Staff member
Joined
Oct 6, 2004
Messages
58,413 (8.21/day)
Location
Oystralia
System Name Rainbow Sparkles (Power efficient, <350W gaming load)
Processor Ryzen R7 5800x3D (Undervolted, 4.45GHz all core)
Motherboard Asus x570-F (BIOS Modded)
Cooling Alphacool Apex UV - Alphacool Eisblock XPX Aurora + EK Quantum ARGB 3090 w/ active backplate
Memory 2x32GB DDR4 3600 Corsair Vengeance RGB @3866 C18-22-22-22-42 TRFC704 (1.4V Hynix MJR - SoC 1.15V)
Video Card(s) Galax RTX 3090 SG 24GB: Underclocked to 1700Mhz 0.750v (375W down to 250W))
Storage 2TB WD SN850 NVME + 1TB Sasmsung 970 Pro NVME + 1TB Intel 6000P NVME USB 3.2
Display(s) Phillips 32 32M1N5800A (4k144), LG 32" (4K60) | Gigabyte G32QC (2k165) | Phillips 328m6fjrmb (2K144)
Case Fractal Design R6
Audio Device(s) Logitech G560 | Corsair Void pro RGB |Blue Yeti mic
Power Supply Fractal Ion+ 2 860W (Platinum) (This thing is God-tier. Silent and TINY)
Mouse Logitech G Pro wireless + Steelseries Prisma XL
Keyboard Razer Huntsman TE ( Sexy white keycaps)
VR HMD Oculus Rift S + Quest 2
Software Windows 11 pro x64 (Yes, it's genuinely a good OS) OpenRGB - ditch the branded bloatware!
Benchmark Scores Nyooom.
And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?
 
Joined
Jul 5, 2013
Messages
25,559 (6.52/day)
How do the titans handle being so closely stacked together heat-wise?
While I can't speak for the OP, being in an open-air case like it is likely gives plenty of room to breath and thus heat may not be a problem. That's my hypothesis.
 
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?

The titans, in the room at 21-22C ambient, run idle at about 29'C. Under load they can get as high as 68'C (on occasion even 71'C)

I removed the back plate to the titans and this allowed more room and heat dissipation. The back plate has a plastic layer under the metal plate and the whole thing is more of an insulator -so I got rid of it. Cards are very happy. So I don't really experience any thermal throttling, thermal throttling (t would occurs between 85'-91'C)

I have attached the temps on medium -light load

IMG_9061.JPG


So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.

I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.

IMG_9071.JPG
 
Last edited:
Joined
May 15, 2014
Messages
235 (0.07/day)
First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.
 
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.

I did not see a priority or sequence setting in the bios for the pcie slots.omly optometrist enable ir dissable.
Disabling prom but for all except the one I want yielded lots of beeping and no post, so I reverted back.

Unfortunately the microcode mitigation’s hurt performance, and I’d rather not have them if they are a negative on perf. I disabled the os mitigation’s by adding ‘3’ to the dword in the feature section in reg/meme I’m HKLM and that helped substantially on raid performance and video performance
 

FordGT90Concept

"I go fast!1!11!1!"
Joined
Oct 13, 2008
Messages
26,259 (4.65/day)
Location
IA, USA
System Name BY-2021
Processor AMD Ryzen 7 5800X (65w eco profile)
Motherboard MSI B550 Gaming Plus
Cooling Scythe Mugen (rev 5)
Memory 2 x Kingston HyperX DDR4-3200 32 GiB
Video Card(s) AMD Radeon RX 7900 XT
Storage Samsung 980 Pro, Seagate Exos X20 TB 7200 RPM
Display(s) Nixeus NX-EDG274K (3840x2160@144 DP) + Samsung SyncMaster 906BW (1440x900@60 HDMI-DVI)
Case Coolermaster HAF 932 w/ USB 3.0 5.25" bay + USB 3.2 (A+C) 3.5" bay
Audio Device(s) Realtek ALC1150, Micca OriGen+
Power Supply Enermax Platimax 850w
Mouse Nixeus REVEL-X
Keyboard Tesoro Excalibur
Software Windows 10 Home 64-bit
Benchmark Scores Faster than the tortoise; slower than the hare.
1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)
Might be ASUS branding for Intel Turbo Boost. Was it enabled by default? If yes, I assume they are one in the same.

2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?
You could find the most recent BIOS for the board that don't have the specter/meltdown fixes. The board is fairly old so most of the non-CPU microcode fixes should have been implemented in earlier BIOS revisions. The board is roughly a year old. The BIOS may have had the specter/meltdown fixes baked in from the first revision. I'd just leave well enough alone.

1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card? I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.
When a computer boots, it enumerates displays and connected monitors. The first card with a connected display gets flagged as primary until the driver loads. If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card. The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot. Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics). Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.
 
Last edited:

FreedomEclipse

~Technological Technocrat~
Joined
Apr 20, 2007
Messages
23,309 (3.77/day)
Location
London,UK
System Name Codename: Icarus Mk.VI
Processor Intel 8600k@Stock -- pending tuning
Motherboard Asus ROG Strixx Z370-F
Cooling CPU: BeQuiet! Dark Rock Pro 4 {1xCorsair ML120 Pro|5xML140 Pro}
Memory 32GB XPG Gammix D10 {2x16GB}
Video Card(s) ASUS Dual Radeon™ RX 6700 XT OC Edition
Storage Samsung 970 Evo 512GB SSD (Boot)|WD SN770 (Gaming)|2x 3TB Toshiba DT01ACA300|2x 2TB Crucial BX500
Display(s) LG GP850-B
Case Corsair 760T (White)
Audio Device(s) Yamaha RX-V573|Speakers: JBL Control One|Auna 300-CN|Wharfedale Diamond SW150
Power Supply Corsair AX760
Mouse Logitech G900
Keyboard Duckyshine Dead LED(s) III
Software Windows 10 Pro
Benchmark Scores (ノಠ益ಠ)ノ彡┻━┻
yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good good friend of mine. :toast::toast:
 
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
Might be ASUS branding for Intel Turbo Boost. Was it enabled by default? If yes, I assume they are one in the same.


When a computer boots, it enumerates displays and connected monitors. The first card with a connected display gets flagged as primary until the driver loads. If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card. The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot. Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics). Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.


The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.



Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like, does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.


I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...

yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good good friend of mine. :toast::toast:

Ok, but there might be a slight handling charge to go with the shipping cost....
 

FordGT90Concept

"I go fast!1!11!1!"
Joined
Oct 13, 2008
Messages
26,259 (4.65/day)
Location
IA, USA
System Name BY-2021
Processor AMD Ryzen 7 5800X (65w eco profile)
Motherboard MSI B550 Gaming Plus
Cooling Scythe Mugen (rev 5)
Memory 2 x Kingston HyperX DDR4-3200 32 GiB
Video Card(s) AMD Radeon RX 7900 XT
Storage Samsung 980 Pro, Seagate Exos X20 TB 7200 RPM
Display(s) Nixeus NX-EDG274K (3840x2160@144 DP) + Samsung SyncMaster 906BW (1440x900@60 HDMI-DVI)
Case Coolermaster HAF 932 w/ USB 3.0 5.25" bay + USB 3.2 (A+C) 3.5" bay
Audio Device(s) Realtek ALC1150, Micca OriGen+
Power Supply Enermax Platimax 850w
Mouse Nixeus REVEL-X
Keyboard Tesoro Excalibur
Software Windows 10 Home 64-bit
Benchmark Scores Faster than the tortoise; slower than the hare.
The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.
I looked through all of the documentation I could find including manual and BIOS updates and there are no references to it.

Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like, does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.
That's pretty stupid. Your work around is probably the best you're going to get then, sadly.

I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...
That's terrible. Maybe got that response because it's technically a workstation/business board. Might be a separate department or tier 2 support that handles them.
 
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
Congratulation for this godbox.

How much did you spend for the hardware ?

Cost: pound of flesh and a pint of blood

Over the years, I've paid for this hobby.

over the course of 20 years, I've built god boxes along the way. At least, in their own time they were consider unique.

19 years ago, I was the first to water-cool a dual cpu rig, chipset and video card.. I used a motorcycle oil cooler for the radiator.

I also was the first to overclock a duallie.

15 years ago I was one of the first to QUAD Sli

I was the first to overclock/watercool video cads from voodoo, specter, stb etc and the obsidian x-24 dual gpu.

I was the first to overclock the 3dlabs willdcat card and 17 years ago had a VR headset working with gyros off. The wildcat card was the first integrated dual GPU.

A few years ago I was the one that implemented 4x quad SLI on pascal as soon as nvidia tried to disable the 3 and 4 way sli on pascal - I have it fully functional and with nice proportional scaling.



here are some of the pictures from those years:

guts.JPG

yes, even the chipset on this duallie

MVC-604F.JPG

MVC-603F.JPG


IMG_1519.JPG


board.JPG

yes, that was really first qua slI

IMG_1609.JPG

dual PSU, cable management, quad sli and all 8 scsi drives are raided, circa mid 2000s

I took 6 years off as I worked on macs, windows vista made me leave the MS world

98469_6239DB49-A56E-4A30-9805-991780AD1D1F.jpeg

98470_697B3C2D-8F61-4149-B57A-CE4B2F27D08D.jpeg

98466_0AC411BB-EC91-40A3-A411-DF377CC93B45.jpeg

98468_BB3D5F14-B99B-4066-96B9-C2FF3B474A06.jpeg

98464_4C6B49D0-9584-4E38-8749-17418109C8BB.jpeg

98465_D20DB152-FC3C-4758-B94A-BE141C69A8A7.jpeg


P5
98463_C31F3EE8-DFAA-4E76-8CB0-842C7C043A2B.jpeg


P3
98282_B26AC72D-508B-47A0-AB75-7853F349C54C.jpeg


P1 - latest
IMG_9013.jpg


ending with where things are right now

I only build mutlti cpu/ multi gpu / with Raid rigs



well, that was 20 years, enjoy
 
Last edited:
Joined
Jun 28, 2016
Messages
3,595 (1.27/day)
I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.
Any noise and heat concerns?
It's basically a server in an open case.
 
Joined
Oct 21, 2005
Messages
6,874 (1.02/day)
Location
USA
System Name Computer of Theseus
Processor Intel i9-12900KS: 50x Pcore multi @ 1.18Vcore (target 1.275V -100mv offset)
Motherboard EVGA Z690 Classified
Cooling Noctua NH-D15S, 2xThermalRight TY-143, 4xNoctua NF-A12x25,3xNF-A12x15, 2xAquacomputer Splitty9Active
Memory G-Skill Trident Z5 (32GB) DDR5-6000 C36 F5-6000J3636F16GX2-TZ5RK
Video Card(s) EVGA Geforce 3060 XC Black Gaming 12GB
Storage 1x Samsung 970 Pro 512GB NVMe (OS), 2x Samsung 970 Evo Plus 2TB (data 1 and 2), ASUS BW-16D1HT
Display(s) Dell S3220DGF 32" 2560x1440 165Hz Primary, Dell P2017H 19.5" 1600x900 Secondary, Ergotron LX arms.
Case Lian Li O11 Air Mini
Audio Device(s) Audiotechnica ATR2100X-USB, El Gato Wave XLR Mic Preamp, ATH M50X Headphones, Behringer 302USB Mixer
Power Supply Super Flower Leadex Platinum SE 1000W 80+ Platinum White
Mouse Zowie EC3-C
Keyboard Vortex Multix 87 Winter TKL (Gateron G Pro Yellow)
Software Win 10 LTSC 21H2
My budget, it would be near impossible.
With your budget, it is possible. :D
I am estimating this machine cost around $40,000.
 
Last edited:
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
Any noise and heat concerns?
It's basically a server in an open case.

Runs silent, runs cool, no heat or noise issues. please see earlier pic with temps.

Even the power supply fan seldom comes on, and that is only when the unit is maxed with all 4 gpus going.
Under load, the gpus do get louder, but its not bad. I have not heard the gpu blower fans max-up yet.

I also made a lot of sections on TIM, example, every gpu was dismantled and reassembled with better TIM and thermal pads that were much better than factory.

I made a lot of effort in selecting fans, fan blade, rpms, and positioning for cool and quiet.
 
Last edited:
Joined
Oct 21, 2005
Messages
6,874 (1.02/day)
Location
USA
System Name Computer of Theseus
Processor Intel i9-12900KS: 50x Pcore multi @ 1.18Vcore (target 1.275V -100mv offset)
Motherboard EVGA Z690 Classified
Cooling Noctua NH-D15S, 2xThermalRight TY-143, 4xNoctua NF-A12x25,3xNF-A12x15, 2xAquacomputer Splitty9Active
Memory G-Skill Trident Z5 (32GB) DDR5-6000 C36 F5-6000J3636F16GX2-TZ5RK
Video Card(s) EVGA Geforce 3060 XC Black Gaming 12GB
Storage 1x Samsung 970 Pro 512GB NVMe (OS), 2x Samsung 970 Evo Plus 2TB (data 1 and 2), ASUS BW-16D1HT
Display(s) Dell S3220DGF 32" 2560x1440 165Hz Primary, Dell P2017H 19.5" 1600x900 Secondary, Ergotron LX arms.
Case Lian Li O11 Air Mini
Audio Device(s) Audiotechnica ATR2100X-USB, El Gato Wave XLR Mic Preamp, ATH M50X Headphones, Behringer 302USB Mixer
Power Supply Super Flower Leadex Platinum SE 1000W 80+ Platinum White
Mouse Zowie EC3-C
Keyboard Vortex Multix 87 Winter TKL (Gateron G Pro Yellow)
Software Win 10 LTSC 21H2
What heatsinks and fans are you running on those 28 core monsters? Looks like Noiseblockers? I wonder if the CPU would cool better on other heatsink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.
 
Joined
Apr 28, 2011
Messages
219 (0.05/day)
System Name VENTURI
Processor Dual Xeon 8280L 56 cores /112HT
Motherboard Asus c621e Sage, Version 2.1, bios 9906
Cooling Air, noctua, heatsinks, silent/low noise, phanteks T-30
Memory 1.5TB 2933mhz LRDIMM ECC REG
Video Card(s) 2x 3090 Ti FE RTX SLI
Storage Rocket 8TB and 4x Micron 9300 (51.2 TB) Raid
Display(s) Asus ProArt PAU32UCG-K
Case TT miniITX P1 (SFF)
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse TT BMW LM10
Keyboard Corsair
Software MS 2022 Data Center Server, Ubuntu
Benchmark Scores GravityMark 79,822
What heat-sinks and fans are you running on those 28 core monsters? Looks like Noiseblockers? I wonder if the CPU would cool better on other heat-sink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.


Orientation is limited. The fans are in pull mode, not push. The air is pulled up and out the top. There wasn't enough room for the fans in push mode. However, I tested it with one less video card in place in push mode and found that pull mode is 1-2 'C cooler (in PULL mode). So the orientation and pull mode ended up being the most efficient.

My primary card is on the bottom, when all 4 gpus are going full the pull mode does not acquire a waft of warm air, instead, all 4 blower gpus push warm ait out the back pulling cool air in. The result is that the chipset and surrounding area are receiving cooler air. Again, tested both ways. But I will be exploring alternative alignments. In push mode (down) the 4th card was being warmed up by the 2 cpu coolers and it was fighting the natural rise of heat.

The fans are noise blocker, 1800rpm, PWM. The heatsinks are Noctua NH-D9 DX-3647 4U

my preferred cpu cooler/fan orientation is push mode from right to left - but the market is sparce and the noctua 92mm is only SILENT configuration i can find. There are Supermicro/Dynatron coolers that will fit but sound like a hair drier compared to the noctuas

please see temp chart under medium load in a 73'F room:

IMG_9061.JPG
 
Last edited:
Joined
Jun 28, 2016
Messages
3,595 (1.27/day)
Runs silent, runs cool, no heat or noise issues. please see earlier pic with temps.
Misunderstanding. I didn't mean temperature of the components. I really meant heat. :)
At full load you'll be near that 1600W the PSU delivers. Essentially, it's like a small heater, isn't it?

As for the noise: Titans are one of the loudest cards around. I had the "pleasure" of working with them lately. Just a single card in a Dell tower and I had to use headphones to concentrate on math. But good luck. ;-)
I wonder if the CPU would cool better on other heatsink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.
It's more efficient to push air into the heatsink than pull from. I bet that's what he's doing.
Similarly, it's better to get air through CPU first and onto GPUs. GPUs are usually operating far from thermal limits and they can take it.
That's also how servers work (disks -> CPUs -> GPUs).
 
Joined
Oct 1, 2018
Messages
134 (0.07/day)
Titan V at full load can consume up to 300W when it peaks, 4 of them so it is 1200W for the GPU, each CPU is rated for 205W but we know intel always lies so count it as 250W, 500W for CPU, I didn't even count the ram sticks and ssds here and it is already reached to 1700W peak, you have 1600W, you'll have PSU issues if you want to run deep learning/machine learning applications at the long run.

but, since you are running windows and not using all the capability of the machine for running deep learning and only have presumably a single 4K screen you will never going to utilize all the gpus with gaming, even if you push all the limits of the hardware you have you'll stay under 1600W but not much, it'd better to have dual PSU or something at the realm of 2000W at least.

there are tons of tweaks on the nvidia hardware, so if we take those into the consideration, when you want to train a dataset you can halve the gpu power and save up to 700W from GPUs, since dataset training uses barely any cpu, we can add 100W, and another 200W for ram and storage, in the minimum consumption you'll again hit ~1000W it is a small heater and you'll need active airflow on the motherboard which I don't see on your photos. servergrade motherboards rely on active airflow and horizontal usage, you'll damage $30000 build no more than 2 years with active use.

if the room is 73F, how the hell SSD's running at 68F?
 
Top