• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

trying the near impossible... most powerful GOD BOX in a miniITX case

Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#1
I wasn't even sure where to post it as I have some questions that don't fall into any particular category.


I decided to ask the question: what is the most powerful build I can do and keep it tiny, and run of a normal household socket.

This was my first list, and I stuck to it:

Asus c621e sage
Dual Xeon 8180M
384GB of ram
4x Titan V including Titan V CEO ed 32GB
960pro nvme (OS)
10x SSD RAID (Apps and Backup 8x 2x )
1600W PSU (digital) noiseless
MS 2016 Data Center, Ubuntu
TT miniITX case



reverse engineering the design to fit in a small enough case proved a bit challenging:

The case had to be deconstructed and rebuilt better, reinforced and be given new hard points.
Wires and connectors had to be remade by hand to do the low profile turns.
Cable management had to be ZEN like.
It would have to run cool all by itself and be rock stable.


so after a week of getting parts in and building the toy (after the daytime job as to be able to afford this little hobby)

well, here is the mostly complete and fully functional final result:

IMG_9013.jpg


It looks harmless enough, but it was a lot of work,

IMG_9019.JPG


The drives had to be RAIDED and stacked in their own cooling tower

IMG_9012.JPG


The 4x Titan V required decent power that had to be shared with the dual 8180M xeons

those required a little order and management

IMG_9023.JPG


IMG_9015.JPG


while I was building it, with all its guts (wires) hanging out, I ran a quick cinebench to see how it would do

score: 8857

IMG_8889.JPG


temps seem reasonable under light load, temps rise about 10-15 'F for cpus under heavy load (up to about 40-41'C)
video cards can get as hot under load as 68'C


IMG_9061.JPG





so it runs fast and stable, games look amazing and are smooth at 4k+, not to mention the little bit of work I use it for (deep learning, clinical decision support, isotopes, medical imaging, Lara Croft, etc)


SO I have some questions that answers seem to exist NOWHERE on the net.

Questions from bios

1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)

2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?

Question on Video cards.

1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card? I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.


IMG_9068.JPG



I really need to tidy those wires up....


Anyhow, any help on the questions would be greatly appreciated.

Grazie


V
 

Attachments

Last edited:
Joined
Oct 26, 2008
Messages
2,084 (0.55/day)
Likes
395
System Name Budget AMD System
Processor Threadripper 1900X @ 4.025Ghz (100x40.25 @ 1.325V)
Motherboard Gigabyte X399 Aorus Gaming 7
Cooling EKWB X399 Monoblock
Memory 4x8GB GSkill TridentZ RGB 14-14-14-34 CR1 @ 2933
Video Card(s) XFX Radeon RX Vega₆⁴ Liquid @ 1,800Mhz Core, 1100 HBM2
Storage 1x ADATA SX8200 NVMe, 1x Segate 2.5" FireCuda 2TB SATA, 1x 500GB HGST SATA
Display(s) Vizio 22" 1080p 60hz TV (Samsung Panel)
Case Corsair 570X
Audio Device(s) Onboard
Power Supply Seasonic X Series 850W KM3
Software Windows 10 Pro x64
#2
Just here to admire the hardware. lol. Not gonna lie, its a damn awesome build. :-o
 
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#3
Thank you, hopefully I'll get enough time off from work to actually play on it ;)
 
Joined
Jul 24, 2009
Messages
511 (0.14/day)
Likes
94
#5
This would belong to XtremeSystems for others to admire, if it was still alive.

Quite insane machine.

Would probably try to watercool it with external big radiator. But ofc it would be bigger. Awesome nonetheless. And insanely expensive..
 

Mussels

Moderprator
Staff member
Joined
Oct 6, 2004
Messages
46,932 (8.89/day)
Likes
14,660
Location
Australalalalalaia.
System Name OCD (Overclocking Compulsive Disorder)
Processor Ryzen R7 2700X (stock/XFR OC)
Motherboard Aorus AX370-Gaming 5 (planned x570 upgrade when they come out)
Cooling Corsair H115i Pro W/ Corsair ML RGB fans
Memory 16GB DDR4 3200 Corsair Vengeance RGB Pro
Video Card(s) MSI GTX 1080 Gaming X (BIOS mod to Gaming Z) w/ NZXT Kraken x52 AIO
Storage 1TB Sasmsung 970 Pro NVME + 1TB Intel 6000 Pro NVME
Display(s) Phillips 328m6fjrmb (32" 1440p 144hz curved) + Sony KD-55X8500F (55" 4K HDR)
Case Fractal Design R6 Gunmetal Grey (Type C TG)
Audio Device(s) Pioneer VSX-519V + Yamaha YHT-270 + Corsair Void pro RGB, Blue Yeti mic
Power Supply Corsair HX 750i (Platinum, fan off til 300W)
Mouse Logitech G703 + PowerPlay mousepad
Keyboard Corsair K65 Rapidfire
Software Windows 10 pro x64 (all systems)
Benchmark Scores Laptops: i7-4510U + 840M 2GB (touchscreen) 275GB SSD + 16GB i7-2630QM + GT 540M + 8GB
#6
And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?
 
Joined
Jul 5, 2013
Messages
5,087 (2.44/day)
Likes
3,246
Location
USA
#7
How do the titans handle being so closely stacked together heat-wise?
While I can't speak for the OP, being in an open-air case like it is likely gives plenty of room to breath and thus heat may not be a problem. That's my hypothesis.
 
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#8
And i thought i had a powerful ITX rig :/

How do the titans handle being so closely stacked together heat-wise?
The titans, in the room at 21-22C ambient, run idle at about 29'C. Under load they can get as high as 68'C (on occasion even 71'C)

I removed the back plate to the titans and this allowed more room and heat dissipation. The back plate has a plastic layer under the metal plate and the whole thing is more of an insulator -so I got rid of it. Cards are very happy. So I don't really experience any thermal throttling, thermal throttling (t would occurs between 85'-91'C)

I have attached the temps on medium -light load

IMG_9061.JPG


So love that case. Why is it on the floor though? I'd move the speakers to the sides of the display and put the case on the left side of the desktop to be displayed with pride! Not to mention keeping away from dust that collects on the floor.
I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.

IMG_9071.JPG
 
Last edited:
Joined
May 15, 2014
Messages
71 (0.04/day)
Likes
30
#9
First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.
 
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#10
First, congrats on the build. Does the c621e bios allow PCIEx16 n slot boot priority? I assume not. Examine the bios & see if there is a hidden option. I'm not suggesting to flash an edit, but send a request through to Asus tech support. Re: microcode edit - yes you can. I wouldn't. It is possible (although I haven't looked at the latest Xeon bioses) that they're encrypted. You'll need the USB flash back feature, etc. You also need to consider the version of microcode in the latest BIOS as well as that bootstrapped by the OSes you're running. You may not achieve much unless you disable OS mitigations (which I don't recommend). Titan V CEO do very well on some data sets, much better than my lowly Quadro.
I did not see a priority or sequence setting in the bios for the pcie slots.omly optometrist enable ir dissable.
Disabling prom but for all except the one I want yielded lots of beeping and no post, so I reverted back.

Unfortunately the microcode mitigation’s hurt performance, and I’d rather not have them if they are a negative on perf. I disabled the os mitigation’s by adding ‘3’ to the dword in the feature section in reg/meme I’m HKLM and that helped substantially on raid performance and video performance
 

FordGT90Concept

"I go fast!1!11!1!"
Joined
Oct 13, 2008
Messages
24,241 (6.36/day)
Likes
13,469
Location
IA, USA
System Name BY-2015
Processor Intel Core i7-6700K (4 x 4.00 GHz) w/ HT and Turbo on
Motherboard MSI Z170A GAMING M7
Cooling Scythe Kotetsu
Memory 2 x Kingston HyperX DDR4-2133 8 GiB
Video Card(s) Sapphire NITRO+ RX 590 Special Edition 8 GiB DVI + HDMI
Storage Crucial MX300 275 GB, Seagate Exos 12 TB 7200 RPM
Display(s) Samsung SyncMaster T240 24" LCD (1920x1200 HDMI) + Samsung SyncMaster 906BW 19" LCD (1440x900 DVI)
Case Coolermaster HAF 932 w/ USB 3.0 5.25" bay
Audio Device(s) Realtek Onboard, Micca OriGen+
Power Supply Enermax Platimax 850w
Mouse SteelSeries Sensei RAW
Keyboard Tesoro Excalibur
Software Windows 10 Pro 64-bit
Benchmark Scores Faster than the tortoise; slower than the hare.
#11
1. what is "Asus Engine Booster"? (no, its not the asus OC tune setting)
Might be ASUS branding for Intel Turbo Boost. Was it enabled by default? If yes, I assume they are one in the same.

2. I like using the latest bios, but the prior bios without the microcode mitigation is a faster machine, is there a way to edit out a microcode mitigation in the bios and keep the latest one?
You could find the most recent BIOS for the board that don't have the specter/meltdown fixes. The board is fairly old so most of the non-CPU microcode fixes should have been implemented in earlier BIOS revisions. The board is roughly a year old. The BIOS may have had the specter/meltdown fixes baked in from the first revision. I'd just leave well enough alone.

1. I like using the lowest card on the stack as primary. This is so because it has the most amount of cool air and I don't hit any thermal limits (I do run mgpu, and I am currently trying to get auto diff sli to work). The question is: how do I get the bios and post screen to come up if the board want to make any card but this one a primary card? I do have a work around for now: I havea DP cable in there and when windows loads it runs fine, I also have an HDMI cable in the 4th card for bios and post so with one monitor I can have both bios, post , and run state. I would like to avoid having to use the HDMI cable -seems not elegant.
When a computer boots, it enumerates displays and connected monitors. The first card with a connected display gets flagged as primary until the driver loads. If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card. The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot. Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics). Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.
 
Last edited:

FreedomEclipse

~Technological Technocrat~
Joined
Apr 20, 2007
Messages
18,906 (4.34/day)
Likes
6,986
Location
London,UK
System Name Codename: Icarus Mk.IV
Processor Intel 8600k@4.8Ghz
Motherboard Asus ROG Strixx Z370-F
Cooling Corsair H105 {2x Corsair ML 120 Pro}
Memory Corsair Vengeance White LED DDR4 3200Mhz
Video Card(s) Gigabyte 1080Ti Gaming OC
Storage Samsung 970Evo 512GB SSD (Boot)|WD Blue 1TB SSD|2x 3TB Toshiba DT01ACA300
Display(s) Asus PB278Q 27"
Case Corsair 760T (White) {1x Corsair ML120 Pro |3x ML140 Pro}
Audio Device(s) Creative SB Z {AVR:Yamaha RX-V573|Speakers: JBL Control One|Auna 300-CN|Wharfedale Diamond SW150}
Power Supply Corsair AX760
Mouse Logitech G900/G502
Keyboard Duckyshine Dead LED(s) III
Software Windows 10 Pro
Benchmark Scores (ノಠ益ಠ)ノ彡┻━┻
#12
yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good good friend of mine. :toast::toast:
 
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#13
Might be ASUS branding for Intel Turbo Boost. Was it enabled by default? If yes, I assume they are one in the same.


When a computer boots, it enumerates displays and connected monitors. The first card with a connected display gets flagged as primary until the driver loads. If all of your displays are plugged into the bottom card, theoretically it should always be considered the primary card. The other cards should be in an idle power state unless SLI give them something to do.

Alternatively, the BIOS may have an option for selecting the first GPU to boot. Usually that choice only enumerates IGP (integrated) and PEG (PCI Express Graphics). Maybe your board lists all of the PCIE slots (PEG1, PEG2, PEG3, etc.) so you can tell it which specific card to load.

The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.



Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like, does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.


I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...

yeah.

If you ever just want to get rid of this thing for free. Im willing to pay for shipping. Its the least i could do for a good good friend of mine. :toast::toast:
Ok, but there might be a slight handling charge to go with the shipping cost....
 

FordGT90Concept

"I go fast!1!11!1!"
Joined
Oct 13, 2008
Messages
24,241 (6.36/day)
Likes
13,469
Location
IA, USA
System Name BY-2015
Processor Intel Core i7-6700K (4 x 4.00 GHz) w/ HT and Turbo on
Motherboard MSI Z170A GAMING M7
Cooling Scythe Kotetsu
Memory 2 x Kingston HyperX DDR4-2133 8 GiB
Video Card(s) Sapphire NITRO+ RX 590 Special Edition 8 GiB DVI + HDMI
Storage Crucial MX300 275 GB, Seagate Exos 12 TB 7200 RPM
Display(s) Samsung SyncMaster T240 24" LCD (1920x1200 HDMI) + Samsung SyncMaster 906BW 19" LCD (1440x900 DVI)
Case Coolermaster HAF 932 w/ USB 3.0 5.25" bay
Audio Device(s) Realtek Onboard, Micca OriGen+
Power Supply Enermax Platimax 850w
Mouse SteelSeries Sensei RAW
Keyboard Tesoro Excalibur
Software Windows 10 Pro 64-bit
Benchmark Scores Faster than the tortoise; slower than the hare.
#14
The intel turbo boost is in another section of the bios. This asus engine booster feature is disabled by default. If it’s enabled it gives a choice of level 1, level 2, and level 3.

It’s not the oc level either, that is in another section as well.
I looked through all of the documentation I could find including manual and BIOS updates and there are no references to it.

Asus made the video boot of the bios dependent on the amount of cards that are plugged into the board, alas, plugging a card in the card like, does not yield video during post/ boot

The bios, unless it’s wording I don’t know, does not have a pcie/card selector. Hence my reaching out.
That's pretty stupid. Your work around is probably the best you're going to get then, sadly.

I called asus tech support, unfortunately I spent the first hour trying to prove to tech support that asus actually made the board and that it was their product... it went downhill from there...
That's terrible. Maybe got that response because it's technically a workstation/business board. Might be a separate department or tier 2 support that handles them.
 
Joined
Oct 24, 2004
Messages
980 (0.19/day)
Likes
596
System Name Seriously ?
Processor Core i7 4790K @ 4.4Ghz
Motherboard MSI Z97 Gaming 5
Cooling Alpenföhne Broken 120 + 2 huge coolermaster chassis fans
Memory 2x8GB DDR3 2133Mhz Crucial Ballistix
Video Card(s) Geforce GT 610 (no longer playing games)
Storage 2x60gb Kingston SSDnow 320 / 1x120gb Crucial SSD C300 / 1 WD caviar black 1tb
Display(s) 40" Samsung UE40ES5500
Case Cooler Master HAF932
Audio Device(s) onboard realtek audio
Power Supply Corsair AX1200
Mouse Microsoft intellimouse optical
Keyboard Logitech K270
Software Windows 10 Pro
Benchmark Scores https://valid.x86.fr/vmb641
#15
Congratulation for this godbox.

How much did you spend for the hardware ?
 
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#17
Congratulation for this godbox.

How much did you spend for the hardware ?
Cost: pound of flesh and a pint of blood

Over the years, I've paid for this hobby.

over the course of 20 years, I've built god boxes along the way. At least, in their own time they were consider unique.

19 years ago, I was the first to water-cool a dual cpu rig, chipset and video card.. I used a motorcycle oil cooler for the radiator.

I also was the first to overclock a duallie.

15 years ago I was one of the first to QUAD Sli

I was the first to overclock/watercool video cads from voodoo, specter, stb etc and the obsidian x-24 dual gpu.

I was the first to overclock the 3dlabs willdcat card and 17 years ago had a VR headset working with gyros off. The wildcat card was the first integrated dual GPU.

A few years ago I was the one that implemented 4x quad SLI on pascal as soon as nvidia tried to disable the 3 and 4 way sli on pascal - I have it fully functional and with nice proportional scaling.



here are some of the pictures from those years:

guts.JPG

yes, even the chipset on this duallie

MVC-604F.JPG

MVC-603F.JPG


IMG_1519.JPG


board.JPG

yes, that was really first qua slI

IMG_1609.JPG

dual PSU, cable management, quad sli and all 8 scsi drives are raided, circa mid 2000s

I took 6 years off as I worked on macs, windows vista made me leave the MS world

98469_6239DB49-A56E-4A30-9805-991780AD1D1F.jpeg

98470_697B3C2D-8F61-4149-B57A-CE4B2F27D08D.jpeg

98466_0AC411BB-EC91-40A3-A411-DF377CC93B45.jpeg

98468_BB3D5F14-B99B-4066-96B9-C2FF3B474A06.jpeg

98464_4C6B49D0-9584-4E38-8749-17418109C8BB.jpeg

98465_D20DB152-FC3C-4758-B94A-BE141C69A8A7.jpeg


P5
98463_C31F3EE8-DFAA-4E76-8CB0-842C7C043A2B.jpeg


P3
98282_B26AC72D-508B-47A0-AB75-7853F349C54C.jpeg


P1 - latest
IMG_9013.jpg


ending with where things are right now

I only build mutlti cpu/ multi gpu / with Raid rigs



well, that was 20 years, enjoy
 
Last edited:
Joined
Jun 28, 2016
Messages
2,013 (2.02/day)
Likes
692
#18
I was going to move it to the top of the desk, just getting organized now. For its size, the server can fit easily on either side of the monitor.
Any noise and heat concerns?
It's basically a server in an open case.
 
Joined
Oct 21, 2005
Messages
4,961 (1.01/day)
Likes
2,549
Location
USA
System Name Small ATX Desktop
Processor Intel i5 8600K @ 4.6 GHz Core & 4.3 GHz Cache @ 1.168V
Motherboard Asrock Z370 Taichi
Cooling Phanteks PH-TC14PE, MonoPlus, Fans: 2xThermalright TY143 / 2xCorsair SP-120L / 2xYate Loon D14BH-12
Memory G-Skill TridentZ 2X8GB DDR4 3200 CL14 F4-3200C14D-16GTZ @ 3200 14-14-14-34 2T @ 1.35V
Video Card(s) Zotac 1060 6GB Mini ZT-P10600A-10L with Arctic MonoPlus and Yate Loon D14BH-12 Fan
Storage Samsung 970 Pro 512GB NVME, Seagate 600 240GB SSD, 2xWD Caviar Blue WD10EZEX 1TB HDD
Display(s) 2x Asus PB258Q 2560x1440 25" IPS
Case Lian Li PC A05NB (Inverted Mobo)
Audio Device(s) Audiotechnica ATH M50X, Antlion Mod Mic 4, SYBA SD-CM-UAUD, Acoustic Research 2Ch Speakers
Power Supply Seasonic SS-660XP2 660 Watt Platinum
Mouse Zowie EC2A Mouse, Corsair MM600 hard pad, on top of a Inateck 900x300 XL pad, Tiger Gaming Skates.
Keyboard Filco Majestouch II Ninja TKL, Goldtouch GTC 0033 Ten Key, PS3 Controller
Software Win7 Pro 64 (Installed on Coffee Lake using AsRock's handy PS/2 Simulator in Bios)
#19
My budget, it would be near impossible.
With your budget, it is possible. :D
I am estimating this machine cost around $40,000.
 
Last edited:
Joined
Oct 24, 2004
Messages
980 (0.19/day)
Likes
596
System Name Seriously ?
Processor Core i7 4790K @ 4.4Ghz
Motherboard MSI Z97 Gaming 5
Cooling Alpenföhne Broken 120 + 2 huge coolermaster chassis fans
Memory 2x8GB DDR3 2133Mhz Crucial Ballistix
Video Card(s) Geforce GT 610 (no longer playing games)
Storage 2x60gb Kingston SSDnow 320 / 1x120gb Crucial SSD C300 / 1 WD caviar black 1tb
Display(s) 40" Samsung UE40ES5500
Case Cooler Master HAF932
Audio Device(s) onboard realtek audio
Power Supply Corsair AX1200
Mouse Microsoft intellimouse optical
Keyboard Logitech K270
Software Windows 10 Pro
Benchmark Scores https://valid.x86.fr/vmb641
#20
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#21
Any noise and heat concerns?
It's basically a server in an open case.
Runs silent, runs cool, no heat or noise issues. please see earlier pic with temps.

Even the power supply fan seldom comes on, and that is only when the unit is maxed with all 4 gpus going.
Under load, the gpus do get louder, but its not bad. I have not heard the gpu blower fans max-up yet.

I also made a lot of sections on TIM, example, every gpu was dismantled and reassembled with better TIM and thermal pads that were much better than factory.

I made a lot of effort in selecting fans, fan blade, rpms, and positioning for cool and quiet.
 
Last edited:
Joined
Oct 21, 2005
Messages
4,961 (1.01/day)
Likes
2,549
Location
USA
System Name Small ATX Desktop
Processor Intel i5 8600K @ 4.6 GHz Core & 4.3 GHz Cache @ 1.168V
Motherboard Asrock Z370 Taichi
Cooling Phanteks PH-TC14PE, MonoPlus, Fans: 2xThermalright TY143 / 2xCorsair SP-120L / 2xYate Loon D14BH-12
Memory G-Skill TridentZ 2X8GB DDR4 3200 CL14 F4-3200C14D-16GTZ @ 3200 14-14-14-34 2T @ 1.35V
Video Card(s) Zotac 1060 6GB Mini ZT-P10600A-10L with Arctic MonoPlus and Yate Loon D14BH-12 Fan
Storage Samsung 970 Pro 512GB NVME, Seagate 600 240GB SSD, 2xWD Caviar Blue WD10EZEX 1TB HDD
Display(s) 2x Asus PB258Q 2560x1440 25" IPS
Case Lian Li PC A05NB (Inverted Mobo)
Audio Device(s) Audiotechnica ATH M50X, Antlion Mod Mic 4, SYBA SD-CM-UAUD, Acoustic Research 2Ch Speakers
Power Supply Seasonic SS-660XP2 660 Watt Platinum
Mouse Zowie EC2A Mouse, Corsair MM600 hard pad, on top of a Inateck 900x300 XL pad, Tiger Gaming Skates.
Keyboard Filco Majestouch II Ninja TKL, Goldtouch GTC 0033 Ten Key, PS3 Controller
Software Win7 Pro 64 (Installed on Coffee Lake using AsRock's handy PS/2 Simulator in Bios)
#22
What heatsinks and fans are you running on those 28 core monsters? Looks like Noiseblockers? I wonder if the CPU would cool better on other heatsink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.
 
Joined
Apr 28, 2011
Messages
33 (0.01/day)
Likes
78
System Name Venturi
Processor Dual Xeon 8180M 56 cores /112HT
Motherboard Asus c621e Sage
Cooling Air, noctua, heatsinks, noiseblocker fans
Memory ecc red 384GB 2666mhz
Video Card(s) 4x Titan V including Titan V CEO ed 32GB
Storage 960pro 2TB and 1x 850pro SSD RAID
Display(s) Asus ProArt 329 4k
Case TT miniITX P1
Audio Device(s) harmon Kardon speakers
Power Supply 1600 AXi digital (silent)
Mouse Z mouse
Keyboard Corsai K65
Software MS 2019 Data Center Server, Ubuntu
Benchmark Scores Cinebench 18708
#23
What heat-sinks and fans are you running on those 28 core monsters? Looks like Noiseblockers? I wonder if the CPU would cool better on other heat-sink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.

Orientation is limited. The fans are in pull mode, not push. The air is pulled up and out the top. There wasn't enough room for the fans in push mode. However, I tested it with one less video card in place in push mode and found that pull mode is 1-2 'C cooler (in PULL mode). So the orientation and pull mode ended up being the most efficient.

My primary card is on the bottom, when all 4 gpus are going full the pull mode does not acquire a waft of warm air, instead, all 4 blower gpus push warm ait out the back pulling cool air in. The result is that the chipset and surrounding area are receiving cooler air. Again, tested both ways. But I will be exploring alternative alignments. In push mode (down) the 4th card was being warmed up by the 2 cpu coolers and it was fighting the natural rise of heat.

The fans are noise blocker, 1800rpm, PWM. The heatsinks are Noctua NH-D9 DX-3647 4U

my preferred cpu cooler/fan orientation is push mode from right to left - but the market is sparce and the noctua 92mm is only SILENT configuration i can find. There are Supermicro/Dynatron coolers that will fit but sound like a hair drier compared to the noctuas

please see temp chart under medium load in a 73'F room:

IMG_9061.JPG
 
Last edited:
Joined
Jun 28, 2016
Messages
2,013 (2.02/day)
Likes
692
#24
Runs silent, runs cool, no heat or noise issues. please see earlier pic with temps.
Misunderstanding. I didn't mean temperature of the components. I really meant heat. :)
At full load you'll be near that 1600W the PSU delivers. Essentially, it's like a small heater, isn't it?

As for the noise: Titans are one of the loudest cards around. I had the "pleasure" of working with them lately. Just a single card in a Dell tower and I had to use headphones to concentrate on math. But good luck. ;-)
I wonder if the CPU would cool better on other heatsink orientations. Looks like it pulls hot air off the back of the top titan and might have better performance blowing sideways from front to back (but then you run into ram clearance) or from top to bottom, (reversing the fans) which has side benefit of cooling the backside of the Titan. Benefit of the open case is you can have the fans pointed any direction you want, there's minimal back pressure.
It's more efficient to push air into the heatsink than pull from. I bet that's what he's doing.
Similarly, it's better to get air through CPU first and onto GPUs. GPUs are usually operating far from thermal limits and they can take it.
That's also how servers work (disks -> CPUs -> GPUs).
 
Joined
Oct 1, 2018
Messages
134 (0.77/day)
Likes
41
#25
Titan V at full load can consume up to 300W when it peaks, 4 of them so it is 1200W for the GPU, each CPU is rated for 205W but we know intel always lies so count it as 250W, 500W for CPU, I didn't even count the ram sticks and ssds here and it is already reached to 1700W peak, you have 1600W, you'll have PSU issues if you want to run deep learning/machine learning applications at the long run.

but, since you are running windows and not using all the capability of the machine for running deep learning and only have presumably a single 4K screen you will never going to utilize all the gpus with gaming, even if you push all the limits of the hardware you have you'll stay under 1600W but not much, it'd better to have dual PSU or something at the realm of 2000W at least.

there are tons of tweaks on the nvidia hardware, so if we take those into the consideration, when you want to train a dataset you can halve the gpu power and save up to 700W from GPUs, since dataset training uses barely any cpu, we can add 100W, and another 200W for ram and storage, in the minimum consumption you'll again hit ~1000W it is a small heater and you'll need active airflow on the motherboard which I don't see on your photos. servergrade motherboards rely on active airflow and horizontal usage, you'll damage $30000 build no more than 2 years with active use.

if the room is 73F, how the hell SSD's running at 68F?
 
Top