• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Penguin Computing Packs 7616 Intel Xeon Platinum Cores in one Server Rack

AleksandarK

News Editor
Staff member
Joined
Aug 19, 2017
Messages
2,233 (0.91/day)
In data centers of hyperscalers like Amazon, Google, Facebook, and ones alike, there is a massive need for more computing power. Being that data centers are space-limited facilities, it is beneficial if there is a system that can pack as much computing power as possible, in a smaller form factor. Penguin Computing has thought exactly about this problem and has decided to launch a TundraAP platform, designed specifically as a high-density CPU system. Using an Intel Xeon Platinum 9200 processor, the company utilizes Intel's processor with the highest core count - 56 cores spread on two dies, brought together by a single BGA package.

The Penguin Computing TundraAP system relies on Intel's S9200WK server system. In a 1U server, Penguin Computing puts two of those in one system, with a twist. The company implements a power disaggregation system, which is designed to handle and remove the heat coming from those 400 W TPD monster processors. This means that the PSU is moved from the server itself and moved on a special rack, so the heat from the CPUs wouldn't affect PSUs. The company uses Open Compute Project standards and says it improves efficiency by 15%. To cool those chips, Penguin Computing uses a direct-to-chip liquid cooling system. And if you are wondering how much cores the company can fit in a rack, look no further as it is possible to have as much as 7616 Xeon Platinum cores in just one rack. This is a huge achievement as the density is quite big. The custom cooling and power delivery system that the company built enabled this, by only allowing compute elements to be present in the system.


View at TechPowerUp Main Site
 
Joined
Feb 15, 2019
Messages
1,531 (0.81/day)
System Name Personal Gaming Rig
Processor Ryzen 7800X3D
Motherboard MSI X670E Carbon
Cooling MO-RA 3 420
Memory 32GB 6000MHz
Video Card(s) RTX 4090 ICHILL FROSTBITE ULTRA
Storage 4x 2TB Nvme
Display(s) Samsung G8 OLED
Case Silverstone FT04
And Highest Heat density ?
 
Joined
May 28, 2020
Messages
752 (0.53/day)
System Name Main PC
Processor AMD Ryzen 9 5950X
Motherboard ASUS X570 Crosshair VIII Hero (Wi-Fi)
Cooling EKWB X570 VIII Hero Monoblock, 2x XD5, Heatkiller IV SB block for chipset,Alphacool 3090 Strix block
Memory 4x16GB 3200-14-14-14-34 G.Skill Trident RGB (OC: 3600-14-14-14-28)
Video Card(s) ASUS RTX 3090 Strix OC
Storage 500GB+500GB SSD RAID0, Fusion IoDrive2 1.2TB, Huawei HSSD 2TB, 11TB on server used for steam
Display(s) Dell LG CX48 (custom res: 3840x1620@120Hz) + Acer XB271HU 2560x1440@144Hz
Case Corsair 1000D
Audio Device(s) Sennheiser HD599, Blue Yeti
Power Supply Corsair RM1000i
Mouse Logitech G502 Lightspeed
Keyboard Corsair Strafe RGB MK2
Software Windows 10 Pro 20H2
Cray was able to fit 8 EPYC CPUs into 1U in 2018. I'll leave the total number of cores as an exercise for the reader ;)
If we assume Epyc Rome (2019), then the math would be 64 (cores per CPU) x 8 (CPUs) x 42 (U's in a rack) = 21504 cores. Goddamn. If we assume the previous gen, we still land at 10752 cores, which still beats Intel by a country mile. Not to mention AMD does it at 280W per CPU (7H12) vs. Intel's 400W (9282).
 
Joined
Oct 27, 2009
Messages
1,133 (0.21/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
Cray was able to fit 8 EPYC CPUs into 1U in 2018. I'll leave the total number of cores as an exercise for the reader ;)

Yeah I was like, oh, I thought only cray said they were going to make a watercooled xeon 9200 box for someone that really wanted Intel (Aurora supercomputer)

Then I see this is a reference design from Intel because literally no one else wasted their money designing anything for 9200...
 
Joined
Oct 27, 2009
Messages
1,133 (0.21/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
...in one rack. OK, but how much space is taken by the external power supplies?
None? The picture shows a sled that clearly slides into an enclosure which according to the article links...
Looks like this....
1596530184622.png


And how many racks worth of A/C cooling equipment to dump all that heat?

Now that is a good question, each generation of watercooled supercomputer gets better... with SGI it used to be 2:1 or 3:1 rack to heat exchange.
Cray currently appears to do 4 cabinets to 1 which isn't exactly a rack... Cray also makes a 9200 version of this, it just isn't their flagship ;)
1596530476528.png


It took awhile to find this... HPE kinda sucks when it comes to product documentation lookup.
 
Joined
Jan 27, 2015
Messages
451 (0.13/day)
System Name Marmo / Kanon
Processor Intel Core i7 9700K / AMD Ryzen 7 5800X
Motherboard Gigabyte Z390 Aorus Pro WiFi / X570S Aorus Pro AX
Cooling Noctua NH-U12S x 2
Memory Corsair Vengeance 32GB 2666-C16 / 32GB 3200-C16
Video Card(s) KFA2 RTX3070 Ti / Asus TUF RX 6800XT OC
Storage Samsung 970 EVO+ 1TB, 860 EVO 1TB / Samsung 970 Pro 1TB, 970 EVO+ 1TB
Display(s) Dell AW2521HFA / U2715H
Case Fractal Design Focus G / Pop Air RGB
Audio Device(s) Onboard / Creative SB ZxR
Power Supply SeaSonic Focus GX 650W / PX 750W
Mouse Logitech MX310 / G1
Keyboard Logitech G413 / G513
Software Win 11 Ent
Presumably the server can't operate without this dedicated PSU. So if it needs its own rack space, how could the server be called 1U?
 
Joined
May 28, 2020
Messages
752 (0.53/day)
System Name Main PC
Processor AMD Ryzen 9 5950X
Motherboard ASUS X570 Crosshair VIII Hero (Wi-Fi)
Cooling EKWB X570 VIII Hero Monoblock, 2x XD5, Heatkiller IV SB block for chipset,Alphacool 3090 Strix block
Memory 4x16GB 3200-14-14-14-34 G.Skill Trident RGB (OC: 3600-14-14-14-28)
Video Card(s) ASUS RTX 3090 Strix OC
Storage 500GB+500GB SSD RAID0, Fusion IoDrive2 1.2TB, Huawei HSSD 2TB, 11TB on server used for steam
Display(s) Dell LG CX48 (custom res: 3840x1620@120Hz) + Acer XB271HU 2560x1440@144Hz
Case Corsair 1000D
Audio Device(s) Sennheiser HD599, Blue Yeti
Power Supply Corsair RM1000i
Mouse Logitech G502 Lightspeed
Keyboard Corsair Strafe RGB MK2
Software Windows 10 Pro 20H2
Presumably the server can't operate without this dedicated PSU. So if it needs its own rack space, how could the server be called 1U?
There's the chassis, the FC2000, that houses 4 1U modules in a 2x2 configuration. So 2U is 4 modules. Each module being 1U in height and half width. The PSUs take up no additional rack space, they're part of the chassis.
 
Joined
Jul 10, 2017
Messages
2,671 (1.08/day)
- How does 7616 reactor core explodes?
- It doesn't!

*famous last words*
 
Joined
Sep 15, 2007
Messages
3,944 (0.65/day)
Location
Police/Nanny State of America
Processor OCed 5800X3D
Motherboard Asucks C6H
Cooling Air
Memory 32GB
Video Card(s) OCed 6800XT
Storage NVMees
Display(s) 32" Dull curved 1440
Case Freebie glass idk
Audio Device(s) Sennheiser
Power Supply Don't even remember
Joined
Oct 27, 2009
Messages
1,133 (0.21/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
Joined
Sep 15, 2007
Messages
3,944 (0.65/day)
Location
Police/Nanny State of America
Processor OCed 5800X3D
Motherboard Asucks C6H
Cooling Air
Memory 32GB
Video Card(s) OCed 6800XT
Storage NVMees
Display(s) 32" Dull curved 1440
Case Freebie glass idk
Audio Device(s) Sennheiser
Power Supply Don't even remember
Joined
Oct 27, 2009
Messages
1,133 (0.21/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
Joined
Sep 15, 2007
Messages
3,944 (0.65/day)
Location
Police/Nanny State of America
Processor OCed 5800X3D
Motherboard Asucks C6H
Cooling Air
Memory 32GB
Video Card(s) OCed 6800XT
Storage NVMees
Display(s) 32" Dull curved 1440
Case Freebie glass idk
Audio Device(s) Sennheiser
Power Supply Don't even remember
Oh, you didn't do the math you just pulled numbers out of your ass got it...
it's 54,400 watts++

Yikes...

No, it's 15,000 roentgen.
 
Joined
Oct 27, 2009
Messages
1,133 (0.21/day)
Location
Republic of Texas
System Name [H]arbringer
Processor 4x 61XX ES @3.5Ghz (48cores)
Motherboard SM GL
Cooling 3x xspc rx360, rx240, 4x DT G34 snipers, D5 pump.
Memory 16x gskill DDR3 1600 cas6 2gb
Video Card(s) blah bigadv folder no gfx needed
Storage 32GB Sammy SSD
Display(s) headless
Case Xigmatek Elysium (whats left of it)
Audio Device(s) yawn
Power Supply Antec 1200w HCP
Software Ubuntu 10.10
Benchmark Scores http://valid.canardpc.com/show_oc.php?id=1780855 http://www.hwbot.org/submission/2158678 http://ww
Yikes...

No, it's 15,000 roentgen.

Ah, the reference would be it's not 3w its 15000w but ok.

These fuckers def require a specialized space... it's pretty hard to power multiple racks >30kw ea. Unless these are detuned/turbo disabled a rack can easily exceed 60kw as 54.4kw is just the cpus not ram, chipset, network fabric...
 
Joined
Jul 10, 2017
Messages
2,671 (1.08/day)
It's not 95W. It's 15,000.
They gave them the propaganda numbers!

Ah, the reference would be it's not 3w its 15000w but ok.

These fuckers def require a specialized space... it's pretty hard to power multiple racks >30kw ea. Unless these are detuned/turbo disabled a rack can easily exceed 60kw as 54.4kw is just the cpus not ram, chipset, network fabric...
Our high-range Watt-meter just arrived. We could cover one of our trucks with lead shielding, mount the Watt-meter on the front.

Intel: We did everything right!
Apple: Do you taste Metal?
 
Last edited:
Top