• Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

HELP! Quad RTX 3090, x2 EVGA 1600 G2: Burnt out +12v on MoBo power connector

FrankJames

New Member
Joined
Jul 20, 2021
Messages
8 (0.01/day)
Hi Guys,

New to this forum.

I just built a custom rig for Deep Learning and after 2-3 months of constant use, I noticed the 2 +12v lines from PSU to MoBo have burned up. Just took it all apart, MoBo is fine, cable is not, +12v pins burnt in place on PSU 1.

SPEC:
- ROG zenith extreme alpha ii
- Threadripper 3990x
- Trident Z royal 32 gig x8
- 4x ASUS RTX 3090 OC (using PCIe X16 riser cables)
- EVGA 1600 G2 x2
- x3 EK waterblock/pumps
- GPUs & CPU are water cooled
- Im using 3 seperate cases, one big center one to house all the main stuff, 2 smaller side towers (all mounted on a wall) to house cooling towers & PSUs

POWER SET UP:
- MoBo, CPU, GPU 1&2 + all fans & pumps on PSU 1.
- GPU 3&4 on PSU 2
- Both PSUs are on one dedicated back up battery to each PSU plugged into the same wall circuit. The battery back-ups are at ~60% max load when GPUs are running flat out.
- A little light overclocking on the RAM in BIOS, nothing serious.
- Temps: Cards run around 50-60 degrees during processing, CPU runs around 55-65 at full power.

CONSUMPTION
- Scenario 1: Im typicially running Neural Net training for 4-8 hours at a time, pulling around 275 W from each card. This is not CPU intensive at all.
- Scenario 2: Occasionally we will run hypotheses simulations through the Threadripper, getting it up to 90% of power usage ~400 W but not at the same time as scenario 1

Thoughts so far:
From everything I have read, it looks like i'm overloading the PCIe power bus on the MoBo.
- I can't add a PCIe power booster as I have all 4 slots in use.
- Potential options: Undervolt cards, pull-out 10-20% performance
- Can you reduce or cap-out the PCIe power draw or cut it completely from the BIOS? If the 2x 8 Pin power connectors on GPUs provide 150W each, should i maybe limit each card to say 280W to stop them pulling from PCIe power bus?


HELP!!! Need to get back to work!
 

Attachments

  • IMG_2704.jpg
    IMG_2704.jpg
    68.2 KB · Views: 582
  • IMG_2705.jpg
    IMG_2705.jpg
    48.2 KB · Views: 570
  • IMG_2706.jpg
    IMG_2706.jpg
    74.5 KB · Views: 653
  • IMG_2707.jpg
    IMG_2707.jpg
    75.8 KB · Views: 518
  • IMG_2708.jpg
    IMG_2708.jpg
    75.8 KB · Views: 534

eidairaman1

The Exiled Airman
Joined
Jul 2, 2007
Messages
40,435 (6.61/day)
Location
Republic of Texas (True Patriot)
System Name PCGOD
Processor AMD FX 8350@ 5.0GHz
Motherboard Asus TUF 990FX Sabertooth R2 2901 Bios
Cooling Scythe Ashura, 2×BitFenix 230mm Spectre Pro LED (Blue,Green), 2x BitFenix 140mm Spectre Pro LED
Memory 16 GB Gskill Ripjaws X 2133 (2400 OC, 10-10-12-20-20, 1T, 1.65V)
Video Card(s) AMD Radeon 290 Sapphire Vapor-X
Storage Samsung 840 Pro 256GB, WD Velociraptor 1TB
Display(s) NEC Multisync LCD 1700V (Display Port Adapter)
Case AeroCool Xpredator Evil Blue Edition
Audio Device(s) Creative Labs Sound Blaster ZxR
Power Supply Seasonic 1250 XM2 Series (XP3)
Mouse Roccat Kone XTD
Keyboard Roccat Ryos MK Pro
Software Windows 7 Pro 64
Your current draw was too much
 

newtekie1

Semi-Retired Folder
Joined
Nov 22, 2005
Messages
28,472 (4.25/day)
Location
Indiana, USA
Processor Intel Core i7 10850K@5.2GHz
Motherboard AsRock Z470 Taichi
Cooling Corsair H115i Pro w/ Noctua NF-A14 Fans
Memory 32GB DDR4-3600
Video Card(s) RTX 2070 Super
Storage 500GB SX8200 Pro + 8TB with 1TB SSD Cache
Display(s) Acer Nitro VG280K 4K 28"
Case Fractal Design Define S
Audio Device(s) Onboard is good enough for me
Power Supply eVGA SuperNOVA 1000w G3
Software Windows 10 Pro x64
Did you plug in the 4-pin molex connector at the bottom of the motherboard?
 
Joined
Jan 2, 2009
Messages
9,899 (1.78/day)
Location
Essex, England
System Name My pc
Processor Ryzen 5 3600
Motherboard Asus Rog b450-f
Cooling Cooler master 120mm aio
Memory 16gb ddr4 3200mhz
Video Card(s) MSI Ventus 3x 3070
Storage 2tb intel nvme and 2tb generic ssd
Display(s) Generic dell 1080p overclocked to 75hz
Case Phanteks enthoo
Power Supply 650w of borderline fire hazard
Mouse Some wierd Chinese vertical mouse
Keyboard Generic mechanical keyboard
Software Windows ten
Can you tell us precisely what you plugged into motherboard?
Or better yet a picture of everything connected :)
 
Joined
Feb 23, 2019
Messages
5,581 (3.00/day)
Location
Poland
Processor Ryzen 7 5800X3D
Motherboard Gigabyte X570 Aorus Elite
Cooling Thermalright Phantom Spirit 120 SE
Memory 2x16 GB Crucial Ballistix 3600 CL16 Rev E @ 3800 CL16
Video Card(s) RTX3080 Ti FE
Storage SX8200 Pro 1 TB, Plextor M6Pro 256 GB, WD Blue 2TB
Display(s) LG 34GN850P-B
Case SilverStone Primera PM01 RGB
Audio Device(s) SoundBlaster G6 | Fidelio X2 | Sennheiser 6XX
Power Supply SeaSonic Focus Plus Gold 750W
Mouse Endgame Gear XM1R
Keyboard Wooting Two HE
Those aren't the stock cables that come with that PSU:
1626761977203.png
 
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
for such an expensive system, i would suggest using stock supplied power cables. your sleeved cables may look good but might be 'cheap' on the inside 'core' meaning AWG.
please ensure that you are not overloading too much current draw on your cooling setup directly from the MB 4 pin PWM connectors including high draw pumps.
you are very poorly load balanced. 3x of the 3090s should be on PSU2- not just 2.
please ensure all 5 power connectors to the MB are fully connected. no 4 pin CPU shortcuts for example.
please do not overclock the CPU and RAM excessively. limit the voltage and current draw equation you're running.
please ensure that your power cable runs are not too long. the longer they are, the more resistive heat is likely to build at the connector interface.
BIOS does not appear to support configurable PCIe power limits.
 
Last edited:
Joined
Feb 23, 2019
Messages
5,581 (3.00/day)
Location
Poland
Processor Ryzen 7 5800X3D
Motherboard Gigabyte X570 Aorus Elite
Cooling Thermalright Phantom Spirit 120 SE
Memory 2x16 GB Crucial Ballistix 3600 CL16 Rev E @ 3800 CL16
Video Card(s) RTX3080 Ti FE
Storage SX8200 Pro 1 TB, Plextor M6Pro 256 GB, WD Blue 2TB
Display(s) LG 34GN850P-B
Case SilverStone Primera PM01 RGB
Audio Device(s) SoundBlaster G6 | Fidelio X2 | Sennheiser 6XX
Power Supply SeaSonic Focus Plus Gold 750W
Mouse Endgame Gear XM1R
Keyboard Wooting Two HE
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
- Potential options: Undervolt cards, pull-out 10-20% performance
- Can you reduce or cap-out the PCIe power draw or cut it completely from the BIOS? If the 2x 8 Pin power connectors on GPUs provide 150W each, should i maybe limit each card to say 280W to stop them pulling from PCIe power bus?
325w x4 should be an absolute non issue just on the GPUs. the MB should be able to handle combined 100w on the PCIe quite easily. nvidia load balances how much draw can occur over PCIe and the algorithm is quite conservative.
 
Joined
Mar 26, 2012
Messages
221 (0.05/day)
System Name Mixed Bag of OC
Processor AMD Ryzen 5800X3D
Motherboard Maxsun MS-iCraft B550M WIFI
Cooling CPU+GPU on Water with 3 X 420 Rad´s
Memory 32GB Patriot Viper RGB @ 3800 Mhz CL14
Video Card(s) XFX Merc 310 RX 7900 XTX
Storage 2TB Kingston Fury + 2TB Samsung PCIe 4 NVME
Display(s) Philips 48OLED806
Case Selfmade Huuuuuge *Case* :)
Audio Device(s) ifi Zen DAC + Monoprice M1060C & Burmester Replica AMP + Selfmade Huuuuuge Speakers :)
Power Supply Seasonic PRIME TX-750
Mouse Kensington Slimblade (main device) + Razer Basilisk V3 (for FPS)
Keyboard Sharkoon PureWriter RGB, Kailh Blue switches
VR HMD None
Software Windows 11
Benchmark Scores do not matter, my PC is fast :)
Yeah, as some mentioned before, those sleeved cables are not designed to handle long sustained loads. Use the stock cables and you should be fine.
 
Joined
May 28, 2020
Messages
752 (0.54/day)
System Name Main PC
Processor AMD Ryzen 9 5950X
Motherboard ASUS X570 Crosshair VIII Hero (Wi-Fi)
Cooling EKWB X570 VIII Hero Monoblock, 2x XD5, Heatkiller IV SB block for chipset,Alphacool 3090 Strix block
Memory 4x16GB 3200-14-14-14-34 G.Skill Trident RGB (OC: 3600-14-14-14-28)
Video Card(s) ASUS RTX 3090 Strix OC
Storage 500GB+500GB SSD RAID0, Fusion IoDrive2 1.2TB, Huawei HSSD 2TB, 11TB on server used for steam
Display(s) Dell LG CX48 (custom res: 3840x1620@120Hz) + Acer XB271HU 2560x1440@144Hz
Case Corsair 1000D
Audio Device(s) Sennheiser HD599, Blue Yeti
Power Supply Corsair RM1000i
Mouse Logitech G502 Lightspeed
Keyboard Corsair Strafe RGB MK2
Software Windows 10 Pro 20H2
Have to agree with @Chomiq, looks like cheap sleeved cables that weren't able to handle the current draw.
 

FrankJames

New Member
Joined
Jul 20, 2021
Messages
8 (0.01/day)
Did you plug in the 4-pin molex connector at the bottom of the motherboard?
Yep, sure did!

Can you tell us precisely what you plugged into motherboard?
Or better yet a picture of everything connected :)
Hey! Yes I have pics during build and after complete. LMK if you can see what you need

Those aren't the stock cables that come with that PSU:
View attachment 208866
Good point, will try this as part of the fix

Can you tell us precisely what you plugged into motherboard?
Or better yet a picture of everything connected :)
wait, took this video, this should cover everything (screenshots)

for such an expensive system, i would suggest using stock supplied power cables. your sleeved cables may look good but might be 'cheap' on the inside 'core' meaning AWG.
please ensure that you are not overloading too much current draw on your cooling setup directly from the MB 4 pin PWM connectors including high draw pumps.
you are very poorly load balanced. 3x of the 3090s should be on PSU2- not just 2.
please ensure all 5 power connectors to the MB are fully connected. no 4 pin CPU shortcuts for example.
please do not overclock the CPU and RAM excessively. limit the voltage and current draw equation you're running.
please ensure that your power cable runs are not too long. the longer they are, the more resistive heat is likely to build at the connector interface.
BIOS does not appear to support configurable PCIe power limits.
- Pump Draw: So the Pumps (while 12v) are 23W max load. We never max them out at full load so I would say around ~60W maximum. But I have an active ROG pump splitter in the back (seen in image 1830 at the bottom left) so I doubt its that?
- Fan Draw: Same; has an active splitter powering all fans.
- Load balancing: Interesting. So I have calculated that with all pumps, fans, MoBo, CPU pull around 300-400w when active. So with current set up im estimating load at max GPU usage to be:
PSU 1: 1100-1300 W
PSU 2: 700 W
If I load up 3 GPUs on PSU 2 I will be looking at:
PSU 1: 800-1100W
PSU 2: 750 W
Feel like its negligible difference?
- Overclocking: CPU has no OC, RAM is minor but will back
- Power: I'm pretty sure RAM and Threadripper does not use the 12v rail. From what I understand, the GPUs, Fans & Pumps all use the 12v rail but nothing else. Since only the 12v pins on the main MoBo cable were burned out, and all the fans and pumps are on Molex power from PSU 1 (with no issue) it leads me to think its a PCIe power bus overload (runs on 12v). I do have 2 SSds, one in DIMM 2 one in an M2.2 slot and im pretty sure DIMM 2 is pulling from the PCIe bus bout doubt its significant.

325w x4 should be an absolute non issue just on the GPUs. the MB should be able to handle combined 100w on the PCIe quite easily. nvidia load balances how much draw can occur over PCIe and the algorithm is quite conservative.
Yes but a combined PCIe bus draw is well above 100w. Did some more research and the 3090 OC can pull up to 400w at max-out. if the 2 8pin per GPU provide 300W combined power and If thats the case (even during transients) then im looking at a combined 400w pull from the PCIe bus JUST from the GPUs... Am i wrong?

Yeah, as some mentioned before, those sleeved cables are not designed to handle long sustained loads. Use the stock cables and you should be fine.
That makes sense, will swap to all the stock cables. I'm pretty sure the AWG is a much higher rating on the stock. My only concern is regarless of this, I'm not sure I like the fact that i'm putting so much strain on the 12v rail regardless. The PSU can handle it fine but (I Think) but im concerned the MoBo & its PCIe bus is under serious strain.I Know the Zenith extreme Alpha is a beast but i'm looking for long-term reliability.

So:
- Swap out all the shitty power cables with stock
- Undervolt the cards?
- Perhaps I shift the active fan & pump splitters to PSU2 just to ensure a load reduction on PSU1 12v? If I think about it:
- PSU1 12v rail running: GPU 1 & 2, MoBo 12v (PCIe bus), active fan splitter, active pump splitter.
- PSU2 running: GPU 3 & 4.
- Only problem is im a litle concerned about having the MoBo (fan & pump headers) on one PSU and the splitters on another.....

Your current draw was too much
Hey thanks for this! Do you mean on the PCIe power bus?
 

Attachments

  • 63540213119__4EA5C985-CA8A-48EC-90DC-CEB570B634BA.jpeg
    63540213119__4EA5C985-CA8A-48EC-90DC-CEB570B634BA.jpeg
    1.7 MB · Views: 258
  • IMG_1806.jpeg
    IMG_1806.jpeg
    2.2 MB · Views: 254
  • IMG_1830.jpeg
    IMG_1830.jpeg
    2 MB · Views: 262
  • IMG_1838.jpeg
    IMG_1838.jpeg
    2.2 MB · Views: 222
  • IMG_1855.jpeg
    IMG_1855.jpeg
    1.8 MB · Views: 275
  • IMG_1856.jpeg
    IMG_1856.jpeg
    1.2 MB · Views: 203
  • Screen Shot 2021-07-20 at 11.15.44 AM.png
    Screen Shot 2021-07-20 at 11.15.44 AM.png
    2.4 MB · Views: 198
  • Screen Shot 2021-07-20 at 11.15.52 AM.png
    Screen Shot 2021-07-20 at 11.15.52 AM.png
    2.7 MB · Views: 198
  • Screen Shot 2021-07-20 at 11.15.58 AM.png
    Screen Shot 2021-07-20 at 11.15.58 AM.png
    2.4 MB · Views: 187
  • Screen Shot 2021-07-20 at 11.16.06 AM.png
    Screen Shot 2021-07-20 at 11.16.06 AM.png
    2.3 MB · Views: 181
  • Screen Shot 2021-07-20 at 11.16.15 AM.png
    Screen Shot 2021-07-20 at 11.16.15 AM.png
    2.2 MB · Views: 205
Last edited:
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
- Pump Draw: So the Pumps (while 12v) are 23W max load. We never max them out at full load so I would say around ~60W maximum. But I have an active ROG pump splitter in the back (seen in image 1830 at the bottom left) so I doubt its that?
- Fan Draw: Same; has an active splitter powering all fans.
if pumps and fans are not directly running off the MB, then that aspect can be discounted as the one causing the issue.
- Load balancing: Interesting. So I have calculated that with all pumps, fans, MoBo, CPU pull around 300-400w when active. So with current set up im estimating load at max GPU usage to be:
PSU 1: 1100-1300 W
PSU 2: 700 W
If I load up 3 GPUs on PSU 2 I will be looking at:
PSU 1: 800-1100W
PSU 2: 750 W
Feel like its negligible difference?
i would still suggest getting 6 of the 8-pin pcie power connectors on PSU2 to get the max 900w off PSU2. as you mentioned next that the 3090 can pull a transient load of 400w. your research would also tell you that those 8pin pcie connectors do well over 150w at times.
- Overclocking: CPU has no OC, RAM is minor but will back
running RAM at XMP equivalent should really not be an issue.
Yes but a combined PCIe bus draw is well above 100w. Did some more research and the 3090 OC can pull up to 400w at max-out. if the 2 8pin per GPU provide 300W combined power and If thats the case (even during transients) then im looking at a combined 400w pull from the PCIe bus JUST from the GPUs... Am i wrong?
so checking again the 3090 is rated for 350w stock. the 8-pin pcie spec is maxed out at 300w even though i'm sure that the card can draw more out from them as required. they rarely overstress the pcie slot power when are already externally powered. that being said, it might be prudent in your use case to back off the cards to nvidia spec base and boost speeds so that you're in theory then only drawing up to 50w max per slot i/o near 75w. then you'll have a lower draw at the MB cable PSU side of 200w i/o up to 400w.

then you need to know that heat built up over time will expand your cable side connection female end, and lead to potential slowly growing problem of heat and arcing. so your cable side connectors can always be an issue, if not correctly rated.

next, huge conjecture on my part but maybe the PSU isn't the best suited for your 24x7 operation. if the distribution back plane on the PSU burned out due to it being somewhat overstressed and not cutting off in the process, that's a bit of a worry. having said that, you definitely need to get a better and correctly rated 24pin MB cable so that perhaps the poor connection issue at the PSU connector side can be avoided in the 1st place.

your setup is quite elaborate, please ensure that you have sufficient physical strain relief on the PSU side MB cable so that it's not tugging or potentially coming loose over time.

don't bunch up the cables too tightly as that leads to further trapped heat that will increase the overall running temp of the power cables. in your 24x7 environment, that heat may potentially have no 'room to breadth' and cool off or have the right thermal management scope.
 
Joined
Dec 29, 2010
Messages
3,439 (0.71/day)
Processor AMD 5900x
Motherboard Asus x570 Strix-E
Cooling Hardware Labs
Memory G.Skill 4000c17 2x16gb
Video Card(s) RTX 3090
Storage Sabrent
Display(s) Samsung G9
Case Phanteks 719
Audio Device(s) Fiio K5 Pro
Power Supply EVGA 1000 P2
Mouse Logitech G600
Keyboard Corsair K95
3090s can draw well over 350w though. Those specific cards the OP is using can draw 420w peak gaming and when really pushed 450w ish. And gaming loads are not constant so if you're slamming them with a constant workload... yikes.
 

FrankJames

New Member
Joined
Jul 20, 2021
Messages
8 (0.01/day)
if pumps and fans are not directly running off the MB, then that aspect can be discounted as the one causing the issue.

i would still suggest getting 6 of the 8-pin pcie power connectors on PSU2 to get the max 900w off PSU2. as you mentioned next that the 3090 can pull a transient load of 400w. your research would also tell you that those 8pin pcie connectors do well over 150w at times.

running RAM at XMP equivalent should really not be an issue.

so checking again the 3090 is rated for 350w stock. the 8-pin pcie spec is maxed out at 300w even though i'm sure that the card can draw more out from them as required. they rarely overstress the pcie slot power when are already externally powered. that being said, it might be prudent in your use case to back off the cards to nvidia spec base and boost speeds so that you're in theory then only drawing up to 50w max per slot i/o near 75w. then you'll have a lower draw at the MB cable PSU side of 200w i/o up to 400w.

then you need to know that heat built up over time will expand your cable side connection female end, and lead to potential slowly growing problem of heat and arcing. so your cable side connectors can always be an issue, if not correctly rated.

next, huge conjecture on my part but maybe the PSU isn't the best suited for your 24x7 operation. if the distribution back plane on the PSU burned out due to it being somewhat overstressed and not cutting off in the process, that's a bit of a worry. having said that, you definitely need to get a better and correctly rated 24pin MB cable so that perhaps the poor connection issue at the PSU connector side can be avoided in the 1st place.

your setup is quite elaborate, please ensure that you have sufficient physical strain relief on the PSU side MB cable so that it's not tugging or potentially coming loose over time.

don't bunch up the cables too tightly as that leads to further trapped heat that will increase the overall running temp of the power cables. in your 24x7 environment, that heat may potentially have no 'room to breadth' and cool off or have the right thermal management scope.
This is awesome help, thanks very much @Ominence !!! Ok im going in for the fix:

- Replacing out all power cables for the stock EVGA (noticed my off-market white ones are around 20-22 AWG vs the 16 on the EVGAs - yikes)
- Ensure no tension on connections
- Swapping GPU 2 to PSU 2
- Am going to de-bundle cable runs & ensure good air circulation
- Im going to limit the power draw on the cards to something less intense.


Side note, i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous. Troubling because it means another issue but more hope that it could just be a shitty 24 pin.


Stupid question for you @Ominence, when powering up the PSUs should i switch on 1 or 2 first or try to do them simultaneous? Do not have them linked on power-up. I have been doing 1 then .5-1 second later #2.
 
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
Those specific cards the OP is using can draw 420w peak gaming and when really pushed 450w ish.
that's were you check your tailored workload and ensure that the frequency doesn't exceed the manageable power draw, if you're having issues with the system, even if it has to be stock or below as the OP recognises. the 3090 is only rated for 1.395Ghz base and 1.695Ghz boost.
having said that, the pcie bus slot power will still not be excessive, even at 450w (single card case of course). you'll end up having 3x 8pins from the factory on those cards.
with a 4x multiplier buff, a bit of caution is obviously required.
GPUTweakII will easily manage any power draw issues those 3090s are having.
 

FrankJames

New Member
Joined
Jul 20, 2021
Messages
8 (0.01/day)
3090s can draw well over 350w though. Those specific cards the OP is using can draw 420w peak gaming and when really pushed 450w ish. And gaming loads are not constant so if you're slamming them with a constant workload... yikes.
Yes that's a good point, however with the type of models we are training they don't run the memory & cores simultaniouslly. Typically you max-out memory for a few mins loading the data-set up and once complete, it start processing in the cores.

I think I remember running them around 270-300w RMS when running hard.

that's were you check your tailored workload and ensure that the frequency doesn't exceed the manageable power draw, if you're having issues with the system, even if it has to be stock or below as the OP recognises. the 3090 is only rated for 1.395Ghz base and 1.695Ghz boost.
having said that, the pcie bus slot power will still not be excessive, even at 450w (single card case of course). you'll end up having 3x 8pins from the factory on those cards.
with a 4x multiplier buff, a bit of caution is obviously required.
GPUTweakII will easily manage any power draw issues those 3090s are having.
Any tips for good software for GPU tuning on Linux?

Also, do you have a recommended, rough idea of how I should reduce power load on tune? I was thinking:
- Reduce core clock down to 1.3
- Reduce voltage 10%
- Reduce boost down to 1.4-1.5
- Reduce power limit by 15%
 
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
Joined
Feb 6, 2021
Messages
2,554 (2.23/day)
Location
Germany
System Name Sunk Cost Fallacy
Processor AMD Ryzen 7 7800X3D
Motherboard ASRock B650E Steel Legend Wifi
Cooling Arctic Liquid Freezer II 360 Rev. 7
Memory 2x16GB G.Skill Trident Z5 NEO 6000 CL30
Video Card(s) Sapphire Nitro+ RX 7900 XTX Vapor-X
Storage WD Black SN850X 1TB + 2x 2TB, 2x 4TB Crucial MX500, 4TB Samsung 870 Evo.
Display(s) Alienware AW2723DF, LG 27GR93U, LG 27GN950-B
Case Lian Li O11 Air Mini
Audio Device(s) Bose Companion Series 2 III, Sennheiser GSP600 and HD599 SE - Creative Soundblaster X4
Power Supply bequiet! Dark Power Pro 12 1500w Titanium
Mouse Logitech GPRO X Superlight & G502 X
Keyboard Corsair K65 RGB Mini, Razer Black Widow V3 TKL
VR HMD Oculus Rift S
i saw your post yesterday and it was deleted a few minutes later.

your PSUs area basically a high wattage low end PSU (EVGA GX PSUs are absolute garbage..)

just get two AX1600I Titanium from Corsair or two bequiet! dark power pro 12 1500W. (since your System costs as much as a decent used car.)
 
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
you can always use a U wire (glued in, and insulated on the exposed part) to bridge the stock 24pin on PSU2 and before you fire PSU1, fire up PSU2 1st after the bridging.
 
Joined
Dec 29, 2010
Messages
3,439 (0.71/day)
Processor AMD 5900x
Motherboard Asus x570 Strix-E
Cooling Hardware Labs
Memory G.Skill 4000c17 2x16gb
Video Card(s) RTX 3090
Storage Sabrent
Display(s) Samsung G9
Case Phanteks 719
Audio Device(s) Fiio K5 Pro
Power Supply EVGA 1000 P2
Mouse Logitech G600
Keyboard Corsair K95
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
Any tips for good software for GPU tuning on Linux?
GreenWithEnvy—Linux Apps on Flathub
Also, do you have a recommended, rough idea of how I should reduce power load on tune? I was thinking:
- Reduce core clock down to 1.3
- Reduce voltage 10%
- Reduce boost down to 1.4-1.5
- Reduce power limit by 15%
reducing voltage will just make the system potentially less stable.

power is a function of V and I. between them, they can still get to the hypothetical 450w, stability permitting. you should try to work on the much simpler power limit settings if the Linux tool allows it to go negative from vendor stock.

Edit- if the PL option is not available, reduce the boost clock 1st (to stock) as that is the stretch target on the clock speed algorithm.
 
Joined
Jan 27, 2015
Messages
1,065 (0.32/day)
System Name loon v4.0
Processor i7-11700K
Motherboard asus Z590TUF+wifi
Cooling Custom Loop
Memory ballistix 3600 cl16
Video Card(s) eVga 3060 xc
Storage WD sn570 1tb(nvme) SanDisk ultra 2tb(sata)
Display(s) cheap 1080&4K 60hz
Case Roswell Stryker
Power Supply eVGA supernova 750 G6
Mouse eats cheese
Keyboard warrior!
Benchmark Scores https://www.3dmark.com/spy/21765182 https://www.3dmark.com/pr/1114767
Side note, i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous. Troubling because it means another issue but more hope that it could just be a shitty 24 pin.
there *should be* 5 lines of 12v in the riser cable:
Capture.PNG

at 1.1 amps each for ~66 watts, the other ~9 is 3.3v (75 total)

that riser, esp w/o getting hot, is perfectly fine . . . it seems.
 
Joined
Jun 26, 2014
Messages
71 (0.02/day)
Location
Mackay, Queensland, Australia
System Name PDOCT18
Processor Intel Core i9-9900X. 2c 4.5GHz, 8c 4.3GHz
Motherboard ASUS Prime X299-Deluxe II
Cooling Noctua NH-D15S (2x NF-A14 industrialPPC-3000 PWM), 5x NF-A14 industrialPPC-3000 PWM
Memory 64GB F4-3200C16-16GVK 3200MHz 16-18-18-38-1T
Video Card(s) nVidia GeForce RTX 3070 Gigabyte Aorus Master 8G, 303W power target frequency curve, stock voltage
Storage Boot- Intel 750 1.2TB NVMe AIC SSD, Game- Samsung 860 Pro 4TB SATA SSD, 960 Pro M.2 512GB, 19TB HDDs
Display(s) CX55 OLED, U3219Q, UP3216Q
Case Fractal Design Define R6 TG USB-C Gunmetal (no stock fans)
Audio Device(s) nVidia HDMI passthru to Denon AVR-X2700H 8ch receiver 1.01kW speakers, Sennheiser HD 800 S 75th AE
Power Supply Thermaltake Toughpower DPS G RGB 1.25kW Titanium
Mouse G903 Hero 16K LightSpeed with PowerPlay Wireless Charging System mousepad
Keyboard G915 brown tactile
Software What is that!
Benchmark Scores Cannot beat K|NGP|N
Side note, i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous. Troubling because it means another issue but more hope that it could just be a shitty 24 pin.
very likely that has been the issue with your nice setup. going for aesthetics, you have to be very careful with all technical aspects too;)

just FYG, my FD flex is rated to 40w 3.33A.
Flex VRC-25 _Product sheet (fractal-design.com)
cable run length would be an issue for you though.
i noticed the PCI-E extension cables im using are totally fine, zero sign of overheating yet they are only rated for 1 amp (30 awg). 1 amp @ 12v = 12 watts so that means they are not pulling anything ridiculous.
not to be alarmist, but they may have already burned out without showing through the insulation. just to be sure, do a simple continuity check on any common multimeter. they look like passive cables without additional electronic circuitry running on them.

That's all I needed when I ran dual psu. If you look closely you can see the jumper cable.
ah, reminds me of the days i used to do quad CFX systems before going to solitary Titans.
 
Last edited:

newtekie1

Semi-Retired Folder
Joined
Nov 22, 2005
Messages
28,472 (4.25/day)
Location
Indiana, USA
Processor Intel Core i7 10850K@5.2GHz
Motherboard AsRock Z470 Taichi
Cooling Corsair H115i Pro w/ Noctua NF-A14 Fans
Memory 32GB DDR4-3600
Video Card(s) RTX 2070 Super
Storage 500GB SX8200 Pro + 8TB with 1TB SSD Cache
Display(s) Acer Nitro VG280K 4K 28"
Case Fractal Design Define S
Audio Device(s) Onboard is good enough for me
Power Supply eVGA SuperNOVA 1000w G3
Software Windows 10 Pro x64
Yep, sure did!
It's hard to tell what is going on in the pictures, but it isn't plugged in in the one picture that shows the connector. Are you sure it was plugged in?
 

eidairaman1

The Exiled Airman
Joined
Jul 2, 2007
Messages
40,435 (6.61/day)
Location
Republic of Texas (True Patriot)
System Name PCGOD
Processor AMD FX 8350@ 5.0GHz
Motherboard Asus TUF 990FX Sabertooth R2 2901 Bios
Cooling Scythe Ashura, 2×BitFenix 230mm Spectre Pro LED (Blue,Green), 2x BitFenix 140mm Spectre Pro LED
Memory 16 GB Gskill Ripjaws X 2133 (2400 OC, 10-10-12-20-20, 1T, 1.65V)
Video Card(s) AMD Radeon 290 Sapphire Vapor-X
Storage Samsung 840 Pro 256GB, WD Velociraptor 1TB
Display(s) NEC Multisync LCD 1700V (Display Port Adapter)
Case AeroCool Xpredator Evil Blue Edition
Audio Device(s) Creative Labs Sound Blaster ZxR
Power Supply Seasonic 1250 XM2 Series (XP3)
Mouse Roccat Kone XTD
Keyboard Roccat Ryos MK Pro
Software Windows 7 Pro 64
Yep, sure did!


Hey! Yes I have pics during build and after complete. LMK if you can see what you need


Good point, will try this as part of the fix


wait, took this video, this should cover everything (screenshots)


- Pump Draw: So the Pumps (while 12v) are 23W max load. We never max them out at full load so I would say around ~60W maximum. But I have an active ROG pump splitter in the back (seen in image 1830 at the bottom left) so I doubt its that?
- Fan Draw: Same; has an active splitter powering all fans.
- Load balancing: Interesting. So I have calculated that with all pumps, fans, MoBo, CPU pull around 300-400w when active. So with current set up im estimating load at max GPU usage to be:
PSU 1: 1100-1300 W
PSU 2: 700 W
If I load up 3 GPUs on PSU 2 I will be looking at:
PSU 1: 800-1100W
PSU 2: 750 W
Feel like its negligible difference?
- Overclocking: CPU has no OC, RAM is minor but will back
- Power: I'm pretty sure RAM and Threadripper does not use the 12v rail. From what I understand, the GPUs, Fans & Pumps all use the 12v rail but nothing else. Since only the 12v pins on the main MoBo cable were burned out, and all the fans and pumps are on Molex power from PSU 1 (with no issue) it leads me to think its a PCIe power bus overload (runs on 12v). I do have 2 SSds, one in DIMM 2 one in an M2.2 slot and im pretty sure DIMM 2 is pulling from the PCIe bus bout doubt its significant.


Yes but a combined PCIe bus draw is well above 100w. Did some more research and the 3090 OC can pull up to 400w at max-out. if the 2 8pin per GPU provide 300W combined power and If thats the case (even during transients) then im looking at a combined 400w pull from the PCIe bus JUST from the GPUs... Am i wrong?


That makes sense, will swap to all the stock cables. I'm pretty sure the AWG is a much higher rating on the stock. My only concern is regarless of this, I'm not sure I like the fact that i'm putting so much strain on the 12v rail regardless. The PSU can handle it fine but (I Think) but im concerned the MoBo & its PCIe bus is under serious strain.I Know the Zenith extreme Alpha is a beast but i'm looking for long-term reliability.

So:
- Swap out all the shitty power cables with stock
- Undervolt the cards?
- Perhaps I shift the active fan & pump splitters to PSU2 just to ensure a load reduction on PSU1 12v? If I think about it:
- PSU1 12v rail running: GPU 1 & 2, MoBo 12v (PCIe bus), active fan splitter, active pump splitter.
- PSU2 running: GPU 3 & 4.
- Only problem is im a litle concerned about having the MoBo (fan & pump headers) on one PSU and the splitters on another.....


Hey thanks for this! Do you mean on the PCIe power bus?

Yes for those cables to fry, it tells me the current draw is higher than what the cables can handle.
 
Top