Watercooling loop troubleshooting help please

[nflesher87]

Hey ya'll so I've got a fun mining rig that crashed over the weekend and I need some help diagnosing please!

Rig:
ASRock TRX40 CREATOR sTRX4
Threadripper 3960X
G.SKILL TridentZ Series 32GB (4 x 8GB) 288-Pin DDR4
1x RTX3080 Asus TUF
3x RTX3080 Dell Reference
2x Phanteks Revolt Pro 1000W

Cooling:
CPU is on it own dedicated loop

GPUs:
Pump: EKWB Quantum Kinetic TBE 200 D5
Rads: 2x XSPC 480mm Ultrathin (Copper)
Tubing: EKWB Duraclear
Coolant: EKWB Cryofuel Mystic Fog
Blocks: 4x EKWB either Nickel or Copper
All fittings were verified non-aluminum

Over night Sunday the rig stopped mining. When I was able to inspect it, the flexible tubing was noticeably discolored towards a yellow tint. The res had some dark debris in it in the foam.
PC boots but I can't get any video feed. Tried all 4 VGAs. At initial setup it was picky about giving video so I'm not surprised about that part.
Any ideas what happened here?
No leaks identifiable, but in 4 months of 24/7 mining the res level dropped an inch and it was getting low.
The liquid appears to flow as it did before this incident.
I can't think of anything else that would help diagnose.
Help!

 

[ThrashZone]

Hi,
Just have to see if any blocks are blocked piping is interesting.

 

[quanash]

I have 2 questions:

- Did you flush the rads before assembling?
-Was the system was somewhere where sunlight hits inside the case?

Decreased coolant level is normal when you consider 7/24 mining on GPUs.

 

[nflesher87]

I have 2 questions:

- Did you flush the rads before assembling?
-Was the system was somewhere where sunlight hits inside the case?

Decreased coolant level is normal when you consider 7/24 mining on GPUs.

I did not flush the rads. Had never heard of that unfortunately.
No sunlight.

 

[ThrashZone]

Hi,
Considering there's a large window wall in your picture your statement doesn't pass the smell test

 

[mx62]

the yellow tint, to me its some kind of metal/paint
can you see the fins in the gpu block maybe an picture?

 

[nflesher87]

Hi,
Considering there's a large window wall in your picture your statement doesn't pass the smell test

Lol. I relocated it to open and diagnose.
99.99% of the time and during operation it is in an interior part of the building with no exposure to sunlight.
It is however close to an xray generating machine if that matters.

the yellow tint, to me its some kind of metal/paint
can you see the fins in the gpu block maybe an picture?

Sorry all of the blocks are opaque. Worst case scenario I'll take them off and analyze.

 

[quanash]

IMHO, that color might be because of 2 reasons; The Flux inside the radiator or the algae growth because of dirty rad + sunlight + extreme heat. Second reason might not be the issue since there are inhibitors inside the coolant.

The color reminded me the pictures inside in the article below;

Flux residues in cheap radiators

Can you disconnect the 3 GPUs from the system and boot it with the single one?

 

[ThrashZone]

Hi,
Just have to start over
Mayhems has better fluids x1 clear is what I use it's best to not get weird with fluids keep it clear.

Mayhem X1 ECO Clear Concentrate Watercooling Fluid 250ml

Mayhem X1 ECO Clear Concentrate Watercooling Fluid 250ml

mayhems.store

Mayhams also has good cleaning products.

Water Cooling Loop Cleaning - Additives & Cleaning

mayhems.store

 

[nflesher87]

IMHO, that color might be because of 2 reasons; The Flux inside the radiator or the algae growth because of dirty rad + sunlight + extreme heat. Second reason might not be the issue since there are inhibitors inside the coolant.

The color reminded me the pictures inside in the article below;

Flux residues in cheap radiators

Can you disconnect the 3 GPUs from the system and boot it with the single one?

Also to clarify if it helps: this all happened suddenly. I glance at the system regularly and there was no sign of color change. So I feel like this happened in a sudden event.

 

[ThrashZone]

Fluid fallout happens without warning.

 

[xtreemchaos]

allways flush rads other wise your just inheriting trouble i allways put 20% white vinger to 80% distlilled water and leave over night and flush in the morrning with just distilled water the crap that comes out of even new rads never disapoints. but if its got something to do with no display coming from the rigg i dont know could just be a software issue but i know nothing about mining riggs so ill just wish you goodluck mate.

 

[quanash]

sudden, how sudden? In a few days, in a few hours?

Only logical reason for a sudden change comes to mind is insane amount of heat dumped into the cooling system. This might be because of failure or inadequate fan / rad cooling or failure of pump.

 

[mx62]

Fluid fallout happens without warning.

doesn't explain the yellow tint

can you see movie liquid when you turn on?
and strange noise?
could it be material from the pump?

 

[nflesher87]

allways flush rads other wise your just inheriting trouble i allways put 20% white vinger to 80% distlilled water and leave over night and flush in the morrning with just distilled water the crap that comes out of even new rads never disapoints. but if its got something to do with no display coming from the rigg i dont know could just be a software issue but i know nothing about mining riggs so ill just wish you goodluck mate.

thanks

sudden, how sudden? In a few days, in a few hours?

Only logical reason for a sudden change comes to mind is insane amount of heat dumped into the cooling system. This might be because of failure or inadequate fan / rad cooling or failure of pump.

On Friday everything was fine. Sunday night the rig crashed and this is what I found.

doesn't explain the yellow tint

can you see movie liquid when you turn on?
and strange noise?
could it be material from the pump?

Loop appears to be operating normally...

 

[ThrashZone]

Hi,
I'd bet fluid why you'd use this crap Cryofuel Mystic Fog for a mining rig all bets are off.

 

[quanash]

Is it possible to dismantle the bottom 3 GPUs from the motherboard and try to boot the system again?

Unplug their power cables and if there is enough slack between gpu block and pump, put those on e box or ath near the case without disassembly of the cooling system.

 

[ThrashZone]

Hi,
Premix fluids are usually only good for 1 year and this is for normal use not 24/7 mining.

Do you have any ph test gear
Check the ph level
Premix is usually ph8+ to start.

 

[quanash]

Hi,
Premix fluids are usually only good for 1 year and this is for normal use not 24/7 mining.

Do you have any ph test gear
Check the ph level
Premix is usually ph8+ to start.

You are missing the real reason of his post. The system is not working, before talking about these first we have to help his system to work. Later we can give our suggestions about the cooling solution or other things we see problematic.

 

[ThrashZone]

You are missing the real reason of his post. The system is not working, before talking about these first we have to help his system to work. Later we can give our suggestions about the cooling solution or other things we see problematic.

Hi,
What's to trouble shoot ?
System needs to be cleaned first period.

Testing the fluids ph level is relevant to what color it is don't you think
OP could save us all a lot of type and send some fluid to a lab for diagnostics while he disassembles/ cleans the loop.

 

[quanash]

Hi,
What's to trouble shoot ?
System needs to be cleaned first period.

Testing the fluids ph level is relevant to what color it is don't you think
OP could save us all a lot of type and send some fluid to a lab for diagnostics while he disassembles/ cleans the loop.

Coolant status is irrelevant right now. If we can't solve the systems issue there is no need to check it.

He might need to disassemble the cooling system and check the GPUs one by one if they are still ok and get video output from the system. After that, he can clean all the cooling system and reassemble correctly.

 

[ThrashZone]

Hi,
Okay sure throw a air cooler on a gpu and see what happens or install a spare air cooled gpu and see what happens.

Only safe item is the cpu cooler seeing it's not part of the loop.

But free free to trouble shoot a toxic loop without cleaning it first that should be ezpz no hard work involved at all just speculation

 

[quanash]

Hi,
Okay sure throw a air cooler on a gpu and see what happens or install a spare air cooled gpu and see what happens.

Only safe item is the cpu cooler seeing it's not part of the loop.

But free free to trouble shoot a toxic loop without cleaning it first that should be ezpz no hard work involved at all just speculation

No need to change the blocks only yo get a video signal from the system. The GPU block has enough thermal mass to keep the GPUs temps in check until it goes to windows login screen. If the GPU pass that, it means is fine for further testing. If not, this means there is an issue about the GPU or the rest of the system ( other GPUs post test will clarify which one is the culprit ).

When he disassembles the system, he can keep a part of coolant as a sample to test its PH value later. But from the pictures it seems that coolant have already been mixed with a clear one. It is to transparent for Mystic fog coolant.

 

[ThrashZone]

Hi,
Since the op is absent here maybe he's busy and being smart about it and draining and disassembling fluid and the sludge that's in his gpu blocks now
Later he can refill and see if it starts again normally with just one gpu in it.
If one fails try another gpu and see a different one works/....

 

[Deleted member 24505]

I would drain it, refill run it, drain it, refill, run it, a few times with the ATX 24 jumped. Hopefully would not mean stripping it.