780 Issues

[Psit]

Hello people, i am having a lot of trouble with my system and after like 30 days of on and off troubleshooting i think my problem is my graphics card which is a Gigabyte GV-N780OC-3GD .

Before going into details with my issue (this will take a huge post i guess) i want to share with you my test result using the TestCudaMemoryBandwidthPerformance application, which is used to check the vram of the graphics cards.

I am attaching some of my logs, can someone take a look and let me know if this looks like a bad VRAM?

Thank you for your time.


POST EDIT

Ok, here goes.
For the past year i have been using my PC for triple screen sim-racing.
Everything was fine until some day i started having system hangs, BSOD's with sound looping.
In the beginning it was very rare but now it has become consistent.
The game i play is Assetto Corsa, and all my crashes only when on triple screen (where GPU usage and vram usage are 100% all the time)
Pretty much after about 10-30 minutes of triple screen gaming my pc:

a) hangs completely
Screen freezes. Some times i lose signal to all 3 monitors and have to pull the plug.
I sometimes saw some artifacts, picture bellow:

b) The driver resets itself
When this happens, the screen freezes, sound stutters for 5 secs, then sound works normally but the screen stays frozen.
I ctrl alt del and acs.exe (game exe) crashes out...

I checked the event viewer and found this (usually :
Display driver nvlddmkm stopped responding and has successfully recovered
Faulting application name: acs.exe, version: 0.0.0.0, time stamp: 0x54b8db1c
Faulting module name: CSERHelper.dll, version: 4.50.0.0, time stamp: 0x43dae554
Exception code: 0xc0000005
Fault offset: 0x00012578
Faulting process id: 0x1074
Faulting application start time: 0x01d03cc2b02d6620
Faulting application path: D:\Program Files (x86)\Steam\steamapps\common\assettocorsa\acs.exe
Faulting module path: D:\Program Files (x86)\Steam\CSERHelper.dll
Report Id: 9afb3b90-a8b6-11e4-8036-1c6f6538d19a

and this (rarely):
Faulting application name: acs.exe, version: 0.0.0.0, time stamp: 0x54d31f55
Faulting module name: kernel32.dll, version: 6.1.7601.18409, time stamp: 0x53159a85
Exception code: 0xc0000005
Fault offset: 0x000e030c
Faulting process id: 0x1674
Faulting application start time: 0x01d04bb4ccdc90fb
Faulting application path: D:\Program Files (x86)\Steam\steamapps\common\assettocorsa\acs.exe
Faulting module path: C:\Windows\syswow64\kernel32.dll
Report Id: 7ef2806b-b7a9-11e4-93b2-1c6f6538d19a

acs.exe
0.0.0.0
54d31f55
kernel32.dll
6.1.7601.18409
53159a85
c0000005
000e030c
1674
01d04bb4ccdc90fb
D:\Program Files (x86)\Steam\steamapps\common\assettocorsa\acs.exe
C:\Windows\syswow64\kernel32.dll
7ef2806b-b7a9-11e4-93b2-1c6f6538d19a
Sometimes, after looking into windows event viewer i can find this :
The computer has rebooted from a bugcheck. The bugcheck was: 0x00000116 ---> After googling this i get the following : The VIDEO_TDR_ ERROR bug check has a value of 0x00000116. This indicates that an attempt to reset the display driver and recover from a timeout failed.

I am attaching pictures of the BSOD viewer to have a look if you want, there are some different cases there...

c) BSOD
Sometimes i get BSOD, i have uploaded the pictures of blue screen viewer above

What i have tried so far:

1) Tried another PSU
2) Ran with custom FAN (80%) RPM for the GPU
3) Loaded safe defaults in bios
4) Limit the game to 60 fps to take some load off
5) Tried a pci soundcard, xonar ds
6)Tried about 10 driver versions, always using DDU first
7) Fresh windows install on another HDD
8) Disabled nvidia sound
9) Overvoltage CPU
10) Disable turbo boost of cpu
11) disable hyperthread
12) disable usb3 controller
13) Properly tested (24hrs) ram sticks
14) Benchmarked GPU with furmark for 1 hour in triple screen mode
15) Benchmarked CPU with many benchmark tools, PRIME crashed my CPU in 5-10 minutes
16) I have troubleshooted this with the Assetto Corsa developers for 4 months, they are claiming its a driver issue... i have no reason to believe them.
17) I have reseated my CPU, changed the thermal paste and blew out all dust from the fan / heatsink
18) Currently running only 3 sticks of RAM instead of 6
19) many more stuff i might be forgetting right now...

Right now, i can game normally without crashing but i have done the following so i wont crash:

Downclocked GPU RAM -1000MHZ.....
Downclocked GPU core -100MHZ.....
Overvoltage GPU 1 step

I am losing a lot of performance but i can still play.
I am trying to find out if it is my GPU or something else in my pc....

Trying to solve this since January.

 

[Law-II]

Hi

Welcome to TPU

Why running an application that was designed and coded primarily for a 4GB GTX 970 ? on a GTX 780

"As I mentioned before, there is a new tool (which is a work in progress, so you will be a VOLUNTARY TESTER, you need to understand this from the start) that can help us dig dipper into the GTX 970 memory issues and configuration."
Source - here

Note: why not address the author of the application

What issue dose the vga card have in games and applications? let the TPU community know they may be able to assist further

go here fill in full system specs hardware manufacturer make; model & OS

EDIT: if possible add a screen shot from GPU-Z showing default core and memory speeds for the vga card

atb

Law-II

 

[MxPhenom 216]

that app is not meant to be used to troubleshoot a video card.

 

[Psit]

OP edited, happy reading

 

[Daemmon]

"I'm the "German programmer", who initially has programmed this benchmark. As I've already stated in this thread several times and immediately after posting the download: This benchmark isn't eligible for benchmarking the VRAM-bug. You're just benchmarking the swapping behaviour of the global memory space in CUDA. However some people spread this benchmark all around the internet, without reading, understanding or knowing its issues. Even some major news sites disgraced themselves by using this benchmark without investigating it further.

There's also a more recent version of the source code hidden within the spoiler down the page. Furthermore this benchmark isn't very well designed, since I've written it within 20 minutes and I've never dreamed of it becoming so notoriously "popular". Thus the quality of the code is very poor. Also I'm very sorry for the unjustified uproar, which this benchmark is causing."

The benchmark tool shouldn't even be used for the 970, and even less for older cards. Stop suggesting it people, please. Quote is from the creator of the tool.

 

[Psit]

"I'm the "German programmer", who initially has programmed this benchmark. As I've already stated in this thread several times and immediately after posting the download: This benchmark isn't eligible for benchmarking the VRAM-bug. You're just benchmarking the swapping behaviour of the global memory space in CUDA. However some people spread this benchmark all around the internet, without reading, understanding or knowing its issues. Even some major news sites disgraced themselves by using this benchmark without investigating it further.

There's also a more recent version of the source code hidden within the spoiler down the page. Furthermore this benchmark isn't very well designed, since I've written it within 20 minutes and I've never dreamed of it becoming so notoriously "popular". Thus the quality of the code is very poor. Also I'm very sorry for the unjustified uproar, which this benchmark is causing."

The benchmark tool shouldn't even be used for the 970, and even less for older cards. Stop suggesting it people, please. Quote is from the creator of the tool.

It was my last resort, after 4 months of trying to figure out... dont crucify me for trying it out, i thought it would stretch the vram of my gpu...

 

[Law-II]

Hi

It was my last resort, after 4 months of trying to figure out... dont crucify me for trying it out, i thought it would stretch the vram of my gpu...

Getting the post back on course;

Most of the error codes look like the CPU IMC [integrated memory controller] is under heavy stress with six slots filled, [remove 3x2Gb sticks of ram to reduce stress in the IMC] **test again
[**use default clock speeds while testing]

Note: upload a screen shot from GPU-Z

0x124 is/can be related to CPU voltage [possible CPU degraded] {occurs when over clocking}
PRIME crashed my CPU in 5-10 minutes [CPU unstable/ or unstable overclock]
0xc5 is Driver pool corruption [monitor.sys]
0x116 is TDR
0x117 is TDR Detected

General Info here

atb

Law-II

 

[Psit]

Hi


Getting the post back on course;

Most of the error codes look like the CPU IMC [integrated memory controller] is under heavy stress with six slots filled, [remove 3x2Gb sticks of ram to reduce stress in the IMC] **test again
[**use default clock speeds while testing]

Note: upload a screen shot from GPU-Z

0x124 is/can be related to CPU voltage [possible CPU degraded] {occurs when over clocking}
PRIME crashed my CPU in 5-10 minutes [CPU unstable/ or unstable overclock]
0xc5 is Driver pool corruption [monitor.sys]
0x116 is TDR
0x117 is TDR Detected

General Info here

atb

Law-II

Hello and thanks for your reply.

For the last 2 weeks i am running with only 3 sticks of ram
CPU degraded, might be but i have NEVER overclocked it.
Concerning the PRIME crash, as i said i have never overclocked my cpu.
Concerning driver crashes, dont know what to say really....

Is there a benchmark of some sort that can fill my VRAM in order to test that?
Sadly i dont have a second GPU to try on my system because a friend borrowed it and he fried it..... yeah i know.

 

[Law-II]

Hi

For the last 2 weeks i am running with only 3 sticks of ram

could you elaborate, this answer dose not tell the TPU community anything [have the artefacts gone?] [is the system more stable?] or just the same as before

Is there a benchmark of some sort that can fill my VRAM in order to test that?

not that I am aware of, other members of the community may be able to chip in

Note: if you have not already done so; reseat the CPU [while the CPU is out of the MOBO reset the bios] reseat the vga card at the same time and reinstall the latest drivers from Nvidia

atb

Law-II

 

[Psit]

In my things-i-tried list i forgot to include that i have reseated my CPU, changed the thermal paste and blew out all dust from the fan / heatsink. I will add it to the OP.

I am running 3 sticks only and i still have the same issue.

 

[Daemmon]

Ok, let's go to the basics. What are the temps of the GPU? The most frequent error you're getting is a driver crash, a TDR. Bad/faulty VRAM is probably one of the last things I will think off. GPU temp, VRAM temp are the usual suspects.

If those are ok then:

1) Tested disabling Aero?
2) Tested raising the TdrDelay? Look for it on google if you don't know how to do it. (nVidia's recommendation on some places, single GPU : TdrDelay of 10 secs, multi GPU : Tdr OFF)
3) Play the game while keeping GPU-Z logging on the background. When/if the game crashes upload the log to the forums.

 

[Psit]

Ok, let's go to the basics. What are the temps of the GPU? The most frequent error you're getting is a driver crash, a TDR. Bad/faulty VRAM is probably one of the last things I will think off. GPU temp, VRAM temp are the usual suspects.

If those are ok then:

1) Tested disabling Aero?
2) Tested raising the TdrDelay? Look for it on google if you don't know how to do it. (nVidia's recommendation on some places, single GPU : TdrDelay of 10 secs, multi GPU : Tdr OFF)
3) Play the game while keeping GPU-Z logging on the background. When/if the game crashes upload the log to the forums.

1) So, the temps of the GPU are absolutely normal, it goes up to 80c i think maximum 84c but this is very rare.
2) I actually tried to do that but for some reason it wouldnt turn off ! Do you have a link of a 100% way to do it?
3) Will do i when i get back home, i was running hardware monitor and the logs were absolutely fine, all sensors were normal but i will upload it later today.

 

[Law-II]

Hi

In my things-i-tried list i forgot to include that i have reseated my CPU, changed the thermal paste and blew out all dust from the fan / heatsink. I will add it to the OP.I am running 3 sticks only and i still have the same issue.

assuming this issue is with all games that are tri-screen enabled

test vga card in another system if same issue raise a ticket and RMA [**if vga card is o.k]
**start to look at the MOBO and CPU/RAM combo

atb

Law-II

 

[Psit]

Hi



assuming this issue is with all games that are tri-screen enabled

test vga card in another system if same issue raise a ticket and RMA [**if vga card is o.k]
**start to look at the MOBO and CPU/RAM combo

atb

Law-II

I can play Rfactor 2 , GSCE fine without crashing, but these games are nowhere near as cpu and gpu intensive as Assetto Corsa. I can also run furmark on triples for 1 hour or longer with no issues.
In AC, GPU and CPU are almost at 100% for the most part of the gaming session.

RAM sticks are fine since i did a long test for them they had 0 errors.
Sadly i dont have a way to test the VGA card to another pc and triple screens....

 

[Daemmon]

1) So, the temps of the GPU are absolutely normal, it goes up to 80c i think maximum 84c but this is very rare.
2) I actually tried to do that but for some reason it wouldnt turn off ! Do you have a link of a 100% way to do it?
3) Will do i when i get back home, i was running hardware monitor and the logs were absolutely fine, all sensors were normal but i will upload it later today.

80ºC+ could be enough if the card has bad VRM cooling. I mean, with no VRM sensors you never know what's exactly happening to them. I had driver crash issues from bad VRM cooling when my old GPU went past 70ºC.

To disable aero all you have to do is pick a non-aero theme on Windows (Basic or High Contrast).

Furmark make 0 crashes after 1 hour, peak temp on it? Is it reaching 100% load on GPU and VRAM?

 

[BiggieShady]

Is there a benchmark of some sort that can fill my VRAM in order to test that?

not that I am aware of, other members of the community may be able to chip in

This one is command line tool IIRC, used to be promoted by folding@home years ago
https://simtk.org/home/memtest
https://simtk.org/frs/download.php?file_id=2466
The only issue is it wasn't updated since v1.0 in 2010, but all new gpus should run code built with older cuda/opencl libraries without issues.

edit: better links
https://folding.stanford.edu/home/download-utilities/
http://fah-web.stanford.edu/MemtestCL/memtestCL-1.00-windows.zip
http://fah-web.stanford.edu/MemtestG80/memtestG80-1.1-windows.zip

 

[Psit]

80ºC+ could be enough if the card has bad VRM cooling. I mean, with no VRM sensors you never know what's exactly happening to them. I had driver crash issues from bad VRM cooling when my old GPU went past 70ºC.

To disable aero all you have to do is pick a non-aero theme on Windows (Basic or High Contrast).

Sorry, i forgot to reply to you that i always disable Aero when i run Assetto Corsa (using the properties option of the .exe)

Concerning the cooling yes, but is there a way to check that... this particular graphics card seems to have some nice cooling to behonest and i have tried gaming with the fans at 80% manual mode.... still got the same results.

 

[Daemmon]

Concerning the cooling yes, but is there a way to check that... this particular graphics card seems to have some nice cooling to behonest and i have tried gaming with the fans at 80% manual mode.... still got the same results.

Yes... and no. Having a temp measuring tool and looking for the hotspot on the PCB is probably the most efficient way (when cards don't have VRM sensors). Also, GPU "general" cooling =/= VRM cooling. There have been countless cases of GPUs running on pretty safe temps but having atrocious VRM temps.

Raising the fans can do the trick some times, but we won't know for sure. 100% honest with you? If you can, RMA that card asap. If you can't lets keep troubleshooting.

 

[Psit]

This one is command line tool IIRC, used to be promoted by folding@home years ago
https://simtk.org/home/memtest
https://simtk.org/frs/download.php?file_id=2466
The only issue is it wasn't updated since v1.0 in 2010, but all new gpus should run code built with older cuda/opencl libraries without issues.

edit: better links
https://folding.stanford.edu/home/download-utilities/
http://fah-web.stanford.edu/MemtestCL/memtestCL-1.00-windows.zip
http://fah-web.stanford.edu/MemtestG80/memtestG80-1.1-windows.zip

Thanks for these, although i am not sure i used it correctly, running the memtestg80 i got errors right away, there is a pic at the beginning of the test and at the end of it attached.

Not sure what to make of this.

 

[Ruyki]

Those errors should probably not be there. I just tried that test and got 0.

OCCT GPU test with error checking enabled is another tool you can use to check for GPU errors.

Anyway, your GPU may be busted.

 

[Psit]

Can anyone else do this test and let me know their results? Just to be sure

 

[BiggieShady]

Can anyone else do this test and let me know their results? Just to be sure

My bad, I guess that cuda version is way too old for newer architectures ... it was made in the days of G80 (GTX 8800). Yeah, I get bogus errors on cuda version too.
OpenCL version seems to work fine as I did a quick test .... 0 errors

Those errors should probably not be there. I just tried that test and got 0.

In cuda or opencl version?
My guess is that Fermi is the last architecture that works fine with that cuda build.

I used to recommend these tools back in the day, no wonder they are not advertised anymore on folding@home

 

[Ruyki]

In cuda or opencl version?

I get no errors in either one of those tests.
This is with fermi card.

 

[BiggieShady]

I get no errors in either one of those tests.
This is with fermi card.

Good to know. Thanks, gotta stop recommending that old cuda build for Keplers and Maxwells

 

[Psit]

So i guess we are back to sq zero?