Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl

[R-T-B]

So, you work for Google. Awesome, right? Yeah. You know what else is awesome? Your 24-Core, 48-thread Intel build system with 64 GBs of ram and a nice SSD. Life is good man. So, you've done your code work for the day on Chrome, because that's what you do, remember? (Yeah, that's right, it's awesome). Before you go off to collect your google-check, you click "compile" and expect a speedy result from your wicked fast system.

Only you don't get it... Instead, your system comes grinding to a lurching halt, and mouse movement becomes difficult. Fighting against what appears to be an impending system crash, you hit your trusty "CTRL-ALT-DELETE" and bring up task manager... to find only 50% CPU/RAM utilization. Why then, was everything stopping?

If you would throw up your arms and walk out of the office, this is why you don't work for Google. For Google programmer Bruce Dawson, there was only one logical way to handle this: "So I did what I always do - I grabbed an ETW trace and analyzed it. The result was the discovery of a serious process-destruction performance bug in Windows 10."




This is an excerpt from a long, detailed blog post by Bruce titled "24-core CPU and I can't move my mouse" on his Wordpress blog randomascii. In it, he details a serious new bug that is only present in Windows 10 (not other versions). Process destruction appears to be serialized.

What does that mean, exactly? It means when a process "dies" or closes, it must go through a single thread to handle this. In this critical part of the OS which every process must eventually partake in, Windows 10 is actually single threaded.

To be fair, this is not a normal issue an end user would encounter. But developers often spawn lots of processes and close them just as often. They use high-end multi-core CPUs to speed this along. Bruce notes that in his case, his 24-core CPU only made things worse, as it actually caused the build process to spawn more build processes, and thus, even more had to close. And because they all go through the same single threaded queue, the OS grinds to a halt during this operation, and performance peak is never realized.

As for whether this is a big bug if you aren't a developer: Well that's up for debate. Certainly not directly, I'd wager, but as a former user of OS/2 and witness to Microsoft's campaign against it back in the day, I can't help but be reminded of Microsoft FUD surrounding OS/2's SIQ issue that persisted even years after it had been fixed. Does this not feel somewhat like sweet, sweet karma for MS from my perspective? Maybe, but honestly, that doesn't help anyone.

Hopefully a fix will be out soon, and unlike the OS/2 days, the memory of this bug will be short lived.

View at TechPowerUp Main Site

 

[P4-630]

 

[eidairaman1]

Win10=MacOS, "crash differently"

 

[Deleted member 50521]

Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

Also you would think they should be using Linux for their work.

 

[notb]

Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

Why does one *need* at least 256GB for his 24-core CPU?

Also you would think they should be using Linux for their work.

Because?

 

[R-T-B]

Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

Not for all workloads, obviously.

 

[cdawall]

Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

Also you would think they should be using Linux for their work.

My 16 core 32 thread server only has 32gb of ram...it doesn't need more for its tasking.

 

[RejZoR]

This is what happens when you buy Intel instead of AMD

 

[R-T-B]

This is what happens when you buy Intel instead of AMD

It happens to any brand.

 

[trparky]

Is this something that can be fixed?

 

[springs113]

It happens to any brand.

Come on man, where's your sense of humor? I think of all ppl on this forum you would have one on his comment, you know he's being sarcastic.

 

[R-T-B]

Come on man, where's your sense of humor? I think of all ppl on this forum you would have one on his comment, you know he's being sarcastic.

Maybe I just failed to see the sarcasm from a guy with an Intel CPU.

Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick.

Is this something that can be fixed?

Yes. It's not even broken in 7.

 

[rtwjunkie]

Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick.

Holy sh#t man!! Hopefully they credit you that lost time.

 

[FordGT90Concept]

Looks to me like the problem is in Chrome's build system. It's likely creating 24 threads because it can, not because it should. Ironic that he calls it "smart" because that's pretty dumb. That said, Microsoft will likely be able to fix it seeing how it is apparently a new issue in NT 6.2, 6.3, or 10.0.

 

[trparky]

Yes. It's not even broken in 7.

Then what the hell changed deep inside the kernel between Windows 7 and Windows 10 that would break something like this?

 

[FordGT90Concept]

Telemetry?

 

[Aquinus]

You would be horrified by how much stuff from NT 4 still exists in Windows 10.

Then what the hell changed deep inside the kernel between Windows 7 and Windows 10 that would break something like this?

How many more threads do you think are running at a given time now on a modern machine running Windows 10 versus a machine running Windows 7 from several years ago. You would be amazed at how many processes and threads are going on all at once.

 

[FordGT90Concept]

...you hit your trusty "CTRL-ALT-DELETE" and bring up task manager...

It's CTRL+SHIFT+ESC.

How many more threads do you think are running at a given time now on a modern machine running Windows 10 versus a machine running Windows 7 from several years ago. You would be amazed at how many processes and threads are going on all at once.

I'm at 110 processes and 1843 threads at 100% CPU load. Garbage collection does take some time. And let's be honest, no one in their right mind should ever be running 1000 processes. Fewer processes with lots of threads is easier to garbage collect.

 

[Prima.Vera]

It's CTRL+SHIFT+ESC.

I learned something new today...

 

[HopelesslyFaithful]

what a shocker...as i keep saying most things in windows is single thread....just like most programs due to shotty programming.

Plus...use Win7....Win 10 blows on so many level...another shocker.

I have 124 proccesses and 2308 threads......

I have seen 200 processes on my PC in the past. Main PC down so I am on server waiting for RMA of MB so server doesnt have all my extra programs installed.

 

[Melvis]

Could this be a reason thread ripper benchmarks the other day was done on Windows 8?

 

[R-T-B]

Holy sh#t man!! Hopefully they credit you that lost time.

They might... depends on how much of the screwup they lay on PSE for knocking a bunch of things offline during pole replacement. Forest internet is... different.

Looks to me like the problem is in Chrome's build system. It's likely creating 24 threads because it can, not because it should. Ironic that he calls it "smart" because that's pretty dumb. That said, Microsoft will likely be able to fix it seeing how it is apparently a new issue in NT 6.2, 6.3, or 10.0.

It is smart assuming you want maxumum build throughput. It doesn't assume the computer will be doing other heavy tasks during the build, but the GUI and other processes should not suffer solely because it literally can't close things fast enough.

 

[FordGT90Concept]

So...

I had to keep going because most of the processes were releasing the lock after holding it for just a few microseconds. But eventually I found several processes (the gomacc.exe processes) that looked like they were holding the lock for a few hundred microseconds. Or, at least, they were readied by somebody holding the lock and then a few hundred microseconds later they readied somebody else by releasing the lock. These processes were all releasing the lock from within NtGdiCloseProcess.

...crosses me as a bug in "gomacc.exe" whatever that is. Literally the only reference to it is articles related to this blog post. "Several processes?" Perhaps they're fighting each other for resources locks. Sure, NtGdiCloseProcess is the call that's actually closing processes but the bug doesn't originate there. I'm not sure I'd even call it a bug at this point. More like gomacc.exe shoddy programming is exposing a low level lock that's cleaning up higher level stupidity.

Depending on why the locks are being used, it might not be possible to remove them...or cause a larger performance problem elsewhere as a queue backs up.

Garbage collection/process closure is likely reflected by the amount of memory the process uses. He says he's using less than half of 64 GiB of RAM. If his compiler software is using, say 20-30 GiB of RAM and it doesn't properly scrub itself as it goes, it's very possible that the operating system has to find 20-30 GiB of memory it is using and reallocate it to free.


They claim to have reported it to Microsoft. Author should have waited for a response before going to the internet. Wouldn't be surprised if Microsoft points out a flaw in the program Google needs to fix and doesn't update Windows at all.


Note: a single lock can cause a cascade of higher level threads to wait. Example, if you have 100 threads that read a single locked variable in another thread, those 100 threads are halted until the lock clears. Application is still multithreaded; it just doesn't act like it.

gomacc.exe could easily be locking a variable that Windows needs access to in order to close the process.

 

[trparky]

Use Win7.... Win 10 blows on so many level...another shocker.

Another Windows 10 hater.

 

[HopelesslyFaithful]

So...

...crosses me as a bug in "gomacc.exe" whatever that is. Literally the only reference to it is articles related to this blog post. "Several processes?" Perhaps they're fighting each other for resources locks. Sure, NtGdiCloseProcess is the call that's actually closing processes but the bug doesn't originate there. I'm not sure I'd even call it a bug at this point. More like gomacc.exe shoddy programming is exposing a low level lock that's cleaning up higher level stupidity.

Depending on why the locks are being used, it might not be possible to remove them...or cause a larger performance problem elsewhere as a queue backs up.

Garbage collection/process closure is likely reflected by the amount of memory the process uses. He says he's using less than half of 64 GiB of RAM. If his compiler software is using, say 20-30 GiB of RAM and it doesn't properly scrub itself as it goes, it's very possible that the operating system has to find 20-30 GiB of memory it is using and reallocate it to free.


They claim to have reported it to Microsoft. Author should have waited for a response before going to the internet. Wouldn't be surprised if Microsoft points out a flaw in the program Google needs to fix and doesn't update Windows at all.

i googled gomacc.exe and nothing really comes up for it.

Another Windows 10 hater.

for good reasons. (I dont like 7 either...plenty to hate but its the best version out for my needs)