Friday, July 14th 2017

Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl

So, you work for Google. Awesome, right? Yeah. You know what else is awesome? Your 24-Core, 48-thread Intel build system with 64 GBs of ram and a nice SSD. Life is good man. So, you've done your code work for the day on Chrome, because that's what you do, remember? (Yeah, that's right, it's awesome). Before you go off to collect your google-check, you click "compile" and expect a speedy result from your wicked fast system.

Only you don't get it... Instead, your system comes grinding to a lurching halt, and mouse movement becomes difficult. Fighting against what appears to be an impending system crash, you hit your trusty "CTRL-ALT-DELETE" and bring up task manager... to find only 50% CPU/RAM utilization. Why then, was everything stopping?

If you would throw up your arms and walk out of the office, this is why you don't work for Google. For Google programmer Bruce Dawson, there was only one logical way to handle this: "So I did what I always do - I grabbed an ETW trace and analyzed it. The result was the discovery of a serious process-destruction performance bug in Windows 10."
This is an excerpt from a long, detailed blog post by Bruce titled "24-core CPU and I can't move my mouse" on his Wordpress blog randomascii. In it, he details a serious new bug that is only present in Windows 10 (not other versions). Process destruction appears to be serialized.

What does that mean, exactly? It means when a process "dies" or closes, it must go through a single thread to handle this. In this critical part of the OS which every process must eventually partake in, Windows 10 is actually single threaded.

To be fair, this is not a normal issue an end user would encounter. But developers often spawn lots of processes and close them just as often. They use high-end multi-core CPUs to speed this along. Bruce notes that in his case, his 24-core CPU only made things worse, as it actually caused the build process to spawn more build processes, and thus, even more had to close. And because they all go through the same single threaded queue, the OS grinds to a halt during this operation, and performance peak is never realized.

As for whether this is a big bug if you aren't a developer: Well that's up for debate. Certainly not directly, I'd wager, but as a former user of OS/2 and witness to Microsoft's campaign against it back in the day, I can't help but be reminded of Microsoft FUD surrounding OS/2's SIQ issue that persisted even years after it had been fixed. Does this not feel somewhat like sweet, sweet karma for MS from my perspective? Maybe, but honestly, that doesn't help anyone.

Hopefully a fix will be out soon, and unlike the OS/2 days, the memory of this bug will be short lived.Source: randomascii Wordpress Blog
Add your own comment

107 Comments on Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl

#1
P4-630
The Way It's Meant to be Played
Posted on Reply
#2
eidairaman1
The Exiled Airman
Win10=MacOS, "crash differently"
Posted on Reply
#3
xkm1948
Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

Also you would think they should be using Linux for their work.
Posted on Reply
#4
notb
xkm1948 said:
Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.
Why does one *need* at least 256GB for his 24-core CPU?
Also you would think they should be using Linux for their work.
Because? :-)
Posted on Reply
#5
R-T-B
xkm1948 said:
Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.
Not for all workloads, obviously.
Posted on Reply
#6
cdawall
where the hell are my stars
xkm1948 said:
Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

Also you would think they should be using Linux for their work.
My 16 core 32 thread server only has 32gb of ram...it doesn't need more for its tasking.
Posted on Reply
#7
RejZoR
This is what happens when you buy Intel instead of AMD :P
Posted on Reply
#8
R-T-B
RejZoR said:
This is what happens when you buy Intel instead of AMD :p
It happens to any brand.
Posted on Reply
#9
trparky
Is this something that can be fixed?
Posted on Reply
#10
springs113
R-T-B said:
It happens to any brand.
Come on man, where's your sense of humor? I think of all ppl on this forum you would have one on his comment, you know he's being sarcastic.
Posted on Reply
#11
R-T-B
springs113 said:
Come on man, where's your sense of humor? I think of all ppl on this forum you would have one on his comment, you know he's being sarcastic.
Maybe I just failed to see the sarcasm from a guy with an Intel CPU. :p

Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick. ;)

trparky said:
Is this something that can be fixed?
Yes. It's not even broken in 7.
Posted on Reply
#12
rtwjunkie
PC Gaming Enthusiast
R-T-B said:
Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick. ;)
Holy sh#t man!! Hopefully they credit you that lost time.
Posted on Reply
#13
FordGT90Concept
"I go fast!1!11!1!"
Looks to me like the problem is in Chrome's build system. It's likely creating 24 threads because it can, not because it should. Ironic that he calls it "smart" because that's pretty dumb. That said, Microsoft will likely be able to fix it seeing how it is apparently a new issue in NT 6.2, 6.3, or 10.0.
Posted on Reply
#14
trparky
R-T-B said:
Yes. It's not even broken in 7.
Then what the hell changed deep inside the kernel between Windows 7 and Windows 10 that would break something like this?
Posted on Reply
#16
Aquinus
Resident Wat-man
You would be horrified by how much stuff from NT 4 still exists in Windows 10.
trparky said:
Then what the hell changed deep inside the kernel between Windows 7 and Windows 10 that would break something like this?
How many more threads do you think are running at a given time now on a modern machine running Windows 10 versus a machine running Windows 7 from several years ago. You would be amazed at how many processes and threads are going on all at once.
Posted on Reply
#17
FordGT90Concept
"I go fast!1!11!1!"
R-T-B said:
...you hit your trusty "CTRL-ALT-DELETE" and bring up task manager...
It's CTRL+SHIFT+ESC.

Aquinus said:
How many more threads do you think are running at a given time now on a modern machine running Windows 10 versus a machine running Windows 7 from several years ago. You would be amazed at how many processes and threads are going on all at once.
I'm at 110 processes and 1843 threads at 100% CPU load. Garbage collection does take some time. And let's be honest, no one in their right mind should ever be running 1000 processes. Fewer processes with lots of threads is easier to garbage collect.
Posted on Reply
#18
Prima.Vera
FordGT90Concept said:
It's CTRL+SHIFT+ESC.
I learned something new today...:toast:
Posted on Reply
#19
HopelesslyFaithful
what a shocker...as i keep saying most things in windows is single thread....just like most programs due to shotty programming.

Plus...use Win7....Win 10 blows on so many level...another shocker.

I have 124 proccesses and 2308 threads......

I have seen 200 processes on my PC in the past. Main PC down so I am on server waiting for RMA of MB so server doesnt have all my extra programs installed.
Posted on Reply
#20
Melvis
Could this be a reason thread ripper benchmarks the other day was done on Windows 8?
Posted on Reply
#21
R-T-B
rtwjunkie said:
Holy sh#t man!! Hopefully they credit you that lost time.
They might... depends on how much of the screwup they lay on PSE for knocking a bunch of things offline during pole replacement. Forest internet is... different.

FordGT90Concept said:
Looks to me like the problem is in Chrome's build system. It's likely creating 24 threads because it can, not because it should. Ironic that he calls it "smart" because that's pretty dumb. That said, Microsoft will likely be able to fix it seeing how it is apparently a new issue in NT 6.2, 6.3, or 10.0.
It is smart assuming you want maxumum build throughput. It doesn't assume the computer will be doing other heavy tasks during the build, but the GUI and other processes should not suffer solely because it literally can't close things fast enough.
Posted on Reply
#22
FordGT90Concept
"I go fast!1!11!1!"
So...
I had to keep going because most of the processes were releasing the lock after holding it for just a few microseconds. But eventually I found several processes (the gomacc.exe processes) that looked like they were holding the lock for a few hundred microseconds. Or, at least, they were readied by somebody holding the lock and then a few hundred microseconds later they readied somebody else by releasing the lock. These processes were all releasing the lock from within NtGdiCloseProcess.
...crosses me as a bug in "gomacc.exe" whatever that is. Literally the only reference to it is articles related to this blog post. "Several processes?" Perhaps they're fighting each other for resources locks. Sure, NtGdiCloseProcess is the call that's actually closing processes but the bug doesn't originate there. I'm not sure I'd even call it a bug at this point. More like gomacc.exe shoddy programming is exposing a low level lock that's cleaning up higher level stupidity.

Depending on why the locks are being used, it might not be possible to remove them...or cause a larger performance problem elsewhere as a queue backs up.

Garbage collection/process closure is likely reflected by the amount of memory the process uses. He says he's using less than half of 64 GiB of RAM. If his compiler software is using, say 20-30 GiB of RAM and it doesn't properly scrub itself as it goes, it's very possible that the operating system has to find 20-30 GiB of memory it is using and reallocate it to free.


They claim to have reported it to Microsoft. Author should have waited for a response before going to the internet. Wouldn't be surprised if Microsoft points out a flaw in the program Google needs to fix and doesn't update Windows at all.


Note: a single lock can cause a cascade of higher level threads to wait. Example, if you have 100 threads that read a single locked variable in another thread, those 100 threads are halted until the lock clears. Application is still multithreaded; it just doesn't act like it.

gomacc.exe could easily be locking a variable that Windows needs access to in order to close the process.
Posted on Reply
#23
trparky
HopelesslyFaithful said:
Use Win7.... Win 10 blows on so many level...another shocker.
Another Windows 10 hater.
Posted on Reply
#24
HopelesslyFaithful
FordGT90Concept said:
So...

...crosses me as a bug in "gomacc.exe" whatever that is. Literally the only reference to it is articles related to this blog post. "Several processes?" Perhaps they're fighting each other for resources locks. Sure, NtGdiCloseProcess is the call that's actually closing processes but the bug doesn't originate there. I'm not sure I'd even call it a bug at this point. More like gomacc.exe shoddy programming is exposing a low level lock that's cleaning up higher level stupidity.

Depending on why the locks are being used, it might not be possible to remove them...or cause a larger performance problem elsewhere as a queue backs up.

Garbage collection/process closure is likely reflected by the amount of memory the process uses. He says he's using less than half of 64 GiB of RAM. If his compiler software is using, say 20-30 GiB of RAM and it doesn't properly scrub itself as it goes, it's very possible that the operating system has to find 20-30 GiB of memory it is using and reallocate it to free.


They claim to have reported it to Microsoft. Author should have waited for a response before going to the internet. Wouldn't be surprised if Microsoft points out a flaw in the program Google needs to fix and doesn't update Windows at all.
i googled gomacc.exe and nothing really comes up for it.

trparky said:
Another Windows 10 hater.
for good reasons. (I dont like 7 either...plenty to hate but its the best version out for my needs)
Posted on Reply
#25
trparky
HopelesslyFaithful said:
I dont like 7 either
Please don't tell me you like Windows XP. :fear:
Posted on Reply
Add your own comment