1. Welcome to TechPowerUp Forums, Guest! Please check out our forum guidelines for info related to our community.

Windows 10 Process-Termination Bug Slows Down Mighty 24-Core System to a Crawl

Discussion in 'News' started by R-T-B, Jul 14, 2017.

  1. R-T-B

    R-T-B

    Joined:
    Aug 20, 2007
    Messages:
    6,598 (1.82/day)
    Thanks Received:
    5,898
    So, you work for Google. Awesome, right? Yeah. You know what else is awesome? Your 24-Core, 48-thread Intel build system with 64 GBs of ram and a nice SSD. Life is good man. So, you've done your code work for the day on Chrome, because that's what you do, remember? (Yeah, that's right, it's awesome). Before you go off to collect your google-check, you click "compile" and expect a speedy result from your wicked fast system.

    Only you don't get it... Instead, your system comes grinding to a lurching halt, and mouse movement becomes difficult. Fighting against what appears to be an impending system crash, you hit your trusty "CTRL-ALT-DELETE" and bring up task manager... to find only 50% CPU/RAM utilization. Why then, was everything stopping?

    If you would throw up your arms and walk out of the office, this is why you don't work for Google. For Google programmer Bruce Dawson, there was only one logical way to handle this: "So I did what I always do - I grabbed an ETW trace and analyzed it. The result was the discovery of a serious process-destruction performance bug in Windows 10."

    [​IMG]

    This is an excerpt from a long, detailed blog post by Bruce titled "24-core CPU and I can't move my mouse" on his Wordpress blog randomascii. In it, he details a serious new bug that is only present in Windows 10 (not other versions). Process destruction appears to be serialized.

    What does that mean, exactly? It means when a process "dies" or closes, it must go through a single thread to handle this. In this critical part of the OS which every process must eventually partake in, Windows 10 is actually single threaded.

    To be fair, this is not a normal issue an end user would encounter. But developers often spawn lots of processes and close them just as often. They use high-end multi-core CPUs to speed this along. Bruce notes that in his case, his 24-core CPU only made things worse, as it actually caused the build process to spawn more build processes, and thus, even more had to close. And because they all go through the same single threaded queue, the OS grinds to a halt during this operation, and performance peak is never realized.

    As for whether this is a big bug if you aren't a developer: Well that's up for debate. Certainly not directly, I'd wager, but as a former user of OS/2 and witness to Microsoft's campaign against it back in the day, I can't help but be reminded of Microsoft FUD surrounding OS/2's SIQ issue that persisted even years after it had been fixed. Does this not feel somewhat like sweet, sweet karma for MS from my perspective? Maybe, but honestly, that doesn't help anyone.

    Hopefully a fix will be out soon, and unlike the OS/2 days, the memory of this bug will be short lived.

    Source: randomascii Wordpress Blog
     
    Last edited by a moderator: Jul 14, 2017
    Kursah, qubit, kruk and 3 others say thanks.
  2. P4-630

    P4-630 The Way It's Meant to be Played

    Joined:
    Jan 5, 2006
    Messages:
    8,310 (1.97/day)
    Thanks Received:
    7,917
    Location:
    Vinewood
    [​IMG]
     
    remixedcat, EntropyZ, qubit and 13 others say thanks.
    10 Year Member at TPU
  3. eidairaman1

    eidairaman1 The Exiled Airman

    Joined:
    Jul 2, 2007
    Messages:
    17,918 (4.87/day)
    Thanks Received:
    4,124
    Win10=MacOS, "crash differently"
     
    Prima.Vera says thanks.
    10 Year Member at TPU
  4. xkm1948

    xkm1948

    Joined:
    Mar 18, 2008
    Messages:
    2,076 (0.61/day)
    Thanks Received:
    1,141
    Who would only pair 64GB of RAM on a 24core server grade system? You will need at least 256GB.

    Also you would think they should be using Linux for their work.
     
  5. notb

    Joined:
    Jun 28, 2016
    Messages:
    675 (1.72/day)
    Thanks Received:
    177
    Location:
    Warsaw, Poland
    Why does one *need* at least 256GB for his 24-core CPU?
    Because? :)
     
    EarthDog and Prima.Vera say thanks.
  6. R-T-B

    R-T-B

    Joined:
    Aug 20, 2007
    Messages:
    6,598 (1.82/day)
    Thanks Received:
    5,898
    Not for all workloads, obviously.
     
  7. cdawall

    cdawall where the hell are my stars

    Joined:
    Jul 23, 2006
    Messages:
    26,006 (6.47/day)
    Thanks Received:
    7,111
    Location:
    Houston
    My 16 core 32 thread server only has 32gb of ram...it doesn't need more for its tasking.
     
    Dethroy, Kursah, Solaris17 and 3 others say thanks.
    10 Year Member at TPU
  8. RejZoR

    RejZoR

    Joined:
    Oct 2, 2004
    Messages:
    11,374 (2.43/day)
    Thanks Received:
    5,188
    Location:
    Europe/Slovenia
    This is what happens when you buy Intel instead of AMD :p
     
    Countryside, yotano211 and KarymidoN say thanks.
    10 Year Member at TPU
  9. R-T-B

    R-T-B

    Joined:
    Aug 20, 2007
    Messages:
    6,598 (1.82/day)
    Thanks Received:
    5,898
    It happens to any brand.
     
    Prima.Vera, KarymidoN and xkm1948 say thanks.
  10. trparky

    trparky

    Joined:
    Mar 6, 2017
    Messages:
    466 (3.28/day)
    Thanks Received:
    135
    Is this something that can be fixed?
     
  11. springs113

    Joined:
    May 24, 2007
    Messages:
    883 (0.24/day)
    Thanks Received:
    212
    Location:
    Florida
    Come on man, where's your sense of humor? I think of all ppl on this forum you would have one on his comment, you know he's being sarcastic.
     
    10 Year Member at TPU
  12. R-T-B

    R-T-B

    Joined:
    Aug 20, 2007
    Messages:
    6,598 (1.82/day)
    Thanks Received:
    5,898
    Maybe I just failed to see the sarcasm from a guy with an Intel CPU. :p

    Or I woke up grumpy because my internet is only now back after comcast had me netless for almost a week. Take your pick. ;)

    Yes. It's not even broken in 7.
     
  13. rtwjunkie

    rtwjunkie PC Gaming Enthusiast

    Joined:
    Jul 25, 2008
    Messages:
    9,006 (2.74/day)
    Thanks Received:
    12,605
    Location:
    Louisiana -Laissez les bons temps rouler!
    Holy sh#t man!! Hopefully they credit you that lost time.
     
  14. FordGT90Concept

    FordGT90Concept "I go fast!1!11!1!"

    Joined:
    Oct 13, 2008
    Messages:
    20,267 (6.32/day)
    Thanks Received:
    9,430
    Location:
    IA, USA
    Looks to me like the problem is in Chrome's build system. It's likely creating 24 threads because it can, not because it should. Ironic that he calls it "smart" because that's pretty dumb. That said, Microsoft will likely be able to fix it seeing how it is apparently a new issue in NT 6.2, 6.3, or 10.0.
     
    Crunching for Team TPU
  15. trparky

    trparky

    Joined:
    Mar 6, 2017
    Messages:
    466 (3.28/day)
    Thanks Received:
    135
    Then what the hell changed deep inside the kernel between Windows 7 and Windows 10 that would break something like this?
     
  16. FordGT90Concept

    FordGT90Concept "I go fast!1!11!1!"

    Joined:
    Oct 13, 2008
    Messages:
    20,267 (6.32/day)
    Thanks Received:
    9,430
    Location:
    IA, USA
    Telemetry?
     
    remixedcat and WaroDaBeast say thanks.
    Crunching for Team TPU
  17. Aquinus

    Aquinus Resident Wat-man

    Joined:
    Jan 28, 2012
    Messages:
    10,102 (5.03/day)
    Thanks Received:
    5,157
    Location:
    Concord, NH
    You would be horrified by how much stuff from NT 4 still exists in Windows 10.
    How many more threads do you think are running at a given time now on a modern machine running Windows 10 versus a machine running Windows 7 from several years ago. You would be amazed at how many processes and threads are going on all at once.
     
    eidairaman1 says thanks.
  18. FordGT90Concept

    FordGT90Concept "I go fast!1!11!1!"

    Joined:
    Oct 13, 2008
    Messages:
    20,267 (6.32/day)
    Thanks Received:
    9,430
    Location:
    IA, USA
    It's CTRL+SHIFT+ESC.

    I'm at 110 processes and 1843 threads at 100% CPU load. Garbage collection does take some time. And let's be honest, no one in their right mind should ever be running 1000 processes. Fewer processes with lots of threads is easier to garbage collect.
     
    Crunching for Team TPU
  19. Prima.Vera

    Prima.Vera

    Joined:
    Sep 15, 2011
    Messages:
    4,012 (1.87/day)
    Thanks Received:
    969
    I learned something new today...:toast:
     
    radusorin, notb and R-T-B say thanks.
  20. HopelesslyFaithful

    Joined:
    Oct 7, 2013
    Messages:
    229 (0.16/day)
    Thanks Received:
    41
    what a shocker...as i keep saying most things in windows is single thread....just like most programs due to shotty programming.

    Plus...use Win7....Win 10 blows on so many level...another shocker.

    I have 124 proccesses and 2308 threads......

    I have seen 200 processes on my PC in the past. Main PC down so I am on server waiting for RMA of MB so server doesnt have all my extra programs installed.
     
  21. Melvis

    Melvis

    Joined:
    Mar 18, 2008
    Messages:
    4,359 (1.28/day)
    Thanks Received:
    971
    Location:
    Australia
    Could this be a reason thread ripper benchmarks the other day was done on Windows 8?
     
    eidairaman1 says thanks.
  22. R-T-B

    R-T-B

    Joined:
    Aug 20, 2007
    Messages:
    6,598 (1.82/day)
    Thanks Received:
    5,898
    They might... depends on how much of the screwup they lay on PSE for knocking a bunch of things offline during pole replacement. Forest internet is... different.

    It is smart assuming you want maxumum build throughput. It doesn't assume the computer will be doing other heavy tasks during the build, but the GUI and other processes should not suffer solely because it literally can't close things fast enough.
     
  23. FordGT90Concept

    FordGT90Concept "I go fast!1!11!1!"

    Joined:
    Oct 13, 2008
    Messages:
    20,267 (6.32/day)
    Thanks Received:
    9,430
    Location:
    IA, USA
    So...
    ...crosses me as a bug in "gomacc.exe" whatever that is. Literally the only reference to it is articles related to this blog post. "Several processes?" Perhaps they're fighting each other for resources locks. Sure, NtGdiCloseProcess is the call that's actually closing processes but the bug doesn't originate there. I'm not sure I'd even call it a bug at this point. More like gomacc.exe shoddy programming is exposing a low level lock that's cleaning up higher level stupidity.

    Depending on why the locks are being used, it might not be possible to remove them...or cause a larger performance problem elsewhere as a queue backs up.

    Garbage collection/process closure is likely reflected by the amount of memory the process uses. He says he's using less than half of 64 GiB of RAM. If his compiler software is using, say 20-30 GiB of RAM and it doesn't properly scrub itself as it goes, it's very possible that the operating system has to find 20-30 GiB of memory it is using and reallocate it to free.


    They claim to have reported it to Microsoft. Author should have waited for a response before going to the internet. Wouldn't be surprised if Microsoft points out a flaw in the program Google needs to fix and doesn't update Windows at all.


    Note: a single lock can cause a cascade of higher level threads to wait. Example, if you have 100 threads that read a single locked variable in another thread, those 100 threads are halted until the lock clears. Application is still multithreaded; it just doesn't act like it.

    gomacc.exe could easily be locking a variable that Windows needs access to in order to close the process.
     
    Last edited: Jul 15, 2017
    Crunching for Team TPU
  24. trparky

    trparky

    Joined:
    Mar 6, 2017
    Messages:
    466 (3.28/day)
    Thanks Received:
    135
    Another Windows 10 hater.
     
  25. HopelesslyFaithful

    Joined:
    Oct 7, 2013
    Messages:
    229 (0.16/day)
    Thanks Received:
    41
    i googled gomacc.exe and nothing really comes up for it.

    for good reasons. (I dont like 7 either...plenty to hate but its the best version out for my needs)
     

Currently Active Users Viewing This Thread: 1 (0 members and 1 guest)