Saturday, June 3rd 2023
Bug in AMD EPYC "Rome" Processors Puts Them to Sleep After 34 Months of Uptime
AMD recently published an errata for their second generation EPYC processors based on Zen 2 which states that, "A core will fail to exit CC6 after about 1044 days after the last system reset." 1044 days is roughly 34 months, or just shy of 3 years of total uptime, and is actually an over estimate according to some sysadmin sleuths on Reddit and Twitter that did the math and discovered the actual time is 1042 days and 12 hours. The problem occurs because the CPU REFCLK counts 10ns ticks in a 54-bit signed integer, and if you count just over 9 quadrillion of these ticks you get the resulting overflow at 1042.4999 days. Once this overflow occurs the cores are stuck forever in a zombie state, and will not take any external interrupt requests. Well, forever until you flip the power switch off and back on again, which will reset the counter.
It's certainly impressive that this problem was discovered at all, as it suggests that more than a single system has been running for almost three years straight without a single restart. Though this does put EPYC "Rome" out of the running for any possible awards for longest running systems, it may serve as a reminder to initiate system updates or patches for other vulnerabilities that have been discovered in the four years since that generation of processor were first launched. AMD does not plan to issue any fix for the CC6 bug, instead recommending that administrators disable CC6 to avoid the cores entering the zombified state, or simply initiating a restart every once in awhile before the time limit expires.
Sources:
AMD, Tom's Hardware
It's certainly impressive that this problem was discovered at all, as it suggests that more than a single system has been running for almost three years straight without a single restart. Though this does put EPYC "Rome" out of the running for any possible awards for longest running systems, it may serve as a reminder to initiate system updates or patches for other vulnerabilities that have been discovered in the four years since that generation of processor were first launched. AMD does not plan to issue any fix for the CC6 bug, instead recommending that administrators disable CC6 to avoid the cores entering the zombified state, or simply initiating a restart every once in awhile before the time limit expires.
58 Comments on Bug in AMD EPYC "Rome" Processors Puts Them to Sleep After 34 Months of Uptime
One year+ uptime not unusual either.
You seem to be mistaking remote management chips with actual BIOS.
"Hey, I haven't restarted our server for 3 years and now it's acting weird."
"Huh? You what now? 3 years?"
:roll:
Naptime!
Definitely Errata, but also... highly unlikely to be a major issue for anyone.
It's in the first line of the news
It's an unexpected bug but also non-critical, since it's very very easy to plan something as simple as yearly maintenance, let alone 3 yearly
en.m.wikipedia.org/wiki/Erratum
A flaw in a processor design is also designated an erratum in tech jargon. It may or may not be correctable, and their severity can range from low to extreme, usually errata which are low severity such as this one are documented but no fix is issued or planned. In these cases, the chipmaker documents the problem and suggests a workaround.
Occasionally higher severity issues are also not fixed (such as Milan's USB stack reset problem), or the fixes come at a performance penalty (eg. Intel's fixes for speculative execution exploits). Very rarely, erratum of extreme severity result in designs being cancelled or permanently recalled. These rarely get past ES stage, but it can occur.