Bug in AMD EPYC "Rome" Processors Puts Them to Sleep After 34 Months of Uptime

[Fouquin]

AMD recently published an errata for their second generation EPYC processors based on Zen 2 which states that, "A core will fail to exit CC6 after about 1044 days after the last system reset." 1044 days is roughly 34 months, or just shy of 3 years of total uptime, and is actually an over estimate according to some sysadmin sleuths on Reddit and Twitter that did the math and discovered the actual time is 1042 days and 12 hours. The problem occurs because the CPU REFCLK counts 10ns ticks in a 54-bit signed integer, and if you count just over 9 quadrillion of these ticks you get the resulting overflow at 1042.4999 days. Once this overflow occurs the cores are stuck forever in a zombie state, and will not take any external interrupt requests. Well, forever until you flip the power switch off and back on again, which will reset the counter.

It's certainly impressive that this problem was discovered at all, as it suggests that more than a single system has been running for almost three years straight without a single restart. Though this does put EPYC "Rome" out of the running for any possible awards for longest running systems, it may serve as a reminder to initiate system updates or patches for other vulnerabilities that have been discovered in the four years since that generation of processor were first launched. AMD does not plan to issue any fix for the CC6 bug, instead recommending that administrators disable CC6 to avoid the cores entering the zombified state, or simply initiating a restart every once in awhile before the time limit expires.



View at TechPowerUp Main Site | Source

 

[natr0n]

reminds me of y2k times

 

[Dirt Chip]

Bug Feature in AMD EPYC "Rome" Processors Puts Them to Sleep After 34 Months of Uptime​

 

[Easo]

While the bug is stupid and should not have happened 3 years of uptime gives me shivers.
Patch and reboot your damn stuff, FFS.

 

[john_]

I read somewhere some comments about Linux being able to update itself without rebooting, so there it does present a kind of a problem if someone needs to have a non stop server running until the Apocalypse.

 

[zmeul]

While the bug is stupid and should not have happened 3 years of uptime gives me shivers.
Patch and reboot your damn stuff, FFS.

live patching in GNU LInux exists, the servers don't need to be rebooted

 

[R0H1T]

Yeah well it's a good idea rebooting your damn systems at least once in 3 years. Though depending on what they're used for, or shared by, it won't be easy.

reminds me of y2k times

You mean the end times? Or 2012

Nostra, Incans(?) & Mayans sure did a number on millions at the time

 

[natr0n]

Yeah well it's a good idea rebooting your damn systems at least once in 3 years. Though depending on what they're used for, or shared by, it won't be easy.


You mean the end times? Or 2012

Nostra, Incans(?) & Mayans sure did a number on millions at the time

year 2000 bios bug/limitation with older pc

 

[R0H1T]

Yes I meant there were also lots of weird predictions around those years, like I mentioned 99/2k & 2012 recently. I wonder when's the next world ending event supposed to come!

 

[natr0n]

Yes I meant there were also lots of weird predictions around those years, like I mentioned 99/2k & 2012 recently. I wonder when's the next world ending event supposed to come!

Well no more world actually ending stuff. Lots of plaques and shit according to prophesy and what a persons beliefs are. If waters turn to blood I would panic if I didnt know about it.

 

[stanleyipkiss]

...or you could just disable sleep state and go about your day.

 

[Wirko]

Yes I meant there were also lots of weird predictions around those years, like I mentioned 99/2k & 2012 recently. I wonder when's the next world ending event supposed to come!

That's easy to answer, unless IBM, Intel and everyone else rip the remaining 32-bit abilities out of their processors by then.

Year 2038 problem - Wikipedia

en.m.wikipedia.org

I read somewhere some comments about Linux being able to update itself without rebooting, so there it does present a kind of a problem if someone needs to have a non stop server running until the Apocalypse.

To reach that goal, you better have servers with hot-swappable CPUs too. I don't know much about that ability but Wikipedia says it's "common". So you pull out a CPU and put it back in, and hopefully that timer will be reset.

Hot swapping - Wikipedia

en.m.wikipedia.org

 

[P4-630]

Rest in peace zombies!...

 

[Zubasa]

...or you could just disable sleep state and go about your day.

In fact many server boards have if off by default.

 

[Nostras]

In fact many server boards have if off by default.

Why this isn't such a widespread problem... Because of this.
Restarting your server every once in a while is dumb, there are many scenarios where rebooting is undesired.
Why CC6 isn't off by default on servers however...

 

[pavle]

AMD sleeping on the job again...

 

[lemonadesoda]

Fascinating read at https://en.m.wikipedia.org/wiki/Leap_second

I’m amazed that such a modern processor as EPYC has an n bit limit clock problem. Where the design teams asleep?

since EPYC is a server processor, rebooting should not have to be part of the standard operating procedures, and i can imagine many use case scenarios where this clock problem can cause chaos, esp, futures markets.

 

[Dr. Dro]

I’m amazed that such a modern processor as EPYC has an n bit limit clock problem. Where the design teams asleep?

I wouldn't be surprised if Xeon had a similar issue, just on a larger (much larger?) time scale.

I bet that the exact same issue would manifest on a consumer-grade Ryzen Threadripper processor, too, maybe even the socket AM4 counterparts.

Other hand, this is some proper uptime...

Reddit - Dive into anything

redd.it

 

[SomeOne99h]

Using Bing AI Image creation with "AMD CPU, transforming to a zombie, depth, 3D", I got these four drawings :

 

[TumbleGeorge]

Nothing, just planned problems that, when accumulated over time, will force you to buy new hardware. Not a total wreck where you can sue them for compensation even after the product's warranty is over, but annoying.

 

[eidairaman1]

Engineer

Fascinating read at https://en.m.wikipedia.org/wiki/Leap_second

I’m amazed that such a modern processor as EPYC has an n bit limit clock problem. Where the design teams asleep?

since EPYC is a server processor, rebooting should not have to be part of the standard operating procedures, and i can imagine many use case scenarios where this clock problem can cause chaos, esp, futures markets.

Engineers aren't perfect

 

[JAKra]

"It's Not a Bug, It's a Feature."
In case of AI uprising, all machines based on EPIC would automatically go to sleep after 34 months of uptime.
Humanity saved thanks to this "feature"!

 

[TheoneandonlyMrK]

Why this isn't such a widespread problem... Because of this.
Restarting your server every once in a while is dumb, there are many scenarios where rebooting is undesired.
Why CC6 isn't off by default on servers however...

While I agree , if you're not performing some sort of upgrade, maintenance or check, in three years of a server part in a normal IT department I would be surprised, filters need cleaning and three years is quite long to use the same server hardware ( considers own IT DPT), maybe though.

It's a surprising oversight for a server part.

@JAKra get a job a Cyberdyne, these firm's need this kind of thought

 

[thesmokingman]

Yea, this is more about lazy admins. If you hit the bug you get fired lol.

 

[AusWolf]

Even computers get tired sometimes.