>The company reports that the 16-bit and 32-bit forms of RDSEED can return zero values, which may be mistakenly interpreted as a successful entropy read.
That's something that should have caught by a simple unit-test in pre-production.

Shadowized>The company reports that the 16-bit and 32-bit forms of RDSEED can return zero values, which may be mistakenly interpreted as a successful entropy read.
That's something that should have caught by a simple unit-test in pre-production.

This sounds like a corner case whose parameters were not exercised until someone noticed something on an actual system, simply because nobody really thought about it (i.e., it's an unknown unknown problem)

I feel like I can count with both hands the amount of times "dumb" things happened from corner cases that nobody thought about, but memory's a little hazy.

ShadowizedThat's something that should have caught by a simple unit-test in pre-production.

shuffy_losysThis sounds like a corner case whose parameters were not exercised until someone noticed something on an actual system, simply because nobody really thought about it (i.e., it's an unknown unknown problem)

I feel like I can count with both hands the amount of times "dumb" things happened from corner cases that nobody thought about, but memory's a little hazy.

It does seem to be a bit of a peculiar situation to reproduce the bug. +10% of the time is a lot, for that matter, but it might be that it's not a common situation? Plus, wouldn't it be more convenient to use the 64-bit variant in most cases anyway?

lore.kernel.org/lkml/20251016182107.3496116-1-gourry@gourry.net/

Under unknown architectural conditions, Zen5 chips running rdseed
can produce (val=0,CF=1) as a "random" result over 10% of the time
(when rdseed is successful). CF=1 indicates success, while val=0
is typically only produced when rdseed fails (CF=0).

This suggests there is an architectural issue which causes rdseed
to misclassify a failure as a success under unknown conditions.

This was reproduced reliably by launching 2-threads per available
core, 1-thread per for hamming on RDSEED, and 1-thread per core
collectively eating and hammering on ~90% of memory.

Fix was modeled after a different RDSEED issue in Zen2 Cyan Skillfish.

windwhirlIt does seem to be a bit of a peculiar situation to reproduce the bug. +10% of the time is a lot, for that matter, but it might be that it's not a common situation? Plus, wouldn't it be more convenient to use the 64-bit variant in most cases anyway?

Yeah, how many times does the thing you bolded happen in any practical sense? It sounds like someone was doing some "what if I did this" kind of experimentation or pen testing.

Honestly, this is bonehead level stupid to let slip out. But well, it's happened before. Some old discrete TPMs from infinion had a similar bug IIRC.

Is htis a repost? I think i read about that months ago.

R-T-BHonestly, this is bonehead level stupid to let slip out. But well, it's happened before. Some old discrete TPMs from infinion had a similar bug IIRC.

I agree. Companies that have been developing CPUs for decades have had plenty of time to develop comprehensive software capable of detecting bugs in their CPU designs, during the design or debugging phases.

If they haven't already developed software that performs a broad-extreme search for bugs in their CPUs, they should do it "to yesterday".

R-T-BHonestly, this is bonehead level stupid to let slip out. But well, it's happened before. Some old discrete TPMs from infinion had a similar bug IIRC.

NhonhoI agree. Companies that have been developing CPUs for decades have had plenty of time to develop comprehensive software capable of detecting bugs in their CPU designs, during the design or debugging phases.

If they haven't already developed software that performs a broad-extreme search for bugs in their CPUs, they should do it "to yesterday".

Here's a scary thought: do you know the Linux kernel doesn't have 100% test coverage? Their tests don't even come close to 100% coverage.

Or to put in another way, if you want absolute 100% test coverage, lets say we want to test every branching path the program has. Every if-statement doubles the amount of paths your program can take. You only need 64 if-statements to reach ~16 quintillion branching paths. I'm looking at a hobbyist project I'm doing and there's 144 if-statements, meaning there's at least 22,300,745,198,530,623,141,535,718,272,648,361,505,980,416 branching paths. Hardware is just hard-coded software, so you can bet you butt there's branching paths. Or even more basic: different logic states.

So, if we want to wait until the heat death of the universe for something to come out, most testing is just "is it doing the feature I need?" and some obvious limits.

shuffy_losysHere's a scary thought: do you know the Linux kernel doesn't have 100% test coverage? Their tests don't even come close to 100% coverage.

Or to put in another way, if you want absolute 100% test coverage, lets say we want to test every branching path the program has. Every if-statement doubles the amount of paths your program can take. You only need 64 if-statements to reach ~16 quintillion branching paths. I'm looking at a hobbyist project I'm doing and there's 144 if-statements, meaning there's at least 22,300,745,198,530,623,141,535,718,272,648,361,505,980,416 branching paths. Hardware is just hard-coded software, so you can bet you butt there's branching paths. Or even more basic: different logic states.

So, if we want to wait until the heat death of the universe for something to come out, most testing is just "is it doing the feature I need?" and some obvious limits.

Yes but at the same time this sounds like a very basic fail, frankly. It should never be doing this.

R-T-BYes but at the same time this sounds like a very basic fail, frankly. It should never be doing this.

No it doesn't. @windwhirl provided an excerpt from the mailing list that discovered the issue and how to reproduce it. Hammering an instruction while hammering memory does not sound like typical use case for anything outside of pen testing.

Does this mean AMD should hire more pen testers? Sure, I guess.

shuffy_losysHammering an instruction while hammering memory does not sound like typical use case for anything outside of pen testing.

Missed that, and indeed it does not.

Thanks for pointing it out.

shuffy_losysHere's a scary thought: do you know the Linux kernel doesn't have 100% test coverage? Their tests don't even come close to 100% coverage.

Or to put in another way, if you want absolute 100% test coverage, lets say we want to test every branching path the program has. Every if-statement doubles the amount of paths your program can take.

Code coverage only checks how much of the code has been touched, in whatever way, by the tests, it doesn't measure the testing of all possible states.

And in any case, if statements only double the amount of paths to the degree that they're nested

shuffy_losysYou only need 64 if-statements to reach ~16 quintillion branching paths. I'm looking at a hobbyist project I'm doing and there's 144 if-statements, meaning there's at least 22,300,745,198,530,623,141,535,718,272,648,361,505,980,416 branching paths.

Only if they're all nested into each other, in which case you certainly have some problem ...

shuffy_losysmost testing is just "is it doing the feature I need?" and some obvious limits.

Oh come on, testing is often very extensive, although it generally can't cover every possible condition that can occur

grupCode coverage only checks how much of the code has been touched, in whatever way, by the tests, it doesn't measure the testing of all possible states.

And this is how you get corner cases that could lead to issues.

In my case, I'm dealing with something at work where the original implementer was fixing a problem, but they didn't realize it caused other side effects because the thing they did broke the original logic of the application.

grupAnd in any case, if statements only double the amount of paths to the degree that they're nested

Only if they're all nested into each other, in which case you certainly have some problem ...

While I acknowledge my statement was oversimplified, it's not true that the code paths only double if there are nested statements. For example, assuming the values being checked are independent from each other, each if statement doubles the amount of code paths.

if (condition1) {
do_something();
}

if (condition2) {
do_something_else();
}

if (condition3) {
do_yet_another_thing();
}

grupOh come on, testing is often very extensive, although it generally can't cover every possible condition that can occur

It depends on what level of testing you're doing, and how they're doing it. A lot of times I've seen, the system level test is just all of the unit tests combined with some adjustments made.

I mean, you also kind of acknowledged it yourself:

Code coverage only checks how much of the code has been touched

shuffy_losysHere's a scary thought: do you know the Linux kernel doesn't have 100% test coverage? Their tests don't even come close to 100% coverage.

Test coverage is one of the most useless and misleading metrics of code testing. It only measures whether the code has been executed, not whether it caused any state change or if the logic is correct. Its only usefulness is when developing a single test, running it across a test suite will only result in users thinking "this code is covered by a test", which inevitably leads inexperienced developers and especially team leadership getting a false sense of the code quality and the need for improvements.

Nor should it be desired or prioritized to have "complete" test coverage. Usually people refer to unit tests (and sometimes integration tests etc.), which should only test a unit in isolation (external states/dependencies should be mocked). And such tests are usually only valuable for libraries or small sections of code, and only for the parts where there's an algorithm or some testable logic. There are a lot of code in the Linux kernel, a driver or any code for that matter that are just declarations and initializations, and there is no point in testing those in this way. I've seen enough unit tests that basically are either just a "write protection" of the code, or worse, will only fail if the CPU is unable to do math or the OS is unable to allocate memory. Such tests have absolutely no value. There are systems tests etc. which should cover those kinds of validations.

What should scare you is that "professional" developers (and probably most large software companies) rely on such metrics along with static analysis and all kinds of metrics to evaluate "code quality" and even "vulnerabilities". While perhaps 1% of the information is useful, the amount of time spent, the bureaucracy, and all the constraints only leads to more bloated, unmaintainable and garbage code. Unfortunately this BS is what people get taught in CS degrees these days, not deep levels of technical understanding.

Virtually all flaws in software that matters are skill issues, the rest are careless mistakes, which you will catch by having proper coding standards and procedures.

shuffy_losysOr to put in another way, if you want absolute 100% test coverage, lets say we want to test every branching path the program has. Every if-statement doubles the amount of paths your program can take. You only need 64 if-statements to reach ~16 quintillion branching paths. I'm looking at a hobbyist project I'm doing and there's 144 if-statements, meaning there's at least 22,300,745,198,530,623,141,535,718,272,648,361,505,980,416 branching paths.

It's true in terms of branching paths, but (probably) not in terms of possible logical states, as most of those branches (hopefully) don't directly interact with each other's state.

-----

To tie this back into the subject matter;
So many of you wonder why so many CPU bugs are uncaught?
Unlike what shuffy_losys claimed, hardware isn't just hardcoded software, there is also physics involved at this level. The vast majority of CPU bugs which sneaks their way into production silicon are due to timing issues, not logical issues. No matter how much they simulate the design beforehand, there will always the possibility of such flaws due to the production node's characteristics, the specific die design and just yield variation. (Unlike software, hardware transistors will have "undefined behavior" between cycles, and this is where most of such bugs and even stability issues comes from.) So even if the managed to simulate enough states, it still would not cover all of these cases.

With that being said, I firmly believe that all CPU designs could benefit from doing another stepping before mass production (and/or having more widely deployed engineering samples), as most of these launch with a few dozen known flaws, not to mention all the flaws they discover afterwards. The amount of "mitigations" in firmware or OS and libraries are often causing major performance regressions, quite often after the purchase of a product.

efikkanTest coverage is one of the most useless and misleading metrics of code testing. It only measures whether the code has been executed, not whether it caused any state change or if the logic is correct. Its only usefulness is when developing a single test, running it across a test suite will only result in users thinking "this code is covered by a test", which inevitably leads inexperienced developers and especially team leadership getting a false sense of the code quality and the need for improvements.

To clear something up, I may have used the term in a simplistic way because in my head I was thinking not just the test touched the code, but the test is exercising the code in various ways. So in my head, if you have a function that takes in a value and its behavior changes depending on certain values of the input, 100% coverage would be you didn't just test the valid, invalid, lower, and upper bounds of the input, you tested all possible values of the input.

efikkanSo many of you wonder why so many CPU bugs are uncaught?
Unlike what shuffy_losys claimed, hardware isn't just hardcoded software, there is also physics involved at this level. The vast majority of CPU bugs which sneaks their way into production silicon are due to timing issues, not logical issues. No matter how much they simulate the design beforehand, there will always the possibility of such flaws due to the production node's characteristics, the specific die design and just yield variation. (Unlike software, hardware transistors will have "undefined behavior" between cycles, and this is where most of such bugs and even stability issues comes from.) So even if the managed to simulate enough states, it still would not cover all of these cases.

With that being said, I firmly believe that all CPU designs could benefit from doing another stepping before mass production (and/or having more widely deployed engineering samples), as most of these launch with a few dozen known flaws, not to mention all the flaws they discover afterwards. The amount of "mitigations" in firmware or OS and libraries are often causing major performance regressions, quite often after the purchase of a product.

That's a good point. But I'd also like to add software still has timing problems as well which can't be easily figured out.

shuffy_losysSo in my head, if you have a function that takes in a value and its behavior changes depending on certain values of the input, 100% coverage would be you didn't just test the valid, invalid, lower, and upper bounds of the input, you tested all possible values of the input.

Testing bounds and especially invalid input is very important, but testing all possible values is most often unfeasible thanks to entropy; you could test until the end of time and still miss very obvious errors. A much better approach is to evaluate all the possible states within that section of code, and all the possible state changes, and write/perform tests accordingly. If this is unrealistic, you're probably overengineering. And very importantly validate result, not just whether the code executed without crashing.

One important note about error handling, whether it is from invalid input or internal state, far too many coders tend to write too "forgiving" code which suppresses errors, which leads to many errors being very hard to detect and eliminate.

shuffy_losysThat's a good point. But I'd also like to add software still has timing problems as well which can't be easily figured out.

Sure, there are most certainly timing issues with software, most commonly know are race conditions, but there is also other synchronization issues, even with event polling (just look at how horribly imprecise IO in Windows is), or when communicating with hardware through drivers etc.

With CPUs on the other hand, the typical timing errors are of a different nature, at least the ones which typically slips through QA. While timing issues with caches could be analogous to race conditions in code, such CPU bugs are rare compared to the ones caused by undefined behavior inside the core. CPUs thanks to their pipelined design actually have ~14-19 states simultaneously, and in-between those some units can have undefined behavior. So if some timing is off (for the reasons mentioned previously), data which should have been flushed can remain, or some data can be under some conditions manipulated, etc. Testing for such cases requires final silicon and cleverly designed tests.

AMD's advisory provides product-family target dates, such as AGESA TurinPI 1.0.0.8 for EPYC 9005, with a target date of November 14, 2025, and consumer families targeted for late November 2025. Users should install vendor BIOS/microcode updates when available and consider regenerating any sensitive keys created on unpatched systems.

Want to point out, AMD has one day left to produce AGESA firmware for client families to meet the bolded promise, and get it to vendors. They really don't seem to be taking this very seriously. Only EPYC 9005 patched so far.