Tuesday, July 23rd 2024
Intel Statement on 13th and 14th Gen Core Instability: Faulty Microcode Causes Excessive Voltages, Fix Out Soon
Long-term reliability issues continue to plague Intel's 13th Gen and 14th Gen Core desktop processors based on the "Raptor Lake" microarchitecture, with users complaining that their processors have become unstable with heavy processing workloads, such as games. This includes the chips that have minor levels of performance tuning or overclocking. Intel had earlier isolated many of these stability issues to faulty CPU core frequency boosting algorithms, which it addressed through updates to the processor microcode that it got motherboard- and prebuilt manufacturers to distribute as UEFI firmware updates. The company has now come out with new findings of what could be causing these issues.
In a statement Intel posted on its website on Monday (22/07), the company said that it has been investigating the processors returned to it by users under warranty claims (which it has been replacing under the terms of its warranty). It has found that faulty processor microcode has been causing the processors to operate under excessive core voltages, leading to their structural degradation over time. "We have determined that elevated operating voltage is causing instability issues in some 13th/14th Gen desktop processors. Our analysis of returned processors confirms that the elevated operating voltage is stemming from a microcode algorithm resulting in incorrect voltage requests to the processor."Modern processor power management runs on an intricate clockwork of collaboration between software, firmware, and hardware, with the software constantly telling the hardware what levels of performance it wants, and the hardware managing its power- and thermal budgets by rapidly altering the power and clock speeds of the various components, such as CPU cores, caches, fabric, and other on-die components. A faulty collaboration between any of the three key components could break this clockwork, as has happened in this case.
Intel is releasing yet another microcode update to its 13th- and 14th Gen Core processors, which will address not just the faulty boosting algorithm issue the company unearthed in June, but also the faulty voltage management the company discovered now. This new microcode should be released some time around mid-August to partners (motherboard manufacturers and PC OEMs), who will then need to validate it on their machines, before passing it along to end-users as UEFI firmware updates.
Meanwhile, an interesting issue has come to light, which that some of Intel's processors built on the Intel 7 node are experiencing chemical oxidation of the die as they age. Intel responded to this, stating that it had discovered the oxidation manufacturing issues in 2023, and addressed it. The company also stated that die oxidation is not related to the stability issues it is embattled with.
Sources:
Intel Community, Intel (Reddit)
In a statement Intel posted on its website on Monday (22/07), the company said that it has been investigating the processors returned to it by users under warranty claims (which it has been replacing under the terms of its warranty). It has found that faulty processor microcode has been causing the processors to operate under excessive core voltages, leading to their structural degradation over time. "We have determined that elevated operating voltage is causing instability issues in some 13th/14th Gen desktop processors. Our analysis of returned processors confirms that the elevated operating voltage is stemming from a microcode algorithm resulting in incorrect voltage requests to the processor."Modern processor power management runs on an intricate clockwork of collaboration between software, firmware, and hardware, with the software constantly telling the hardware what levels of performance it wants, and the hardware managing its power- and thermal budgets by rapidly altering the power and clock speeds of the various components, such as CPU cores, caches, fabric, and other on-die components. A faulty collaboration between any of the three key components could break this clockwork, as has happened in this case.
Intel is releasing yet another microcode update to its 13th- and 14th Gen Core processors, which will address not just the faulty boosting algorithm issue the company unearthed in June, but also the faulty voltage management the company discovered now. This new microcode should be released some time around mid-August to partners (motherboard manufacturers and PC OEMs), who will then need to validate it on their machines, before passing it along to end-users as UEFI firmware updates.
Intel is delivering a microcode patch which addresses the root cause of exposure to elevated voltages. We are continuing validation to ensure that scenarios of instability reported to Intel regarding its Core 13th/14th Gen desktop processors are addressed. Intel is currently targeting mid-August for patch release to partners following full validation. Intel is committed to making this right with our customers, and we continue asking any customers currently experiencing instability issues on their Intel Core 13th/14th Gen desktop processors reach out to Intel Customer Support for further assistance, the company stated.It's important to note here, that the microcode update won't fix the issues on processors already experiencing instability, but prevent it on chips that aren't. The instability is caused by irreversible physical degradation of the chip. These chips will, of course, be covered under warranty.
Meanwhile, an interesting issue has come to light, which that some of Intel's processors built on the Intel 7 node are experiencing chemical oxidation of the die as they age. Intel responded to this, stating that it had discovered the oxidation manufacturing issues in 2023, and addressed it. The company also stated that die oxidation is not related to the stability issues it is embattled with.
We can confirm that the via Oxidation manufacturing issue affected some early Intel Core 13th Gen desktop processors. However, the issue was root caused and addressed with manufacturing improvements and screens in 2023. We have also looked at it from the instability reports on Intel Core 13th Gen desktop processors and the analysis to-date has determined that only a small number of instability reports can be connected to the manufacturing issue, the company stated.If you feel your chip might be affected, you can file for an RMA.

387 Comments on Intel Statement on 13th and 14th Gen Core Instability: Faulty Microcode Causes Excessive Voltages, Fix Out Soon
Remember Turbo specs/clocks aren't guaranteed & technically it's still OCing ~ Intel/AMD only guarantee base clocks!
Speaking of Xeons, the Xeons with the highest clocks are the 6 core E-2486 and 8 core E-2488, both boosting up to 5.6GHz. And their recommended prices are $506.00 and $606.00 respectively. So not sure where some people got the idea that people are using 14900K on servers to save money, since the recommended price for a 14900K is $589.00-$599.00.
Yea those with a Max boost of 4.8 GHz (w9-3495X)
That's a massive stretch and we both know it.
Be it server or non-server processor, the processor itself must remain fully functional till the warranty ends. The CPU has to be configured by a manufacturer in a way that it won't allow to damage itself over time. That's what the thermal throttling and TDP, current limits are for. We have at work multiple desktop PCs equipped with 10th Gen i7 Ks that fill a role of small camera servers, web servers and database server. We chose them as they are perfectly fine for this purpose in terms of their performance and were much cheaper than regular rack server. They have been working 24/7 for a few years now without any problems.
You don't need Xeon or EPYC or Threadripper just because the Xeon/EPYC/TRP primarily used for servers. Any other CPU is capable of handling same tasks or work but may lack some instruction sets so it may not be as effective. Although I agree on a fact that server CPUs tend to be optimized for maximum efficiency. Also, it goes the other way around - meaning you can enjoy playing games on server-grade CPUs without any problems, but you will get lower single-thread performance this way.
As I said, it doesn't matter whether it is OCed or non OCed CPU, it must be configured by the manufacturer to be fully operational at least till it's warranty period ends.
Even running desktop CPU at 6 GHz 24/7 must be perfectly safe and the CPU must not die, provided there is sufficient cooling.
So no, your point of view is not right, as in this case the manufacturer (Intel) let their products run outside of their safe limits for such a long time that it caused unrecovereable damage to them. It's a long-known general information that 1.5V is way too much for modern chips. Personally, I wouldn't go past 1.3V for VCore and 1.2V for VSoC.
Btw Xeons are quite pricy for what they offer. Using Core CPU desktop as a server is nothing special and there's nothing wrong about it as long as your power and performance requirements are met.
In my particular case, with the 13900KF, going over that TDP is not only "sanctioned" by them, it is actually recommended. And about using the baseline power profile, which would enforce the TDP, they say this: "Intel does not recommend Baseline power delivery profiles for the 13th gen and 14th gen K Sku processors unless required for compatibility."
Furthermore, they say this: "Intel recommends using the 'Extreme' power delivery profile if supported by the voltage regulator (VR) and motherboard design". Also: "Intel strongly recommends these values to be applied as BIOS defaults". Which is exactly what ASUS does with their latest BIOS updates. It defaults to the Intel Extreme profile. I can as an option switch to the Intel Performance profile that lowers the current limits. And the Baseline profile, which is not recommended by Intel, is not even an option.
So no, they don't just "sanction" it, they "strongly recommend" the power profiles that allow you to go over the TDP, which will allow you to run all the cores over their base frequency in MT workloads.
ark.intel.com/content/www/us/en/ark/products/236182/intel-xeon-e-2488-processor-24m-cache-3-20-ghz.html
ark.intel.com/content/www/us/en/ark/products/236185/intel-xeon-e-2486-processor-18m-cache-3-50-ghz.html
BTW, the base frequency, as you can see, has everything to do with TDP. The 6 core CPU has a higher base frequency because that TDP budget is split between fewer cores, so they can clock higher. The boost frequency is the same because it's not affected by the TDP.
The entry level server chips are called the E-2400 series, e.g. Xeon E-2488, which is the Xeon part that closely resembles i9-13900K/14900K, except for it lacking aggressive boost and voltage. If they had gone for proper server grade parts, they likely would never have seen these issues.
I must add that several outlets calls W680 boards "server grade", they are not, they are workstation boards. Don't get me wrong, they are good boards, but wouldn't stop the consumer CPUs from aging prematurely.
If we assume these i9s "age" 4-5x faster than expected due to too much voltage, then this would easily explain why people using these as "servers" would see them fail after ~3-6 months. This only tells us that they've gotten away with consumer grade hardware in the past, because CPUs from the past few generations have been very reliable.We don't know what they did, and until we have evidence we shouldn't speculate.
What we should do instead is to encourage those with contacts within Intel to publicly address this more precisely;
- Which product ranges were affected?
- For how long did this problem happen?
- Was this limited to certain production lines or everything?As mentioned, the Xeon E-2400 series offers similar performance, probably ~95% of the same performance for more sustained loads when loaded up with many threads.
As for the argument about "blast radius" which several mentioned; servers crashing regularly shouldn't be an issue like that, and servers crashing regularly is normally fairly unheard of, and the amount of management overhead could easily justify having fewer servers with more cores. What this sounds to me is some companies are cutting corners and being unprofessional. (not that this in any way reduces the issues for Intel, this just exposes what practices some companies have)
As for the similar performance, you mentioned many threads. But the companies doing this do not need many threads, so that's irrelevant. They wanted the highest clocks possible because their use case is limited by single thread performance.
Furthermore, when the Raptor Lake i9s have been launched, those 2400 series Xeon CPUs were not even an option. The Raptor Lake i9s were released in Q4'22. The Xeons one year later, in Q4'23. The only option before that were the Rocket Lake Xeons, that clocked only up to 5.30 GHz.
Yes, people using the i9s on servers thought they took a just slight risk to get more performance. But "a slight risk", in my opinion, would be to lose 1%, 2%, maybe up to 5% of the CPUs after a year. This is much worse.
And, even ignoring the server discussion, this is unacceptable for regular consumers. And you can't say that the consumers should have bought Xeons instead.
In fact, almost every other generation(major microarchitecture) have had some kind of major issue (and the same goes for the other team, but we're apparently not permitted to mention that), but most have a fairly short memory. What's unusually here is that the issue(s) are discovered so long after the product release. That usually only happens with security vulnerabilities or bugs which are very hard to pin down. And for historical context, just a few examples: Sandy Bridge (desktop) was a nightmare in the beginning with CPU and chipset issues, people predicted a disaster for Intel back then. On top of that Sandy Bridge-E and server has bugs too. But all of which were resolved, and are now remembered as one of the great milestones in CPU history. :)Well, it's not linear, it depends on how aggressively the CPU applies too much voltage (which is probably why we hear about more i9-13900Ks than i5-13600s), and the Xeons don't have this aggressive voltage for extreme boosting. So there is presumably no issue there at all.It's not like they're hosting a single game on one server at the time ;)
A possible defect in 11th gen CPUs. (finally fixed with BIOS update 1601 11/24/22 M13H MB) | Overclock.net
You're right -- this is the worst one by a mile -- im not at all saying it's good. But consumer cpus are not generally considered 'STABLE' for 'PRODUCTION' workloads.
The main issue is that it's taking intel FOREVER to respond, and it feels like they're purposely dragging their feet in admitting what's actually wrong with the hardware -- which is not common for these types of issues. It feels like even Spectre/Meltdown was addressed faster than this... and that was a 10-25% hit inperformance to alot of existing processors at the time.
This is the 'Chernobyl' response strategy and it never ends up well.
Of course intel used to run desktop chips to Xeons only after a good while of selling them for general usage, so they could weed out some problems early on, which does explain some of the stuff.
Anyway, none of the listed instability problems were of this scale, just a few% of chips being bad from the first couple of batches, or in the case of the temperature reading bug - a total non issue for running PRODUCTION servers.
And just to reiterate, there are differences in PRODUCTION workloads. Some need the utmost data integrity and stability, while others DO NOT. A bank is very different to a game server hosting service. For the game hosting service only the facilitating and load management servers need to be very reliable, others you just spin more up if some fail. None the less, they are PRODUCTION workloads.
Production workloads, by definition, require stability and performance, but first and foremost stability. You can build a mac mini farm, or a rasberry pi farm, or a farm of blades running game servers run a specific workload and call it 'Production', but if your processor comes with marketing materials with the words 'Exxxtreme' or 'Overclocking' or 'Gamers' it is not a production class system.
Keep in mind im not disagreeing with you -- this is a total disaster for them on the prosumer side, and highlights a huge gap in their product line. But it's much more common to have problems when you repurpose consumer gear.
en.wikipedia.org/wiki/Deployment_environment
Marketing jargon is its own thing, and has nothing to do with the topic. Saying something is "production class" is also just marketing.Yes, smaller, but failure rate has likely been something around 1% for consumer stuff and 0.1% for xeon stuff - pretty small for both. For some production use it makes total sense to use non xeon parts, as they are faster and cheaper. Now suddenly the failure rates are 50% and the message from some users here has been like "They should have know", which is total bullcrap.
100% sympathy to the end-users and the SIs that Intel did screw over. My point was that "SERVER" crash rates in a conversation about desktop overclocking cpus is less relevant since other sub-optimal decisions had to be made for that to happen.