ShrekWhy 4 threads per core? Two threads keeps the core almost fully busy, so more threads gains little to nothing.

ShrekWhy 4 threads per core? Two threads keeps the core almost fully busy, so more threads gains little to nothing.

AleksandarKIIRC IBM Power has 4 SMT, very specific scientific workloads. Think nuclear decomposition.

stickleback123Power8 has 8 way SMT!

stickleback123I think it's possible, not certain but possible, that the team of hundreds of highly trained and qualified microprocessor engineers might know something about this? Other high end CPUs have used more than 2 way SMT in the past.

They'll have simulated this every which way to Sunday.

Chinese server CPU maker Hygon, which owns an x86 CPU license from AMD

AMD Does Due Diligence

Simply stating ‘AMD sublicensed the IP of one of its x86 designs’ sounds a bit farfetched on most days of the week. If either AMD or Intel believed that the opportunity to let others sell its CPU designs was profitable, how come it took until 2015/2016 to ever come to fruition? Part of this story covers that while there was clearly some money in it for AMD here, it didn’t fall foul of any Intel-AMD licensing agreements. And most importantly, it didn’t contravene any US laws regarding the export of high-performance computing intellectual property.

This last point is important. The US government gives every CPU that comes out of Intel, AMD, and others, a value based on its performance. This is some combination of FLOPs and power, and those that surpass a specific threshold are deemed too powerful to be sold in certain markets. This includes semi-custom processors, where AMD/Intel fiddle with the core count/frequency and provide off-roadmap parts.

AMD at the time made the following statement:

Starting in 2015, AMD diligently and proactively briefed the Department of Defense, the Department of Commerce and multiple other agencies within the U.S. Government before entering into the joint ventures. AMD received no objections whatsoever from any agency to the formation of the joint ventures or to the transfer of technology – technology which was of lower performance than other commercially available processors. In fact, prior to the formation of the joint ventures and the transfer of technology, the Department of Commerce notified AMD that the technology proposed was not restricted or otherwise prohibited from being transferred. Given this clear feedback, AMD moved ahead with the joint ventures.

AMD had contacted the DoD and DoC, as well as all others, and had been given the green light. The new microarchitecture was deemed of low enough performance to not hit any of the export bans. AMD was also given crystal clear confirmation that the ‘technology proposed was not restricted or otherwise prohibited from being transferred’, which is a rather stark statement. At this point it should be clear that AMD may have submitted a modified version of its IP to the relevant US departments, rather than the microarchitecture we saw in the Ryzen 1000-series. This is part of what this review is about.

ncrsWhat is the source for this statement? As far as I know AMD is not capable of sub-licensing x86 without Intel's approval, and that's not what happened with Hygon.
AnandTech's analysis from 2020:

stickleback123I think it's possible, not certain but possible, that the team of hundreds of highly trained and qualified microprocessor engineers might know something about this? Other high end CPUs have used more than 2 way SMT in the past.

They'll have simulated this every which way to Sunday.

ShrekWhy 4 threads per core? Two threads keeps the core almost fully busy, so more threads gains little to nothing.

stickleback123I think it's possible, not certain but possible, that the team of hundreds of highly trained and qualified microprocessor engineers might know something about this? Other high end CPUs have used more than 2 way SMT in the past.

They'll have simulated this every which way to Sunday.

efikkanRemember, the 4 threads will compete over caches and front-end resources, so the effective throughput for a single thread for the intended workload would have to be pretty miserable in order to justify 4-way SMT (or even 8-way like with PPC).

efikkanIt could also be that their SMT implementation works different from Intel and AMD, e.g. executing two of four threads intermixed (where Intel/AMD switches between two threads). If this happens to be the case, the saturation for each thread would be dreadful.

efikkanThis is limited to workloads where the core is stalled most of the time thanks to cache misses and mispredictions, each worker thread is async, and the only thing that matters is overall throughput (not latency).

efikkan"21 custom security instructions"
I do wonder what those entails :(

efikkanSMT is a relic of the past, and stopped making sense for user-interactive workloads after quad cores, but will stick around for a while in the server space, partly due to marketing reasons, but also because there are certain server workloads where it sort-of "makes sense", but that rationale is still shrinking. This is limited to workloads where the core is stalled most of the time thanks to cache misses and mispredictions, each worker thread is async, and the only thing that matters is overall throughput (not latency). Remember, the 4 threads will compete over caches and front-end resources, so the effective throughput for a single thread for the intended workload would have to be pretty miserable in order to justify 4-way SMT (or even 8-way like with PPC).

While modern x86 microarchitectures from Intel and AMD aren't anywhere close to saturating the CPU resources, their continuing advancement have made SMT less and less useful over time. So the less idle cycles there are, the less "free performance" can be extracted through SMT, which is probably what you're thinking about.

efikkanMeanwhile, Intel's upcoming Diamond Rapids and hopefully Nova Lake will introduce APX, which according to their documentation should bring a significant uplift in throughput.

efikkanThey probably have extracted the performance they could the easiest way within their time-frame and constraints, and the end result is a CPU with lots of resources on the execution side, but with a very weak front-end to feed it.
It could also be that their SMT implementation works different from Intel and AMD, e.g. executing two of four threads intermixed (where Intel/AMD switches between two threads). If this happens to be the case, the saturation for each thread would be dreadful.

efikkanFor instance PPC with its 8-way SMT is(was?) popular for certain java workloads, which are so inefficient that they barely execute at all. :p (more like a traffic jam…)

igormpZen 5, as an example, has 2x 4-wide decoders…

igormpGoing for SMT makes your front-end way simpler as well, and allows you to do some fancy strategies to increase IPC and maximize EU utilization for some given scenarios.<snip>
A single thread will end up bottlenecked by it, and is not able to make use of both decoders, both with SMT then it's possible to basically double up the IPC.

ncrsAPX is solving different issues, chiefly of register pressure.

ShrekWhy 4 threads per core? Two threads keeps the core almost fully busy, so more threads gains little to nothing.

stickleback123I think it's possible, not certain but possible, that the team of hundreds of highly trained and qualified microprocessor engineers might know something about this? Other high end CPUs have used more than 2 way SMT in the past.

They'll have simulated this every which way to Sunday.

efikkanZen 5 implements two-ahead branch prediction, in an effort to reduce the cost of mispredicitons by having the alternative branch ready to be executed. Such improvements are just another example of reducing idle clock cycles which in return means gains from SMT will be reduced.

efikkanJust to be clear, SMT the way Intel and AMD implements it doesn't improve IPC at all. It just tries to keep the core fed, as if it was one single thread saturating the core.

igormpIt does improve IPC in absolute terms in practice, given that a single thread is not able to effectively saturate the core. As an example, see this micro-benchmark from chips and cheese:
chipsandcheese.com/p/amds-ryzen-9950x-zen-5-on-desktop

efikkanZen 5 implements two-ahead branch prediction, in an effort to reduce the cost of mispredicitons by having the alternative branch ready to be executed. Such improvements are just another example of reducing idle clock cycles which in return means gains from SMT will be reduced.

I haven't seen any evidence of significant gains from this yet, but with refinement and combined with APX it has some great (theoretical) potential.

Both the fetch+decode and op cache pipelines can be active at the same time, and both feed into the in-order micro-op queue.

efikkanJust to be clear, SMT the way Intel and AMD implements it doesn't improve IPC at all. It just tries to keep the core fed, as if it was one single thread saturating the core.

efikkanActually not, the benefits from less register shuffling is just an added bonus, and a rather minimal one to be honest.
APX is about maximizing the efficiency of the branch predictor to saturate the CPU which is very clearly explained in the official documentation:

The performance features introduced so far will have a limited impact on workloads that suffer from a large number of conditional branch mispredictions. As out-of-order CPUs continue to become deeper and wider, the cost of mispredictions increasingly dominates the performance of such workloads. Branch predictor improvements can mitigate this only to a limited extent as data-dependent branches are fundamentally hard to predict.

To address this growing performance issue, we significantly expand the conditional instruction set of x86, which was first introduced with the Intel® Pentium® Pro in the form of CMOV/SET instructions. These instructions are used quite extensively by today’s compilers, but they are too limited for the broader use of if-conversion (a compiler optimization that replaces branches with conditional instructions).

Intel APX adds conditional forms of load, store, and compare/test instructions and adds an option for the compiler to suppress the status flag writes of common instructions. These enhancements expand the applicability of if-conversion to much larger code regions, cutting down on the number of branches that may incur misprediction penalties.

So as you can clearly see, this is very much about saturating the CPU.

How successful it will be, remains to be seen. This is pretty much in line with what myself and other programmers have requested for many years, if anything I'm wondering it it's enough.

igormpIt does improve IPC in absolute terms in practice, given that a single thread is not able to effectively saturate the core.

ncrsWhat you quoted affects one type of workload, and doesn't invalidate SMT in any way. As I wrote before I haven't read anything that makes APX SMT-phobic ;)

efikkan"21 custom security instructions"
I do wonder what those entails :(


SMT is a relic of the past, and stopped making sense for user-interactive workloads after quad cores, but will stick around for a while in the server space, partly due to marketing reasons, but also because there are certain server workloads where it sort-of "makes sense", but that rationale is still shrinking. This is limited to workloads where the core is stalled most of the time thanks to cache misses and mispredictions, each worker thread is async, and the only thing that matters is overall throughput (not latency). Remember, the 4 threads will compete over caches and front-end resources, so the effective throughput for a single thread for the intended workload would have to be pretty miserable in order to justify 4-way SMT (or even 8-way like with PPC).

While modern x86 microarchitectures from Intel and AMD aren't anywhere close to saturating the CPU resources, their continuing advancement have made SMT less and less useful over time. So the less idle cycles there are, the less "free performance" can be extracted through SMT, which is probably what you're thinking about.

Meanwhile, Intel's upcoming Diamond Rapids and hopefully Nova Lake will introduce APX, which according to their documentation should bring a significant uplift in throughput.


They probably have extracted the performance they could the easiest way within their time-frame and constraints, and the end result is a CPU with lots of resources on the execution side, but with a very weak front-end to feed it.
It could also be that their SMT implementation works different from Intel and AMD, e.g. executing two of four threads intermixed (where Intel/AMD switches between two threads). If this happens to be the case, the saturation for each thread would be dreadful.

For instance PPC with its 8-way SMT is(was?) popular for certain java workloads, which are so inefficient that they barely execute at all. :p (more like a traffic jam…)

efikkanAbsolutely not, it's a common misconception that IPC is performance per clock, when it's not, it's the amount of instructions the CPU is able to churn through. Whether there is one, two or more threads sharing a core's resources, the IPC remains constant. SMT does improve the saturation of the core for some workloads, but the total performance will only converge towards a fully thread fully saturating the core, never above that. This should be basic knowledge about CPUs.

efikkanThe more saturated the core is from one thread, the less gains there will be from SMT, this should be obvious and is basic logical deduction, and is why we've seen fewer and fewer cases where SMT is significantly beneficial as CPUs advances.

efikkanOn top of that, the intricate complexity of implementing SMT in the pipeline in modern CPUs, with the resulting transistor "costs" and design constraints, and all the nasty security implications, it naturally comes to a point where the efforts can be better spent by creating a more efficient architecture without SMT. This is why Intel's client CPUs have already moved on, and others will eventually follow.

The 2T point gets emphasis here. AMD is well aware that Intel is planning to leave SMT out of their upcoming Lunar Lake mobile processor. Zen 5 takes the opposite approach, maintaining SMT support even in mobile products like Strix Point. AMD found that SMT let them maintain maximum 1T performance while enjoying the higher throughput enabled by running two threads in a core for multithreaded workloads. They also found SMT gave them better power efficiency in those multithreaded loads, drawing a clear contrast with Intel’s strategy.

ncrsYou are redefining what "IPC" means to suit your argument. I gave you detailed test results which you simply ignore. There's not much more I can do here.