Monday, October 2nd 2023
NVIDIA Chief Scientist Reaffirms Huang's Law
In a talk, now available online, NVIDIA Chief Scientist Bill Dally describes a tectonic shift in how computer performance gets delivered in a post-Moore's law era. Each new processor requires ingenuity and effort inventing and validating fresh ingredients, he said in a recent keynote address at Hot Chips, an annual gathering of chip and systems engineers. That's radically different from a generation ago, when engineers essentially relied on the physics of ever smaller, faster chips.
The team of more than 300 that Dally leads at NVIDIA Research helped deliver a whopping 1,000x improvement in single GPU performance on AI inference over the past decade (see chart below). It's an astounding increase that IEEE Spectrum was the first to dub "Huang's Law" after NVIDIA founder and CEO Jensen Huang. The label was later popularized by a column in the Wall Street Journal.
The advance was a response to the equally phenomenal rise of large language models used for generative AI that are growing by an order of magnitude every year. "That's been setting the pace for us in the hardware industry because we feel we have to provide for this demand," Dally said.
In his talk, Dally detailed the elements that drove the 1,000x gain. The largest of all, a sixteen-fold gain, came from finding simpler ways to represent the numbers computers use to make their calculations.
The New Math
The latest NVIDIA Hopper architecture with its Transformer Engine uses a dynamic mix of eight- and 16-bit floating point and integer math. It's tailored to the needs of today's generative AI models. Dally detailed both the performance gains and the energy savings the new math delivers.
Separately, his team helped achieve a 12.5x leap by crafting advanced instructions that tell the GPU how to organize its work. These complex commands help execute more work with less energy. As a result, computers can be "as efficient as dedicated accelerators, but retain all the programmability of GPUs," he said.
In addition, the NVIDIA Ampere architecture added structural sparsity, an innovative way to simplify the weights in AI models without compromising the model's accuracy. The technique brought another 2x performance increase and promises future advances, too, he said. Dally described how NVLink interconnects between GPUs in a system and NVIDIA networking among systems compound the 1,000x gains in single GPU performance.
No Free Lunch
Though NVIDIA migrated GPUs from 28 nm to 5 nm semiconductor nodes over the decade, that technology only accounted for 2.5x of the total gains, Dally noted.
That's a huge change from computer design a generation ago under Moore's law, an observation that performance should double every two years as chips become ever smaller and faster.
Those gains were described in part by Denard scaling, essentially a physics formula defined in a 1974 paper co-authored by IBM scientist Robert Denard. Unfortunately, the physics of shrinking hit natural limits such as the amount of heat the ever smaller and faster devices could tolerate.
An Upbeat Outlook
Dally expressed confidence that Huang's law will continue despite diminishing gains from Moore's law.
For example, he outlined several opportunities for future advances in further simplifying how numbers are represented, creating more sparsity in AI models and designing better memory and communications circuits.
Because each new chip and system generation demands new innovations, "it's a fun time to be a computer engineer," he said.
Dally believes the new dynamic in computer design is giving NVIDIA's engineers the three opportunities they desire most: to be part of a winning team, to work with smart people and to work on designs that have impact.
Source:
NVIDIA Blog
The team of more than 300 that Dally leads at NVIDIA Research helped deliver a whopping 1,000x improvement in single GPU performance on AI inference over the past decade (see chart below). It's an astounding increase that IEEE Spectrum was the first to dub "Huang's Law" after NVIDIA founder and CEO Jensen Huang. The label was later popularized by a column in the Wall Street Journal.
In his talk, Dally detailed the elements that drove the 1,000x gain. The largest of all, a sixteen-fold gain, came from finding simpler ways to represent the numbers computers use to make their calculations.
The New Math
The latest NVIDIA Hopper architecture with its Transformer Engine uses a dynamic mix of eight- and 16-bit floating point and integer math. It's tailored to the needs of today's generative AI models. Dally detailed both the performance gains and the energy savings the new math delivers.
Separately, his team helped achieve a 12.5x leap by crafting advanced instructions that tell the GPU how to organize its work. These complex commands help execute more work with less energy. As a result, computers can be "as efficient as dedicated accelerators, but retain all the programmability of GPUs," he said.
In addition, the NVIDIA Ampere architecture added structural sparsity, an innovative way to simplify the weights in AI models without compromising the model's accuracy. The technique brought another 2x performance increase and promises future advances, too, he said. Dally described how NVLink interconnects between GPUs in a system and NVIDIA networking among systems compound the 1,000x gains in single GPU performance.
No Free Lunch
Though NVIDIA migrated GPUs from 28 nm to 5 nm semiconductor nodes over the decade, that technology only accounted for 2.5x of the total gains, Dally noted.
That's a huge change from computer design a generation ago under Moore's law, an observation that performance should double every two years as chips become ever smaller and faster.
Those gains were described in part by Denard scaling, essentially a physics formula defined in a 1974 paper co-authored by IBM scientist Robert Denard. Unfortunately, the physics of shrinking hit natural limits such as the amount of heat the ever smaller and faster devices could tolerate.
An Upbeat Outlook
Dally expressed confidence that Huang's law will continue despite diminishing gains from Moore's law.
For example, he outlined several opportunities for future advances in further simplifying how numbers are represented, creating more sparsity in AI models and designing better memory and communications circuits.
Because each new chip and system generation demands new innovations, "it's a fun time to be a computer engineer," he said.
Dally believes the new dynamic in computer design is giving NVIDIA's engineers the three opportunities they desire most: to be part of a winning team, to work with smart people and to work on designs that have impact.
41 Comments on NVIDIA Chief Scientist Reaffirms Huang's Law
Do these fools not know what the first Graphics processor was, as I said on the shoulders of giants shouting I'm biggest, Noice.
Huang's ego knows no bounds him musk and that other 7#@7 should have a cage match, last man alive takes all with gates as referee hopefully then suicide bombed out of my world.
So you're random fact is irrelevant.
1986, Intel introduced the 82786 chip, the first discrete graphics coprocessor. The 82786 integrates a graphics processor, a display processor with a CRT controller, a bus interface unit (BIU), and its DRAM/VRAM controller.
So if we're talking MSDT then you have quad cores (2700k/3770k) vs 13900k/14900k on the Intel side & for AMD 7950x ~ even then they're at least 10-15x faster in lots of tasks. Talking about MT performance of course.
Sandy Bridge is older than 10 years old by now. It released in early 2011, so you'd have to stretch it even further, not to mention you're also compounding all frequency increases and node improvements since that.
To calculate 10 years from now you'd have to take Haswell as a base and we already had 18 core Haswell chips. Their frequency were in the low 2 GHz's range as well, so compound the frequency gains the IPC and all, 8 to 10x seems about as reasonable as we get in MT, and that's bringing all of the improvements we've had since into account: from the weakest link to the strongest one.
I still believe GPUs have outperformed CPUs in this interim, from an evolutionary perspective. But CPUs also rely on more external factors, such as memory, motherboard, exploit fixes and such.
Making GPUs faster is simpler than making CPUs faster as each GPU core is unbelievably dumb. Even a 486 is smarter than an Ada SMX. GPUs rely on massive register files and a lot of independent simultaneous work to achieve their performance. The Ada equivalent for a CPU would be SMT-48. CPUs are designed to minimize the latency of individual instructions and as such, the challenges for a CPU designer are much harder. Predicting control flow, recovering from mispredictions, bypass networks, multi-ported register files, instruction reordering, and sophisticated prefetchers for both instructions and data have no equivalent on the GPU side. Don't forget that they have to try and do all of this securely too even though that is a never-ending race.
And how does that not relate to GPU's ~ which are even more node(improvements) dependent?You mean Xeons? It was released in Q3 2014 o_O
ark.intel.com/content/www/us/en/ark/products/codename/42174/products-formerly-haswell.html#@Server
You're seriously making a mess of timelines!
ark.intel.com/content/www/us/en/ark/products/52214/intel-core-i72600k-processor-8m-cache-up-to-3-80-ghz.html
Haswell also debut in 2013
ark.intel.com/content/www/us/en/ark/products/75123/intel-core-i74770k-processor-8m-cache-up-to-3-90-ghz.html
Both of them now exceed a decade of age, HEDT and high-end server always follows a year later or so but they are very much part of the same architectural family and follow a similar support path, as you can see, servicing ended only 6 months after the client processors
ark.intel.com/content/www/us/en/ark/products/81061/intel-xeon-processor-e52699-v3-45m-cache-2-30-ghz.html
I've got a 4669 v3 myself, 12115 on R23, 550 single
Compared to my i9-13900KS at 41430/2368
So that's the absolute highest end CPU that were available back then for the scalable server segment x the same for consumer client today, in 10 years comparing 18x Haswell cores at 2.4 GHz x 8x Raptor Cove + 16x Gracemont config, we have, at least on Cinebench: 3.42x the multicore clout and 4.30x the single-core performance, accounting for going from quad-channel DDR4-2133 to dual DDR5-6400
We'd be arguing semantics, I brought this as a somewhat reasonable/realistic benchmark, but the truth is that it will change wildly with the SIMD optimizations available to each processor and is extremely workload-dependant.
I find it reasonable to believe GPU growth has outpaced CPU growth, but idk, YMMV.
So horse shit.
I've had 1000000% less sick days this month then last, And, at least I mentioned a reciprocal period, see.
Also no constraints mentioned at all.
Second.
They're on about AI and machine learning performance, or were.
Games have not scaled anywhere near that rate even if you're drinking all the dlss cool ade you can while saluting AMD.
So gaming improvements are irrelevant.
You also limit the time span of viable CPU in the list,why.
In comparison terms year dot to now on CPU verses the first AI hardware Nvidia gpu to now is apt.
Yes GPU out scaled CPU but then WiFi 7 chips will exponentially out pace WiFi six last year.
I hope tenstorrent shows Huang his shit ain't all that like ASICS did to Huang's mining interest or did you all think GPU mining died on GPU just because?!?.
You're saying GPU demand has stagnated, but really, it hasn't, look at new engine performance and how Ada struggles, and RDNA3 isn't killing it either. Dozens of fresh games out of the gate this year struggle with performance and even post bug fix I still can't say I understand why shit runs as it does, and that's definitely not because of CPU constraints.
And then there is the RT tax, or any non proprietary version of it. If Huang wants that implemented, he's gonna have to get a lot more serious than this paltry 4090.
I don't take 'most games doing a thing wrong' for granted as 'this is the new normal'. I just know shit code when I see it. And every once in a while, there is a game/a dev that proves everyone wrong, and proves me right. Blame commerce and money for all the bad things, but don't let it cloud your view of what's what. A lot of limitations and problems are self-imposed.
The real question is how much you are willing to invest as a consumer in brute forcing past all those bills passed to you from the developer side. The reality is, once a baseline of perf is achieved, everything above it is bonus. The application runs. It doesn't crash. Market that shit. And here we are, playing a half finished product and throwing 1500 bucks of GPU at it to get the frames we want ;) GPUs just travel along with the demand in gaming, simple as. There is no Huang's Law. There is Huang's 80+% market share and the resulting ability to dictate nonsense that feels like a law - while stocks last.
GPU mining died because Ethereum made good on their proof of stake transition. The other cryptocurrencies are too small to be profitable, so a lot fewer people kept on mining. To compound that, the lockdowns ended and people returned to work, with that a lot of the speculative potential of crypto disappeared overnight. ASICs will not push GPU out of the business... And I'm unsure of the relation between Wi-Fi (radio comms) and raw compute performance?I did not say that GPU demand has stagnated, I said that current generation CPUs bottleneck the 4090 in all but a handful of very intensive scenarios involving super high resolutions and complex workloads. It's a different claim, one that you'll pretty easy to measure when we're talking high refresh scenarios.
And that's the point of contention ~ GPU's IMO have not outpaced CPU growth in the last decade or so :ohwell:
And limiting the chips to just MSDT also wouldn't make sense because they're artificially limited, in terms of core count, by Intel/AMD due to the market they serve.
The scaling part is important because the vast majority of tasks on a GPU scales nicely for larger chips.