Next‑Gen HBM4 to HBM8: Toward Multi‑Terabyte Memory on 15,000 W Accelerators

[AleksandarK]

In a joint briefing this week, KAIST's Memory Systems Laboratory and TERA's Interconnection and Packaging group presented a forward-looking roadmap for High Bandwidth Memory (HBM) standards and the accelerator platforms that will employ them. Shared via Wccftech and VideoCardz, the outline covers five successive generations, from HBM4 to HBM8, each promising substantial gains in capacity, bandwidth, and packaging sophistication. First up is HBM4, targeted for a 2026 rollout in AI GPUs and data center accelerators. It will deliver approximately 2 TB/s per stack at an 8 Gbps pin rate over a 2,048-bit interface. Die stacks will reach 12 to 16 layers, yielding 36-48 GB per package with a 75 W power envelope. NVIDIA's upcoming Rubin series and AMD's Instinct MI500 cards are slated to employ HBM4, with Rubin Ultra doubling the number of memory stacks from eight to sixteen and AMD targeting up to 432 GB per device.

Looking to 2029, HBM5 maintains an 8 Gbps speed but doubles the I/O lanes to 4,096 bits, boosting throughput to 4 TB/s per stack. Power rises to 100 W and capacity scales to 80 GB using 16‑high stacks of 40 Gb dies. NVIDIA's tentative Feynman accelerator is expected to be the first HBM5 adopter, packing 400-500 GB of memory into a multi-die package and drawing more than 4,400 W of total power. By 2032, HBM6 will double pin speeds to 16 Gbps and increase bandwidth to 8 TB/s over 4,096 lanes. Stack heights can grow to 20 layers, supporting up to 120 GB per stack at 120 W. Immersion cooling and bumpless copper-copper bonding will become the norm. The roadmap then predicts HBM7 in 2035, which includes 24 Gbps speeds, 8,192-bit interfaces, 24 TB/s throughput, and up to 192 GB per stack at 160 W. NVIDIA is preparing a 15,360 W accelerator to accommodate this monstrous memory.



Last but not least is the HBM8 by 2038 with 32 Gbps, 16,384 lanes, 64 TB/s, 200-240 GB stacks, and embedded cooling at 180 W. Beyond pure DRAM, the briefing also teased hybrid HBM-HBF (High-Bandwidth Flash) architectures to meet the demands of large-scale AI inference, pairing NAND-based flash with DRAM stacks through ultra-fast through-silicon vias. With accelerators evolving toward petabyte‑class bandwidth and multi‑chiplet designs, these emerging HBM and HBF standards are the only way forward.

View at TechPowerUp Main Site | Source

 

[Vayra86]

Yay for climate!

I mean we are using this to fix that issue, right?

 

[Prima.Vera]

Basically you will need your own Nuclear Power Plan to run a Datacenter with those abominations.
This is not even a sarcastic remark sadly...

 

[dgianstefani]

People somehow still don't get efficiency does not equal power usage. It's power usage against work done. The work is going to be done regardless if investment in large scale hardware is done. So it's this, or worse power numbers.

See 450 W 5080 and 600 W 5090 topping efficiency charts even at stock.

I can essentially guarantee these units will be more efficient than lower wattage systems.

Furthermore even if they do require their own reactors, that's exactly what the nuclear industry needs. Modular, cheap, efficient and safe reactors (which have been in development before the issue of AI) rather than the regulated to death designs from decades ago super sized jokes governments build. These being used and developed will cut their cost and make them more accessible to other industries.

 

[Vayra86]

People somehow still don't get efficiency does not equal power usage. It's power usage against work done. The work is going to be done regardless if investment in large scale hardware is done. So it's this, or worse power numbers.

See 400 W 5080 and 600 W 5090 topping efficiency charts even at stock.

I can essentially guarantee these units will be more efficient than lower wattage systems.

Furthermore even if they do require their own reactors, that's exactly what the nuclear industry needs. Modular, cheap, efficient and safe reactors (which have been in development before the issue of AI) rather than the regulated to death designs from decades ago super sized jokes governments build. These being used and developed will cut their cost and make them more accessible to other industries.

Of course not.

The work is going to become economically more feasible so more of it gets done and that in turn drives growth, ad infinitum.
This is what progress in tech ultimately brings us. It opens up new avenues, opportunities, possibilities... and new ways to use all that power.

Car analogy works here. As cars get more efficient, they've also progressively gotten heavier and bulkier. You can chalk that up to 'demand'... but demand is created. And it is created because it is enabled.

The bottom line is, progress leads to more power usage.

The bottom line is this - note the acceleration as well. We're not just growing, there is an escalation of growth since around 2000. And that is despite all the fantastic green products we have.

This whole idea technological progress leads to savings is top level bullshit that never got proven in real life. Its a utopian thought that allows us to keep consuming and developing; wishful thinking. And in isolation, its true, and fine that way. Too bad there's also a planet.

Furthermore even if they do require their own reactors, that's exactly what the nuclear industry needs. Modular, cheap, efficient and safe reactors (which have been in development before the issue of AI) rather than the regulated to death designs from decades ago super sized jokes governments build. These being used and developed will cut their cost and make them more accessible to other industries.

There is nothing stopping governments from running similar projects to provide energy. Well yeah. Public opinion.

 

[AleksandarK]

Of course not.

The work is going to become economically more feasible so more of it gets done.
This is what progress in tech ultimately brings us. It opens up new avenues, opportunities, possibilities... and new ways to use all that power.

Car analogy works here. As cars get more efficient, they've also progressively gotten heavier and bulkier. You can chalk that up to 'demand'... but demand is created. And it is created because it is enabled.

In the case of LLMs, reasoning requires 10x more compute, but if you need to run it in an entire cluster it costs you 30kW of power. Now imagine a single 15kW chip running an entire reasoning model because it can fit it in memory and a 15kW processor can run it with more tokens per second. That now equals 2x efficiency, and another x from boosted token throughput. Of course models will become more complex, but that is only allowed by having more powerful hardware. And that more complicated AI will solve more problems, which is also efficiency increase. It is an endless loop. Efficiency is always increasing.

 

[Vayra86]

In the case of LLMs, reasoning requires 10x more compute, but if you need to run it in an entire cluster it costs you 30kW of power. Now imagine a single 15kW chip running an entire reasoning model because it can fit it in memory and a 15kW processor can run it with more tokens per second. That now equals 2x efficiency, and another x from boosted token throughput. Of course models will become more complex, but that is only allowed by having more powerful hardware. And that more complicated AI will solve more problems, which is also efficiency increase. It is an endless loop. Efficiency is always increasing.

And so is energy usage. That's the point, and the problem. Its also paradoxical, to solve 'ever more complicated problems' while not getting even remotely in the vicinity of solving the biggest problem. In fact, AI nor companies ever want to fix that problem, because they don't benefit from it.

I'm not opposing the technicality of efficiency increases. But its not bringing us anywhere. Its just a gateway to do more, and 'more' is like a drug humans can't seem to stop taking. We're addicts and we'll find ways to keep using.

 

[AleksandarK]

And so is energy usage. That's the point, and the problem. Its also paradoxical, to solve 'ever more complicated problems' while not getting even remotely in the vicinity of solving the biggest problem. In fact, AI nor companies ever want to fix that problem, because they don't benefit from it.

I'm not opposing the technicality of efficiency increases. But its not bringing us anywhere. Its just a gateway to do more, and 'more' is like a drug humans can't seem to stop taking. We're addicts and we'll find ways to keep using.

"More" is what got us here: more tech, more innovation. There are billions of R&D that are "thrown away" just to bring you a 1% more interesting/efficient product.

 

[dgianstefani]

Car analogy works here. As cars get more efficient, they've also progressively gotten heavier and bulkier. You can chalk that up to 'demand'... but demand is created. And it is created because it is enabled.

False example unfortunately.

The reason cars have gotten bigger, particularly in the USA, is due to so called environmental standards. Ever wondered why you can't buy a (new) small truck with a big engine and a big weight/towing capacity? Why the Japanese pick up trucks are so valued and imported at great expense? You have to make the vehicle bigger to fit the standards. A lot of the rest is simply the peak aerodynamic efficiency has been found, and crumple zones (safety standards) dictate x amount of space etc. Why all cars look the same. Oh, and fatter people who want more amenities.

Of course not.

The work is going to become economically more feasible so more of it gets done and that in turn drives growth, ad infinitum.
This is what progress in tech ultimately brings us. It opens up new avenues, opportunities, possibilities... and new ways to use all that power.

Car analogy works here. As cars get more efficient, they've also progressively gotten heavier and bulkier. You can chalk that up to 'demand'... but demand is created. And it is created because it is enabled.

The bottom line is, progress leads to more power usage.

The bottom line is this - note the acceleration as well. We're not just growing, there is an escalation of growth since around 2000. And that is despite all the fantastic green products we have.

View attachment 403875

This whole idea technological progress leads to savings is top level bullshit that never got proven in real life. Its a utopian thought that allows us to keep consuming and developing; wishful thinking. And in isolation, its true, and fine that way. Too bad there's also a planet.


There is nothing stopping governments from running similar projects to provide energy. Well yeah. Public opinion.

Population growth drives demand drives westernisation. I don't want to hear about the so called inefficient Western supercomputers powered by the evil nuclear when SEA exists.

Anyway, veering slightly off topic.

And so is energy usage. That's the point, and the problem. Its also paradoxical, to solve 'ever more complicated problems' while not getting even remotely in the vicinity of solving the biggest problem. In fact, AI nor companies ever want to fix that problem, because they don't benefit from it.

I'm not opposing the technicality of efficiency increases. But its not bringing us anywhere. Its just a gateway to do more, and 'more' is like a drug humans can't seem to stop taking. We're addicts and we'll find ways to keep using.

The alternative is "Industrial society and it's future" which some people found too extreme, but has predictions that certainly came true since publishing.

 

[evernessince]

Basically you will need your own Nuclear Power Plan to run a Datacenter with those abominations.
This is not even a sarcastic remark sadly...

Hence why three-mile island is being recommissioned and other nuclear plants are planned. The former is a project sponsored by Microsoft. Honestly a much better alternative than doing what the Chinese are doing and firing up coal plants.

 

[Hakker]

Of course not.

The work is going to become economically more feasible so more of it gets done and that in turn drives growth, ad infinitum.
This is what progress in tech ultimately brings us. It opens up new avenues, opportunities, possibilities... and new ways to use all that power.

Car analogy works here. As cars get more efficient, they've also progressively gotten heavier and bulkier. You can chalk that up to 'demand'... but demand is created. And it is created because it is enabled.

The bottom line is, progress leads to more power usage.

The bottom line is this - note the acceleration as well. We're not just growing, there is an escalation of growth since around 2000. And that is despite all the fantastic green products we have.

View attachment 403875

This whole idea technological progress leads to savings is top level bullshit that never got proven in real life. Its a utopian thought that allows us to keep consuming and developing; wishful thinking. And in isolation, its true, and fine that way. Too bad there's also a planet.


There is nothing stopping governments from running similar projects to provide energy. Well yeah. Public opinion.

Funny you are proving his point exactly. Europe and NA power usage has stayed somewhat the same over the past 25 years yet computational speed has increased dramatically during that time hence more work got done with the same amount of power.
Heck even the 2nd slide shows it. 50x the Total Bandwidth, 20x the HBM capacity against 7x the power requirements.

 

[dgianstefani]

Funny you are proving his point exactly. Europe and NA power usage has stayed somewhat the same over the past 25 years yet computational speed has increased dramatically during that time hence more work got done with the same amount of power.
Heck even the 2nd slide shows it. 50x the Total Bandwidth, 20x the HBM capacity against 7x the power requirements.

Yep. Plus EU/NA population has increased over this time (due to migration). So per capita power usage has decreased. You can see this in all sorts of similar charts graphs etc. Not really the West driving these explosions of resource usage. We did that decades ago when we invented all this tech, and now do things a bit more efficiently.

I think France is a decent model with their nuclear focus, cheap clean energy etc. Shame more countries didn't do that during the golden years when it was affordable. 10-20% power from nuclear on average is just too low.

AI might have interesting second order effects, on paying for small scale modular nuclear models to actually be built, that can later be scaled up, funny how that works.

 

[Vayra86]

Good point and also true. But what matters is global and we do outsource a lot too...

 

[TheinsanegamerN]

Good point and also true. But what matters is global and we do outsource a lot too...

Well, if computers and their energy use is a major problem, I have a pretty good solution. Find your local Amish community. At some point, the buck has to stop.

Yep. Plus EU/NA population has increased over this time (due to migration). So per capita power usage has decreased. You can see this in all sorts of similar charts graphs etc. Not really the West driving these explosions of resource usage. We did that decades ago when we invented all this tech, and now do things a bit more efficiently.

I think France is a decent model with their nuclear focus, cheap clean energy etc. Shame more countries didn't do that during the golden years when it was affordable. 10-20% power from nuclear on average is just too low.

AI might have interesting second order effects, on paying for small scale modular nuclear models to actually be built, that can later be scaled up, funny how that works.

France absolutely did it right and the dumbest thing the rest of the globe ever did was allowing the Environmentalists to spook everyone with Chernobyl.

Coal/Natgas power should have been outmoded by 1990.

Hence why three-mile island is being recommissioned and other nuclear plants are planned. The former is a project sponsored by Microsoft. Honestly a much better alternative than doing what the Chinese are doing and firing up coal plants.

I swear, if MS can manage that in a reasonable timeframe, the "its too expensive" crown wont have a leg to stand on.

 

[Caring1]

And that more complicated AI will solve more problems, which is also efficiency increase. It is an endless loop. Efficiency is always increasing.

The more you buy, the more you save.

 

[AleksandarK]

The more you buy, the more you save.

We may laugh, but somehow that almost always makes sense in scaling compute.

 

[Hakker]

Good point and also true. But what matters is global and we do outsource a lot too...

You're point still isn't true. The current smartphones have roughly 3000 times more computational power than a Cray supercomputer from 1985. That thing ate your house in space and ate juice enough for an electricity station. Now you have way more computational power dangling near your swimmers that you power with a battery!!!!