NVIDIA's and AMD's next-generation accelerators will definitely utilize a next-generation HBM4 memory, too. However, interesting news, confirmed by SemiAnalysis, is that these accelerators will utilize a customized HBM memory for the first time. In an HBM stack, consisting of stacked DRAM dies up to a 12-high, connected with TSVs, there is the possibility of embedding an optional base die with customized logic/accelerator circuitry tailored to a specific need. It appears that NVIDIA and AMD are not only exploring this option to gain more performance, but are also actively working on an implementation with a custom base that will push these accelerators ahead of any third-party ASIC solution by a third. Since everyone is trying to grab a bit of market share in the AI ASIC world, NVIDIA and AMD have to innovate their designs to stay ahead constantly.

"We expect NVIDIA and AMD to come out with their custom HBM implementations with HBM4, with other accelerator designers likely to only have a custom base die implementation by HBM4E in the 2027 timeframe," notes the SemiAnalysis post. This means that while ASIC designers, such as Broadcom and MediaTek, will have a working solution in 2027 chips, NVIDIA and AMD will have a working custom HBM4 in 2026 products. For NVIDIA, this is the "Rubin" architecture, supposed to outperform the current "Blackwell" implementation by a wide margin. For AMD, this is the Instinct MI400 series of accelerators, which is coming for the first time in a rack-scale flavor.Companies usually opt for off-the-shelf HBM memory for the standardized pricing model implemented across suppliers like Samsung, Micron, and SK Hynix. However, when ordering quantities so large, like NVIDIA and AMD do, they can request special accommodation features. Since the HBM is open to the base die modification, NVIDIA and AMD prepared their own modifications. While this may not be necessarily a compute die that brings TeraFLOPS of power, it will likely be a data processing/logic die that helps route data packets more efficiently, cutting latency and improving performance. Especially during inference, where latency is the most important factor, having a "smarter" HBM could yield sizable, double-digit gains in throughput of tokens.