Micron is reportedly exploring the idea of developing high-endurance NAND Flash modules that would be positioned closer to the GPU, rather than being located far away in the storage pool behind multiple protocols. The company is said to be investigating ways to bring NAND Flash closer to handle workloads that don't require the latency and bandwidth of traditional HBM or regular DRAM modules. This architecture, currently referred to as "near-GPU NAND," would serve as a middle ground between density and durability. This concept would allow GPUs to access a pool of NAND Flash that has a lower density than conventional TLC or QLC NAND, but instead focuses on improving I/O speeds, bandwidth, and read times.

For instance, this would enable a GPU to access a potential storage pool with hundreds of gigabytes of storage directly on the GPU package or PCB, similar to what DRAM like HBM or GDDR7/LPDDR6 already does. This tier would sit between GPU memory and regular system storage, acting as a very fast buffer, which would be ideal for the inference of massive LLMs. At that point, LLMs wouldn't be memory-bound and could run on systems with fewer GPUs, but with more storage and memory tiers to compensate for the required inference size. Additionally, this NAND would be significantly cheaper than HBM or regular DRAM and would be sized according to system specifications. Images below are HBF slides from SK hynix and Sandisk.

Kioxia Corporation, a world leader in memory solutions, has successfully developed a prototype of a large-capacity, high-bandwidth flash memory module essential for large-scale artificial intelligence (AI) models. This achievement was made within the "Post-5G Information and Communication Systems Infrastructure Enhancement R&D Project (JPNP20017)" commissioned by the New Energy and Industrial Technology Development Organization (NEDO), Japan's national research and development agency. This memory module features a large capacity of five terabytes (TB) and a high bandwidth of 64 gigabytes per second (GB/s).

To address the trade-off between capacity and bandwidth that has been a challenge with DRAM-based conventional memory modules, Kioxia has developed a new module configuration utilizing daisy-chained connections with beads of flash memories. We have also developed high-speed transceiver technology enabling bandwidths of 128 gigabits per second (Gbps), along with techniques to enhance flash memory performance. These innovations have been effectively applied to both memory controllers and memory modules.

Sandisk Corporation (NASDAQ: SNDK) today announced it has signed a landmark Memorandum of Understanding (MOU) with SK hynix to work together to establish the specification for High Bandwidth Flash - a new technology designed to deliver breakthrough memory capacity and performance for the next generation of AI inferences. Through this collaboration, the companies expect to standardize the specification, define technology requirements and explore the creation of a technology ecosystem for High Bandwidth Flash.

"By collaborating with SK hynix to define the High Bandwidth Flash specification, we are addressing the critical need for scalable memory in the AI industry," said Alper Ilkbahar, Executive Vice President and Chief Technology Officer, and HBF Technical Advisory Board member at Sandisk. "This collaboration accelerates innovation and will offer the industry new tools to handle the exponential data demands of tomorrow's applications. Our work will help provide an effective solution to meet the world's technology needs and exceeds expectations of our respective customers."

Sandisk Corporation today announced the formation of a Technical Advisory Board to guide the development and strategy of its groundbreaking High Bandwidth Flash (HBF) memory technology. The board includes industry experts and senior technical leaders from both within and outside the company. Appointed today, Professor David Patterson and Raja Koduri will provide strategic guidance, technical insight, market perspective, and shape open standards as Sandisk prepares to launch HBF.

"We're honored to have two distinguished computer architecture experts join our Technical Advisory Board," said Alper Ilkbahar, Executive Vice President, Chief Technology Officer, and HBF Technical Advisory Board member at Sandisk. "Their collective experience and strategic counsel will be instrumental in shaping HBF as the future memory standard for the AI industry, and affirming we not only meet but exceed the expectations of our customers and partners."

In a joint briefing this week, KAIST's Memory Systems Laboratory and TERA's Interconnection and Packaging group presented a forward-looking roadmap for High Bandwidth Memory (HBM) standards and the accelerator platforms that will employ them. Shared via Wccftech and VideoCardz, the outline covers five successive generations, from HBM4 to HBM8, each promising substantial gains in capacity, bandwidth, and packaging sophistication. First up is HBM4, targeted for a 2026 rollout in AI GPUs and data center accelerators. It will deliver approximately 2 TB/s per stack at an 8 Gbps pin rate over a 2,048-bit interface. Die stacks will reach 12 to 16 layers, yielding 36-48 GB per package with a 75 W power envelope. NVIDIA's upcoming Rubin series and AMD's Instinct MI500 cards are slated to employ HBM4, with Rubin Ultra doubling the number of memory stacks from eight to sixteen and AMD targeting up to 432 GB per device.

Looking to 2029, HBM5 maintains an 8 Gbps speed but doubles the I/O lanes to 4,096 bits, boosting throughput to 4 TB/s per stack. Power rises to 100 W and capacity scales to 80 GB using 16-high stacks of 40 Gb dies. NVIDIA's tentative Feynman accelerator is expected to be the first HBM5 adopter, packing 400-500 GB of memory into a multi-die package and drawing more than 4,400 W of total power. By 2032, HBM6 will double pin speeds to 16 Gbps and increase bandwidth to 8 TB/s over 4,096 lanes. Stack heights can grow to 20 layers, supporting up to 120 GB per stack at 120 W. Immersion cooling and bumpless copper-copper bonding will become the norm. The roadmap then predicts HBM7 in 2035, which includes 24 Gbps speeds, 8,192-bit interfaces, 24 TB/s throughput, and up to 192 GB per stack at 160 W. NVIDIA is preparing a 15,360 W accelerator to accommodate this monstrous memory.

During its first post-Western Digital spinoff investor day, SanDisk showed something it has been working on to tackle the AI sector. High-bandwidth flash (HBF) is a new memory architecture that combines 3D NAND flash storage with bandwidth capabilities comparable to high-bandwidth memory (HBM). The HBF design stacks 16 3D NAND BiCS8 dies using through-silicon vias, with a logic layer enabling parallel access to memory sub-arrays. This configuration achieves 8 to 16 times greater capacity per stack than current HBM implementations. A system using eight HBF stacks can provide 4 TB of VRAM to store large AI models like GPT-4 directly on GPU hardware. The architecture breaks from conventional NAND design by implementing independently accessible memory sub-arrays, moving beyond traditional multi-plane approaches. While HBF surpasses HBM's capacity specifications, it maintains higher latency than DRAM, limiting its application to specific workloads.

SanDisk has not disclosed its solution for NAND's inherent write endurance limitations, though using pSLC NAND makes it possible to balance durability and cost. The bandwidth of HBF is also unknown, as the company hasn't put out details yet. SanDisk Memory Technology Chief Alper Ilkbahar confirmed the technology targets read-intensive AI inference tasks rather than latency-sensitive applications. The company is developing HBF as an open standard, incorporating mechanical and electrical interfaces similar to HBM to simplify integration. Some challenges remain, including NAND's block-level addressing limitations and writing endurance constraints. While these factors make HBF unsuitable for gaming applications, the technology's high capacity and throughput characteristics align with AI model storage and inference requirements. SanDisk has announced plans for three generations of HBF development, indicating a long-term commitment to the technology.