Memory & Storage

Explainer

Memory & Storage explained

Memory and storage hold the data that processors work on. Memory such as DRAM is fast and loses its contents when power is removed. Storage such as flash (NAND) is slower, cheaper per bit and keeps data without power.

For AI chips the key product is HBM, or high bandwidth memory. HBM stacks several DRAM dies on top of each other and connects them with vertical wires called through-silicon vias. The stack sits right beside the processor and delivers far more bandwidth than ordinary memory modules.

How it differs from the usual approach

Processors get faster more quickly than memory does, which leaves chips waiting for data. This is often called the memory wall. Memory technology is therefore judged on bandwidth, capacity, energy per bit and cost, not on raw compute. Many AI workloads, especially generating text from a trained model, are limited by memory bandwidth more than by arithmetic speed.

Where it is used

• HBM beside GPUs and AI accelerators in data centers.
• DRAM modules (DDR) in servers, PCs and phones.
• NAND flash in solid-state drives and phones.
• Embedded non-volatile memory such as MRAM and ReRAM in microcontrollers and chips.
• Memory expansion over CXL to add capacity to servers.

What to watch

• DRAM is made mainly by three companies: SK hynix, Samsung and Micron. HBM supply has been a limiting factor for AI accelerator output.
• New HBM generations arrive on a regular cycle with more bandwidth and taller stacks, and the JEDEC standards body has published a specification for HBM4.
• Stacking more layers, in both DRAM and NAND, is the main way to raise capacity as flat scaling slows.
• Emerging memories such as MRAM, ReRAM and phase-change memory are moving into niche products but have not replaced DRAM or NAND.

Frequently asked questions