The Memory Wall: HBM, CXL and In-Memory Computing

What the memory wall is

The term comes from a 1995 paper by Wulf and McKee. Their point was that processor speed had been improving faster than memory speed, so memory would eventually dominate.

Two numbers matter. Bandwidth is how much data memory can deliver per second. Latency is how long a single request takes. A chip with enormous arithmetic power but modest bandwidth will spend much of its time waiting.

Moving data also costs energy. Reading a value from off-chip memory takes far more energy than doing arithmetic on it. For many AI workloads, data movement is the larger part of the power bill.

Why AI makes it worse

Large language models store billions of numbers called weights. To produce each new token, a unit of text, the chip generally has to read most or all of the weights from memory.

Here is a simple illustration. A model with 70 billion weights stored in 16-bit form takes about 140 gigabytes. If memory delivers 3 terabytes per second, reading all the weights once takes about 47 milliseconds. That sets a floor on the time to produce one token for a single request, however fast the arithmetic is. The figures are illustrative, not those of any specific chip.

Engineers describe this with arithmetic intensity, the number of calculations done for each byte fetched. Generating text token by token has low arithmetic intensity, so it is memory-bound. Processing a long prompt has higher intensity and is more often limited by arithmetic.

SRAM and DRAM

Two memory types matter here. SRAM is built from several transistors per bit. It is very fast and sits on the chip, but it takes a lot of area, so capacities are limited to tens or hundreds of megabytes on a typical processor.

DRAM stores each bit with a single transistor and a tiny capacitor. It is far denser and cheaper per bit, but it is slower and usually sits off the chip.

Nearly every fix below is a different way of getting more of the right memory closer to the compute.

HBM: stacking memory next to the chip

High bandwidth memory, or HBM, stacks several DRAM dies on top of each other and connects them with vertical wires called through-silicon vias. The stack is placed beside the processor on a shared base layer, called an interposer, using advanced packaging.

The key feature is width. Each HBM3 stack has a 1,024-bit interface, many times wider than a standard memory chip. HBM3 offers roughly 800 gigabytes per second per stack, and HBM3E roughly 1.2 terabytes per second. HBM4 doubles the interface to 2,048 bits. A processor typically carries several stacks.

HBM has limits. It is expensive, supply is concentrated among three manufacturers, SK hynix, Samsung and Micron, and the advanced packaging it needs has been a bottleneck. Stacks also generate heat and offer limited capacity per processor.

CXL: more capacity, not more speed

Compute Express Link, or CXL, is an open standard that uses the same physical connection as PCIe. It lets processors share and expand memory. Memory can sit in a separate box or module and be pooled between servers.

CXL does not match the bandwidth or latency of HBM. Its value is capacity. It can add large amounts of DRAM to a system at a lower cost, which helps with large databases and with serving big models where capacity matters more than peak speed.

Companies in this area include Astera Labs, Panmnesia and Rambus, along with software firms such as MemVerge.

Big on-chip memory

Another approach avoids off-chip memory by placing much more SRAM on the chip. Cerebras builds a processor the size of a whole silicon wafer with tens of gigabytes of on-chip SRAM. Groq built chips around SRAM and a scheduled data flow. The trade-off is that SRAM holds much less than DRAM, so large models must be split across many chips.

In-memory computing

In-memory computing does the arithmetic where the data is stored, so the data does not travel at all.

Analog designs store weights in a grid of memory cells such as flash, resistive RAM or phase-change memory. Applying voltages along the rows produces currents that add up along the columns, which performs a matrix-vector multiplication using Ohm's law and Kirchhoff's current law. The difficulties are noise, device drift over time, limited precision and the cost of converting signals to and from digital form. Mythic and Encharge AI are examples.

Digital designs put logic next to or inside SRAM arrays and avoid analog noise. d-Matrix is one example.

Processing-in-memory, or PIM, adds simple processors to DRAM itself. UPMEM did this on standard memory modules, and memory makers have shown PIM variants of HBM.

Software and model fixes

Not all fixes are hardware. Lower numerical precision, such as 8-bit or 4-bit weights, shrinks the data to read. Batching serves many requests per read of the weights. Mixture-of-experts models activate only part of the network per token. Compressing the key-value cache, which stores earlier context, saves memory too.

In practice, real products combine several of these. When judging a chip, always ask for memory capacity and bandwidth next to its peak compute number.