What is High-bandwidth memory?

High-bandwidth memory, or HBM, is the fast memory attached to the chips that run AI. It matters because for large models the bottleneck is often not how quickly a chip can calculate, but how quickly it can be fed data.

What it is

Ordinary computer memory (DRAM) sits in separate modules, connected to the processor over a relatively narrow path. HBM takes memory chips and stacks them vertically, like floors of a building, then places the stack beside the processor on the same package. Because the stacks connect over a very wide interface across a short distance, data moves much faster and uses less energy per bit moved.

Bandwidth is the amount of data that can flow per second, usually quoted in terabytes per second. Newer HBM generations are numbered, such as HBM3 and HBM4, with each generally offering more capacity and bandwidth than the last.

Why AI needs it

A large model is made of billions of parameters. To produce each token of an answer, the chip must read a large share of those numbers from memory. If memory cannot deliver them fast enough, the processor's math units sit idle, waiting. Think of a fast chef with a tiny pantry door: the cooking is quick, the ingredient delivery is the limit.

This is especially true for inference, where answering one user's request can be limited by memory bandwidth rather than raw compute. It is also a reason models with a mixture of experts design still need plenty of fast memory: all the experts must be stored, even if only a few run per token.

Capacity matters too

HBM also determines how much fits on a chip. The model's weights have to be held somewhere, and the context window of a long conversation takes memory as well. Bigger models and longer contexts need more of it, or need many chips working together.

Why it shows up in the news

HBM is made by only a few companies and is hard to produce, since stacking and packaging the chips is complex. That makes it a frequent constraint on how many AI accelerators can be built. When a chip maker announces a new AI processor, the number of HBM stacks and the memory bandwidth are headline specifications alongside raw compute.

What it means for you

You rarely choose HBM yourself. It shows up indirectly: in how fast a model responds, how long a context it can handle, and what a provider charges per token. Better memory bandwidth is one of the main ways hardware makers claim lower serving costs.