What is Speculative decoding?

Speculative decoding makes a large language model answer faster without changing what it says. It works by pairing the big model with a much smaller helper.

Why generation is slow

A language model writes one token at a time. Each new token needs a full pass through the model, and that pass spends most of its time moving the model's weights through memory, not doing math. So a big model is slow per token even when the hardware has plenty of spare compute.

How the trick works

Think of a junior writer drafting a few sentences ahead while a senior editor reads them in one sitting.

1. Draft. A small, fast "draft" model guesses the next several tokens.
2. Verify. The large "target" model scores all of those guesses in a single pass. Checking several tokens at once costs about the same as generating one.
3. Accept or correct. The target keeps the guesses it agrees with. At the first guess it would not have made, it substitutes its own token and the loop starts again from there.

When the draft guesses well, you get several tokens for the price of one big-model pass. When it guesses badly, you lose little, because the target still produces the right token at that point.

The key property is that the output is statistically the same as running the large model alone. The method was described in 2022 by researchers at Google (Leviathan, Kalman and Matias) and, at nearly the same time, by a DeepMind team. Reported speedups are commonly in the range of 2x to 3x, depending on how often the draft matches the target.

What affects the speedup

• Draft quality. Predictable text, like boilerplate code or a formatted list, is easy to guess. Open-ended creative writing is harder.
• Draft cost. The helper has to be much cheaper than the target, or the savings vanish.
• Serving setup. It pays off most when you serve one request at a time and the hardware is waiting on memory.

Why you might care

If you run inference yourself, speculative decoding is one of the few speedups that costs you no quality. Some open model releases ship a ready-made draft model for exactly this reason. It sits alongside other ways to cut cost, such as quantization and distillation, and you can combine them.