Definition
Prefill and decode is Prefill and decode are the two stages of running a language model: prefill reads your whole prompt at once, and decode then writes the answer one token at a time.
Every time you send a prompt to a language model, inference happens in two stages.
Prefill is the reading stage. The model takes your entire prompt, including any documents and chat history, and processes it in one big pass. All of the tokens can be handled in parallel, so this stage keeps a GPU busy doing math. It ends when the model is ready to write its first word. The wait before that first word is what people call time to first token.
Decode is the writing stage. The model produces its reply one token at a time, because each new token depends on the one before it. This stage does less math per step but has to keep reading the saved state of the conversation from memory, so it is limited by memory speed rather than raw computing power.
The two stages stress hardware in opposite ways. Prefill wants lots of computing power. Decode wants fast memory. Running both on the same chip means one of them is always wasting some of it, and a long prefill can make other users' replies stutter.
Some providers therefore run them on separate groups of GPUs. This is often called disaggregated serving. One group does prefill, and then hands the model's saved working state, the KV cache, to another group that does decode. Each group can be sized and tuned for its own job.
The handoff is the tricky part. The cache can be many gigabytes, and it has to arrive intact and be tied to the right request. Mistakes there are rare, but they are a different kind of bug from ordinary model errors.
You see the two stages as two numbers on a provider's dashboard. Latency before the first word comes mostly from prefill, so it grows with prompt length. The speed of words appearing afterward comes from decode.
Long prompts are expensive mostly in prefill. Long answers are expensive mostly in decode, and a reasoning model that thinks at length spends a lot of time there.
Techniques such as prompt caching cut prefill work for repeated input, and speculative decoding speeds up the decode stage.
Prefill is the stage where the model reads your whole prompt in one pass before it writes the first word of its answer.
Decode produces one token at a time and must read the saved conversation state from memory at every step, so memory speed becomes the limit.
Cloudflare says a misconfiguration and an SGLang bug let 155 GLM 5.3 responses on Workers AI include data from other customers' requests between Aug. 27 and Sept. 9. Shared inference has a new failure mode, and long prompts found it.