Definition
KV cache is a KV cache is the working memory an AI model keeps of the text it has already read, stored on the GPU so it does not have to recompute that text each time it generates a new token.
When a language model reads your prompt, it turns every token into two sets of numbers called keys and values. The model uses them to decide which earlier words matter for the next word. The KV cache (key-value cache) is the saved copy of those numbers.
Think of a chef who writes down the prep work for a recipe. Without the notes, every new step means redoing all the chopping from the start. With the notes on the counter, each step takes a glance.
A model writes its answer one token at a time. For each new token it needs the keys and values of everything before it: your prompt, the chat history and the words it has already written. Recomputing all of that for every token would be painfully slow. So the model computes each token's keys and values once, stores them, and reads them back for every later step.
The cache grows with every token. A short chat uses a little. A prompt of hundreds of thousands of tokens can use many gigabytes. That memory has to sit on the GPU, usually in fast memory such as high-bandwidth memory, next to the model's weights.
The KV cache is a big reason long conversations cost more and slow down. It is also why a long context window is expensive for a provider to offer: each active request holds its own cache in memory while it runs.
Providers can also reuse a cache. If many requests start with the same long text, such as a system prompt, the provider can keep the cache from the first one and skip the work for the rest. That is the idea behind prompt caching, and it is why repeated input can be priced lower.
Because a cache holds a copy of someone's text, a hosted service has to keep each customer's cache separate and clear it properly. A bug that mixes them up can expose one customer's data to another.
Serving systems often split the work into prefill and decode, and the KV cache is what gets handed from one stage to the other. Compression methods such as quantization can shrink the cache as well as the model, so more requests fit on one GPU.
The cache is not a memory in the sense of AI memory. It lasts for one request, or at most a short time on the provider's side, and then it is discarded.
KV stands for key-value. The cache stores the key and value numbers a model computes for every token it has already read.
The KV cache grows with every token in the prompt and the reply, so very long inputs can take many gigabytes of GPU memory.
Apple's new MacBook Pro with M5 Pro or M5 Max adds a touch OLED screen and up to 128GB of memory, from $2,899, shipping Nov. 6. Apple says M5 Max runs models with hundreds of billions of parameters locally but gives no speeds, so wait for real numbers.
Cloudflare says a misconfiguration and an SGLang bug let 155 GLM 5.3 responses on Workers AI include data from other customers' requests between Aug. 27 and Sept. 9. Shared inference has a new failure mode, and long prompts found it.