Definition
Latency is Latency is the delay between sending a request to an AI model and getting a response back, usually measured as time to the first token and time to the full answer.
Latency is how long you wait. In AI, it is the time between sending a prompt and receiving what the model produces. Low latency feels instant. High latency feels like staring at a spinner.
Most AI products report latency in two parts:
A model can start fast and finish slowly, or the reverse. Which one matters depends on the job. A chat window cares about the first token. A background task that processes a thousand documents cares about total time and cost.
Several things add delay:
Teams shorten prompts, reuse repeated context with prompt caching, pick a smaller model for easy tasks, stream the reply so words appear as they are generated, and use techniques such as speculative decoding. Lowering reasoning effort also trades some depth for speed.
Latency is about one request. Throughput is how many requests a system handles at once. Optimizing for one can hurt the other, and faster service often costs more. For a voice assistant, latency decides whether the product works at all, because a pause of a couple of seconds breaks a conversation. See voice agents for an example.
Published speeds are averages under ideal conditions. Test with your real prompts, at your usual times of day, and record both time to first token and total time. Then choose the model and settings that meet the speed you need at a price you accept.
It is the delay before a model returns the first piece of its answer, and it largely determines whether a chat feels responsive.
Shorten prompts, cache repeated context, use a smaller model for simple tasks, stream the response and lower reasoning effort where depth is not needed.
Mistral released Voxtral 3 on Oct. 20: new transcription and speech models plus a voice agent API at $0.04 a minute when it reasons with Mistral Small 4. Skip the benchmark charts. The per-minute price and the choice of where audio is processed are the story.
OpenAI put its Decisions API in public beta on October 6, with gpt-6-luna at $0.10 per million input tokens and no charge for output. If you classify, route or grade at volume through a chat model, this is the endpoint to price against.
Microsoft says Maia 200 is in production and Maia 300 starts multi-gigawatt deployments in 2027, making its own chip the primary inference system for its products. GitHub Copilot and Foundry move to Maia by year end, so your requests will soon run on Microsoft silicon.
TypeSafe AI raised $870M at a $7.5B valuation for Jev, its decision model. Cloudflare answered with open-weight Clef-omni and a Clef-flash price cut to $0.038 per million input tokens, and a category this young just got a price war.
Microsoft released Decision-1, a Qwen-based model that scores fixed choices, at $0.042 per million input tokens with free output. The benchmarks are Microsoft's own. The price is the reason to move your agent's small calls off big models.