What is Tokens per second?

Tokens per second, often written tok/s or TPS, tells you how quickly an AI model writes its output. Since a token is a chunk of text, roughly three quarters of an English word, a model running at 50 tokens per second writes a few dozen words every second.

What it measures

The number usually refers to the model's output speed once it starts answering. This is the "decode" stage described in prefill and decode. It is different from the wait before the first word appears, which is called time to first token and depends mostly on how long your prompt is.

So two speeds matter:

• Time to first token: how long you wait before anything shows up.
• Tokens per second: how fast the rest streams in.

A system can be quick to start and slow to finish, or the reverse. Always ask which one a claim refers to.

What controls it

Several things set the speed:

• Hardware. Generating each token requires reading the model's weights from memory, so memory bandwidth is often the limit. A fast GPU with fast memory generates faster.
• Model size. More parameters means more to read and compute per token, so bigger models are slower on the same machine.
• Precision. Quantization shrinks the weights, which usually raises the speed.
• Software tricks. Techniques like speculative decoding let a small model draft tokens that a large model checks in bulk.
• Load. On a shared service, many users at once can slow each person's stream.

Why you should care

For reading a chat answer, anything faster than you read feels instant, and a few dozen tokens per second is plenty. For agents and reasoning models, speed matters more, because they may write thousands of hidden tokens before they answer. A slow model turns a one-minute task into a ten-minute one.

It also affects cost. Providers often charge per token, so a model that generates more tokens costs more, even if each one is cheap. A reasoning model that thinks at length can use many times the tokens of one that answers directly.

How to read a claim

When a vendor or reviewer quotes a speed, check the model, the precision, the length of the prompt and the hardware. A figure for a small model says nothing about a large one. For local AI, wait for independent tests on the exact model you plan to run.