What is Test-time compute?

Test-time compute is the amount of computing a model uses at the moment it answers you. It is also called inference-time compute. The idea: instead of only making a model bigger or training it longer, you let it spend more effort on each hard question.

How it works

Most of the cost of building a model is paid up front, during pretraining. Test-time compute is the other budget, the one you pay on every request. A model can spend that budget in a few ways.

• Thinking longer. The model writes out a long chain of thought before it answers, working through steps and checking itself. This is what a reasoning model is trained to do.
• Trying several times. The model produces many candidate answers and a checker, or a vote, picks the best. A simple version is best-of-N: generate N answers and keep the one that scores highest.
• Searching. The model explores different lines of reasoning and uses a verifier to steer toward promising ones.

All of these use more tokens and more time, which is why they show up on your bill and in your wait.

Why it matters

It changes the trade-off between model size and effort. A research paper from Snell and colleagues, "Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning," found that on certain math problems a smaller model given extra test-time compute could beat a model 14 times larger under matched compute. That result covered math reasoning, so it should not be read as true for every task.

It also means difficulty matters. The same paper found the best way to spend the extra effort depends on how hard the question is. An easy question does not need a long think, and a hard one may.

For you as a user, this explains the "thinking" or "reasoning effort" setting many AI tools now offer. More effort usually means a slower, costlier answer that is more likely to be right on hard problems like math, coding and planning. For simple requests it is often wasted.

Related ideas

Test-time compute sits next to scaling laws, which describe how results improve with more training compute, data and size. It adds a second dial: spend more at inference time. Together they describe the two main ways to buy better answers.

A simple example

Ask a model for a quick dinner idea and it answers in a second. Ask it to find the bug in a 300-line program and, with a high reasoning setting, it might think for a minute, test several hypotheses and check its own fix. Same model, very different spend. That extra minute is test-time compute.

What to remember

More thinking is not free and not always better. Match the effort to the difficulty of the job, and test it on your own tasks before paying for the highest setting.