What is Compute?

What compute means

In AI, compute is shorthand for computing power: the number of calculations a system performs, and the chips and data centers that do them. When someone says a lab is "short on compute," they mean it lacks enough chips running to train or serve the models it wants.

Compute is usually counted in floating-point operations, or FLOPs: individual math steps on numbers. Training a large model takes an enormous count of them, which is why it needs thousands of GPUs working together for weeks or months.

Two places compute gets spent

Training. This is when a model learns. Pretraining on huge amounts of data is the biggest bill, followed by post-training. More compute, used well, tends to produce a more capable model, a pattern described by scaling laws.

Running the model. Every answer you get is inference, and it uses compute too. A short chat reply needs little. A reasoning model that thinks for a long time before answering uses much more, which is the idea behind test-time compute.

Why it matters to you

Compute explains many things that look unrelated:

• Price. Providers charge per token partly because each token costs real chips and electricity.
• Limits. Rate limits and usage caps exist because capacity is finite.
• Speed and availability. When demand spikes, responses slow or features roll out to fewer people.
• Strategy. Labs sign multi-year deals for chips and data centers because whoever has more compute can train bigger models and serve more customers.

An everyday picture

Compute is like electricity for a factory. The factory's output depends on how much power it can draw and how efficiently it uses it. A smarter process, such as distillation or quantization, gets more done from the same power. A bigger power supply lets you build a bigger factory.

Where you can get it

Most teams do not own chips. They buy compute as an API from a model provider, or rent GPUs from a large cloud or a neocloud. Which route is cheaper depends on how steady your workload is and how much control you need.

Related terms

Compute pairs with training data and model size (parameters) as the three ingredients labs trade off when building a model.