What is GPU?

A GPU, short for graphics processing unit, is a processor with thousands of small cores that all work at once. It was invented to draw video game graphics, which means doing the same math on millions of pixels. It turns out AI needs the same kind of math.

Why AI runs on GPUs

An AI model is mostly huge grids of numbers multiplied together. A normal computer processor (CPU) has a few powerful cores that handle one complicated task after another. A GPU is the opposite: thousands of simple cores that each handle a small piece of the same job in parallel.

Think of a CPU as one expert accountant and a GPU as a room of a thousand clerks who each add one column. For a job that splits into many identical small tasks, the room wins easily.

Where you meet GPUs

Two stages of an AI model's life need them. Training a model, such as pretraining, can use thousands of GPUs wired together for weeks. Running the finished model to answer your prompt, called inference, uses far fewer, but it still needs them to answer quickly.

When a company talks about "compute" or "capacity", it usually means how many GPUs it can get. The chips are expensive, and supply has often limited how fast labs can grow.

What limits a GPU

Speed is only part of it. Memory matters just as much. A model's weights must fit in the GPU's own memory, called VRAM, or be split across several GPUs. Newer AI chips use high-bandwidth memory to feed the cores data fast enough. This is why a bigger model, one with more parameters, needs more or better GPUs, and why quantization is popular: smaller numbers mean the model fits on cheaper hardware.

GPUs and other AI chips

GPUs are not the only AI hardware. Companies also design custom accelerators built only for AI math, such as Google's TPUs and Amazon's Trainium. Those can be cheaper or more efficient for some jobs, but they usually need software that supports them. Most AI software was first written for Nvidia GPUs through a toolkit called CUDA, which is a big reason Nvidia's chips remain the default.

Why it matters to you

You rarely buy a GPU yourself. You rent time on one, directly through a cloud provider or indirectly through the per-token price of an AI service. When prices fall or a model gets faster, GPU supply and efficiency are usually part of the reason. If you run models on your own computer, your graphics card's memory decides which models you can use.