What is Transformer?

The transformer is a neural network architecture introduced in 2017 in a Google research paper titled "Attention Is All You Need." It is the design underneath most of today's large language models, and it is also used in many image, audio and video models.

The key idea: attention

Older language models read text mostly in order, one word after another, and tended to lose track of what came early. A transformer instead uses attention. For each token in the text, the model works out how much every other token matters to it, then blends that information together.

Take the sentence "The trophy did not fit in the suitcase because it was too big." To know what "it" refers to, the model has to connect "it" with "trophy," not "suitcase." Attention is the mechanism that lets the model make that link, even when the two words are far apart.

How it is built

A transformer stacks many identical layers. Each layer has an attention step and a small feed-forward network that processes each token. Text first becomes tokens, and each token is turned into an embedding, a list of numbers. The layers refine those numbers step by step until the model can predict the next token. Everything it learned is stored in its parameters.

Why it took over

Transformers process all the tokens in a prompt at the same time during the reading step, which suits GPUs that do many calculations in parallel. That made it practical to train enormous models on enormous data. Reading a prompt in parallel and then writing the answer token by token is the split described in prefill and decode.

The cost

Attention compares tokens with each other, so the work grows quickly as the text gets longer. This is one reason the context window is limited, and why systems keep a KV cache so they do not redo earlier work. Many designs reduce the cost, such as mixture of experts, which activates only part of the network for each token.

Why it matters

You do not need to know the math to use AI well. But the transformer explains several things you see in practice: why a model can use a whole document at once, why longer prompts cost more and run slower, and why there is a hard limit on how much it can read.

Related terms

A transformer is the architecture. Pretraining is how it learns. A diffusion model is a different approach to generating images, although some use transformer layers inside. The older convolutional neural network was the standard for images before transformers spread.