Skip to content
LiveNext 10:46:03

Definition

What is Attention mechanism?

Attention mechanism is an attention mechanism is the part of a transformer that lets a model weigh how relevant every other token in the input is to the one it is processing, so it can use context from anywhere in the text.

An attention mechanism is how a modern AI model decides which parts of its input matter for each step. It is the core idea inside the transformer, the design behind nearly every major large language model.

The idea in plain terms

Read this sentence: "The trophy didn't fit in the suitcase because it was too big." To know what "it" means, you glance back at "trophy" and "suitcase" and decide which one fits. Attention does something similar in math. For every token, the model scores how relevant every other token is, then blends information from the most relevant ones.

How it works

Each token is turned into an embedding, a list of numbers. The model then computes three versions of it, usually called query, key and value. Think of the query as a question ("what am I looking for?"), each key as a label on a drawer, and each value as what is inside the drawer. The model compares a token's query against every key, turns the scores into weights that add up to one, and takes a weighted mix of the values.

Models run many of these in parallel, called attention heads. Different heads tend to pick up different relationships, such as grammar, references or nearby words.

Why it matters to you

  • Context length has a cost. Standard attention compares every token with every other token, so the work grows quickly as the context window grows. This is a main reason long inputs are slower and pricier.
  • Memory. To avoid recomputing, models store keys and values in a KV cache, which uses expensive GPU memory.
  • Position matters. Models can attend less reliably to material buried in the middle of a very long prompt, so put key instructions where they are easy to find.
  • Efficiency tricks. Many models use variants that cut the cost, and designs like mixture of experts save compute elsewhere in the network.

What attention is not

Attention is not understanding or memory. It is a calculation that mixes information across the tokens already in the input. Nothing from a past conversation survives unless it is included again, which is why AI memory features are built separately.

If you remember one thing: attention is how a model looks back at its own input. That is why what you put in the prompt shapes the answer so strongly.

Questions people ask

What is attention in a transformer?

It is the calculation that lets each token pull in information from the other tokens in the input, weighted by how relevant they are.

Why do long prompts cost more?

Standard attention compares every token with every other token, so the work and memory grow quickly with input length.