What is Retrieval-augmented generation?

What it is

Retrieval-augmented generation, usually shortened to RAG, is a way of giving an AI model the right reading material at the moment you ask a question. Instead of relying only on what the model absorbed in training, the system searches a collection of documents, picks the most relevant passages, and places them in the prompt. The model then writes its answer using that text.

Think of an open-book exam. A model without retrieval has to answer from memory. A model with retrieval gets to flip to the right page first.

How it works

Most RAG systems follow the same steps:

1. Prepare. Documents are split into chunks. Each chunk is turned into a list of numbers called an embedding, which captures its meaning, and stored in a searchable index.
2. Retrieve. When you ask a question, the system turns it into an embedding too and finds the chunks closest in meaning. Some systems also use ordinary keyword search, or both.
3. Augment. The best chunks are added to the prompt, so they take up space in the context window.
4. Generate. The model answers, ideally quoting or citing the chunks it was given.

Web search inside a chatbot is the same idea with the whole internet as the collection.

Why it matters

RAG solves three common problems. A model has a knowledge cutoff and knows nothing newer, but retrieval can supply fresh facts. A model has never seen your private files, but retrieval can bring them in without retraining. And a model that is shown the source is less likely to produce a hallucination, and you can check where an answer came from.

It is also usually cheaper and faster to update than fine-tuning. To change what the system knows, you change the documents, not the model.

Where it goes wrong

RAG is only as good as its retrieval. If the search pulls the wrong chunks, the model confidently answers from the wrong material. Chunks that are too small lose context, and chunks that are too large waste space. The model can also ignore the supplied text or blend it with its own guesses.

Retrieved text is untrusted input, too. A web page or document can contain instructions meant for the model, which is how prompt injection reaches RAG systems.

RAG compared with a long context window

When models accept very long inputs, you can sometimes paste whole documents in and skip retrieval. That works for a handful of files. For a large or changing collection, retrieval still wins on cost and speed, because you send only the relevant pieces instead of everything on every request. Many AI agents use both, searching first and then reading the best results in full.