What Is RAG in AI: A Simple Definition and What It's For

RAG (retrieval-augmented generation) is a technique that lets a language model answer using documents it never saw during training. First the chunks relevant to the question are retrieved, then they’re passed to the model along with the question, and the model writes the answer with that text in front of it.

To follow the rest, you need one piece of background: what a language model is and why it makes things up. If that’s not clear yet, start with what an LLM is, explained without the jargon and come back here.

The problem it solves: the model doesn’t know your data

A language model only knows what was in the text it trained on, and that training ended on a specific date. That’s where the two gaps RAG is meant to close come from: it hasn’t read your company’s private documents, and it knows nothing about what’s happened since its cutoff date.

Picture someone who has memorized half the internet but has never opened your company’s Drive. You ask them about the vacation policy and they don’t say “I don’t know”: they answer with whatever policy sounds most plausible, with the same confidence they’d use to tell you the capital of France. That’s a hallucination.

Retraining the model every time someone uploads a PDF isn’t viable. So the alternative is to leave the model as it is and hand it the right text at the moment you ask.

How RAG works under the hood

RAG is three steps, and none of them touch the model.

The first is retrieve. When the question comes in, a retriever goes through your document collection and pulls out the chunks that talk about that topic. It’s not an exact word-for-word search like Ctrl+F: the usual approach is embeddings, numeric representations of a text’s meaning. With them, “how many days off do I have left” finds a document about “annual leave allowance” even though they don’t share a single word.

The second is augment, which is where the technique gets its name. The retrieved chunks get pasted into the prompt — the text you send the model. The question no longer travels alone: it travels along with the material it should be answered from.

The third is generate. The model does what it always does, writing the most plausible continuation, except now the most plausible continuation is the one grounded in the text sitting in front of it.

// The three steps of RAG, no libraries involved
const question = "How many vacation days do I have left this year?";

// 1. Retrieve: search your documents for the 4 most relevant chunks
const chunks = await searchDocuments(question, { top: 4 });

// 2. Augment: put those chunks into the prompt, before the question
const prompt = `Answer using ONLY this context:
${chunks.join("\n---\n")}

Question: ${question}`;

// 3. Generate: the model writes with the context already in front of it
const answer = await model.complete(prompt);

That one-line searchDocuments is where all the real difficulty concentrates: how to chunk documents, how to combine semantic search with keyword search, how to rerank results before handing them to the model. That’s covered in the guide to a RAG pipeline that works in production, and you don’t need it to understand what RAG is.

An example: the support assistant that actually knows your product

Think of a support chat for a product that shipped a new version two months ago. Without RAG, the model answers with whatever it learned during training and describes an outdated version. With RAG, the question triggers a search over the current documentation, the two or three articles that cover the topic go into the prompt, and the answer comes out matching the behavior of the version released today. And since you know exactly which chunks you handed it, you can show the sources underneath: the user no longer has to just trust the model and can go check for themselves.

When RAG is worth it, and when it isn’t

RAG makes sense when the knowledge you need is private, changes often, or has to be verifiable. If none of those conditions apply, it’s usually complexity that buys you nothing.

Situation	RAG?
Questions about internal documentation that changes every week	Yes, this is exactly the case it was built for
You need to show where each answer came from	Yes: since you control which chunks go in, you can cite them
You want to change the tone or format the model writes in	No. That’s adjusted with prompt instructions, or fine-tuning if the style needs to be very specific
General, stable knowledge: grammar, history, popular programming languages	Not needed, the model already has it built in
You have four PDFs and check them once a month	Probably not. Just paste them into the conversation and be done with it

That last row is the one that saves the most money. Building a semantic search system for a handful of documents that fit in a single conversation is wasted work.

What RAG doesn’t fix

RAG shifts the problem rather than eliminating it: the quality of the answer now depends on the quality of what the retriever finds. If the search returns the wrong chunk, the model will write a wrong answer with the same confident tone as always. Most failures in a RAG system aren’t failures of the model. They’re failures of retrieval.

It also doesn’t save you from having to keep your documents in order. A wiki with three contradictory versions of the same policy hands the retriever three equally valid candidates, and the model will pick one without warning you about the conflict.

If you want to see how these pieces, RAG included, fit inside a complete AI agent, the course AI Agent Design Patterns walks through them with interactive exercises.

Frequently asked questions

Is RAG the same as fine-tuning?

No. RAG doesn’t touch the model: it changes what the model has in front of it when answering, by putting chunks of your documents into the prompt. Fine-tuning does modify the model, training it on additional examples. Rule of thumb: if the problem is that the model doesn’t know something, use RAG; if the problem is how it says something, use fine-tuning.

When someone says “a RAG” in a meeting, what do they mean?

In AI, “a RAG” is the whole system, not the technique. “We’ve built a RAG” means there’s an indexed document collection, a retriever that queries it, and a model that writes the answer using whatever the retriever returns.

Do I need RAG if the model already knows a ton?

It depends on whether your questions are about public knowledge or your own. A large model can answer plenty about what was on the internet before its training, and it knows nothing about your contract with a vendor or last week’s production incident. RAG is only needed for the second kind.

What is agentic RAG?

It’s the version where the model decides when to search, what to search for, and whether what it found is good enough, instead of always running a fixed search before answering. It can rephrase the query, run several searches in a row, or conclude it doesn’t need one at all. It’s RAG placed inside an agentic loop.

Does RAG eliminate hallucinations?

No. It reduces them a lot: anchoring the answer in a specific text removes most of the fabrication, but the model can still blend two chunks together or extrapolate beyond what they say. That’s why serious systems show their sources — so the error can be caught.