Large language models are trained on a fixed snapshot of text and then frozen. Ask one about your company’s internal API, last week’s earnings call, or anything published after its training cutoff, and it will either say it doesn’t know or, more dangerously, make something up that sounds plausible. Retrieval-augmented generation, or RAG, is the standard architectural answer to that problem. Instead of relying on what the model memorized during training, you hand it relevant source material at the moment of the query and ask it to answer using that material.

The idea is simple enough that it’s easy to underestimate how much engineering sits underneath it.

The basic pipeline

A RAG system has two halves: retrieval and generation. Generation is just the LLM call. Retrieval is where the real work happens.

First, a corpus of documents gets broken into chunks, small enough to fit usefully into a context window, large enough to preserve meaning. Each chunk is run through an embedding model that turns it into a vector, a list of numbers capturing its semantic content. Those vectors get stored in a vector database or a vector index bolted onto an existing database.

At query time, the user’s question is embedded the same way, and the system searches for chunks whose vectors are closest to the question’s vector. Those chunks get stuffed into the prompt alongside the original question, and the LLM generates an answer grounded in that retrieved text rather than purely in its training data.

That’s the core loop. Everything else in a production RAG system is refinement on top of it.

Why not just fine-tune the model instead

Fine-tuning bakes new knowledge into the model’s weights, which sounds like a cleaner solution. In practice it has three problems RAG avoids. Fine-tuning is expensive and slow to iterate on, which matters when your source data changes daily. It doesn’t give you provenance: a fine-tuned model can’t point to which document it got an answer from, which matters a lot for anything involving compliance or user trust. And it doesn’t reliably teach a model new facts so much as nudge its style and behavior, models are notoriously bad at absorbing precise, low-frequency information through fine-tuning alone.

RAG sidesteps all three. Update the source documents, re-embed them, and the system’s knowledge is current within minutes. Every answer can cite the chunks it came from. And no retraining is required at all.

Where it gets hard

The failure modes of RAG are mostly retrieval failures, not generation failures. If the retrieval step pulls back the wrong chunks, the model will confidently generate a wrong answer from them, which can look even more convincing than a plain hallucination because it’s dressed up with citations.

Chunking strategy matters more than most teams expect going in. Split a document badly and you sever the sentence that actually answers the question from the paragraph that gives it context. Pure vector similarity search also has known blind spots: it’s good at semantic matching but weak on exact keyword matches, acronyms, or numeric lookups, which is why many production systems pair it with traditional keyword search in a hybrid retrieval setup, then use a reranking model to sort the combined results before they ever reach the LLM.

There’s also a context budget problem. Retrieving more chunks improves recall but dilutes the prompt, and stuffing a context window with marginally relevant text tends to degrade answer quality even when the truly relevant chunk is in there somewhere. Retrieval systems increasingly spend effort on getting fewer, better chunks rather than more chunks.

Where it’s heading

The pattern is evolving from a single retrieve-then-generate pass into something closer to an agentic loop, where the model can decide it needs more information, issue a follow-up retrieval, and iterate before answering. That blurs the line between RAG as an architecture and RAG as one tool among several that a model orchestrates for itself. The underlying motivation hasn’t changed though: models are static, the world isn’t, and someone has to bridge that gap at query time rather than at training time.