RAG (Retrieval-Augmented Generation)
RAG (Retrieval-Augmented Generation) is an architectural pattern that supplies an LLM with content from external knowledge sources at answer time: first, documents matching the request are retrieved (retrieval), then the model generates the answer on this basis (generation).
How does RAG work?
A RAG system operates in two phases. In the preparation phase, the knowledge base is processed: documents are broken into manageable sections (chunking) and indexed so they can be searched by content — usually via vector embeddings that capture semantic similarity, often combined with classic full-text search.
At answer time, every request then runs through three steps:
- Retrieval: The sections of the knowledge base that best match the user’s question are retrieved.
- Augmentation: The retrieved sections are inserted into the input to the LLM together with the question.
- Generation: The model formulates the answer based on the supplied content — and can name the sources used.
The decisive effect: the model no longer answers only from its training knowledge, but from the current, controlled content of the company.
When is RAG useful?
RAG plays to its strengths when the required knowledge is current, company-internal or subject to verification. Typical reasons for RAG:
- Currency: Content changes continuously — the knowledge base is simply updated, without retraining the model.
- Traceability: Answers can be supported with source references, which considerably eases review and trust.
- Access control: Who may see which content can be controlled at the level of the knowledge base — the model only receives what the requester is allowed to see.
- Hallucination reduction: Grounding in supplied documents noticeably lowers the risk of freely invented statements — but does not replace review of critical statements.
RAG or fine-tuning?
The two approaches solve different problems and are often confused:
| RAG | Fine-tuning | |
|---|---|---|
| Changes | the knowledge available at answer time | the behavior of the model itself |
| Suitable for | facts, documents, current company knowledge | tone, format, domain language, specialized tasks |
| Updating | maintaining the knowledge base is enough | retraining required |
| Source attribution | possible, answers are verifiable | not possible, knowledge is “baked into” the model |
As a rule of thumb: if it is about knowledge, RAG is the first choice. If it is about behavior, fine-tuning (or training adapters) is the appropriate means. In practice, the two are frequently combined.
Limits of RAG
RAG is not a sure-fire success. Answer quality stands and falls with retrieval: if the wrong sections are found — because the documents were chunked unfavorably, the search misses the domain terminology or the knowledge base is outdated — the model still produces a fluent-sounding answer from poor material. Ambiguous questions or questions spread across many documents also remain challenging. And RAG does not teach the model new skills: it delivers content, not competence.
RAG in practice: data quality decides
In real projects, in our experience, the largest part of the work goes not into the model but into the knowledge base: reviewing content, clarifying responsibilities for its maintenance, mapping access rights and measuring retrieval against the actual questions users ask. A RAG system built on top of an unmaintained document repository only makes its contradictions visible faster — the real investment is a knowledge base you can trust.