Retrieval-augmented generation (RAG) is a pattern where your application first searches your own data for passages relevant to a question, then passes those passages to a language model so it can answer using that context. RAG explained in one line: retrieval supplies the facts, generation supplies the wording. It is the most common way to make an LLM answer accurately about private or recent information without retraining it.
Why retrieval-augmented generation exists
A large language model only knows what was in its training data, up to a cutoff date. It knows nothing about your internal wiki, your product catalog, or a policy changed last week. Ask it anyway and it may hallucinate a plausible answer. RAG fixes this by putting the right documents into the prompt at request time. It also gives you citations, access control at retrieval time, and a way to update knowledge by re-indexing documents rather than retraining models. For a shorter conceptual overview, see what is RAG.
How does RAG work?
A RAG system has two pipelines: an offline indexing pipeline and an online query pipeline.
The indexing pipeline
- Load documents from their sources: docs, tickets, PDFs, databases.
- Clean them: strip navigation, boilerplate, and duplicated content.
- Chunk each document into passages small enough to be specific but large enough to carry meaning.
- Embed each chunk into a vector using an embedding model.
- Store vectors, chunk text, and metadata (source, title, date, permissions) in a vector index.
The query pipeline
- Embed the user's question with the same embedding model.
- Retrieve the top-k most similar chunks, applying metadata filters such as tenant or permissions.
- Rerank candidates with a more precise model to keep only the best few.
- Generate an answer by sending the question plus selected chunks to the LLM, with instructions to use only that context and cite sources.
If vectors and similarity are unfamiliar, read embeddings explained first.
A minimal RAG implementation
The Python sketch below shows the moving parts. The embed and llm functions stand in for whichever provider you use, and the in-memory index would be a real vector database in production.
import numpy as np
def chunk(text: str, size: int = 800, overlap: int = 100) -> list[str]:
step = size - overlap
return [text[i:i + size] for i in range(0, max(len(text) - overlap, 1), step)]
class Index:
def __init__(self, embed):
self.embed = embed
self.vectors, self.chunks = [], []
def add(self, doc_id: str, text: str):
for c in chunk(text):
v = np.array(self.embed(c), dtype=np.float32)
self.vectors.append(v / np.linalg.norm(v))
self.chunks.append({"doc_id": doc_id, "text": c})
def search(self, query: str, k: int = 4):
q = np.array(self.embed(query), dtype=np.float32)
q /= np.linalg.norm(q)
scores = np.stack(self.vectors) @ q
top = np.argsort(-scores)[:k]
return [self.chunks[i] | {"score": float(scores[i])} for i in top]
def answer(index: Index, llm, question: str) -> str:
hits = index.search(question, k=4)
context = "\n\n".join(f"[{h['doc_id']}] {h['text']}" for h in hits)
prompt = (
"Answer using only the context below. Cite sources as [doc_id]. "
"If the context does not contain the answer, say you do not know.\n\n"
f"<context>\n{context}\n</context>\n\nQuestion: {question}"
)
return llm(prompt)
This is enough to demo. Making it work reliably on real data is where the engineering happens.
How to build RAG that actually works
Chunk by structure, not just by length
Fixed-size character windows split sentences, tables, and code blocks in awkward places. Prefer chunking on natural boundaries such as headings, paragraphs, and list items, then merge or split to a target size. Prepend the document title and section heading to each chunk so it stays meaningful out of context. Small overlap between neighbors helps when an answer straddles a boundary.
Use hybrid search
Pure vector search is good at meaning but can miss exact terms like error codes, SKUs, or function names. Keyword search (BM25) is the opposite. Running both and merging results, for example with reciprocal rank fusion, is a reliable upgrade for most corpora.
Filter with metadata
Store tenant, permissions, product, language, and date with every chunk, and filter at query time. This is both a relevance feature and a security requirement: never retrieve documents the current user is not allowed to see, because the model will happily repeat them.
Rerank before generating
Retrieve generously, for example 20 to 50 candidates, then use a cross-encoder or reranking model to score each against the question and keep the top few. Reranking typically improves precision more than tweaking embeddings, and it shrinks the prompt, which helps latency and cost. See how to reduce LLM API costs for top-k tuning.
Write a grounded generation prompt
Tell the model to answer only from the provided context, to cite sources, and to say when it does not know. Put context inside clear delimiters. The prompt engineering best practices article covers these patterns in detail.
Rewrite queries when needed
User questions are often short or depend on chat history, such as "what about the enterprise plan?" A cheap model can rewrite the latest message into a standalone query before retrieval. For multi-part questions, decompose them into sub-queries and retrieve for each.
Common RAG failure modes
| Symptom | Likely cause | Fix |
|---|---|---|
| Answer ignores the obvious document | Retrieval missed it | Hybrid search, better chunking, query rewriting |
| Right document retrieved, wrong answer | Too much noisy context | Rerank, lower top-k |
| Confident answer with no support | Weak grounding instructions | Require citations and an explicit I do not know path |
| Exact codes or names not found | Vector-only search | Add keyword search |
| Outdated answers | Stale index | Incremental re-indexing on document change |
| Users see restricted content | No permission filtering | Filter by access metadata at retrieval |
How to evaluate a RAG pipeline
Evaluate retrieval and generation separately, because they fail for different reasons.
- Retrieval metrics: for a set of questions with known relevant chunks, measure recall at k (did the right chunk appear?) and ranking quality such as mean reciprocal rank.
- Faithfulness: does every claim in the answer appear in the retrieved context?
- Answer relevance: does the answer actually address the question?
- Citation accuracy: do the cited sources support the claims?
Start with a few dozen real questions from users or support tickets, label the relevant documents, and rerun the set whenever you change chunking, embeddings, retrieval, or prompts.
RAG vs fine-tuning vs agentic RAG
RAG is the right default when the problem is missing knowledge. Fine-tuning is better for teaching consistent style or a narrow skill, and it does not reliably add facts. Agentic RAG lets the model decide when and how to search, iterating across multiple retrievals, which helps with complex research questions at the cost of latency and predictability. Compare them in RAG vs fine-tuning and RAG vs agentic RAG.
Key takeaways
- RAG retrieves relevant passages from your data and passes them to the LLM as context.
- Quality depends mostly on retrieval: chunking, hybrid search, filters, and reranking.
- Always filter by permissions at retrieval time; the model cannot unsee what you send it.
- Instruct the model to cite sources and to admit when the context is insufficient.
- Measure retrieval recall and answer faithfulness separately with a labeled question set.
Frequently asked questions
What is RAG in simple terms?
RAG is a search step followed by a writing step. The application finds relevant documents for a question, then asks a language model to answer using those documents, so the answer is grounded in your data rather than the model's memory.
Does RAG eliminate hallucinations?
No, but it reduces them significantly when retrieval is good and the prompt requires grounded, cited answers. Hallucinations still happen when retrieval misses the right content or the model over-generalizes, which is why evaluation matters.
What chunk size should I use for RAG?
There is no universal best size. Start with chunks of a few hundred tokens split on natural boundaries, include the section title, and tune based on retrieval recall measured on your own questions.
Do I need a vector database for RAG?
For small corpora, an in-memory index or a database extension can be enough. As data, filtering needs, and traffic grow, a proper vector store helps; see how to choose a vector database.