Overview
Retrieval-augmented generation (RAG) answers a question in two moves. First it retrieves the most relevant passages from your own data, such as docs, tickets, or policies. Then it passes those passages to an LLM with instructions to answer using only that context. The model contributes language skill; your data contributes facts.
RAG solves three problems of a plain LLM. The model's knowledge is frozen at training time, it has never seen your private documents, and it may invent answers. With retrieval, answers can cite sources, stay current as soon as documents are re-indexed, and respect access control because you choose which documents each user can retrieve.
A plain LLM takes a closed-book exam from memory. RAG hands it the relevant pages of the textbook right before each question. The student still writes the answer, but now it is based on the pages in front of them, and they can point to where each fact came from.
When to use it
- Question answering over internal documentation or a knowledge base.
- Support assistants that must cite policy or help articles.
- Any LLM feature where facts change often or are private.
- When you need citations and auditability.
Where it shows up in interviews
Recognize it when: answers must be correct, current, or cite sources.
- Design a RAG system
- Design an AI customer-support system
Recognize it when: users ask questions over a large document collection.
- Design an AI search engine
- Design Large-Scale RAG
Where it is used in real software
Support bots retrieve articles and past resolved tickets, then draft an answer with links.
Workplace assistants search wikis, drives, and chat with the user's own permissions before answering.
Documentation sites answer "how do I" questions by retrieving the right reference pages.
Key terms
- Chunk
- A small passage of a document, often a few hundred tokens, that is indexed and retrieved.
- Embedding
- A vector that represents the meaning of text, so similar passages have nearby vectors.
- Vector database
- A store optimized for finding the nearest vectors to a query vector.
- Grounding
- Making the model's answer depend on supplied evidence rather than memory.
- Top-K
- The number of best-matching chunks passed to the model.
How it works, step by step
- 1Ingest and chunk
Parse documents, split them into overlapping chunks, and keep metadata such as source, date, and permissions.
- 2Embed and index
Convert each chunk to a vector and store it with its metadata in a vector index.
- 3Retrieve for a question
Embed the user's question, find the nearest chunks, and filter by metadata such as user permissions.
- 4Build the prompt
Place the top chunks in the prompt with instructions to answer only from them and cite sources.
- 5Generate and check
The LLM writes the answer; optionally verify citations and fall back to "I don't know" when context is weak.
STEP 1User asks "How many days of parental leave do we offer?"
Plain LLM vs RAG
Question: "What is our refund window for opened electronics?"
| Approach | What the model sees | Likely answer |
|---|---|---|
| Plain LLM | Only the question | A generic guess such as 30 days |
| RAG | Question plus current refund policy chunk | 15 days, with citation to policy section 4.2 |
NOWApproach: Plain LLM | What the model sees: Only the question | Likely answer: A generic guess such as 30 days
The policy text in the prompt turns a plausible guess into a verifiable answer.
Implementation
import OpenAI from "openai";const openai = new OpenAI(); async function embed(text: string): Promise<number[]> { const res = await openai.embeddings.create({ model: "text-embedding-3-small", input: text }); return res.data[0].embedding;} export async function answer(question: string, userRegion: string) { const queryVector = await embed(question); const chunks = await vectorStore.search(queryVector, { topK: 5, filter: { region: userRegion } }); const context = chunks.map((c, i) => `[${i + 1}] (${c.source})\n${c.text}`).join("\n\n"); const res = await openai.chat.completions.create({ model: "gpt-4o-mini", temperature: 0, messages: [ { role: "system", content: "Answer only from the context. Cite sources like [1]. If the answer is not in the context, say you don't know." }, { role: "user", content: `Context:\n${context}\n\nQuestion: ${question}` }, ], }); return { text: res.choices[0].message.content, sources: chunks.map((c) => c.source) };}Complexity and performance
Milliseconds over millions of chunks.
Drives token cost and time to first token.
New documents are answerable as soon as they are embedded.
Trade-offs
Larger top-K raises recall but adds noise and cost. Re-ranking a larger candidate set down to a few chunks balances both.
Small chunks match precisely but lose surrounding context. Large chunks keep context but dilute relevance.
Variants and related techniques
Combine keyword search and vector search to catch exact terms like product codes.
An agent decides when and where to search, reformulates queries, and retrieves multiple times.
Retrieves over a knowledge graph of entities and relationships for multi-hop questions.
Common mistakes
- Ignoring document permissions.
Fix: Store access metadata with each chunk and filter at retrieval time.
- No fallback when retrieval fails.
Fix: Instruct the model to say it does not know and log low-similarity queries for review.
- Never measuring retrieval quality.
Fix: Build a test set of questions with expected sources and track recall and answer accuracy.
Interview questions
Why use RAG instead of putting all documents in the prompt?
Context windows are limited, long prompts are slow and expensive, and models attend less reliably to details buried in huge contexts. Retrieval selects only the relevant passages, keeps cost predictable, and allows per-user access control.
How do you evaluate a RAG system?
Separately measure retrieval (did the right chunks appear in top-K) and generation (is the answer correct, grounded in the chunks, and properly cited). Use a labeled question set plus sampled human review.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Build a RAG bot over a folder of Markdown docs | Medium | Chunking, embedding, and citations. |
| Design a permission-aware RAG system for a large company | Hard | Access control and freshness. |