Overview
RAG and fine-tuning change different things. RAG changes what the model knows at request time by putting relevant documents into the prompt. Fine-tuning changes how the model behaves by further training its weights on examples. A useful rule is that RAG supplies knowledge while fine-tuning teaches skills, style, and format.
Most teams should start with prompting, add RAG when answers need private or changing facts, and consider fine-tuning only when the model still fails at a repeatable behavior, such as a strict output format, a domain writing style, or a narrow classification task. The two can also be combined, with a fine-tuned model reading retrieved context.
RAG is handing a new employee a binder of current policies to look things up. Fine-tuning is sending them on a training course so they naturally write and act the company way. The binder is easy to update tomorrow; the course changes habits but gets out of date when policies change.
When to use it
- Deciding how to adapt an LLM to a company's data or domain.
- Explaining cost, freshness, and accuracy trade-offs to stakeholders.
- Planning an AI roadmap from prototype to production.
Where it shows up in interviews
Recognize it when: the interviewer asks how to make the model know company information.
- Design a RAG system
- Design an AI customer-support system
Recognize it when: a large model works but is too slow or expensive.
- Design an LLM inference platform
- Design an AI coding assistant
Where it is used in real software
Use RAG because content changes weekly and answers need citations.
Teams fine-tune small models to output a precise JSON schema from invoices or forms at low cost.
Marketing tools fine-tune on approved copy so drafts match tone without long style instructions.
Key terms
- Fine-tuning
- Continuing training of a pre-trained model on task-specific examples.
- LoRA
- A parameter-efficient method that trains small adapter matrices instead of all weights.
- Knowledge cutoff
- The date after which the model has no training data.
- Catastrophic forgetting
- Losing general ability when fine-tuning pushes the model too far toward narrow data.
How it works, step by step
- 1Try prompting first
Clear instructions and a few examples solve many problems with no infrastructure.
- 2Add RAG for facts
If answers depend on private, large, or changing information, retrieve it into the prompt.
- 3Measure the remaining failures
Build an evaluation set and classify errors as missing knowledge or wrong behavior.
- 4Fine-tune for behavior
If failures are about format, tone, or a narrow skill, fine-tune with a few hundred to thousands of high-quality examples.
- 5Combine when needed
A fine-tuned model can still use RAG, getting consistent behavior and fresh facts.
STEP 1Start with instructions and examples. It is the cheapest and fastest to change.
Side-by-side comparison
Adapting a model for an insurance company.
| Factor | RAG | Fine-tuning |
|---|---|---|
| Adds new facts | Yes and instantly | Poorly and goes stale |
| Changes style or format | Somewhat through prompt | Yes and reliably |
| Citations | Natural | Not available |
| Upfront cost | Indexing pipeline | Training data and compute |
| Per-request cost | Longer prompts | Shorter prompts and possibly smaller model |
| Access control | Filter documents per user | Hard since knowledge is in weights |
NOWFactor: Adds new facts | RAG: Yes and instantly | Fine-tuning: Poorly and goes stale
Use RAG for policy answers and fine-tune only for the claim-summary format the adjusters need.
Implementation
type Failure = { kind: "missing-fact" | "stale-fact" | "wrong-format" | "wrong-tone" | "reasoning" }; function recommend(failures: Failure[]): string[] { const count = (k: Failure["kind"]) => failures.filter((f) => f.kind === k).length; const plan: string[] = []; if (count("missing-fact") + count("stale-fact") > 0) plan.push("Add or improve RAG retrieval"); if (count("wrong-format") + count("wrong-tone") > failures.length * 0.2) plan.push("Fine-tune (LoRA) on curated examples"); if (count("reasoning") > 0) plan.push("Try a stronger model or break the task into steps"); return plan.length ? plan : ["Keep prompting; failures are rare"];}Complexity and performance
Re-embed changed documents.
Retrain and redeploy the model.
Quality matters more than quantity.
Trade-offs
RAG is always current but depends on retrieval quality. Fine-tuning gives consistent behavior but bakes in a snapshot.
RAG adds tokens to every request. Fine-tuning costs upfront but can shorten prompts and allow a smaller model.
Variants and related techniques
LoRA and similar methods train a small fraction of weights, making tuning cheaper and adapters swappable.
Train the model on examples that include retrieved context so it learns to use and cite it well.
Common mistakes
- Fine-tuning to teach facts.
Fix: Facts in weights go stale and cannot be cited; keep facts in a retrievable store.
- Fine-tuning before building an evaluation set.
Fix: Measure the baseline first so you can prove the tuned model is better.
Interview questions
A company wants the model to know its product catalog. RAG or fine-tuning?
RAG. The catalog changes, answers need exact prices and availability, and retrieval allows citations and per-region filtering. Fine-tuning would bake in stale data and still hallucinate details.
When is fine-tuning the right choice?
When the model must reliably follow a specific format, style, or narrow skill that prompting cannot achieve consistently, or when you want a smaller, cheaper model to match a larger one on a focused task.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Classify ten product requirements as RAG, fine-tuning, or both | Easy | Knowledge vs behavior. |
| Design an adaptation plan for a legal document assistant | Medium | Evaluation-driven decisions. |