LoRA - System Design Guide | Interview Forge

Overview

LoRA (low-rank adaptation) is a way to fine-tune a large model cheaply. Instead of updating every weight, it freezes the original model and adds two small matrices beside selected weight matrices. Their product is a low-rank update that is learned during training. Only these adapters are trained, often less than one percent of the parameters.

This makes fine-tuning practical on a single GPU, produces adapter files of a few megabytes instead of full model copies, and lets one base model serve many customers or tasks by swapping adapters. QLoRA goes further by loading the frozen base model in 4-bit precision, so even large models can be tuned on modest hardware.

Sticky notes on a textbook

Rewriting a whole textbook for each class is expensive. LoRA leaves the textbook unchanged and adds a thin stack of sticky notes with corrections for a specific course. Swap the sticky notes and the same book serves a different class.

When to use it

Fine-tuning an open model on limited GPU memory.
Serving many task- or tenant-specific variants of one base model.
Teaching a consistent format, tone, or narrow skill.

Where it shows up in interviews

Multi-tenant model customization

Recognize it when: many customers want their own tuned model.

Design an LLM inference platform
Design an AI content-generation platform

Where it is used in real software

Hosted fine-tuning services

Many providers use adapter-based tuning behind the scenes so custom models are cheap to train and serve.

Image generation styles

Diffusion model communities share small LoRA files that add a style or character to a base model.

Multi-adapter serving

Inference servers load hundreds of adapters on top of one base model and pick one per request.

Key terms

Rank (r): The inner dimension of the adapter matrices. Higher rank means more capacity and more parameters.
Adapter: The small trained matrices added to the frozen model.
Alpha: A scaling factor applied to the adapter's update.
QLoRA: LoRA training on a base model quantized to 4 bits to save memory.
Merging: Adding the adapter update into the base weights for zero-overhead inference.

How it works, step by step

1
Freeze the base model
Original weights W are not updated, so optimizer memory for them is not needed.
2
Add adapters
For chosen layers (often attention projections), add matrices A (d x r) and B (r x d) with small r such as 8 or 16.
3
Train only A and B
The effective weight becomes W + (alpha / r) x A x B. B starts at zero so training begins from the original behavior.
4
Save the adapter
Store only A and B, typically a few megabytes.
5
Serve
Merge adapters into W for one task, or keep them separate and swap per request for many tasks.

Full fine-tuning vs LoRA

Base weights W

Adapter A

Adapter B

W + A x B

Output

STEP 1The base weight matrix (for example 4096 x 4096 is about 16.7M values) stays frozen.

Trainable parameters for one 4096 x 4096 layer

Comparing full fine-tuning with LoRA at different ranks.

Method	Trainable values	Share of full
Full fine-tuning	16,777,216	100%
LoRA r=64	524,288	3.1%
LoRA r=16	131,072	0.8%
LoRA r=8	65,536	0.4%

NOWMethod: Full fine-tuning | Trainable values: 16,777,216 | Share of full: 100%

Rank 8 to 16 is often enough for style and format tasks at under one percent of the parameters.

Implementation

1from transformers import AutoModelForCausalLM2from peft import LoraConfig, get_peft_model3 4base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B")5config = LoraConfig(6    r=16,7    lora_alpha=32,8    lora_dropout=0.05,9    target_modules=["q_proj", "v_proj"],   # attention projections10    task_type="CAUSAL_LM",11)12model = get_peft_model(base, config)13model.print_trainable_parameters()          # a small fraction of the total14 15# ... train with your usual Trainer loop on curated examples ...16model.save_pretrained("adapters/claims-summary")   # a few MB

Complexity and performance

Trainable parameters2 x d x r per adapted matrix

vs d x d for full fine-tuning.

Inference overheadzero when merged

Small extra matmul when kept separate.

Trade-offs

Rank vs capacity

Low rank is cheap and resists overfitting but may underfit complex new skills. Increase rank or adapted layers if quality plateaus.

LoRA vs full fine-tuning

Full fine-tuning can reach slightly higher quality on big domain shifts, but costs far more memory and produces full model copies.

Variants and related techniques

QLoRA

Quantizes the frozen base to 4 bits so larger models fit on one GPU during training.

DoRA and other PEFT methods

Variants that change how the update is parameterized; prefix tuning and adapters are older alternatives.

Common mistakes

Expecting LoRA to add large amounts of new knowledge.
Fix: Use it for behavior and format; supply facts with retrieval.
Training on noisy examples.
Fix: A few hundred clean, consistent examples beat thousands of messy ones.

Interview questions

Why is LoRA cheaper than full fine-tuning?

It freezes the base weights and trains only small low-rank matrices, so gradients and optimizer state are needed for a tiny fraction of parameters. Memory drops sharply and the saved artifact is megabytes instead of a full model.

How would you serve 500 customer-specific fine-tuned models?

Train a LoRA adapter per customer on one shared base model, load the base once per GPU, and attach the right adapter per request using a multi-adapter inference server, caching hot adapters in GPU memory.

Practice problems

Problem	Difficulty	What it trains
Compute trainable parameters for LoRA on all attention layers of a model	Easy	Parameter math.
Design a multi-tenant fine-tuning and serving platform	Hard	Adapter storage and routing.