Overview
The transformer is the neural network design behind modern language models. Its key idea is attention. Instead of reading text strictly left to right like older recurrent networks, every token looks at the other tokens and decides how much each one matters to its meaning. In "The bank raised rates", attention lets "bank" draw on "rates" and resolve to the financial sense.
A transformer block has two parts, multi-head self-attention followed by a feed-forward network, each wrapped with residual connections and normalization. Models stack dozens of these blocks. Because attention over a sequence can be computed in parallel, transformers train efficiently on GPUs, which is what made scaling to billions of parameters practical.
In a relay, each person only hears the one before them, and details get lost. In a transformer, every participant can ask every other participant a question (query), each person advertises what they know (key), and answers (values) are blended according to how relevant each person is. After several rounds, everyone has a rich shared understanding.
When to use it
- Explaining how LLMs, embedding models, and many vision models work internally.
- Reasoning about context length, memory, and latency limits in AI systems.
- Understanding why the KV cache and batching matter for inference.
Where it shows up in interviews
Recognize it when: the interviewer asks why long prompts are slow or expensive.
- Design an LLM inference platform
- Design a ChatGPT-like system
Recognize it when: the task is search or classification rather than generation.
- Design a large-scale embedding service
- Design a vector search system
Where it is used in real software
Decoder-only transformers that generate text one token at a time.
Encoder transformers turn sentences into vectors for semantic search.
Encoder-decoder transformers read the source fully, then generate the target.
Key terms
- Self-attention
- Each token computes weighted connections to other tokens in the same sequence.
- Query, key, value
- Three projections of each token. Query and key scores decide weights, and values are blended by those weights.
- Multi-head attention
- Several attention computations in parallel, each free to focus on a different relationship.
- Positional encoding
- Information added to tokens so the model knows word order.
- Causal mask
- Prevents a token from attending to later tokens during generation.
How it works, step by step
- 1Embed tokens
Each token ID maps to a learned vector, and positional information is added.
- 2Compute attention
Every token produces a query, key, and value. Scores are query-key dot products, scaled and softmaxed into weights.
- 3Blend values
Each token's new representation is the weighted sum of values, so it now carries context.
- 4Feed-forward and residual
A small per-token network transforms the result, and residual connections keep earlier information flowing.
- 5Stack and predict
After many blocks, the last layer maps each position to vocabulary probabilities (decoder) or a sentence vector (encoder).
STEP 1Each token starts as a vector plus positional information.
Attention for one word
Sentence "The bank raised rates". Attention weights from the token "bank".
| Token | Weight from "bank" | Effect |
|---|---|---|
| The | 0.05 | Little information |
| bank | 0.30 | Keeps its own identity |
| raised | 0.20 | Suggests an action by an institution |
| rates | 0.45 | Strong signal for the financial meaning |
NOWToken: The | Weight from "bank": 0.05 | Effect: Little information
The weighted blend pushes "bank" toward the finance sense rather than a river bank.
Implementation
import numpy as np def softmax(x): e = np.exp(x - x.max(axis=-1, keepdims=True)) return e / e.sum(axis=-1, keepdims=True) def self_attention(X, Wq, Wk, Wv, causal=True): Q, K, V = X @ Wq, X @ Wk, X @ Wv # (n, d) each scores = Q @ K.T / np.sqrt(K.shape[-1]) # (n, n) if causal: # block attention to future tokens scores += np.triu(np.full(scores.shape, -1e9), k=1) return softmax(scores) @ V # weighted blend of values n, d = 4, 8X = np.random.randn(n, d)out = self_attention(X, *(np.random.randn(d, d) for _ in range(3)))print(out.shape) # (4, 8)Complexity and performance
n tokens, d dimensions. Long contexts grow quadratically.
Stored per request during generation.
The reason transformers replaced recurrent networks.
Trade-offs
Longer windows let the model see more but raise attention compute and KV cache memory. Retrieval often beats stuffing everything into the prompt.
Training processes all tokens at once, but generation still produces one token per step, so output length dominates latency.
Variants and related techniques
GPT-style models for generation with a causal mask.
BERT-style models that see the full sentence, used for embeddings and classification.
Routes each token to a few of many feed-forward experts, increasing capacity without using all weights per token.
Common mistakes
- Assuming the model reads text in order like a human.
Fix: Remember order comes only from positional encoding; attention itself is order-agnostic.
- Ignoring quadratic attention cost.
Fix: Budget context length and use retrieval, summarization, or efficient attention for long inputs.
Interview questions
Why did transformers replace RNNs?
RNNs process tokens one at a time, which is slow to train and forgets distant context. Transformers let every token attend to every other token in parallel, capturing long-range relationships and scaling well on GPUs.
What is the KV cache and why does it matter?
During generation, the keys and values of earlier tokens do not change, so they are cached instead of recomputed. This makes each new token cheap to compute but uses GPU memory that grows with context length and concurrent requests.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Implement scaled dot-product attention with a causal mask | Medium | Q, K, V math. |
| Estimate KV cache memory for a 32-layer model at 8K context and batch 16 | Hard | Serving capacity. |