Overview
Language models do not read characters or whole words. They read tokens, which are frequent chunks of text learned from training data. Common words like "the" are one token, while a rare word like "tokenization" may split into "token" and "ization". Each token maps to an integer ID, and the model only ever sees those IDs.
Tokenization affects almost every production concern. Pricing, rate limits, and context windows are measured in tokens. Non-English text, code, and numbers often use more tokens per character, which raises cost. Odd model behavior, such as miscounting letters in a word, often comes from the model seeing tokens instead of letters.
A stenographer does not write every letter. Common words and phrases get short symbols, while rare words are spelled out in pieces. Tokenization is the model's shorthand. Frequent text is compact, and unusual text takes more symbols.
When to use it
- Estimating LLM cost and latency.
- Checking whether a prompt and retrieved documents fit in the context window.
- Chunking documents for retrieval by token count.
- Explaining strange model behavior with spelling, numbers, or rare words.
Where it shows up in interviews
Recognize it when: the design has long documents, chat history, or per-token billing.
- Design a RAG system
- Design an LLM gateway
Where it is used in real software
LLM providers charge separately for input and output tokens, so tokenization directly sets cost.
Chat apps count tokens to decide which older messages to summarize or drop.
Teams measure tokens per language because some scripts need two to three times more tokens for the same meaning.
Key terms
- Token
- A unit of text the model reads, often a word or word piece.
- Vocabulary
- The fixed set of tokens a model knows, typically 30,000 to 200,000 entries.
- Byte-pair encoding (BPE)
- A method that builds tokens by repeatedly merging the most frequent adjacent pairs of symbols.
- Special tokens
- Reserved tokens that mark roles, message boundaries, or the end of text.
How it works, step by step
- 1Normalize the text
Some tokenizers adjust whitespace or Unicode before splitting.
- 2Split into pieces
The tokenizer applies learned merge rules to break text into the largest known chunks.
- 3Map to IDs
Each chunk becomes an integer from the vocabulary.
- 4Model processes IDs
IDs are turned into embedding vectors and processed by the network.
- 5Decode the output
Generated IDs are mapped back to text pieces and joined.
STEP 1Input text is "Tokenization helps".
Token counts vary by content
Approximate token counts for 100 characters of different content (exact numbers depend on the tokenizer).
| Content | Approx. tokens | Why |
|---|---|---|
| Plain English prose | 22-25 | Common words are single tokens |
| Source code | 30-40 | Symbols and indentation split often |
| Long numbers | 40-60 | Digits split into small groups |
| Non-Latin scripts | 40-80 | Fewer merged pieces in the vocabulary |
NOWContent: Plain English prose | Approx. tokens: 22-25 | Why: Common words are single tokens
Always measure tokens with the model's own tokenizer instead of guessing from characters.
Implementation
import tiktoken enc = tiktoken.get_encoding("cl100k_base")text = "Tokenization helps models read text."ids = enc.encode(text)print(len(ids), ids)print([enc.decode([i]) for i in ids]) # see each piece def fits(prompt: str, context_limit: int, reserve_for_output: int = 500) -> bool: return len(enc.encode(prompt)) + reserve_for_output <= context_limitComplexity and performance
Fast compared with model inference.
For English prose only.
Trade-offs
Larger vocabularies make text shorter in tokens but increase the embedding table size and can waste capacity on rare tokens.
Byte-level tokenizers never hit unknown words but split rare text into many small pieces.
Variants and related techniques
Used by BERT-style models; similar goal to BPE with a different merge score.
Treats text as raw characters including spaces, which suits languages without spaces.
Common mistakes
- Estimating cost from word counts.
Fix: Count tokens with the exact tokenizer for the model you call.
- Chunking documents by characters only.
Fix: Chunk by tokens so each chunk reliably fits the embedding model and prompt budget.
Interview questions
Why can an LLM struggle to count the letters in a word?
It sees tokens, not letters. A word may be one or two tokens, so individual characters are not directly visible to the model.
How would you keep a long chat within the context window?
Count tokens per message, keep the system prompt and the most recent turns, summarize older turns into a short memory, and reserve space for the answer.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Build a token counter that warns before a prompt exceeds the limit | Easy | Budgeting. |
| Design a chat memory policy with summarization | Medium | Context management. |