Overview
In messaging systems, a partition is an ordered, append-only subset of a topic's messages. Partitions are the unit of parallelism (one consumer per partition per group), ordering (guaranteed only within a partition), and replication (each partition has a leader and followers). A message's key determines its partition, usually by hashing.
Choosing the key and the number of partitions is the central design decision for a Kafka or Kinesis topic. A good key spreads load evenly while keeping related events (same order, same user) together so they are processed in order.
More lanes carry more cars at once, and cars in the same lane stay in order. If everyone crowds into one lane (a hot key), the other lanes do not help.
When to use it
- Designing Kafka topics or Kinesis streams.
- Guaranteeing per-entity ordering.
- Scaling consumer throughput.
- Diagnosing hot partitions and lag.
Where it shows up in interviews
Recognize it when: events for an entity must stay ordered.
- Design order event streams
- Design a chat message pipeline
Recognize it when: how many partitions for this load?
- Design a logging pipeline
- Design IoT ingestion
Where it is used in real software
Uses murmur2 hash of the key modulo partition count; keyless messages use sticky batching across partitions.
The same concept: partition keys hash to shards with fixed throughput.
Celebrity accounts or a single large tenant can overload one partition, causing lag for everyone sharing it.
Key terms
- Partition key
- Value hashed to select a partition.
- Leader replica
- Handles reads and writes for the partition.
- Hot partition
- Partition receiving disproportionate traffic.
- Repartitioning
- Changing partition count, which remaps keys.
How it works, step by step
- 1Identify ordering needs
Which events must be processed in order relative to each other?
- 2Choose the key
The entity ID for that ordering (orderId, userId).
- 3Estimate throughput
Partitions >= peak throughput / per-partition throughput.
- 4Plan consumer parallelism
Partitions >= max consumers you expect.
- 5Check key distribution
Look for skew; handle heavy keys specially.
Choosing a partition key for order events
Events: OrderPlaced, OrderPaid, OrderShipped
| Key | Ordering | Distribution | Verdict |
|---|---|---|---|
| orderId | Per order (what we need) | Even | Good |
| customerId | Per customer | Skewed by big buyers | OK if per-customer order matters |
| country | Per country | Very skewed | Bad |
| null (no key) | None | Even | Only if order does not matter |
NOWKey: orderId | Ordering: Per order (what we need) | Distribution: Even | Verdict: Good
Key by the smallest entity whose events must stay in order; that usually gives both correctness and even load.
Implementation
// Estimate partitions and check key skew before creating a topicfunction partitionsNeeded(peakMBps: number, perPartitionMBps: number, maxConsumers: number) { return Math.max(Math.ceil(peakMBps / perPartitionMBps), maxConsumers);} function keySkew(keys: string[], partitions: number, hash: (k: string) => number) { const counts = new Array(partitions).fill(0); for (const k of keys) counts[hash(k) % partitions]++; const avg = keys.length / partitions; return Math.max(...counts) / avg; // 1.0 = perfectly even; > 2 means a hot partition} console.log(partitionsNeeded(120, 10, 8)); // 12Complexity and performance
hash(key) mod N.
Until broker limits.
Trade-offs
Coarser keys preserve more ordering but concentrate load; finer keys spread load but order less.
Increasing partitions remaps keys, breaking per-key order during the transition.
Variants and related techniques
Route heavy keys to dedicated partitions.
Append a suffix to spread a hot key, at the cost of ordering for that key.
Common mistakes
- Too few partitions.
Fix: Cannot scale consumers later without remapping keys; plan for growth.
- Low-cardinality keys.
Fix: Few distinct keys means few busy partitions.
Interview questions
How do you keep events for one order in sequence?
Use the order ID as the partition key so all its events go to the same partition, which is consumed in order by a single consumer in the group.
What happens when you add partitions to a Kafka topic?
New messages for some keys go to different partitions, so per-key ordering can break between old and new messages; existing data is not moved.
Practice problems
| Problem | Difficulty | What it trains |
|---|---|---|
| Choose keys for 4 topics | Easy | Ordering vs distribution. |
| Fix a hot partition caused by a large tenant | Medium | Salting and dedicated partitions. |