Consumer groups - System Design Guide

Overview

A consumer group is a set of consumers that cooperate to read a topic. In Kafka, each partition is assigned to exactly one consumer in the group, so partitions are processed in parallel while order within each partition is preserved. Different groups read the same topic independently, each with its own offsets.

When consumers join or leave, the group rebalances partition assignments. The number of partitions caps parallelism: with 6 partitions, a seventh consumer sits idle. Understanding groups is key to scaling consumers, preserving ordering, and avoiding duplicate processing during rebalances.

Checkout lanes with assigned customers

Each lane (partition) has its customers in order. Each cashier (consumer) takes one or more lanes, but no lane is served by two cashiers at once. A second store (another group) processes the same customers for its own purpose.

When to use it

Scaling consumers horizontally on Kafka, Kinesis, or Redis Streams.
Allowing several services to read the same stream independently.
Preserving per-key ordering while processing in parallel.
Planning partition counts.

Where it shows up in interviews

Scaling stream consumers

Recognize it when: consumer lag keeps growing.

Scale an order events consumer
Design a log processing pipeline

Independent consumers

Recognize it when: several services need all events.

Design analytics and notifications from one event stream

Where it is used in real software

Kafka consumer groups

Coordinated by a group coordinator broker; cooperative sticky assignment minimizes rebalance disruption.

Kinesis Client Library

Uses a DynamoDB lease table to assign shards to workers.

Redis Streams

XREADGROUP with pending entry lists for per-message acknowledgment.

Key terms

Group ID: Identifies the group; offsets are stored per group.
Assignment: Mapping of partitions to consumers.
Rebalance: Reassigning partitions when membership changes.
Consumer lag: Latest offset minus committed offset.
Static membership: Stable member IDs to avoid rebalances on restarts.

How it works, step by step

1
Consumers join the group
Same group ID.
2
Coordinator assigns partitions
Range, round robin, or cooperative sticky.
3
Each consumer processes its partitions
In order, committing offsets.
4
Membership changes trigger rebalance
Partitions move to other consumers.
5
Monitor lag
Scale consumers (up to the partition count) when lag grows.

Rebalance when a consumer joins

Consumer A

Consumer B

STEP 1Consumer A alone owns all four partitions.

Consumers vs partitions

Topic with 4 partitions

Consumers in group	Partitions per consumer	Effect
1	4	All work on one instance
2	2 each	2x parallelism
4	1 each	Maximum parallelism
6	1 each, 2 idle	Extra consumers do nothing

NOWConsumers in group: 1 | Partitions per consumer: 4 | Effect: All work on one instance

Choose partition counts for future peak parallelism; adding partitions later changes key-to-partition mapping.

Implementation

1Properties props = new Properties();2props.put("bootstrap.servers", System.getenv("KAFKA_BROKERS"));3props.put("group.id", "shipping-service");4props.put("enable.auto.commit", "false");5props.put("partition.assignment.strategy", "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");6props.put("group.instance.id", System.getenv("POD_NAME")); // static membership: fewer rebalances7props.put("key.deserializer", StringDeserializer.class.getName());8props.put("value.deserializer", StringDeserializer.class.getName());9 10try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {11    consumer.subscribe(List.of("order-events"));12    while (true) {13        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));14        for (ConsumerRecord<String, String> r : records) {15            handle(r.key(), r.value());          // idempotent16        }17        consumer.commitSync();                   // commit after processing the batch18    }19}

Complexity and performance

Max parallelism= partition count

Per group.

Rebalance pausems to seconds

Cooperative protocols reduce it.

Trade-offs

More partitions vs overhead

More partitions allow more consumers but increase broker overhead and rebalance time.

Commit frequency

Frequent commits reduce reprocessing after failures but add load.

Variants and related techniques

Share groups (queues for Kafka)

Newer Kafka feature allowing multiple consumers per partition with per-record acks.

Parallel consumer libraries

Process one partition's records concurrently by key while keeping per-key order.

Common mistakes

Slow processing exceeding max.poll.interval.ms.
Fix: The consumer is kicked out and triggers rebalances; process faster or reduce batch size.
More consumers than partitions.
Fix: Extra consumers are idle; increase partitions.

Interview questions

How do you scale Kafka consumers?

Add consumers to the same group up to the number of partitions; if more parallelism is needed, increase partitions (knowing it remaps keys) or process records concurrently within a partition per key.

How do two services both receive every event?

Give them different consumer group IDs; each group tracks its own offsets and receives the full stream.

Practice problems

Problem	Difficulty	What it trains
Plan partitions for 50k events/s with 5k/s per consumer	Easy	Sizing.
Diagnose growing consumer lag	Medium	Rebalances and slow handlers.