Kafka

Overview

Apache Kafka is a distributed, append-only event log. Producers write records to topics, which are split into partitions; each partition is an ordered, immutable sequence stored on disk and replicated across brokers. Consumers read by offset and track their own position, so many independent consumer groups can read the same data, and data can be replayed.

Kafka achieves very high throughput through sequential disk I/O, batching, compression, and zero-copy transfers. Ordering is guaranteed only within a partition, so records with the same key (such as a user ID) go to the same partition. It is used for event streaming, log aggregation, change data capture, and stream processing.

When to use it

• High-throughput event streams (clicks, logs, metrics, IoT).
• Multiple consumers need the same events, possibly replayed.
• Change data capture and data pipelines.
• Stream processing with Kafka Streams or Flink.
• Event sourcing and audit logs.

Where it shows up in interviews

Where it is used in real software

Key terms

Topic / partition

Named stream / ordered shard of a topic.

Offset

Position of a record within a partition.

Consumer group

Consumers sharing work; each partition is read by one member.

Replication factor / ISR

Copies per partition / replicas that are caught up.

Retention

How long records are kept (time or size), or compacted by key.

How it works, step by step

1. 1
   Producer chooses a partition

   hash(key) mod partitions, so the same key goes to the same partition.

2. 2
   Leader broker appends the record

   Followers replicate; acks=all waits for in-sync replicas.

3. 3
   Consumers poll

   Each group member reads its assigned partitions from its offset.

4. 4
   Consumers commit offsets

   After processing, so they resume correctly after restarts.

5. 5
   Retention cleans up

   Delete after N days, or compact to keep the latest value per key.

Kafka vs a traditional queue

Key differences

Use Kafka for event streams that several systems consume or replay; use queues for task distribution with per-message retries.

Implementation

1import { Kafka } from "kafkajs";2 3const kafka = new Kafka({ clientId: "orders", brokers: process.env.KAFKA_BROKERS!.split(",") });4 5// Producer: idempotent, waits for all in-sync replicas6const producer = kafka.producer({ idempotent: true, maxInFlightRequests: 5 });7await producer.connect();8await producer.send({9  topic: "order-events",10  acks: -1,11  messages: [{ key: "order-981", value: JSON.stringify({ type: "OrderPaid", orderId: "order-981" }) }],12});13 14// Consumer group: partitions are shared among instances with the same groupId15const consumer = kafka.consumer({ groupId: "shipping-service" });16await consumer.connect();17await consumer.subscribe({ topic: "order-events", fromBeginning: false });18await consumer.run({19  autoCommit: false,20  eachMessage: async ({ topic, partition, message }) => {21    await handleEvent(JSON.parse(message.value!.toString()));      // idempotent handler22    await consumer.commitOffsets([{ topic, partition, offset: (BigInt(message.offset) + 1n).toString() }]);23  },24});

Complexity and performance

Trade-offs

Variants and related techniques

Common mistakes

• Expecting global ordering.

  Fix: Ordering holds only within a partition; key by the entity that needs order.

• Hot partitions from skewed keys.

  Fix: Choose keys with even distribution; split heavy keys.

• Committing offsets before processing.

  Fix: Commit after processing for at-least-once; make handlers idempotent.

Interview questions

Practice problems