Kinesis - System Design Guide

Overview

Amazon Kinesis Data Streams is a managed streaming service similar in concept to Kafka. Records are written to a stream divided into shards; each shard is an ordered sequence with a fixed capacity (1 MB/s or 1,000 records/s in, 2 MB/s out). Records are retained (24 hours by default, up to 365 days) and can be read by multiple consumers and replayed.

The Kinesis family also includes Data Firehose (load streams into S3, Redshift, OpenSearch with no code) and Managed Service for Apache Flink (stream processing). It is a common choice for clickstreams, logs, and IoT on AWS when you do not want to operate Kafka.

Conveyor belts in a factory

Each shard is a conveyor belt with a speed limit. Items stay in order on each belt, and several inspectors can watch the same belt. For more capacity, you add belts.

When to use it

Real-time ingestion of logs, clickstreams, IoT telemetry on AWS.
Multiple consumers reading the same ordered stream.
Loading streaming data into S3 or a warehouse (Firehose).
Real-time analytics with Flink.

Where it shows up in interviews

Streaming ingestion on AWS

Recognize it when: millions of events into analytics.

Design a clickstream analytics system
Design IoT telemetry ingestion

Real-time processing

Recognize it when: detect patterns within seconds.

Design real-time fraud detection
Design a live dashboard

Where it is used in real software

Netflix

Has used Kinesis for network flow log analysis at massive scale.

Kinesis Data Firehose

Delivers logs to S3 in Parquet with batching and compression, feeding Athena queries.

DynamoDB Streams

Uses a Kinesis-like shard model for change data capture.

Key terms

Shard: Unit of capacity and ordering.
Partition key: Hashed to choose a shard.
Sequence number: Order within a shard.
Enhanced fan-out: Dedicated 2 MB/s per consumer per shard.
On-demand mode: Automatically scales shard capacity.

How it works, step by step

1
Producers put records
With a partition key (for example deviceId).
2
Kinesis assigns shards
By hash of the key.
3
Consumers read shards
KCL or Lambda track checkpoints per shard.
4
Process and checkpoint
Resume after failures from the checkpoint.
5
Scale
Split or merge shards, or use on-demand mode.

Kinesis vs Kafka (MSK)

Streaming options on AWS

Aspect	Kinesis Data Streams	Kafka / MSK
Operations	Fully managed, serverless option	Managed brokers, more tuning
Scaling unit	Shard (1 MB/s in)	Partition / broker
Retention	Up to 365 days	Configurable, tiered storage
Ecosystem	AWS-native integrations	Kafka Connect, Streams, huge ecosystem
Portability	AWS only	Any cloud or on-prem

NOWAspect: Operations | Kinesis Data Streams: Fully managed, serverless option | Kafka / MSK: Managed brokers, more tuning

Kinesis minimizes operations on AWS; Kafka offers portability and a richer ecosystem.

Implementation

1import { KinesisClient, PutRecordsCommand } from "@aws-sdk/client-kinesis";2 3const kinesis = new KinesisClient({});4 5export async function sendClicks(events: { userId: string; page: string; ts: number }[]) {6  const res = await kinesis.send(new PutRecordsCommand({7    StreamName: "clickstream",8    Records: events.map((e) => ({9      PartitionKey: e.userId,                         // per-user ordering10      Data: new TextEncoder().encode(JSON.stringify(e)),11    })),12  }));13  if (res.FailedRecordCount) {14    const failed = events.filter((_, i) => res.Records![i].ErrorCode);15    await retryWithBackoff(() => sendClicks(failed)); // throttled records must be retried16  }17}

Complexity and performance

Per shard write1 MB/s or 1,000 records/s

Throttled beyond.

Per shard read2 MB/s shared

Or 2 MB/s per consumer with enhanced fan-out.

Trade-offs

Managed vs flexible

Kinesis removes cluster operations but has fixed per-shard limits and AWS lock-in.

Shard sizing

Provisioned shards are cheaper when load is steady; on-demand is simpler for spiky load.

Variants and related techniques

Data Firehose

No-code delivery to S3, Redshift, OpenSearch, Splunk.

Managed Flink

Stateful streaming SQL and Java applications.

Common mistakes

Hot shards from skewed partition keys.
Fix: Use high-cardinality keys; add random suffixes if order is not needed.
Ignoring partial failures in PutRecords.
Fix: Retry failed records.

Interview questions

How do you size a Kinesis stream?

Shards = max(ingest MB/s / 1, records/s / 1000, consumer read needs / 2), with headroom, or use on-demand mode for unpredictable traffic.

How do you guarantee per-device ordering?

Use the device ID as the partition key so all its records land on one shard, where sequence numbers preserve order; process each shard with one consumer at a time.

Practice problems

Problem	Difficulty	What it trains
Design clickstream ingestion to S3 with Firehose	Easy	Managed pipeline.
Design real-time fraud detection on Kinesis + Flink	Hard	Stateful streaming.