Batch vs streaming

Overview

Batch processing collects data over a period and processes it all at once, for example a nightly job computing yesterday's sales. Stream processing handles each event (or small window) continuously as it arrives, producing results within seconds. Batch is simpler, cheaper, and easier to reprocess; streaming gives low latency but must handle late, duplicate, and out-of-order events.

Modern data platforms often combine both. The Lambda architecture runs a batch layer for accuracy and a speed layer for freshness; the Kappa architecture uses one streaming pipeline and replays the log to reprocess. Engines like Apache Flink and Spark Structured Streaming unify the two.

Laundry

Batch is doing all the laundry on Sunday: efficient, but a shirt worn Monday is not clean until next week. Streaming is washing each item right after use: always fresh, but you are running the machine constantly.

When to use it

Batch: reports, billing, ML training, backfills, data warehouse loads.
Streaming: fraud detection, live dashboards, alerts, recommendations updates.
Deciding freshness requirements for a data pipeline.
Interview designs for analytics and metrics.

Where it shows up in interviews

Real-time analytics

Recognize it when: results needed within seconds.

Design real-time fraud detection
Design a live leaderboard of trending topics

Periodic aggregation

Recognize it when: daily or hourly results are enough.

Design a billing system
Design ad click aggregation reports

Where it is used in real software

Apache Spark and Hadoop

Batch processing of massive datasets in data lakes.

Apache Flink

Stateful stream processing with event-time windows and exactly-once state, used by Alibaba, Uber, and Netflix.

Ad click aggregation

Ad platforms stream clicks for near-real-time budgets and reconcile with batch jobs for billing accuracy.

Key terms

Event time vs processing time: When it happened vs when it was processed.
Window: Group events by time: tumbling, sliding, session.
Watermark: Estimate that events up to time T have arrived.
Late data: Events arriving after their window closed.
Lambda / Kappa architecture: Batch plus stream / stream-only with replay.

How it works, step by step

1
Define freshness needs
Seconds, minutes, hours, or daily.
2
Pick the model
Batch for large periodic jobs, streaming for continuous low-latency results.
3
Handle time correctly
Use event time, windows, and watermarks for streams.
4
Ensure correctness
Idempotent sinks, exactly-once state, deduplication.
5
Plan reprocessing
Replay the log or rerun batch jobs after bugs.

Key differences

Aspect	Batch	Streaming
Latency	Minutes to hours	Milliseconds to seconds
Data	Bounded (a day's files)	Unbounded (continuous)
Complexity	Lower	Higher (state, late data, ordering)
Cost	Efficient bulk processing	Always-on resources
Tools	Spark, dbt, Airflow	Flink, Kafka Streams, Spark Streaming

NOWAspect: Latency | Batch: Minutes to hours | Streaming: Milliseconds to seconds

Start with batch unless the business needs low latency; add streaming for the specific use cases that do.

Implementation

1-- Count clicks per ad in 1-minute tumbling windows by event time2CREATE TABLE clicks (3  ad_id STRING,4  user_id STRING,5  ts TIMESTAMP(3),6  WATERMARK FOR ts AS ts - INTERVAL '10' SECOND   -- tolerate 10 s of lateness7) WITH ('connector' = 'kafka', 'topic' = 'ad-clicks', 'format' = 'json');8 9SELECT ad_id,10       TUMBLE_START(ts, INTERVAL '1' MINUTE) AS window_start,11       COUNT(*) AS clicks12FROM clicks13GROUP BY ad_id, TUMBLE(ts, INTERVAL '1' MINUTE);

Complexity and performance

Batch latencyJob interval + runtime

Hours typical.

Streaming latencyWindow + watermark delay

Seconds typical.

Trade-offs

Freshness vs complexity and cost

Streaming delivers fresh results but needs always-on infrastructure and careful handling of time and state.

Accuracy

Streams must decide how long to wait for late data; batch sees complete data.

Variants and related techniques

Micro-batching

Spark Structured Streaming processes small batches every few seconds.

Kappa architecture

One streaming codebase; reprocess by replaying Kafka.

Common mistakes

Using processing time for business metrics.
Fix: Delays and replays shift events into the wrong windows; use event time.
Choosing streaming for daily reports.
Fix: Batch is cheaper and simpler when freshness is not needed.

Interview questions

How would you aggregate ad clicks for billing and a live dashboard?

Stream clicks through Kafka into Flink with event-time windows for the dashboard (seconds of latency), and run a batch job over the raw, deduplicated click log for billing, reconciling differences.

What is a watermark?

A marker in a stream asserting that no more events older than a given time are expected, letting windows close and emit results while tolerating a bounded amount of lateness.

Practice problems

Problem	Difficulty	What it trains
Choose batch or streaming for 6 use cases	Easy	Freshness needs.
Design an ad click aggregation system	Hard	Streaming + batch reconciliation.