Batch vs Stream Processing: How to Choose for Your Data Pipeline

Batch vs stream processing is a choice about when data gets processed. Batch processing collects data over a period and processes it all at once on a schedule, such as every hour or every night. Stream processing handles each event, or small groups of events, continuously as they arrive, producing results within seconds. Batch is simpler and cheaper to operate; streaming is worth its extra complexity only when fresher results create real value.

What is batch processing?

A batch job reads a bounded dataset, such as yesterday's orders or all files that landed in the last hour, processes it, and writes the output. When it finishes, it stops. The next run picks up the next slice of data.

Batch processing is a natural fit for:

• Daily and hourly reporting and dashboards.
• Building warehouse tables from raw data (the transform step of ETL vs ELT).
• Large historical backfills and reprocessing.
• Machine learning training datasets.

The biggest strength of batch is that the input is complete and fixed while the job runs. That makes results deterministic and reruns straightforward: if a job fails or has a bug, fix it and run it again over the same input. Engines such as Apache Spark (see Apache Spark explained) and warehouse SQL are the usual tools.

What is stream processing?

A stream processor reads from an unbounded source, usually an event log such as Kafka or a managed equivalent, and keeps running. It updates results continuously as new events arrive.

Stream processing fits:

• Fraud and anomaly detection where decisions must happen within seconds.
• Real-time metrics, alerting, and operational monitoring.
• Keeping search indexes, caches, or feature stores in sync with source data.
• Change data capture pipelines that replicate database changes downstream.

Common engines include Apache Flink, Spark Structured Streaming, and Kafka Streams. Because the input never ends, streaming introduces concepts that batch jobs can mostly ignore: windows, event time, watermarks, and state.

Batch vs stream processing compared

Event time vs processing time

Every event has at least two timestamps:

• Event time: when the thing actually happened, for example when a user clicked a button on their phone.
• Processing time: when your system gets around to handling the event.

These can differ significantly. A mobile device might go offline and upload a burst of events an hour later. If you aggregate by processing time, those clicks land in the wrong hour. For correct analytics you almost always want to group by event time, which means the processor must handle events that arrive out of order.

How do windows work in stream processing?

Aggregations over an infinite stream need boundaries. Windows provide them:

1. Tumbling windows: fixed, non-overlapping intervals, for example every 5 minutes.
2. Sliding (hopping) windows: fixed size with overlap, for example a 10-minute window every minute.
3. Session windows: dynamic windows that close after a period of inactivity per key, useful for user sessions.

Watermarks and late data

If you group by event time, when is a window "done"? A watermark is the processor's estimate that no more events older than a certain timestamp are expected. When the watermark passes the end of a window, the window can emit its result.

Choosing the watermark delay is a trade-off. A short delay gives faster results but drops or separately handles more late events. A long delay captures more late events but increases latency and state size. Events that arrive after the watermark are "late data," and most engines let you drop them, route them to a side output, or update previously emitted results.

Here is a tumbling-window aggregation in Flink SQL with an event-time watermark that tolerates 30 seconds of disorder:

CREATE TABLE clicks (
  user_id    STRING,
  url        STRING,
  event_time TIMESTAMP(3),
  WATERMARK FOR event_time AS event_time - INTERVAL '30' SECOND
) WITH (
  'connector' = 'kafka',
  'topic' = 'clicks',
  'properties.bootstrap.servers' = 'kafka:9092',
  'scan.startup.mode' = 'latest-offset',
  'format' = 'json'
);

SELECT window_start, window_end, url, COUNT(*) AS clicks
FROM TABLE(
  TUMBLE(TABLE clicks, DESCRIPTOR(event_time), INTERVAL '5' MINUTES)
)
GROUP BY window_start, window_end, url;

What does exactly-once really mean?

Many streaming engines advertise exactly-once processing. It is a real and useful guarantee, but it is narrower than it sounds:

• It usually means exactly-once effects on the engine's internal state, achieved through checkpoints and replay. Events may be physically processed more than once after a failure, but state ends up as if each was processed once.
• End-to-end exactly-once also requires the sink to cooperate, either through transactional writes tied to checkpoints or idempotent writes such as upserts keyed by a unique ID.
• Side effects outside the pipeline, like sending an email or calling an external API, are not covered. Retries can repeat them unless you design them to be idempotent.

In practice, many teams aim for at-least-once delivery plus idempotent sinks, which is simpler to reason about. The delivery guarantees guide goes deeper into these semantics.

Lambda vs kappa architecture

Two well-known patterns describe how to combine batch and streaming.

Lambda architecture runs two parallel paths: a batch layer that periodically recomputes accurate results from all historical data, and a speed layer that produces approximate real-time results. A serving layer merges them. It provides both correctness and freshness, but you maintain the same logic twice, often in two different frameworks, and the two can drift.

Kappa architecture uses only a streaming path. All data flows through a replayable log, and reprocessing is done by replaying the log through a new version of the streaming job. It avoids duplicated logic, but depends on long log retention and a stream processor capable of handling historical replays efficiently.

Many modern platforms land somewhere in between: streaming for fresh operational data, and batch jobs over the same data in a lakehouse for heavy historical computation.

How to choose between batch and stream processing

Ask these questions in order:

1. What latency does the business actually need? If a dashboard is checked each morning, hourly batch is plenty. If a fraudulent transaction must be blocked before approval, you need streaming.
2. What is the cost of being late? Quantify it. Streaming is justified when freshness changes a decision or an outcome.
3. Can your team operate it? Streaming jobs run continuously, hold state, and need on-call attention.
4. Is micro-batching enough? Running small batches every few minutes often covers "near real-time" needs with much less complexity.
5. How will you reprocess? Plan for bugs. Batch reruns are easy; streaming replays need log retention and state handling.

Key takeaways

• Batch processes bounded data on a schedule; streaming processes unbounded data continuously.
• Batch is simpler, cheaper, and easier to rerun; streaming offers low latency at higher operational cost.
• Use event time, windows, and watermarks to get correct streaming results when events arrive late or out of order.
• Exactly-once guarantees usually cover engine state; end-to-end correctness also needs transactional or idempotent sinks.
• Start with batch or micro-batch and move to streaming only when fresher data changes outcomes.

Frequently asked questions

Is stream processing always better than batch?

No. Streaming gives lower latency but adds complexity in state management, late data handling, and operations. For many analytics workloads, scheduled batch or micro-batch is cheaper and entirely sufficient.

What is micro-batching?

Micro-batching processes data in small batches at short intervals, such as every few seconds or minutes. It offers near real-time results while keeping much of the simplicity of batch processing, and it is the model used by Spark Structured Streaming by default.

What is a watermark in stream processing?

A watermark is a marker of event-time progress that tells the processor it can assume no older events are coming. It lets windowed aggregations emit results while still tolerating a bounded amount of out-of-order data.

Should I use lambda or kappa architecture?

Kappa is simpler if your stream processor can replay history efficiently and your log retains enough data. Lambda can make sense when historical recomputation is very heavy and better suited to batch engines, but it requires maintaining two code paths.