Skip to content
AI Engineer Guide.

concept · data · intermediate · concept 7 of 17

Batch vs streaming

0 of 17 complete

Why it matters

Every pipeline answers one question: how old may this number be when someone acts on it? A nightly job that rebuilds a table is simple to reason about, cheap to run and trivial to replay, but its output is a day old. A streaming job updates within seconds and inherits watermarks, out-of-order events, checkpointing and a cluster that must stay up. Teams that pick streaming for a dashboard nobody watches hourly pay that operational cost for no product benefit.

Mental model

Batch is a scheduled delivery; streaming is a tap that never closes. The delivery can be re-sent if the boxes are wrong, while a tap you already drank from cannot. The metaphor breaks because streams can be replayed from retained logs, so streaming is not strictly lossy, only more expensive to rewind.

How it works

One window, two paths, one late eventInteractive
workwaitriskorphan traceevent
14 min

Event · step 1 of 5

14 min · the stream converges

At 14 min

Four events arrive between 0 and 10 minutes and the stream path processes each one as it lands, publishing by 14 minutes. The dashboard is fresh at the cost of a job that must stay up and keep state.

On screen at this event

  • events
  • run
  • sink
  • 10 min
  • 4

Next at 15 min: 15 min · the watermark closes

Stream count
4
Batch rebuilt
0
Queries see
4
Late dropped
0
Agreement
yes
Step 1/5 · 14 min · the stream converges

Figure. A synthetic 60-minute window with four on-time events and one event that arrives seven minutes after its window closed. The stream publishes four; the served count is stale from minute 22 to 54, when the batch rebuild publishes five. Counts and times are illustrative.

What to notice. Freshness is the decision. The stream is right within seconds and drops the late event because the watermark closed at minute 15 with five minutes of allowed lateness. The batch job reads the raw log and recovers it, so queries are wrong for 32 minutes and right afterwards. Every write needs a deterministic key so the replay does not double-count, and the two paths need a scheduled comparison or they will drift silently.

Text version

Five lanes over 60 minutes: arrivals, watermark, stream, batch and what queries see. Four events arrive from 0 to 10 minutes; a fifth, labelled late, arrives at 22 to 23 in red. A 10-minute window runs 0 to 10 with five minutes of allowed lateness to 15, when a marker reads window closed. The stream processes 0 to 12, sinks four by 14, and drops the late event at 22 to 23. The batch lane holds the raw log from 0 to 45, runs a job over five events 45 to 52 and sinks five at 52 to 54, when a reconciled marker appears. Queries see a normal count of four from 14 to 22, a stale count from 22 to 54, then five from 54. Counters: stream count 4 throughout; batch rebuilt 0, 0, 0, 5, 5; queries see 4, 4, 4, 4, 5, with the fifth value only after the sink completes at 54; late dropped 0, 0, 1, 1, 1; agreement yes, yes, no, no, yes.

  1. Write down the freshness requirement per consumer, because a single pipeline often serves a dashboard at hourly granularity and an alert path at second granularity.
  2. Model the data as bounded or unbounded before choosing a tool; an event feed with no end needs windowing and state, a daily extract does not.
  3. For streaming, define event time, watermark and allowed lateness so late records have a rule instead of a special case.
  4. Make every write idempotent with a deterministic key, because retries and replays are normal in both models.
  5. Keep a batch path over the same raw data as the reconciliation source, and compare the two outputs on a schedule.
  6. Measure end-to-end lag from event creation to query visibility, not just consumer throughput.

Trade-offs

ChoiceGood forCostFails when
Micro-batchNear-real-time with simpler semanticsLatency floor set by batch intervalThe product needs per-event reactions
Continuous streamingLow latency and per-event logicState, checkpointing and on-call burdenThe team has no streaming operations experience
Nightly batchCheap, replayable, easy to reason aboutStale output and a large daily spikeDecisions must react within minutes
Streaming with batch reconciliationFresh reads plus a trustworthy backfillTwo code paths to keep consistentThe two paths drift and nobody compares them
Trade-off mapInteractive

Good for

Near-real-time with simpler semantics

Cost

Latency floor set by batch interval

Fails when

The product needs per-event reactions

  1. The product needs per-event reactions
  2. The team has no streaming operations experience
  3. Decisions must react within minutes
  4. The two paths drift and nobody compares them

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Micro-batch is good for Near-real-time with simpler semantics, costs Latency floor set by batch interval, and fails when The product needs per-event reactions. Continuous streaming is good for Low latency and per-event logic, costs State, checkpointing and on-call burden, and fails when The team has no streaming operations experience. Nightly batch is good for Cheap, replayable, easy to reason about, costs Stale output and a large daily spike, and fails when Decisions must react within minutes. Streaming with batch reconciliation is good for Fresh reads plus a trustworthy backfill, costs Two code paths to keep consistent, and fails when The two paths drift and nobody compares them

Failure modes

  • Replay duplicates. A reprocessed window inserts the same events twice and totals inflate. Debug: count rows per event id and compare against the raw log.
  • Watermark too aggressive. Late events are dropped silently and counts run low for some partitions. Debug: compare per-partition counts with the batch path for the same day.
  • Clock skew in event time. Producers stamp events with local clocks and windows close early. Debug: histogram the difference between ingest time and event time.

In production

Track end-to-end lag as a percentile and alert on the oldest unprocessed event, not on CPU. Cap allowed lateness explicitly and rehearse a replay of one day into a staging table before you need it.

Common misconceptions

People think: Streaming is strictly better because it is faster.

Actually: Latency is one requirement; batch is cheaper per record, simpler to backfill and easier to audit.

People think: Exactly-once delivery removes the need for idempotent writes.

Actually: Delivery guarantees are scoped to a pipeline boundary; sinks still need deterministic keys to survive retries.

Interview angle

A strong answer starts from the freshness each consumer needs, then picks the simplest mechanism that meets it and names the cost of the harder one. It should mention event time, watermarks, idempotency and reconciliation. Weak answers compare tool names and claim exactly-once as a property of the framework rather than of the whole path.

Sources