Batch versus streaming (MapReduce and friends)

Exact answers over everything at night, or approximate answers now.

System design conceptQueues, streams and batch1 Nov 2026

01The intuition

The problem it solves

A school of 5,000 students wants two things from its attendance data. The principal wants a list, on the first of every month, of everyone who missed more than three days, correct to the day. The gate guard wants to know, right now, roughly how many students are inside the building. The same data serves both, but the questions are opposites: one is exact and can wait a month, the other is approximate and cannot wait a minute.

At web scale this is 10 billion events a day. Counting all of them exactly takes a job that reads everything and runs for an hour; that is batch. Keeping a running count that is updated as events arrive, and is right within a percent, is streaming. Most real systems need both, and the design question is which answer goes to which path.

Build it one piece at a time

Step 1: land every event in cheap storage. Each event is appended to a log and copied, hourly, into files in object storage: events/2026-11-01/hour=14/part-0007. These files are the source of truth. Nothing is ever computed that cannot be recomputed from them.

Step 2: the batch job, as MapReduce. To count views per video for the day, split the day's files across 1,000 machines. Each machine reads its files and emits (video, 1) pairs: the map step. Pairs are shuffled so that all pairs for the same video land on the same machine, which adds them up: the reduce step. The output is exact, and the job can be re-run tomorrow if a bug is found.

Stage	Machine 1	Machine 2	After shuffle
map	(42,1) (42,1) (7,1)	(42,1) (9,1)	42: [1,1,1], 7: [1], 9: [1]
reduce	42: 3, 7: 1, 9: 1

Step 3: the streaming job beside it. The same log feeds a stream processor that keeps a running count per video in memory and writes it to a serving store every ten seconds. It is a little wrong, because of late and duplicate events, and it is now.

Step 4: serve both through one view. The dashboard reads the exact batch number for everything up to the last completed hour, and the streaming number for the current hour, and adds them. This is the lambda architecture. The simpler cousin, the kappa architecture, drops the batch job and replays the stream from the log whenever an exact recomputation is needed, which works when the stream job is itself exact enough.

Step 5: handle the straggler. In a 1,000-machine batch, one machine is always slow. The scheduler notices a task running far longer than its siblings and starts a duplicate on another machine; whichever finishes first wins. Without this, the job's time is the slowest machine's time.

Why it works, with the numbers

The batch job's time is data divided by aggregate throughput. Ten billion events at 200 bytes is 2 TB; with 1,000 machines each reading 100 MB a second:

time ≈ 2,000,000 MB / (1,000 × 100 MB/s) = 20 s of reading, plus shuffle and reduce, so a few minutes

The streaming job's cost is one addition per event and state proportional to the number of keys, as the stream-processing article shows. The guarantee of batch is exactness and reproducibility; the guarantee of streaming is bounded delay. The failure modes: a batch job that finishes after the deadline (yesterday's totals arrive at noon), a stream job whose approximate numbers are mistaken for exact ones, and two code paths that compute the "same" metric slightly differently and never agree.

The trade-off you must say out loud

Freshness against exactness, and one pipeline against two. Say which numbers are billed or reported (batch, exact) and which are displayed (stream, approximate), and how the two are reconciled. If you choose kappa, say how long a replay of the whole log takes. The follow-up is "why not just make the stream exact", and the answer is that late data and duplicates make that expensive, and the batch over immutable files is the cheap way to be sure.

Where it shows up in the 100 Google problems

Ad click aggregation: minute counts from the stream for dashboards, the exact daily bill from a batch over the log.
Sort 1 TB on 1,000 machines: a pure batch question: partition by range, sort locally, merge, and handle the straggler.
Play Store payouts: monthly money is computed by batch, with refunds and tax, and never by a running counter.
Data lake ingestion: the landing layer that both paths read from.
Google News: clustering runs as a batch every few minutes while a stream handles breaking stories.