Message queues and pub/sub

Decoupling producers from consumers with a durable log in between.

System design conceptQueues, streams and batch1 Nov 2026

01The intuition

The problem it solves

A student uploads a video. The server has to store the file, make three smaller copies for phones, scan it for copyright, update the search index and tell her followers. If the server does all of that before replying, she stares at a spinner for two minutes, and if the copyright scanner is down the upload fails even though the file is safely stored. Worse, a popular moment brings 10,000 uploads a minute while the transcoders can only do 2,000, and there is nowhere for the other 8,000 to wait.

In a post office, the counter clerk does not deliver your letter. She stamps it, drops it in a sack, and says "done". Sorting and delivery happen later, by other people, at their own pace, and the sack holds the backlog when a festival brings too much mail. A message queue is the sack. It separates the person who produces work from the people who consume it, and it survives if either of them is temporarily missing.

Build it one piece at a time

Step 1: a queue between two services. The upload server stores the file and appends one message, {video: 42, path: ...}, to a queue, then replies to the user in 50 ms. A transcoder process reads the queue, does its two minutes of work, and acknowledges the message, which removes it. If the transcoder crashes mid-way, the unacknowledged message is handed to another transcoder.

Event	Queue contents	User sees
upload 42 arrives	[42]	done, in 50 ms
uploads 43, 44 arrive; transcoder takes 42	[43, 44]	done, done
transcoder crashes with 42	[42, 43, 44] after timeout	nothing; video appears later

Step 2: many consumers. Run 50 transcoders reading the same queue. Each message goes to exactly one of them. Throughput grows with the number of consumers, and a backlog simply means the queue is long for a while.

Step 3: many interested parties, publish and subscribe. The copyright scanner, the indexer and the notifier all want to know about the upload. Instead of the uploader sending three messages, it publishes one event to a topic, video-uploaded. Each subscriber group gets its own copy and its own position. Adding a fourth consumer next year needs no change to the uploader.

Step 4: make the queue a durable log. Modern systems (Kafka, Pub/Sub) store a topic as an append-only log on disk, replicated to three machines, and split into partitions. A consumer remembers its offset, the position it has read up to. Because messages are not deleted on read, a consumer can rewind and replay yesterday's events, and a new consumer can start from the beginning.

Step 5: ordering and keys. Within a partition, messages keep their order. Messages with the same key, say the same user ID, are always sent to the same partition, so one user's events are consumed in order even though users are spread over 100 partitions.

Why it works, with the numbers

The queue converts a spike into a delay. With arrivals a = 10,000 a minute and capacity c = 2,000 a minute for 10 minutes, the backlog grows at a − c = 8,000 a minute:

backlog after the spike = 10 × 8,000 = 80,000 messages; time to drain = 80,000 / 2,000 = 40 minutes

Nothing failed, and the user was never made to wait. The guarantee is at-least-once delivery: a message is delivered until a consumer acknowledges it, so a crash causes a redelivery, never a loss. That is also the main failure mode: duplicates, which is why consumers must be idempotent. The other failure is unbounded lag, when consumers fall behind for hours and nobody is watching the queue length.

The trade-off you must say out loud

You trade immediacy and simplicity for resilience and elasticity. The user no longer knows when the work finished, so you need a status field, a notification or a page that updates. Say which work is synchronous (must be done before replying, like charging a card) and which is asynchronous (can be done later, like sending the receipt). The follow-up is "at-least-once or exactly-once", and the honest answer is at-least-once delivery plus idempotent consumers.

Where it shows up in the 100 Google problems

YouTube: the upload event feeds transcoding, scanning and indexing through one topic.
Notification system: a queue per channel absorbs bursts and lets senders fail without losing a message.
Managed pub/sub: the log, partitions, offsets and replay are the design itself.
Log pipeline: every machine publishes, many consumers index and archive.
Cross-service transactions: a saga is a chain of messages, each step publishing the next.