Design a managed pub/sub or message queue with replay

01The design, in the six-step framework

1. Requirements

Functional

• Producers publish messages to a topic; many independent subscriber groups each receive every message.
• Messages within a key are delivered in order; across keys there is no order.
• At-least-once delivery with acknowledgements; unacknowledged messages are redelivered.
• Replay: a subscriber can rewind to a timestamp or offset and reprocess.
• Out of scope: schema registry, exactly-once transactions across topics, cross-topic joins.

Non-functional

• 10 million messages a second across the service; the largest topic takes 1 million a second.
• Publish acknowledged in under 10 ms, p99; end-to-end to a healthy subscriber under 100 ms.
• Durability: once acknowledged, a message survives the loss of any two machines.
• Retention of 7 days by default, configurable to 30, for replay.
• Subscribers may be slow or offline for hours without affecting producers.

2. Core entities

• Topic: topic_id, partition count, retention, owner.
• Partition: topic_id, partition_no, leader replica, follower replicas, log segments on disk.
• Message: offset within the partition, key, payload, publish timestamp, producer id and sequence.
• Subscription: subscription_id, topic_id, consumer group, acknowledged offset per partition (the cursor).
• Consumer: consumer_id, subscription_id, assigned partitions, last heartbeat.

3. API

• POST /topics/{t}/publish with key, payload, producer_id, sequence; returns partition and offset. Called by producers, usually batched.
• POST /subscriptions with topic and group name; returns subscription_id.
• POST /subscriptions/{s}/pull with max messages; returns a batch with offsets. Called by consumers in a loop (or a streaming variant over a long-lived connection).
• POST /subscriptions/{s}/ack with the highest contiguous offset per partition.
• POST /subscriptions/{s}/seek to a timestamp or offset: replay.

4. High-level design

One message from publish to ack:

1. A producer sends a batch with keys; the front end hashes each key to a partition and forwards the batch to that partition's leader.
2. The leader appends the messages to the active log segment on local disk, assigns offsets, and sends the batch to its two followers.
3. When one follower (the in-sync majority) has written to disk, the leader acknowledges the producer with the offsets: the message is now durable against two failures.
4. A consumer in a subscription group pulls from its assigned partitions; the front end reads from the leader (or an up-to-date follower) starting at the group's cursor.
5. The consumer processes the batch and acks the highest contiguous offset; the coordinator records the cursor in the consensus-backed metadata store.
6. Segments older than the retention are moved to object storage, from where a seek to an old timestamp can still be served.

5. Deep dives

How is order guaranteed per key when there are a hundred partitions and many regions?

The simple answer, order by publish timestamp, fails as soon as two producers in two regions have clocks that disagree by 50 ms, which they do. The guarantee offered is weaker and precise: all messages with the same key go to the same partition (hash of key), a partition has one leader, and the leader assigns offsets in arrival order, so per-key order is the leader's append order. Cross-region producers sending the same key therefore race, and the service documents that the order between them is arrival order at the leader, not wall-clock order. If an application needs a global order it must use one key, which caps it at one partition's throughput, about 50,000 a second. Producer sequence numbers let the leader drop a retried duplicate and reject a gap, which keeps a single producer's stream exactly ordered even across its own retries.

What does at-least-once cost the consumer, and how close can we get to exactly-once?

Exactly-once end to end is a property of the consumer's store (write the result and the cursor in one transaction), not of the queue; the queue's job is to make duplicates detectable and loss impossible after ack.

How does replay work without keeping everything on expensive disks?

A seek sets the group's cursor to the first offset at or after a timestamp; each segment carries a sparse time index so the lookup is a binary search over a few hundred entries. Hot segments (the last day) stay on local disk for low-latency tail reads; older segments are uploaded to object storage and read through a cache when a replay touches them. Replay throughput is therefore limited by object storage bandwidth, not by the brokers, and a consumer rewinding seven days does not slow the live subscribers sharing the partition.

How are partitions moved and leaders chosen when brokers die?

The coordinator keeps partition-to-broker assignments and leader identity in a small store backed by consensus so that there is exactly one leader per partition even during a network partition. A broker that stops heartbeating loses its leaderships; the coordinator promotes the most up-to-date in-sync follower, which is safe because acks required a majority. Consumer group membership works the same way: a consumer that stops heartbeating has its partitions reassigned to the group's other members after a 10 s session timeout, and the cursor it last acked is where the next one starts.

What stops a slow subscriber from hurting producers?

Because the log is on disk and retention is time-based, a subscriber that is hours behind simply reads older segments; producers never wait for consumers. Backpressure applies on the producer side only when a partition's leader falls behind on disk or replication, in which case publish requests are throttled with a retry hint rather than buffered in memory without bound.

6. What a strong answer sounds like

• Define the delivery and ordering guarantees precisely before drawing anything; the interviewer is testing whether you know what a queue can and cannot promise.
• Quote the sizing: 10 GB/s written, 18 PB retained, storage sizes the fleet, and the 1 M/s topic needs about 100 partitions.
• Explain why timestamp ordering is wrong and per-partition offsets are right, and what that means for cross-region producers.
• Expect follow-ups on exactly-once, on hot partitions from a skewed key, on tiered storage and on how consumer rebalancing avoids a stampede.
• A common mistake is to put the cursor in the consumer's memory; it must be stored centrally so a replacement consumer resumes correctly.

02Where this question was reported

• Exponent · 'managed pub/sub or message queue used internally across Google'
• PracHub · 'pub-sub replay system', 'distributed message queue service', 'high-throughput event subscription system'
• gitgood.dev · Candidate reports: managed pub/sub used internally; mock interviewer rejected timestamp ordering across regions, probed partition sequencing, idempotency, back
• PracHub · 'High-throughput event subscription system'.
• grokkingsystemdesigninterview.com · Deep dive 'build a message queue from scratch'; probes storage, ordering, consumer recovery, duplicate delivery.
• onsites.fyi · Suggested L5 practice topic.