Send 100 million emails with five machines

01The design, in the six-step framework

1. Requirements

Functional

• Send one campaign of 100 million personalised emails from a template and a recipient list.
• Every recipient receives the email exactly once, or is recorded as undeliverable.
• Progress and a final report: sent, bounced, deferred.
• Out of scope: building the recipient list, tracking opens and clicks, replies.

Non-functional

• Hardware is fixed: five machines, each with 8 cores, 32 GB RAM, a 1 Gbps link.
• Finish within 24 hours; the interviewer wants to know whether that is even possible.
• No duplicates after a crash; a machine may die and be replaced mid-run.
• Stay within receiving providers' limits so the sending domain is not blacklisted.

2. Core entities

• Campaign: campaign_id, template_id, total recipients, state.
• Recipient batch: batch_id, campaign_id, range of recipient ids, assigned worker, state (pending, in progress, done).
• Recipient: recipient_id, address, personalisation fields, provider domain.
• Send record: recipient_id, campaign_id, state (sent, bounced, deferred), attempts, last_error.
• Provider budget: domain, max concurrent connections, max per minute, current window counters.

3. API

• POST /campaigns with template_id and recipient list location; returns campaign_id. Called by the operator.
• POST /campaigns/{id}/start: splits the list into batches and makes them claimable.
• POST /batches/claim with worker_id: hands out the next pending batch with a lease. Called by workers.
• POST /batches/{id}/progress: the worker reports the last recipient completed and renews its lease.
• GET /campaigns/{id}: counts by state, estimated finish time. Called by the operator's dashboard.

4. High-level design

How the run proceeds:

1. The operator starts the campaign. The coordinator (one of the five machines, with its state replicated to a second) splits 100 M recipients into 10,000 batches of 10,000, sorted so that each batch mixes provider domains.
2. Each of the four worker machines claims a batch with a 10-minute lease, loads the 10,000 recipient rows, and renders the template for each.
3. The worker sends through a pool of long-lived SMTP connections per provider domain, each pool capped by that provider's rate limit; messages for a saturated provider wait while others proceed.
4. Every 500 recipients the worker reports progress, which renews the lease and persists the high-water mark.
5. Bounces come back to a dedicated mailbox; a small consumer marks the send records and feeds a suppression list.
6. When all batches are done the coordinator produces the report.

5. Deep dives

Can five machines finish in a day, and what is the real bottleneck?

The naive calculation says yes with room to spare: 1,160 sends a second is light work for four senders. The bottleneck is reputation. Receiving providers throttle unknown senders to a few hundred messages an hour and back off further on complaints. A cold domain cannot send 100 M in a day; it is blocked by lunch. So the honest answer is: feasible in 24 hours only from a domain that has been warmed over weeks and whose list is clean. The capacity plan therefore has two numbers: our send rate (thousands per second available) and their accept rate (the binding constraint), and the schedule is set by the smaller.

How do we avoid sending anyone two copies after a crash?

A worker that dies mid-batch leaves up to 10,000 recipients in doubt. Replaying the whole batch would resend to everyone already done. The progress report every 500 messages bounds the doubt to 500; the replacement worker restarts from the high-water mark. For those 500 the design accepts a small chance of duplicates, or, better, writes a per-recipient sent flag before each SMTP call and checks it on resume, which makes the resend idempotent at the cost of one local write per message. The flag write is cheap because it goes to the worker's local disk and is reconciled with the coordinator when the batch closes.

How do retries and deferrals work without blocking the pipeline?

Deferred messages go to a separate retry queue per provider so that one provider's greylisting does not stall the others; the retry queue is drained by the same workers at lower priority.

Why batches instead of one shared queue of 100 million?

A single queue needs a message broker that is itself a sixth machine and a point of failure. Batches with leases give the same at-least-once claim semantics using a small replicated table: a claim is an atomic update of one row, progress is a single column, and a lost lease is detected by a scan every minute. With 10,000 batches and four workers the scheduling overhead is negligible and a slow provider in one batch does not block the next claim.

How do we know we are being throttled, and what do we change?

Each worker records, per provider domain, the accept rate, the share of 4xx responses and the average response time in one-minute windows. A rising 4xx share with constant request volume is the throttle signal; the worker halves that provider's connection pool, waits five minutes, and grows it again by one connection a minute while the 4xx share stays low, which is the same additive-increase, multiplicative-decrease idea TCP uses. The coordinator's dashboard shows per-provider progress so the operator can see that, for example, one provider will finish in three hours and another will spill into tomorrow, and can decide whether to slow the whole campaign to protect the domain's reputation.

6. What a strong answer sounds like

• Do the arithmetic out loud first: 1,160 a second, 2 TB total, 23 MB/s, and conclude that neither CPU nor bandwidth is the limit.
• Name the real constraint, provider reputation and rate limits, and say the schedule is set by their accept rate, not our send rate.
• Describe the batch-and-lease model and the high-water mark that bounds duplicates after a crash.
• Expect follow-ups on warming a domain, on what to do if one machine is much slower, and on how you would know you are being throttled.
• A common mistake is to design a Kafka cluster and a microservice fleet for a problem that fits five machines and a table.

02Where this question was reported

• interviewkickstart.com · Listed
• interviewkickstart.com · Listed
• interviewkickstart.com · Listed verbatim.