Design an A/B experimentation platform

01The design, in the six-step framework

1. Requirements

Functional.

• Product teams define experiments: a population (all users, 10 percent of users in India on Android), variants with traffic splits, and the metrics to compare.
• Every request can ask "which variant is this user in for experiment E" and get the same answer every time, in well under a millisecond, in every service.
• Exposures (user saw variant V of E at time T) are logged, joined with outcome metrics, and reported as differences with confidence intervals.
• Experiments can be ramped, paused and rolled back; overlapping experiments must not corrupt each other.
• Out of scope: the metric definitions themselves, feature-flag use without measurement (though the assignment path is shared).

Non-functional.

• 1,000 concurrent experiments, 500 million users, 2 million assignment lookups per second across the fleet.
• Assignment latency under 100 microseconds (it is in every hot path) and it must work offline from any central service.
• Exposure logging: 100,000 per second, no loss; results refreshed hourly, final daily.
• Config propagation: a pause reaches every server within 30 seconds.

2. Core entities

• Experiment: id, hypothesis, state (draft, running, paused, concluded), start, end, owner, layer.
• Variant: experiment, name, traffic share, parameter overrides.
• Layer: a namespace; each user has one bucket per layer, experiments in the same layer split the buckets and never overlap, experiments in different layers are independent.
• Targeting rule: predicate over user attributes (country, platform, app version, cohort).
• Exposure: user_id hash, experiment, variant, timestamp, context; the first exposure per user-experiment is the one that counts.
• Result: experiment, metric, per-variant mean, delta, confidence interval, sample size, computed_at.

3. API

• In-process SDK: assign(user, experiment) → variant, computed locally from the cached config; and expose(user, experiment, variant), which emits an event when the variant is actually applied.
• GET /config?version= → full experiment set, long-polled by every server; returns 304 when unchanged.
• POST /experiments, POST /experiments/{id}/ramp {shares}, POST /experiments/{id}/pause: the control plane, with validation that layer capacity is not exceeded.
• GET /experiments/{id}/results → metrics table with intervals and the sample ratio check.

4. High-level design

A request and an experiment, end to end:

1. A team creates an experiment in a layer with a 50/50 split over 10 percent of eligible users; the config service validates it and publishes version n+1.
2. Every server long-polls the distribution service and swaps in the new config atomically; a server that cannot reach it keeps using the last version.
3. On a request, the SDK evaluates targeting, computes hash(user_id, layer_salt) mod 10000 to get the user's bucket in that layer, and maps the bucket to a variant from the config; no network call.
4. When the code path actually applies the variant, the SDK emits an exposure to the log; exposures are sunk into the warehouse alongside product events.
5. The stats engine joins first exposures with outcome events per user, computes per-variant metrics with confidence intervals and the sample-ratio check, and writes results the console displays.

5. Deep dives

How do assignments stay consistent, fast and independent across experiments?

Consistent because they are deterministic: the same user and the same layer salt always hash to the same bucket, on any server, with no state to replicate; that is hashing doing the work of a database. Fast because the config is local memory. Independent across experiments because of layers: each layer has its own salt, so a user's bucket in layer A is uncorrelated with their bucket in layer B, and two experiments that would interact (both change the checkout button) are forced into the same layer where they split disjoint bucket ranges and never see the same user. Ramping from 10 to 50 percent expands the bucket range so users already in a variant stay in it; reshuffling (a new salt) is only allowed when the experiment restarts, otherwise a user flips variants mid-experiment and the analysis is garbage.

What is an exposure, and why log it rather than the assignment?

Assignment says a user would get variant B; exposure says the user actually hit the code path and saw it. If the experiment changes the fifth screen of onboarding, most users never reach it, and counting them dilutes the effect to nothing. So the SDK logs on exposure, exactly once per user-experiment (the first exposure is kept; later ones are deduplicated by (user, experiment) in the stats job). The exposure carries the config version, so a result can be traced to the exact split that was live.

How are results computed so people can trust them?

The stats engine is a batch job per experiment: take first exposures, join each user's outcome events in the window, compute the metric per user, then per variant mean, variance, delta and a confidence interval; for ratio metrics use the delta method, for heavy-tailed metrics cap or use a variance-reduction technique such as using the pre-experiment value of the metric as a covariate. Peeking is the human failure mode: a dashboard refreshed hourly invites stopping the moment p dips under 0.05, so the console shows intervals with a fixed-horizon warning or uses a sequential test designed for continuous monitoring. The platform's job is to make the statistically honest path the default path.

How does a pause reach every server in 30 seconds without a thundering herd?

Config is versioned and small; servers long-poll the distribution tier with their current version, and the tier holds the connection until a new version exists or 30 seconds pass. A publish fans out to the distribution tier, which answers all held polls at once with the 2 MB payload (or a delta), served from memory and cached at the edge of each region. A server that misses the update keeps running the old config, which is safe for a pause only if the old config is at most a minute stale, hence the timeout. Kill switches for a broken variant are the same mechanism with a higher priority channel.

6. What a strong answer sounds like

• Say assignment is a local hash with a layer salt, and explain layers as the mechanism for both independence and mutual exclusion.
• Distinguish assignment from exposure and log the latter, once per user.
• Name the sample-ratio check; it is the single most useful guardrail and shows you have run experiments.
• Treat results as a batch computation with explicit statistics, and mention the peeking problem.
• Expected follow-ups: experiments on logged-out users (device ids, and the mess when they log in); network effects (cluster-based assignment); privacy (hashed ids, aggregated results only).
• Common mistake: storing assignments in a database and calling it on every request, or re-randomising when the split changes.

02Where this question was reported

• PracHub · PM