A/B testing and experimentation
Assigning users consistently and measuring a difference you can trust.
01The intuition
The problem it solves
The team has a new ranking model. On the test set it scores 2 percent better. Does it make real users happier? The only way to know is to show it to some users and compare with the rest. But comparisons are easy to get wrong. If the new model is shown on Tuesday and the old one on Monday, Tuesday's weather, news and holidays pollute the result. If a user sees the new model on their phone and the old one on their laptop, their behaviour on each is not independent. If the team looks at the numbers every hour and stops when they look good, random noise will eventually look good.
An experimentation platform exists to make these mistakes hard: it assigns users consistently, logs what each saw, and reports a difference with an honest uncertainty.
Build it one piece at a time
Step 1: assign by hashing, not by storing. For experiment E, compute hash(user id, E) and take the result modulo 100. Users with 0 to 49 get the control, 50 to 99 the treatment. The same user always lands in the same group, on every device, with no database lookup, because the hash is deterministic. Different experiments use different salts so one user's group in E says nothing about their group in F.
| User | hash(user, E) mod 100 | Group in E | hash(user, F) mod 100 | Group in F |
|---|---|---|---|---|
| asha | 17 | control | 83 | treatment |
| ravi | 64 | treatment | 22 | control |
Step 2: log the exposure. The moment a user is actually shown the experiment's surface, write an exposure event: user, experiment, group, time. Users who were assigned but never saw the feature are excluded from the analysis, which removes a lot of noise.
Step 3: join exposures with outcomes. Outcome events (clicks, watch time, purchases) flow through the normal event pipeline. A nightly job joins each outcome to the user's exposure and computes the metric per group.
Step 4: report a difference with a confidence interval. The platform computes the mean per group, the standard error, and a confidence interval for the difference. It also runs an A/A check in the background: two groups with the same experience should show no difference, and if they do, the pipeline is biased.
Step 5: decide the sample size before starting. To detect a 1 percent lift in a metric with a 20 percent standard deviation, at 95 percent confidence and 80 percent power, each group needs about
n ≈ 16 × (σ ÷ δ)2 = 16 × (0.20 ÷ 0.01)2 = 6,400 users per group, more for rarer outcomes
The experiment runs until that many users have been exposed, for at least one full week to cover weekday and weekend behaviour, and the result is read once.
Why it works, with the numbers
Hashed assignment makes the groups random with respect to everything except the treatment, which is what lets the difference in outcomes be attributed to the treatment. The confidence interval turns "the treatment was 1.3 percent better" into "between 0.4 and 2.2 percent better", which is the statement the team can act on. The guarantee is statistical: with the stated sample size the test will detect a real effect of the chosen size 80 percent of the time and will raise a false alarm 5 percent of the time. The honest failure modes: peeking at the result repeatedly inflates the false-alarm rate far above 5 percent unless a sequential method is used; network effects (a treated user messaging a control user) break independence and need cluster-level assignment; and running fifty experiments at once means a few will "win" by chance, which is why platforms adjust for multiple comparisons.
The trade-off you must say out loud
More traffic per experiment gives faster, surer answers but fewer concurrent experiments; layering experiments across orthogonal hashes lets many run at once at the cost of interactions you must watch for. The interviewer's follow-up is how to ship safely when the metric cannot wait two weeks: guardrail metrics (errors, latency, crashes) that stop an experiment within hours, with the success metric read later.
Where it shows up in the 100 Google problems
- Design an A/B experimentation platform: hashed assignment, exposure logging, the join with outcomes, intervals, A/A checks and layering.
- Design a recommendation system: every model change is an experiment, and the serving path reads its group from the assignment hash.
- Monitor an ML model's performance in real time: the control group is the baseline that tells drift in the world from a worse model.