Build an authentication platform
Passwords, sessions, OAuth and two-factor for a billion accounts with no single key to steal.
- Authentication and authorization — Proving who you are, and checking what you may do, on every request.
- Encryption and key management — Keys that encrypt keys, rotation, and hardware that never gives up a secret.
- Replication — Keeping copies so reads scale and a dead machine loses nothing.
- Rate limiting — Token buckets, sliding windows, and enforcing them across many servers.
- Privacy and PII handling — Minimising, isolating and deleting personal data on purpose.
01The design, in the six-step framework
1. Requirements
Hello Interview notes that Google interviewers ask about their own area; an identity engineer asks this one. Scope it to the account platform behind sign-in for a billion users and for the services that need to know who is calling.
Functional
- Sign up, sign in with password, passkeys and third-party identity providers; two-factor with codes and security keys.
- Issue sessions to browsers and apps, and short-lived tokens to services acting for a user; refresh and revoke them.
- Account recovery, suspicious-sign-in detection, and sign-out everywhere.
- An authorization check that any service can call: is this token valid, who is it, what scopes does it hold.
- Out of scope: fine-grained resource permissions inside products, enterprise directory sync.
Non-functional
- Scale: 1 billion accounts, 100 million sign-ins a day, 5 million token validations a second across services.
- Latency: sign-in under 300 ms; token validation under 1 ms (must be local to the caller).
- Security: no single key whose theft compromises every session; passwords never recoverable; revocation effective within 10 seconds.
- Availability: 99.999 percent for validation and sign-in; a region loss must not sign anyone out.
108 sign-ins/day ≈ 1,200/s, 10,000/s at peak; password hashing at ~100 ms of CPU each = 1,000 cores at peak, so hashing capacity is a real cost
5 × 106 validations/s cannot hit a store; tokens must be verifiable locally with cached public keys
Account data: 109 × 2 KB = 2 TB, replicated to every region; sessions: 3 × 109 active × 200 bytes = 600 GB
2. Core entities
- Account: account id, identifiers (email, phone), credential records, recovery options, status, security settings.
- Credential: type (password hash with parameters, passkey public key, provider link, TOTP secret), created at, last used.
- Session: session id, account id, device, created, last active, risk level, revoked flag; the long-lived browser state.
- AccessToken: short-lived signed token with account id, session id, scopes, expiry, key id; RefreshToken: opaque, bound to the session.
- SigningKey: key id, public key, private half in an HSM, validity window.
3. API
POST /signinwith identifier and credential (password, passkey assertion, provider code); may return a challenge requirement; on success sets a session cookie and returns tokens.POST /tokenwith a refresh token → new access token;POST /revokefor a session or all sessions of an account.GET /.well-known/jwkspublic keys for local validation;POST /introspectfor callers that need revocation-precise answers.POST /recovery/startand/complete;POST /mfa/enroll.
4. High-level design
- A sign-in arrives at the gateway, which applies rate limits per account and per IP and consults the risk engine for a step-up decision.
- The sign-in service verifies the credential against the account store (a slow memory-hard hash for passwords, a signature check for passkeys), then asks the token service for a session.
- The token service creates a session record, mints a short-lived access token signed by an HSM-held key, and returns an opaque refresh token bound to the session.
- Product services validate access tokens locally with cached public keys and subscribe to a revocation feed; they call introspection only for high-value operations.
5. Deep dives
Why short-lived signed tokens plus a revocation feed, and what is the revocation gap?
Validating 5 million tokens a second against a store is impossible, so tokens are self-contained: signed with an asymmetric key, carrying account, session, scopes and a 10-minute expiry. A service verifies the signature in microseconds with public keys fetched hourly. The cost is that a revoked session's tokens stay valid until expiry, so the token service publishes revoked session ids on a feed every service subscribes to, held in a small in-memory set until the tokens would have expired anyway. Revocation is then effective within the feed's propagation time, a few seconds, and the set stays small (sessions revoked in the last 10 minutes). High-value operations (changing a password, a payment) call introspection, which checks the session store directly. Say both halves; the gap is the point (authentication and authorization).
How are credentials stored and keys protected so one theft is not everything?
Passwords are hashed with a memory-hard function and a per-user salt, with parameters stored alongside so they can be raised over time; a breach of the store yields hashes that cost seconds of CPU each to test. Passkeys store only public keys. Signing keys for tokens live in HSMs, rotate every few days with overlapping validity, and each region signs with its own key so a compromised region's key can be retired without invalidating the world; the key id in the token tells validators which public key to use. Secrets such as TOTP seeds are encrypted with envelope keys from the key management service, so a database dump is useless without the HSM. Session identifiers are random, never derived from account data.
How does the platform stay up and keep users signed in through a region loss?
Accounts are replicated to every region with a home region for writes; sign-in reads can be served anywhere, and credential changes are rare enough to pay a cross-region write. Sessions are replicated too, so a refresh request landing in another region finds the session. Access tokens need no store at all, which is why a region loss does not sign anyone out: existing tokens keep validating with cached keys, refreshes are served by the surviving regions, and only sign-ins briefly slow while traffic moves. Hashing capacity is the fragile part under a sign-in surge; keep headroom and shed credential-stuffing traffic at the gateway before it reaches the hashers.
How do recovery and risk avoid becoming the weakest link?
Recovery is the attack surface attackers prefer. Require two independent factors to recover (a recovery code and a verified phone, or a trusted device approval), add a delay with notifications to all existing devices so the real owner can intervene, and never let support staff bypass it without a logged, dual-approved process. The risk engine scores every sign-in with device familiarity, location, and velocity, and asks for a second factor on a new device by default; it never reveals whether an identifier exists. Privacy rules bound what the risk engine keeps: hashed device identifiers, coarse location, short retention, and deletion when the account is deleted.
6. What a strong answer sounds like
- Split the two problems: proving identity (slow, rare, risk-scored) and checking tokens (fast, constant, local), and size each.
- Explain self-contained tokens with a revocation feed and the gap it closes; mention introspection for sensitive operations.
- Describe key hygiene concretely: per-region HSM keys, rotation with overlap, memory-hard password hashing, envelope encryption of secrets.
- Show the region-loss story: tokens validate, refreshes move, sign-ins slow.
- Follow-ups to expect: OAuth for third-party apps (consent, scopes, refresh-token rotation), service-to-service identity (workload certificates, not user tokens), and session management UX (list and revoke devices).
02Where this question was reported
- Hello Interview · Google example; page notes no central question bank, interviewers ask about their own area