Manage user preferences across many Google products

01The design, in the six-step framework

1. Requirements

Dozens of products (Search, Maps, YouTube, Gmail, Photos) each need to read a user's settings on every request: language, region, dark mode, safe search, ad personalisation, notification choices. The question is to design the one store they all share.

Functional

• Read all preferences for a user, or one namespace (a product's own settings), on any request.
• Write a preference from any product or from the account settings page; the change is visible everywhere quickly.
• Namespaces owned by products, with schemas, defaults and validation; a product cannot write another product's namespace.
• History and audit of changes; a user can see what changed and when.
• Out of scope: per-device state like window size, and anything that is not a user-chosen setting.

Non-functional

• Scale: 2 billion accounts, 1 million reads a second across products, 5,000 writes a second.
• Latency: read under 5 ms inside the product's data centre at p99; write under 200 ms.
• Consistency: read-your-writes for the user who changed a setting; other products see it within 1 second; never an old value after a new one was seen.
• Availability: 99.999 percent for reads, since every product depends on it; reads must survive a region loss.

2. Core entities

• Namespace: namespace id, owning product, schema (keys, types, allowed values, defaults), version.
• Preference: user id, namespace, key, value, version (monotonic per user), updated by, updated at.
• UserBundle: user id, map of namespace → values, bundle version (max of preference versions); the unit that is cached.
• ChangeEvent: user id, namespace, key, old, new, version, actor; the audit record and the replication payload.

3. API

• GET /users/{id}/prefs?ns=youtube,core returns the requested namespaces with the bundle version; called by product backends, never by browsers directly.
• PATCH /users/{id}/prefs/{ns} with key-value changes and an expected version; returns the new version or a conflict. The caller must own the namespace.
• GET /users/{id}/prefs/history?ns= for the audit view.
• POST /namespaces/{ns}/schema for product teams to register keys and defaults.

4. High-level design

1. A product backend asks its in-process client library for a user's bundle. The library holds a short-TTL cache keyed by user and bundle version.
2. On a miss, the library calls the read service in the same region, which reads the regional replica.
3. A write goes to the user's home region, is applied to the primary store with a version check, and is appended to the change log.
4. Replication workers apply the change to every regional replica and send invalidation messages to the client libraries, which drop the user's cached bundle.

5. Deep dives

How do we get 5 ms reads at a million a second with five nines?

Three layers, each cheaper than the one below. The in-process cache in the client library answers most reads in microseconds; it is bounded by memory (10,000 hot users per process is 20 MB) and by a 60-second TTL. The regional read service fronts a replica in the same region, so a miss costs one local round trip. The replica is a full copy of the 4 TB dataset, sharded by user id, so a region never crosses the ocean to read. For availability, reads never depend on the home region: if the primary is unreachable, reads continue from the replica, writes queue. This is the deliberate trade: reads are highly available and slightly stale; writes are strongly ordered and may wait.

How does "never an old value after a new one" hold with caches everywhere?

Every bundle carries a version, monotonic per user, assigned by the home-region write service. Replicas and caches accept a bundle only if its version is higher than what they hold, so replication arriving out of order cannot regress a value. Read-your-writes for the user who changed a setting is done by returning the new version to the settings UI, which passes it on subsequent requests as a minimum version; a read service that holds an older version waits briefly for replication or forwards the read to the home region. Other products are allowed up to a second of staleness, which the invalidation fan-out meets comfortably at 5,000 writes a second.

How do we keep products from stepping on each other?

Namespaces are the unit of ownership and schema. A product registers its keys with types and defaults; writes to a namespace must come from the owning service, enforced by service identity on the store's write path. Cross-cutting settings (language, region, personalisation consent) live in a core namespace owned by the account team, readable by everyone. Defaults are applied at read time from the schema, so adding a key never requires backfilling 2 billion rows: a missing value means the default. Schema changes are versioned and additive; removing a key is a two-step deprecation so old readers keep working.

What happens when a region is lost?

Each user has a home region for writes, chosen by residency rules. If the home region fails, a failover promotes a replica in another region to primary for the affected users; because the change log is replicated synchronously to at least one other region before a write is acknowledged, the promoted region has every acknowledged write. Reads were never affected. The cost is write latency: synchronous cross-region replication adds about 50 to 100 ms, which is inside the 200 ms write budget and buys zero data loss on failover.

How are hot users and cache invalidation kept cheap?

Invalidation is a fan-out of a tiny message (user id, new version) to every process that might hold the bundle. Rather than tracking which processes cache which users, publish invalidations on a per-region topic that every client library subscribes to; at 5,000 writes a second that is 5,000 small messages a second per region, well within a pub/sub channel. A library that receives a version higher than its cached one drops the entry; the next read refetches. If the subscription lags, the 60-second TTL bounds staleness. For the read service's own cache, use the same rule and add request coalescing so that a thousand concurrent misses for one user (a celebrity whose settings every product reads at once) become one replica read.

6. What a strong answer sounds like

• Size it immediately: 4 TB is small, so replicate everywhere and make reads local; the hard part is versioning, not storage.
• State the three-layer read path and the staleness each layer may introduce.
• Give the monotonic version rule and how read-your-writes rides on it.
• Explain namespaces as ownership plus schema with defaults at read time.
• Follow-ups to expect: privacy deletion (delete the user's rows and emit a tombstone event that purges every cache), consent changes that must be seen immediately by ad systems (treat them as a strongly consistent namespace read from the home region), and a thundering herd on a popular user (the bundle is cached per process, so a celebrity's settings are read from a million processes, which is fine).

02Where this question was reported

• IGotAnOffer · Google TPM list