Design a global configuration and feature-flag service

01The design, in the six-step framework

1. Requirements

The onsite framing is "a globally consistent configuration service like FlagService": thousands of services read flags on every request, and a change must reach a million servers in seconds, in order, with a way to undo a mistake instantly.

Functional

• Store configuration as versioned keys in namespaces: feature flags (boolean, percentage, per-cohort), tunables (numbers, strings), and small structured values.
• Read any key in-process with no network call on the hot path; see a change within 5 seconds of it being published.
• Staged rollouts: a flag change applies to 1 percent of servers or users, then more, with automatic halt on bad metrics.
• Full history and instant rollback; every change has an author and a review.
• Out of scope: secrets (the key management service), large files.

Non-functional

• Scale: 1 million server processes, 500,000 keys across 5,000 namespaces, 10 billion reads a second in-process, 100 changes a minute.
• Propagation: p99 under 5 seconds globally; ordering: no server ever applies version v after v+1.
• Availability: a server that starts with the service down still gets a valid configuration; a reader never blocks.
• Safety: a change cannot take down the fleet faster than it can be rolled back.

2. Core entities

• Namespace: namespace id, owning team, ACL, current version.
• Key: namespace, name, type, default value, targeting rules (percentage, cohorts, server labels), schema.
• Version: namespace, version number, author, review id, the full set of key values at that version (snapshot), created at.
• Rollout: version, stages (percentage, duration, guard metrics), current stage, state (in progress, paused, completed, rolled back).
• Subscription: process id, namespaces, acknowledged versions.

3. API

• In-process: flag("checkout.new_flow", context) → value; on_change(namespace, callback).
• Client library ↔ service: WATCH namespaces since versions over a streaming connection; the service pushes (namespace, version, delta) messages.
• POST /namespaces/{ns}/versions with changes and review id; POST /rollouts with stages; POST /rollouts/{id}/rollback.
• GET /namespaces/{ns}/versions/{v} for history and diff.

4. High-level design

1. An author submits a change through review; the API writes a new immutable namespace version to a consensus-replicated store, so the version number is a total order per namespace.
2. Distribution roots in each region stream versions from the store; edge distributors in each cluster subscribe to roots and hold the latest version of every namespace their cluster uses.
3. Each process's client library keeps a watch stream to its edge distributor and holds an in-memory snapshot per namespace; a delta arrives, is validated, and swaps in atomically.
4. Reads are a map lookup against the snapshot with targeting rules evaluated locally; nothing on the request path leaves the process.

5. Deep dives

How do a million processes get a change in 5 seconds without a thundering herd?

A fan-out tree: one store, a few roots per region, hundreds of edge distributors, a million clients. Each level pushes deltas (changed keys only, typically a few hundred bytes) downstream over long-lived streams, so a change is 1,000 messages at the root level and 1,000 messages per edge, all in parallel: under a second per level. Edges cache full snapshots so a newly started process gets its configuration from the edge in one round trip, and the store sees none of the fleet's traffic. If an edge dies, clients reconnect to another edge and resume from their acknowledged versions; the replicated store and roots are the only parts that must be highly available for writes, and reads never touch them.

How is ordering guaranteed, and what does a client do with a gap?

Every delta carries (namespace, from version, to version). A client applies a delta only if from version equals what it holds; otherwise it has missed one and asks the edge for a snapshot at the latest version, or for the missing deltas. Versions never go backwards: a client ignores any message for a version lower than its current one, which makes replays and duplicate messages harmless. The store assigns versions through consensus, so two authors changing the same namespace concurrently produce versions v+1 and v+2 with a well-defined order and the second author sees the first's change in review.

How do staged rollouts work and what halts them?

A rollout publishes a version whose targeting rules say "value X for 1 percent of servers (by a stable hash of the server id) and the old value otherwise". The client library evaluates the rule locally, so the staged state is just another version. The rollout controller advances stages on a timer only if guard metrics (error rate, latency for the affected service) stay within bounds, else it publishes a rollback version automatically. Because every stage is itself an ordinary version, rollback is publishing the previous values as a new version: instant, ordered and audited. Flags that are known to be dangerous (kill switches) can be marked to skip staging in emergencies, with an alert.

What does a process do when the service is unreachable at startup or for an hour?

The client library persists its last snapshot to local disk and starts from it; a brand-new machine ships with a baked snapshot from the image build. A process never blocks on configuration: it serves with what it has and keeps retrying the watch with backoff and jitter so a recovering edge is not stampeded. Defaults declared in code are the last resort for keys missing from the snapshot. The staleness risk is explicit: a flag flipped during an outage will not reach isolated processes, so the console shows the acknowledgement coverage per version (the fraction of subscribed processes at that version), which is also how the rollout controller knows a stage has actually applied before measuring it.

6. What a strong answer sounds like

• Size the configuration (100 MB total) and the fleet (a million readers), and conclude: consensus for writes, a push tree for reads, in-process snapshots for lookups.
• Give the ordering rule with from and to versions, and the gap recovery.
• Explain staged rollouts as versions with targeting rules, auto-halt on guard metrics, and rollback as a new version.
• Describe startup and outage behaviour: disk cache, baked snapshot, never block, show coverage.
• Follow-ups to expect: per-user flags evaluated at request time (hash the user id against the percentage locally, no lookup), large values (store in object storage and distribute a pointer plus hash), and cross-region consistency when a region is partitioned (the partitioned region keeps serving its last version and catches up in order).

02Where this question was reported

• resumegeni.com · 'Design Spanner' (TrueTime, external consistency) as canonical problem.
• gitgood.dev · L6 deep-dive example: consistency layer for a multi-region database.
• onsites.fyi · L6: globally consistent configuration management system (like Google FlagService).
• interviewkickstart.com · 'Design of a globally distributed database'; data consistency in globally distributed system.