Feature stores
The same features at training time and serving time.
01The intuition
The problem it solves
A ranking model for a video feed uses, among other inputs, "how many videos did this user watch in the last 7 days". When the model was trained, that number came from a nightly batch job over the logs: for each user and each day, count the watches in the previous week. At serving time, the number must be available in a millisecond for a user who has just opened the app, so an engineer writes a different piece of code that reads a counter from a cache. The two numbers should be the same. They are not: the batch counts completed watches, the counter counts started ones; the batch runs at midnight UTC, the counter is live.
The model learned the batch definition and is being fed the live one. Its predictions are quietly wrong, and nobody can see it, because both pipelines work. This is training-serving skew, and it is the most common silent failure in production ML.
Build it one piece at a time
Step 1: define each feature once. A feature has a name, an entity it belongs to (user, video), a definition as code, and a freshness. "watches_7d: for user, count of completed watches in the trailing 7 days, updated hourly." The definition is the single source of truth, written once.
| Feature | Entity | Definition | Freshness | Computed by |
|---|---|---|---|---|
| watches_7d | user | completed watches, trailing 7 days | hourly | batch |
| video_ctr_1h | video | clicks ÷ impressions, last hour | 1 minute | stream |
| country | user | account country | on change | lookup |
Step 2: an offline store with history. Batch and stream jobs write every feature value with the time it became valid into a table partitioned by time. Training builds examples by joining each logged impression with the feature values that were valid at that moment, never later ones. That "as of" join is what keeps the model from seeing the future.
Step 3: an online store with only the latest. The same jobs also write the newest value of each feature into a key-value store keyed by entity id. Serving reads one row per user and one per candidate video in under a millisecond. Because both stores are filled by the same job from the same definition, the number the model sees in production is the number it was trained on.
Step 4: streaming features. Some features must be fresh within a minute: a video's click rate in the last hour. A stream job computes it from the event log and writes to both stores, with the same definition applied to the same events.
Step 5: serve by feature set. A model declares which features it needs. At request time one call fetches all of them for the user and the 1,500 candidates, batched, from the online store, and the training pipeline fetches the same set by name. Renaming or redefining a feature creates a new version, so a running model never changes under its feet.
Why it works, with the numbers
The online read is the serving cost: for a feed request with 1,500 candidates and 30 features each, the fetch is 1,500 rows, which a key-value store returns in a few milliseconds when batched, and far less when the per-video features are cached because popular videos appear in many users' candidates. The offline join is the training cost: for a day of 10 billion impressions, an as-of join against hourly feature snapshots is a large but routine batch job.
online: 1,500 candidates × 30 features × 8 B ≈ 360 KB per request, one batched fetch; offline: 1010 rows/day joined as-of, nightly
The guarantee is consistency of definition and time: the value a model sees at serving time is produced by the same code as the value it saw in training, and training never uses a value that was not yet known. The honest failure modes: freshness still differs between the hourly batch value and the live world (a user's watches since the last run are missing, which the model must have been trained with too); an online store outage must degrade to default values rather than fail the request; and feature pipelines are a shared dependency across many models, so a bad definition change breaks all of them at once.
The trade-off you must say out loud
Two stores and a registry are more infrastructure than a cache and a batch job, and they pay off only once several models share features. Say which features need streaming freshness and which can be a day old. The interviewer's follow-up is how you detect skew anyway: log the features used at serving time and compare their distribution with the training set every day.
Where it shows up in the 100 Google problems
- Design a recommendation system: the registry, the as-of join for training and the batched online fetch for the ranker.
- Design an ad-serving system: per-ad click rates and budgets as streaming features with one-minute freshness, read inside a 50 ms budget.
- Monitor an ML model's performance in real time: logged serving features compared with training distributions to catch skew and drift.