Observability: metrics, logs and traces

The three signals that explain why a request was slow.

System design conceptReliability and operations1 Nov 2026

01The intuition

The problem it solves

A user reports that checkout was slow this morning. The checkout request passed through a load balancer, an API server, a cart service, a pricing service, a payments call to a bank and two databases. Somewhere in that chain, 2 seconds went missing. Without instrumentation the only evidence is the user's memory. With a single log line per request on one machine you know it took 2.3 seconds, but not where.

Three questions need three different kinds of evidence. "Is the service healthy right now?" needs a number over many requests. "What happened to this one request?" needs the story of that request. "Where inside the chain did the time go?" needs the request's path across machines, with timing on each hop. These are metrics, logs and traces.

Build it one piece at a time

Step 1: metrics, the numbers. Each server counts: requests served, errors, and a histogram of latencies. Every 10 seconds it reports the counters to a time-series store. Counters are tiny no matter how much traffic flows, so a million requests a minute still produce a handful of numbers.

Metric	Service	Value, last minute
requests	checkout	12,400
errors	checkout	31 (0.25%)
latency p50 / p99	checkout	180 ms / 2,100 ms
latency p99	pricing	1,900 ms

Already the table points a finger: pricing's p99 matches checkout's. Dashboards and alerts are built on these, and an alert on "p99 above 1 s for 5 minutes" pages someone before the user emails.

Step 2: logs, the story. Each service writes one structured line per request with a timestamp, the user id, the request id, the result and the duration, as key-value pairs rather than prose, so machines can search them. Logs ship through a queue to a store that indexes them. Searching for the user id finds the slow request and its error text.

Step 3: a request id that travels. The load balancer generates a random id for each incoming request and puts it in a header. Every service passes it on to every call it makes and writes it into every log line. Now one search for that id returns the lines from all seven services, in order.

Step 4: traces, the path with timing. Go one step further: each service records a span, which is the request id plus "I started at t₁, finished at t₂, my parent span was X". Collected together, the spans form a tree that draws as a waterfall: checkout 2,300 ms, inside it pricing 1,900 ms, inside that a database call of 1,850 ms. The missing 2 seconds has an address.

Step 5: keep the bill sane. Metrics are cheap and kept for a year. Logs are big; keep them for weeks. Traces are the biggest, so sample: record every span for 1 percent of requests, and always for requests that erred or were slow, decided at the first hop so the whole trace is kept or dropped together.

Why it works, with the numbers

The three signals cost very differently per request. A metric update is a counter increment, a few nanoseconds and no bytes per request. A log line is about 500 bytes. A trace with 20 spans is about 10 KB. At 10,000 requests a second:

logs ≈ 10,000 × 0.5 KB = 5 MB/s ≈ 430 GB/day; traces at 1% ≈ 100 × 10 KB = 1 MB/s ≈ 86 GB/day

which is why metrics can be complete, logs need retention limits, and traces need sampling. The guarantee is that the three agree, because they share the request id and the same clocks (synchronised to within milliseconds). The honest failure modes: a metric is an average that hides the one slow user, so always look at p99 and not the mean; logs can be lost when the queue backs up, so they must never be the system of record; and sampling can miss the one trace you wanted, which is why errors and slow requests are always kept.

The trade-off you must say out loud

Detail costs money: the more you record, the more you store and the more every request pays in overhead. Say which signal answers which question and how long each is kept. The interviewer's follow-up is how you alert without waking people for noise: alert on symptoms users feel (error rate, p99) rather than causes (CPU), and require the condition to hold for minutes.

Where it shows up in the 100 Google problems

Distributed tracing for a microservice fleet: the span tree, propagation headers, head-based sampling and the storage that answers "show me this trace".
Metrics and monitoring system: the time-series side, with alerting and dashboards over a year of data.
High-scale API gateway: the first hop that mints the request id and records the first span.
Global code deployment pipeline and NoSQL to SQL migration: the error-rate and latency metrics that decide whether a rollout continues.
ML model monitoring: the same pipeline carrying prediction quality instead of latency.