Design a log collection and aggregation pipeline

01The design, in the six-step framework

1. Requirements

Functional.

• Every machine and service emits logs (structured JSON lines and free text). Collect all of them centrally.
• Engineers search by service, time range, severity, request id and free text, and see results within a minute of the log being written.
• Logs are retained for 30 days hot, one year cold; some are deleted earlier for privacy.
• Different log types have different priorities: audit logs must never be dropped; debug logs may be sampled under load.
• Out of scope: metrics and traces (separate systems), alerting rules.

Non-functional.

• Ingest 2 TB per hour steady, 10 TB per hour at peak, from 100,000 hosts.
• Freshness: a line is searchable within 60 seconds at p99.
• Search latency: under 2 seconds for a filtered query over the last hour; under 30 seconds over a week.
• Durability: audit and error logs lost with probability under 10^-9 per line; debug logs best-effort under overload.

2. Core entities

• Log line: timestamp, host, service, severity, body, structured fields (request_id, user_id hash, region), ingest_time.
• Stream: service + log type, with a priority class (audit, error, info, debug) and retention policy.
• Segment: an immutable block of lines for one stream over a time window, with its compressed blob and its index.
• Query: filters, time range, text terms, requester, cost budget.
• Retention rule: stream pattern, hot days, cold days, deletion triggers.

3. API

• Agent → ingest: POST /ingest batch of lines, gzip, with stream id and a batch sequence number; returns the highest sequence acknowledged. Called by the on-host agent every second or 1 MB.
• POST /search {streams, from, to, filters, text, limit, cursor} → {lines, cursor, scanned_bytes}. Called by the UI and by CLI tools.
• GET /streams/{id}/tail → server-sent events of fresh lines matching a filter, for live debugging.
• PUT /streams/{id}/retention {hot_days, cold_days} → policy; admin only.

4. High-level design

A log line's journey:

1. The agent on the host tails files and stdout, batches lines, compresses them and posts them with a sequence number; it keeps a local disk buffer so a slow network does not lose lines.
2. The gateway authenticates the host, tags the batch with stream and priority, and appends it to the partitioned log, partitioned by stream so one stream's lines stay ordered.
3. Indexers consume partitions, parse structured fields, drop or hash personal fields per policy, and build segments: a compressed block of a few minutes of lines plus an inverted index over fields and terms.
4. Segments are written to the hot store, sharded by stream and time; after 30 days they move to object storage as-is, index included.
5. A query names streams and a time range, which selects the segments; the query service fans the work out to the shards holding them, merges, and returns a page.

5. Deep dives

How do we process ten million lines a second and stay under a minute?

The pipeline is a sequence of buffers, and the freshness target is the sum of the time spent in each. The agent batches for at most one second. The partitioned log adds a few hundred milliseconds. Indexers work in micro-batches of about 10 seconds per segment; a line becomes searchable when its segment is sealed and registered, so segment size sets the floor on freshness, roughly 15 to 20 seconds in practice. Throughput comes from partitioning: 10 million lines per second over 2,000 partitions is 5,000 lines per second per indexer, well within one core's parsing budget. The interviewer who wrote "processed in under a minute" wants the processing layer named: parsing and enrichment are CPU-bound and stream-shaped, so they scale by adding consumers to the partition group, and the index build is the expensive step that justifies micro-batching over per-line handling.

What happens when ingest spikes and the indexers cannot keep up?

The log absorbs it; that is what the 24-hour buffer is for. Freshness degrades (lag grows) while throughput is held, which is the correct trade for logs: a line that arrives late is far better than one that is lost. Backpressure runs the other way too: if the log itself is full, the gateway rejects debug-priority batches first with a retry-after, agents sample debug lines locally (keep 1 in 10, record the sampling rate), and audit and error lines are always accepted. Priority is attached at the stream level and enforced at every queue: separate partitions, separate consumer capacity, and a guarantee that audit streams never share a disk with debug streams. This is also the answer to "prioritised log manager with eviction" from the reports: evict and sample low-priority data, never high-priority.

How does search over a week stay under 30 seconds?

By not reading most of the data. Segments are keyed by stream and time, so a query touches only the segments in its range; a query with no time range is refused. Within a segment, the inverted index narrows to matching lines before any decompression. Columnar storage of structured fields lets a filter on severity or request_id skip whole blocks via min/max metadata. The query service runs scatter-gather across the shards holding the selected segments, streams results back as they arrive, and enforces a byte budget per query so one careless search over a month cannot occupy the cluster; past the budget it returns partial results and says so.

How are privacy deletions handled in an append-only store?

Two mechanisms. At ingest, fields known to be personal (user ids, emails, IP addresses) are hashed or dropped by stream policy, so most logs never contain raw identifiers. For the remainder, deletion requests are satisfied by rewriting segments: a deletion job finds segments containing the key (via the index), writes new segments without those lines, and swaps the pointers; the old segments are destroyed after a grace period. Rewrites are batched daily, which is acceptable for the usual 30-day deletion deadline, and the design should state that deletion is eventual, not instant, and what the bound is.

6. What a strong answer sounds like

• State the ingest rate in lines per second and bytes per second, then size the buffer, hot store and cold store from it.
• Walk the freshness budget stage by stage and show where the 60 seconds goes; segment size is the knob.
• Say that logs are append-only and partitioned by stream, and that search works because every query is bounded by time.
• Make priority explicit: audit never drops, debug samples first; explain how the queue enforces it.
• Expected follow-ups: multi-tenant isolation (per-team quotas and query budgets); why a general-purpose search engine is expensive here (index everything versus index fields plus text); cost per GB and how sampling and retention control it.
• Common mistake: proposing a database with full-text search as the whole answer and skipping the buffering and prioritisation that keep it alive at peak.

02Where this question was reported

• Exponent · Detailed report: collect several log/metric types, process in under a minute; follow-ups on the processing layer
• DesignGurus · 'Log Collection System'
• System Design Handbook · 'Metrics or Logging System'
• interviewkickstart.com · 'distributed logging system, such as the Google logging infrastructure'; 'log search system'
• PracHub · 'distributed log storage service', Hard
• PracHub · 'log management with auto-deletion'; 'prioritized log manager with eviction'
• scaleengineer.com · 'store and retrieve user activity logs for a global service'
• PracHub · PM
• LeetCode Discuss · Creating a log aggregator system; recruiter later said design skills not satisfactory (search snippet)
• Taro · Design a distributed logging system; probed concurrency, load balancing, fault tolerance
• Glassdoor · Design a Logging system with high availability (London SRE loop; review ID from search snippet, page not fetchable)
• Blind · Design logger (OP list of Google questions)