The 100 problems, solved / Search and indexing / 012 · Design a web crawler

Design a web crawler

Fetch billions of pages politely, dedupe them, and keep the index fresh.

Problem 012 of 100Search and indexingL51 Nov 2026
Concepts this design leans on

01The design, in the six-step framework

1. Requirements

Functional

  • Starting from seed URLs, fetch pages, extract links, and keep going; store page content and metadata for the indexer.
  • Obey robots.txt and per-host politeness; never fetch the same URL twice within a refresh interval; detect duplicate content.
  • Prioritise important and frequently changing pages for re-crawl.
  • Out of scope: rendering JavaScript (a separate renderer fleet), ranking, the index itself (see Design Google Search).

Non-functional

  • Crawl 10 billion pages per month; refresh news sites hourly and most pages monthly.
  • At most one request every few seconds to any single host unless it allows more; distributed across many IPs.
  • Resilient to slow or malicious hosts (crawler traps, infinite calendars), huge pages, and worker crashes.
  • Storage for raw pages for reprocessing: months of history.
Back of the envelope

1010 pages / (30 × 86,400 s) ≈ 3,900 pages/s average, about 8,000/s at peak; at 500 KB average that is 2 to 4 GB/s of download bandwidth.

Fetch workers: a fetch takes about 500 ms of mostly waiting, so one worker with 1,000 concurrent connections does 2,000 pages/s; 10 to 20 fetch workers suffice, with hundreds more for parsing and politeness headroom.

Storage: 1010 × 500 KB = 5 PB per month raw, about 1 PB compressed; keep 6 months → 6 PB. The URL frontier: 1011 known URLs × 100 bytes = 10 TB, far too big for memory on one machine and sharded by host.

2. Core entities

  • URL record: normalised URL, host, last fetched, last changed, priority score, fetch status, content fingerprint.
  • Host record: hostname, robots.txt rules and fetch time, crawl-delay, last request time, IP addresses, health.
  • Page: URL, fetch time, HTTP status, headers, body (compressed), extracted links, content hash.

3. API

  • Internal pipeline, not a public API. Key interfaces: frontier.next(host_shard) → the next URL whose host is allowed to be fetched now; fetcher.fetch(url) → page or error; frontier.add(urls, source) → new or refreshed candidates with priority.
  • GET /pages/{url} → for the indexer: latest stored copy and metadata.
  • POST /seeds, POST /blocklist → operator controls.

4. High-level design

FrontierFetchProcessStorageranked URLshost allowed nowraw pagelinks, hashnew / changedcandidatesPrioritiser(front queues)Politeness scheduler(per-host queues)DNS cacheFetch workers(async HTTP)Parser + linkextractorURL seen test+ content dedupPage store(object storage)URL DB(sharded by host)Fetched-page queue

The life of one URL:

  1. The prioritiser scores candidates (page importance, change frequency, time since last fetch) and feeds them into the politeness scheduler, which keeps one queue per host and a timer saying when that host may next be hit.
  2. A fetch worker takes a URL for a host whose timer has expired, resolves DNS through a cache, fetches with a timeout and size cap, stores the raw page, and enqueues it.
  3. A parser extracts links and text, computes a content fingerprint, and passes links to the URL-seen test, which deduplicates against the URL database and adds the new ones with an initial priority.
  4. The indexer reads the page store on its own schedule.

5. Deep dives

Politeness: how do we crawl 8,000 pages a second and still never hammer one host?

A single global queue of URLs would let a burst of links from one site be fetched together. The frontier is therefore two-tiered: front queues by priority, and a back queue per host, with a heap of (next-allowed-time, host). A worker pops the host whose time has come, fetches one URL, and reinserts the host with the delay from robots.txt (or a default of a few seconds, adapted to the host's response time). Hosts are assigned to frontier shards by hostname hash so every URL of a host lives on one shard and the timer is consistent. With 10 million hosts crawled in parallel at one request per few seconds each, the aggregate is millions per second available, so politeness never limits throughput; the long tail of slow hosts is handled by per-host concurrency of one.

Have we seen this URL? Have we seen this content?

The URL-seen test runs 10 to 100 times per fetched page (every extracted link) at 8,000 pages/s, so about 500,000 checks a second against 1011 URLs. A Bloom filter of 1011 entries at 1 percent false positives needs about 120 GB, sharded across frontier nodes by host; a hit goes on to the exact check in the URL database, a miss is definitely new. The false positives mean a few new URLs are wrongly thought seen, which is tolerable for a crawler. Content dedup uses a similarity hash (SimHash) of the extracted text so near-identical pages (the same article with a different session id in the URL) are stored once and their links deprioritised; exact duplicates use the content hash. URL normalisation before any check (lower-case host, remove default ports and fragments, sort known tracking parameters) removes most trivial duplicates.

Traps, giant pages and misbehaving hosts

Crawler traps are pages that generate infinite links (calendars, faceted search). Defences: cap URL length and depth per host, cap the number of pages fetched per host per day in proportion to its importance, and detect patterns such as a steadily growing query string. Every fetch has a timeout and a byte cap (say 10 MB) so one host cannot hold a connection open or stream forever. A host that times out repeatedly gets an exponential backoff and a lower share of workers. Fetchers are stateless, so a crashed worker loses only the in-flight fetches; the frontier re-issues them after a lease on the URL expires.

Freshness: which pages get re-crawled when?

Re-crawl priority is a product of importance (links in, traffic) and the observed change rate: a page that changed on each of the last five fetches is fetched more often, one that never changes decays to monthly. The signal comes from the content hash: fetch, compare with the last fingerprint, update the change estimate. Conditional requests (If-Modified-Since, ETag) make a refresh of an unchanged page cost a few hundred bytes. Site maps and publisher pings feed the prioritiser directly for news, which is how hourly freshness is affordable on a crawl that otherwise takes a month.

Running the crawl from several data centres

Fetching a host in Tokyo from Virginia adds 150 ms per request and wastes trans-Pacific bandwidth, so the crawl is partitioned geographically: hosts are assigned to the data centre nearest their IP, and each data centre runs its own frontier shards, fetchers and page store. Links that point to a host owned by another data centre are forwarded in batches to that centre's URL-seen test rather than fetched locally. The page store is replicated to wherever the indexer runs. Exchanging URLs between centres is a trickle compared with fetching, which keeps the design simple: the only globally coordinated state is the host-to-data-centre assignment, which changes rarely and is published through a configuration service.

6. What a strong answer sounds like

  • Do the throughput arithmetic (8,000 pages/s, a few GB/s, tens of fetch workers) and then explain why the frontier, not the fetchers, is the hard part.
  • Draw the two-tier frontier and the per-host timer; it is the single structure that gives both prioritisation and politeness.
  • Size the URL-seen Bloom filter and say what a false positive costs.
  • Name the traps and the caps that defuse them.
  • Expect follow-ups on JavaScript rendering (a separate, far more expensive fleet with its own budget), on distributing across data centres (shard by host, DNS locality), and on how the indexer consumes the page store.
  • Common mistake: a single global queue with no per-host state, which either hammers hosts or serialises the crawl.

02Where this question was reported