The 100 problems, solved / Storage and files / 003 · Design a blob / object storage service (Google Cloud Storage)

Design a blob / object storage service (Google Cloud Storage)

Store and serve billions of immutable objects with eleven nines of durability.

Problem 003 of 100Storage and filesL61 Nov 2026
Concepts this design leans on

01The design, in the six-step framework

1. Requirements

Functional

  • Put, get and delete objects by (bucket, key); objects are immutable blobs from 1 byte to 5 TB.
  • List keys in a bucket by prefix, in lexicographic order, paginated.
  • Per-object metadata (content type, custom headers), versioning per bucket, and access control per bucket and object.
  • Out of scope: in-place edits, a file-system interface, cross-object transactions.

Non-functional

  • Durability 99.999999999 percent (eleven nines) a year; availability 99.99 percent for reads and writes.
  • Exabyte scale: 1012 objects, 100 EB total, millions of requests a second.
  • Strong read-after-write consistency for a single key: a successful put is visible to the next get.
  • First-byte latency under 50 ms for small objects in region; large objects limited by bandwidth.
Back of the envelope

1012 objects with 200 bytes of metadata each is 200 TB of metadata: far too much for one machine, fine for a sharded store.

Durability: with three full copies, data is lost only if all three disks holding an object die before re-replication. With a 3 percent annual disk failure rate and a 1-hour repair window:

p(one disk fails in 1 h) ≈ 0.03 / 8,760 ≈ 3.4 × 10-6; three independent failures in the same hour ≈ (3.4 × 10-6)3 ≈ 4 × 10-17

Eleven nines is 10-11, so three replicas with fast repair clears it with room to spare; erasure coding at 1.5× overhead clears it too, which is why it wins on cost.

Bandwidth: 100 EB read twice a year is 200 EB / 3.15 × 107 s ≈ 6.3 TB/s of egress across all regions.

2. Core entities

  • Bucket: name (globally unique), owner, region, versioning flag, ACL.
  • Object: bucket, key, version id, size, content hash, user metadata, creation time, the storage id of its bytes.
  • Blob: storage id, list of (stripe or replica) locations, checksum, erasure-coding group.
  • Node: data server id, disks, free space, health.

3. API

  • PUT /{bucket}/{key} → body is the object; headers carry content type and metadata; returns the version id and ETag (content hash).
  • POST /{bucket}/{key}?uploads then PUT ...?partNumber=n then POST ...?uploadId= → multipart upload for objects over 64 MB.
  • GET /{bucket}/{key} with optional Range header → bytes; supports conditional gets by ETag.
  • DELETE /{bucket}/{key} → with versioning on, writes a delete marker.
  • GET /{bucket}?prefix=&startAfter=&maxKeys= → ordered listing with a cursor.

4. High-level design

ClientsEdgeServicesDatametadata opsbyteswrite stripescommitre-replicateheartbeatSDK / HTTP clientLoad balancerAPI gateway(auth, routing)Metadata serviceData service(stream, stripe, checksum)Placement + repairMetadata store(sharded by key range)Data nodes(disks)

A 100 MB put:

  1. The gateway authenticates the request and routes it to a data service node, which streams the body, computes a checksum on the fly and cuts it into 8 MB blocks.
  2. Each block is encoded into stripes (for example 6 data + 3 parity) and written to 9 data nodes chosen by the placement service across racks and power domains. The write is acknowledged when every stripe is durable.
  3. The data service then asks the metadata service to commit the object: one write on the metadata shard that owns the (bucket, key) range. Only after this commit is the object visible, which is what makes a get after a put always succeed.
  4. A get reverses it: metadata lookup, then the data service reads 6 of the 9 stripes per block (any 6) and streams the bytes.

5. Deep dives

Three replicas or erasure coding?

Three copies cost 3× raw storage and make reads trivial. Erasure coding at 6+3 costs 1.5× and survives any three stripe losses, which is more durable than three replicas surviving two, at half the disk bill: on 100 EB that is 150 EB of disk instead of 300 EB. The price is CPU for encoding and the need to read six stripes to rebuild one block, which raises small-object latency. The usual answer is both: replicate small and hot objects (under about 1 MB) for latency, erasure-code everything large and everything cold. Repair bandwidth matters too: rebuilding one lost stripe reads six others, so repair traffic is six times the lost bytes.

How does a listing by prefix stay fast on a trillion keys?

If keys were hashed to shards, a prefix listing would need to ask every shard and merge (scatter-gather), with p99 set by the slowest shard. Instead the metadata store is range-partitioned by (bucket, key): keys that share a prefix sit on the same shard or on adjacent shards, so a listing is a single ordered scan with a cursor. The cost is hot spots: a bucket with keys like 2026-11-01/... writes all day to one range. The fixes are splitting hot ranges automatically (a range that exceeds a size or QPS threshold divides in two and one half moves) and advising customers to put high-entropy prefixes first; the real services document exactly that advice.

What does "strong consistency" cost here, and how is it done?

The guarantee is per key: after a put returns, every get of that key sees the new version. It falls out of the write order above: bytes first, then one atomic metadata commit, and gets always consult metadata. The metadata shard itself is a replicated group that reaches agreement through consensus before acknowledging, so a shard leader crash cannot roll back a committed put. Cross-region buckets are the hard case: either writes go to one home region (strong, but far for some writers) or each region accepts writes and the service reconciles later (fast, but a get may briefly see the old version in another region). Say which one you pick and why.

How do you keep eleven nines when disks lie?

Durability is not only about copies. Disks return wrong bytes silently, so every block carries a checksum verified on read, and a background scrubber reads every stripe every few weeks and repairs mismatches. Heartbeats and disk SMART data feed the placement service, which keeps a priority queue of under-replicated blocks and limits repair bandwidth so it does not starve customer traffic. Placement spreads stripes across racks, power feeds and ideally buildings so that a correlated failure removes at most the number of stripes the code tolerates.

How does a 5 TB upload work without a 5 TB request?

Multipart upload: the client starts an upload, sends parts of 8 MB to 5 GB in any order and in parallel, each acknowledged with an ETag, then completes the upload with the list of part ETags. Each part is stored as its own blob, and completion writes one metadata record that stitches them in order, so a failed part is retried alone and an abandoned upload is garbage collected after a day. The same mechanism gives resumable uploads from a laptop and lets a 5 TB object be written by 100 machines at once.

6. What a strong answer sounds like

  • Derive eleven nines from a failure rate and a repair window; the number is the design, not a slogan.
  • Draw two planes: a range-sharded metadata store with consensus, and a flat pool of data nodes with placement and repair as a separate service.
  • Explain the write order that gives read-after-write for free.
  • Pick erasure coding for cold and large data and replication for small and hot, and say what each costs in disk and in read latency.
  • Expect follow-ups on multipart uploads, lifecycle rules (move to colder storage after 90 days), and a region outage.
  • Common mistake: hashing keys across shards and then being unable to list by prefix.

02Where this question was reported