Asynchronous workers
Taking slow work off the request path and reporting back later.
01The intuition
The problem it solves
A photo app receives a 12 MB picture from a phone. To show it properly the service must make a 200 KB preview, a 50 KB thumbnail, read the location from the file, detect faces, and find which album it belongs to. All of that takes about four seconds of computer time. If the server does it before answering, the phone waits four seconds per photo, a holiday of 300 photos takes twenty minutes to upload, and the user gives up.
At a restaurant, the waiter takes your order and leaves. The kitchen cooks while you talk. The waiter's job is to accept the order quickly and come back when the food is ready, not to cook. A request handler should behave like the waiter: do the one thing that must happen now, hand the rest to a kitchen of worker processes, and reply.
Build it one piece at a time
Step 1: split the request into "now" and "later". Now: store the original file and write a database row photo 42, status = processing. That takes about 100 ms. Later: everything else. The phone gets a reply after the "now" part, with the photo ID.
Step 2: a job and a worker. After the row is written, the server appends a job {photo: 42, tasks: [preview, thumb, faces]} to a queue. A pool of worker processes on other machines pulls jobs, runs them, and updates the row: status = ready, preview = ..., faces = [...].
| Time | Request server | Queue | Worker | Row for photo 42 |
|---|---|---|---|---|
| 0 ms | stores file, writes row | [] | idle | processing |
| 100 ms | replies to phone | [42] | picks up 42 | processing |
| 4,100 ms | serving other users | [] | done | ready, preview, faces |
Step 3: show the user something meanwhile. The app displays the original or a blurred placeholder, and either polls GET /photos/42 every few seconds or receives a push when status flips to ready. The design must say which, because "the user sees nothing for four seconds" is the whole thing you were avoiding.
Step 4: make workers safe to repeat. A worker may crash after making the preview but before updating the row. The job is redelivered and the preview is made again, which is fine because the result is written under the same name. Every step of a worker should be idempotent, and the job should carry everything it needs so it can run on any machine.
Step 5: size and separate the pools. Face detection is slow and uses a GPU; thumbnails are fast. Put them on separate queues with separate worker pools so a backlog of face jobs does not delay thumbnails. Scale each pool by its own queue length.
Why it works, with the numbers
The user-facing latency drops from the full work to the "now" part. With 1,000 uploads a second and 4 seconds of processing each, the workers need
1,000 uploads/s × 4 s = 4,000 worker-seconds per second, so about 4,000 busy workers, or 500 machines with 8 cores
and that number is independent of how fast the request servers reply, which is the point: the two scale separately. The guarantee is that every accepted job eventually completes, as long as the queue is durable and the workers retry. The failure modes are a growing backlog nobody notices, a poison job that crashes every worker that picks it up (send it to a dead-letter queue after three tries), and a user who waits for a status that never flips because the final update was lost.
The trade-off you must say out loud
You trade a simple "request, work, reply" for eventual completion and a status to track. Say what the user sees while the work is pending, how they learn it finished, and what happens if it never does. Say which steps are idempotent. The follow-up is "how long can the backlog get", and the answer is a queue-length alert and autoscaling of the worker pool.
Where it shows up in the 100 Google problems
- Google Photos: thumbnails, face grouping and content indexing run after the upload reply.
- YouTube: transcoding is the archetypal long job; the video appears first in low quality and improves.
- Street View ingestion: stitching and blurring faces happen in worker pools fed by the upload queue.
- Comments service: spam scoring and notifications happen after the comment is stored and shown.