Video transcoding
Turning one upload into many renditions, in parallel, by segment.
01The intuition
The problem it solves
Ravi records a two-minute video on his phone: 4K, 60 frames a second, about 1.2 GB. He uploads it. Asha wants to watch it on a bus with a weak signal that carries about 1 megabit per second. Sending her the original would take
1.2 GB × 8 bits per byte ÷ 1 Mbps ≈ 9,600 s, more than two and a half hours
for a two-minute clip. Her phone also cannot decode 4K at 60 fps smoothly, and her screen cannot show it anyway. The upload is the wrong shape for almost every viewer. One file must become several: a small one for weak networks, a medium one for phones, a large one for televisions, in formats every device can decode.
Build it one piece at a time
Step 1: store the original and acknowledge. The upload goes straight into object storage and Ravi's app is told "received". Nothing else happens on his request. Converting video is slow, minutes of CPU per minute of footage, so it must run as a background job.
Step 2: one job per rendition. A worker reads the original and produces one output: 360p at 0.8 Mbps. Another produces 720p at 2.5 Mbps, another 1080p at 5 Mbps. These jobs are independent, so they run on different machines at the same time. After this step the catalogue looks like this:
| Rendition | Resolution | Bitrate | Size for 2 min | Status |
|---|---|---|---|---|
| original | 3840×2160 | 80 Mbps | 1.2 GB | stored |
| 360p | 640×360 | 0.8 Mbps | 12 MB | done |
| 720p | 1280×720 | 2.5 Mbps | 38 MB | running |
| 1080p | 1920×1080 | 5 Mbps | 75 MB | queued |
Step 3: split by time, not only by rendition. A one-hour film at 1080p takes one machine an hour or more. Cut the original into segments of a few seconds at keyframes, transcode each segment on a different worker, and stitch the results. Fifty workers finish a one-hour film in a couple of minutes. The segments also become the units that adaptive streaming serves.
Step 4: a pipeline with a state machine. Each video moves through states: uploaded, probed (what codec, duration, resolution), segmented, transcoded per rendition, packaged (manifests written), published. Store the state per video and per segment in a database so a crashed worker's piece is retried, not the whole film.
Step 5: make the lowest rendition first. Publish 360p as soon as it finishes so the video is watchable in a minute, then add the higher ones as they land. Thumbnails, audio-only tracks and subtitles are more jobs in the same pipeline.
Why it works, with the numbers
Transcoding is embarrassingly parallel because video is already a sequence of independent groups of pictures starting at keyframes. If a film has S segments and R renditions, the pipeline creates S × R small jobs; with W workers the wall-clock time is about
time ≈ (S × R × tsegment) ÷ W, for a 1 h film: 720 segments × 4 renditions × 20 s ÷ 200 workers ≈ 5 minutes
The guarantee is eventual: the video exists in all renditions some minutes after upload, and every segment is produced exactly once because each job is keyed by (video, segment, rendition) and is safe to rerun. The costs are honest: CPU time is the big bill (hardware encoders help), and storage multiplies, since a video is kept in five or more sizes, though the originals can move to cold storage.
The trade-off you must say out loud
More renditions mean a better fit for every viewer and more storage and encoding cost; a popular video deserves eight ladders, a video with ten views deserves two, which is why some platforms transcode lazily on first demand. The interviewer will ask how a user sees progress and how you avoid redoing a whole film when one worker dies, which is the per-segment state machine and idempotent jobs.
Where it shows up in the 100 Google problems
- Design YouTube: the upload lands in object storage, a segment-parallel pipeline produces the ladder, and 360p publishes first.
- Design TikTok: short clips let the whole pipeline finish in seconds, which is what makes "live in seconds" possible.
- Image and GIF hosting: the same pattern for images and GIFs, producing thumbnails and web-friendly formats.
- Design Google Meet: transcoding happens live, per participant, which is why a forwarding server is preferred over re-encoding.