Job scheduling
Running work on time, once, with dependencies and retries.
01The intuition
The problem it solves
A company needs a report emailed at 6 a.m. every day, a backup every hour, and a cleanup every Sunday. On one computer, cron does this: a text file lists times and commands, and a small program runs them. It works until the computer is off at 6 a.m., or the report takes two hours and the 7 a.m. run starts while the 6 a.m. run is still going, or there are 200,000 such jobs across a company and one machine cannot run them all.
A distributed scheduler has to answer three questions that cron ignores. Who runs a job when there are a thousand machines? How do we make sure it runs once, not twice and not zero times, when machines crash? And how do we run jobs that depend on other jobs, like "build the report after the backup finishes"?
Build it one piece at a time
Step 1: a table of jobs. Store every job in a database: its ID, schedule (0 6 * * *), command, and next_run time. Keep an index on next_run.
Step 2: a poller that finds what is due. Every few seconds, a scheduler process asks for jobs with next_run less than now, puts each on a queue as a run request, and advances next_run to the next occurrence. Worker machines pull run requests from the queue and execute them.
| Time | Jobs due | Queue | next_run updated |
|---|---|---|---|
| 05:59:58 | none | [] | report: 06:00 |
| 06:00:01 | report | [report-2026-11-01] | report: next day 06:00 |
| 06:00:02 | none | [] (worker took it) | unchanged |
Step 3: more than one scheduler, without double runs. If two scheduler processes poll at once, both find the report due and both enqueue it. Fix it with a claim: the poller updates the row to claimed_by = me only if it is still unclaimed, in one atomic statement, and only the winner enqueues. Each run also carries an ID made from the job and its scheduled time, report-2026-11-01T06:00, so a duplicate enqueue is deduplicated by the worker.
Step 4: survive a worker crash. A worker picks a run, marks it running, and sends a heartbeat every 30 seconds. If heartbeats stop, the run is marked failed and retried on another worker, with backoff. The job must be safe to re-run, which means idempotent steps or checkpoints for long jobs.
Step 5: dependencies as a graph. "Report after backup" is an edge in a directed acyclic graph. A job becomes due when its schedule fires and all its upstream jobs have succeeded for that same date. The scheduler stores each run's state per date, so the 6 a.m. report of 1 November waits for the 1 November backup, not for any backup.
Why it works, with the numbers
With 200,000 jobs and an average of 10 runs a day each, the scheduler creates about 23 runs a second, which one database and one queue handle easily; the workers are where the compute lives. The time a job starts late is bounded by the polling interval:
worst-case lateness = poll interval + queue wait ≈ 5 s + a few seconds, well under the minute most jobs tolerate
The invariant is: for each job and scheduled time, at most one run is executing and at least one run eventually completes. The claim gives at-most-once dispatch; the heartbeat and retry give at-least-once completion; the run ID makes a duplicate harmless. Failure modes: a job that overlaps itself when a run outlasts its interval (set a concurrency policy: skip, queue, or allow), a thundering herd when 50,000 jobs are all scheduled at midnight (add jitter), and a dependency cycle that never fires (reject at job creation).
The trade-off you must say out loud
Precision of start time against the cost of polling and claiming. Say your polling interval, your concurrency policy for overlapping runs, and your retry limit. Say that the scheduler is small and the workers are where you add machines. The follow-up is "what if the scheduler database is down", and the answer is replicated storage and a scheduler that catches up on missed runs when it returns, with a rule for whether to run them or skip them.
Where it shows up in the 100 Google problems
- Distributed job scheduler: this design, with the claim and heartbeat as the deep dives.
- GPU job scheduler: the same loop with scarce resources, priorities, preemption and checkpoints for multi-hour runs.
- CI with a million tests: one commit becomes thousands of test-shard runs with a dependency on the build.