Design a scheduler for long-running GPU jobs

01The design, in the six-step framework

1. Requirements

The 2026 framing is "design Sora's job scheduler": opaque jobs (video generation, model training) that run for hours on scarce GPUs, submitted by many teams with different priorities. The interviewer wants placement, fairness and failure handling for expensive, long work.

Functional

• Submit a job with resource needs (number and type of GPUs, memory, whether the GPUs must be on one machine or can be across machines), a priority, a deadline hint and a checkpoint interval.
• Queue jobs when GPUs are busy; start them when resources free up; report position and expected start.
• Preempt low-priority jobs for high-priority ones, with checkpointing so preempted work resumes rather than restarts.
• Fair sharing across teams with quotas and borrowing of idle capacity.
• Out of scope: the training framework, data loading, billing.

Non-functional

• Scale: 50,000 GPUs in 10 clusters, 20,000 jobs a day, job length from 10 minutes to 3 days, 2,000 jobs queued at peak.
• Utilisation: above 85 percent of GPU hours used; GPUs idle for under 2 minutes between jobs.
• Latency: scheduling decision within 10 seconds of resources freeing; a preempted job resumes within 5 minutes.
• Correctness: a GPU is never double-booked; a job is never silently lost; checkpoints are never corrupted by a concurrent writer.

2. Core entities

• Job: job id, team, priority class, GPU spec (count, type, topology constraint), memory, max runtime, checkpoint interval, state, submit time.
• Allocation: job id, the set of (machine, GPU index) pairs, lease, start time.
• Machine: machine id, cluster, GPU inventory with health, free GPUs, network topology position.
• Checkpoint: job id, sequence number, storage path, size, written at, complete flag.
• Quota: team, guaranteed GPUs, maximum GPUs, current usage.

3. API

• POST /jobs with spec and priority; returns job id, queue position and an estimated start.
• GET /jobs/{id} for state, allocation and checkpoint list; POST /jobs/{id}/cancel.
• Agent-facing: POST /machines/{id}/heartbeat with GPU health and running jobs; POST /allocations/{id}/preempt sent to the agent with a grace period; POST /jobs/{id}/checkpoints to record a completed checkpoint.
• PUT /quotas/{team} for administrators.

4. High-level design

1. A job is stored with its spec and enters the cluster's queue; the API answers with a position computed from the policy engine's current ordering.
2. One scheduler leader per cluster (elected via a lease) runs a scheduling cycle every few seconds: it takes the queue ordered by priority and fair share, and the fleet's free GPUs from heartbeats, and finds placements.
3. Placements are written as allocations with a version, then sent to machine agents, which start the job's container with the GPUs pinned.
4. Jobs checkpoint to object storage on their interval; agents report completed checkpoints, which the scheduler uses when it must preempt.

5. Deep dives

How does placement respect topology and avoid fragmentation?

An 8-GPU training job wants all 8 GPUs on one machine with fast interconnect; a 64-GPU job wants 8 machines on the same network block. Treat the fleet as a tree (cluster → block → rack → machine → GPU) and place to minimise the spread at each level. The failure mode is fragmentation: many machines with 2 free GPUs each cannot host an 8-GPU job. Counter it with gang scheduling (a multi-GPU job starts only when all its GPUs can be reserved at once) and with a bin-packing bias that fills partially used machines before touching empty ones, so empty machines remain for big jobs. Keep a small reserved pool of whole machines for the largest class. The decision rate is tiny, so the scheduler can afford a real optimisation pass each cycle (capacity planning in miniature).

How do priorities, fairness and preemption interact?

Each team has a guaranteed share and a maximum. Order the queue by priority class first, then by how far below its guarantee a team is (dominant resource fairness over GPUs), then by submit time. Idle capacity is borrowed by anyone; when a team under its guarantee submits a job and the cluster is full, the scheduler preempts the most recently started borrowed job of the lowest priority class whose freed GPUs satisfy the request, after a grace period that lets it checkpoint. Preemption is a last resort and is rate-limited per job (a job preempted twice is protected for an hour) so nothing thrashes. State the policy in this order; interviewers want to hear who gets displaced and why.

How do checkpoints make preemption and failure cheap, and how are they kept safe?

A job writes a checkpoint every interval to object storage under a path with a sequence number, then reports it; only reported, complete checkpoints are used for resume. Writing to a temporary name and renaming on completion prevents a half-written checkpoint from being read. On preemption or machine failure the job is re-queued with its last checkpoint; a resumed job loses at most one interval of work. The interval is a trade: 30 minutes on a 3-day job loses 2 percent on a failure and costs 220 GB/s of fleet-wide bandwidth at peak; stagger intervals across jobs so the storage tier sees a smooth stream instead of a burst. Old checkpoints are deleted after the next two succeed.

What happens when the scheduler or a machine dies?

The scheduler leader is stateless beyond its in-memory snapshot; it rebuilds from the state store on takeover, and allocations carry versions so a stale leader cannot overwrite a newer one. Running jobs are unaffected by scheduler loss, because agents keep running what they have. A machine that stops heartbeating for 2 minutes has its GPUs marked unavailable and its jobs re-queued from their checkpoints; if it returns, the agent is told to kill the orphaned containers before taking new work, which prevents a job running twice. GPU-level health (ECC errors, thermal throttling) is reported in heartbeats and taken out of the inventory per GPU, not per machine (failure detection). Backpressure is a queue cap per team so a runaway pipeline cannot submit a million jobs.

6. What a strong answer sounds like

• Show that the decision rate is tiny and the cost of a bad decision is huge, so the scheduler can be a careful single leader per cluster.
• Explain gang scheduling, topology-aware placement and the fragmentation trap.
• Give the preemption policy precisely, with grace periods and rate limits.
• Treat checkpoints as the mechanism that makes preemption and failure affordable, with the bandwidth arithmetic.
• Follow-ups to expect: estimated start times (simulate the queue forward with expected durations), multi-cluster placement (pick the cluster with the earliest predicted start, move data accordingly), and inference versus training (short latency-sensitive jobs get a separate partition so a training backlog never delays them).

02Where this question was reported

• DesignGurus · Listed.
• Prepfully · Candidate reports: cron jobs across machines with variable CPU/RAM.
• PracHub · 'Task scheduler with dependencies' (Hard).
• PracHub · Jun 2026 Senior+: task scheduler for opaque long-running GPU jobs ('Design Sora').
• Blind · Blind list: 'job scheduler'.