Design a Distributed Job Scheduler

A distributed job scheduler runs work at the right time: send this email in 2 hours, rebuild the search index every night at 2 AM, retry a failed webhook in 5 minutes. Think of cron for a whole company, plus one-off delayed jobs. It must run every job on time and at least once, even when machines die mid-job.

Requirements#

Functional

• Schedule one-off jobs at a given time and recurring jobs with cron expressions.
• Jobs have a payload, a priority, a timeout and a retry policy.
• Users can pause, cancel and inspect jobs and their run history.

Non-functional

• Jobs start within a few seconds of their scheduled time.
• At-least-once execution; no job is silently skipped.
• Scale to millions of jobs per day and thousands of workers.

Data model#

jobs (
  id            UUID PRIMARY KEY,
  schedule      TEXT,         -- cron expression, or null for one-off
  next_run_at   TIMESTAMP,    -- indexed
  payload       JSONB,
  status        TEXT,         -- scheduled | paused | done
  max_attempts  INT
);

runs (
  job_id, attempt, status, worker_id, started_at, finished_at, error
);

The key index is next_run_at: "which jobs are due now?" must be a cheap query.

Architecture#

• Job API: creates and manages jobs.
• Scheduler: every second, finds due jobs and pushes them onto a ready queue, then computes the next run time for recurring jobs.
• Workers: pull from the queue, run the job, and record the result.

Claiming due jobs without duplicates#

With several scheduler instances, two could grab the same due job. Two common approaches:

• Partition the work: each scheduler owns a range of job IDs (assigned through a coordinator), so they never overlap.
• Lock rows while claiming:

SELECT id FROM jobs
WHERE next_run_at <= now() AND status = 'scheduled'
ORDER BY next_run_at
LIMIT 500
FOR UPDATE SKIP LOCKED;

SKIP LOCKED lets each scheduler take a different batch without blocking the others.

Reliability#

• Visibility timeouts: when a worker takes a job, the queue hides it for the job's timeout. If the worker dies, the job reappears and another worker runs it. That is what makes execution at-least-once.
• Heartbeats: long jobs extend their lease periodically, so they aren't re-run while still healthy.
• Retries with backoff: failed jobs are rescheduled with exponential backoff; after max_attempts they move to a dead letter queue.
• Idempotent jobs: because a job can run twice, job handlers should be safe to repeat (use idempotency keys for side effects like payments or emails).

Scaling#

• Shard the jobs table by job ID, with one scheduler per shard.
• For very large volumes of near-term jobs, keep the next few minutes in a timing wheel or a Redis sorted set scored by run time, and leave far-future jobs in the database.
• Separate queues per priority or team, so a flood of low-priority jobs can't starve critical ones.

Recurring jobs and missed runs#

After each run, compute the next fire time from the cron expression in the job's time zone. Decide on a misfire policy: if the system was down at 2 AM, should the nightly job run once on recovery, or be skipped until tomorrow? Make it a per-job setting.