sydepsystem design atlas

Distributed Job Scheduler

Queue jobs, dispatch them across workers, and persist execution state.

AdvancedPlatformData
SchedulerComputeJob QueueMessagingWorker PoolComputeJob State DBStorage

A scheduler enqueues recurring or ad-hoc jobs, workers pull them from a durable queue, and the scheduler persists status updates so jobs can be retried or resumed if a worker crashes.

When to use it

  • Work needs to be executed on a schedule or in a distributed worker pool, with retries and visibility into state
  • You need jobs to outlive a single process and recover after failure

Trade-offs

  • A scheduler becomes an operational dependency and a source of hidden backlogs if misconfigured
  • Long-running jobs need durable checkpoints and a clear replays/retries strategy

Components used

Workflow OrchestratorMessage QueueContainer ServiceRelational Database

How it works

  • Jobs are persisted with their schedule and state, so a scheduler restart does not lose pending work.
  • A dispatcher claims due jobs and hands them to a worker pool, marking each claimed to prevent two workers running it simultaneously.
  • Execution state, attempts and results are written back, making the system inspectable and retryable.

Used in the wild

  • Cron replacement that survives the loss of any single machine.
  • Delayed and scheduled user-facing actions such as reminders or publishing.
  • Orchestrating recurring data pipeline runs with dependency ordering.

Good to know

  • Exactly-once execution is not achievable in general. A worker can complete a job and die before recording it, so jobs should be idempotent and the system should target at-least-once.
  • Scheduling everything on the hour creates a thundering herd where thousands of jobs fire simultaneously. Jittering start times spreads the load without changing the schedule meaningfully.