Distributed Job Scheduler
Queue jobs, dispatch them across workers, and persist execution state.
AdvancedPlatformData
A scheduler enqueues recurring or ad-hoc jobs, workers pull them from a durable queue, and the scheduler persists status updates so jobs can be retried or resumed if a worker crashes.
When to use it
- Work needs to be executed on a schedule or in a distributed worker pool, with retries and visibility into state
- You need jobs to outlive a single process and recover after failure
Trade-offs
- A scheduler becomes an operational dependency and a source of hidden backlogs if misconfigured
- Long-running jobs need durable checkpoints and a clear replays/retries strategy
Components used
Workflow OrchestratorMessage QueueContainer ServiceRelational Database
How it works
- Jobs are persisted with their schedule and state, so a scheduler restart does not lose pending work.
- A dispatcher claims due jobs and hands them to a worker pool, marking each claimed to prevent two workers running it simultaneously.
- Execution state, attempts and results are written back, making the system inspectable and retryable.
Used in the wild
- Cron replacement that survives the loss of any single machine.
- Delayed and scheduled user-facing actions such as reminders or publishing.
- Orchestrating recurring data pipeline runs with dependency ordering.
Good to know
- Exactly-once execution is not achievable in general. A worker can complete a job and die before recording it, so jobs should be idempotent and the system should target at-least-once.
- Scheduling everything on the hour creates a thundering herd where thousands of jobs fire simultaneously. Jittering start times spreads the load without changing the schedule meaningfully.
Related system design
Ad Click Aggregation & Analytics
Ingest high-volume click events and aggregate them for billing and reporting.
URL Shortener
Create short aliases, then redirect quickly from a cache-backed read path.
Pastebin / Snippet Hosting
Store snippets as text or markdown, retrieve them by id, and optionally render preview.
Web Crawler / Search Indexer
Discover pages, crawl them, extract content, and index for search.