dispatch¶

Submit and supervise multi-node training jobs on the scheduler you already have, behind one contract.

A job's outcome is a coarse state plus a typed cause (preempted, node lost, out of memory, timeout, no capacity...) that is never collapsed into "failed". Capability support is reported per cluster, not per adapter. Recovery policy is data: the plan names who will enforce each rule before the job starts, and a separate supervisor carries it out, through conditional commits to the job's record and launches that start gated.

dispatch is in development and has no release yet. This module carries the contract, the supervisor, the conformance suites, an in-memory adapter, an in-memory record store and a durable one over go/ledger, and the checkpoint package with cmd/publish, the helper a job script runs to publish checkpoints. Its scheduler adapters live in their own modules and are documented on this site: the Slurm adapter (go/dispatch-slurm) and the Kubeflow adapter (go/dispatch-kubeflow).

Start here¶

Submit and supervise a job takes you from nothing to a job that's preempted, recovered and finished, in one Go program with no cluster.

Do a task¶

• Read a plan before submitting
• Choose an unmatched policy
• Receive and answer escalations
• Watch a job's state changes
• Read a job's output
• Publish and resolve checkpoints from a job script
• Write an adapter and pass the conformance suite
• Run the simulator and replay a seed

Understand it¶

• Why classification never collapses to "failed"
• Safety from conditional commits and gated launches
• The event-sourced record
• Isolated checkpoints and append-only publication
• Preflight, read-only and observe-only
• How the simulator tested the supervisor

Look something up¶

• Errors and sentinels
• Capabilities and support
• The adapters compared
• Status, receipts and filters
• The steps Next returns
• Escalations
• The record's schemas
• Record stores
• The execution environment
• Timings and defaults
• Checkpoint publication
• The publish image
• The memory adapter
• The Go API on pkg.go.dev

Run on a scheduler¶

• The Slurm adapter: sbatch over the local shell or SSH, holds as a far-future start time, and a deadline enforced on the cluster
• The Kubeflow adapter: one gated TrainJob per submission, and the dispatch-admission controller for Kueue

The adapters compared puts the two side by side.

The design is recorded as specs in the project wiki.