Skip to content

Isolated checkpoints and append-only publication

When a scheduler says a job has ended, its processes haven't necessarily stopped. A trainer that traps SIGTERM to save a checkpoint keeps running for the scheduler's kill wait. A node cut off by the network keeps training until it rejoins. In the Slurm spike behind the supervisor's design (spec 0004 D9), old processes outlived their job every time it was measured: by about 24 seconds after a cancel, 29 after a preemption, and around 140 after a network partition.

So when dispatch starts a replacement, the old run may still be writing. If both write into the same checkpoint folder, two things go wrong. Frameworks like Hugging Face Trainer resume from the highest-numbered checkpoint-<step>, so the replacement can resume from the old run's later save, which is a different lineage. And their rotation deletes old folders, so the old run can delete the replacement's saves.

dispatch doesn't try to prove the old run has stopped. It makes the overlap harmless.

Each execution writes into its own folder

Every execution has an opaque token, DISPATCH_ATTEMPT, different for every execution. It writes into <Checkpoint>/exec/<DISPATCH_ATTEMPT>/, which nothing else writes into. An old run's saves land in its own folder.

That's what lets dispatch start a replacement as soon as the scheduler reports the old object ended, without waiting for proof that its processes stopped.

Publication is append-only

A save isn't visible to the next execution until it's published, and a publication is an empty file in <Checkpoint>/pub/ whose name says everything:

<epoch, 20 digits>-<step, 20 digits>-<the save's path under the root, base32>
  • It's created with O_CREAT|O_EXCL, so two writers can't both create it, and then pub/ and the root are synced before Publish returns. A returned publication is durable.
  • It's never renamed over, rewritten or appended to. A crash leaves a whole publication or none, and there's no content to be half-written.
  • The path is base32 (A-Z, 2-7, no padding): one case, so a case-folding filesystem can't merge two names, and no -, so the name splits one way only.

The latest checkpoint is the entry with the greatest epoch, then step, found by listing pub/. There's no lock and no check-then-rename, which matters because locks are unreliable on Lustre, NFS and GPFS. In the design review recorded in spec 0001 D16, an earlier design that checked, then renamed a pointer into place, lost to a stalled old run in 4 of 40 overlapping trials; this one in none of 40.

A name in pub/ that doesn't parse is an error naming it. It's never skipped, because a skipped entry could be the latest one.

The epoch orders executions

DISPATCH_EPOCH is a number that only increases across a job's executions:

DISPATCH_EPOCH = G × 10^12 + N × 10^6 + R
  • G is the job's generation: 0 for a new job, one more than the job it supersedes for a job created by Supersede;
  • N is the submission number, 1 to 999,999;
  • R is the backend's restart count for the object, 0 to 999,999.

A resubmission raises N, a requeue raises R, and a supersession raises G, so each raises the epoch. model.Epoch computes it and refuses values outside those bounds.

An old run's late publication carries its own, lower epoch, so it never outranks the replacement's saves, however high its step. And a superseding job that reuses the same checkpoint root outranks every save of the job it superseded, while still being able to resume from them.

When a job can't isolate

Some jobs share a mutable checkpoint and can't write per execution. A job declares that with the dispatch.SharedCheckpoint{} extension. dispatch then can't make overlap harmless, and the scheduler's word that a job ended doesn't prove its processes stopped. So every recovery rule becomes a hold for a person, for 24 hours (dispatch.SharedHoldFor), and the plan shows the substitution before the job starts. The person decides when it's safe to run again.

A site can give that person's answer itself. The adapter interface has an optional CessationSignal, a site's own answer on whether an object's processes are gone. While such a hold stands, the supervisor asks it about the ended object every dispatch.CessationInterval, a minute, and records each answer in the record as a cessation fact. A ceased answer lifts the hold, and the rule it stood in for applies again: a resubmit rule with budget left starts the next submission. Without the interface, or while the answer is no, the job waits for a person. The Slurm adapter's signal is a site command; the Kubeflow adapter's reads the state of the nodes the object ran on.

Limits

  • watch stops at the first error. A store error ends watch, on POSIX as on S3, where a scan that doesn't finish within -timeout is one. Saves written after that aren't published until something publishes them, so a job script should treat watch exiting early as a failure.
  • A publication isn't a data sync. It's durable when Publish returns, but the save's own files are only as durable as the framework made them.
  • dispatch never reads the checkpoint itself. The core passes JobSpec.Checkpoint through; only the helper, inside the job, touches it.