Safety from conditional commits and gated launches¶

The supervisor's job sounds simple: start the job, and when it ends, do what the plan says. The hard part is doing it when the supervisor itself can crash halfway through, stall for minutes, be restarted, or end up running twice, and when the scheduler can take its time answering, or not answer at all.

dispatch makes three promises about that:

• no launch runs that the record doesn't authorize. A launch submitted twice, or one the record no longer wants, stays held on the scheduler, then is retired and stopped;
• a committed cancel is final. Nothing new is authorized after it;
• nothing is forgotten. A finished job's record stays open until every launch it authorized is accounted for.

None of them depends on there being only one supervisor.

Why not a lock¶

The obvious design is a lock: one supervisor per job, and everyone else keeps out. It doesn't work, for a reason that's easy to miss. Slurm and Kubernetes don't check fencing tokens. A supervisor that stalls after sending sbatch can have that call land long after another supervisor has taken over, and no lock on dispatch's side can stop a call that's already on its way to the scheduler.

So dispatch splits the problem into three, and gives each its own mechanism:

Conditional commits¶

Every change to a job's record is a commit against the revision the writer read. If anything else was committed in between, the store refuses it with records.ErrConflict, and the writer reads again before deciding again. A decision made on a stale view is rejected, never written and judged void later.

The supervisor makes two kinds of commit:

• decisions (authorize a submission, choose the canonical object, retire one, commit an outcome, settle) are committed against the revision the decision was made from, and carry the supervisor's ownership lease where the store can check it;
• evidence (what the scheduler showed, what a Start call replied) is committed at whatever the revision is now, with no lease, because a late result is still a fact.

Each pass of the supervisor reads the record, then asks the scheduler what it has, then decides. Reading first matters: a judgement like "this object is gone, so it stopped" is committed against a revision read before the Find it rests on, so it can never cover an object created after that Find began.

Gated launches¶

Every object is created with its start held back: sbatch --hold on Slurm, an admission gate under Kueue, spec.suspend on Kubeflow without Kueue. Nothing runs until dispatch calls Open. The gate isn't a hold: a hold is what a recovery rule or a person asks for, and each adapter keeps the two apart (see Slurm's and Kubeflow's gates). Launching submission N goes like this:

1. authorize N in the record, reserving the rule's budget under a key that can only be used once;
2. commit the attempt number, then call Start. The object carries its submission and attempt numbers, so it can be found and matched later;
3. record the reply, whatever it was. A reply that isn't a clear refusal is recorded as unknown, and the call may be retried, up to two more times. A timed-out sbatch still creates its job, so "no reply" never means "not created";
4. choose the canonical object: the lowest attempt number among N's objects in the latest Find that aren't retired, ended ones included, so an object that ended before it was chosen still decides the submission rather than leaving it undecided. That's committed once and never recomputed, because Slurm purges finished jobs, and recomputing from what's visible later could pick a duplicate;
5. retire every other object of N, permanently, and stop it with a per-object Stop that can't touch the canonical one;
6. open the canonical object's gate, but only when it's committed as canonical, the job is meant to run (or is held, with its hold already placed on the gated object), every duplicate is retired and gone, and every earlier submission's canonical object has been seen ended by the scheduler.

That last condition is what stops two runs overlapping. Submission N+1 can be authorized and started while N is still ending, but its gate stays shut until the scheduler shows N ended.

A duplicate costs nothing while it's gated: it holds no GPUs, and it's retired before anything is opened.

Cancel¶

Cancel commits first, then stops. The commit sets the outcome to cancelled, lifts every hold in the record, and drops any requeue or resubmit intent not yet carried out. Then StopJob stops every object of the job by name, gated ones included.

A launch can't escape that, because each side writes, then reads the other. If the stop runs after the launch's object exists, it catches it. If it runs before, the cancel commit came earlier still, so the launch's re-read before Open sees it and doesn't open.

That covers the decision to open, not the moment an Open lands. An Open sent from a read just before the cancel can still land after it. What's guaranteed is that no Open is sent from a read that saw the cancel, and that everything opened is stopped before the record settles.

Slurm will requeue a cancelled job. So the supervisor runs StopJob a second time once every requeue call recorded for the job has landed or expired, and the cancel's receipt only reports clean-up complete once they have.

Calls that can land late¶

A hold, a release or a requeue sent from a stale view could land after a newer decision and undo it. So each carries a deadline, committed to the record before the call is sent, and the scheduler's host refuses to act on a call that started after its deadline. A call counts as expired at its deadline plus the adapter's declared CallBound, the longest a started command can take to land.

From those recorded deadlines any later actor can work out when every call is safely over. The supervisor places a new hold instance on an object only once every earlier-instance call on it has expired, and settlement and N+1's gate both wait for requeue calls to expire before trusting a Find.

Settlement¶

A job that has reached an outcome isn't finished with. Its record stays open, and Next keeps stopping anything of the job that's alive, until the settled predicate holds: every object ever recorded and every object the latest Find returned has ended or been verified stopped, no foreign object is undisposed, no submission is stuck with every attempt unknown, and (under the manual unmatched policy) no attempt is unmatched. Only then does the supervisor commit the record as settled.

The lock is for liveness¶

The supervisor does take ownership of a job, through the store's optional records.Owner interface: Run returns records.ErrOwned if another supervisor holds it, and ErrDisplaced if it loses it. That stops two supervisors doing the same work. It isn't part of the safety argument: everything above holds for any number of actors. The simulator checks it by running up to three supervisors contending for one job, and by taking ownership away from a live one, so a displaced supervisor's calls keep landing while another works.

What the lock does give you is a signal. A job with a supervisor-enforced rule and nobody holding it is reported as Unattended in its status.

No state outside the record¶

The supervisor keeps nothing the record doesn't hold. A supervisor restarted mid-pass reads the record and carries on: an attempt with no reply is treated as unknown, and retried only within the committed attempt count. It starts no goroutines of its own; each job is one loop that wakes on its context, the adapter's watch stream or one timer, and stopping it drains the step in flight within WithDrain's bound.

Related¶

• The steps Next returns, and how each is committed.
• How the simulator tested the supervisor, which tests these promises against the real code.