The start gate and the readmission gate¶

go/dispatch's contract has two rules the adapter has to keep on Kubernetes. No object may run before dispatch has recorded it and opens it. And every execution of an object, including each one after a preemption, must start with an identity that is higher than the last. Kubernetes does neither by itself. The adapter uses two gates.

The start gate¶

Start creates the TrainJob closed. Nothing is scheduled until Open. The point of a separate Open is that dispatch records the object's identity, a name and a UID, before anything runs, so that a crash between creating a job and writing it down cannot leave an untracked job burning GPUs.

How the TrainJob is closed depends on the mode.

In Kueue mode it carries the annotation kueue.x-k8s.io/admission-gated-by set to dispatch.phpboyscout.uk/gate. Kueue copies the annotation to the TrainJob's Workload and will not admit a Workload while the annotation is non-empty. Kueue allows the annotation to be removed after creation, never added, which is exactly the asymmetry a gate needs: a job cannot be re-gated by accident. Open removes dispatch's entry from the annotation's comma list, in an update preconditioned on the resourceVersion it read. When dispatch's was the last entry it deletes the annotation rather than leaving it empty, because an absent key is the one state every reader agrees means ungated.

If someone else's gate is still on the list, the object stays gated by them, and the adapter reports it as held by an administrator, which dispatch escalates.

Kueue copies the annotation to the Workload only while the Workload is active. A hold placed before Open deactivates the Workload, so after Open the Workload may still carry dispatch's gate until the hold is released. For that reason the adapter reads its own gate from the TrainJob alone.

In plain mode the TrainJob is created with spec.suspend: true. Open sets suspend: false, adds the annotation dispatch.phpboyscout.uk/opened: "true" and raises the admission count in one update. The opened mark is what lets a suspended object be told apart: suspended and never opened is a gated object; suspended after opening is a hold or someone else's suspension.

Open on an object that already ended returns ErrRejected: it is a fence, because the object finished without ever being opened. Open on an object already past its gate succeeds.

After Start creates the TrainJob, it reads it back and refuses it, stopping the object, if anything the adapter wrote has changed: labels, annotations, runtimeRef, trainer, activeDeadlineSeconds, its runtime patch, or a managedBy set by someone else. It does not compare fields others own by design, such as spec.suspend in Kueue mode. In Kueue mode it then waits up to timeouts.workload for the Workload, and checks the Workload carries the gate and lists the configured AdmissionCheck. No Workload in time means Kueue is not managing the TrainJob, and Start stops the object and refuses.

Preflight proves the gate rather than assuming it. It creates a probe TrainJob that cannot be scheduled, and watches it for ten seconds: no pod may appear, and in Kueue mode the Workload may not be admitted. If the gate leaks, the adapter is observe-only.

The readmission gate¶

The start gate guards the first execution. A job that Kueue preempts, or that is deactivated and reactivated, is admitted again by Kueue with no dispatch call involved, and each execution needs a new admission count A (see execution identity). Something has to raise A before each admission, and that has to be in the cluster, because Kueue re-admits at any time, including when no supervisor is running.

Kueue offers one hook: an AdmissionCheck. Kueue resets every check to Pending on each eviction and will not admit the Workload until the checks are Ready. cmd/dispatch-admission serves dispatch's check. For each Workload whose check is Pending and whose quota is reserved, it:

1. finds the Workload's owning TrainJob;
2. if the TrainJob is not dispatch's, sets the check Ready at once, so a shared ClusterQueue is not held up by it;
3. if it is dispatch's and a pod of the previous execution may still run, leaves the check Pending and writes the remaining pods into its message (never two executions at once);
4. otherwise raises A in the TrainJob and then sets the check Ready.

A TrainJob is dispatch's when its name matches dj-<ULID>-s<N> or it carries a runtime patch whose manager is dispatch.phpboyscout.uk/adapter. Labels are not consulted, so stripping them does not wave a job past the check.

The controller keeps no state. Each decision is made from one read of the Workload, and each write is preconditioned on that read's resourceVersion. That is what makes a slow or stalled controller safe. If a former leader reads Pending, stalls while Kueue evicts and resets the check, and then wakes to write Ready, its write fails with a conflict, because Kueue's reset changed the version. Without the precondition that stale Ready would admit the next execution without raising A, and two executions would share an identity.

Leader election exists only so replicas do not do the same work twice. A Lease is not a fence, and nothing relies on it for safety.

It waits for quota to be reserved before it acts, though Kueue would accept an earlier Ready. A Ready written before quota is reserved is banked until the next admission, whose timing nobody controls. Waiting keeps the pod check, the raise of A and the admission it approves in one cycle.

If the controller crashes between raising A and setting Ready, the next pass raises A again. A skips a number, which does no harm: R only has to increase.

Plain mode needs no readmission gate. Nothing re-admits a plain-mode TrainJob: it runs once between Open and its end, and Release after a hold is itself an adapter call that raises A. So preflight reports ReadmissionGate as no, with the reason that it is not needed; the capability says whether a gate runs, as it does for the Slurm adapter, which needs none either.

What happens when the controller is down¶

Jobs that have reserved quota wait at the Pending check, holding their quota. Nothing runs. Status shows the check's state and message. The controller does not set the check inactive on its way down (why), so Kueue does not stop the queue; the wait is at the check.

A new adapter opened while the controller is down is observe-only, since it cannot prove the gate (the Lease heartbeat).

See Install the admission controller.