Never two executions at once¶

go/dispatch promises that one object never has two executions at once. On Kubernetes that is harder than it sounds, and the adapter's first design broke it.

What went wrong¶

A TrainJob that is suspended is not necessarily stopped. Suspending deletes its pods, but a pod takes up to its termination grace period to go on a reachable node, and indefinitely on a node that has been lost or frozen: its kubelet cannot confirm the container has stopped, so the pod stays Terminating.

The first design assumed Trainer would not let the object be changed or re-admitted while those pods were still there, because Trainer's webhook refuses to change the runtime patch while a replicated job still has active pods. It was tested and the assumption was wrong. JobSet's active count drops to zero the moment the TrainJob suspends, with the pods still Terminating, and the change is accepted. So a re-admission could start a new execution beside the old one's still-running processes.

The new execution's identity would have been fine, since A is higher (execution identity). What would have broken is the thing identity cannot supply: two executions writing to one checkpoint path at once. On Kueue it could not be left to a supervisor either, because Kueue re-admits by itself and the controller would approve at once.

The rule¶

Nothing re-admits an object while any pod of it may still run.

Before A is raised for an admission, the writer lists the namespace's pods in one consistent read, Terminating ones included, and selects the TrainJob's in two independent ways:

• by dispatch's labels dispatch.phpboyscout.uk/job-id and /submission;
• by JobSet's label jobset.sigs.k8s.io/jobset-name equal to the TrainJob's name.

The second is there so that a rewritten runtime patch, which carries the first pair of labels, cannot hide the pods. A pod whose phase is Succeeded or Failed has no running container and does not block, even while it waits to be deleted. Without that exception, a Job that keeps a finished rank's pod across a suspend would block re-admission for good.

If any other pod remains, nothing is raised and nothing is admitted. No pod can appear between the list and the admission, because the object stays suspended until the write that admits it.

Who enforces it¶

• Kueue mode: the admission controller leaves the check Pending and writes the remaining pods into its message: waiting for pods of the previous execution to go: <pod> on <node>, .... The Workload keeps its reserved quota while it waits. Status shows the message.
• Plain mode: Release writes nothing and returns ErrPodsRemain, which names the pods and says the previous execution may still run, so it is not admitted. The hold stays in force. ErrPodsRemain is classified as ErrUnreachable: it is a failed call to retry, not a refusal. Open meets the same check, though a never-run object has no pods.

How it ends¶

On a reachable node the pods go when their kubelet confirms termination, at most the termination grace period later. That is the cost of the rule: each re-admission is delayed by up to that long.

On a lost node the pods stay Terminating until the Node is deleted or tainted node.kubernetes.io/out-of-service. Kubernetes never adds that taint itself: an operator adds it once they have confirmed the node is shut down (Kubernetes' non-graceful node shutdown), and Kubernetes then force-deletes the node's pods. This is also the moment CessationSignal answers ceased: the adapter reports that the object's processes are gone once every Node it ran on is deleted or has the taint. So the re-admission waits exactly until cessation is proven, which is what makes restarting safe. The wait costs the quota the Workload holds.

Seeing it¶

While an object is not running, Inspect and Find put remaining pods: <pod> on <node> (<node state>), ... in the cause detail, where the node state is Ready=True, Ready=False, Ready=Unknown, gone or not readable, with out-of-service added when the taint is present. A pending object in Kueue mode also gets admission check <name> <state> and the check's message. So status explains the stall without a kubectl.

Ceased writes nothing and works on a read-only adapter, but needs permission to get Nodes.