Execution identity: R = A + J¶

go/dispatch gives every execution of a job two values that the job's own code can read: an epoch, DISPATCH_EPOCH, which only increases, and an attempt token, DISPATCH_ATTEMPT, which is unique. Checkpoint publication orders saves by epoch, so a late save from an old execution can never overwrite a newer one. On Kubernetes the adapter has to build both from what the cluster exposes.

Two counters¶

A single execution of a TrainJob can end and a new one begin in two ways.

• JobSet restarts the job. JobSet counts these in status.restarts and stamps the count on every pod as the label jobset.sigs.k8s.io/restart-attempt. Call it J. Nothing lowers it, and Trainer keeps one JobSet per TrainJob, so J survives holds and re-admissions.
• Kueue admits the job again, after a preemption or a hold. JobSet does not restart anything here, so J does not move. The adapter therefore keeps its own count, the admission count A. It starts at 0 in the TrainJob's runtime patch, as the annotation dispatch.phpboyscout.uk/admission on the pod template, and is raised by at least one before every admission.

Every boundary between executions raises at least one of the two, so their sum R = A + J only increases. The first execution has R = 1.

DISPATCH_EPOCH   = G × 10^12 + N × 10^6 + R
DISPATCH_ATTEMPT = <JobID>.<N>.<attempt>.<A>.<J>

G is the job's generation, N the submission number and attempt the attempt, all from the StartRequest. R below 1,000,000 is required; at the bound the prologue refuses to run.

Who raises A¶

A can change only while the TrainJob is suspended, which Trainer's validating webhook enforces. So the raise happens at the one moment the object is closed:

• Kueue mode: the admission controller raises A in the TrainJob, then approves the admission (the two gates);
• Plain mode: Open and Release raise A in the same update that unsuspends.

Each raise writes max(A in the patch, the floor) + 1 into both the runtime patch and the TrainJob annotation dispatch.phpboyscout.uk/admission-floor, in one update preconditioned on what was read.

How the container gets them¶

A pod's container reads A once, when it starts, so A never changes under a running pod. The adapter writes the following into spec.trainer.env at Start. Trainer does not let a runtime patch set env on the node container, and trainer is immutable after creation, so the values that vary at run time come through the Downward API instead of being edited:

Your job's own Env goes before these, sorted by key. Keys starting with DISPATCH_ are reserved.

The prologue¶

The trainer's command is not your command. It is /bin/sh -c <prologue> dispatch-prologue <your command...>. The prologue turns the DISPATCH_K8S_* values into the two the job sees, then execs your command, so your process is PID 1 of the container:

set -eu
if [ -n "${DISPATCH_K8S_WORKDIR:-}" ]; then
  cd "$DISPATCH_K8S_WORKDIR" || { echo "dispatch: cannot enter $DISPATCH_K8S_WORKDIR" >&2; exit 125; }
fi
case "$DISPATCH_K8S_ADMISSION$DISPATCH_K8S_RESTART" in *[!0-9]*) false ;; esac &&
  [ -n "$DISPATCH_K8S_ADMISSION" ] && [ -n "$DISPATCH_K8S_RESTART" ] ||
  { echo "dispatch: admission or restart count missing or not a number" >&2; exit 125; }
R=$((DISPATCH_K8S_ADMISSION + DISPATCH_K8S_RESTART))
if [ "$R" -ge 1000000 ]; then echo "dispatch: restart count $R reached the epoch bound" >&2; exit 125; fi
DISPATCH_EPOCH=$(printf '%s%06d' "$DISPATCH_K8S_EPOCH_PREFIX" "$R")
DISPATCH_ATTEMPT="$DISPATCH_K8S_TOKEN_PREFIX.$DISPATCH_K8S_ADMISSION.$DISPATCH_K8S_RESTART"
export DISPATCH_EPOCH DISPATCH_ATTEMPT
unset DISPATCH_K8S_ADMISSION DISPATCH_K8S_RESTART DISPATCH_K8S_EPOCH_PREFIX DISPATCH_K8S_TOKEN_PREFIX DISPATCH_K8S_WORKDIR
exec "$@"

What this means for your image:

• It must provide /bin/sh and printf. A FROM scratch training image will not run.
• It refuses to run on a missing or non-numeric A or J, exiting 125. A Downward API reference to an annotation that has been removed yields an empty string, which shell arithmetic reads as 0 without complaint; that would silently repeat an R, so the prologue checks for it.
• The epoch is built by concatenation, not arithmetic, so it does not depend on the shell's integer width. Only R is computed.
• WorkDir is honoured by the prologue's cd, since a TrainJob has no field for it. If the directory does not exist, the container exits 125.
• The DISPATCH_K8S_* variables are unset before your command runs.

Resets are detected, not prevented¶

Both counters can be reset from outside. If someone deletes the JobSet, Trainer builds a new one with restarts: 0, and if someone deletes or rewrites dispatch's runtime patch, A falls. Either can repeat an R. The adapter notices the accidental cases and flags the object as foreign, with the reason in its cause detail, so that dispatch escalates and a person decides. It does not stop the object itself, because recovery is not the adapter's call.

• The floor. A is raised from max(patch A, admission-floor) + 1, so a rewritten patch does not lower it.
• The JobSet's UID is recorded in the annotation dispatch.phpboyscout.uk/jobset-uid the first time the adapter, or the controller, sees it. A different UID later is reported as JobSet UID <new> is not the recorded <old>: J may have been reset.
• A pod with no admission annotation, or one whose A is below the floor, is reported too.

Detection is the trust boundary. Someone with update rights on the TrainJob who deletes the annotations as well can defeat the check. go/dispatch does not defend against a principal who can rewrite the job's objects.

Why the runtime must not restart in place¶

A kubelet restarting a container in place raises neither A nor J, so two executions would share an identity. That is why the adapter refuses a runtime whose trainer job does not state restartPolicy: Never and backoffLimit: 0, and, unless allowRankRestart is set, one that lets a failed rank be replaced. See runtime requirements.