Annotations, labels and names¶

Everything the adapter writes is under the prefix dispatch.phpboyscout.uk/, except three Kueue keys it is required to use: the labels kueue.x-k8s.io/queue-name and kueue.x-k8s.io/priority-class, and the annotation kueue.x-k8s.io/admission-gated-by. All writes use the field manager dispatch.phpboyscout.uk/adapter, and every update after creation carries the object's UID and resourceVersion as a precondition.

Names¶

Object	Name
TrainJob	`dj-<lowercase JobID>-s<N>`, at most 37 characters
runtime copy (a TrainingRuntime)	the same name as its TrainJob
runtime patch entry	manager `dispatch.phpboyscout.uk/adapter`
the start gate	`dispatch.phpboyscout.uk/gate`
AdmissionCheck controller name	`dispatch.phpboyscout.uk/admission`
orphan object reference	`<namespace>/jobset/<name>`, `<namespace>/pod/<name>` or `<namespace>/trainingruntime/<name>`; a TrainJob is `<namespace>/<name>`

On the TrainJob¶

Labels¶

Label	Value	When
`dispatch.phpboyscout.uk/job-id`	the JobID, a ULID	always
`dispatch.phpboyscout.uk/submission`	N	always
`kueue.x-k8s.io/queue-name`	`kueue.localQueue`	Kueue mode
`kueue.x-k8s.io/priority-class`	`kueue.priorityClass`	Kueue mode, when set
the site's `labels`	as configured	always

TrainJob labels do not reach pods. The pod labels are in the runtime patch, below.

Annotations¶

Annotation	Value	Written by
`dispatch.phpboyscout.uk/comment`	`dispatch:s=<N>;a=<attempt>`	`Start`
`kueue.x-k8s.io/admission-gated-by`	`dispatch.phpboyscout.uk/gate`, a comma list if others are added	`Start`; `Open` removes dispatch's entry
`dispatch.phpboyscout.uk/opened`	`true`	plain-mode `Open`
`dispatch.phpboyscout.uk/admission-floor`	the highest A raised so far	each raise of A
`dispatch.phpboyscout.uk/jobset-uid`	the JobSet's UID	the adapter, or the admission controller, the first time it sees the JobSet
`dispatch.phpboyscout.uk/hold-instance`, `hold-since`, `hold-who`, `hold-until`, `hold-reason`	see holds	plain-mode `Hold`
the site's `annotations`	as configured	`Start`

Spec¶

The TrainJob's spec, field by field, is in how a job becomes a TrainJob.

Environment variables¶

Variable	Visible to the job	Value
`DISPATCH_JOB_ID`	yes	the JobID
`DISPATCH_SUBMISSION`	yes	N
`DISPATCH_EPOCH`	yes, set by the prologue	see execution identity
`DISPATCH_ATTEMPT`	yes, set by the prologue	`<JobID>.<N>.<attempt>.<A>.<J>`
`DISPATCH_K8S_EPOCH_PREFIX`, `DISPATCH_K8S_TOKEN_PREFIX`, `DISPATCH_K8S_ADMISSION`, `DISPATCH_K8S_RESTART`, `DISPATCH_K8S_WORKDIR`	no, unset by the prologue	inputs to the prologue; `DISPATCH_K8S_WORKDIR` only when `WorkDir` is set

DISPATCH_PUBLISH_DEST (value /opt/dispatch) is set on the publish init container under publish.via: initContainer.

The runtime patch¶

One entry in spec.runtimePatches, manager dispatch.phpboyscout.uk/adapter, rewritten and never appended, on the trainer replicated job's pod template:

Part	Value
pod labels	`dispatch.phpboyscout.uk/job-id`, `dispatch.phpboyscout.uk/submission`
pod annotation	`dispatch.phpboyscout.uk/admission`: A, `"0"` at creation
volumes	`dispatch-credential-<i>` for each credential (a Secret volume, `defaultMode` 0400, `optional: true`); `dispatch-publish` when `publish` is set (an `image` volume with `pullPolicy: IfNotPresent`, or an `emptyDir` under the init container route)
`node` container mounts	each credential at its `mountPath`, read-only; `dispatch-publish` at `/opt/dispatch`, read-only
init container (init container route only)	the named container, with `DISPATCH_PUBLISH_DEST=/opt/dispatch` and the volume mounted there

On the runtime copy¶

Key	Value
label `dispatch.phpboyscout.uk/job-id`, `dispatch.phpboyscout.uk/submission`	the TrainJob's
annotation `dispatch.phpboyscout.uk/runtime-source`	`<kind>/<name>@<resourceVersion>` of the runtime it was copied from
`ownerReferences`	the TrainJob, `controller: false`, `blockOwnerDeletion: false`
`spec`	the source runtime's, verbatim, never updated by the adapter

A metadata.generation above 1 means someone edited it; the object is then reported as foreign.

On the Workload (Kueue mode)¶

Key	Value
`spec.active`	`false` while held
annotations `dispatch.phpboyscout.uk/hold-instance`, `hold-since`, `hold-who`, `hold-until`, `hold-reason`	the hold, see holds
`status.admissionChecks[]` for dispatch's check	`Pending`, `Ready`; set by the admission controller, with the messages in `dispatch-admission`

What the adapter reads from others¶

Key	Where	Used for
`kueue.x-k8s.io/job-uid`	label on a Workload	finding the TrainJob's Workload
`jobset.sigs.k8s.io/jobset-name`	label on a pod	finding a TrainJob's pods
`jobset.sigs.k8s.io/restart-attempt`	label on a pod	J
`batch.kubernetes.io/job-completion-index`	label on a pod	finding rank 0, whose log `Output` reads
`trainer.kubeflow.org/trainjob-ancestor-step: trainer`	label on a runtime's replicated job	finding the trainer replicated job
`kueue.x-k8s.io/admission-gated-by`	annotation on the TrainJob	dispatch's own gate, which is read from the TrainJob alone, and other parties' gates
`kueue.x-k8s.io/admission-gated-by`	annotation on a Workload	other parties' gates
`node.kubernetes.io/unreachable`, `node.kubernetes.io/not-ready`, `node.kubernetes.io/out-of-service`, and the site's `nodeLossTaints`	taints on a Node a pod ran on	telling node loss from another taint eviction; `out-of-service` alone also means the node's processes are gone, for `Ceased` and for a pod that the pod garbage collector deleted

Prefixes a site may not use¶

Keys in the configured labels and annotations are refused when they begin with dispatch.phpboyscout.uk/, kueue.x-k8s.io/, trainer.kubeflow.org/ or jobset.sigs.k8s.io/.

Other fixed names¶

Name	Meaning
`node`	the trainer container, in every runtime
`/opt/dispatch`	where `cmd/publish` is mounted; credentials may not overlap it
`dispatch-system` / `dispatch-admission`	the Lease's default namespace and name
`dispatch.invalid/never:probe`	the preflight probe's image, in a reserved TLD so it never resolves
`dispatch.phpboyscout.uk/probe: never`	the probe's node selector, a label no node carries