Skip to content

Capabilities

go/dispatch asks every adapter for a Support per capability. The Kubeflow adapter's answers, from Capabilities(ctx, spec):

Capability Answer Source Reason or what changes it
ContainerImage yes declared spec.trainer.image
NativeRequeue no declared no push-down in v1
NativeRestart no declared no push-down in v1
Hold yes or no; unknown if read-only probed; declared when read-only the preflight probe's hold round trip, and in Kueue mode permission to update Workloads. Read-only: not probed: the backend is read-only
PreTerminationSignal unknown catalogue terminationGracePeriodSeconds is honoured, but SIGTERM delivery was not observed
AccountingHistory no declared a deleted TrainJob is gone
AttemptInjection yes in plain mode; Kueue mode: probed declared or probed plain: Open and Release raise the admission count. Kueue: the same as ReadmissionGate, since the controller injects it
ReadmissionGate no in plain mode; Kueue mode: probed declared or probed plain: not needed: nothing re-admits a TrainJob without Kueue. Kueue: proven from the ClusterQueue, the AdmissionCheck and the Lease
WorkingDirectory yes declared the prologue changes into WorkDir
ProcsPerNode unknown with no spec; yes on a runtime with mlPolicy.torch; no on one with no ML policy declared a runtime with no ML policy ignores numProcPerNode
OutputPath no declared no stable output location
OutputRetained no declared logs go with the pods
OutputRead yes; no without get pods/log declared, or probed when lowered rank 0's log of the current execution
OutputFollow yes; no without get pods/log declared, or probed when lowered the same

AttemptInjection must be yes for the adapter to write at all, and in Kueue mode so must ReadmissionGate; when they are not, it is observe-only. In plain mode ReadmissionGate is no because no gate is needed, and the adapter writes. See Read a preflight result.

With a spec, Capabilities reads the spec's runtime and refuses with ErrUnsupported one that breaks the rules.

Rules

For each rule in a recovery policy the adapter says whether its action can be carried out here:

Action Answer
Resubmit, Escalate yes
Hold the Hold capability's answer
Requeue no: Kubeflow has no requeue
Restart no: no push-down in v1
any other unknown, naming the action

Output

Output serves rank 0's log of submission N's current execution, from the node container, through pods/log, following it when asked. Kubernetes interleaves standard output and standard error in one log, so Stdout returns it and Stderr is ErrUnsupported. Earlier executions' logs went with their pods. A job with no rank 0 pod yet returns ErrNotFound.

Watch

The Backend implements Watcher, which is also how it keeps the evidence of deleted pods. See classification.

Cessation

The Backend implements CessationSignal. Ceased is true when every Node of the object's current or last execution is deleted or carries the taint node.kubernetes.io/out-of-service. Otherwise it reports not ceased with each Node's state. Without permission to get Nodes it returns ErrUnsupported. It writes nothing, so it works on a read-only adapter.