Skip to content

How a job becomes a TrainJob

Start turns one model.StartRequest into two objects in the configured namespace: a copy of the job's Trainer runtime, and a TrainJob that runs on that copy with its start gate closed. This page lists what goes into them and what Start refuses. The labels, annotations, volumes and environment variables on each object are in annotations, labels and names; what a runtime must state is in runtime requirements.

What Start does, in order

  1. Refuses a closed Backend with ErrUnreachable, a read-only one with ErrReadOnly, and an observe-only one with ErrUnsupported, naming the gate that is not proven.
  2. Reads the runtime the kubeflow.RuntimeRef extension names: a TrainingRuntime from the configured namespace, a ClusterTrainingRuntime from the cluster. A runtime outside the configured runtimes, or one that does not exist, is refused with ErrUnsupported.
  3. Checks the runtime against the runtime rules, refusing with ErrUnsupported and naming every reason.
  4. Renders the TrainJob from the request and its spec, refusing what the tables below refuse. Nothing has been created yet.
  5. Creates the runtime copy. If a retried Start finds a copy of that name, it reuses it only if the copy has no owner and its spec equals the source runtime's; otherwise it refuses with ErrRejected. A create that goes unanswered is ErrUnreachable, and no TrainJob is created.
  6. Creates the TrainJob. A create that goes unanswered is ErrOutcomeUnknown, and Find reports the job InFlight for CallBound. A create that fails definitely deletes the copy and returns the cause.
  7. Sets the copy's owner reference to the TrainJob, so that deleting the TrainJob collects it.
  8. Reads the TrainJob back and compares what it wrote: labels, annotations, spec.runtimeRef, spec.trainer, spec.activeDeadlineSeconds, dispatch's runtime patch, a spec.managedBy set to anything but Trainer's controller, and in plain mode spec.suspend.
  9. In Kueue mode, waits up to timeouts.workload, checking every second, for the TrainJob's Workload to exist, carry dispatch's gate and list kueue.admissionCheck.

When step 7, 8 or 9 fails, Start stops the TrainJob it created and returns the failure, which is ErrRejected for an owner that could not be set, a TrainJob that changed, or a Workload that did not appear or lacks the gate or the check. If the stop fails too, it returns ErrOutcomeUnknown. Either way it also returns the object's reference. On success the reference is <namespace>/<name> with the TrainJob's UID. Error mapping lists every refusal and how the supervisor records it.

The request

StartRequest field Becomes Refused when
Job the TrainJob's name, the job-id labels, DISPATCH_JOB_ID it is not a valid JobID
Generation G in the epoch prefix, DISPATCH_K8S_EPOCH_PREFIX = G × 10^6 + N it is negative or above 18,000,000
Submission N: the name's suffix, the submission labels, the comment, DISPATCH_SUBMISSION it is outside 1 to 999,999
Attempt the comment's a=<attempt> and the attempt token's prefix it is negative
Spec the TrainJob, below
Native nothing: the adapter declares NativeRequeue and NativeRestart no

Each of these refusals is ErrInvalid.

The spec

Start first runs go/dispatch's JobSpec.Validate, which refuses with ErrInvalid, reporting every problem at once: no command, a NUL in the command or env, fewer than one node, a negative count or time limit, a reserved or malformed env key, and a checkpoint that is neither an absolute path nor an s3:// URL.

JobSpec field Becomes The adapter also refuses
Command spec.trainer.command: /bin/sh, -c, the prologue, dispatch-prologue, then the command more than 124 items, because Trainer allows 128 and the prologue takes four: ErrInvalid
Env spec.trainer.env, sorted by key, followed by dispatch's six variables, seven with WorkDir more than 122 entries, 121 with WorkDir, because Trainer allows 128: ErrInvalid. Keys beginning DISPATCH_ are reserved
WorkDir DISPATCH_K8S_WORKDIR, which the prologue enters; a missing directory makes the container exit 125
Nodes spec.trainer.numNodes
GPUsPerNode spec.trainer.resourcesPerNode.limits[<gpuResource>], when above 0; gpuResource defaults to nvidia.com/gpu
ProcsPerNode spec.trainer.numProcPerNode, when above 0 a runtime without mlPolicy.torch, which would ignore it: ErrUnsupported
TimeLimit spec.activeDeadlineSeconds, rounded up to a whole second, when above 0
Name nothing
Checkpoint nothing in the TrainJob
Recovery nothing in the TrainJob; Rules answers for it
Extensions below

These checks run only at Start. Capabilities(ctx, &spec) checks the runtime, not the rest of the spec.

Extensions

Extension Required Becomes
dispatch/Image (model.Image) yes: missing, or with an empty Ref, is ErrInvalid spec.trainer.image
kubeflow/RuntimeRef yes: missing is ErrInvalid the runtime that is copied
dispatch/SharedCheckpoint no accepted; it changes nothing in the TrainJob
any other refused with ErrUnsupported, extension <namespace>/<name> is not supported on Kubeflow, never ignored

The TrainJob

The TrainJob and its runtime copy are both named dj-<lowercase JobID>-s<N>, at most 37 characters.

Field From
spec.runtimeRef {apiGroup: trainer.kubeflow.org, kind: TrainingRuntime, name: <the TrainJob's name>}: the runtime copy
spec.suspend true at creation in plain mode; Kueue's in Kueue mode, never written by the adapter. Set by Open, Hold and Release in plain mode
spec.trainer.image the dispatch.Image extension
spec.trainer.numNodes Nodes
spec.trainer.numProcPerNode ProcsPerNode, torch runtimes only
spec.trainer.resourcesPerNode.limits[gpuResource] GPUsPerNode, when above 0
spec.trainer.command /bin/sh, -c, the prologue, dispatch-prologue, then the job's command
spec.trainer.env the job's Env sorted by key, then dispatch's variables
spec.activeDeadlineSeconds TimeLimit rounded up to a second, when set
spec.runtimePatches one entry, dispatch's runtime patch

Its labels and annotations are in annotations, labels and names. In Kueue mode they include the queue label and the start gate; in plain mode the gate is spec.suspend.

The runtime copy

A TrainingRuntime with the TrainJob's name, whose spec is the source runtime's, copied verbatim and never updated, so a later edit to the source does not change a job already submitted. Its keys are in annotations, labels and names.

The preflight probe

The probe that Factory runs is a Start like any other, with a spec of its own: the image dispatch.invalid/never:probe, the command true, one node, submission 1, attempt 1, and the runtime named by probeRuntime. Its runtime patch adds the node selector dispatch.phpboyscout.uk/probe: never, which no node matches, so it can never be scheduled. Because it goes through the same steps, probeRuntime must pass the same checks as a job's runtime.