Skip to content

Runtime requirements

A job names a Trainer runtime, a ClusterTrainingRuntime or a TrainingRuntime, with its kubeflow.RuntimeRef. The adapter reads the runtime before it creates anything and refuses the job with ErrUnsupported, naming every reason it finds, when the runtime breaks a rule below. Capabilities(ctx, &spec) applies the same rules, so a job on such a runtime can be refused at planning time. The checks on the rest of the job, including the ProcsPerNode one below, run only at Start; how a job becomes a TrainJob lists them.

The rules exist because the adapter's guarantees depend on what the runtime does when a pod fails: anything that restarts work invisibly would run a rank twice under one identity. Execution identity explains it.

The rules

Rule Because
The runtime is in runtimes, if that list is set; and it exists. the site's allow-list
mlPolicy names nothing but torch, besides numNodes. v1 runs plain and torch runtimes only. MPI, Flux and JAX are refused
The runtime's replicated jobs are the trainer one and nothing else. The trainer job is the one whose template carries the label trainer.kubeflow.org/trainjob-ancestor-step: trainer. initializers, MPI launchers and other steps were never exercised
The trainer job's pod template states restartPolicy: Never. a kubelet restarting a container in place raises neither A nor J
The trainer job states backoffLimit: 0. with a limit above 0, the Job replaces a failed rank's pod within one execution, which nothing above the Job shows
The trainer job has no podFailurePolicy rule with action Ignore, and no backoffLimitPerIndex. Waived by allowRankRestart. each lets the Job replace a failed pod without failing
Every podFailurePolicy rule has a known action: FailJob, FailIndex, Count or Ignore. an unknown action is meaning the adapter does not understand
The trainer pod has a container named node. dispatch's runtime patch names it
Under publish.via: initContainer, the trainer pod has an init container with the configured name. Trainer accepts a patch naming a missing container and the job then holds quota with no pods
The JobSet failurePolicy, if there is one, has maxRestarts 0 (or unset) and every rule's action is FailJobSet. v1 restarts nothing natively; a restart the plan never chose is unplanned recovery

An absent failurePolicy is accepted: JobSet then fails on the first Job failure.

FailJob and Count rules in a podFailurePolicy are accepted, since they only fail the Job sooner or count against backoffLimit: 0.

A job asking for ProcsPerNode on a runtime without mlPolicy.torch is refused by Start too, because the runtime would ignore it. Capabilities(ctx, &spec) does not refuse it; it answers the ProcsPerNode capability no for a runtime with no ML policy.

allowRankRestart

Setting allowRankRestart: true in the configuration accepts a runtime whose trainer job has an Ignore rule or a backoffLimitPerIndex. The cost is the weaker identity guarantee: a failed rank can rerun under the same DISPATCH_ATTEMPT, so one execution can contain a restarted rank. The adapter implements go/dispatch's IdentityWeakener and answers, for a spec on such a runtime:

allowRankRestart: the runtime's trainer job states a podFailurePolicy Ignore rule and backoffLimitPerIndex, so a
failed rank can rerun under the same DISPATCH_ATTEMPT and one execution can contain a restarted rank

with whichever of the two it finds. The reason appears in the job's plan and in its status.

What Start does with the runtime

Start copies the runtime's spec into a TrainingRuntime named for the TrainJob, and the TrainJob runs on the copy. A runtime edited after submission therefore does not change a job already submitted. See annotations, labels and names.

An example

This is the ClusterTrainingRuntime the test cluster uses (internal/kindtest/testdata/base.yaml), which satisfies every rule:

apiVersion: trainer.kubeflow.org/v1alpha1
kind: ClusterTrainingRuntime
metadata:
  name: dispatch-kind
spec:
  mlPolicy:
    numNodes: 2
  template:
    spec:
      replicatedJobs:
        - name: node
          template:
            metadata:
              labels:
                trainer.kubeflow.org/trainjob-ancestor-step: trainer
            spec:
              backoffLimit: 0
              template:
                spec:
                  restartPolicy: Never
                  terminationGracePeriodSeconds: 5
                  containers:
                    - name: node
                      image: localhost/dispatch-busybox:kind
                      resources:
                        requests: {cpu: 100m, memory: 32Mi}
                        limits: {cpu: 100m, memory: 32Mi}

The harness also sets securityContext and imagePullPolicy; they are left out here because they have nothing to do with the rules.