Skip to content

Read a preflight result

The adapter proves what it depends on once, when Factory opens it, and does not look again. This page shows where the result appears and what to do about each outcome. The result has three possible shapes:

  • Factory returns an error. The cluster is refused outright.
  • The adapter opens, but is observe-only. It reads and classifies; it does not write.
  • The adapter opens normally, and some capability is lowered. Its Support says why.

Read the answers

Ask for the capabilities with no spec, which returns the cluster-level answers preflight fixed:

caps, err := backend.Capabilities(ctx, nil)
if err != nil {
    return err
}

for _, c := range []model.Capability{
    model.ReadmissionGate, model.AttemptInjection, model.HoldCapability,
    model.OutputRead, model.OutputFollow,
} {
    s := caps[c]
    fmt.Printf("%s: %s (%s) %s\n", c, s.Value, s.Source, s.Reason)
}

Source is probed when preflight established the answer and declared when the adapter states it. Five capabilities are settled when the adapter opens: ReadmissionGate, AttemptInjection, Hold, and, by the pods/log permission, OutputRead and OutputFollow. Capabilities lists them all, with the source of each.

Observe-only

An observe-only adapter either may not write, or has no proof that one of its gates works, so it will not launch anything. Find, Inspect, Watch, Output, Ceased and Capabilities work. Writes do not:

  • Start returns ErrUnsupported;
  • every other write (Open, Stop, StopJob, Hold, Release) returns ErrReadOnly.

Both errors carry the reason in their text: the cluster is observe-only: <reason>. The reason is also in the Hold capability's Reason. A ReadOnly adapter, one opened with ReadOnly: true, returns ErrReadOnly from every write, including Start, and does not run the probe. Its Hold is unknown with the reason not probed: the backend is read-only.

Three things make an adapter observe-only. Find the reason in the table.

The reason says Because Fix
not permitted to create trainjobs.trainer.kubeflow.org, ... (a list of verbs and resources, which can include delete jobsets.jobset.x-k8s.io and delete pods) the adapter may read but not write grant the verbs
the readmission gate is not proven: kueue.admissionCheck is not set Kueue mode, no admissionCheck set it
the readmission gate is not proven: LocalQueue <ns>/<name> does not exist the configured queue is missing create it, or correct kueue.localQueue
... ClusterQueue <name> does not list AdmissionCheck <check> the ClusterQueue does not list the check, or lists it with onFlavors list it without onFlavors
... AdmissionCheck <check> does not exist the object is missing create it
... AdmissionCheck <check> names controller "<x>", not dispatch.phpboyscout.uk/admission wrong controllerName correct it
... AdmissionCheck <check> is not Active the controller has not started deploy or start it
... Lease <ns>/<name> does not exist no controller has ever led deploy it, or correct kueue.admissionLease
... the admission controller's Lease has no holder or ... has never been renewed nothing leads start the controller
... the admission controller's Lease was last renewed <age> ago, beyond <limit> the controller is down or stalled restart it
the start gate is not proven: pods appeared for the gated probe: ... pods ran for a TrainJob that should have been held something other than the gate lets TrainJobs through: look at what admits them
the start gate is not proven: Kueue admitted the gated probe's Workload <name> Kueue admitted a gated Workload check the Kueue version and the AdmissionGatedBy feature

In the readmission rows the full text begins with the readmission gate is not proven:. When the adapter may not read a ClusterQueue, an AdmissionCheck or the Lease, the readmission gate is unknown, not no, with the reason not permitted to get <resource> and the apiserver's own explanation; it is observe-only all the same. Grant those get permissions: they are cluster-scoped for ClusterQueues and AdmissionChecks, and namespaced to the Lease's namespace.

The start gate's probe needs the permission to write. It runs only if the adapter is not observe-only for another reason and not ReadOnly.

Factory refuses the cluster

These come back from Factory itself:

Error Text contains Means
ErrUnsupported the cluster does not serve <group-versions>; this adapter supports Trainer v2.3, JobSet v0.12 and Kueue v0.20 an API the adapter needs is not served. It checks v1, trainer.kubeflow.org/v1alpha1, jobset.x-k8s.io/v1alpha2, and in Kueue mode kueue.x-k8s.io/v1beta2 and coordination.k8s.io/v1
ErrUnauthorized the adapter may not <verbs>, so it cannot observe the cluster a read permission is missing; the adapter cannot even watch
ErrUnsupported Kueue made Workload <name> for a TrainJob with no queue ... plain mode on a cluster where Kueue manages queue-less jobs
any preflight probe: ... the probe's TrainJob could not be created, or could not be removed. An ErrUnreachable here says objects remain after stopping it
any preflight: ... a request preflight made failed: discovery, a SelfSubjectAccessReview, or a read of the readmission gate's objects, for example because the apiserver could not be reached
ErrInvalid kubeflow settings: ... the configuration is wrong; nothing was contacted

The full text starts with the sentinel's own words, such as dispatch: unsupported: the cluster does not serve ..., so match on the part shown.

A probe left behind by a failure is named dj-<lowercase ULID>-s1. The adapter tries to stop it; if it reports objects remaining, delete the TrainJob by hand.

A capability is lowered

Capability Lowered when Reason
OutputRead, OutputFollow the adapter may not get pods/log not permitted to get pods/log
Hold the probe's hold steps failed, or in Kueue mode the adapter may not update Workloads preflight probe step <name> failed: ..., or not permitted to update workloads.kueue.x-k8s.io

When Hold is no, Hold and Release return ErrUnsupported with the reason. The probe's steps are: place hold instance 1, inspect it, Open and see the object still held, and a Release naming instance 2 refused as stale.

If the adapter may not get nodes, nothing is lowered in Capabilities, but Ceased returns ErrUnsupported, and node-loss classification cannot read the Node's taints; its cause says the taint was not checked.

With a spec, Capabilities(ctx, &spec) also reads the spec's runtime and refuses with ErrUnsupported one the adapter cannot run. See runtime requirements.