Skip to content

Error mapping

Every error the adapter returns is classified by a go/dispatch sentinel, and the cause beneath it is kept in the chain: test with errors.Is. The sentinels are described in go/dispatch's errors reference. This page says which one the adapter returns, and when.

From the apiserver

The adapter's client classifies every failed request. The first matching row applies.

The request failed with Sentinel Notes
the token source failed or returned an empty token ErrUnauthorized nothing was sent
404 Not Found ErrNotFound
409 and AlreadyExists ErrRejected a create of a name that exists
409 Conflict ErrRejected a write whose precondition failed: the object changed since it was read, or is another instance of the name
401 Unauthorized, 403 Forbidden ErrUnauthorized
410 Gone, or the resource version expired ErrUnreachable a list or watch whose resourceVersion is no longer served. Lists restart up to three times before giving up
422 Invalid, 400 Bad Request, 405, 413, 406, 415 ErrRejected
a write with no definite answer, that may have reached the apiserver ErrOutcomeUnknown a timeout, a reset or any failure after the request could have been sent
a read with no definite answer, or a write that provably was not sent ErrUnreachable

A write is provably not sent when the connection could not be made (a failed dial or DNS lookup), when its token could not be obtained, or when its call's deadline had passed before it went out.

Two cases are decided before a request goes out:

  • a context that has ended before a request begins is ErrUnreachable, except in a hold-family call whose context ended by passing a deadline, which is ErrRejected with the message the call's deadline passed before the request was sent. A hold-family call whose context was cancelled is ErrUnreachable;
  • a hold-family call without a deadline is ErrInvalid.

Requests made directly rather than through the dynamic client map plainly: for pods/log, 404 is ErrNotFound, 401 and 403 are ErrUnauthorized, 400 is ErrRejected, and any other status is ErrUnreachable. The group-version discovery check maps the same way, except that 404 is not an error: it means the group-version is not served, which Factory reports as ErrUnsupported.

Why a create can be ErrOutcomeUnknown

A create the apiserver received can still commit until its request timeout. So when Start gets no answer to its create, it returns ErrOutcomeUnknown and marks the job in flight for CallBound, which Find then reports as InFlight. A create that fails definitely, say with a rejection, also deletes the runtime copy it made, and returns the cause.

The in-flight mark lives only in the memory of the Backend that made the create. Another process, or this one after a restart, does not report InFlight for it.

From the adapter

Situation Sentinel
any call on a closed Backend ErrUnreachable
any write on a read-only Backend ErrReadOnly
Start on an observe-only adapter ErrUnsupported, naming the missing gate
any other write on an observe-only adapter ErrReadOnly
Hold or Release where Hold is not yes ErrUnsupported, with the capability's reason
Hold or Release naming hold instance 0 ErrInvalid
settings that are malformed, unknown, null, or fail validation ErrInvalid
a kubeconfig or in-cluster configuration that cannot load ErrInvalid
a missing dispatch/Image or kubeflow/RuntimeRef extension, a kubeflow/RuntimeRef whose kind is neither TrainingRuntime nor ClusterTrainingRuntime or whose name is not a Kubernetes object name (also from Capabilities with a spec), a reserved DISPATCH_ env key, an env or command over Trainer's limits ErrInvalid
an extension other than dispatch/Image, dispatch/SharedCheckpoint and kubeflow/RuntimeRef ErrUnsupported
a runtime outside runtimes, a runtime that does not exist, or one that breaks a runtime rule ErrUnsupported
ProcsPerNode on a runtime without mlPolicy.torch ErrUnsupported
a cluster that does not serve a supported group-version, at Factory ErrUnsupported
plain mode on a cluster where Kueue manages queue-less jobs, at Factory ErrUnsupported
a missing read permission, at Factory ErrUnauthorized
an invalid JobID given to Find, Watch, StopJob or Output ErrInvalid
an object read from the cluster whose apiVersion or kind is not the one the adapter supports ErrUnsupported
an object reference that is not this adapter's, names another backend, or carries no UID ErrInvalid
an object reference outside the configured namespace ErrNotFound
an object whose name is now held by another UID ErrNotFound
Open on a TrainJob that has ended: the message says ended without being opened when the gate was never lifted, and has run and ended when it was ErrRejected
Open, Hold or Release on an orphaned JobSet, pod or runtime copy ErrRejected
Hold on a TrainJob that has ended; Hold or Release, in Kueue mode, on a TrainJob with no Workload ErrRejected
Hold or Release naming an instance that is not the one in force kubeflow.ErrStaleHold, which is ErrRejected
Open or Release while pods of the object remain, plain mode kubeflow.ErrPodsRemain, which is ErrUnreachable
Open, Hold or Release that keeps conflicting after eight tries ErrRejected (a conflict)
Start, when read-back finds the TrainJob changed since it was written ErrRejected, naming the difference. The object is stopped
Start, when setting the runtime copy's owner to the TrainJob fails ErrRejected. The object is stopped
Start, when Kueue makes no Workload in time or the Workload lacks the gate or the check ErrRejected. The object is stopped
Start, when its context ends while it waits for the Workload ErrOutcomeUnknown: the stop it then attempts cannot be sent either, so the TrainJob may remain
Start, when stopping the object it refuses also fails ErrOutcomeUnknown
Start retried over a runtime copy that exists and is owned, or has a different spec ErrRejected
the runtime copy's create goes unanswered ErrUnreachable: no TrainJob was created
Output on Stderr ErrUnsupported
Output when OutputRead is not yes ErrUnsupported
Output with no rank 0 pod ErrNotFound
Ceased without permission to get Nodes, at preflight or when it reads one ErrUnsupported
Stop of an object already gone, or whose UID no longer matches success, no error
Stop or StopJob deleting an orphaned JobSet or pod without permission to delete it ErrUnauthorized
StopJob, when its context ends while it waits for the job's objects to go ErrUnreachable

ErrPodsRemain is a failed call to retry, not a refusal: nothing was written and the hold stays in force.

How the supervisor records a Start error

go/dispatch's supervisor records a refused attempt only when Start proves nothing was left running: ErrRejected, ErrInvalid or ErrUnsupported. Any other error is an attempt with an unknown outcome, which it may retry. The rule is in go/dispatch's errors reference. For this adapter:

Start returned When Recorded as
ErrInvalid the request, the spec or an extension is refused before anything is created a refused attempt
ErrUnsupported the adapter is observe-only, an extension is not supported, or the runtime is outside runtimes, missing or breaks a rule; nothing is created a refused attempt
ErrRejected a runtime copy of the name exists and is owned or differs; the apiserver refused the TrainJob's create; or the TrainJob changed on read-back, its runtime copy's owner could not be set, or its Workload did not come, and the TrainJob was stopped a refused attempt
ErrOutcomeUnknown the TrainJob's create went unanswered, so it may exist and Find reports InFlight; or Start could not stop a TrainJob it was refusing an attempt with an unknown outcome
ErrUnreachable the runtime copy's create went unanswered, so no TrainJob was created, though the copy may exist; a read failed; or the Backend is closed an attempt with an unknown outcome; a retried Start reuses an unowned copy whose spec matches
anything else, such as ErrUnauthorized for a create the apiserver forbade an attempt with an unknown outcome