Errors and sentinels¶

Sentinel	Message prefix	Means
`dispatch.ErrNotFound`	`dispatch: not found`	no such job, adapter or object
`dispatch.ErrUnsupported`	`dispatch: unsupported`	this cluster or adapter can't do it
`dispatch.ErrInvalid`	`dispatch: invalid`	the request is malformed, or not allowed in the job's current state
`dispatch.ErrRejected`	`dispatch: rejected`	refused: nobody would enforce a rule, an adapter's answer is incoherent, the scheduler said no
`dispatch.ErrUnauthorized`	`dispatch: unauthorized`	this actor may not do it
`dispatch.ErrUnreachable`	`dispatch: unreachable`	the scheduler can't be reached
`dispatch.ErrReadOnly`	`dispatch: read-only`	a write on a read-only backend
`dispatch.ErrUnavailable`	`dispatch: record unavailable`	the record store can't be reached, or a commit couldn't get through before the context ended
`dispatch.ErrOutcomeUnknown`	`dispatch: outcome unknown`	a backend call whose effect can't be determined
`dispatch.ErrAlreadyRegistered`	`dispatch: already registered`	a second registration of an adapter or extension

Test with errors.Is. A sentinel classifies an error and is added to its chain, so the error beneath it (a store's, a scheduler's) survives. The dispatch.Err* values are the same values as model.Err*; the root package re-exports them. model.Invalidf builds an error classified as ErrInvalid.

Validation errors¶

JobSpec.Validate, RecoveryPolicy.Validate and Extensions.Validate report every refusal they find, not only the first, joined with errors.Join. Each refusal is a *dispatch.ValidationError (the same type as *model.ValidationError) and is classified as ErrInvalid, so errors.Is(err, dispatch.ErrInvalid) holds for the joined error and for each refusal in it.

Field	Holds
`Path`	a JSON Pointer (RFC 6901) to the offending value, over the JSON field names of the record schema
`Msg`	what is wrong with it

JobSpec.Validate paths start at the spec: /nodes, /command/0, /env/FOO, /recovery/rules/0/for, /recovery/rules/1/on/0, /extensions/1. RecoveryPolicy.Validate paths start at the policy (/rules/0/for), and Extensions.Validate paths at the list (/1). An env key containing / or ~ is escaped as RFC 6901 says (~1, ~0).

A refusal prints as dispatch: invalid: <path>: <msg>, for example dispatch: invalid: /recovery/rules/0/for: Hold needs a positive For; the joined error prints one refusal per line. errors.As(err, &v) on the joined error finds the first refusal. To read every one:

if joined, ok := err.(interface{ Unwrap() []error }); ok {
    for _, e := range joined.Unwrap() {
        var v *dispatch.ValidationError
        if errors.As(e, &v) {
            fmt.Println(v.Path, v.Msg)
        }
    }
}

Other packages' sentinels¶

Sentinel	Means
`records.ErrConflict`	the record's revision moved since it was read; read again and decide again
`records.ErrNotOwner`	a commit carried a lease that's no longer the current owner's
`records.ErrOwned`	`Acquire` found the record owned by another holder
`records.ErrNotFound`	no such record; wraps `dispatch.ErrNotFound`
`records.ErrSchema`	an entry or header whose schema is unknown or whose body doesn't decode strictly
`records.ErrUnknownName`	an enum name in a record entry this build doesn't know
`supervise.ErrDisplaced`	the supervisor lost ownership of the job, and stopped
`checkpoint.ErrMalformed`	an entry name, epoch, step or framework marker that doesn't parse
`checkpoint.ErrInvalid`	an argument the operation refuses
`checkpoint.ErrExists`	the publication's entry already exists; it wasn't touched
`memory.ErrFinished`	`Open` of an object that already ended; wraps `ErrRejected`
`memory.ErrExpired`	a hold, release or requeue call that started past its deadline; wraps `ErrRejected`
`memory.ErrStaleHold`	a release naming a hold instance other than the one in force; wraps `ErrRejected`
`memory.ErrNotApplicable`	an `Env` action the object's state or the shape doesn't allow; wraps `ErrRejected`
`memory.ErrCrashed`	a `Requeue` ended between its steps by the harness's `WithCrashBetweenSteps`; wraps `ErrOutcomeUnknown`
`creds.ErrNoAnswer`	a credential source whose location is absent from this node; the next source is tried
`creds.ErrRefused`	a credential source that is present but refused, for its content or its permissions; resolving stops
`creds.ErrNoSource`	no credential source in the list answered
`creds.ErrInvalid`	a source list or location that doesn't parse
`proxy.ErrInvalid`	a proxy URL that isn't `http(s)://host[:port]`, or carries userinfo, a path, a query or a fragment
`s3.ErrNoConditionalWrites`	the store overwrote an object that a put with `If-None-Match: *` should have left alone
`s3.ErrInvalid`	an argument or configuration the S3 store refuses
`s3.ErrProtocol`	an answer from the store that breaks the S3 protocol

creds, proxy and s3 are checkpoint/creds, checkpoint/proxy and checkpoint/s3, the packages under cmd/publish.

Each adapter defines its own sentinels as well, classified by the ones above: slurm.ErrExpired and kubeflow.ErrStaleHold wrap ErrRejected, and kubeflow.ErrPodsRemain wraps ErrUnreachable. Their pages list the rest, and which scheduler answers map to which sentinel: Slurm's errors and Kubeflow's error mapping.

What each call returns¶

Registry and Open¶

Call	Returns
`dispatch.Register`	`ErrInvalid` for an empty name or nil factory; `ErrAlreadyRegistered` for a name already taken
`model.RegisterExtension`	`ErrInvalid` for a nil, unnamed or pointer prototype; `ErrAlreadyRegistered` for a namespace and name already taken
`dispatch.Open`	`ErrNotFound` when no adapter is registered under the name; `ErrInvalid` when `Config.Adapter` doesn't encode as JSON; otherwise whatever the factory returns

Observer¶

Call	Returns
`Get`	the backend's `Find` error; with a store, the store's error and `ErrInvalid` from a record that doesn't fold
`List`	`ErrUnsupported` without a store; otherwise as `Get`, for each job

Operator¶

Every Operator call returns ErrInvalid for an actor with no kind or no id. Store errors reach you as ErrUnavailable, ErrNotFound, ErrInvalid, records.ErrConflict or records.ErrNotOwner; any other store error is wrapped in ErrUnavailable. Control intents are committed with records.CommitControl, which retries a revision conflict until the context ends, then returns ErrUnavailable wrapping the context's error.

Call	Refuses with
`Submit`	`ErrInvalid` for an invalid actor, or a negative `WithSubmitRetries`, `WithCleanupDeadline` or `WithUnresolvedAge`; everything `MakePlan` refuses; `ErrRejected` when the idempotency key names a job with a different spec
`Cancel`	the backend's `StopJob` error, returned with a receipt whose `Committed` is true: the cancel is recorded even if the stop failed
`Hold`	`ErrInvalid` for a deadline not in the future, or a job that's cancelled or finished
`Release`	`ErrInvalid` when the job isn't held, the hold is a recovery hold, or the job is cancelled or finished
`Resubmit`	`ErrInvalid` unless a recovery hold is in force and the job isn't cancelled or finished
`Supersede`	`ErrInvalid` unless the job is finished; then as `Submit`
`GiveUp`	nothing of its own; on a finished job it records the intent and changes nothing
`Abandon`	`ErrUnauthorized` unless the actor is a human; `ErrInvalid` for a target naming neither or both of a submission and a foreign object, or a negative attempt; `ErrInvalid` for a submission or attempt no escalation names yet, or a foreign target the record doesn't hold as a foreign object
`Requeue`	`ErrUnsupported` when the adapter has no `Requeuer`; `ErrInvalid` for a job that's cancelled or finished

Stopper¶

Call	Returns
`EmergencyStop`	`ErrInvalid` for an invalid actor; otherwise the backend's `StopJob` error, with its report

Supervisor¶

Call	Returns
`supervise.New`	`ErrInvalid` for a nil backend or store, a drain or poll that isn't positive, or a poll slower than an evidence window the adapter declares (when it has no `Watcher`)
`Supervisor.Run`	`nil` once the record settles; the context's error once it ends; `records.ErrOwned` if another supervisor holds the job; `ErrDisplaced` if ownership is lost; `records.ErrNotFound` when the store has no such record; an error wrapping `ErrInvalid` when the record doesn't fold or a step can't be executed
`Supervisor.Reconcile`	the store's `List` error; otherwise every unsettled record is visited, and the errors are joined, one per job: an `Acquire` error other than `ErrOwned`, or a pass's error

Run retries every other error at its poll interval: a backend Find that fails, a store read that fails, a commit whose reply was lost.

How the supervisor reads an adapter's errors¶

From	Error	Recorded as
`Start`	`ErrRejected`, `ErrInvalid`, `ErrUnsupported`	a refused attempt
`Start`	anything else, `ErrOutcomeUnknown` included	an attempt with an unknown outcome, which may be retried
`Hold`	`ErrRejected`, `ErrUnsupported`	the hold refused; the record stops holding
`Open`, `Stop`, `Release`, `Requeue`, `RestoreRequeueBit`	any	nothing; the next pass's `Find` shows what happened
`Ceased`	any	a `cessation` fact that says not ceased, with the error's text as its detail
`StopJob`, `Find`	any	the pass ends and is retried

Each adapter's error page says which of its Start errors are refusals and which leave the outcome unknown: Slurm and Kubeflow.