Errors and error mapping¶
The adapter returns go/dispatch's sentinels, wrapped with detail. Test with errors.Is. The
sentinels are re-exported by dispatch and defined in model; their meaning to the core is in
go/dispatch's error reference.
The distinction that matters most is between three answers to "did it happen?":
| Sentinel | The adapter knows | The core does |
|---|---|---|
ErrRejected, ErrInvalid, ErrUnsupported |
nothing was created or changed | records a refusal |
ErrOutcomeUnknown |
the call may or may not have taken effect | records it as unknown, and looks |
ErrUnreachable |
the Slurm host could not be read, or (for Open, Stop and StopJob) a write provably never ran |
treats it as a failure to reach the scheduler, not as a statement about the job |
The adapter chooses the first only when it can prove it. A timeout, a dropped connection, an unfamiliar message and a killed command are all unknown.
Exported errors¶
| Value | Is | Wraps |
|---|---|---|
slurm.ErrExpired |
a hold-family or requeue call started after its deadline on the Slurm host, so it did nothing | dispatch.ErrRejected |
ssh.ErrInvalidOption, in transport/ssh |
an ssh option the transport cannot pass safely, from ssh.Options.Validate and ssh.New |
nothing. slurm.New wraps it in dispatch.ErrInvalid when it opens the transport |
slurmtest.ErrUnexpectedScript |
the replaying slurmtest.Executor was sent a script its transcript did not expect next, or ran out of transcript |
nothing |
Everything else is a sentinel from dispatch or model, or one of the transport errors below.
slurm.ErrExpired carries the text "the call started past its deadline".
Transport results¶
The transport reports one of these to the adapter, which translates them below.
| Transport error | Means |
|---|---|
transport.ErrNeverRan |
the script is proven not to have reached the host |
transport.ErrDropped |
the script started, or may have, and its outcome was lost |
transport.ErrInvalidScript |
no transport will run it: over 90 KiB, a NUL byte, a negative bound, or a detached script with no bound |
transport.ErrClosed |
Run after Close. Always wrapped with ErrNeverRan |
Over ssh, ErrNeverRan needs ssh to exit 255, the script's begin marker to be absent, and ssh's log
to hold only lines it recognises as before a session (a refused connection, a name that does not
resolve, Permission denied, a failed host key check, a control path too long) with at least one
of them. An unrecognised line makes it ErrDropped. With the local transport, only the shell
failing to start is ErrNeverRan.
By call¶
Opening a backend: slurm.New¶
| Cause | Error |
|---|---|
| the configuration does not decode or validate | ErrInvalid, naming the key |
the transport is unregistered, or cannot be opened: an ssh value with characters ssh would misread (wrapping ssh.ErrInvalidOption), or a default control directory another user could reach |
ErrInvalid |
a command in the tool list is missing (munge and unmunge too, under auth/munge), or Slurm is not 25.11 or 26.05, or $HOME is not absolute |
ErrUnsupported |
| a read of the cluster fails or exits non-zero | ErrUnreachable |
scontrol show config, Slurm's JSON, the host's clock, unmunge's TTL or an AuthInfo ttl= that is not whole seconds is not understood |
an error with no sentinel, saying what was not understood |
reading environment.pass for the probe failed |
ErrRejected |
the probe's script cannot be built, for example because output.dir holds % or \ |
ErrInvalid |
the probe's submission never reached the host, or sbatch refused it with one of the refusals below |
ErrRejected |
| the probe's submission failed any other way, printed no job id, or its read-back could not run | ErrOutcomeUnknown |
| the cluster changes the probe job | ErrRejected, naming the difference |
| stopping an old probe left by a crashed preflight failed, or stopping this run's probe failed or left it alive | the stop's own error, which may carry no sentinel, prefixed stopping stale probe job or slurm: the preflight probe failed |
| the preflight cache directory is not private | an error for an unsafe directory, with no sentinel |
An error from submitting, reading back or stopping the probe begins
slurm: the preflight probe failed. A failing hold step does not fail
slurm.New: it makes the Hold capability no, with the step as its reason. See
Capabilities and declarations.
Start¶
| Cause | Error | The supervisor records |
|---|---|---|
| read-only backend | ErrReadOnly |
an unknown outcome |
| the request is malformed: a bad job id, a spec that fails validation, a negative attempt, a relative work directory, an unusable output path, a script over 90 KiB | ErrInvalid |
a refused attempt |
| a container image, or an extension from another namespace | ErrUnsupported |
a refused attempt |
reading environment.pass failed |
ErrRejected, which also matches ErrUnreachable |
a refused attempt |
| the script never reached the host | ErrRejected |
a refused attempt |
the transport would not run the script (transport.ErrInvalidScript) |
ErrInvalid |
a refused attempt |
sbatch printed one of the refusals below |
ErrRejected, with sbatch's message |
a refused attempt |
| any other failure, a non-zero exit, a timeout, or output with no job id | ErrOutcomeUnknown |
an unknown outcome |
| read-back could not run | ErrOutcomeUnknown, naming the job id |
an unknown outcome |
| read-back found the job differs from the submission, and it was stopped | ErrRejected, naming the difference |
a refused attempt |
| read-back found a difference, and stopping the job failed | ErrOutcomeUnknown |
an unknown outcome |
Every refusal of the request itself comes before any command runs.
The last column is what go/dispatch's supervisor makes of the reply. A refused attempt is final: no
object exists. An unknown outcome may have created a job, so the supervisor looks for it with
Find and may retry the attempt. See
how the supervisor reads an adapter's errors.
sbatch refusals. Only these messages, as the sole line of stderr after
sbatch: error: Batch job submission failed:, are ErrRejected:
- Invalid partition name specified
- Access/permission denied
- User's group not permitted to use this partition
- More processors requested than permitted
- Node count specification invalid
- Requested node configuration is not available
- Requested partition configuration not available now
- Invalid node name specified
- Invalid feature specification
- Memory required by task is not available
- Invalid account or account/partition combination specified
- Invalid qos specification
- Job violates accounting/QOS policy (job submit limit, user's size and/or time limits)
- Requested time limit is invalid (missing or exceeds some limit)
- Access denied to requested reservation
- Requested reservation is invalid
- Required partition not available (inactive or drain)
- Invalid generic resource (gres) specification
- Pathname of a file, directory or other parameter too long
These are messages slurmctld returns only after deciding not to create the job. A socket timeout
is not among them, because it may have created one.
Read-back compares the job with what was asked: its name, its comment, that requeue is off, that only the owner's hold is on it, and its dependency. It does not compare partition, account or resources, because a site routing plugin may legitimately change them.
The identity lookup¶
Every call that names an object (Open, Stop, Hold, Release, Requeue,
RestoreRequeueBit, Ceased and Inspect) first reads the job with that id from squeue, as
How identity works describes. That lookup fails
the same way for each of them:
| Cause | Error |
|---|---|
| the ref has no decimal job id, or no name | ErrInvalid, for Stop as for the rest |
squeue failed, exited non-zero for any reason but an unknown id, or timed out |
ErrUnreachable |
squeue's JSON is not understood |
an error with no sentinel |
The tables below start after the lookup.
Find, FindMany, Inspect¶
| Cause | Error |
|---|---|
| a malformed job id or ref | ErrInvalid |
| the command failed, exited non-zero, or timed out | ErrUnreachable |
a job id Slurm no longer holds, from squeue |
not an error: the object is absent |
| Slurm's output is not understood | an error with no sentinel |
Inspect: the id is not in squeue, not in sacct (or accounting is not yes), or belongs to another object |
ErrNotFound |
Inspect asks sacct only when AccountingHistory is yes. On a read-only backend with no cached
probe it is unknown, so a job that has left squeue is ErrNotFound there.
Open¶
| Cause | Error |
|---|---|
| read-only backend | ErrReadOnly |
| the id is gone or belongs to another object | ErrNotFound |
| the job is held by an administrator, or held by Slurm after a requeue, so the gate is not dispatch's | ErrRejected |
| the script never ran | ErrUnreachable |
Job has already finished |
ErrRejected |
Invalid job id specified |
ErrNotFound |
| anything else | ErrOutcomeUnknown |
Stop and StopJob¶
| Cause | Error |
|---|---|
| read-only backend | ErrReadOnly |
| the object is gone, belongs to another job, or has ended | success |
the scancel script never ran |
ErrUnreachable |
| the object is still alive after two checks, one second apart | an error with no sentinel, saying it is still alive |
StopJob: a malformed job id |
ErrInvalid |
Stop does not trust scancel's own result; it asks Slurm whether the object ended. StopJob
cancels every object of the job by name, then checks, and repeats the cancel each round until
nothing is alive and every cancelled job's nodes are up, or until KillWait + SlurmdTimeout + 60s
has passed. It returns a StopReport with the objects stopped, still alive, and unproven (a node
is down, not responding, or not reported). Reaching the bound is not an error: read the report.
Hold, Release, Requeue, RestoreRequeueBit¶
| Cause | Error |
|---|---|
| read-only backend | ErrReadOnly |
Hold or Release with Hold not yes: the cluster failed the hold probe, or has no known landing bound |
ErrUnsupported, with the reason |
Requeue or RestoreRequeueBit where CallBound is unknown, because the AuthType has no known landing bound |
ErrUnsupported. Only a *slurm.Backend from slurm.New can be asked: the backend dispatch.Open returns then has no Holder or Requeuer |
| instance 0, or no deadline | ErrInvalid |
| the id is gone or belongs to another object | ErrNotFound |
Hold: the job is completing, so its submit time, and with it the hold's encoding, is about to change |
ErrRejected |
Release: the hold in force is not the instance named |
ErrRejected |
Requeue: a running job carries a dormant hold, which a requeue would bring back held |
ErrRejected |
| the deadline had passed on the host | ErrExpired (an ErrRejected) |
| the script never reached the host | ErrRejected |
the transport would not run the script (transport.ErrInvalidScript) |
ErrInvalid |
scontrol refused with a known message, such as Job is no longer pending execution for a hold of a job that has started |
ErrRejected |
scontrol was killed, or timeout failed, or anything unrecognised |
ErrOutcomeUnknown |
| a requeue whose bit may be left set | ErrOutcomeUnknown, saying so |
go/dispatch's supervisor treats ErrRejected and ErrUnsupported from Hold as the hold refused,
and stops holding. It records nothing for any error from Open, Stop, Release or Requeue, and
reads what happened from its next Find.
Exit codes of the deadline wrapper has the statuses and messages behind the last four rows.
Ceased¶
| Cause | Error |
|---|---|
no cessation is configured |
ErrUnsupported |
| the job is not known | ErrNotFound |
| exit 0 or 1 | evidence: ceased, or not yet |
| any other exit, a kill, or a transport failure | ErrUnreachable, never evidence |
Output¶
| Cause | Error |
|---|---|
| a bad job id, a submission below 1, or a stream other than stdout and stderr | ErrInvalid |
| the manifest cannot be read | ErrUnreachable if the command failed, ErrNotFound if it exited non-zero |
| a manifest or file listing that is not understood | an error with no sentinel |
Errors with no sentinel¶
Some errors match no dispatch sentinel, so errors.Is cannot classify them. Among them: Slurm
output the adapter does not understand, including scontrol show config and an AuthInfo ttl=
that is not whole seconds; a preflight cache directory that is not private; a job that is still
alive after Stop; and a requeue whose bit is known to be set because its restore certainly did
not run. Spec 0001
gives them no sentinel either. Treat an error from these calls that matches no sentinel as "the
adapter could not complete the call".