Skip to content

Capabilities and declarations

slurm.New runs preflight and fixes these answers for the life of the Backend. Read them with Declarations() and Capabilities(ctx, spec). What each capability means to go/dispatch is in Capabilities and support; what to do about an answer is in Read a preflight result.

Declarations

Declarations() returns:

Field Value Meaning
NodeLossWindow 2 × SlurmdTimeout + 60s how long dispatch lets an object report running with cause node-lost before it treats the execution as ended. Slurm needs about SlurmdTimeout to mark a silent node down
CallBound see Choose deadlines, TTL and skew how long after its deadline a hold, release or requeue call may still land. 0s when the cluster's AuthType is neither auth/munge nor auth/slurm
PolledLoss[state], [cause], [exit-code] MinJobAge how long a finished job stays in squeue. Poll more slowly than this and you can miss how a job ended
PolledLoss[execution] model.Unbounded if AccountingHistory is yes, otherwise 0 whether a requeue's earlier execution can still be read once it has been replaced

SlurmdTimeout, MinJobAge and MessageTimeout are read from scontrol show config on every slurm.New.

NodeLossWindow was chosen from timings measured on sind, not derived from Slurm's behaviour, and is a bound on a cluster where Slurm fails to end the job itself.

model.Unbounded is -1ns and means "never lost", not a length. The supervisor refuses a poll interval longer than the shortest bounded window, because the adapter implements no Watcher.

A read-only backend with no cached probe has AccountingHistory unknown, so its execution window is 0, and works out CallBound with a skew of 0.

Capabilities

Capabilities gives the same answers whatever spec it is passed, and never returns an error.

Capability Answer Source Reason or what changes it
ContainerImage no declared container images are not mapped in v1. A spec with the dispatch.Image extension is refused
NativeRequeue no declared jobs are submitted with --no-requeue, so Slurm never restarts one on its own
NativeRestart no declared the same
Hold yes, no or unknown probed the preflight probe. See Hold
PreTerminationSignal yes declared SIGTERM, then SIGKILL after KillWait <the cluster's value>
AccountingHistory yes, no or unknown probed the preflight probe. See AccountingHistory
AttemptInjection yes declared the batch script exports the DISPATCH_* variables
ReadmissionGate no declared under --no-requeue Slurm never starts an execution on its own authority
WorkingDirectory yes declared --chdir
ProcsPerNode yes declared --ntasks-per-node
OutputPath yes declared the dispatch.Output extension; see Output files
OutputRead yes declared stdout and stderr, each execution's file in turn
OutputFollow yes declared polled every two seconds
OutputRetained yes declared the files stay in output.dir on the cluster

A reason in code font is the Support.Reason text. A cached probe answer keeps probed as its source.

Hold

Hold is yes only if the probe could place a hold, open the gate and see the hold survive, refuse a release naming a stale hold instance, and leave the hold unchanged by that refusal. See Holds as a far-future start time.

Value Reason Means
yes none holds work as the protocol needs
no AuthType "<type>" has no known landing bound the cluster authenticates with something the adapter has no measured bound for. The probe does not try a hold. Requeue and RestoreRequeueBit are refused too, and the backend dispatch.Open returns implements neither Holder nor Requeuer
no begins placing hold 1 on the probe:, hold 1 did not decode from the probe's start time:, opening the held probe:, the hold did not survive the gate's release: or a stale release changed the hold:, or is slurm: the preflight probe failed: a release naming instance 2 was not refused the step that failed, with the error behind it
unknown read-only, and no cached probe a read-only backend runs no probe

With Hold not yes, Hold and Release return dispatch.ErrUnsupported with the reason.

AccountingHistory

AccountingHistory says whether sacct can describe a job that Slurm has already forgotten. The probe polls sacct for the stopped probe job, every two seconds, for up to MinJobAge.

Value Reason Means
yes comments are kept sacct returned the probe, comment included
yes comments are not kept: AccountingStoreFlags lacks job_comment sacct works, but without the comment a finished job's submission and attempt numbers cannot be read back from it
no sacct's stderr, which contains Slurm accounting storage is disabled or Problem talking to the database this user has no accounting
unknown sacct: <error>, sacct exited <n>: <stderr>, or no sacct row for probe job <id> within MinJobAge it could not be established
unknown read-only, and no cached probe a read-only backend runs no probe

Every probed reason ends with ; AccountingStoreFlags=<the cluster's value>.

Only yes lets Inspect and Ceased fall back to sacct for a job that has left squeue. Otherwise such a job is dispatch.ErrNotFound.

Rules

For each rule in a recovery policy, Rules says whether its action can be carried out here:

Action Answer Source Reason
Resubmit, Escalate yes declared
Restart, Requeue no declared a job that has ended cannot be restarted or requeued under --no-requeue
Hold the Hold capability's answer probed the Hold capability's reason
any other unknown declared action <name>

Requeue as a recovery action is no even though the backend implements dispatch.Requeuer. That interface serves an operator's Requeue command through go/dispatch, not a recovery rule.

Optional interfaces

*slurm.Backend implements dispatch.Backend and these optional interfaces:

Interface Notes
Holder Hold and Release, when Hold is yes
Requeuer Requeue and RestoreRequeueBit, when the landing bound is known
CessationSignal Ceased, when a cessation command is configured. It works on a read-only backend
OutputReader Output, per submission; see Output files
MultiFinder FindMany: one squeue for many jobs

It does not implement Watcher or NativePolicyCompiler.