How Slurm states become observations¶

Find, FindMany and Inspect turn each Slurm job into a dispatch.ObjectObs: an object state, a cause with a kind, a provenance and a detail, and the facts around them. This page is the table the adapter applies. The state and cause names are go/dispatch's text forms, such as gated and node-lost.

A job is read from squeue or scontrol show job JSON while Slurm still holds it. Inspect reads it from sacct instead once it has left squeue, when AccountingHistory is yes; see From sacct.

Pending jobs¶

A job whose base state is PENDING is classified by its reason, state_reason, and its restart count, restart_cnt. The first row that matches applies.

JobHeldUser with no restart is the gate that sbatch --hold closes. After a requeue Slurm holds the job with the same reason, which is why the restart count tells the two apart.

A start time decodes as a hold when start − submit − 315,576,000 seconds is between 1 and 4,294,967,295; Holds as a far-future start time explains the encoding. The reason None counts with BeginTime because Slurm reports None for about two seconds after the gate is opened, until its next scheduling pass.

HoldInstance is filled from the start time for every pending job, whatever its reason, so a gated job that carries a hold shows both.

Running jobs¶

Ended jobs¶

These rows apply to squeue and to sacct alike.

A non-zero-exit cause carries the job's return code as ExitCode when Slurm set one. No other cause carries an exit code.

The adapter never reports the causes no-capacity, held, released or duplicate.

Unknown states and flags¶

Slurm prints job_state as a list: one base state and any flags. The adapter knows the twelve base states above and the flags of Slurm's data_parser/v0.0.44.

COMPLETING and CONFIGURING change nothing: a CANCELLED,COMPLETING job is ended with cause cancelled. LAUNCH_FAILED matters only under FAILED.

An end time still ahead says nothing, because for a live job it is only Slurm's expected end.

A node list in a form the adapter does not recognise makes the call fail with an error that has no sentinel. It is never classified without its nodes.

From sacct¶

Inspect reads a job from sacct only when the job id is no longer in squeue and AccountingHistory is yes. It uses the ended rows, with these differences:

• sacct has no preempt_time, so a PREEMPTED row is always preempted.
• The launch-failure test reads only the reason JobLaunchFailure; sacct prints no flags.
• A non-zero-exit cause always carries ExitCode, from the return code in sacct's ExitCode field, 0 included.
• A state with no row in the table is ended, cause unknown, with the state and what follows it as the detail. The row is gone from slurmctld, so the job has ended; what it ended as is not known.
• A row that is still PENDING, RUNNING, SUSPENDED, REQUEUED or RESIZING is refused with an error that has no sentinel. sacct cannot tell gated, held and pending apart, so the adapter does not guess.

A sacct observation carries no restart count, no execution token, no hold instance and no requeue bit.

Foreign objects, submission and attempt¶

The adapter reads dispatch:s=<N>;a=<attempt> from the job's comment, with canonical decimals.

A foreign object is still reported, with its state and cause. Only objects named dj-<ULID> are found at all; see How identity works.

The other facts¶

Related¶

• How identity works
• Holds as a far-future start time
• Capabilities and declarations