How identity works¶

Slurm gives every job a number, and numbers get reused. dispatch has its own idea of a job, which can be submitted several times. The adapter has to connect the two so that it never acts on the wrong Slurm job: a stale scancel or a late hold landing on somebody else's work would be worse than doing nothing. This page explains how it does that.

Three levels of name¶

The name¶

Every Slurm job belonging to a dispatch job has the same name: dj- followed by the job's 26-character upper-case ULID. For example dj-01JZ8Q2N5V0000000000000000. It is what model.JobID.DiscoveryName() returns, and the adapter sets it with sbatch --job-name.

The name is how Find looks a job up. It asks Slurm for your jobs (squeue --me) with that name, in any state. squeue matches names without regard to case and splits a list on commas, so the adapter repeats the match itself, byte for byte, and keeps only exact matches. Find sees only your own jobs.

Because the name is the same for every submission of the job, it cannot tell one Slurm job of a dispatch job from another. The comment and the id do that.

The comment¶

--comment=dispatch:s=<N>;a=<attempt> records the submission number N and the attempt number. The adapter reads them back with a strict parse: exactly that shape, with canonical decimals (no leading zeros). A Slurm job named dj-... whose comment does not parse is reported as foreign, never dropped and never guessed at. How Slurm states become observations has the cases.

Slurm's accounting keeps the comment only if the cluster has job_comment in AccountingStoreFlags. Without it, a finished job read from sacct has its submission and attempt unknown, not foreign. Preflight tells you which you have; see AccountingHistory.

The object, and Submit as its anchor¶

A model.ObjectRef has four parts, and the adapter fills them like this:

Submit is the anchor. It is the part that survives what the other two cannot: Slurm can hand an old job id to a new job once its counter wraps, and a later submission of the same dispatch job carries the same name.

The identity check¶

Before every write, the adapter reads the job with that id from squeue and compares it with the ObjectRef you hold. It is the same object only if all of these hold:

1. the job id is the same;
2. the job's name equals UID exactly;
3. the job's submit time equals the ref's Submit, or, if the job has been requeued by Slurm (its restart count is above zero), is the same or later.

The third rule bends for requeues because a requeue moves the submit time forward and never back. Submit stays the time recorded when the object was created, and the check accepts what has moved on from it.

If the check fails, the call acts on nothing. Stop, which cancels by name and id together, finds nothing to cancel and succeeds. Operations that need the object, such as Open, Hold, Release and Requeue, return dispatch.ErrNotFound. Stop also checks again afterwards that the job has ended.

What else rides on Submit¶

A hold is a start time, and the start time is computed from the job's submit time. The adapter decodes it against the job's current submit time. A hold placed on a job that is later requeued, or on a reused id, therefore does not decode to the instance that was placed. See Holds as a far-future start time.

Identity inside the job¶

The same ideas continue inside the running job. The batch script exports dispatch's identity variables before it runs your command, and DISPATCH_ATTEMPT is built from the same parts as the object: the submission number, the Slurm job id, and Slurm's restart count. The restart count is read when the script starts, so every execution of one Slurm job gets a larger DISPATCH_EPOCH than the last, even though Slurm reuses the job id for it. How a job becomes an sbatch submission has the values, and the execution environment says what they mean to a job.

Why it is built this way¶

Each part answers a failure that would otherwise be silent:

• A name alone would match every submission of the job.
• An id alone could match somebody else's job after reuse.
• Neither can tell you the job was requeued underneath you; the submit time can.

The adapter would sooner report that it cannot find the object than act on something that merely looks like it.