Exit codes of the deadline wrapper¶

Hold, release, requeue and restore calls run as deadline-wrapped scripts. Each scontrol call in a script ends with a status, which the script prints as a framed line and which becomes the script's exit status if it is the last call.

Statuses¶

Anything over 255, or not a plain decimal, is an unreadable status and an unknown outcome.

Messages accepted as refusals¶

A failed scontrol is a refusal only if every non-blank line of its stderr is <message> for job <id>, with the job's own id and a message allowed for that call. A call whose output has any other line, or two different messages, is an unknown outcome.

The messages are the ones slurmctld returns after it has decided and changed nothing. Open (scontrol release) is not wrapped, but reads its refusals the same way: Job has already finished is ErrRejected, and Invalid job id specified is dispatch.ErrNotFound. The text of the lines may also arrive with a slurm_suspend error: prefix there.

The status line¶

Every call prints one line to stderr before the next call starts:

<RS>DSL <nonce> call <i> <status>

<RS> is the byte 0x1E. <nonce> is the script's random nonce, in hexadecimal. <i> is the call's number in the script: 1 for a hold, release or restore; 1 to 3 for a requeue. <status> is the statuses above. Text before the line is that call's own stderr.

A call that the check skipped for want of a prior success (the requeue step after a failed set) prints no line. The script's exit status is the last printed call's.

The reader refuses a line with another script's nonce, a number that is not greater than the previous, a status over 255, text after the last line, no line at all, and an exit status that is not the last call's. Each is an unknown outcome.

A requeue's combined result¶

A requeue script has three calls, and its result combines them.

• Step 1 (set the bit) did not succeed. Step 2 does not run. The result is step 1's: ErrExpired for 75, ErrRejected for a refusal. If step 1's outcome is unknown, the result is ErrOutcomeUnknown with a note that the requeue bit may be left set, since the setting may have landed unseen. Step 3 still runs in every case but 75.
• Step 3 (clear the bit) succeeded, or found the job finished. The result is step 2's own: success, ErrRejected for a refusal, ErrExpired, or ErrOutcomeUnknown.
• Step 3 did not clear the bit. The result says the requeue bit may be left set on the job. It is ErrOutcomeUnknown if step 3 ran (any status other than 75, 126 and 127) or if step 2's outcome was unknown. If step 3 certainly did not run, the bit is known to be set, and the error carries no dispatch sentinel. It is never ErrRejected, even when step 2 was refused, because something may have changed.
• The script's output cannot be read, or reports the wrong calls. ErrOutcomeUnknown, with the same note.

A set bit shows as RequeueBit on the object, and RestoreRequeueBit clears it.

Constants¶