The deadline wrapper and the landing bound¶

A hold, a release and a requeue each change a job that may be running or waiting right now. They are also the calls most likely to be retried: a supervisor that gets no answer decides again. If the first call is only slow, and lands after the decision, it undoes what the second did. The landing bound on this page is what go/dispatch calls CallBound, the value the adapter declares in Declarations().CallBound.

The adapter prevents a late landing with two things: a deadline wrapper that stops a late call from starting, and the landing bound, which says when a call that did start can no longer take effect.

The wrapper¶

Each of these calls is sent as one short shell script, and the script does the checking on the Slurm host. This is the script for a hold, as the adapter's tests record it (the numbers vary):

trap '' PIPE HUP
_dsl_s=75
if [ "$(date +%s)" -le 1791797399 ]; then
  TZ=UTC0 timeout -s KILL 25s scontrol update JobId=4242 StartTime=2036-10-11T20:00:07 </dev/null
  _dsl_s=$?
fi
printf '\036DSL 0123456789abcdef0123456789abcdef call 1 %d\n' "$_dsl_s" >&2
exit "$_dsl_s"

In order, it:

1. Ignores hangup and broken pipe. If the SSH session drops, the shell is hung up and what read its stderr is closed. Either would kill the script between two calls, and a requeue's last step would never run. scontrol inherits the ignored signals, so a call in flight completes.
2. Starts with status 75. If the check refuses, 75 is what is reported and nothing runs.
3. Checks the deadline on the host. date +%s is the host's clock, truncated to whole seconds. The script goes ahead only while that is no later than the deadline's second minus one, which means only while the host's clock is before the deadline.
4. Runs scontrol under timeout -s KILL, so a hung scontrol is killed after the bound B rather than waiting for ever. TZ=UTC0 makes the start time mean UTC.
5. Reports the status as a framed line on stderr, and exits with it.

The check is on the host for a reason: the caller's clock tells you when the caller sent a call, not when it began to run. Only the machine that runs scontrol knows whether it is still in time. The script is delivered whole, and over ssh carried base64 encoded, so a connection that drops partway cannot run half of it.

The script is also detached: it runs to its own bound even if the context you passed is cancelled. Abandoning a call that is already on the host would turn a call that may have landed into one whose outcome you cannot know.

The landing bound¶

Passing the check means the call started in time. It does not mean it has finished. scontrol sends its request to slurmctld and can be killed at B while the request is still on its way, or waiting in the controller's socket. When the controller reads it, it applies it, unless the authentication credential in the request has expired. Four things decide how late that can be:

Over SSH there are two more, because the deadline is in your clock and the check is in the host's: the offset between the two clocks, and half the round trip of reading the host's clock.

Their sum is Declarations().CallBound. The core treats a call as expired at its deadline plus the call bound: past that point the call either landed or never will, and it is safe to decide again. Choose deadlines, TTL and skew has the formula and what to change.

The TTL was measured, not assumed. With slurmctld stopped, every update younger than munge's TTL was applied when it resumed, and every older one was refused as Munge decode failed: Expired credential; the cut-off fell within 5 seconds either side of 300 seconds. That was on 25.11.8 and 26.05.4 with munge. For auth/slurm the 60-second TTL comes from Slurm's source and has not been measured.

The bound is tested, not just reasoned. TestLandingBound in the sind suite freezes slurmctld between a call's lookup and its scontrol, so the request waits in the socket while timeout kills the client, then lets the controller run. A call older than the TTL must change nothing; a younger one may land, but before its deadline plus CallBound. See Run the conformance suite against sind.

Why a requeue is three calls¶

Jobs are submitted with --no-requeue, so that Slurm never restarts one on its own authority: every start of a job is dispatch's decision, with a new identity. To requeue one anyway, the adapter sets the job's requeue bit for the length of the call. So Requeue is one script with three calls, each behind the same deadline check:

1. scontrol update JobId=<id> Requeue=1
2. scontrol requeue <id>, run only if call 1 succeeded
3. scontrol update JobId=<id> Requeue=0, run whenever call 1 was attempted at all, whatever call 2 returned

Call 3 runs even when call 1's result was lost, because the bit may have been set without your seeing it. A script that is cut off after call 1, as a crash of the caller would, leaves the bit set. That is visible: the object reports RequeueBit, and RestoreRequeueBit (one wrapped Requeue=0) clears it. A requeue whose last step cannot be shown to have cleared the bit returns an error saying the bit may be left set, so the caller knows to restore it. It is never reported as a clean refusal while the bit may still be set.

How the answers are read¶

Each call prints one status line to stderr: a record separator, DSL, the script's random nonce, the call's number and its status. The nonce means noise from a login script, or from scontrol itself, can't be mistaken for a status. The reader refuses a line without the script's nonce, calls out of order, text after the last status line, and a script exit status that is not the last call's. Any of these is an outcome unknown, never a guess.

A failed scontrol counts as a refusal, meaning slurmctld decided and changed nothing, only if every line it printed is <message> for job <id> with a message the adapter knows. Anything else leaves the outcome unknown. Exit codes of the deadline wrapper lists the statuses and the messages.