Choose deadlines, TTL and skew¶

Hold, release, requeue and restore calls carry a deadline. The adapter declares a call bound: how long after its deadline such a call may still take effect. Together they let a supervisor know when a call can no longer land, and so when it is safe to decide again. This page is about the numbers you can set. The deadline wrapper explains why they matter.

What you control and what you don't¶

Number	Who sets it	Where
the deadline	the caller of `Hold`, `Release`, `Requeue` and `RestoreRequeueBit`	`CallMeta.Deadline`. A supervisor sets it 30 seconds ahead of now (`dispatch.CallDeadline`)
`MessageTimeout`	you, as the Slurm administrator	`slurm.conf`
the credential TTL	you, as the Slurm administrator	`AuthInfo=ttl=<seconds>` in `slurm.conf`; otherwise munge's own lifetime
clock skew, controller to login node	measured by preflight	not set; keep your clocks in step
clock offset, you to login node	measured by preflight, with `transport: ssh` only	not set; keep your clocks in step
`timeouts.command`, `ssh.connectTimeout`, `preflight.maxAge`	you, in the adapter's configuration	Configuration

How the call bound is made¶

CallBound = B + TTL + |skew| + 5s + |ssh offset| + ssh round trip / 2
B         = 2 × MessageTimeout + 5s, rounded up to a whole second

Term	Is	Comes from
`B`	how long the wrapper lets `scontrol` run before it kills it. `scontrol` may retry inside that bound, each try with a fresh credential	`MessageTimeout` from `scontrol show config`
`TTL`	how long the authentication credential in a request stays valid	`AuthInfo` `ttl=` if the cluster sets it; otherwise the TTL printed by `munge -n \\| unmunge` for `auth/munge`; otherwise 60 seconds for `auth/slurm`
`skew`	the difference between the login node's clock, where munge encodes the request, and `slurmctld`'s, where it is decoded	the probe: the larger distance between the controller's submit time and the login node's clock read before and after, plus one second
`5s`	a fixed margin for `slurmctld` applying a request it has decoded, seen at 0.3 to 0.8 seconds	the adapter
`ssh offset` and `round trip`	the difference between your clock and the login node's, and the time a message takes	preflight, with `transport: ssh` only. Zero on the local transport and on a custom transport

The AuthInfo value, when set, wins over munge's live value. It must be a positive whole number of seconds; anything else fails preflight.

If the cluster's AuthType is neither auth/munge nor auth/slurm, there is no known bound. CallBound is 0s, Hold is no, and requeue is refused. Through dispatch.Open or dispatch.NewBackend the backend then offers no Holder or Requeuer at all, because dispatch allows a zero CallBound only without them; see Build a Backend for a supervisor and Capabilities and declarations.

A worked case, taken from the adapter's own tests: MessageTimeout 10 seconds gives B of 25 seconds; munge's TTL is 300 seconds; the probe measured 2 seconds of skew. On the local transport the call bound is 25 + 300 + 2 + 5 = 332 seconds. The TTL is nearly all of it.

Make the call bound shorter¶

The TTL dominates on a default munge cluster. To shorten it, lower ttl= in AuthInfo. Lowering MessageTimeout shortens B by twice the change. Neither is the adapter's to tune; they are your cluster's settings and affect everything else on it.

The probe has to see the new values, and so does anything cached: scontrol show config is part of the cache's hash, so a changed AuthInfo or MessageTimeout makes the next slurm.New probe again.

Choose a deadline¶

A call is expired if the clock on the login node, when the command starts, has passed the deadline. It then does nothing and returns slurm.ErrExpired, which is a dispatch.ErrRejected. So:

Give it room for the trip. The deadline has to outlast opening an SSH connection and running the script's first lines. The adapter refuses only a call with no deadline (ErrInvalid); a deadline that is near, or already past on your clock, is sent anyway, and the host refuses it only if its own clock has passed the deadline when the script starts (status 75, slurm.ErrExpired). 30 seconds, as a supervisor uses, suits a connection that opens in a second or two.
Do not make it long. Once the deadline plus CallBound has passed, you know the call either landed or never will. A far deadline pushes that moment out for nothing.
A deadline is checked against the login node's clock, not yours. The adapter does not shift it by the ssh offset. The offset only widens CallBound. If your clock runs ahead of the login node's, calls land later than you think; if behind, they expire early. Run NTP, or chrony, on both.

The call is detached from the context you pass: cancelling the context does not abandon a call that is already on the host, because turning a landed call into an unknown one is worse than waiting. The script's own bound ends it: B + 1s + ssh.connectTimeout for one scontrol call, and three times B + 1s plus the connect timeout for a requeue.

Choose how stale the skew may be¶

On a managed cluster with NTP the drift stays far below the 5-second margin. Skew is measured once, by the probe, and kept in the cache with the hold answer. A cached result is reused until preflight.maxAge (24 hours by default). If the clocks on your cluster drift, a 24-hour-old skew understates the truth. Lower preflight.maxAge, at the price of a probe job every time it runs out. Or leave preflight.cacheDir unset, and every slurm.New probes afresh.

Other bounds you may set¶

Key	Default	Bounds
`timeouts.command`	2 minutes	every ordinary script: reads, `sbatch`, `scancel`. Not the hold-family and requeue scripts, which carry their own bound
`ssh.connectTimeout`	10 seconds	opening the connection. It is also added to the bound of each hold, release, requeue and cessation script
`cessation.timeout`	60 seconds	your cessation command, which is killed at the limit