Skip to content

Holds as a far-future start time

dispatch can hold a job: keep it from starting for a while, for example until a checkpoint is published, and later release it. A hold has a number, its instance, so that a release meant for one hold cannot undo a newer one. Slurm has no such thing, so the adapter builds it from a job's start time.

Two things that look like holds

A job the adapter submits is already held when it is created. sbatch --hold makes it pending with the reason JobHeldUser and priority 0. That is the gate. It stops the job from running until dispatch has recorded it and calls Open, which is scontrol release. The gate belongs to the launch protocol, not to the recovery policy.

A hold is what a recovery rule asks for. If the adapter used the gate for it, Open would lift the hold, and a hold could not be told apart from a launch waiting to be opened. So the adapter holds a different way, and the two are independent: a job can be gated, held, both or neither.

The encoding

A hold sets the job's earliest start time:

StartTime = submit time + 315,576,000 s + instance s

315,576,000 seconds is ten Julian years (model.HoldBase). instance is the hold's number, at least 1. The adapter sends it as

scontrol update JobId=<id> StartTime=<YYYY-MM-DDTHH:MM:SS>

in UTC. A job that cannot start for ten years stays pending without further help.

To read a hold, the adapter takes a pending job's start time, subtracts its submit time and the hold base, and checks the remainder. A value from 1 to 4,294,967,295 (2^32 − 1) is a hold of that instance. Anything else is no dispatch hold: instance 0. The result is shown as the object's HoldInstance, and a pending job with a hold and the reason BeginTime is reported as held.

Two consequences follow from using the submit time:

  • The hold is bound to the job as it was submitted. If Slurm requeues the job, its submit time moves and the hold no longer decodes, so a hold does not carry over a requeue.
  • A start time at or past the hold base that does not decode is reported as pending with a note in the object's cause detail, not as a hold.

Held and gated together

While the gate is closed, Slurm reports the reason JobHeldUser, so the object reads as gated even if a hold is placed on it. The hold is still there: its instance decodes from the start time at once, and it survives Open. Once the gate is open, the reason becomes BeginTime, but only at the scheduler's next pass. For about two seconds after scontrol release it reads None (measured on sind 25.11.8). The adapter reads an encoded start time as held in that window as well, and its preflight probe waits for the pass before it judges the hold.

Release

Release sets StartTime=now, and only if the hold in force, decoded against the job's current submit time, is the instance you name. A release for any other instance is refused as dispatch.ErrRejected and changes nothing. The probe checks this: it places hold 1 and then releases instance 2, which must fail and leave hold 1 in place.

What is refused

Because a hold is decoded against the submit time, the adapter refuses to hold a job that is completing: its submit time is about to change, and the hold with it. It also refuses to requeue a running job on which Slurm still shows a dormant hold, because the requeue would bring the job back held. Errors and error mapping lists every refusal and the error each returns.

Why every hold-family call has a deadline

A hold is placed by scontrol on the Slurm host, possibly through a flaky connection. A call that the caller has given up on can still arrive later and place a hold nobody wants, or release one somebody does. Each call therefore carries a deadline, and the host itself refuses to start it after the deadline. The deadline wrapper is that mechanism.