Skip to content

Prepare a cluster for the adapter

Work through this list on a cluster before the first slurm.New, or when preflight refuses one. Each item says what preflight reports if it is missing. Some gaps fail slurm.New; others only lower a capability, which you read as described in Read a preflight result.

On the login node

The login node is the host the transport runs scripts on: the machine itself for local, the ssh.host for ssh.

  1. Run Slurm 25.11 or 26.05. Any patch level. Anything else fails with ErrUnsupported: Slurm <version>; supported are 25.11, 26.05.
  2. Put the commands on the PATH that /bin/sh sees, without a login profile doing it interactively: sbatch, squeue, scontrol, scancel, sacct, timeout, date and base64. A missing one fails with ErrUnsupported: missing on the Slurm host: <commands>. The scripts also use mkdir, printf, cat and wc, which preflight assumes.
  3. Use GNU coreutils for timeout, date and base64. The scripts run timeout -s KILL, date +%s, and over ssh date +%s.%N and base64 -d. Preflight checks only that the commands exist; a date that cannot print nanoseconds fails step 6 with an error naming its output.
  4. Give the user an absolute $HOME. Otherwise ErrUnsupported: the cluster home "<value>" is not an absolute path.

Authentication

  1. Use AuthType=auth/munge or auth/slurm. Under any other type there is no known landing bound: Hold is no with the reason AuthType "<type>" has no known landing bound, requeue is refused with ErrUnsupported, and CallBound is 0s. slurm.New still succeeds, and the backend dispatch.Open returns implements neither Holder nor Requeuer.
  2. Under auth/munge, put munge and unmunge on the login node's PATH, unless AuthInfo sets ttl=. Without them, preflight fails with ErrUnsupported, naming them.
  3. If you set AuthInfo=ttl=, use a positive whole number of seconds. Anything else fails preflight. The TTL is most of the call bound; see Choose deadlines, TTL and skew.

Accounting

  1. Keep job accounting the user can read, so sacct can describe a job after it leaves squeue. Without it AccountingHistory is no, and such a job is dispatch.ErrNotFound.
  2. Add job_comment to AccountingStoreFlags. Without it AccountingHistory is still yes, with the reason comments are not kept: AccountingStoreFlags lacks job_comment, and a finished job read from sacct has no submission or attempt number.

What the user may do to their own jobs

  1. Let a user submit held jobs to a partition they can use. The probe is one real one-node submission with --hold, using the configured defaults. If sbatch refuses it, preflight fails with ErrRejected and sbatch's message.
  2. Let the owner set StartTime on, and release, their own pending jobs. That is how a hold is placed and the gate opened. If the probe cannot, Hold is no, and the reason names the step: placing hold 1 on the probe: ... or opening the held probe: ....
  3. Let the owner requeue their own jobs (scontrol update JobId=<id> Requeue=1, then scontrol requeue <id>), if you want requeue. Preflight does not try a requeue, so a cluster that forbids it is found by the first Requeue call. That call fails with ErrRejected only for a message on the adapter's list, and otherwise with ErrOutcomeUnknown; see Exit codes of the deadline wrapper.

job_submit plugins

  1. Leave the adapter's submission as it asked for a job named dj-<ULID>. A job_submit plugin may change partition, account and resources, but not:

    • the job's name or comment;
    • the requeue flag, which must stay off;
    • the owner's hold: pending, reason JobHeldUser, priority 0;
    • the dependency, which must be empty, or singleton with singleton: true.

    A plugin that changes one of these fails preflight with ErrRejected: the cluster changes a submission: <difference>. Without preflight, every Start would be stopped and refused the same way. How a job becomes an sbatch submission has the options the adapter sends.

Output and working directories

  1. Make output.dir the same path on the login node and on every compute node. Start creates the job's output directory on the login node, the batch script on the first allocated node appends each execution's output there, and Output reads it back on the login node. A path that only exists on one side fails each job, not preflight. The default, $HOME/.local/state/dispatch-slurm, assumes a shared home.
  2. Do the same for any WorkDir a job names, and for the cluster home, which is the working directory when a job names none.
  3. Keep the commands a job runs at the same paths on the compute nodes. The batch environment starts empty (--export=NIL); PATH is copied from the login node's environment by environment.pass.

Timings

  1. Set MinJobAge at least as long as your supervisor's poll interval. It is how long a finished job stays in squeue, and the adapter declares it as the PolledLoss window for state, cause and exit code. It also bounds how long the probe waits for sacct.
  2. Know your SlurmdTimeout and MessageTimeout. The node-loss window is 2 × SlurmdTimeout + 60s, and MessageTimeout enters the call bound twice. See Capabilities and declarations.
  3. Run NTP or chrony on the login node and the controller, and on the machine the adapter runs on if it uses ssh. The probe measures the skew once and adds it to the call bound; a drifting clock makes a cached skew wrong.

On the machine that runs the adapter

  1. Make preflight.cacheDir, if you set one, a directory of mode 0700 owned by the user, or leave it to be created. A writable slurm.New refuses a directory anyone else could reach, with an unsafe-directory error.
  2. If you want Ceased, provide a cessation command on the login node and name it in cessation.command. Without one, Ceased returns ErrUnsupported. See Configuration.