How a job becomes an sbatch submission¶

Start turns a model.StartRequest into one script, runs it on the Slurm host, and checks the job it created by reading it back. This page lists what goes into the script. The golden scripts in testdata/golden in the repository are the exact text, and the adapter's tests compare against them.

The script¶

Start sends one script of three steps, in this order. Each step stops the script if it fails.

1. mkdir -p the job's output directories.
2. Write the output manifest, a file naming where this submission's output goes.
3. Run sbatch, with the batch script in a quoted here-document.

The whole script may be at most 90 KiB.

The sbatch options¶

JobSpec.Name and JobSpec.Checkpoint do not reach sbatch.

All values are shell-quoted. Placement and defaults merge field by field: each field Placement sets replaces the same field of defaults.

The batch script¶

sbatch stores a script and runs it on the first allocated node. It is a POSIX sh script that:

1. sets DISPATCH_JOB_ID and DISPATCH_SUBMISSION;
2. reads SLURM_RESTART_COUNT as R (0 if unset) and exits with an error if R is 1,000,000 or more;
3. sets DISPATCH_ATTEMPT and DISPATCH_EPOCH, and exports all four, with the values below;
4. redirects the shell's stdout and stderr, by exec, to this execution's files (below);
5. exports each environment.pass variable, with the value it had on the host at Start;
6. exports SLURM_EXPORT_ENV=ALL, so steps launched by srun inherit the environment;
7. exports each variable of JobSpec.Env, sorted by name;
8. execs the command, each argument quoted.

The variables the job sees, for job <ULID>, generation G, submission N and restart count R:

What they mean to a job is in the execution environment.

The command runs once, on the batch host, as sbatch runs it. It is never wrapped in srun; launching ranks (srun, torchrun) is the command's job.

Under --export=NIL the batch environment has no PATH and no HOME. The script exports the environment.pass variables first (PATH by default), so a job that needs HOME must list it there.

Every value is a literal in the script, and slurmctld stores the script. An administrator can read it, and accounting keeps it where the cluster records job scripts. Keep secrets out of JobSpec.Env and environment.pass.

JobSpec.Env names and environment.pass names must be shell identifiers, and neither may start DISPATCH_. JobSpec.Validate refuses such an Env key, so Start returns ErrInvalid; the adapter refuses such a pass name when it decodes the configuration.

Output files¶

For job <ULID> and submission <N>, with output.dir resolved:

Output reads a stream by listing the files under its manifest prefix and suffix, in restart order, numerically, so r10 follows r9. With follow it polls every two seconds for new bytes and for a new execution's file.

A job that carries model.Output writes elsewhere. Its stdout path P becomes P.s<N>-<job id>-r<R>, with no suffix, and the same for stderr. Both must be absolute and free of newlines and tabs, and the directories are created.

Extensions¶

A JobSpec carries extensions by namespace and name. The adapter registers two in the slurm namespace:

slurm.Placement chooses where the job runs. Each non-empty field overrides the same defaults key for this job.

slurm.Resources asks for what JobSpec has no field for.

Of go/dispatch's own extensions, model.Output is mapped as above and model.SharedCheckpoint is accepted and needs nothing from Slurm. model.Image is refused with ErrUnsupported, as is any extension from another namespace.

What is refused before anything runs¶

An invalid job id, a spec that fails JobSpec.Validate, a negative attempt, a submission outside 1 to 999,999, a relative WorkDir, an output.dir containing %, \, a newline or a tab, an unusable extension, or a script over 90 KiB is refused with ErrInvalid or ErrUnsupported, and no command runs. See Errors and error mapping.