Skip to content

Run the conformance suite against sind

go test ./... uses recorded transcripts of Slurm's output and never touches a cluster. This page is for the other test: running go/dispatch's conformance suite against a real slurmctld, using sind to bring up a small Slurm cluster in docker. Run it before you change how the adapter reads or writes Slurm, and when a new Slurm release needs checking.

What you need

  • docker, able to run containers as you;
  • ssh and ssh-keygen on the PATH;
  • a sind binary, on the PATH or named by DISPATCH_SLURM_SIND_BIN;
  • just, for the recipe below. Without it, run the command it wraps;
  • the module checked out, and Go at the version go.mod names;
  • time and memory. One run takes about 35 minutes per Slurm version. The cluster is one controller capped at 1 GiB and two workers capped at 512 MiB each, swap included, plus the two helper containers sind shares (sind-ssh and sind-dns), each capped at 256 MiB if it has no memory limit yet, whoever started it. Leave about 3 GiB free.

Run it

just sind 25.11.8
just sind 26.05.4

The argument is a sind-node image tag, and the tags tested are those two. The recipe is:

DISPATCH_SLURM_SIND=25.11.8 CGO_ENABLED=0 go test ./internal/sind/ -v -count=1 -timeout 150m

CGO_ENABLED=0 matters: one part of the run copies the test binary onto the cluster's controller and runs it there, and a binary linked against your host's C library may not start in the node image.

The images are ghcr.io/gsi-hpc/sind-node:<tag>, so the host needs to be able to fetch them.

Environment variables

Variable Meaning
DISPATCH_SLURM_SIND The sind-node image tag to run, such as 25.11.8 or 26.05.4. Unset, every test in the package skips and says so.
DISPATCH_SLURM_SIND_BIN The sind binary. Unset, sind is looked up on the PATH.
DISPATCH_SLURM_SIND_KEEP Any non-empty value leaves the cluster running when the tests end, for you to look at. Delete it yourself with sind delete cluster dsl<version without dots>, for example dsl25118.
DISPATCH_SLURM_SIND_ONHOST Set by the run itself, on the copy of the test binary it runs on the controller. Do not set it.

What it runs

The cluster has one controller, two workers, partitions main and high (so a job in high preempts one in main), TaskPlugin=task/cgroup, and an unprivileged user, trainer, who is reachable over SSH. A small job_submit plugin is compiled in the controller and loaded but passive until a test arms it.

TestConformance runs conformance.Backend over three ways of delivering the adapter's scripts:

Subtest Delivery Faults injected
docker-exec the local transport's scripts, carried into the controller by docker exec from outside all of them
ssh the SSH transport, to the controller's sshd all of them
local the test binary copied onto the controller and run there as trainer, so the real local transport runs all but the ones that need root or docker

The faults are a higher-priority partition to preempt a job, cutting a node's power, the owner setting Requeue=1 and requeueing, freezing slurmctld to delay a submit's reply, killing the host shell after a requeue's first step, and making the host-side check read a clock past the deadline.

Two more tests cover what the shared suite cannot:

  • TestReadBack arms the plugin to rewrite the job's comment, set Requeue=1, or strip the hold, and checks that Start stops the job and refuses it, and that preflight's probe refuses the cluster, naming the difference. It runs over the docker-exec delivery only.
  • TestLandingBound freezes slurmctld between a call's lookup and its scontrol, and checks that a call older than the credential TTL changes nothing while a younger one lands before its deadline plus CallBound. It runs over the docker-exec and ssh deliveries.

To run one part, add -run to the command. For example, only the SSH delivery:

DISPATCH_SLURM_SIND=25.11.8 CGO_ENABLED=0 go test ./internal/sind/ -v -count=1 -run 'TestConformance/ssh'

The cluster still comes up first.

Read the skips

A skip is a check nobody made. Expect these, and treat others as new:

  • ReadBack and Preflight inside conformance.Backend skip for every adapter, because the shared suite has no fault that rewrites a submission and the SPI has no preflight entry point. TestReadBack and the preflight run in slurm.New cover them here.
  • LoseNode and DelaySubmitReply are not injected on the local delivery, which has neither root nor docker. The other two deliveries cover them.
  • TestReadBack/adminhold always skips. No job_submit plugin can turn the owner's hold into an admin hold, because slurmctld picks the reason from the priority after the plugin has run. The refusal of an administrator's hold is covered by the recorded-transcript unit tests.
  • NativePolicyCompiler and Watcher skip, because the adapter implements neither.

Clean up

On success or failure the run deletes its cluster, unless DISPATCH_SLURM_SIND_KEEP is set. If a run is killed, look for leftovers with docker ps -a --filter name=sind-dsl.