Connect to a cluster and run preflight¶

By the end of this page you will have opened the Slurm adapter against your cluster's login node, let it run its preflight, and printed what preflight found out about the cluster. Allow about ten minutes.

Preflight is what slurm.New does before it hands you a backend. It checks the cluster is one the adapter can drive, reads the settings that matter, and submits one held probe job to find out whether holds and accounting work. The probe never runs: it is held until a start time ten years away, and preflight stops it when its checks are done.

Before you start¶

You need:

• a login node of a Slurm 25.11 or 26.05 cluster that you can reach with ssh, using a key and without a password prompt;
• that login node's host key already in your known_hosts. The adapter runs ssh with BatchMode=yes and StrictHostKeyChecking=yes, so it will never ask you to accept a key. Run ssh login.example.org true once by hand first, and accept the key there;
• a partition you may submit to. The probe is a real submission, so a partition with no access for you fails preflight;
• Go, at the version go.mod names.

The login node needs these commands on the PATH that /bin/sh sees: sbatch, squeue, scontrol, scancel, sacct, timeout, date and base64. On a cluster that authenticates with munge it also needs munge and unmunge. Preflight checks for all of them and names any that are missing.

Make a module and fetch the adapter. It has no release yet, so go get resolves a pseudo-version.

mkdir preflight-tutorial && cd preflight-tutorial
go mod init example.com/preflight-tutorial
go get gitlab.com/phpboyscout/go/dispatch-slurm

Write the program¶

Put this in main.go. Change login.example.org, the user, the partition and the cache directory to yours.

package main

import (
    "context"
    "fmt"
    "log"
    "time"

    "gitlab.com/phpboyscout/go/dispatch/model"

    slurm "gitlab.com/phpboyscout/go/dispatch-slurm"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
    defer cancel()

    cfg := slurm.Config{
        Transport: slurm.TransportSSH,
        SSH:       &slurm.SSH{Host: "login.example.org", User: "ada"},
        Defaults:  slurm.Defaults{Partition: "gpu"},
        Preflight: slurm.Preflight{CacheDir: "/home/ada/.cache/dispatch-slurm"},
    }

    b, err := slurm.New(ctx, cfg)
    if err != nil {
        log.Fatal(err)
    }
    defer b.Close()

    d := b.Declarations()
    fmt.Println("node loss window:", d.NodeLossWindow)
    fmt.Println("call bound:      ", d.CallBound)

    for evidence, window := range d.PolledLoss {
        fmt.Printf("polled loss %-10s %s\n", evidence, window)
    }

    caps, err := b.Capabilities(ctx, nil)
    if err != nil {
        log.Fatal(err)
    }

    for _, c := range model.Capabilities() {
        s := caps[c]
        fmt.Printf("%-22s %-8s %-9s %s\n", c, s.Value, s.Source, s.Reason)
    }
}

slurm.New takes the Config and returns a *slurm.Backend. The transport is ssh, so the adapter will run its scripts on the login node through your OpenSSH client. Everything you leave out takes its default; the configuration reference lists them.

Preflight.CacheDir gives the adapter somewhere to keep the probe's result. The directory is created with mode 0700 if it is missing. Leave it out and the probe runs on every call to slurm.New.

Run it¶

go run .

The first run takes longer than later ones, because the probe job has to be submitted, held, released, stopped and looked up in accounting. On a quiet cluster it takes some seconds. If the cluster keeps accounting but is slow to record the probe, preflight waits for it for up to the cluster's MinJobAge.

You will see output of this shape. Your numbers and reasons will differ, and the polled loss lines come in no fixed order.

node loss window: 2m0s
call bound:       5m32s
polled loss state      2m0s
polled loss cause      2m0s
polled loss exit-code  2m0s
polled loss execution  -1ns
ContainerImage         no       declared  container images are not mapped in v1
NativeRequeue          no       declared
NativeRestart          no       declared
Hold                   yes      probed
PreTerminationSignal   yes      declared  SIGTERM, then SIGKILL after KillWait 30s
AccountingHistory      yes      probed    comments are kept; AccountingStoreFlags=job_comment
...

A few things to notice:

• Hold and AccountingHistory say probed: preflight found them out by trying them on your cluster. The rest say declared: they are true of the adapter on any cluster.
• A polled loss of -1ns is model.Unbounded: that evidence stays readable however late you poll. It is what execution shows when accounting works.
• Hold could say no, with a reason. Then this cluster cannot hold jobs the way dispatch needs. If the cluster's authentication is one the adapter does not know, CallBound is 0s as well. Capabilities and declarations lists each answer.

Run it again. This time Hold and AccountingHistory come from the cache, so the program returns faster, and the values are the same. The cache is good for 24 hours by default, and is discarded sooner if the cluster's configuration or the adapter's own settings change.

If it fails¶

slurm.New returns the first problem preflight meets. The common ones:

Errors and error mapping lists the rest.

What you did¶

You opened a Backend over SSH, which ran preflight, and you read its declarations and capabilities. Next:

• Configure the transports to use the local transport, or to tune SSH.
• Read a preflight result for what each answer means for the jobs you will run.
• How identity works for what the adapter does to a job's name.