Skip to content

Configure the transports

A transport is how the adapter's scripts reach the host that has the Slurm commands. The adapter ships two, local and ssh. Pick one with the transport key.

Choose a transport

Run the supervisor... Use
on the login node itself, or any host where sbatch and squeue work local (the default)
anywhere else, with a login node you can ssh to ssh

Use the local transport

Leave transport out, or set it to local. Every script runs as /bin/sh -c <script> on the machine the adapter runs on. There is nothing else to configure, and an ssh section is refused.

{ "transport": "local", "defaults": { "partition": "gpu" } }

Use the SSH transport

Set transport to ssh and give an ssh section with at least a host. In JSON:

{
  "transport": "ssh",
  "ssh": {
    "host": "login.example.org",
    "user": "ada",
    "identityFile": "/home/ada/.ssh/id_ed25519",
    "knownHostsFile": "/home/ada/.ssh/known_hosts",
    "connectTimeout": "15s"
  }
}

The same in Go:

cfg := slurm.Config{
    Transport: slurm.TransportSSH,
    SSH: &slurm.SSH{
        Host:           "login.example.org",
        User:           "ada",
        IdentityFile:   "/home/ada/.ssh/id_ed25519",
        KnownHostsFile: "/home/ada/.ssh/known_hosts",
        ConnectTimeout: slurm.Duration(15 * time.Second),
    },
}

An ssh section with any other transport, or transport: ssh with no ssh section, is refused when the configuration is read. The configuration reference lists every ssh key with its limits.

What OpenSSH has to provide

The transport runs your ssh binary, so everything it needs is plain OpenSSH:

  • An ssh client on the PATH of the process running the adapter.
  • Authentication that needs no prompt. Every connection runs with BatchMode=yes, in a new session with no controlling terminal, so nothing can ask for a password, a passphrase or a two-factor code. Use a key (identityFile), your agent, or an ssh_config entry. If your site needs two-factor, open a connection with a ControlMaster by hand on the control path the adapter uses, and let it reuse that.
  • A host key you already trust. Every connection runs with StrictHostKeyChecking=yes. A host that is not in known_hosts is refused, never added. Give knownHostsFile to use a file other than your default.
  • A remote login shell that can run one echo ... | base64 -d | /bin/sh -s command line. The script is sent base64 encoded, so it arrives intact under bash, tcsh and others. The login node needs base64 and /bin/sh.

Both of those are given on the command line, which wins over any ssh_config, so a configFile you name cannot turn either of them off.

Options the adapter does not expose

There is no key for an arbitrary ssh option. The ssh section is a closed set, because a free option once let StrictHostKeyChecking=no run a command against a host whose key had changed. Anything else you need goes in an ssh_config file named by configFile, which is passed as ssh -F. Cipher choices, ProxyCommand and the like belong there.

proxyJump takes ssh's [user@]host[:port] list, with commas between hops.

All values are checked for characters ssh would misread. A path may contain only letters, digits and - _ . / +; a host only letters, digits and - _ . :; a host, user or jump list may not begin with -. These character checks run when slurm.New opens the ssh transport, not when the configuration is decoded, so they do not run at all under slurm.WithExecutor.

Connection sharing

The transport keeps one connection open and reuses it, so each script does not pay for a new login. It sets ControlMaster=auto and ControlPersist to controlPersist (default 60 seconds).

The socket lives at controlPath. The default, /tmp/dispatch-slurm-<uid>/%C, is on local disk, never an NFS home, with <uid> replaced by your numeric user id. The adapter creates that directory with mode 0700, and refuses to start if it exists with another mode or another owner, because a socket there is a session anyone who could reach it could use.

If you set your own controlPath:

  • it must be absolute and at most 90 bytes once <uid> is replaced and %C counted as 40 characters (ssh's socket address limit);
  • %C and %% are the only ssh tokens it may use;
  • the adapter creates only the default directory, so create this one yourself, with a mode only you can reach, before the first run.

Set controlPath to none (in Go, slurm.ControlPathNone) to turn sharing off. Each script then makes its own connection.

Closing the backend does not close the shared connection, because other processes, or you, may be using it. It ends when controlPersist runs out.

When a connection fails

The adapter has to know whether a script ran. For ssh failures that happen before any session exists (a refused connection, a name that does not resolve, Permission denied, a failed host key check, an over-long control path) it can tell, and the call fails as one that never ran. For any other ending without the script's own end marker, such as a dropped connection or a kill at the bound, it cannot, and the outcome is unknown. Errors and error mapping shows what that means for each call.

A script may be at most 90 KiB (transport.MaxScriptSize). A larger one is refused before anything is sent.

Use docker exec, or any other delivery

docker exec is not a value of transport. The module's own tests deliver scripts with it, from an executor in internal/sind that is test-only and not importable. If you need that, or kubectl exec, or anything else, write a transport.Executor and hand it over in one of two ways.

An Executor has two methods, Run(ctx, transport.Script) (transport.Result, error) and Close() error. Run must deliver Script.Text whole, run it under /bin/sh, and return its stdout, stderr and exit status. It must honour Script.Bound and Script.Detached; both Script.RunContext and Script.Validate are there to help.

If your executor still uses OpenSSH, the transport/ssh package exports its pieces: ssh.Options and ssh.New build the adapter's own ssh executor, ssh.RemoteCommand is the command line it sends, and ssh.CheckControlPath and ssh.DefaultControlPath apply its control path rules.

To get the output back reliably, wrap the text with transport.NewFrame() and Frame.Wrap, and read the result with Frame.Parse. Frame.Started tells you whether the script reached the host. Return an error wrapping transport.ErrNeverRan only when you can prove the script never started; return one wrapping transport.ErrDropped when it may have. The adapter treats ErrNeverRan as "safe to say nothing happened", so claiming it wrongly can lose a job.

Then either:

  • Pass it to slurm.New, which then skips only the transport factory. The backend owns the executor and closes it.
b, err := slurm.New(ctx, cfg, slurm.WithExecutor(myExecutor))

The transport key still counts. It is validated as usual, so transport: ssh still needs its ssh section, and with transport: ssh preflight still reads the host's clock through your executor and every hold-family, requeue and cessation bound still includes ssh.connectTimeout.

  • Register it by name, so a configuration can select it and dispatch.Open can build it:
err := slurm.RegisterTransport("dockerexec", func(ctx context.Context, cfg slurm.Config) (transport.Executor, error) {
  return newDockerExec(cfg), nil
})

Then set "transport": "dockerexec". The function is a slurm.TransportFactory. A custom transport takes no ssh section. Registering a name twice returns dispatch.ErrAlreadyRegistered.

What the call bound does not cover on a custom transport

Preflight reads the host's clock against yours only when transport is ssh, and only the ssh section has a connect timeout. Under any other transport name, registered or local, the adapter assumes your clock and the host's are the same clock and that a script starts as soon as it is sent:

  • CallBound has no clock-offset term and no round-trip term;
  • the bound on each hold-family, requeue and cessation script has no connect allowance.

If your executor crosses a network, or runs the script on a machine whose clock is not yours, keep both clocks in step with NTP and make sure delivery is quick. Otherwise a call can land later than its deadline plus CallBound says, or be killed at its bound before it starts.

A registered name passes the Go decoder but not the published JSON Schema, which lists only local and ssh; a host that validates against the schema first must widen it.