The adapters compared¶

Each row links to the adapter's own page, which has the detail. Both adapters register themselves with dispatch.Register when imported, under the names slurm and kubeflow.

Reaching the cluster¶

	Slurm	Kubeflow
Module	`gitlab.com/phpboyscout/go/dispatch-slurm`	`gitlab.com/phpboyscout/go/dispatch-kubeflow`
How it talks to the scheduler	runs Slurm's commands as scripts, through the local shell or SSH, or a transport you register (configure the transports)	the Kubernetes apiserver, through a kubeconfig or the pod's service account, optionally with `AdapterConfig.Token` as a bearer token (`cluster`)
Settings	configuration	configuration
What it needs on the cluster	an account that can submit jobs, and the commands preflight checks for; a cessation command of the site's own, if it wants one	Kubeflow Trainer and JobSet; in Kueue mode, Kueue, an AdmissionCheck on the ClusterQueue and the `dispatch-admission` controller (choose Kueue mode or plain mode)
Permissions	the user's own on Slurm	the RBAC in grant the adapter its RBAC
Tested against	Slurm 25.11 and 26.05, in sind (tested versions)	Kubernetes 1.37, Trainer v2.3.0, JobSet v0.12.0, Kueue v0.20.1, on kind (tested versions)

Opening it¶

	Slurm	Kubeflow
Preflight at `Factory`	reads the cluster on every open; on a writable open, submits a held probe job and tests holds and accounting, unless a fresh cached result exists (read a preflight result)	checks the API group-versions and permissions, proves the readmission gate in Kueue mode, and on a writable open creates a probe TrainJob that can't run and watches it for ten seconds (read a preflight result)
Observe-only	no such mode: a cluster that fails preflight is refused, or opens with the capability lowered	when a gate isn't proven, or it may read but not write: reads work, `Start` is `ErrUnsupported`, other writes `ErrReadOnly`

Preflight, read-only and observe-only explains the difference.

What a job carries¶

	Slurm	Kubeflow
Extensions it requires	none	`dispatch.Image` and `kubeflow.RuntimeRef` (name a runtime in a job)
Extensions it accepts	`slurm.Placement`, `slurm.Resources`, `dispatch.Output`, `dispatch.SharedCheckpoint` (extensions)	`dispatch.Image`, `kubeflow.RuntimeRef`, `dispatch.SharedCheckpoint`
A container image	refused with `ErrUnsupported`	required
What one submission becomes	one `sbatch` job (how a job becomes an sbatch submission)	one TrainJob, with a copy of its runtime (annotations, labels and names)
The start gate	`sbatch --hold`; `Open` is `scontrol release`	Kueue's `admission-gated-by` annotation, or `spec.suspend` in plain mode (the two gates)
Execution identity	`<N>.<job id>.<restart>`, R the restart count	`<JobID>.<N>.<attempt>.<A>.<J>`, R = A + J

Optional interfaces¶

Found by type assertion on each adapter's Backend.

Interface	Slurm	Kubeflow
`dispatch.Holder`	yes	yes
`dispatch.Requeuer`	yes	no: Kubeflow has no requeue
`dispatch.NativePolicyCompiler`	no	no
`dispatch.Watcher`	no: it is polled	yes
`dispatch.CessationSignal`	yes: the site's command (`cessation`)	yes: from the state of the nodes the object ran on (cessation)
`dispatch.IdentityWeakener`	no	yes: a runtime accepted under `allowRankRestart` (runtime requirements)
`dispatch.MultiFinder`	yes	no
`dispatch.OutputReader`	yes	yes

Declarations¶

	Slurm	Kubeflow
`NodeLossWindow`	`2 × SlurmdTimeout + 60s`	the `nodeLossWindow` setting, `5m` by default
`CallBound`	worked out by preflight (how the call bound is made)	`timeouts.apiserverRequest` plus 5 seconds, 65 seconds by default (declarations)
`PolledLoss`	`MinJobAge` for state, cause and exit code; execution `Unbounded` where accounting works, `0` otherwise (declarations)	none, because it implements `Watcher`

Capabilities¶

Answers as each adapter gives them, declared unless marked. Where an answer depends on the cluster, the cell says what decides it. ReadmissionGate is on each adapter's page.

Capability	Slurm	Kubeflow
`ContainerImage`	no	yes
`NativeRequeue`, `NativeRestart`	no	no
`Hold`	probed by the preflight probe	probed by the preflight probe; `no` when observe-only, `unknown` when read-only
`PreTerminationSignal`	yes: SIGTERM, then SIGKILL after `KillWait`	unknown, from the catalogue: `terminationGracePeriodSeconds`, with SIGTERM delivery not observed
`AccountingHistory`	probed with `sacct`	no: a deleted TrainJob is gone
`AttemptInjection`	yes	yes in plain mode; in Kueue mode, probed with the readmission gate
`WorkingDirectory`	yes	yes, through the prologue
`ProcsPerNode`	yes	per runtime: yes for a torch runtime, no for one with no ML policy, unknown otherwise
`OutputPath`	yes	no
`OutputRead`, `OutputFollow`	yes	yes, rank 0's log of the current execution; `no` without `pods/log`
`OutputRetained`	yes	no: logs go with the pods

The full tables, with each Reason, are Slurm's capabilities and Kubeflow's.

Rules¶

Action	Slurm	Kubeflow
`resubmit`, `escalate`	yes	yes
`hold`	as the `Hold` capability	as the `Hold` capability
`requeue`, `restart`	no	no

Holds¶

	Slurm	Kubeflow
How a hold is placed	a far-future start time that encodes the hold instance (holds as a far-future start time)	the Workload deactivated in Kueue mode, the TrainJob suspended in plain mode (holds are Workload deactivation)
A hold on a running execution	refused with `ErrRejected`: the job is no longer pending	ends the execution, with cause `held`
A gate held by someone else	`JobHeldAdmin`, escalated as `admin-held`	another entry in `admission-gated-by`, escalated as `admin-held`

Output¶

	Slurm	Kubeflow
Streams	stdout and stderr, in separate files	one log; `Stdout` returns it, `Stderr` is `ErrUnsupported`
Executions	every execution of the submission, each in its own file	rank 0 of the current execution only

Read a job's output covers both.

Errors¶

Slurm's errors and Kubeflow's error mapping say which scheduler answers become which sentinel, and so how the supervisor records each one; see how the supervisor reads an adapter's errors.