Capabilities and support¶

Capability	Means	Needed at submit when	`Unknown` may proceed
`ContainerImage`	the job can name the image it runs in, with the `dispatch.Image` extension	the spec carries `dispatch.Image`	no
`NativeRequeue`	the scheduler can requeue the job itself, under a rule pushed down to it	not checked at submit
`NativeRestart`	the scheduler can restart the job in place itself, under a rule pushed down to it	not checked at submit
`Hold`	the adapter can place and release a numbered hold that survives `Open` and refuses a release naming another instance, as `Holder` requires	not checked at submit
`PreTerminationSignal`	the job's processes get a signal, and time to act on it, before they are killed	not checked at submit
`AccountingHistory`	the scheduler keeps a job's history after it drops the job from its live view, so an end can be read late	not checked at submit
`AttemptInjection`	every rank of every execution sees the four `DISPATCH_` variables	always	no, and nothing waives it
`ReadmissionGate`	the adapter runs a gate that stops each later admission of the same object until it has injected the next admission id; only Kubeflow with Kueue needs one, so `no` is the normal answer elsewhere	not checked at submit
`WorkingDirectory`	`JobSpec.WorkDir` is honoured	`JobSpec.WorkDir` is set	no
`ProcsPerNode`	`JobSpec.ProcsPerNode` is honoured	`JobSpec.ProcsPerNode` > 0	no
`OutputPath`	the job can say where its stdout and stderr go, with the `dispatch.Output` extension	the spec carries `dispatch.Output`	no
`OutputRead`	a submission's output can be read through the `Observer`	not checked at submit	yes
`OutputFollow`	output can be followed while it is written	not checked at submit	yes
`OutputRetained`	output stays readable once the execution that wrote it has gone	not checked at submit	yes

The Go names are constants of type dispatch.Capability (re-exported from model), and the string value of each is the name in the first column. The Hold capability's Go name is dispatch.HoldCapability. model.Capabilities() lists all fourteen in the order of this table.

The core reads only the five capabilities a spec can need. The others are there for a host to show, and for an adapter to report what it found.

Support is three-valued¶

An adapter answers each capability with a dispatch.Support:

Field	Values
`Value` (`Tristate`)	`unknown` (the zero value), `yes`, `no`
`Source`	`declared` (the zero value), `probed`, `catalogue`
`Reason`	free text; shown in the refusal when `Value` is `no`

`Source`	Means
`declared`	the adapter answers from what it knows of the scheduler, and of this cluster's configuration where that decides it, without testing it
`probed`	the adapter tested it on this cluster, usually in its preflight when it was opened
`catalogue`	the answer is the scheduler's documented behaviour, neither tested on this cluster nor fixed by the adapter

Support is per cluster, and for a spec: Backend.Capabilities takes the JobSpec, because what a cluster can do can depend on what the job names. It may be called with a nil spec.

How Submit uses the answers¶

For each capability the spec needs (the first five rows above), MakePlan and Submit:

proceed on yes;
refuse on no, with ErrUnsupported and the support's Reason;
on unknown, refuse with ErrUnsupported, unless the capability is visible on failure, in which case the job proceeds and the capability is listed in Plan.Unverified.

Capability.VisibleOnFailure() is true only for OutputRead, OutputFollow and OutputRetained: the capabilities where an unsupported request surfaces as an error to the caller rather than a job that quietly runs differently. None of them is needed by a field today, so Plan.Unverified is empty in practice.

Adapter-specific extensions aren't checked here. An adapter refuses an extension from another namespace itself, with ErrUnsupported.

Rule support¶

Recovery rules are checked separately, through Backend.Rules, which answers one dispatch.RuleSupport per rule, in policy order. Its one field, Perform, is a Support saying whether the rule's action can be carried out on this cluster by anyone.

`Perform`	At submit
`no`	refused, `ErrUnsupported`; nothing can accept it
`unknown`	refused, `ErrUnsupported`, unless `AcceptUnverified(rule, enforcer)` names who enforces it; the rule is then listed in `Plan.UnverifiedRules`
`yes`	assigned an enforcer: `backend` if the adapter's native offer includes it, else `supervisor` with `WithSupervisor()`, else `nobody` with `AcceptUnenforced(rule)`, else refused with `ErrRejected`

An adapter that implements NativePolicyCompiler is asked which rules it would enforce itself (Native), then offered the whole policy with every rule's full budget (Compile). Under a shared checkpoint it is asked neither, because every recovery rule becomes a hold for a person. A rule it compiles must be one it answered yes for natively, and a native yes must have a Perform of yes; otherwise the submit is refused with ErrRejected. The accepted offer is recorded in Plan.Native and passed to every Start.

Execution identity¶

Every execution of a job runs under its own identity: its DISPATCH_ATTEMPT differs from every other execution's, and its DISPATCH_EPOCH is higher than any before it, so a rerun can't be taken for the execution it replaces, and neither can its checkpoints. Some site opt-ins give that up. Kubeflow's allowRankRestart, for example, accepts runtimes whose podFailurePolicy or backoffLimitPerIndex let the backend rerun one rank under the identity it already had.

An adapter that can weaken the guarantee implements dispatch.IdentityWeakener, and its IdentityWeakened(ctx, spec) answers why for that spec, or "" when the spec keeps it. MakePlan records the answer in Plan.IdentityWeakened, so it is in the record's header and every actor sees it, and Status.IdentityWeakened() returns it for a host to print. An adapter without the interface never weakens it, and the field is absent from the record.

The core refuses nothing on it and changes no step: the plan records the weakening, and the host shows it. An error from IdentityWeakened fails MakePlan and Submit.

Each adapter's answers¶

The Slurm adapter's answers, and which of them its preflight probe decides;
the Kubeflow adapter's answers, which depend on the runtime a job names.

The adapters compared puts the two side by side.

The memory adapter's answers¶

The memory adapter declares AttemptInjection, Hold, WorkingDirectory and ProcsPerNode as yes, and every other capability no. Shaped like Kubeflow, it also declares ReadmissionGate yes and WorkingDirectory no. For rules, resubmit, hold and escalate are yes, and requeue and restart are no. It leaves execution identity intact unless identityWeakened is set. Its configuration can override every table; see the memory adapter.

Its shapes model the protocol, not each adapter's capability table, so they differ from the real adapters. Every source is declared, where the real adapters probe some answers. Shaped like Slurm, it answers no where the Slurm adapter answers yes for PreTerminationSignal and the four output capabilities. Shaped like Kubeflow, it answers no for ContainerImage, WorkingDirectory, OutputRead and OutputFollow, all of which the Kubeflow adapter can support, and yes for ProcsPerNode, which the Kubeflow adapter answers per runtime.