Skip to content

Preflight, read-only and observe-only

dispatch's safety rests on a few things the scheduler has to do exactly: keep a new object from running until Open, keep a hold through Open, refuse a release that names the wrong hold. A cluster can quietly fail any of them. A site plugin rewrites a submission, a queue isn't configured the way the adapter expects, a permission is missing, a controller isn't running. Configuration can't show these, and a failure found while a job is running is found too late.

So each adapter checks before it is used. Its Factory runs a preflight: it reads the cluster, and on a backend that may write, it submits a probe built exactly as a real submission is, but one that can never run. It then checks what the protocol relies on against that probe (that the gate holds, that a hold survives Open, that a stale release is refused) and stops it. An answer the probe decides is reported with source probed, so a reader can tell a capability that was tested from one that was only declared.

Why when it is opened

What an adapter answers at open is fixed for that backend. Capabilities and Rules are what MakePlan reads to accept or refuse a job, and Declarations are copied into the header of every job the backend creates, so every supervisor of that job uses the same call bound and windows. An answer that changed under a job would change the plan the job was accepted on. Proving the answers once, before anything is submitted, is what lets the plan be trusted.

It costs time at every open. The Kubeflow adapter's probe watches for ten seconds that nothing runs. The Slurm adapter reads the cluster on every open, but can keep the probe's answers in a cache, so an open with a fresh cached result submits nothing.

Three outcomes

A preflight ends one of three ways.

  • The cluster is refused. Factory returns an error, and there is no backend: an unsupported Slurm release or Kubernetes API, a missing command, a cluster the adapter can't even read.
  • A capability is lowered. The backend opens, and the answer it couldn't prove is no or unknown, with the reason. A job that needs it is refused at submit; one that doesn't is unaffected. A Slurm cluster whose holds fail the probe opens with Hold no, so a hold rule is refused while a resubmit rule is still accepted.
  • The adapter is observe-only. The backend opens, but without proof that a gate works it will launch nothing. It finds, inspects, watches and classifies, and refuses every write: Start returns ErrUnsupported, the other writes ErrReadOnly, each naming the gate that isn't proven. The Kubeflow adapter does this when its start gate leaks, its readmission gate isn't proven, or it may read but not write.

Read-only is your choice, observe-only is the adapter's

Both refuse writes, so they look alike from outside. They differ in who decided.

A read-only backend is one you asked for, with ReadOnly: true in dispatch.Config or dispatch.AdapterConfig: a status command, a dashboard, a login node that shouldn't change anything. It submits no probe, so the answers the probe would decide aren't proven. The Slurm adapter takes them from a fresh cached result if it has one, and reports them unknown if not; the Kubeflow adapter reports Hold as unknown. Every write returns ErrReadOnly. To write, open it again without the flag.

An observe-only backend is the adapter's verdict on the cluster: you asked to write, and the cluster gave it no grounds to. Opening it again changes nothing. The reason in the error says what to fix on the cluster, and the next open proves it again.

A cluster with neither is fully writable, but still reports what its preflight found in each answer's Reason.