Choose Kueue mode or plain mode¶

The adapter runs in one of two modes, chosen by a single key: the kueue section of its configuration. Present means Kueue mode. Absent means plain mode. There is no other switch.

	Kueue mode	Plain mode
Configuration	`kueue.localQueue` set	no `kueue` section
Start gate	annotation `kueue.x-k8s.io/admission-gated-by`	`spec.suspend: true`
`Open`	removes dispatch's gate from the annotation	sets `suspend: false` and the `opened` annotation, raises A; with a hold in place, writes only the `opened` annotation (holds)
Readmission gate	`dispatch-admission`, an AdmissionCheck controller	none needed
Hold	Workload `spec.active: false`	TrainJob `spec.suspend: true`
Who writes `spec.suspend`	Kueue; the adapter never does	the adapter
Needs in the cluster	Kueue, a LocalQueue and ClusterQueue, `dispatch-admission`	nothing beyond Trainer and JobSet

Use Kueue mode when your cluster queues and shares GPUs with Kueue. Use plain mode when it does not.

Kueue mode¶

Set the kueue section:

{
  "namespace": "team-a",
  "runtimes": [{"kind": "ClusterTrainingRuntime", "name": "torch-distributed"}],
  "kueue": {
    "localQueue": "lq",
    "priorityClass": "low",
    "admissionCheck": "dispatch-admission"
  }
}

localQueue is required. Every TrainJob gets the label kueue.x-k8s.io/queue-name with it.
priorityClass is optional. When set, every TrainJob gets kueue.x-k8s.io/priority-class with it; it names a Kueue WorkloadPriorityClass.
admissionCheck is optional in the schema but is needed to write. See below.
admissionLease names the controller's Lease; the default is dispatch-system/dispatch-admission.

You need, in the cluster: Kueue v0.20 (the adapter requires the API group-version kueue.x-k8s.io/v1beta2 and coordination.k8s.io/v1), the LocalQueue, its ClusterQueue listing the AdmissionCheck for every flavor, the AdmissionCheck object, and the controller running. Install the admission controller does that.

With admissionCheck unset, preflight sets ReadmissionGate to no with the reason kueue.admissionCheck is not set, and the adapter is observe-only: it reads and classifies, and Start is refused. Treat the key as required.

Plain mode¶

Leave the kueue section out:

{
  "namespace": "team-a",
  "runtimes": [{"kind": "ClusterTrainingRuntime", "name": "torch-distributed"}]
}

Nothing re-admits a TrainJob without Kueue, so no readmission gate runs and none is needed; preflight declares ReadmissionGate no with the reason not needed: nothing re-admits a TrainJob without Kueue. That no is not a fault: the capability says whether a gate runs, and plain mode writes normally without one.

Kueue must not manage queue-less jobs. If Kueue is installed with manageJobsWithoutQueueName on, it would make a Workload for every TrainJob and take over spec.suspend. When it runs the probe, preflight catches this: the probe is a TrainJob with no queue, and if a Workload appears for it the adapter refuses the cluster for plain mode with ErrUnsupported, naming the setting. If Kueue's API is not served, or the adapter may not list Workloads, the check is skipped without a word, so where Kueue is installed, grant list on workloads even in plain mode.

In plain mode Open and Release do the work that Kueue and the controller do in Kueue mode, and both refuse to raise the admission count while a pod of the object may still run. Release on a plain-mode hold returns ErrPodsRemain until the pods are gone, and the caller retries. Never two executions at once explains it.

Changing mode¶

The mode is a property of the adapter instance, not of the object. An object created gated by the annotation is opened by removing the annotation, and one created suspended is opened by unsuspending it, so an adapter in the other mode would not operate on it correctly. Do not change the configuration under jobs in flight: stop or drain them first.