Skip to content

Install the admission controller

cmd/dispatch-admission is the readmission gate for Kueue mode. It runs in the cluster as a Deployment, and the adapter refuses to write to a cluster where it is not running. You need it only in Kueue mode; plain mode has no controller to install.

The start gate and the readmission gate says what it does and why it has to be in the cluster. This page installs it.

You'll need cluster-admin rights, Kueue v0.20 and Trainer v2.3 already installed, and a container registry your nodes can pull from.

1. Build an image

The repository ships no image and no manifests, so build your own. The binary is static:

CGO_ENABLED=0 go build -trimpath -o dispatch-admission ./cmd/dispatch-admission

The test harness wraps it in a FROM scratch image that runs as a non-root user (internal/kindtest/testdata/Dockerfile.admission):

FROM scratch
COPY dispatch-admission /dispatch-admission
USER 65532:65532
ENTRYPOINT ["/dispatch-admission"]

Push the result to your registry.

2. Create the AdmissionCheck

The controller serves every AdmissionCheck whose controllerName is dispatch.phpboyscout.uk/admission, whatever the check is called. Create one:

apiVersion: kueue.x-k8s.io/v1beta2
kind: AdmissionCheck
metadata:
  name: dispatch-admission
spec:
  controllerName: dispatch.phpboyscout.uk/admission

3. Deploy the controller with RBAC and a Lease

This is the manifest the test harness applies (internal/kindtest/testdata/admission.yaml), with four changes for a real cluster. The image is a placeholder, and three harness settings are left out: imagePullPolicy: Never (the harness loads its image into the nodes rather than pulling it), the argument -resync=10s (a short pass interval for the tests; the default is 30s) and terminationGracePeriodSeconds: 5. Change the image to yours:

apiVersion: v1
kind: Namespace
metadata: {name: dispatch-system}
---
apiVersion: v1
kind: ServiceAccount
metadata: {name: dispatch-admission, namespace: dispatch-system}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: dispatch-admission}
rules:
- apiGroups: [kueue.x-k8s.io]
  resources: [workloads]
  verbs: [get, list, watch, update]
- apiGroups: [kueue.x-k8s.io]
  resources: [workloads/status, admissionchecks/status]
  verbs: [update]
- apiGroups: [kueue.x-k8s.io]
  resources: [admissionchecks]
  verbs: [get, list, watch]
- apiGroups: [trainer.kubeflow.org]
  resources: [trainjobs]
  verbs: [get, list, update]
- apiGroups: [jobset.x-k8s.io]
  resources: [jobsets]
  verbs: [get]
- apiGroups: [""]
  resources: [pods]
  verbs: [list]
- apiGroups: [coordination.k8s.io]
  resources: [leases]
  verbs: [get, list, watch, create, update]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata: {name: dispatch-admission}
roleRef: {apiGroup: rbac.authorization.k8s.io, kind: ClusterRole, name: dispatch-admission}
subjects:
- {kind: ServiceAccount, name: dispatch-admission, namespace: dispatch-system}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: dispatch-admission, namespace: dispatch-system}
spec:
  replicas: 1
  selector: {matchLabels: {app: dispatch-admission}}
  template:
    metadata: {labels: {app: dispatch-admission}}
    spec:
      serviceAccountName: dispatch-admission
      containers:
      - name: controller
        image: registry.example/dispatch-admission:v0   # your image
        args: [-in-cluster]
        env:
        - name: POD_NAME
          valueFrom: {fieldRef: {fieldPath: metadata.name}}
        resources:
          requests: {cpu: 20m, memory: 32Mi}
          limits: {memory: 128Mi}

Points to know:

  • -in-cluster makes it use the pod's service account. -in-cluster together with -kubeconfig or -context is refused at startup.
  • POD_NAME becomes the replica's identity in the Lease. Without it the identity is the hostname. An empty identity is refused.
  • Leader election uses a Lease named dispatch-admission in dispatch-system. The two flags -lease-namespace and -lease-name move it. Only the leader reconciles, so you can run more than one replica; the others wait as candidates. Safety does not rest on the Lease: every write the controller makes is preconditioned on the version it read.
  • The Lease name is also the adapter's heartbeat. The adapter's kueue.admissionLease setting must name the same Lease. Its default is dispatch-system/dispatch-admission, which is what the flag defaults give, so leave both alone unless you have a reason. See the Lease heartbeat.
  • Losing the Lease ends the process with a non-zero exit, so the Deployment restarts it as a candidate. Stopping on SIGTERM or interrupt exits cleanly.
  • The controller sets the AdmissionCheck's Active condition True when it starts leading, and never sets it False. See why.

Wait for it to roll out, and check the Lease has a holder:

kubectl -n dispatch-system rollout status deploy/dispatch-admission
kubectl -n dispatch-system get lease dispatch-admission -o jsonpath='{.spec.holderIdentity}'
kubectl get admissionchecks dispatch-admission \
  -o jsonpath='{.status.conditions[?(@.type=="Active")].status}'

The last command should print True. dispatch-admission reference lists every flag.

The ClusterRole above is the one the harness runs with and is tested. The calls the code makes are narrower: list on admissionchecks and update on admissionchecks/status; list and watch on workloads and update on workloads/status; get and update on trainjobs; get on jobsets; list on pods; and get, create and update on leases in the Lease's namespace. You can trim the role to those if your policy needs it, but the narrower form has not been run.

4. List the check on the ClusterQueue, for every flavor

Add the check to the ClusterQueue that your jobs' LocalQueue points at:

apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata: {name: team-a}
spec:
  admissionChecksStrategy:
    admissionChecks:
    - name: dispatch-admission
  # namespaceSelector, resourceGroups and the rest as you have them

Do not add onFlavors. The adapter treats a ClusterQueue that lists the check with onFlavors as not listing it, and goes observe-only. The reason is in Kueue: before a Workload is admitted, Kueue lists on it only the checks that cover all of its ClusterQueue's flavors (v0.20.1, pkg/workload/workload.go). A check limited to some flavors would not appear on a gated Workload, so the adapter's Start would not find it there.

A shared ClusterQueue is fine. Jobs that are not dispatch's are approved as soon as they have quota, so the check does not hold them up while the controller runs. While it is down, they wait at the check with dispatch's, because only the controller approves it. Why the controller never sets Active to False explains the trade, and why to run more than one replica.

5. Point the adapter at them

In the adapter's configuration, add a kueue section:

{
  "namespace": "team-a",
  "probeRuntime": {"kind": "ClusterTrainingRuntime", "name": "torch-distributed"},
  "kueue": {
    "localQueue": "lq",
    "admissionCheck": "dispatch-admission"
  }
}

admissionCheck is the name of the AdmissionCheck object. Without it the adapter reports kueue.admissionCheck is not set and is observe-only. admissionLease is optional and defaults to dispatch-system/dispatch-admission.

Then open the adapter and read the preflight result. ReadmissionGate should be yes, probed.

If it does not work

The preflight reasons say which step is missing. In order, the adapter checks that the LocalQueue exists, that its ClusterQueue lists the check for every flavor, that the AdmissionCheck exists, that its controllerName is dispatch's and that it is Active, and that the Lease has a holder renewed within twice its leaseDurationSeconds.