Skip to content

dispatch-admission

dispatch-admission is the Kueue AdmissionCheck controller for dispatch's readmission gate. It runs in the cluster, beside the ClusterQueue configuration that names its AdmissionCheck. Install the admission controller deploys it; the start gate and the readmission gate says what it is for.

dispatch-admission [flags]

Go's flag package parses the flags, so -name and --name both work, and booleans take -name or -name=false.

Flags

Flag Default Meaning
-kubeconfig empty Kubeconfig path. Empty follows client-go's loading rules.
-context empty Kubeconfig context. Empty is the current one.
-in-cluster false Use the pod's service account. Refused together with -kubeconfig or -context.
-lease-namespace dispatch-system Namespace of the leader-election Lease.
-lease-name dispatch-admission Name of the Lease. The adapter's preflight reads this Lease as the controller's heartbeat.
-identity $POD_NAME, else the hostname This replica's identity in the Lease. Refused if empty.
-lease-duration 15s Leader-election lease duration.
-renew-deadline 10s Leader-election renew deadline.
-retry-period 2s Leader-election retry period.
-resync 30s Interval of a full pass over every Workload.
-request-timeout 30s Bound on each apiserver request.

The adapter's kueue.admissionLease setting must name the same Lease as -lease-namespace and -lease-name. Its default is the same pair as the flags' defaults.

What it does

  1. Leader election on a Lease, using client-go's leaderelection. Replicas that do not hold it wait as candidates. The Lease is not released on shutdown.
  2. A full pass, on becoming leader and then once per -resync:
  3. finds every AdmissionCheck whose spec.controllerName is dispatch.phpboyscout.uk/admission, whatever it is called, and sets its Active condition True (reason ControllerStarted) if it is not. It never sets False;
  4. lists every Workload and reconciles it.
  5. A watch on Workloads for the rest of the interval, reconciling each Workload as it changes. DELETED events are ignored. A failed watch waits out the interval before the next pass.

The checks the controller serves are those it found at its last full pass that completed the first step. An AdmissionCheck created later is served from the next pass, up to -resync later. If the first step fails, for example because the controller may not update admissionchecks/status, it logs full pass at error level and skips that pass's list of Workloads; the watch still runs, for the checks found before.

To reconcile a Workload, the controller acts only when the Workload has quota reserved (QuotaReserved=True) and has a check of ours whose state is Pending. Then:

The Workload belongs to Outcome
a TrainJob that is not dispatch's, or to none the check is set Ready, with the message not a dispatch TrainJob
a dispatch TrainJob with a pod of the previous execution that may still run the check stays Pending, with a message waiting for pods of the previous execution to go: <pod> on <node>, .... Nothing is written to the TrainJob
a dispatch TrainJob with no dispatch runtime patch the check is set Pending, with a message saying A cannot be raised
a dispatch TrainJob otherwise A is raised in the TrainJob (and the JobSet's UID recorded if it is not yet), then the check is set Ready, with the message admission <A>

A TrainJob is dispatch's when its name matches dj-<ULID>-s<N> or it has a runtime patch whose manager is dispatch.phpboyscout.uk/adapter. A Workload's TrainJob is the one named by its owner reference, whose UID must match.

Every write to the Workload is preconditioned on the Workload as read, and the TrainJob update on the TrainJob as read. A conflict is logged at info level and the Workload is reconciled again when it next changes.

Logging and exit

  • Logs go to standard error as log/slog text lines. leading and stopped leading are logged with the identity; raised the admission count with the TrainJob and the new A; failures at error level.
  • On SIGINT or SIGTERM it stops and exits 0.
  • On losing the Lease, or if its reconciliation loop ends, it exits 1 with dispatch-admission: lost the leader-election Lease, so that its Deployment restarts it as a candidate.
  • A flag error, or a conflicting -in-cluster, exits non-zero with a message on standard error.

Verbs it uses

Resource Verbs
admissionchecks (kueue.x-k8s.io) list; update on the status subresource
workloads (kueue.x-k8s.io) list, watch; update on the status subresource
trainjobs (trainer.kubeflow.org) get, update
jobsets (jobset.x-k8s.io) get
pods list
leases (coordination.k8s.io), in the Lease's namespace get, create, update

The ClusterRole in the install guide, which the test cluster uses, grants a little more than this.