dispatch-admission¶
dispatch-admission is the Kueue AdmissionCheck controller for dispatch's readmission gate. It runs
in the cluster, beside the ClusterQueue configuration that names its AdmissionCheck. Install the
admission controller deploys it; the start gate and
the readmission gate says what it is for.
Go's flag package parses the flags, so -name and --name both work, and booleans take -name
or -name=false.
Flags¶
| Flag | Default | Meaning |
|---|---|---|
-kubeconfig |
empty | Kubeconfig path. Empty follows client-go's loading rules. |
-context |
empty | Kubeconfig context. Empty is the current one. |
-in-cluster |
false |
Use the pod's service account. Refused together with -kubeconfig or -context. |
-lease-namespace |
dispatch-system |
Namespace of the leader-election Lease. |
-lease-name |
dispatch-admission |
Name of the Lease. The adapter's preflight reads this Lease as the controller's heartbeat. |
-identity |
$POD_NAME, else the hostname |
This replica's identity in the Lease. Refused if empty. |
-lease-duration |
15s |
Leader-election lease duration. |
-renew-deadline |
10s |
Leader-election renew deadline. |
-retry-period |
2s |
Leader-election retry period. |
-resync |
30s |
Interval of a full pass over every Workload. |
-request-timeout |
30s |
Bound on each apiserver request. |
The adapter's kueue.admissionLease setting must name the same Lease as -lease-namespace and
-lease-name. Its default is the same pair as the flags' defaults.
What it does¶
- Leader election on a Lease, using client-go's
leaderelection. Replicas that do not hold it wait as candidates. The Lease is not released on shutdown. - A full pass, on becoming leader and then once per
-resync: - finds every AdmissionCheck whose
spec.controllerNameisdispatch.phpboyscout.uk/admission, whatever it is called, and sets itsActiveconditionTrue(reasonControllerStarted) if it is not. It never setsFalse; - lists every Workload and reconciles it.
- A watch on Workloads for the rest of the interval, reconciling each Workload as it changes.
DELETEDevents are ignored. A failed watch waits out the interval before the next pass.
The checks the controller serves are those it found at its last full pass that completed the first
step. An AdmissionCheck created later is served from the next pass, up to -resync later. If the
first step fails, for example because the controller may not update admissionchecks/status, it
logs full pass at error level and skips that pass's list of Workloads; the watch still runs, for
the checks found before.
To reconcile a Workload, the controller acts only when the Workload has quota reserved
(QuotaReserved=True) and has a check of ours whose state is Pending. Then:
| The Workload belongs to | Outcome |
|---|---|
| a TrainJob that is not dispatch's, or to none | the check is set Ready, with the message not a dispatch TrainJob |
| a dispatch TrainJob with a pod of the previous execution that may still run | the check stays Pending, with a message waiting for pods of the previous execution to go: <pod> on <node>, .... Nothing is written to the TrainJob |
| a dispatch TrainJob with no dispatch runtime patch | the check is set Pending, with a message saying A cannot be raised |
| a dispatch TrainJob otherwise | A is raised in the TrainJob (and the JobSet's UID recorded if it is not yet), then the check is set Ready, with the message admission <A> |
A TrainJob is dispatch's when its name matches dj-<ULID>-s<N> or it has a runtime patch whose
manager is dispatch.phpboyscout.uk/adapter. A Workload's TrainJob is the one named by its owner
reference, whose UID must match.
Every write to the Workload is preconditioned on the Workload as read, and the TrainJob update on the TrainJob as read. A conflict is logged at info level and the Workload is reconciled again when it next changes.
Logging and exit¶
- Logs go to standard error as
log/slogtext lines.leadingandstopped leadingare logged with the identity;raised the admission countwith the TrainJob and the new A; failures at error level. - On
SIGINTorSIGTERMit stops and exits 0. - On losing the Lease, or if its reconciliation loop ends, it exits 1 with
dispatch-admission: lost the leader-election Lease, so that its Deployment restarts it as a candidate. - A flag error, or a conflicting
-in-cluster, exits non-zero with a message on standard error.
Verbs it uses¶
| Resource | Verbs |
|---|---|
admissionchecks (kueue.x-k8s.io) |
list; update on the status subresource |
workloads (kueue.x-k8s.io) |
list, watch; update on the status subresource |
trainjobs (trainer.kubeflow.org) |
get, update |
jobsets (jobset.x-k8s.io) |
get |
pods |
list |
leases (coordination.k8s.io), in the Lease's namespace |
get, create, update |
The ClusterRole in the install guide, which the test cluster uses, grants a little more than this.