Skip to content

Why the controller never sets Active to False

An AdmissionCheck has a status condition Active. A controller is expected to set it True while it is serving the check. The obvious thing for a well-behaved controller to do is set it False when it goes away. dispatch-admission does not. It sets Active True when it starts leading, sets it True again on each full pass if something cleared it, and never writes False, including on shutdown.

Why not

Kueue treats an AdmissionCheck that is not active as a reason to stop the entire ClusterQueue that references it. The Workloads on that queue are not admitted, with the message "Can't admit new workloads: references inactive AdmissionCheck(s)". On a shared queue that stalls every workload on it, dispatch's or not, because of one controller restarting.

A controller that is merely down, with the check left Active, shows up where it belongs: at the check. Workloads that have reserved quota sit with their check Pending, holding it, and their status says so. Nothing is hidden behind a queue-wide message.

That is not free, and you should plan for it. A Workload on any ClusterQueue that lists the check waits at it while the controller is down, whether it is dispatch's or not, because only the controller approves it. List the check on queues where you accept that dependency, and run more than one replica.

The cost, and what pays for it

If Active is never cleared, it says nothing about whether a controller exists. It stays True with no controller process at all, and nothing in Kueue clears it. So Active cannot be what the adapter checks to decide that the readmission gate works.

The adapter checks that the ClusterQueue lists the check, that the check names dispatch's controller and is Active, and, in addition, that there is a live heartbeat: the controller's leader-election Lease must have a holder and a recent renewal. A running leader renews the Lease; a dead one does not. The Lease heartbeat explains the rule.

Taken together:

  • Active tells Kueue the check is usable, so it does not stop the queue.
  • The Lease tells the adapter the controller is really there.

The two jobs are separate because they have to be: the first must never go false, and the second has to.

ReleaseOnCancel is off for the same reason

The controller's leader election does not release the Lease on shutdown. A cleared holder reads as no heartbeat, and every adapter opened during a rolling restart would then be observe-only. The old holder's name stays on the Lease until it expires or a new leader takes it.