Skip to content

The Lease heartbeat

In Kueue mode the adapter will not write to a cluster whose readmission gate it cannot prove. The gate is a controller, and proving that a controller exists is harder than it looks.

What cannot prove it

The AdmissionCheck's Active condition is no evidence. The controller sets it and never clears it (why), and nothing in Kueue clears it either, so it stays True after the controller is gone. The AdmissionCheck existing, with the right controllerName, proves only that someone wrote the object.

What can

The controller holds a leader-election Lease, and a running leader renews it every few seconds. A Lease that has a holder and a recent renewal is a signal only a live leader can produce. The adapter reads it as a heartbeat.

At preflight, with kueue.admissionLease naming the Lease (default dispatch-system/dispatch-admission), the adapter requires:

  1. the Lease has a holderIdentity that is not empty;
  2. it has both a renewTime and a leaseDurationSeconds;
  3. renewTime is no more than twice leaseDurationSeconds before now.

With the controller's default -lease-duration of 15 seconds, that is 30 seconds. The reasons it gives when a rule fails are, in order: the admission controller's Lease has no holder; ... has never been renewed; ... was last renewed <age> ago, beyond <limit>. A Lease that does not exist is reported too, as Lease <namespace>/<name> does not exist.

The Lease check is the last of the readmission gate's checks, after the ClusterQueue lists the AdmissionCheck and the check names dispatch's controller and is Active. Any failure gives ReadmissionGate no, and the adapter is observe-only. If the adapter may not get the Lease, the gate is unknown, which is also observe-only.

The adapter needs permission to get Leases in the Lease's namespace.

What it does not prove

The heartbeat proves the controller was running when preflight ran. It does not watch afterwards. If the controller goes down after the adapter opens, the adapter does not notice, and jobs that reserve quota wait at their Pending check until the controller comes back. Inspect shows the check's state and message, so the stall is visible, and a new adapter opened during the outage will find the Lease stale and open observe-only.

Reopen the adapter, rather than keeping one open for ever, if the controller has been replaced.

Why a restart does not break it

On a rolling restart the old pod's Lease would normally be released as it shuts down. The controller does not release it (ReleaseOnCancel is off). A released Lease has no holder, which the rule above reads as no heartbeat, so every adapter opened during the restart would be observe-only. Left in place, the old holder's name and last renewal stay on the Lease until the new leader takes it, and the adapter's two-duration window covers the gap. A restart that takes longer than that does show as a stale Lease.

The Lease is not a fence

The controller's safety does not depend on the Lease. Two replicas might each believe they lead for a moment; every write either makes is preconditioned on the version it read, so the second loses. The Lease keeps replicas from duplicating work, and gives the adapter its heartbeat.

If you change the Lease's namespace or name with the controller's flags, set kueue.admissionLease to match. See dispatch-admission.