Holds are Workload deactivation¶

A hold is go/dispatch's way of saying "stop this object and do not run it again until I say so". The supervisor uses it, for example, to wait until a lost node's processes are provably gone before it recovers a job. A hold has an instance number, so that a late call to place or lift an old hold cannot undo a newer one.

In Kueue mode, the Workload is held¶

Hold is one update of the job's Workload: spec.active becomes false, and the hold's annotations are written on the Workload in the same update. The TrainJob is not touched. Kueue evicts a deactivated Workload, suspends the TrainJob itself, and Trainer deletes the pods.

The adapter never writes spec.suspend in Kueue mode. Kueue owns that field. Two controllers writing it fight, and the loser's change is silently undone. Deactivating the Workload is Kueue's own way of saying stop, so the hold is something Kueue respects and reports.

Release reads the Workload, refuses unless the hold annotation names the instance it was asked to lift, and in one update sets active back to true and removes the hold annotations. The next execution then goes through Kueue admission and the readmission gate, so A rises (the two gates). The gate's pod check means the release does not run the object until the held execution's pods are gone (never two executions).

Kueue's own deactivations carry no hold-instance annotation, so dispatch's holds and someone else's can be told apart, and a deactivation Kueue made for its own reasons (a time limit, a requeueing limit) is classified from its reason, not as a hold.

If the TrainJob has no Workload, Hold returns ErrRejected. Start waits for the Workload, so every started object has one.

In plain mode, the TrainJob is held¶

The same, on the TrainJob itself, with spec.suspend: true as the hold. Release removes the hold annotations and, only if the object was opened and has not ended, unsuspends it and raises A, in the same update, and only when no pod of it remains. If the pods are still there it writes nothing and returns ErrPodsRemain.

An object that was held before it was ever opened stays suspended after Release, back at its start gate, waiting for Open. A release must never run what the gate still holds. The reverse also holds: Open on an object with a hold in place writes only the opened mark and does not unsuspend, so a stale Open cannot lift a hold.

The annotations¶

All of them go on the held object, the Workload in Kueue mode and the TrainJob in plain mode, in the same update as the hold, and Release removes all of them:

Instances, and stale calls¶

Every write is an update preconditioned on the resourceVersion of the read it was decided on. That defeats a call delayed between its read and its write: if hold 2 lands in between, the version has changed, and the stale write fails.

A conflict is not simply retried, because re-reading and re-writing would throw away what the precondition protected. After a conflict the adapter checks that the call's deadline has not passed and re-reads the instance in force:

• Hold(m): if m.Instance is already in force, it succeeds, since an earlier try landed. If a newer instance is in force, or, after a conflict, an instance other than the one the last read saw, it returns kubeflow.ErrStaleHold. Otherwise the conflict came from some other writer, a controller's status update for example, and the adapter tries again. Those conflicts are routine, since Kueue and Trainer update status every few seconds.
• Release(m): if m.Instance is in force, it tries again; any other instance, or none, is ErrStaleHold.

Each call retries at most eight times, and then fails with a conflict error.

ErrStaleHold is classified as ErrRejected: the call was refused because a newer one has overtaken it.

Deadlines¶

A precondition does not defeat a call delayed before its read: that call reads the current state, hold 2 included, and its write then succeeds. What prevents that is the deadline. Every hold-family call carries CallMeta.Deadline, which the adapter checks immediately before it sends each request, and which is also the request's context deadline, so nothing goes out after it. A call with no deadline is ErrInvalid; one whose deadline has passed is ErrRejected.

A request already on the wire can still commit after its deadline, up to the apiserver's request timeout. So the adapter declares a CallBound to dispatch that is a little longer than timeouts.apiserverRequest (the value). The supervisor places a new hold instance only after every earlier call has landed or expired at its deadline plus that bound. Set timeouts.apiserverRequest to the apiserver's --request-timeout; if it is lower than the real value the bound is too small.

When Hold is not available¶

Hold is a probed capability. If preflight's hold round trip failed, or in Kueue mode the adapter may not update Workloads, Hold and Release return ErrUnsupported with the reason. See Read a preflight result.