Background deletion¶

Stop deletes the TrainJob with a DELETE whose preconditions.uid is the UID from the object reference and whose propagationPolicy is Background. Background propagation means the TrainJob is removed at once, and Kubernetes' garbage collector deletes the things it owns (the JobSet, its Jobs, the pods and the runtime copy) afterwards.

It was not always so. The adapter's first design deleted in the foreground, so that the TrainJob stayed visible until everything beneath it was gone, which would have made "the TrainJob exists" mean "something of it may still run". On Trainer v2.3.0 that never completes, in two cases that were observed on a kind cluster.

What goes wrong in the foreground¶

The TrainJob never finishes terminating. The adapter gives each job a copy of its runtime, a TrainingRuntime with the TrainJob's own name, and the collector deletes that copy along with the other dependants. Trainer's validating webhook then refuses the update that lifts the TrainJob's foregroundDeletion finalizer, because the runtime the TrainJob names no longer exists. The TrainJob stays Terminating for ever. This is kubeflow/trainer#4203, open against v2.3.0 at the time of writing.

A suspended TrainJob being deleted gets its JobSet back. Trainer recreates the JobSet each time the collector deletes it: a new JobSet UID about every three seconds, for 30 seconds to over a minute.

Background deletion avoids both: the TrainJob is gone at once, so there is no finalizer to wedge and nothing left to recreate the JobSet.

What it costs¶

Because the TrainJob is gone before its pods, processes are tracked by pods, not by the TrainJob. A pod on a reachable node goes when its kubelet confirms termination; one on an unreachable node stays until the Node is deleted or tainted node.kubernetes.io/out-of-service.

While they terminate, Find reports the leftover JobSet and pods as orphans of a TrainJob that is gone. Each orphan is its own object with Foreign set, and an object reference whose ID is <namespace>/jobset/<name>, <namespace>/pod/<name> or <namespace>/trainingruntime/<name>, with that object's UID. dispatch escalates a foreign object and it blocks settlement. Stop on such a reference deletes that one object, under its UID precondition. Nothing is re-admitted while the pods remain (never two executions).

An orphan is found the same way a stray of an earlier crash is: by its name prefix and labels. A runtime copy that Start created but never got to own, because the adapter died between the two steps, is found the same way and deleted by StopJob. Because orphans are routine, writing requires delete on JobSets and pods: an adapter without it opens observe-only rather than leave a StopJob it cannot finish.

Stop and StopJob¶

• Stop deletes one object instance. A missing object is success. A UID that no longer matches, meaning the name now belongs to another instance, is success too: the instance the reference names is gone, and Stop must not touch its replacement.
• StopJob deletes every object Find returns, orphans included, each under its own UID precondition. It then repeats Find until nothing of the job remains or timeouts.stopJob passes (ten minutes by default). It reports, per object:

Unproven needs permission to read Nodes. Without it, nothing is proven and remaining pods count as Alive.

Cancel means delete: a stopped object leaves no trace in the cluster. dispatch's record remembers it, and AccountingHistory is no.