Classification¶

Find, Inspect and Watch report each object as a model.ObjectObs: a state, whether an administrator holds it, a cause with a kind, a provenance and a detail, and the execution it is on. This page lists every rule the adapter applies, in the order it applies them, and the exact text it puts in the detail. go/dispatch's classification explanation says what the states and causes mean to the supervisor.

What one classification reads¶

Each object is classified from one read of:

• the TrainJob;
• its JobSet, the one with the TrainJob's name, if there is one;
• in Kueue mode, its Workload, the one labelled kueue.x-k8s.io/job-uid with the TrainJob's UID, if there is one;
• its pods, from one list of the namespace's pods, Terminating ones included, selected by dispatch's job-id and submission labels and separately by JobSet's jobset-name label;
• the last state of pods that have been deleted, which only an open Watch captures;
• the Nodes those pods are bound to, when the adapter may get Nodes.

In the tables below, words means a condition's reason and message as <reason>: <message>, or the reason alone when the message is empty. Kinds and states are written as dispatch prints them: node-lost, non-zero-exit, gated and so on.

The state, and the cause it comes with¶

The first rule that matches decides. Rules 1 to 5 read the TrainJob and, in Kueue mode, the Workload; rule 6 covers everything else.

1. Complete¶

2. Failed¶

The TrainJob condition Failed=True. The state is ended, and the cause depends on the condition's reason:

3. Gated¶

dispatch's own gate is read from the TrainJob alone. Kueue copies a gate's removal to the Workload only while the Workload is active, so a Workload held before Open can still carry dispatch's gate. The two gates explains it.

4. Held by dispatch¶

A dispatch hold is in force when the held object carries dispatch.phpboyscout.uk/hold-instance as a decimal of 1 or more, with no sign or leading zero. The held object is the Workload in Kueue mode and the TrainJob in plain mode.

The detail of a held object is empty apart from the situation notes. The hold's instance is reported as HoldInstance whatever the state.

5. Suspended by someone else¶

6. Pending or running¶

Everything else is running when it is admitted and not suspended, and pending otherwise. Admitted means, in plain mode, always; in Kueue mode, the Workload has Admitted=True. Not suspended means spec.suspend is not true and the TrainJob has no Suspended=True condition.

The cause is the first of these that applies:

Without permission to read Nodes, a running object is never judged node-lost from a Node's Ready condition.

A condition the adapter does not know¶

When the rules leave the cause at none and the TrainJob has a condition that is True and is not Suspended, Complete or Failed, the cause becomes unknown, with no provenance, and the detail TrainJob <type>: <words>, one per such condition, separated by ;. The state is unchanged.

Pod findings¶

A pod reports up to two findings: one from its DisruptionTarget condition and one from its node container's termination. A pod is read the same way whether it is live or a deleted pod's captured last state.

DisruptionTarget=True¶

A finding from the condition is dated by the condition's transition.

The node container terminated¶

A finding from the container is dated by its finishedAt.

Which finding wins¶

The earliest finding among the pods is the cause. On a tie, node-lost beats preempted, which beats out-of-memory, which beats non-zero-exit, which beats anything else. The detail is pod <name>: <the finding's detail>, followed by ; other ranks: <pod> <kind> (<detail>), ... for every other finding, since a rank that exits because its peer died is a consequence, not the cause.

The execution¶

An object's current execution is the set of its pods, live or captured, with the highest R = A + J, preferring the higher A on a tie. A comes from the pod annotation dispatch.phpboyscout.uk/admission and J from the label jobset.sigs.k8s.io/restart-attempt; a pod lacking either is not counted towards one. Execution identity explains the two counters.

Foreign¶

An object is reported Foreign when anything below is true. Each adds its note to the detail.

A label that has been removed contradicts nothing, so stripping labels alone does not make an object foreign.

Situation¶

Last, the detail says what status needs to explain a stall:

The parts of the detail are joined with ;, in the order of this page: the cause's own detail, the foreign notes, then the situation.

Orphans¶

Find also reports what is left of a TrainJob that is gone: a JobSet, a pod or a runtime copy of the job with no live TrainJob of its UID. Each is its own object, always Foreign, with an object reference naming its kind, a cause of unknown with no provenance, and the submission from its name or its submission label.

Inspect reports an orphan the same way for as long as it lasts, even if a TrainJob of its name appears again. Background deletion says why orphans are routine.

Watch¶

The adapter implements go/dispatch's Watcher, so it declares no PolledLoss.

Watch(ctx, job) opens watches on the namespace's TrainJobs and JobSets, on the pods labelled with the job's job-id, and in Kueue mode on the namespace's Workloads. It then runs a Find and sends the observation of every object of the job. After that, each change to the job's TrainJobs (by name prefix or job-id label), JobSets (by name prefix), pods, or the Workload of an object already sent, makes it run Find again and send each observation that differs from the last one sent for that object, ignoring the time it was seen.

The stream closes, without an error, when any of the watches expires or disconnects, when a Find fails, or when ctx ends. The caller subscribes again. go/dispatch's supervisor does so on its next pass, and treats what the stream carries as a prompt: it records only what its own Find shows.

Watch is how the adapter keeps a deleted pod's evidence. When a watched pod is deleted, the adapter keeps its last state, exit code and DisruptionTarget included, against its TrainJob, up to the latest 256 pods per TrainJob. Every later classification by that Backend reads those pods as well as the live ones, which is how a TrainJob whose pods are already gone can still be classified from them. The evidence:

• is kept only while a Watch is open on that Backend. A pod deleted while no stream is open, including in the gap between a stream closing and the next subscription, leaves none;
• lives in that Backend's memory. Another process, or this one after a restart, does not have it;
• is only for pods that carry the job's job-id label.

Without it, a TrainJob that failed with FailedJobs after its pods went is classified unknown, with no pod evidence was captured in the detail.