Skip to content

Why classification never collapses to "failed"

A multi-node training job stops for many reasons, and most of them aren't failures. A higher-priority job preempted it. A node died under it. The scheduler killed it at its time limit. It ran out of GPU memory. The training script exited 1. Each wants a different response: preemption is usually worth retrying straight away, a lost node is worth retrying elsewhere, and a script that exits 1 will exit 1 again. A tool that reports all of these as failed has thrown away the one thing you needed to know.

So dispatch reports a job as two things: a coarse state, and a cause that says why the job is in it.

State and cause

There are eight states: pending, held, running, recovering, succeeded, failed, cancelled and unknown. They answer "is it doing anything, and is it finished?"

The cause carries the rest:

  • a kind, from a closed list: none, preempted, node-lost, out-of-memory, time-limit, non-zero-exit, launch-failed, no-capacity, held, released, requeued, cancelled, duplicate, unknown;
  • a provenance: whether the scheduler said so, the application said so (an exit code), or an actor did it (a person cancelling, a supervisor holding);
  • the actor, when there was one;
  • the detail: the scheduler's own words, kept verbatim, such as Slurm's NODE_FAIL reason;
  • the exit code, when there was one, and when it happened.

The kind list is closed, with unknown for a scheduler answer the adapter can't map. It's never stretched to fit: an adapter that sees something new reports unknown with the scheduler's words in the detail, rather than guessing non-zero-exit. And unknown is a thing the scheduler said. A scheduler you can't reach is an ErrUnreachable error, never a state.

Recovery rules are written against causes

Because causes survive, a recovery policy can name the ones it covers:

{On: []dispatch.CauseKind{dispatch.CausePreempted, dispatch.CauseNodeLost}, Action: dispatch.ActionResubmit, Budget: 3}

Preemption doesn't spend a budget meant for real failures, because a rule for one doesn't cover the other. RecoveryPolicy.Validate refuses a cause covered by two rules, so there's never a question of which applies. And Plan.Uncovered lists, before the job starts, every cause no rule covers.

Some causes can't be covered at all. cancelled from the scheduler is final: a cancel that reached the scheduler is never undone by recovery, even if the record missed it. none is success. unknown may only be covered by a hold or escalate rule, because resubmitting on a cause nobody understands is a guess.

An object ending is not the job ending

The scheduler knows about objects: a Slurm job, a Kubeflow TrainJob. dispatch knows about the training job, which may need several objects over its life. When an object ends, the scheduler is done with it, but the job isn't necessarily done. The plan decides:

  • a rule with budget left: the job is recovering, and the supervisor starts the next submission;
  • a hold rule: the job is held under a recovery hold, until a person resubmits, cancels or gives up;
  • no rule, a spent budget, an escalate rule, or a rule nobody enforces: the job is failed, with the cause that ended the execution, and an escalation.

Only a committed outcome is terminal. succeeded, failed and cancelled never change once committed. Running a failed job again is Supersede, which creates a new job naming the old one.

recovering never goes back to pending. A job whose second submission is waiting to start is recovering, because it's been taken back to run again, and that's different from a job that has never run.

Today the supervisor carries out resubmit and hold itself. restart and requeue are carried out only by a backend that enforces them natively; a rule with one of those actions assigned to the supervisor fails the job, with an escalation saying supervisor recovery for that action isn't implemented.

Where the state comes from

No adapter computes a job's state. The adapter reports what it sees of each object (its state, cause and executions), and dispatch computes the job's state in one place:

  • dispatch.Fold replays the job's record into a Job: its submissions, holds, outcome and escalations;
  • dispatch.Project combines that Job with what the backend shows now (a Discovery) to give the Status a caller reads.

Both are pure functions with the time passed in, so the state can't drift between adapters or between the processes reading it. The event-sourced record covers the record they read.

This is also why Status.History holds the transitions committed to the record (recoveries, holds, outcomes) and not running. Whether a job is running comes from what the scheduler shows when you ask; it's in Status.State, not the history.

The node-loss fence

Sometimes a scheduler never says the job ended. A Kubeflow job that loses a node can sit at FailureTarget indefinitely, reporting itself running. Each adapter declares a node-loss window (Declarations.NodeLossWindow): how long an object may report running with cause node-lost before dispatch treats the execution as ended.

Inside the window the job is running with cause node-lost: the one case where a running job carries a cause. Past it, Next asks for the object to be stopped, then applies the plan for node-lost. A replacement may be started, but its gate isn't opened until the scheduler shows the old object ended.

Without a record

A reader with no record store can still ask what the scheduler shows. Its Status has Level object-only, and its state is pending or running, from whether any object of the job is alive. It's never terminal, because an object ending isn't the job ending, and only the record knows which it was.

Refusing what it doesn't understand

Every state, cause kind and other enum encodes as its name, and decoding refuses a name it doesn't know. A record written by a newer build that added a cause is refused by an older one, rather than misread as something it isn't.