Skip to content

Receive and answer escalations

sup, err := supervise.New(backend, store, supervise.WithDeliver(
    func(ctx context.Context, job dispatch.JobID, e dispatch.Escalation) error {
        return pager.Send(ctx, fmt.Sprintf("dispatch job %s: %s (%s)", job, e.Detail, e.ID))
    },
))

An escalation is dispatch asking a person to decide something it won't decide alone. After each pass's steps, the supervisor calls your Deliver function for each escalation the record held, undelivered, when the pass began, and records it as delivered when your function returns nil. An escalation the pass itself committed is delivered after the next pass; when that pass settled the job, the supervisor's last look, which finds the record settled, delivers it before Run returns. Return an error and it's offered again after the next pass, except on that last look: once the job has settled, Run returns after one attempt.

pager.Send stands for whatever reaches a person in your setup.

Read escalations without a supervisor

Escalations are in the record, not in Status. Fold the record to read them:

v, err := store.Read(ctx, job)
if err != nil {
    return err
}

j, err := dispatch.Fold(v, time.Now())
if err != nil {
    return err
}

for _, e := range j.Escalations {
    fmt.Println(e.ID, e.At, e.Delivered.IsZero(), e.Detail)
}

Each escalation is committed once, keyed by its ID, so you can deduplicate on it.

Answer it by its id

The id's prefix says what happened. All the calls below are on the cluster's Operator, and each needs an Actor naming who answered.

overdue:<instance>, a hold past its deadline

The hold stays in force; dispatch never releases or fails a job on its own. Decide, then:

  • for a hold an actor placed, Release(ctx, job, by) lets the job run again;
  • for a recovery hold, which a hold rule placed after an execution ended, Release is refused. Use Resubmit(ctx, job, by, reason) to run a new submission, Cancel, or GiveUp(ctx, job, by, reason) to fail the job with the cause that ended it.

Status.Hold.Kind says which kind it is. Releasing a hold someone else placed works, and the record notes that you overrode it.

all-unknown:<N> or unmatched:<N>/<a>, a launch whose outcome nobody knows

A Start call's outcome stayed unknown for at least the job's unresolved age, 10 minutes unless WithUnresolvedAge set it, and no object turned up for it. Check the scheduler for an object named dj-<job id>. If there's none, close it:

target := dispatch.AbandonTarget{Submission: 2}             // all-unknown:2
target = dispatch.AbandonTarget{Submission: 2, Attempt: 1}  // unmatched:2/1
err := op.Abandon(ctx, job, target, human, "no such job in squeue")

Only a human actor may abandon. After an all-unknown submission is abandoned, the supervisor launches the next one. Choose an unmatched policy explains when unmatched escalations happen.

foreign:<object id>, an object with the job's name that isn't dispatch's

The scheduler has an object named for this job whose submission and attempt can't be read from it, perhaps because a site plugin rewrote it. It blocks the record from settling. dispatch never stops or adopts it. Find out what it is, stop it on the scheduler yourself if it shouldn't run, then dispose of it in the record:

err := op.Abandon(ctx, job, dispatch.AbandonTarget{Foreign: ref}, human, "site plugin rewrite, stopped by hand")

ref is the object's ObjectRef as recorded, from j.Object(...) or the job's Objects.

admin-held:<object id>, a gate an administrator holds

A launched object is held by someone other than dispatch: on Slurm a JobHeldAdmin hold, on Kubeflow another gate left in the TrainJob's kueue.x-k8s.io/admission-gated-by annotation (see the start gate). dispatch never opens it. Ask whoever holds it, or cancel the job.

failed:<cause> or refused:<N>, the job failed

The outcome is already committed and final. The detail says why: no rule covers the cause, the rule's budget is spent, or (for refused) the scheduler refused every attempt to launch. To run the job again, create a new job from it:

plan, err := op.Supersede(ctx, job, human, "fixed the data path", dispatch.WithSupervisor())

The new job has the same spec, a new id, and names the old one in Status.Supersedes. Its executions get a higher DISPATCH_EPOCH, so they outrank every checkpoint the old job published and can still resume from them.

cleanup-deadline, a cancelled job still stopping

The job has an outcome, and its record still hasn't settled more than the job's clean-up deadline after the cancel, or after the outcome if nobody cancelled. The deadline is 30 minutes unless WithCleanupDeadline set it. The detail names the objects still alive. Either something of the job is still alive on the scheduler, or something else holds the record open: a launch whose outcome is unknown, or a foreign object, each of which has its own escalation above. For objects that won't stop, stop them on the scheduler or with cluster.Stopper().EmergencyStop(ctx, job, by, reason). This escalation is committed once per job.

give-up

Someone called GiveUp. It records the decision; there's nothing to answer.