Escalations¶

Id	Committed when	Committed with
`foreign:<object id>`	`Find` shows an object with the job's name whose submission and attempt can't be read, and it hasn't been disposed of	nothing else; it blocks settlement until a person abandons it
`admin-held:<object id>`	`Find` shows a gated object held by someone other than dispatch: on Slurm a `JobHeldAdmin` hold, on Kubeflow another entry left in the `admission-gated-by` annotation	nothing else; the object is never opened
`all-unknown:<N>`	every attempt of open submission N is unresolved, none was refused, no retries are left (or the job is cancelled or finished), and the last attempt is at least the job's unresolved age old	nothing else; it blocks N+1 and settlement
`unmatched:<N>/<a>`	under the manual unmatched policy, attempt a of submission N has been unresolved for at least the job's unresolved age	nothing else; it blocks settlement
`refused:<N>`	the scheduler refused every attempt of submission N	the outcome `failed`, cause `launch-failed` with the scheduler's words
`failed:<cause kind>`	an execution ended with a cause no rule recovers: no rule covers it, its rule is `escalate` or enforced by nobody, its budget is spent, or the supervisor can't carry out its action	the outcome `failed`, with that cause
`overdue:<hold instance>`	the hold in force is past its deadline	nothing else; the hold stays
`cleanup-deadline`	a terminal job's record is still unsettled more than the job's clean-up deadline after the cancel, or after the outcome when there was no cancel	nothing else; the detail names the objects still alive
`give-up`	a person called `GiveUp`	committed by `Fold` from the control intent, with the outcome `failed`

<object id> is the scheduler's id from the object's ref: a Slurm job id, or namespace/name on Kubernetes. <cause kind> is a cause name such as node-lost.

The unresolved age and the clean-up deadline are the job's own: 10 and 30 minutes, unless dispatch.WithUnresolvedAge and dispatch.WithCleanupDeadline set them at submit. They are recorded in the header, so every supervisor uses the same ones.

Each id is committed at most once per job, and Job.Escalated(id) reports whether it has been. The supervisor commits all of them but give-up, which GiveUp brings with it. The Detail says, in words, what happened.

An escalation is delivered by the supervisor's Deliver function, if one is set (supervise.WithDeliver), and marked delivered in the record when that returns nil. Until then Escalation.Delivered is zero.

Receive and answer escalations says what to do about each.