Skip to content

Choose an unmatched policy

plan, err := op.Submit(ctx, spec, by, dispatch.WithUnmatched(dispatch.UnmatchedManual))

WithUnmatched decides how an unmatched attempt, a Start call whose outcome stayed unknown and to which no object was ever matched, affects whether the job's record can settle. Leave it out and you get dispatch.UnmatchedResidual. It's recorded in the plan as plan.Unmatched, so it can't be changed for a job after submit, and every supervisor of the job applies the same one.

When an attempt goes unmatched

The supervisor commits each attempt's number before it calls Start, and records the reply after. A reply that's an error other than ErrRejected, ErrInvalid or ErrUnsupported, or a reply that never came, is recorded as unknown, because the object may have been created anyway: a timed-out sbatch still creates its job. The attempt is matched later if an object carrying its submission and attempt number turns up in Find. Until then it's unmatched.

The supervisor retries an unknown Start up to two more times for the same submission (records.DefaultSubmitRetries), or as many as WithSubmitRetries set at submit. Any duplicate that lands is created gated, so it never runs, and is retired and stopped once the canonical object is chosen.

Pick a policy

UnmatchedResidual (default) UnmatchedManual
An unmatched attempt blocks settlement no yes, until a person abandons it
An unmatched attempt escalates no yes, once, the job's unresolved age after the attempt: 10 minutes unless WithUnresolvedAge set it (unmatched:<N>/<a>)
After settlement Job.Residuals() lists the attempt, for the host to keep sweeping nothing is left over
A submission whose every attempt is unknown escalates once (all-unknown:<N>) and blocks, under both policies same

Choose residual when unknown outcomes are common and nearly always mean the call never reached the scheduler, which is the usual case over an unreliable connection. Records settle without a person, and the host stops any late object by name when it appears.

Choose manual when nothing may be left behind unexamined, and someone will answer escalations. A record stays open until each unmatched attempt is closed.

Close an unmatched attempt

Under either policy, a person closes what escalated with Abandon, naming the target the escalation names:

human := dispatch.Actor{Kind: dispatch.ActorHuman, ID: "ada"}

// unmatched:2/1, one attempt
err := op.Abandon(ctx, job, dispatch.AbandonTarget{Submission: 2, Attempt: 1}, human, "checked squeue, never created")

// all-unknown:2, every unresolved attempt of the submission
err = op.Abandon(ctx, job, dispatch.AbandonTarget{Submission: 2}, human, "never created")

Abandon refuses with ErrUnauthorized unless the actor is a human, and with ErrInvalid unless the escalation naming its target was committed first. A foreign object needs no escalation of its own, only to be recorded as foreign; see foreign:<object id>. It's allowed after a cancel. Abandoning a whole submission whose every attempt was unknown lets the supervisor authorize the next one.

Sweep residuals after settlement

dispatch.Job.Residuals() returns the unmatched attempts of a folded record as AbandonTargets. dispatch doesn't sweep them itself. The host is expected to find the job's objects by name after the record settles, and stop any that are alive with the per-object Backend.Stop.