Skip to content

Read a plan before submitting

plan, err := dispatch.MakePlan(ctx, backend, "", spec, dispatch.WithSupervisor())

MakePlan makes the same decision Submit makes, against the same backend and with the same options, and returns the plan without creating a job or a record. Pass it the options you'll pass to Submit; the job id can be empty, because nothing is recorded.

Use it to show a person what will happen before they commit: who enforces each recovery rule, which causes no rule covers, and whether a shared checkpoint has turned recovery into holds.

Get a backend

MakePlan takes a dispatch.Backend, not a Cluster. dispatch.NewBackend builds one by the adapter's registered name, from the same dispatch.Config that dispatch.Open takes:

backend, err := dispatch.NewBackend(ctx, "slurm", dispatch.Config{Adapter: settings})

Compose it into a Cluster with dispatch.Compose(backend, cfg) when you need one as well, so the plan and the submit ask the same backend. An adapter's own constructor works too; the memory adapter's is memory.New.

Read the refusal first

When MakePlan returns an error, Submit would have refused the same job. The error is classified by a sentinel, and its message says which rule or capability caused it.

errors.Is(err, …) What it means What to do
dispatch.ErrInvalid the spec or policy is malformed: no command, fewer than one node, a DISPATCH_ environment key, a cause covered twice, a budget on a Hold rule, and so on; or an accept option naming a rule the policy doesn't have fix the spec or the option
dispatch.ErrUnsupported a field or extension the job uses is No on this cluster, or Unknown where an ignored request would go unseen; or a rule's action can't be carried out here (Perform is No), or is unverified with no enforcer named drop the field or rule, or for an unverified rule pass AcceptUnverified(rule, enforcer)
dispatch.ErrRejected nobody would enforce a rule, or the adapter gave an incoherent answer pass WithSupervisor() and run a supervisor, or AcceptUnenforced(rule) to accept that the cause fails the job

The order it checks in is fixed, and it stops at the first refusal:

  1. the spec;
  2. the unmatched policy;
  3. the capabilities the spec's fields and extensions need;
  4. whether execution identity is weakened, where the adapter implements IdentityWeakener;
  5. the adapter's rule support: one answer per rule, or ErrRejected;
  6. that every AcceptUnverified and AcceptUnenforced names a rule the policy has;
  7. the native offer, where the adapter implements NativePolicyCompiler and the checkpoint isn't shared;
  8. each rule's enforcer, in policy order.

AttemptInjection must be Yes. It's the capability that gives every execution its DISPATCH_ATTEMPT and DISPATCH_EPOCH, and no option waives it.

Read who enforces each rule

Each entry of plan.Rules pairs a rule with its Enforcer:

for _, a := range plan.Rules {
    fmt.Println(a.Index, a.Effective.Action, a.Effective.On, "enforced by the", a.Enforcer)
}

0 resubmit [preempted node-lost] enforced by the supervisor
Enforcer Chosen when
backend the adapter implements NativePolicyCompiler and its Compile offered the rule
supervisor you passed WithSupervisor()
nobody you passed AcceptUnenforced(rule); a matching cause fails the job, with an escalation

A rule whose Perform support was Unknown takes the enforcer you named with AcceptUnverified, and is also listed in plan.UnverifiedRules.

Rule is what you asked for and Effective is what will be applied. They differ only under a shared checkpoint.

Check what no rule covers

plan.Uncovered lists every cause that can end an execution and that no rule covers. An execution ending with one of these fails the job.

fmt.Println(plan.Uncovered)

[node-lost out-of-memory time-limit non-zero-exit launch-failed no-capacity requeued unknown]

none (success) and cancelled never appear: a policy may not cover them.

Check for substituted holds

If the spec carries the dispatch.SharedCheckpoint{} extension, plan.Checkpoint is shared, and every resubmit, restart or requeue rule becomes a hold for 24 hours (dispatch.SharedHoldFor), so a person decides when it's safe to run again:

a := plan.Rules[0]
fmt.Println("asked for:", a.Rule.Action, "budget", a.Rule.Budget)
fmt.Println("will do:", a.Effective.Action, "for", a.Effective.For)

asked for: resubmit budget 3
will do: hold for 24h0m0s

Isolated checkpoints says why a shared checkpoint can't be recovered automatically.

Check fields proceeding on Unknown

plan.Unverified lists capabilities the job needs whose support is Unknown but which proceed anyway, because a failure would surface as an error rather than as a job quietly running differently. Only OutputRead, OutputFollow and OutputRetained qualify. See capabilities.

Check whether execution identity is weakened

plan.IdentityWeakened is empty when every execution keeps its own identity. When it isn't, the adapter has said why, usually a site opt-in that lets the backend rerun a rank under the identity it already had. Nothing is refused on it, so show it to the person submitting:

if plan.IdentityWeakened != "" {
    fmt.Println("execution identity weakened:", plan.IdentityWeakened)
}

After submit, Status.IdentityWeakened() returns the same reason from the record. See execution identity.

  • ExampleMakePlan and ExampleMakePlan_sharedCheckpoint in the repository's example_test.go are the runnable versions of the snippets above.
  • Choose an unmatched policy covers plan.Unmatched.