Skip to content

The record's schemas

Schema Go type Written Holds
dispatch.header.v1 records.Header once, by Create the spec, plan and lineage
dispatch.control.v1 records.Control appended by the Operator a person's control intent; frozen
dispatch.fact.v1 records.Fact appended by the supervisor a decision, a call about to be made, or evidence
Label Value
dispatch.settled false at create; true from the commit that settles the record
dispatch.settled-at the settling commit's time, RFC 3339, UTC
dispatch.name the job's discovery name, dj-<job id>

Bodies are JSON. records.EncodeHeader, EncodeControl and EncodeFact write them, and DecodeHeader and DecodeEntry read them strictly: an unknown field, trailing data or an unknown schema is records.ErrSchema, and an unknown enum name is refused. Times are RFC 3339 with nanoseconds; durations are integer nanoseconds; enums are their names (below).

Field Type Notes
schema string dispatch.header.v1; a build that doesn't know the value refuses the header
job job id a 26-character upper-case ULID
spec JobSpec as submitted
specDigest string hex SHA-256 of the spec's JSON encoding; a repeated idempotency key with a different digest is refused
plan Plan as MakePlan returned it
supersedes job id omitted for an original job
generation int 0 for an original job; the superseded job's plus 1
declarations object the adapter's at create: nodeLossWindow, polledLoss (evidence name to duration), callBound
submitRetries int how many times an unknown Start may be retried per submission: WithSubmitRetries' value, 2 by default
cleanupDeadline duration WithCleanupDeadline' value; omitted for the default, 30 minutes
unresolvedAge duration WithUnresolvedAge' value; omitted for the default, 10 minutes
createdBy actor
createdAt time

Control

Field Type Used by
kind control kind all
by actor all
reason string all but release
at time all
until time hold: its deadline
target object: submission, attempt, foreign (an object ref) abandon
override bool release: the releaser isn't the holder
instance int release, resubmit: the hold instance the intent was aimed at, so a retry landing after that hold was replaced leaves the new one alone; absent means whichever hold is active

Control kinds: cancel, hold, release, requeue, give-up, resubmit, abandon. (unset exists as the zero value and is refused by Fold.)

Fact

Every fact has kind, by and at. The other fields are set only for the kinds that use them:

Field Type Fact kinds
submission int authorize, attempt, reply, canonical, hold-placed
triggerKey string authorize: the budget reservation; unique per record
rule int authorize: the rule's index, or -1 for none
attempt int attempt, reply, canonical
reply reply reply
ref object ref reply, canonical, opening, retire, verified-stopped, hold-call, hold-refused, requeue-call, restore-call, cessation
findAt time verified-stopped: when the Find the judgement rests on was taken
obs object observation observe
outcome state outcome: succeeded, failed or cancelled
cause cause outcome, hold-placed
escalation string escalate, delivered: the escalation id
detail string escalate, reply, hold-refused, cessation
hold hold hold-placed: the recovery hold
instance int hold-call, hold-refused
op call op hold-call
ceased bool cessation: the site's signal says the object's processes are gone; omitted when it doesn't
deadline time hold-call, requeue-call, restore-call: the call's deadline, committed before it's sent
expires time hold-call, requeue-call, restore-call: the deadline plus the adapter's callBound

Fact kinds:

Kind Class Records
authorize decision submission N, with its trigger key and rule
attempt decision attempt a of submission N, before its Start
reply evidence what that Start returned
observe evidence an object as the scheduler showed it
canonical decision submission N's canonical object, once
opening decision the decision to open an object's gate
retire decision an object retired, permanently
verified-stopped decision an object judged stopped from its absence
outcome decision the job's terminal state and cause
escalate decision an escalation, by id
delivered evidence an escalation delivered to a person
hold-placed decision a recovery hold, placed by a hold rule
hold-call decision a hold placement or release about to be sent
hold-refused evidence or decision a hold the scheduler or adapter refused
requeue-call decision a requeue about to be sent
restore-call decision a restore of a requeue bit left set, about to be sent
stop-job decision a StopJob sweep ran
settled decision the settled predicate held
cessation evidence a site cessation signal's answer for one object; a ceased answer lifts the hold a shared checkpoint substituted

(unset exists as the zero value and is refused by Fold.) Cancel also commits a stop-job fact after its own sweep.

Reply Call op
none, created, refused, unknown place, release (unset as the zero value)

JobSpec

Field Type Notes
name string a label for people, not an identity
command string array required; no NUL bytes
env object no key starting DISPATCH_
workDir string needs WorkingDirectory
nodes int at least 1
gpusPerNode int not negative
procsPerNode int not negative; needs ProcsPerNode when set
timeLimit duration per execution; not negative
checkpoint string an absolute POSIX path or s3://bucket/prefix
recovery object rules: an array of rules
extensions array each {"namespace", "name", "value"}; decoding refuses an unregistered type

dispatch's own extensions are in namespace dispatch: Image ({"ref"}), SharedCheckpoint ({}) and Output ({"stdout", "stderr"}).

Rule

Field Type Notes
on cause kind array at least one; a cause in one rule only; never none, cancelled or duplicate; unknown only with hold or escalate
action action resubmit, restart, requeue, hold, escalate
budget int at least 1 for resubmit, restart and requeue; absent otherwise
for duration positive for hold; absent otherwise

Plan

Field Type Notes
job job id
rules array each {"index", "rule", "effective", "enforcer", "firings"}; firings is filled in by status, not stored
uncovered cause kind array causes that end an execution and that no rule covers
unverified capability array needed capabilities proceeding on unknown
unverifiedRules rule array rules accepted with AcceptUnverified
checkpoint checkpoint mode isolated, shared
unmatched unmatched policy residual, manual
native object the accepted native offer: rules (indexes), data (base64, the adapter's own encoding)
identityWeakened string the adapter's reason the job's execution identity is weakened; absent when it is not. See execution identity

Shared shapes

Actor

{"kind", "id"}. Kind: human, supervisor, adapter. (unset is the zero value, and an actor with it, or with no id, is refused.)

Cause

Field Type
kind cause kind
provenance none, scheduler, application, actor
actor actor, when an actor caused it
detail the scheduler's own words
exitCode int, when there is one
at time

Cause kinds: none, preempted, node-lost, out-of-memory, time-limit, non-zero-exit, launch-failed, no-capacity, held, released, requeued, cancelled, duplicate, unknown.

Hold

{"kind", "by", "since", "until", "reason", "overdue"}. Kind: backend or recovery.

Object ref and observation

An object ref is {"backend", "id", "uid", "submit"}: the registered adapter name, the scheduler's id, the Kubernetes UID where there is one, and the Slurm submit time where there is one.

An observation (obs) is {"ref", "submission", "attempt", "foreign", "execution", "state", "adminHeld", "holdInstance", "requeueBit", "cause", "seen"}. Its execution is {"token", "restart", "nodes", "started", "ended"}, and its state one of gated, pending, held, running, ended.

Other enum names

Type Names
state pending, held, running, recovering, succeeded, failed, cancelled, unknown
enforcer backend, supervisor, nobody (unset is the zero value)
tristate unknown, yes, no
source declared, probed, catalogue
evidence state, cause, exit-code, execution
level job, object-only
clean-up state clear, pending, blocked