Skip to content

The event-sourced record

Schedulers forget. Slurm drops a finished job after MinJobAge, five minutes by default. Kubeflow deletes pods, and their exit codes and causes with them, on every restart, preemption and hold. Cancelling on Kubernetes means deleting. So dispatch keeps its own record of every job, and never reconstructs history from what the scheduler happens to remember.

Written once, then appended to

A record has two parts.

The header is written once, when the job is created, and never changes. It holds the spec, a digest of it, the plan, the adapter's declarations at the time, the number of Start retries allowed, who created it and when, and the job's lineage: the job it supersedes, and its generation.

Recording the plan and the declarations matters. Every actor that reads the record, whatever its own configuration, decides with the same plan and the same node-loss window and call bound.

Everything after creation is an entry appended to the history. There are two kinds:

  • control intents, what a person wants: cancel, hold, release, requeue, give up, resubmit and abandon. They're appended by the Operator;
  • facts, everything else: the supervisor's decisions (authorize a submission, choose a canonical object, retire one, commit an outcome, escalate, settle), the calls it's about to make (an attempt number before each Start, a deadline before each hold or requeue call), and the evidence it gathered (what the scheduler showed, what a Start replied).

Entries are never changed or removed. A commit can only add them, and only if the revision it read is still current.

Fold, Project and Next

Three pure functions turn the record into everything else. None of them does I/O, and each takes the current time as an argument rather than reading a clock.

  • Fold(view, now) replays the history over the header and returns a Job: the desired state (run, hold or cancel), the outcome if one was committed, the submissions with their attempts and canonical objects, every object ever recorded, the holds, the recorded call deadlines and the escalations. Every actor that folds the same revision gets the same Job.
  • Project(job, discovery, now) adds what the scheduler shows now and returns the Status a caller reads.
  • Next(job, discovery, now) returns the steps that should happen next. The supervisor carries them out; see the steps.

Because they're pure, the simulator can drive the real ones, and any actor (a supervisor, a CLI answering status, a test) computes the same answer from the same inputs.

Fold refuses a history the protocol couldn't have written, with ErrInvalid: an entry that's both or neither kind, submission 3 authorized after submission 1, a trigger key reserved twice, a second canonical object for one submission, an outcome that isn't terminal. It never folds such a record into something plausible.

Desired state and outcome

Control intents change what the user wants. Most of them leave the outcome alone, and the supervisor commits outcomes as facts. Two are different, because a person must be able to end a job whatever any supervisor is doing:

  • cancel sets the desired state to cancel and the outcome to cancelled at once, lifts every hold, drops any pending requeue or resubmit, and marks clean-up pending;
  • give up sets the outcome to failed, with the cause of the recovery hold it ends if there is one, and an escalation.

An outcome, once committed, never changes. A cancel that arrives after the job succeeded leaves the outcome succeeded; its receipt says so.

Schemas that refuse what they don't know

Each entry is stored as a schema name and a JSON body: dispatch.control.v1 for control intents and dispatch.fact.v1 for facts. The header is dispatch.header.v1. Every body is decoded strictly: an unknown schema, an unknown field, trailing data or an unknown enum name is refused, with records.ErrSchema or records.ErrUnknownName, and never skipped.

Control intents are kept in a schema of their own for a reason. Facts change as the protocol does, and a new fact gets a new schema name that older builds refuse. Control intents are frozen. So an older build that can't decode a newer record's facts can still append a cancel to it.

Settling

A record with an outcome stays open until the settled predicate holds: every object ever recorded, and every object the latest Find returned, has ended or been verified stopped, judged on a Find taken after every recorded requeue call has landed or expired; no foreign object is undisposed; no submission has every attempt unknown; and under the manual unmatched policy, no attempt is unmatched. The supervisor then commits it settled, and the store sets the dispatch.settled and dispatch.settled-at labels so List can find records that are, or aren't, settled.

A settled record can be tombstoned. records.Records requires the store to keep its idempotency entry when it does, so a late Submit with the same key can't recreate the job.

The store

The core reads and writes records through the records.Records interface: create once, read, commit entries only while the revision holds, list by the settled labels, and tombstone. Ownership is the optional records.Owner; the owner's display-only loop health is the optional records.Health. records.CommitControl is how control intents are committed: it re-reads and retries on a revision conflict until its context ends, so a stream of evidence commits can't starve a cancel.

recordstest.Memory is the in-memory implementation, with ownership, for tests. The durable store a host uses is records/ledgerstore, which keeps the record in a go/ledger store (spec 0002); the simulator runs it too, over go/ledger's file store on a crash-modelling filesystem. Record stores says how to open one.