Run the simulator and replay a seed¶

go test ./internal/sim -run TestSim -seed=42 -simtrace -v

This replays seed 42 and prints its whole trace. A seed fixes everything about a run, so replaying it reproduces a failure exactly, as many times as you like.

The simulator lives in internal/sim. It isn't importable from outside the module; it's how the supervisor's protocol is tested. How the simulator tested the supervisor explains what it does and why.

Run a batch¶

go test ./internal/sim -run TestSim -count=1 -v

With no flags this runs 20 seeds, from seed 1, each twice to check it's deterministic. The last log line reports how many seeds passed, then how often each fault and protocol event happened across them, as a map from event name to count: 20 seeds passed; event counts map[cancel:… crash:… …].

-count=1 stops Go from reusing a cached pass.

These are test flags, so they go after the package path. TestKnobs adds two more, -knob and -knobseeds; see the knobs.

The simulator runs inside a testing/synctest bubble on a fake clock, so a seed that simulates hours of scheduler time takes milliseconds of real time. Large batches still use memory: cap them on a shared machine, for example systemd-run --user --scope -p MemoryMax=4G env GOMEMLIMIT=3GiB go test ./internal/sim ....

Read a failure¶

A failing seed stops the batch. The last lines give the seed's configuration, the invariant it broke, and the command that replays it. This one is real: seed 1 with the RetireCanonical knob turned on, so a fix is deliberately missing (see the knobs):

seed 1 shape slurm sups 3 unmatched residual faults {lostReply:true lateLanding:true failBefore:true lostCommit:true crash:false stop:false revoke:true powerCut:false omit:true}: 1502 steps
    invariant 6: retired handle 1 was opened
    replay: go test ./internal/sim -run TestSim -seed=1

Above that, the test log holds the trace, one line per scheduler step, then a dump of the record and the scheduler's ground truth at the failure:

 1473 00:09:24.651 sup2#0 Find
 1474 00:09:24.651 sup2#0 Commit retire
 1475 00:09:24.651 sup1#0 fs open .tmp-d9fecc14e2ed9629
 1476 00:09:24.651 sup1#0 fs sync .tmp-d9fecc14e2ed9629
 1477 00:09:24.651 sup2#0 flock tx.lock
 1478 00:09:24.732 sup0#0 Commit retire
 ...
 1482 00:09:24.732 sup1#0 fs rename record
record rev 35 desired 1 outcome &{State:failed ...} settled false holds 4 requeue false
  sub 1 key initial disp 1 canonical &{memory 1  2000-01-01 00:01:25 +0000 UTC} attempts: [1 reply 3 matched false abandoned false] [2 reply 1 matched true abandoned false]
  obj 1 n1 a2 retired true ended false vstop false seen true foreign false disposed false last held
  hold 4 kind backend active true refused false until 01:05:06
  escalation give-up: sim
  truth 1 n1 a2 held opened true hold 4 purged false foreign false

Each trace line is the step number, the fake time, the process and its incarnation (sup0#1 would be the first supervisor after one restart), and what it did. A deliver line is a call the fault layer held back and landed later. On a seed that runs the file store, as this one does, every filesystem and lock call is a step too (fs, flock). In the dump, obj lines are what the record says about each object and truth lines are what the memory adapter knows: here the record has object 1 retired while the scheduler shows it opened.

Replay a seed¶

go test ./internal/sim -run TestSim -seed=1 -v

The replay prints the same trace, because the run is deterministic: same seed, same configuration, same order of every call. Add -simtrace to see the trace of a seed that passes.

The replay command a failing TestSim prints doesn't carry a knob. To replay a seed a knob was caught at, turn the same knob on, as the line TestKnobs logs for each caught knob does:

DISPATCH_MUTANT=RetireCanonical go test -tags dispatch_mutants ./internal/sim -run TestSim -seed=1 -v

If -twice reports "is not deterministic", the two runs of one seed diverged. It prints the lines around the first difference. That's a bug in the simulated code or the simulator, usually a map ranged without sorting, a draw from a shared random source, or a goroutine started on the simulated path.

Gate on coverage¶

go test ./internal/sim -run TestSim -count=1 -seeds=500 -coverage

With -coverage, the batch fails for any required event that never happened in a shape, naming it:

coverage: sweep-stopped never happened in the kubeflow shape (spec 0005 D8)

A fault that never fired is a gap in the simulator, not a pass. A small batch is expected to miss rare events. The required lists are in internal/sim/coverage.go: coverageBoth, coverageSlurm and coverageKubeflow per shape, and coverageFile once over the seeds that ran the file store, which fails as coverage: tombstoned never happened on the file store (spec 0005 D8).

Check the mutation knobs¶

go test -tags dispatch_mutants ./internal/sim -run TestKnobs -count=1 -v

Each knob turns one fix back off, one at a time, and the simulator must then find a violation within the seed budget (-knobseeds, default 500, starting at -first). A caught knob is logged with the seed and what broke:

knobs_test.go:56: knob RetireCanonical caught at seed 1: invariant 6: retired handle 1 was opened
    replay: DISPATCH_MUTANT=RetireCanonical go test -tags dispatch_mutants ./internal/sim -run TestSim -seed=1

Run one knob with -knob=<name>. The names are in internal/mutants/off.go. The knobs only exist in a build tagged dispatch_mutants; in every other build they're constants the compiler removes.

A knob not caught fails the test, unless the maintainer has accepted a written defence-in-depth argument for it. Those are in acceptedExempt in internal/sim/knobs_test.go; an accepted knob that isn't caught passes, and the log quotes its argument. Two are accepted, provisionally, as of 2026-10-12: SingleCancelSweep and VerifiedStoppedAsEvidence.

What CI runs¶

The commands are in .gitlab-ci.yml.