Skip to content

Write an adapter and pass the conformance suite

func TestConformance(t *testing.T) {
    conformance.Backend(t, conformance.Harness{
        Open: func(t *testing.T, readOnly bool) dispatch.Backend { return openMyBackend(t, readOnly) },
    })
}

conformance.Backend runs the backend conformance suite as subtests of t, through the dispatch.Backend interface alone. Every check your adapter can't be tested for, because the harness can't inject a fault or the adapter lacks an optional interface, is skipped with a reason, never passed.

An adapter works at the level of scheduler objects and their runs. It never sees jobs, plans or the record: the core and the supervisor own those.

Implement Backend

dispatch.Backend has ten methods. What the core relies on, method by method:

Method Must
Capabilities(ctx, spec) answer a Support per capability, for this cluster and, where it matters, this spec; spec may be nil. AttemptInjection must be Yes or no job is accepted
Rules(ctx, spec, policy) answer one RuleSupport per rule, in policy order, saying whether the rule's action can be carried out here by anyone
Start(ctx, req) create the object with its gate closed, so nothing runs before Open, carrying req.Submission and req.Attempt where Find can read them back. Return ErrOutcomeUnknown when the scheduler's answer doesn't say whether it was created
Open(ctx, ref) lift the gate. An object already past its gate is success; an object that ended before it was opened is an error
Find(ctx, job) return every object named for the job (job.DiscoveryName()), with Complete, Horizon and At set
Inspect(ctx, ref) observe one object
Stop(ctx, ref, reason) stop this one object instance and nothing else. Already gone is success
StopJob(ctx, job, reason) stop every object of the job by name, then verify, reporting each object as stopped, alive or unproven
Declarations() return fixed answers: the node-loss window, how long a hold-family or requeue call can take to land (CallBound), and how long each kind of evidence survives between polls (PolledLoss)
Close() release what the adapter holds

Beyond the methods:

  • Never guess a refusal. The supervisor records an error from Start as a refusal only when it is ErrRejected, ErrInvalid or ErrUnsupported. Anything else is recorded as unknown and retried, because the object may exist. Return a refusal sentinel only when you know nothing was created.
  • Map causes, don't collapse them. Each ObjectObs carries a Cause with a Kind from the closed set, a Provenance, and the scheduler's own words in Detail. A scheduler answer you can't map is CauseUnknown, never CauseNonZeroExit.
  • Report foreign objects. An object with the job's discovery name whose submission and attempt you can't parse is reported with Foreign: true. Never drop it.
  • Give every execution an identity. Every rank of an execution sees DISPATCH_JOB_ID, DISPATCH_SUBMISSION, DISPATCH_ATTEMPT (an opaque token, different for every execution) and DISPATCH_EPOCH, computed with model.Epoch(req.Generation, req.Submission, restart). See the execution environment.
  • Refuse what you don't understand. An extension from another namespace is ErrUnsupported, and your settings decode with unknown fields refused.
  • Honour read-only. On a backend opened with AdapterConfig.ReadOnly, Start, Open, Stop, StopJob, Hold, Release and Requeue return ErrReadOnly; Find, Inspect, Watch and Ceased still work.

Add the optional interfaces you can support

The core finds optional capabilities by type assertion on your Backend:

Interface Adds Declare with it
dispatch.Holder Hold and Release of a numbered hold instance, each carrying a deadline the scheduler's host checks. A hold survives Open, and Release of any instance but the one in force is refused CallBound > 0, and the Hold capability Yes; where it isn't, Hold returns ErrUnsupported
dispatch.Requeuer Requeue, re-running the same object, and RestoreRequeueBit, clearing a requeue bit left set outside Requeue, both under the same deadline rule CallBound > 0
dispatch.NativePolicyCompiler Native and Compile: rules the scheduler enforces itself, with the budget left, not the rule's whole budget
dispatch.Watcher Watch, a stream of observations, for evidence a poll would lose empty PolledLoss
dispatch.CessationSignal Ceased, a site's own answer on whether an object's processes are gone
dispatch.IdentityWeakener IdentityWeakened, why a spec's execution identity would be weakened, such as a site opt-in that lets the backend rerun a rank under the same identity; recorded in Plan.IdentityWeakened, never refused
dispatch.MultiFinder FindMany, the objects of many jobs in one scheduler query, optionally only those in given states; dispatch.FindAll uses it and otherwise calls Find per job
dispatch.OutputReader Output, a submission's stdout or stderr where the cluster keeps it, optionally followed; the cluster's Observer passes it through, as in read a job's output. The conformance suite doesn't check it OutputRead, OutputFollow and OutputRetained as they are

CallBound must be zero exactly when the adapter has neither Holder nor Requeuer; the suite checks it. Without Watcher, PolledLoss must name a window for every kind of evidence (state, cause, exit-code, execution). Evidence that stays readable however late it is polled declares dispatch.Unbounded, never the largest time.Duration; any other negative window is refused. supervise.New refuses a poll interval longer than the shortest bounded one.

A hold-family or requeue call whose command started after its CallMeta.Deadline must do nothing. That's what stops a stale call landing late and undoing a newer decision.

Register it

Register a Factory under your adapter's name, usually in init, so dispatch.Open can find it:

func init() {
    if err := dispatch.Register("myscheduler", Factory); err != nil {
        panic(err)
    }
}

func Factory(ctx context.Context, cfg dispatch.AdapterConfig) (dispatch.Backend, error) {
    var s settings
    dec := json.NewDecoder(bytes.NewReader(cfg.Settings))
    dec.DisallowUnknownFields()
    if err := dec.Decode(&s); err != nil {
        return nil, fmt.Errorf("%w: myscheduler settings: %w", dispatch.ErrInvalid, err)
    }

    return newBackend(ctx, s, cfg.ReadOnly)
}

A second registration of the same name returns ErrAlreadyRegistered; it never replaces the first. cfg.Settings is the JSON encoding of dispatch.Config.Adapter, and is nil when that was nil. cfg.Backend is the name the adapter was opened under; put it in every ObjectRef.Backend.

Write the harness

conformance.Harness has five fields:

h := conformance.Harness{
    // Every call with the same t must see the same objects, read-only or not.
    Open: func(t *testing.T, readOnly bool) dispatch.Backend { ... },
    Faults: conformance.Faults{
        Preempt:  func(t *testing.T, ref model.ObjectRef) { ... },
        LoseNode: func(t *testing.T, ref model.ObjectRef) { ... },
        // RequeueExternally, DelaySubmitReply, LateCall, CrashBetweenSteps: as your test cluster allows
    },
    // The DISPATCH_* environment each rank of ref's current execution sees; nil if you can't read it.
    Ranks: func(t *testing.T, ref model.ObjectRef) []map[string]string { ... },
    // The job every check submits; nil submits Command ["true"] on two nodes.
    Spec: func(t *testing.T) model.JobSpec {
        return model.JobSpec{Command: []string{"sleep", "600"}, Nodes: 2}
    },
    // Zero fields take conformance.DefaultTimings.
    Timings: conformance.Timings{Wait: 5 * time.Minute, Quiet: 5 * time.Second, Poll: time.Second},
}
  • Open is called many times per test. Calls with the same t must reach the same cluster, so an object one call creates is visible to the next, including through a read-only backend.
  • Each Faults entry may be nil. Preempt, LoseNode and RequeueExternally act on the running execution of ref. DelaySubmitReply gets a zero ref and delays the reply of the next Start. RequeueExternally should leave the requeue bit set, as an operator does, so the suite can check RestoreRequeueBit. LateCall makes the next hold-family, requeue or restore call on ref reach the scheduler only after its deadline plus CallBound. CrashBetweenSteps ends the next Requeue on ref after it has set Requeue=1, as a crash of the caller would; the suite then checks that Inspect shows the bit and that re-running the requeue restores it.
  • On a real cluster, set Spec to a job that runs long enough for the suite to see it Running and read its ranks. The default ["true"] ends at once anywhere that actually runs it.
  • Without Ranks, the identity check can't see a rank's environment and is skipped.

The memory adapter's harness in memory/conformance_test.go is a complete one, for both of its shapes.

Run it, and read the skips

go test -run TestConformance -v ./...

The suite's top-level subtests are Declarations, Capabilities, Rules, Gate, OpenIdempotent, OpenFence, Find, StopDuplicate, StopJob, Identity, ReadBack, Preflight, ReadOnly, Classification, PolledLoss, Holder, Requeuer, NativePolicyCompiler, Watcher, MultiFinder and CessationSignal, in that order.

Read every SKIP line. Some are the suite's own gaps, the same for every adapter:

  • ReadBack: no fault injects a site plugin that rewrites a submission;
  • Preflight: the SPI has no preflight entry point.

The rest are yours: a fault your harness didn't provide, or an optional interface you don't implement. A skip is not a pass, so each one is a check nobody has made of your adapter.

By default the suite waits up to a minute for the scheduler to show a state, watches 250 milliseconds for something that must not happen, and polls every 10 milliseconds. Those suit the memory adapter; set Timings to what your test cluster needs.

Run the core's suite too

conformance.Core checks the core's own rules (spec 0001's D4 to D10) through the real Operator and a supervise.Supervisor, with your adapter underneath. Each check runs in a testing/synctest bubble, so it suits an adapter whose time is the fake clock's, such as an in-memory model of your scheduler:

func TestCoreConformance(t *testing.T) {
    conformance.Core(t, conformance.CoreHarness{
        Open: func(t *testing.T, s conformance.CoreSetup) (*conformance.CoreCluster, bool) {
            return openMyCluster(t, s)
        },
    })
}

Open returns a fresh cluster for each check, set up as its CoreSetup asks: with or without a store, read-only or not, without the adapter's Requeuer, and with the capability, Perform and native answers it names overridden. Return false when your adapter can't be set up that way, and the check is skipped with a reason. The CoreCluster carries the Cluster, the Backend under it, the store, and a CoreEnv whose Run, End, Preempt and LoseNode make the scheduler act; a nil one skips the checks that need it. memory/core_conformance_test.go is a complete harness.

Prove what you depend on when opened

Both adapters run a preflight inside their Factory: they read the cluster, and on a writable backend submit a probe that can't run, to test the gate and the hold round trip. An answer the probe decides is reported with source probed; a cluster that fails what the adapter can't work without is refused from Factory. Preflight, read-only and observe-only explains why.

Worked examples

The two adapters on this site are complete adapters, each with its own way of running the suite against a real scheduler: