Run the conformance suite against kind¶

go test ./... runs the adapter's unit tests, which classify recorded objects and never touch a cluster. This page is for the other test, internal/kindtest: it stands up a kind cluster with Trainer, JobSet, Kueue and dispatch-admission, and runs go/dispatch's backend conformance suite against the adapter in both modes, with cluster cases of its own. Run it before you change how the adapter reads or writes the cluster, and when a new release of a component needs checking.

What you need¶

docker, able to run containers as you;
a kind binary, on the PATH or named by DISPATCH_KUBEFLOW_KIND_BIN. The tested run used kind v0.33.0. The suite uses kind's default node image, which for v0.33.0 is Kubernetes 1.37.0;
kubectl and gunzip on the PATH, which the setup script uses;
the module checked out, and Go at the version go.mod names;
just, for the recipe below. Without it, run the command it wraps;
the busybox:1.38.0 image, or access to Docker Hub to pull it;
memory and time. The cluster is three node containers, each capped at 2500 MB, so leave about 7.5 GB free. A run is allowed 90 minutes, and the setup alone up to 20.

Run it¶

From the repository root:

just test-kind

The recipe is:

DISPATCH_KUBEFLOW_KIND=1 go test ./internal/kindtest -v -timeout 90m

The run stands the cluster up, runs every test against it, and tears it down.

Environment variables¶

Variable	Meaning
`DISPATCH_KUBEFLOW_KIND`	`1` runs the suite. Anything else, and every test in the package skips, saying `the kind cluster tests need DISPATCH_KUBEFLOW_KIND=1, docker and a kind binary (DISPATCH_KUBEFLOW_KIND_BIN)`.
`DISPATCH_KUBEFLOW_KIND_BIN`	The kind binary. Unset, `kind` is looked up on the `PATH`.
`DISPATCH_KUBEFLOW_KIND_KEEP`	`1` leaves the cluster running when the tests end. The next run reuses the cluster, named `dkkind`, and provisions it again rather than creating it. Delete it with `internal/kindtest/testdata/down.sh dkkind`.

A setup that fails prints kindtest: <error> and fails the run; a teardown that fails prints kindtest: teardown: <error>.

What it sets up¶

The setup is internal/kindtest/testdata/up.sh, the script the tutorial walks through: a cluster of one control-plane node and two workers; JobSet v0.12.0, Kueue v0.20.1 and Trainer v2.3.0 from manifests kept in internal/kindtest/testdata/manifests; dispatch-admission built from the checkout and deployed; cmd/publish built from the go/dispatch version go.mod pins; and the controllers pinned to the control-plane node, so that pausing a worker loses only training pods.

The adapter reaches the apiserver through a fault proxy in the test process, which can hold, delay or swallow a request or its reply. Each test runs in a namespace of its own, which in Kueue mode has a ClusterQueue whose quota fits one job.

What it runs¶

TestConformanceKueue and TestConformancePlain run go/dispatch's conformance.Backend against the adapter in each mode, with these faults:

Fault	How
lose a node	pause the worker a rank runs on, wait for it to go `NotReady`, and taint it `node.kubernetes.io/out-of-service`, as an operator confirming a shut-down node does
delay a submit's reply	let the next TrainJob create reach the apiserver, and withhold its reply until the adapter gives up waiting
preempt, Kueue mode only	admit a higher-priority Job to the job's ClusterQueue and remove it once the job's Workload is evicted

The suite's external requeue is not injected, because Kubernetes has no external requeue of a TrainJob. Nor is its late call, whose deadline is an hour away, which no proxy in front of a real apiserver can wait out; TestLateCall covers it instead.

The cluster cases cover what the shared suite cannot:

Test	Checks
`TestKueueListsCheckOnGatedWorkload`	Kueue lists the AdmissionCheck on a gated Workload before any quota is reserved
`TestStaleGateRemovalRefused`	removing the gate on a stale resourceVersion is refused with a conflict
`TestRuntimeLackingPatchedContainer`	`Start` refuses a runtime without the publish init container, and leaves nothing behind
`TestSnapshotOwnedAndCollected`	the runtime copy is owned by its TrainJob, reported foreign once edited, and collected with the TrainJob
`TestWorkDirAndPublishVolume`	a job runs as a non-root user with `WorkDir` set and `cmd/publish` mounted as an image volume
`TestNoAdmissionWhilePodsRemainPlain`	a plain-mode `Release` while the held execution's pods terminate returns `ErrPodsRemain`, writes nothing, and succeeds once they are gone
`TestNoAdmissionWhilePodsRemainKueue`	the controller keeps the check `Pending`, naming the pods, while pods that ignore SIGTERM remain
`TestControllerDown`	with the controller stopped, the check stays `Active`, a job waits at its `Pending` check, a newly opened adapter is observe-only, and a stale `Ready` is refused
`TestCessationSignal`	`Ceased` answers true only once every Node of the execution is tainted `out-of-service`
`TestLateCall`	a call past its deadline sends nothing, and a `Release` delayed past its deadline lands within deadline + `CallBound` and is refused by its precondition
`TestNoReadmissionWhileALostNodeHoldsPods`	a Kueue re-admission waits, with no second execution, while a rank's pod is stuck on a paused worker, until the node is tainted `out-of-service`

Tested versions lists the versions the suite was run against.