Configuration¶

The adapter's settings are one JSON object, passed to the adapter as AdapterConfig.Settings. The Go package exposes the JSON Schema of that object as kubeflow.Schema() (the same bytes as schema.json in the repository, $id https://gitlab.com/phpboyscout/go/dispatch-kubeflow/schema.json, JSON Schema 2020-12), and the parser as kubeflow.ParseSettings. Omit a key for its default.

The schema sets additionalProperties: false at every level, and so does the parser: an unknown key anywhere is refused with ErrInvalid. So are duplicate keys, trailing data, and null for any value. A null would read as unset and take the default without saying so.

An example¶

{
  "cluster":   {"kubeconfig": "/home/u/.kube/config", "context": "train"},
  "namespace": "team-a",
  "runtimes":  [{"kind": "ClusterTrainingRuntime", "name": "torch-distributed"}],
  "kueue": {
    "localQueue": "lq",
    "priorityClass": "low",
    "admissionCheck": "dispatch-admission",
    "admissionLease": {"namespace": "dispatch-system", "name": "dispatch-admission"}
  },
  "labels":      {"team": "a"},
  "annotations": {"cost-centre": "42"},
  "gpuResource": "nvidia.com/gpu",
  "credentials": [{"secretName": "team-a-s3", "mountPath": "/var/run/secrets/s3"}],
  "nodeLossWindow": "5m",
  "nodeLossTaints": ["example.com/node-dead"],
  "allowRankRestart": false,
  "timeouts": {"request": "30s", "apiserverRequest": "60s", "workload": "60s", "stopJob": "10m"},
  "probeRuntime": {"kind": "ClusterTrainingRuntime", "name": "torch-distributed"},
  "publish": {"image": "registry.example/dispatch-publish@sha256:<64 hex digits>", "via": "imageVolume"}
}

Only namespace is required, and either runtimes (non-empty) or probeRuntime.

Keys¶

Durations are Go duration strings such as 30s or 1h30m, and must be positive.

Top level¶

Key	Type	Default	Rule
`namespace`	string	none, required	An RFC 1123 label (at most 63 characters). One adapter instance serves one namespace.
`cluster`	object	the kubeconfig loading rules	How to reach the apiserver. See below.
`runtimes`	array of runtime references	none: any runtime	An allow-list. A job naming a runtime outside it is refused with `ErrUnsupported`. No duplicates.
`probeRuntime`	runtime reference	the first of `runtimes`	The runtime the preflight probe uses. Required when `runtimes` is empty or absent. When `runtimes` is set it must be one of them, and it must pass the runtime rules, including the publish init container under `publish.via: initContainer`; otherwise the probe's `Start` is refused and `Factory` fails with `ErrUnsupported` (`preflight probe: ...`). The parser does not check either.
`kueue`	object	absent: plain mode	Present selects Kueue mode. See below.
`labels`	object, string values	none	Added to every TrainJob.
`annotations`	object, string values	none	Added to every TrainJob.
`gpuResource`	string	`nvidia.com/gpu`	The resource `GPUsPerNode` maps to, in the TrainJob's `resourcesPerNode.limits`. A qualified name.
`credentials`	array	none	Secret volumes mounted read-only on the trainer container.
`nodeLossWindow`	duration	`5m`	Declared to dispatch as `Declarations.NodeLossWindow`, the longest a lost node can go unreported. The default is the `tolerationSeconds` Kubernetes gives pods for the not-ready and unreachable taints, after which it evicts the pod itself.
`nodeLossTaints`	array of qualified names	none	Site `NoExecute` taints that mean a node is lost, in addition to Kubernetes' own `node.kubernetes.io/unreachable`, `not-ready` and `out-of-service`. No duplicates.
`allowRankRestart`	boolean	`false`	Accept runtimes whose trainer job lets a failed rank rerun under the same identity. See runtime requirements.
`timeouts`	object	below
`publish`	object	absent	How `cmd/publish` reaches the trainer container. See below.

labels and annotations keys must be qualified names and must not begin with dispatch.phpboyscout.uk/, kueue.x-k8s.io/, trainer.kubeflow.org/ or jobset.sigs.k8s.io/, which dispatch, Kueue, Trainer and JobSet read. Label values must be valid label values (at most 63 characters).

`cluster`¶

Exactly one of a kubeconfig or the pod's service account.

Key	Type	Rule
`kubeconfig`	string	A kubeconfig path. Empty or absent follows client-go's loading rules: `KUBECONFIG`, then `~/.kube/config`. If those give no configuration and the adapter runs in a pod, client-go falls back to the pod's service account.
`context`	string	A kubeconfig context. Absent is the current context. An empty string is refused.
`inCluster`	boolean	Use the pod's service account. Refused together with `kubeconfig` or `context`.

If AdapterConfig.Token is set, it supplies a bearer token on every request and replaces the credentials the kubeconfig carries.

`kueue`¶

Key	Type	Default	Rule
`localQueue`	string	none, required	A LocalQueue name that is also a valid label value (at most 63 characters). Set as `kueue.x-k8s.io/queue-name`.
`priorityClass`	string	none	A name of the same shape. Set as `kueue.x-k8s.io/priority-class`.
`admissionCheck`	string	none	The name of the AdmissionCheck object. Without it `ReadmissionGate` is `no` and the adapter is observe-only, so treat it as required.
`admissionLease`	object	`{"namespace": "dispatch-system", "name": "dispatch-admission"}`	The admission controller's leader-election Lease, read as its heartbeat. Both `namespace` (a label) and `name` (an object name) are required when the object is given.

`credentials[]`¶

Each entry mounts a Secret by name with defaultMode 0400 and optional: true, read-only, on the node container. The adapter never reads a Secret, and the value never enters an environment variable.

Key	Type	Rule
`secretName`	string	An object name. Required.
`mountPath`	string	A clean absolute path. Required. Must not be `/` and must not overlap `/opt/dispatch` or another credential's path.

optional: true lets a pod start without the Secret, so a missing one fails the job's first step with a message naming the source, instead of leaving a pod stuck waiting for a mount.

`timeouts`¶

Key	Default	Bounds
`request`	`30s`	Each non-streaming apiserver request.
`apiserverRequest`	`60s`	Set this to the apiserver's `--request-timeout`. `Declarations.CallBound` is derived from it.
`workload`	`60s`	How long `Start` waits for Kueue to make the TrainJob's Workload, and how long the preflight probe's `StopJob` waits.
`stopJob`	`10m`	How long `StopJob` repeats `Find` waiting for everything of the job to go.

`publish`¶

Key	Type	Default	Rule
`image`	string	none, required	A reference pinned by digest: `name@sha256:` and 64 lower-case hex digits.
`via`	`imageVolume` or `initContainer`	`imageVolume`	The route.
`initContainer`	string	`dispatch-publish` under `initContainer`	The init container the runtime defines. A container name (an RFC 1123 label). Refused unless `via` is `initContainer`.

Both routes put the binary at /opt/dispatch/publish. See Mount the publish image.

Runtime references¶

runtimes[] and probeRuntime share one shape:

Key	Type	Rule
`kind`	`ClusterTrainingRuntime` or `TrainingRuntime`	Required.
`name`	string	A Kubernetes object name. Required.

A TrainingRuntime is read from namespace.

What the schema does not say¶

The schema is the contract for a host's own configuration validation. The parser enforces a few things beyond it, and the schema carries one the parser enforces differently.

credentials[].mountPath: the parser also requires a clean path (path.Clean leaves it unchanged) and refuses overlaps between credentials. The schema checks only the leading slash and the /opt/dispatch prefix.
runtimes and nodeLossTaints lists: no duplicates, in both.
publish.initContainer with via absent: the schema requires via to be present and equal to initContainer whenever initContainer is. The parser reads an absent via as imageVolume and then refuses the combination. The result is the same refusal.
Qualified names (labels and annotations keys, gpuResource, nodeLossTaints): the parser limits the name part to 63 characters and the prefix before the / to 253. The schema's pattern sets no length, so a longer key passes the schema and is refused by the parser.
null: refused by the parser anywhere. The schema's types already exclude it.
Defaults: the schema documents the default for most keys; the parser applies them.

Declarations¶

Two values reach dispatch from this section, fixed when the adapter opens:

Declaration	Value
`NodeLossWindow`	`nodeLossWindow`, `5m` by default
`CallBound`	`timeouts.apiserverRequest` plus 5 seconds, 65 seconds by default

PolledLoss is empty, because the adapter implements Watcher.