Mount the publish image¶

A job saves and resolves checkpoints with go/dispatch's publish binary, which the job's own command runs. The training image stays yours, so the adapter delivers the binary separately, to /opt/dispatch/publish in the trainer container, by one of two routes.

The image that holds the binary, registry.gitlab.com/phpboyscout/go/dispatch/publish, is described in go/dispatch's publish image reference: its tags, its layout and how to read a release's digest. This page does not repeat it.

Route 1: an image volume (the default)¶

Set publish.image to the image, pinned by digest:

{
  "publish": {
    "image": "registry.gitlab.com/phpboyscout/go/dispatch/publish@sha256:<64 hex digits>"
  }
}

A reference with a tag, or without @sha256: and 64 lower-case hex digits, is refused when the settings are parsed.

For each job, the adapter's runtime patch then adds a volume named dispatch-publish with an image source, pullPolicy: IfNotPresent, and mounts it read-only on the node container at /opt/dispatch. The image holds one file at its root, so the binary is at /opt/dispatch/publish.

Requirements:

• Kubernetes with image volumes enabled. They are on by default in 1.37, the version tested, and need no configuration there.
• A container runtime that mounts them without noexec. This was observed with containerd on kind; on other runtimes, test it.
• Nodes that can pull the image.

Refer to the binary by its full path in the job's command:

/opt/dispatch/publish resolve -checkpoint "$CKPT"
/opt/dispatch/publish watch -checkpoint "$CKPT" -framework hf &

Route 2: an init container your runtime defines¶

Where image volumes are unavailable, set publish.via to initContainer:

{
  "publish": {
    "image": "registry.gitlab.com/phpboyscout/go/dispatch/publish@sha256:<64 hex digits>",
    "via": "initContainer",
    "initContainer": "dispatch-publish"
  }
}

initContainer is the name of an init container that the runtime must already define. Its default is dispatch-publish. A runtime patch cannot set a container's image, so the image of that init container is whatever the runtime says, and publish.image is checked for its shape but not placed on the pod. Because the publish image has no shell and no cp, the init container's image needs a copy tool. The test harness's dispatch-kind-init runtime shows the shape of the runtime side only: its init container runs true and copies nothing.

This route has not been run end to end. The kind suite checks that Start refuses a runtime without the named init container, and its preflight probe creates a gated TrainJob with the init container's patch on dispatch-kind-init, but no pod on this route has ever started and found the binary. Test it on your cluster before you rely on it.

For each job, the adapter's runtime patch:

• adds an emptyDir volume named dispatch-publish;
• sets DISPATCH_PUBLISH_DEST=/opt/dispatch on the named init container and mounts the volume there;
• mounts the volume read-only on the node container at /opt/dispatch.

Your init container must copy the binary to $DISPATCH_PUBLISH_DEST, as publish, so that the job still finds it at /opt/dispatch/publish.

Trainer does not check that the init container exists: a patch naming a missing one is accepted and the job then holds quota with no pods. The adapter therefore checks the runtime itself, and Start refuses a runtime without the named init container with ErrUnsupported, before creating anything. See runtime requirements.

Things to avoid¶

• No credential may be mounted at or under /opt/dispatch. A credentials[].mountPath that overlaps it is refused when the settings are parsed.
• initContainer is only valid with via: initContainer. Naming it under the default route is refused.
• Publishing is optional. With no publish section the adapter mounts nothing at /opt/dispatch.

Configuration reference lists the publish keys.