Skip to content

Grant the adapter its RBAC

The adapter runs as whatever identity its kubeconfig, its in-cluster service account or its Token supplies. At Factory it asks the apiserver, with a SelfSubjectAccessReview for each verb below, whether that identity may do it, and sets itself up accordingly. It does not assume. Your identity needs create on selfsubjectaccessreviews, which Kubernetes grants to all authenticated users by default (system:basic-user).

Read a preflight result shows how the answers surface. This page is the other half: what to grant.

The adapter serves one namespace, so most rules go in a Role in that namespace. Three kinds of permission are different:

  • Required to observe. Without them Factory fails with ErrUnauthorized.
  • Required to write. Without them the adapter opens observe-only.
  • Optional. Without them a named capability is lowered.

The namespaced Role

Replace team-a with the namespace in namespace. Everything here is for a Role bound to the adapter's identity in that namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: dispatch-adapter, namespace: team-a}
rules:
# Required to observe, and to write (create, update, delete).
- apiGroups: [trainer.kubeflow.org]
  resources: [trainjobs]
  verbs: [get, list, watch, create, update, delete]
- apiGroups: [trainer.kubeflow.org]
  resources: [trainingruntimes]
  verbs: [get, list, create, update, delete]
- apiGroups: [jobset.x-k8s.io]
  resources: [jobsets]
  verbs: [get, list, watch]
- apiGroups: [""]
  resources: [pods]
  verbs: [get, list, watch]
# Required to write: StopJob deletes the JobSets and pods background deletion orphans. pods/log is rank 0's log.
- apiGroups: [jobset.x-k8s.io]
  resources: [jobsets]
  verbs: [delete]
- apiGroups: [""]
  resources: [pods]
  verbs: [delete]
- apiGroups: [""]
  resources: [pods/log]
  verbs: [get]
# Kueue mode only.
- apiGroups: [kueue.x-k8s.io]
  resources: [workloads]
  verbs: [get, list, watch, update]
- apiGroups: [kueue.x-k8s.io]
  resources: [localqueues]
  verbs: [get]

The adapter creates, updates and deletes TrainingRuntimes because each job runs on a namespaced copy of its runtime, named for the TrainJob. It never touches a Secret. Credentials are mounted by name and the adapter does not read them.

The cluster-scoped rules

Runtimes, Nodes, ClusterQueues and AdmissionChecks are cluster-scoped, so these go in a ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: {name: dispatch-adapter}
rules:
# Required to observe.
- apiGroups: [trainer.kubeflow.org]
  resources: [clustertrainingruntimes]
  verbs: [get]
# Optional: the Node reads behind node-loss classification and CessationSignal.
- apiGroups: [""]
  resources: [nodes]
  verbs: [get]
# Optional, Kueue mode: proving the readmission gate.
- apiGroups: [kueue.x-k8s.io]
  resources: [clusterqueues, admissionchecks]
  verbs: [get]

And the Lease, in the controller's namespace (dispatch-system unless you moved it):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: dispatch-adapter-heartbeat, namespace: dispatch-system}
rules:
- apiGroups: [coordination.k8s.io]
  resources: [leases]
  verbs: [get]

Bind each to the adapter's identity with a RoleBinding (the namespaced Roles) and a ClusterRoleBinding (the ClusterRole).

What each permission is for

Permission Class Without it
get, list, watch trainjobs, jobsets, pods required to observe Factory fails with ErrUnauthorized
get clustertrainingruntimes; get, list trainingruntimes required to observe Factory fails
get, list, watch workloads; get localqueues (Kueue mode) required to observe Factory fails
create, update, delete trainjobs; create, update, delete trainingruntimes required to write observe-only; Start refuses
update workloads (Kueue mode) optional Hold is no with the reason not permitted to update workloads.kueue.x-k8s.io
get pods/log optional OutputRead and OutputFollow are no
get nodes optional Ceased returns ErrUnsupported; node-loss classification cannot read taints and says so in the cause; StopJob never reports Unproven, and counts remaining pods as Alive
get clusterqueues, admissionchecks; get leases (Kueue mode) optional ReadmissionGate is unknown, so the adapter is observe-only
delete jobsets, pods required to write observe-only; Start refuses. StopJob deletes the orphaned JobSets and pods it finds, which background deletion makes routine, so a writable adapter must be able to

Plain mode needs none of the Workload, LocalQueue, ClusterQueue, AdmissionCheck or Lease permissions to run. Where Kueue is installed, grant it list on workloads all the same: preflight's check that Kueue does not manage queue-less jobs lists the probe's Workloads, and without that permission it skips the check silently. See Choose Kueue mode or plain mode.

update on TrainJobs matters beyond Open. The first time the adapter observes a TrainJob with a JobSet, it records the JobSet's UID in an annotation, which needs update. An observe-only adapter does not do this.

The admission controller's RBAC

The controller has its own identity and its own ClusterRole. It is in Install the admission controller.

Running as a bearer token

If the host supplies AdapterConfig.Token, the adapter calls it for every request and uses the result as the bearer token, replacing whatever credentials the kubeconfig carries. The permissions above apply to the identity that token names.