The execution environment¶
| Variable | Value |
|---|---|
DISPATCH_JOB_ID |
the job's id, a 26-character ULID |
DISPATCH_SUBMISSION |
the submission number N, from 1 |
DISPATCH_ATTEMPT |
an opaque token: the same on every rank of one execution, different for every execution |
DISPATCH_EPOCH |
a decimal integer, G × 10^12 + N × 10^6 + R, that only increases across the job's executions |
An adapter guarantees all four wherever it reports the AttemptInjection capability as yes, and
Submit refuses a job on a cluster where it doesn't. Where a site opt-in lets the backend rerun a
rank under the identity it already had, the plan says so in Plan.IdentityWeakened; see
execution identity.
DISPATCH_ is reserved: JobSpec.Validate refuses an Env key that starts with it. Adapters use
further names under it for their own plumbing, such as the Kubeflow adapter's DISPATCH_K8S_*
inputs to its prologue and DISPATCH_PUBLISH_DEST for a publish init container. A job script
shouldn't read them.
DISPATCH_ATTEMPT¶
Treat it as a string, and don't parse it. It identifies an execution, not an attempt: it is not
the attempt number a Start call carried, and one attempt's object can run several executions,
each with its own token. It's the execution's rendezvous identity, and the name of the folder the
execution writes checkpoints into, <Checkpoint>/exec/<DISPATCH_ATTEMPT>/. Each adapter builds it
its own way; see per adapter.
DISPATCH_EPOCH¶
| Term | Is | Range |
|---|---|---|
| G | the job's generation: 0, or the superseded job's plus 1 | 0 to 18,000,000 |
| N | the submission number | 1 to 999,999 |
| R | a per-object execution counter the adapter guarantees only increases | 0 to 999,999 |
model.Epoch(generation, submission, restart) computes it and returns ErrInvalid outside those
ranges. checkpoint.ParseEpoch parses it. What R counts is the adapter's choice, so long as every
later execution of the object gets a larger one.
cmd/publish reads DISPATCH_EPOCH for watch and publish, and DISPATCH_ATTEMPT for watch
with no folder given; see checkpoint publication.
Per adapter¶
| Adapter | DISPATCH_ATTEMPT |
R |
|---|---|---|
| Slurm | <N>.<Slurm job id>.<restart count> |
Slurm's restart count, from 0; see identity inside the job |
| Kubeflow | <JobID>.<N>.<attempt>.<A>.<J> |
A + J, the admission count plus JobSet's restart count, from 1; see execution identity |
| memory, Slurm shape | <N>.<Slurm job id>.<restart> |
the restart count |
| memory, Kubeflow shape | <N>.a<execution>.r<restart> |
every execution of the object, because the restart counter it shows doesn't count re-admissions |