Skip to content

Read a job's output

r := cluster.Observer().(dispatch.OutputReader)

out, err := r.Output(ctx, job, 1, dispatch.Stdout, false)
if err != nil {
    return err
}
defer out.Close()

_, err = io.Copy(os.Stdout, out)

This copies submission 1's stdout, as the cluster has kept it so far, to your own stdout. Output is not part of Status: it is read through an operation, and only where the cluster keeps it.

Get the reader

A Cluster's Observer is always a dispatch.OutputReader, so the type assertion above holds for any cluster built by dispatch.Open or dispatch.Compose. Whether output can actually be read depends on the adapter underneath: one that doesn't read output answers every call with dispatch.ErrUnsupported, naming itself.

Output needs no record store, so a read-only cluster, or one opened without a store, reads it too.

Choose the submission and the stream

Output(ctx, id, submission, stream, follow) names the job, the submission number N (from 1, as in Status.Submissions[i].N), and dispatch.Stdout or dispatch.Stderr. Each submission is a separate object on the scheduler, with output of its own. To read what the job is running now, take the last submission in its status:

st, err := cluster.Observer().Get(ctx, job)
if err != nil {
    return err
}

latest := st.Submissions[len(st.Submissions)-1].N

A job with no submissions yet has nothing to read.

Follow it while it is written

Pass follow as true to keep reading as the execution writes. How the stream ends is the adapter's: the Slurm adapter's stays open, looking for new bytes and new executions every two seconds, until you close it or its context ends; the Kubeflow adapter's follows rank 0's log through the apiserver. Read it line by line:

out, err := r.Output(ctx, job, latest, dispatch.Stdout, true)
if err != nil {
    return err
}
defer out.Close()

lines := bufio.NewScanner(out)
for lines.Scan() {
    fmt.Println(lines.Text())
}

Cancel ctx, or close out, to stop following.

Check what the cluster can do first

Three capabilities describe output: OutputRead, OutputFollow and OutputRetained. They are visible on failure: a cluster that can't do one tells you with an error from Output, rather than running the job differently. So a job is never refused at submit for them, and you can ask before you read:

caps, err := cluster.Observer().Capabilities(ctx, nil)
if err != nil {
    return err
}

if caps[dispatch.OutputFollow].Value != dispatch.Yes {
    fmt.Println("this cluster can't follow output:", caps[dispatch.OutputFollow].Reason)
}

An answer of unknown means the adapter couldn't tell. Try the read and handle the error.

What each adapter keeps

The two adapters keep output very differently:

Slurm Kubeflow
Streams stdout and stderr, each in its own files one log with both interleaved: Stdout returns it, Stderr is ErrUnsupported
Executions every execution of the submission, each in its own file, read in restart order rank 0 of the current execution only; earlier executions' logs went with their pods
Ranks every task's output, as Slurm writes it rank 0
After the execution ends still readable gone once its pods are
Not there yet ErrNotFound while the submission has no output manifest ErrNotFound while there is no rank 0 pod

The Slurm adapter's files are under its output.dir, as how a job becomes an sbatch submission says. The Kubeflow adapter reads pods/log, which needs the permission its RBAC lists as optional; without it, both reads are ErrUnsupported. See its capabilities.