Job Runners

A build is a job: boot, do one thing, exit. The platform runs three kinds of job on the kata_jobs island, each in its own Kata microVM, each from an image derived from the release worker so what is compiled matches what will run. This chapter covers the runners, how work reaches them, and the budget they share.

#The Three Jobs

Each runner is a small module, bundled at build time into one self-contained file of a few tens of kilobytes. The harness that gives a runner its logging, uploads and completion callback is bundled in, so a runner has no import graph to resolve inside the pod.

#Images Derived From the Worker

The three runner images are built from the worker image at the same commit. The method compiler resolves and installs a release's dependencies inside an image whose Node.js, operating system and C library are identical to the release sandbox that will later load them. A native module compiled here loads there. The ABI stamp on the artifact records the Node version, platform, architecture and C library, and the worker checks it before extracting.

This is also why the build toolchain lives in the compiler image and not in the worker: compilation happens once at publish, so the hundreds of short-lived serving pods that follow never carry it.

Two workflows build the images, off the platform's deploy train: one for the worker and the three derived runners, one for the dev box. Both are path-filtered so a change to a runner rebuilds only what changed. A push to the default branch moves the staging alias; production is a manual dispatch. Because the orchestrator's pods reference the alias, a push is the deploy.

#How Work Reaches a Runner

The release build calls the orchestrator to create a job sandbox with a spec: the runner image, the bundled runner script as a file, the job's parameters as environment, and a job label. The orchestrator mounts the script as a ConfigMap owned by the pod and creates the pod detached, returning as soon as it exists rather than waiting for readiness. Nothing will ever connect to a runner, so readiness buys nothing, and on an island that scales from zero a node can take minutes to arrive.

The runner reads its inputs from environment, does its work, uploads its outputs to S3 through presigned grants the platform minted, posts a completion callback with its result, and exits. Logs stream to the platform as it runs. A runner that times out still posts a failure completion, so the platform learns what happened instead of waiting for a poll to give up.

Inputs travel as files and environment because a sandbox cannot reach the orchestrator: the egress policy denies internal ranges, and pulling executable code from a public endpoint would first require standing one up.

#One Budget

Every job on the island shares one budget: a Kubernetes deadline of one hour on the pod, and a runner wallclock of fifty-five minutes below it. The gap is what lets a timing-out runner post its failure and flush its logs before Kubernetes stops it.

The budget is deliberately generous. A method compile, a web build and a browser audit do not need three different clocks; they are jobs on a node group with one timeout. Real builds take seconds to a few minutes, so the deadline is a backstop and not a target, and an unusual app with a slow build is not killed for being unusual.

#Scale From Zero

The jobs island sits at zero nodes when nothing is building. The cluster autoscaler cannot inspect a node that does not exist, so the node-group module publishes tags describing what a node would look like: the Kata label the runtime class requires, the taint, the GPU and ephemeral-storage resources. Those tags are what let the autoscaler conclude that a pending job pod would fit and bring a node up for it.