ReferenceAsk
Remy Reference/Architecture/Architecture/Job Runners
07Architecture

Job Runners

The three build jobs the platform runs, how work reaches a runner that nothing else can reach, and the one budget they share.

A build is a job: boot, do one thing, exit. The platform runs four kinds of job on the kata_jobs island, each in its own Kata microVM, each from an image derived from the release worker so what is compiled matches what will run. This chapter covers the runners, how work reaches them, and the budget they share.


#The Four Jobs

JobWhat it doesConsumed by
Method compilerBundles each method to JavaScript and builds the release's dependency artifactThe release build
Web interface compilerRuns the web interface's build and uploads the outputThe release build
Frontend diagnosticsAudits the built interface with a headless browserPost-deploy checks
Browser sessionHolds one Chrome open and runs an app's actions on it, one at a timeAn app's browser calls

Each runner is a small module, bundled at build time into one self-contained file of a few tens of kilobytes. The harness that gives a runner its logging, uploads and completion callback is bundled in, so a runner has no import graph to resolve inside the pod.


#Images Derived From the Worker

The four runner images are built from the worker image at the same commit. The method compiler resolves and installs a release's dependencies inside an image whose Node.js, operating system and C library are identical to the release sandbox that will later load them. A native module compiled here loads there. The ABI stamp on the artifact records the Node version, platform, architecture and C library, and the worker checks it before extracting.

This is also why the build toolchain lives in the compiler image and not in the worker: compilation happens once at publish, and the hundreds of short-lived serving pods that follow carry none of its weight.

Two workflows build the images: one for the worker and the four derived runners, one for the dev box. Both are path-filtered so a change to a runner rebuilds only what changed. A push to the default branch moves the staging alias; a release moves the production alias when it changed those paths. Because the orchestrator's pods reference the alias, a push is the deploy, and it rolls no platform pod.


#How Work Reaches a Runner

The release build calls the orchestrator to create a job sandbox with a spec: the runner image, the bundled runner script and the job's input as files, its settings as environment, and a job label. The orchestrator mounts the files as a ConfigMap owned by the pod and creates the pod detached, returning as soon as it exists rather than waiting for readiness. Nothing will ever connect to a runner, so readiness buys nothing, and on an island that scales from zero a node can take minutes to arrive.

The runner reads what it was asked to do from its input file, does its work, uploads its outputs to S3 through presigned grants the platform minted, posts a completion callback with its result, and exits. Logs stream to the platform as it runs. A runner that times out still posts a failure completion, so the platform learns what happened instead of waiting for a poll to give up. A runner that dies before it can post (its exec refused, killed for memory, its VM gone) is posted for: the orchestrator's reaper finds the failed pod and sends the failure, saying how the runner ended, to the same completion callback.

The input is a file and not environment because it grows with the app, and environment has a cliff. Linux won't start a process whose environment holds a string over 128 KiB, and everything the runner starts inherits its environment. An app of 1152 methods once put its method list past that limit, and every compile pod died before its runner ran. Every sandbox spec is now held to a small size per environment variable and a total size for its files. Inputs travel as files and environment at all because a sandbox cannot reach the orchestrator: the egress policy denies internal ranges, and fetching executable code from a public endpoint would mean standing one up.

A browser session is the one job that isn't handed all its work at boot. Its runner holds Chrome open and long-polls the platform for the session's next action, the way a dev box's tunnel polls for method calls. It runs the action, posts the outcome and polls again, until the app closes the session, it sits idle past its timeout, or the budget below runs out. A heartbeat beside the loop tells the platform the pod is still alive through a long action, so a pod that died is noticed within a minute rather than at its deadline.


#One Budget

Every job on the island shares one budget: a Kubernetes deadline of one hour on the pod, and a runner wallclock of fifty-five minutes below it. The gap is what lets a timing-out runner post its failure and flush its logs before Kubernetes stops it.

The budget is deliberately generous. A method compile, a web build and a browser audit do not need three different clocks; they are jobs on a node group with one timeout. Real builds take seconds to a few minutes, so the deadline is a backstop and not a target, and an unusual app with a slow build is not killed for being unusual.


#Scale From Zero

The jobs island sits at zero nodes when nothing is building. The cluster autoscaler cannot inspect a node that does not exist, so the node-group module publishes tags describing what a node would look like: the Kata label the runtime class requires, the taint, the GPU and ephemeral-storage resources. Those tags are what let the autoscaler conclude that a pending job pod would fit and bring a node up for it.

What This Gives an App
Its dependencies are resolved and compiled in the exact environment they will run in.
A build storm on one app never takes a core from a serving sandbox.
Builds are isolated in their own VMs, like everything else that runs app code.
A slow build is given room; a failed build reports why.
DiagramFIG. 07 — JOB RUNNERS
imagescript · fileparams · envlabelRelease worker imagethe serving runtimeRelease buildthe callerOrchestratorcreates the job sandboxS3outputs uploaded · presigned grantsCompletion callbackposts the result, then exitsLogsstreamed while it runsIMAGES DERIVED FROM THE WORKERmethod-compilerweb-interface-compilerfrontend-diagnosticssame commit · ABI-matchednode · platform · arch · libctoolchain ships in the runner images, not theworker, so serving pods stay leanruns askata_jobs · JOBS ISLANDKATA microVMMethodcompilerdoes bundle methods to JS· build the dep artifactfeeds the release buildKATA microVMWeb interfacecompilerdoes run the web build· upload the outputfeeds the release buildKATA microVMFrontenddiagnosticsdoes audit the built UI· headless browserfeeds post-deploy checkseach runner is a self-contained bundle · its harness (logging, uploads,completion callback) is bundled in · no import graph to resolve in-podcreate ·detachedreturns as soon as the pod exists,never waits for readinessSANDBOX SPECthe runner only reaches outnothing ever connects ina timed-out runner still posts a failure completiona sandbox cannot reach the orchestrator (egress denies internal ranges), so inputs arrive as files + env
image derived from the workerjob runner · its own Kata microVMplatform build & createthe runner's only path: outwardkata_jobs grouping
A build is a job: boot, do one thing, exit. Three kinds of job run on the kata_jobs island, each in its own Kata microVM, each built from an image derived from the release worker at the same commit. What is compiled matches what runs, ABI-matched down to the Node version, platform, architecture, and C library. The method compiler and web interface compiler feed the release build; frontend diagnostics feeds post-deploy checks. The release build calls the orchestrator to create a job sandbox from a spec. The orchestrator creates the pod detached, returning as soon as it exists rather than waiting for readiness, since nothing ever connects to a runner. A runner reads its inputs from files and environment (a sandbox cannot reach the orchestrator, so inputs are pushed in at creation), does its work, uploads outputs to S3 through presigned grants, posts a completion callback, and exits, streaming logs throughout.

``` Figure 07, "Job Runners": the kata_jobs island, read as one caller fanning out to disposable runners that only ever reach back out. Top, provenance: one release worker image distributes down a bus into four images built from it at the same commit (method compiler, web interface compiler, frontend diagnostics, browser session), ABI-matched down to the Node version, platform, architecture and C library; the build toolchain ships in the runner images, not the worker. Center, the kata_jobs island: each image runs as a pod in its own Kata microVM, four parallel columns compared across the row. Each pod shows its runner script and input as files (a ConfigMap owned by the pod) and its settings as environment. The method compiler bundles methods and builds the dependency artifact, the web interface compiler runs the web build and uploads it, and both feed the release build; frontend diagnostics audits the built interface in a headless browser and feeds post-deploy checks; the browser session holds one Chrome open, long-polls for the next action and serves an app's browser calls, with a heartbeat beside its loop. Each runner is a self-contained bundle with its harness bundled in, so it has no import graph to resolve in the pod. A clock on the island reads deadline one hour, runner fifty-five minutes. The isolation axis: from the left, the release build drives the orchestrator, which creates each job sandbox detached from a spec (image, files, environment, a label), returning as soon as the pod exists rather than waiting for readiness. From the island, a dashed egress fan is the runner's only path, and it is outbound only: outputs uploaded to S3 through presigned grants, a completion callback that posts the result, and streamed logs. Nothing ever connects in, and a timed-out runner still posts a failure completion. A second dashed arrow runs from the orchestrator's reaper to the same callback, labeled as a runner that died, posted for. Inputs arrive as files and environment because a sandbox cannot reach the orchestrator. A small cluster autoscaler box beside the island carries the note "node-template tags, scale from zero". No trust boundary in this figure, so no crimson.