Release Sandboxes
The sandbox a release's methods run in, the warm pool it comes from, and what is inside it once assigned.
A deployed app's methods run in a release sandbox: a Kata microVM on the kata_apps island, assigned to one release, kept warm while the app is in use, and replaced when it is not. This chapter covers the lifecycle of that sandbox, the warm pool it comes from, what is inside it, and how it talks to the platform.
#Persistent, Not Per-Invocation
A release sandbox is not created per request. When a method is first invoked for a release, the platform claims a warm sandbox from the pool, configures it for that release, and records the assignment. Subsequent invocations for the same release go to the same sandbox over a persistent WebSocket connection. The sandbox stays alive as long as it is being used.
The pod owns its own death. At creation, the platform injects its timings as environment variables from one configuration file. The worker reads them and hardcodes nothing.
| Clock | Value | What it does |
|---|---|---|
| Unclaimed max age | 4 hours | A warm pod that is never assigned exits on its own |
| Idle timeout | 30 minutes for a person's call; for background work, 30 seconds, up to about 2 minutes to wait for a cron's next run | Each execute keeps the pod this long after it arrives and after it finishes. The pod exits when the latest of these has passed with nothing running |
| Handler timeout | 30 minutes | The cap on a single method execution |
| Background cutoff | 30 minutes | How long post-response background work may keep flushing |
| Shutdown drain | 25 seconds | In-flight work finishes before the process exits on SIGTERM |
| Kubernetes deadline | 24 hours | The hard backstop for a wedged pod that never self-exits |
A release has one sandbox for all its work, reused across invocations, and each call says how long it keeps the pod. A call a user is waiting on (interactive) keeps it for 30 minutes. Background work keeps it briefly. This covers cron ticks, inbound email, jewel runs, and data-source mapper and reader frames. A cron run keeps the pod until just past the app's next cron run when that run is within two minutes, so an app with a run every minute or two holds one pod rather than booting one per run. A provision for background work draws only from the warm pool's shared tier, never from the part held for people waiting.
#The Warm Pool
Every release sandbox comes from a warm pool: pods booted ahead of demand, labeled warm, waiting to be claimed. A claim turns a warm pod into this release's pod in tens of milliseconds, not the seconds a cold boot takes.
Global, not per replica. The orchestrator runs as several replicas. The pool is one pool across all of them. Which warm pod a caller receives is decided by an atomic pop in a Redis registry. The pool label on the pod then flips to claimed, and that flip is the durable record. The registry is an index that can be rebuilt from the pods, not a second source of truth.
Two tiers. A reserved tier is claimable only by interactive work, where a person is waiting. A shared tier is claimable by anyone, including background work. The tier is stamped on the pod at birth, so the claim enforces the partition instead of relying on callers to respect it.
One leader sizes, every replica admits. The control plane observes demand and decides how many pods should exist. It runs on exactly one replica, chosen by leadership over a Valkey lease. The data plane (the atomic claim and the waiter queue) runs on every replica. This is the standard Kubernetes controller shape: one leader-elected reconciler, everything else stateless.
The sizing is textbook queueing. The controller measures arrival rate and boot time. It holds the in-flight quantity those imply plus square-root staffing headroom, or stock for the largest burst it has recently observed, or everyone currently queued, whichever is largest, since each counts the same arrivals. The target is the maximum of recent wants over a stabilization window, so scale-down happens by samples expiring rather than by a decay constant. Every term is a pure function of declared inputs, with no clock and no I/O in the decision path. That is what lets the controller's scenarios run deterministically in a regression harness.
Admission. When no warm pod is available, an interactive caller waits, within a bounded budget, for a fill. If the pool cannot honor that wait, the caller is refused with a clear error instead of being left hanging. Creates are paced per pass to stay inside the Kubernetes API server's fairness limits, and a rate-limit response from the API server pauses fills with jitter so replicas do not resume in lockstep.
#What Is Inside
The worker image is a pure runtime: a pinned Node.js, the platform SDK, and a worker process that listens on the worker port. It runs as an unprivileged user. It carries no compiler toolchain. A release's native dependencies are built once at publish and shipped in the dependency artifact.
At assignment the platform configures the sandbox with everything the release needs:
- The dependency artifact. A prebuilt dependency tarball for the release, fetched by the pod from a short-lived presigned URL, stamped with the ABI it was built against so the worker can verify a match before extracting. See Job Runners.
- The app's own secrets. The values of the secrets the builder stored, prod for a live release and dev for a branch preview, decrypted by the platform and injected as environment variables. See Secrets.
- The callback origin. The one hostname the sandbox reaches back on.
- Its own sandbox id. Every execute frame names the sandbox it is for, and the worker rejects a frame addressed to a different id. Pod IPs are recycled quickly, and this check keeps a stale routing entry from running on an honest pod. The check runs in the same process as the tenant's code, so what keeps misaddressed frames rare in the first place is that a route always expires before its pod can.
Each execute frame carries the compiled handler by its immutable artifact key, the handler name, the parameters, a hook token scoped to this execution, the auth context, and the database bindings for the environment. The worker caches compiled handlers by key, so code travels only on a cold miss.
The sandbox holds only what belongs to this app. Every managed capability the app uses is a call back into the platform, authorized by the hook token: database queries go to the app database role, platform actions go to the hook endpoints, and model calls go to the model service. No provider key, no platform credential, and no other app's data is ever inside the VM.
#Assignment, Reuse and Invalidation
The HTTP role owns the mapping from release to sandbox in Redis. When a release is promoted, the previous release's sandboxes are stopped by app label, and the new release claims fresh ones on first use. A stopped sandbox retires: it finishes the calls it is running, for up to the 30-minute handler limit, and takes no new ones, which go to a fresh sandbox.
When a secret changes, the app's sandboxes are stopped the same way, and the write waits for the stop. If the orchestrator cannot confirm it, the builder is told the running app still has the old value, and saving again retries.
Dead sandbox detection and retry live in the HTTP role. A sandbox that provably never ran a call is dropped from the mapping, and the call runs on another. A call whose frame reached the sandbox and whose result was lost fails instead, because running it again could run it twice.
The worker image is pinned to a moving alias. Every pod start re-checks the digest, and the pool controller recycles warm pods running a stale image, so a push to the image rolls the warm pool without a platform deploy.
``` Figure 06, "Release Sandboxes": the lifecycle of one sandbox on kata_apps, read left to right. Left, the warm pool: a leader-elected pool controller (one replica sizes it from arrival rate, boot time, the largest recent burst, and the queue; every replica admits) over two shelves of warm pod glyphs, reserved (interactive only) and shared (anyone, including background). A claim is an atomic pop from a Redis registry, in tens of milliseconds. Center, the claimed sandbox as the hero: a "claimed" state pill marks the flip from warm; it carries its release and app id. It serves the app's users and its cron, email and jewel runs alike. Its anatomy is a divided list: worker runtime (pinned Node.js, platform SDK, unprivileged), the dependency artifact (a prebuilt dependency tarball, ABI-stamped, fetched from a presigned URL, shown as a highlighted row), app secrets (injected as environment variables), a compiled-handler cache (by artifact key), and its sandbox id (named in every frame). The clocks are drawn on the pod: idle deadline (each call's own timeout: 30 minutes interactive, 30 seconds to 2 minutes background), handler timeout at 30 minutes, and the 24-hour deadline. A footer sets apart the negative claim: it holds only this app's code, dependencies, and secrets, and no provider key, platform credential, or other app's data. Right, the platform: the HTTP role configures the sandbox (dependency URL, secrets, callback origin, sandbox id), then drives it over a persistent WebSocket with execute frames (artifact key, handler, parameters, hook token). The sandbox's only path back is the callback origin, over HTTPS with a per-execution hook token, reaching three surfaces shown by a dashed egress fan: the app database role, the hook endpoints, and the model service. A bottom note carries invalidation (stop-by-app on promote and on secret change; the sandbox retires after its running calls finish). No trust boundary in this figure, so no crimson.