Execution Islands
The sandbox app code runs in, the three islands it becomes, and the contract that keeps the arrangement swappable.
App code runs in a Kata microVM with its own guest kernel, scheduled by Kubernetes onto EC2 nodes with nested virtualization, on a cluster in a VPC that cannot reach the platform. This chapter covers the sandbox primitive, the three islands it becomes, the hardening on every pod, and the provider contract that makes the arrangement swappable.
#The Primitive
A sandbox is a pod whose runtime class is Kata Containers. Instead of sharing the node's kernel the way an ordinary container does, each pod boots a lightweight virtual machine with its own kernel, and the container runs inside it. The boundary between app code and the node is a hypervisor, and the boundary between two apps is two hypervisors.
The nodes are EC2 instances with nested virtualization enabled, in dedicated node groups on the untrusted cluster. A node-group module owns the launch template that turns on nested virtualization, sets the instance metadata hop limit, applies the Kata taint and labels, and carries the tags the cluster autoscaler needs to model a node that does not exist yet. The same module is instantiated three times, once per island.
Kata's node configuration is asserted twice. A privileged DaemonSet keeps the node-level defaults correct, and every pod also carries runtime-scope annotations that pin the two settings that matter: the seccomp profile is applied inside the guest, and the sandbox's cgroup is anchored so the VM survives systemd's reclaim. A sandbox asserts its own hardening and does not depend on landing on a freshly configured node.
#Three Islands
The platform runs untrusted code in three places: separate node groups, separate resource profiles, separate lifetimes, and one shared construction path.
| Release sandboxes | Job runners | Dev boxes | |
|---|---|---|---|
| Node group | kata_apps | kata_jobs | kata_dev |
| What runs | A deployed app's compiled methods | A build: method compile, web build, diagnostics | A person's development environment |
| Lifetime | Assigned to a release, exits on idle | Seconds to minutes, exit 0 | Hours, on a lease |
| Image | The worker image | Runner images derived from the worker | The dev-box image |
| Reachability | The worker port, from the platform | None | The dev-box port, from the sandbox proxy |
| Scaling | A warm pool sized to demand | On demand, from zero | Warm-ish, few |
| Chapter | Release Sandboxes | Job Runners | Dev Boxes |
The rule is: share code, never share runtime. One module builds the pod manifest, applies the hardening, stamps the labels, and reaps terminated pods, so a hardening fix lands once. Three provider instances apply it with three configurations. A janitor on one island never sees another island's pods, because every pod carries an island label and every selector is rooted in it.
#What Every Pod Carries
The manifest builder produces one shape for every sandbox, and the release worker's container is byte-for-byte what the warm pool boots.
| Setting | Value | Why |
|---|---|---|
runtimeClassName | The Kata runtime class | A VM per pod |
seccompProfile | RuntimeDefault, asserted in the guest by annotation | The syscall filter applies inside the VM |
capabilities | Drop all | The worker binds an unprivileged port and needs none |
allowPrivilegeEscalation | false | Nothing in the pod escalates |
automountServiceAccountToken | false | No credential to present to the cluster |
dnsPolicy | None, public resolvers | Cluster-internal names do not resolve at all |
restartPolicy | Never | A dead sandbox is replaced, never resurrected in place |
activeDeadlineSeconds | Per island | Kubernetes hard-kills a wedged pod regardless of what it is doing |
imagePullPolicy | Always | Images are moving aliases; every start re-checks the digest |
| Resource limits | Per island | Bounded CPU and memory per sandbox |
| Labels | app, island, pool, tier, release, app id, role | What the egress policy and the janitors select on |
Files a runner needs before it starts arrive as a ConfigMap named after the pod, mounted into the image's working directory, and owned by the pod so Kubernetes garbage-collects it. Nothing is ever written into a running sandbox from outside.
The sandbox NetworkPolicy from The Trust Boundary selects on the app label every sandbox carries, so the value is defined once and the policy follows it.
#The Provider Contract
The orchestrator talks to sandboxes through one interface. A sandbox is a self-starting OCI image, booted from a spec. The contract is declarative: no exec into a running sandbox, no file writes after boot, no snapshots. Everything a workload needs is in the spec at create time.
The spec. An image, a command and arguments, environment, a working directory, files to place before start, labels to stamp at birth, and three hints: one to return as soon as the pod exists, one to fail fast when a node cannot be found, and one to report boot milestones to whoever is waiting on it. Every field is optional, and absent means exactly the provider's default, which is the release worker on its configured image.
Six operations. Create a sandbox, fetch one, find the sandboxes for a release, stop an app's sandboxes, relabel a sandbox, and reap terminated pods. A running sandbox reports its address on a port, and stops on request.
A required warm pool. The release path provisions only from the pool and has no cold-boot path, so a provider that cannot boot ahead of demand cannot serve a release. The pool is global across every orchestrator replica: which warm sandbox a caller receives is decided by an atomic pop in a Redis registry, and the pool label on the pod is the durable record of the claim.
Optional capabilities. A fleet view for the admin console, showing per-pod health, labels, resource usage, and boot timing broken into schedule, image and VM, and worker start; and log streaming for a pod's output. A consumer checks a capability once and calls it.
The Kubernetes provider implements the contract today. The same shape lets a Docker Engine provider, or a remote sandbox service, implement the same operations and pool, and slot in beneath the release manager unchanged.
Figure 05, "Execution Islands": a three-tier plate. One hardened primitive, three islands, one construction path. Top left, the primitive as a nested-containment box: an EC2 node (nested virtualization) contains a Kata microVM (its own guest kernel), which contains the container (app code: a worker, a runner, or a dev box). The isolation claim below it: code to node is one hypervisor; app to app is two. Top right, a Manifest builder bar (the Kubernetes provider, one module to three configs) fans down into a dashed "sandbox namespace" zone holding three island columns, each a pool of pods. The columns compare across the row: Release sandboxes (kata_apps: compiled methods, idle-exit, worker port, warm pool), Job runners (kata_jobs: builds and diagnostics, seconds then exit 0, no reachability, from zero), and Dev boxes (kata_dev: a dev environment, hours on a lease, dev-box port, warm-ish). The middle band, "what every pod carries", is a strip of hardening chips: runtimeClass Kata, seccomp RuntimeDefault, drop all capabilities, no privilege escalation, no ServiceAccount token, dnsPolicy None, restartPolicy Never, a per-island active deadline, imagePull Always, and CPU and memory limits; plus the labels (app, island, pool, tier, release, appId, role) the egress NetworkPolicy and the janitors select on. The bottom band, "the provider contract", is the swappable point: the sandbox orchestrator (the only caller) drives one declarative contract (no exec, no writes after boot, no snapshots; six operations; a required warm pool claimed by an atomic pop in a Redis registry), and a dashed "other providers" box (Docker Engine, a remote sandbox service) implements the same contract and slots in unchanged. No trust boundary in this figure, so no crimson.