Scale and Reliability
How each tier scales on the signal that describes its load, what keeps it available through rollouts and failures, and the bounds that keep one workload from taking down another.
Each tier of the platform scales on the signal that describes its load, and each is arranged so that a rollout, a node replacement, or a pod failure is absorbed without a user noticing. This chapter covers how each tier scales, what keeps it available, and the bounds that keep one workload from taking down another.
#How Each Tier Scales
| Tier | Signal | Mechanism |
|---|---|---|
| HTTP and WebSocket roles | CPU and memory utilization | Horizontal Pod Autoscaler with a floor of several replicas |
| Voice worker | Active calls | HPA; surge-only rollouts so a deploy never ends a call |
| General worker fleet | Queue depth across steps, ingest and file scans | KEDA ScaledObject; the busiest queue wins |
| Bulk worker fleet | Bulk-ingest batches in flight | KEDA ScaledObject on its own node group; sized to batches, not backlog |
| Model trainer | Training queue | KEDA ScaledJob; one Job per message, from zero |
| Release sandboxes | Arrival rate, boot time, burst size, queue | The adaptive pool controller, on the apps island |
| Job runners | Pending build pods | Cluster autoscaler from zero on the jobs island |
| Dev boxes | Pending box pods, with a prepull balloon holding capacity | Cluster autoscaler on the dev island |
| Nodes, every group | Pending pods | Cluster autoscaler, with node-template tags for groups that start at zero |
| Platform database | Load | Aurora Serverless v2 capacity within configured bounds |
The worker fleets are split by what saturates them. A corpus load saturates one JavaScript thread per batch, for minutes at a stretch; a step is a caller waiting seconds. They run in different pods on different node groups, so a load never takes a core from interactive work. Each fleet's scaling target matches its shape: the interactive fleet scales early on a small backlog, while the ingest and scan queues scale only on a real backlog, because routine bursts are absorbed by baseline concurrency.
The warm-pool controller is the one tier sized by a model rather than a threshold. See Release Sandboxes.
#Redundancy
- Compute. Every role runs several replicas across availability zones. PodDisruptionBudgets on the API, socket, git, app-db, mail-inbound, sandbox-proxy, and voice tiers keep a minimum available through rollouts and node drains; they are integers rather than percentages, so they never round to zero at the floor.
- Database. Aurora with a writer and a reader across zones, behind RDS Proxy, with continuous backups and a cross-region copy.
- Cache. Valkey Multi-AZ with automatic failover.
- Object storage. S3, versioned, with per-region buckets.
- Queues. SQS, with dead-letter queues on the paths where a poison message would otherwise cycle.
- Edge. Cloudflare's network in front of every public hostname.
#Graceful Everything
A rollout, a scale-down, and a node replacement all arrive at a pod as SIGTERM, and every role has a drain designed for its work:
- Request pods stop accepting connections, finish in-flight requests, and exit within a short grace period.
- The app-db role refuses new ownership claims, stops accepting connections, flushes every dirty database, releases ownership one database at a time, and deregisters last, so a surviving pod can reclaim anything a kill interrupted.
- The git role flushes pending repository backups before it deregisters.
- The worker fleets stop taking messages and let in-flight jobs finish. A long copy chain interrupted by a rollout is handed back to the queue rather than cut, and visibility heartbeats keep a running job's message invisible to other consumers.
- The voice worker drains for up to fifteen minutes so a live call finishes.
- Release sandboxes finish in-flight handlers within a drain window that sits inside the pod's grace period.
Node groups roll blue/green: new nodes join on the new image before old nodes drain, and the disruption budgets pace the move.
#Bounds
Every unit of work has a ceiling, so a hung or runaway job is contained.
| Bound | Value |
|---|---|
| A method execution | 30 minutes |
| A queued job | 30 minutes, with visibility heartbeats until then |
| A build job | 1 hour on the pod, 55 minutes on the runner |
| A dev box | 12 hours on the pod, a 30-minute lease on the session |
| An idle release sandbox | 30 minutes |
| A sandbox's memory | A per-island limit, with the worker's heap capped beneath it |
| Realtime event grants | 15 minutes by default, 1 hour at most |
| Builds per push | Capped, with the default branch always inside the cap |
| Creates per pool pass | Paced to the Kubernetes API server's fairness limits |
Vendor capacity is a bound too. A gate keyed to the key, the provider, and the model admits calls up to what that key allows, shared fleet-wide through Redis leases that shrink on a refusal and regrow on success. Bulk loads wait for a slot and see the ceiling minus a reserve; interactive calls wait briefly, then fail with a clear error rather than hang behind a load.
#Failure Handling
- A dead sandbox is detected by the HTTP role, dropped from the release's mapping, and replaced by a fresh claim on the next request.
- A dead app-db pod loses its ownership records by TTL; surviving pods reclaim its databases from the reverse index without scanning.
- A stale git pod refuses reads until its copy catches up, and pushes repair it.
- A poison message parks in a dead-letter queue after a bounded number of attempts.
- An unroutable message returns to the queue rather than being deleted, so a routing bug leaves evidence.
- A failed migration fails the deploy before any pod rolls.
- A failed build leaves live untouched.
- A superseded build that finishes late is marked superseded rather than promoted.
- Redis unreachable degrades the capacity gate to a per-process floor with a warning rather than blocking calls.