Observability

The platform is instrumented at four layers: application performance and errors from every role, traces and metrics in an open format, fleet metrics from the clusters, and the AWS account's own detection and logging services. This chapter covers each layer and what it watches.

#Application

New Relic APM runs in every role. Transaction traces carry the endpoint, the workspace, and the app, so a slow method dispatch or a slow query is attributable. A change marker is recorded on every successful production deploy, so a regression lines up against the release that introduced it.

Sentry receives every unhandled error and every explicitly reported one from every role, with environment tags so staging and production are distinguishable. The DSN comes from configuration rather than code, so the destination is visible in the deploy diff.

OpenTelemetry exports traces and metrics over OTLP from every role through a collector on the cluster. Custom metrics cover what the vendors do not see by default: sandbox provisioning phases, pool depth and queue, method execution timing, and the queue processors' in-flight counts.

Structured logs from every role use one logger with request context. Log volume is shaped deliberately: audit-grade facts go to the audit log, per-workspace request history goes to the request log, and everything else stays in the operational logs.

#Platform Metrics

Beyond the vendors, the platform records its own operational history in Postgres:

• Request logs per app: every invocation with its interface, environment, duration, and outcome.
• Method metrics and usage metrics in five-minute buckets, with the model route as a dimension, so cost and error rates can be read per app, per model, and per serving route.
• Sandbox metrics: provisioning origin (warm, queued, reuse), boot timing by phase, and execution timing including dependency install time.
• Frontend error metrics from deployed apps' browsers, per release.
• Cron run history, with lifecycle events for every transition.

The admin surface reads these for a fleet view of every sandbox, with health, labels, resource usage, and boot latency per pod.

#Clusters

CloudWatch Container Insights runs on the untrusted cluster for time-series fleet observability: sandbox and pod counts, per-node and per-pod CPU and memory, out-of-memory kills, restarts, node count, and network, with CloudWatch retention, dashboards, and alarms. It is configured metrics-only, with no container-log shipping, because it is the cluster where customer code runs.

The Kubernetes metrics API on both clusters feeds the fleet view and the autoscalers.

#The Account

• GuardDuty on the account, with EKS runtime monitoring on the clusters. Findings route to a chat channel and to email through EventBridge.
• CloudTrail for every control-plane call, with log-file validation and global service events, delivered to a dedicated bucket.
• VPC flow logs on both VPCs.
• Database audit logging through pgaudit, exported to CloudWatch Logs, with lock-wait and temp-file logging for performance diagnosis, and Performance Insights on the cluster.
• S3 server access logs on every private bucket, date-partitioned so they can be queried by day.
• Backup and restore events through EventBridge, alerting on a failed backup or restore-test job.
• Audit-log delivery watched by a CloudWatch alarm on the Firehose stream.

#Alerting

Alarms from CloudWatch and New Relic, GuardDuty findings, and backup events route to an on-call rotation through an incident management service, and to chat. Every infrastructure resource carries an ownership tag, enforced through the compliance monitoring described in Compliance and Attestation, so an alarm always has a named owner.