ReferenceAsk
Remy Reference/Architecture/Architecture/Observability
25Architecture

Observability

The four layers the platform is instrumented at, from application performance and errors to the AWS account's own detection and logging, and how an alert always reaches a named owner.

The platform is instrumented at four layers: application performance, traces, metrics and errors from every role, the platform's own operational history in Postgres, fleet metrics from the clusters, and the AWS account's own detection and logging services. This chapter covers each layer and what it watches.


#Application

Every role process starts with one preload, before anything it instruments has loaded, so every role is instrumented the same way.

New Relic APM runs in every role wherever the deployment has its key. Each role is its own application, under the name its traces carry. Its transactions, like the OpenTelemetry server spans below, carry the endpoint, the workspace, the app and the trace id, so a slow method dispatch or a slow query is attributable and leads to its trace and log lines. It counts an error by the same rule the traces do, so its error rate reflects the platform's own failures. A served app call is named for the surface it reached, and the transaction for an open-ended stream ends once the stream is open, so averages measure work rather than how long a tab stayed open. A change marker is recorded on every successful production deploy, so a regression lines up against the release that introduced it.

OpenTelemetry sends traces, metrics and logs over OTLP from every role to the endpoint the deployment names, using the specification's standard settings. Any backend that speaks OTLP, or a collector in front of one, can receive them. With no endpoint configured, nothing is built or sent.

  • Traces follow a request across processes. They cover the request a role serves, the HTTP and fetch calls it makes (each carrying the trace onward), the background job it queues, and the handler that runs that job.
  • Errors are part of the trace. A platform fault is recorded on the span where it happened and marked failed. A caller's mistake or an app's own failure is answered and logged, not reported.
  • Metrics cover the Node runtime (event loop, garbage collection, heap) and what the platform counts itself: the model capacity gate's waits, windows and refusals, and a bulk load's stage timings.
  • Credentials in URLs never leave. Share and sign-in tokens, OAuth codes and the signatures of signed URLs are redacted from every span before export, and no request header is recorded.
  • The health report says whether export is working. It shows whether the endpoint accepts what is sent and, when it does not, the exporter's own error.

The console's browser errors go to New Relic Browser when the deployment names an application. Page loads and their timings are reported too, and credentials are obfuscated in every URL the agent sends. Stacks read as source: each deploy uploads the build's source maps to the application and serves none of them. Without a configured application, the console reports nowhere. What reaches it:

  • unhandled errors and rejections from the page's first byte (anything that occurs before the agent starts is held and handed to it);
  • scripts and chunks that fail to load;
  • render crashes caught by the error boundaries;
  • a saga that dies;
  • errors the code catches that are a fault of the platform's own, each named by where it was caught.

Each error names the build it came from.

Structured logs from every role use one logger, writing JSON in production to stdout and, where the deployment wants it, over OTLP. The cloud reads logs from stdout, which a crash writes to as well. A line written inside a trace carries the trace and span ids and the app and workspace the work is for, so a log search and a trace lookup lead to each other. Log volume is shaped deliberately: audit-grade facts go to the audit log, per-workspace request history goes to the request log, and operational logs stay operational.


#Platform Metrics

Beyond the vendors, the platform records its own operational history in Postgres:

  • Request logs per app: every invocation with its interface, environment, duration and outcome.
  • Method metrics and usage metrics in five-minute buckets, with the model route as a dimension, so cost and error rates can be read per app, per model and per serving route.
  • Sandbox metrics: provisioning origin (warm, queued, reuse), boot timing by phase, and execution timing including dependency install time.
  • Frontend error metrics from deployed apps' browsers, per release.
  • Cron run history, with lifecycle events for every transition.

The admin surface reads these for a fleet view of every sandbox, with health, labels, resource usage and boot latency per pod.


#Clusters

OpenTelemetry collectors run on both clusters. They are defined in the platform's infrastructure and deployment configuration rather than installed by hand, and they send to the same backend as the roles.

  • On the cluster that runs the platform:
    • an agent on every node takes the roles' telemetry from their own node, along with the node's and pods' resource use and the platform's container logs, with trace ids lifted from the logs so a log line and its trace lead to each other;
    • a gateway samples traces whole, keeping every trace with an error, every trace over two seconds, and a tenth of the rest;
    • one more collector reads the cluster's state and events.
  • On the cluster where customer code runs: metrics only, covering the nodes' resource use and the cluster's state and warnings. It has no input any sandbox could reach and ships no logs, since that cluster's output is customer code's.

CloudWatch Container Insights runs on the untrusted cluster for time-series fleet observability: sandbox and pod counts, per-node and per-pod CPU and memory, out-of-memory kills, restarts, node count and network, with CloudWatch retention, dashboards and alarms. It is configured metrics-only, with no container-log shipping, because it is the cluster where customer code runs.

The Kubernetes metrics API on both clusters feeds the fleet view and the autoscalers.


#The Account

  • GuardDuty on the account, with EKS runtime monitoring on the clusters. Findings route to a chat channel and to email through EventBridge.
  • CloudTrail for every control-plane call, with log-file validation and global service events, delivered to a dedicated bucket.
  • VPC flow logs on both VPCs.
  • Database audit logging through pgaudit, exported to CloudWatch Logs, with lock-wait and temp-file logging for performance diagnosis and Performance Insights on the cluster.
  • S3 server access logs on every private bucket, date-partitioned so they can be queried by day.
  • Backup and restore events through EventBridge, alerting on a failed backup or restore-test job.
  • Audit-log delivery watched by a CloudWatch alarm on the Firehose stream.

#Alerting

New Relic's synthetic checks of each public system (the API, the editor, app hosting, app databases, git, realtime, files and media) route to an incident management service. GuardDuty findings route to chat and email. Failed backup and restore jobs, RDS failure events and the audit log's delivery alarm publish to the operations alarm topic. Every infrastructure resource carries an ownership tag, enforced through the compliance monitoring described in Compliance and Attestation, so an alarm always has a named owner.

What This Gives a Workspace
A platform whose operators can see a slow request, a failing sandbox, or a saturated queue as it happens.
Detection at the account and cluster level, not only in application code.
Per-app history of every invocation and every cost, readable in the dashboard.
Deploys correlated with behavior, so a regression is found by release.