Models and Routing
The one model service every AI call passes through, how a call's route is resolved and where inference runs (Bedrock, first-party providers, the in-cluster pool, or a workspace's own key), and the catalog and controls around it.
Every AI model call on the platform, from an app's method, from the agent, or from the editor's coding agent, passes through one model service. Voice sessions are the one exception: the voice worker runs its realtime and streaming speech models through LiveKit's provider plugins, and the voice domain meters each session from its usage when it ends. The catalog of models is code; each model declares who made it, who serves it, what it costs, and where its data goes; and a workspace controls which providers it uses and with whose keys. This chapter covers the catalog, the routing of a call, and the controls around it.
#The Catalog
The catalog lives in code, one file per model, keyed by publisher: who made the model. Each entry declares the model's id, type, capabilities, pricing, the adapter that speaks to its serving vendor, and the product facts the picker shows. A generated registry indexes the whole catalog at build time; adding a model is a new file and a regeneration, never an admin API call.
Adapters are keyed by provider: who serves the model. The two are deliberately different trees. Many publishers have no serving API of their own and are reached through a hosting vendor, and one publisher's model can be served by several providers. Adapters extend a base class and take a resolved credential in their constructor, so an adapter never reads a key from configuration.
Model types cover chat and reasoning, embeddings, image generation and editing, video, speech-to-text, text-to-speech, realtime voice, document extraction, reranking and more, each with its own input and output vocabulary.
#Routes
A model can declare several routes: serving arrangements for the same product. Each route names its adapter, the vendor's wire name for the model, the service it egresses through, the capabilities it supports, and its facts: the data boundary and the region the data is processed in. A route can carry its own rate card when the vendor's price differs from the product's.
A call picks one route, in this order:
- A pin. A caller can name a route explicitly with model@route. A pin is hard: if the route is not usable for this workspace, the call fails rather than silently rerouting.
- The workspace's own key. A route whose service the workspace holds a key for is preferred.
- The capability filter. A request that needs server-side tools, a hosted connector, or image input drops routes that cannot serve it.
- The preferred route set by the operator for the model, or the primary route when none is set.
A route whose service the workspace has disabled is removed before selection; the model as a whole is unavailable only when no route survives. The choice is deterministic rather than load-balanced, which keeps prompt caches coherent: a conversation stays on one provider unless a later request needs a capability its route lacks (an image, a server-side tool), and the capability filter moves it.
The facts on each route are shown to the user in the model picker, so a workspace can see and choose where inference runs.
#Where Inference Goes
| Route class | Path | Boundary |
|---|---|---|
| Amazon Bedrock | A PrivateLink interface endpoint inside the trusted VPC | Traffic never leaves the AWS network |
| First-party providers | HTTPS through the NAT gateways | The provider's own boundary, declared in the registry with its BAA and retention facts |
| The platform's own serving pool | An in-cluster Service on the GPU node group | Never leaves the cluster |
| A workspace's own key | The same paths, under the workspace's account | The workspace's own agreement with the vendor |
Every provider is a surface in the counterparty registry, with its hosts, data classes, retention mode, training policy and agreements. When a provider's agreement excludes a feature, the adapter strips that feature from requests rather than attempting it. See Counterparties and Egress.
#The Model Service
The model service is the single chokepoint for every call outside a voice session. For each call it:
- resolves the model and route as above, through the model factory and the clearinghouse;
- checks the workspace's credit and plan state;
- admits the call under the capacity gate, keyed by whose key, which provider and which model, shared fleet-wide through Redis leases, shrinking on a vendor refusal and regrowing on success, with bulk loads waiting behind a reserve and interactive calls failing fast rather than hanging;
- executes through the adapter, streaming where the caller streams;
- bills the workspace: every adapter declares the billing events it can emit, each event has a declared price in the catalog, and a call under a workspace's own key records distinct own-key events that bill nothing, since the workspace pays the vendor directly;
- records metrics with the route as a dimension, so cost, latency and error rates are readable per app, per model and per serving route;
- classifies errors so a workspace-state condition, such as a disabled model, is reported to the caller as what it is and never logged as a platform failure.
#Workspace Controls
- Enable and disable any provider, or a single model of a provider.
- Bring your own key for a provider or a single model, honored by every route through that provider.
- Default models per workspace for the agent, the editor and the picker.
- Capability overrides granted by the operator, additive per workspace, for models or features outside a plan.
- Custom and local models: on a plan with custom models, a workspace can register its own public endpoint as a model, billed at no platform charge. Such a model runs only on the custom-model adapter, never on a vendor's.
``` Figure 32, "Models and Routing": one chokepoint for model calls, with voice as the exception. Left: callers. A method (through the SDK, with its hook token), the agent, and the editor's coding agent. An untrusted app holds no model key and calls no provider; it asks the platform. Apart, below these: the voice worker, calling vendors through LiveKit plugins and metered by the voice domain when a session ends. Center (a dashed grouping, "the model service, one chokepoint"): a vertical pipeline. Resolve the model and route (pin, own key, capability filter, preferred; via the model factory and the clearinghouse) into a credit and plan check, into the capacity gate (key, provider, model; Redis leases), into the adapter, and into bill (declared events; own-key variants that bill nothing), metrics (by route), and error classification. Right: from the adapter, three egress paths: Amazon Bedrock over a PrivateLink endpoint (stays on the AWS network), first-party providers via the NAT gateways (registry facts: BAA, retention, training), and the in-cluster serving pool on the GPU node group. A workspace's own key uses the same paths under its own account. Bottom: the catalog. A publisher tree and a provider tree (adapters), joined by routes carrying facts (data boundary, region) and capabilities, indexed by a generated registry. A cross-reference notes that every provider is a surface in Figure 20. Side: workspace controls as toggles: enable/disable, bring your own key, defaults, overrides, custom models. No trust boundary is drawn in this figure, so there is no crimson.