ReferenceAsk
Remy Reference/Architecture/Architecture/Voice
30Architecture

Voice

How an app answers the phone and holds a realtime voice conversation, with LiveKit for media, Telnyx for telephony, and a voice worker that bridges to the model and exposes the app's methods as tools.

An app can answer the phone, place calls, and hold a realtime voice conversation in the browser, with its methods available to the conversation as tools. The media runs through LiveKit and telephony through Telnyx; a dedicated voice worker bridges the two to the realtime model providers and back to the platform. This chapter covers the components and a call's lifecycle.


#The Components

The voice worker is its own image and Deployment, a LiveKit Agents process that registers with LiveKit Cloud and receives a job per session. It imports nothing from the platform's code; the whole boundary is HTTP, typed as a wire contract that the two sides evolve together. It holds no keys of its own either: its one secret is a standing token, with which it fetches the deployment's LiveKit and model keys from the platform, once at boot to register and again as each call starts, so a key rotated in the admin reaches the next call. Its lifecycle is call-shaped: surge-only rollouts, a long drain on SIGTERM so a live call finishes, and scaling on CPU.

LiveKit carries the media. A browser joins a room directly over WebRTC. A phone call arrives from Telnyx over SIP into a room. The worker joins the room as the agent.

Telnyx provides the phone numbers. The platform searches, orders, and configures numbers on the workspace's behalf, binds them to apps, sets caller-ID names, and bills monthly rental for dedicated numbers. A sandbox tier of shared numbers serves development.

The platform mints every session, holds the configuration, resolves the engine, receives tool calls, stores transcripts, and bills usage.


#Engines

A voice interface declares its engine in its interface config. Two shapes are supported: a native speech-to-speech model with a voice, or a cascaded pipeline of a language model, a speech-to-text model, a text-to-speech model, and a voice. The platform normalizes either into a canonical engine and resolves each slot against the model catalog. Slot credentials follow the workspace's vendor controls: a workspace's own key for a provider is used when present, and the platform's otherwise. The same resolution runs at build time to reject an engine that is not wired, and at billing time to price what ran.


#A Call's Lifecycle

  1. Admit. A browser calls the app's voice routes, a method places an outbound call, or an inbound call arrives on a bound number. One admission sequence serves all three: the workspace must be able to pay (balance and budgets, with one auto-refill), and the app and the caller must be under their concurrent-call limits. The platform then creates the session row, with its channel recorded explicitly as web, phone-in, or phone-out, and mints the worker's per-session credential.
  2. Dispatch. The platform sends the worker into the room itself, with dispatch metadata carrying the credential, the caller's identity, and, for a phone call, how the caller may verify. The browser's LiveKit token only joins the room. An inbound call is the exception: LiveKit's dispatch rule starts the job, and the worker registers the call with the platform using its standing token, which answers with the same things. From then on the worker authenticates every request with the session's own credential.
  3. Configure. The worker fetches the runtime configuration: the resolved engine, the tools the interface exposes, and the app's instructions.
  4. Converse. The worker bridges the room's audio to the realtime model. When the model calls a tool, the worker posts the call to the platform, which runs the method through the normal execution path with the voice session as its originating identity, and returns the result to the model. Tools can be fast, slow, or background; a background tool releases the turn and resolves later.
  5. Transcribe. Spoken turns and tool rounds append to the session's transcript as they happen, with previews rather than full payloads for tool traffic.
  6. Finalize. When the call ends the worker finalizes the session; the platform closes the transcript, records duration and model usage per slot, and bills the workspace. A call the platform ends itself (a suspension, a deletion, a transfer), or whose worker died, is billed from the usage snapshot the worker flushes every few seconds.

Outbound calls start from a method through the SDK, authenticated with the hook token, and follow the same lifecycle. A caller can list, read, and delete their own finished calls; the app's workspace sees every call its app has had.

What This Gives an App
A phone number and a browser voice channel with nothing to host.
Its methods as tools in a spoken conversation, authorized like every other call.
A transcript per session, with tool activity on the timeline.
Model choice per slot, including the workspace's own keys.
DiagramFIG. 30 — VOICE
Realtime modelspeech-to-speech, orLLM + STT + TTS + voiceBrowserjoins over WebRTCPhone callPSTN via Telnyxarrives over SIPLiveKit roomcarries the mediaWebRTC and SIP meet hereVoice workera LiveKit Agents processjoins the room as the agentThe platformsession: web · phone-in · phone-outengine · transcript · billingA method runstool call · voice session identityVOICE · A CALL'S PATHbridges audio and textper-session credential: register, config, tools, transcript, finalizeeach tool callthe execution path · Fig 15TELNYX · NUMBER MANAGEMENTsearch → order → bind to app → caller-ID → monthly billing
caller · component · the workermedia and control flow
One room, one worker, the app’s methods on the line. A call reaches a LiveKit room two ways: a browser joins over WebRTC, and a phone call arrives from Telnyx over SIP. The voice worker, a LiveKit Agents process that imports none of the platform’s code, joins the room as the agent and bridges its audio to a realtime model, either a native speech-to-speech model or a cascaded pipeline of a language model, speech-to-text, text-to-speech, and a voice, each slot resolved against the model catalog under the workspace’s vendor controls. The worker talks to the platform with a per-session credential: it registers an inbound call, fetches the runtime configuration, posts each tool call, appends to the transcript, and finalizes. The platform mints every session with its channel recorded as web, phone-in, or phone-out, resolves the engine, stores the transcript, and bills usage per slot; each tool call runs through the normal execution path as a method, with the voice session as its identity. Telnyx number management, searching, ordering, binding a number to an app, setting caller-ID, and monthly billing, runs alongside. No trust boundary is drawn here, so there is no crimson.

``` Left: a browser (WebRTC) and a phone (PSTN → Telnyx → SIP) both entering a LiveKit room. Center: the voice worker joins the room as the agent. Arrows: worker ⇄ realtime model provider (audio/text) worker → platform: the deployment's keys (standing token, at boot and each call's start) · register (standing token, inbound only) · runtime config · tool calls · transcript appends · finalize (all with the per-session credential) platform → method execution for each tool call (voice session as identity) Right: platform: session row (channel: web | phone-in | phone-out) · engine resolution (S2S or LLM+STT+TTS+voice, catalog-backed, workspace keys honored) · transcript · billing. Bottom: Telnyx number management: search → order → bind to app → caller-ID → monthly billing.