ReferenceAsk
Back to Remy
Remy Reference/Architecture/Architecture/Voice
30Architecture

Voice

How an app answers the phone and holds a realtime voice conversation, with LiveKit for media, Telnyx for telephony, and a voice worker that bridges to the model and exposes the app's methods as tools.

An app can answer the phone, place calls, and hold a realtime voice conversation in the browser, with its methods available to the conversation as tools. The media runs through LiveKit and telephony through Telnyx; a dedicated voice worker bridges the two to the realtime model providers and back to the platform. This chapter covers the components and a call's lifecycle.

#The Components

The voice worker is its own image and Deployment, a LiveKit Agents process that registers with LiveKit Cloud and receives a job per session. It imports nothing from the platform's code; the whole boundary is HTTP, typed as a wire contract that the two sides evolve together. Its lifecycle is call-shaped: surge-only rollouts, a long drain on SIGTERM so a live call finishes, and scaling on active calls.

LiveKit carries the media. A browser joins a room directly over WebRTC. A phone call arrives from Telnyx over SIP into a room. The worker joins the room as the agent.

Telnyx provides the phone numbers. The platform searches, orders, and configures numbers on the workspace's behalf, binds them to apps, sets caller-ID names, and bills monthly rental for dedicated numbers. A sandbox tier of shared numbers serves development.

The platform mints every session, holds the configuration, resolves the engine, receives tool calls, stores transcripts, and bills usage.

#Engines

A voice interface declares its engine in the manifest. Two shapes are supported: a native speech-to-speech model with a voice, or a cascaded pipeline of a language model, a speech-to-text model, a text-to-speech model, and a voice. The platform normalizes either into a canonical engine and resolves each slot against the model catalog. Slot credentials follow the workspace's vendor controls: a workspace's own key for a provider is used when present, and the platform's otherwise. The same resolution runs at build time to reject an engine that is not wired, and at billing time to price what ran.

#A Call's Lifecycle

  1. Mint. A browser calls the app's voice routes, or an inbound call arrives on a bound number. The platform creates the session row, with its channel recorded explicitly as web, phone-in, or phone-out, and mints a LiveKit room token whose dispatch metadata carries everything the worker needs.
  2. Dispatch. LiveKit dispatches a job to the worker. For an inbound call the worker first registers the call with the platform using its standing token, since no session exists yet; from then on it authenticates every request with the session's own credential.
  3. Configure. The worker fetches the runtime configuration: the resolved engine, the tools the interface exposes, the app's instructions, and the caller's verification state.
  4. Converse. The worker bridges the room's audio to the realtime model. When the model calls a tool, the worker posts the call to the platform, which runs the method through the normal execution path with the voice session as its originating identity, and returns the result to the model. Tools can be fast, slow, or background; a background tool releases the turn and resolves later.
  5. Transcribe. Spoken turns and tool rounds append to the session's transcript as they happen, with previews rather than full payloads for tool traffic.
  6. Finalize. When the call ends the worker finalizes the session; the platform closes the transcript, records duration and model usage per slot, and bills the workspace.

Outbound calls start from a method through the SDK, authenticated with the hook token, and follow the same lifecycle from dispatch.

What This Gives an App
A phone number and a browser voice channel with nothing to host.
Its methods as tools in a spoken conversation, authorized like every other call.
A transcript per session, with tool activity on the timeline.
Model choice per slot, including the workspace's own keys.
DiagramFIG. 30 — VOICE
Realtime modelspeech-to-speech, orLLM + STT + TTS + voiceBrowserjoins over WebRTCPhone callPSTN via Telnyxarrives over SIPLiveKit roomcarries the mediaWebRTC and SIP meet hereVoice workera LiveKit Agents processjoins the room as the agentThe platformsession: web · phone-in · phone-outengine · transcript · billingA method runstool call · voice session identityVOICE · A CALL'S PATHbridges audio and textper-session credential: register, config, tools, transcript, finalizeeach tool callthe execution path · Fig 15TELNYX · NUMBER MANAGEMENTsearch → order → bind to app → caller-ID → monthly billing
caller · component · the workermedia and control flow
One room, one worker, the app’s methods on the line. A call reaches a LiveKit room two ways: a browser joins over WebRTC, and a phone call arrives from Telnyx over SIP. The voice worker, a LiveKit Agents process that imports none of the platform’s code, joins the room as the agent and bridges its audio to a realtime model, either a native speech-to-speech model or a cascaded pipeline of a language model, speech-to-text, text-to-speech, and a voice, each slot resolved against the model catalog under the workspace’s vendor controls. The worker talks to the platform with a per-session credential: it registers an inbound call, fetches the runtime configuration, posts each tool call, appends to the transcript, and finalizes. The platform mints every session with its channel recorded as web, phone-in, or phone-out, resolves the engine, stores the transcript, and bills usage per slot; each tool call runs through the normal execution path as a method, with the voice session as its identity. Telnyx number management, searching, ordering, binding a number to an app, setting caller-ID, and monthly billing, runs alongside. No trust boundary is drawn here, so there is no crimson.

Figure 30, "Voice": a call's path, a bridge topology. Two callers reach a LiveKit room: a browser joins over WebRTC, and a phone call arrives from Telnyx over SIP. The voice worker (the heavy node) joins the room as the agent, and bridges its audio and text to a realtime model drawn above it (two arrows, one each way): a native speech-to-speech model, or a cascaded pipeline of a language model, speech-to-text, text-to-speech, and a voice. To the right, the worker talks to the platform on a per-session credential: register, config, tool calls, transcript appends, and finalize. Each tool call drops to a method that runs with the voice session as its identity. The platform mints the session (channel: web, phone-in, or phone-out), resolves the engine, stores the transcript, and bills. A footer band carries Telnyx number management: search, order, bind to app, caller-ID, and monthly billing. A cross-reference points to Figure 15 for the execution path each tool call rides. No trust boundary is drawn in this figure, so there is no crimson.