Ditching the Call Center SaaS: Building a Voice Agent from Scratch

The short answer

You build an AI voice agent by connecting a telephony provider, a speech-to-text and text-to-speech pipeline, an LLM, and a call orchestration layer, then wiring it to your own knowledge base and business logic instead of renting a seat-based contact center suite. Platforms like Retell AI package the voice infrastructure so a small team can ship a working agent in days, at a per-minute cost instead of a per-seat license.

This matters because the SaaS math on the other side is bad and getting worse. Entry-level contact center seats run $65 to $119 a month per agent, with full-suite enterprise plans reaching $149 and up, before you add telephony, WFM, or analytics add-ons.¹ Other benchmarks put the range even wider, from $20 to $150-plus per agent per month depending on features and scale.² Multiply that by a 20-person support team and you are paying for headcount capacity whether or not the phones are ringing.

Why teams are walking away from seat-based pricing

Seat pricing was built for a world where a human had to be sitting in the seat. It charges you for capacity, not usage. A support team staffed for peak volume is overpaying for every quiet Tuesday afternoon. The industry itself is aware this model is cracking: 63% of software leaders believe AI will fully change their business model within three to five years, and McKinsey projects at least 40% of enterprise SaaS spend shifting to usage-based or outcome-based pricing by 2030.³

The fully-loaded cost of the humans behind those seats compounds the problem. A US-based call center agent's wage looks like $18 to $22 an hour, but benefits, attrition replacement, QA, supervision, and shrinkage roughly double that by the time it hits the P&L, landing the true cost around $29 to $42 an hour.⁴ The industry benchmark for cost per call across large operations sits between $2.70 and $5.60.⁴

That gap is why teams are rebuilding the stack themselves. This is the same build-vs-buy logic we have laid out for orchestrating your own office of clones: once the usage-based primitives exist and are cheap enough, renting a seat stops making sense for anything you run at real volume.

Step 1: Price the usage, not the seats

Before writing code, run the numbers you actually have. Voice infrastructure providers like Retell price per minute of conversation, not per seat. A current Retell configuration with GPT-4.1 as the LLM and standard telephony runs about $0.11 per minute all in. That is the voice engine ($0.055/min), the LLM ($0.045/min), and text-to-speech ($0.015/min) added together, with telephony often free through the provider's own carrier relationships.⁵

A four-minute call costs roughly $0.44 on that stack, against an industry-benchmarked $2.70 to $5.60 for the equivalent human-handled call, a 6x to 13x gap before you count management overhead or missed-call revenue. A million minutes a month on the AI side runs about $110,000. The same volume routed through human agents, at five-minute average handle time, lands between $540,000 and $1.12 million at the low and high ends of the per-call benchmark.

Add the realistic enterprise extras, a knowledge base at $0.005/min, safety guardrails at $0.005/min, PII redaction at $0.01/min, and AI quality assurance at $0.10/min after the first 100 free minutes, and a fully loaded rate still stays under $0.25 per minute.⁵ That is your baseline number. Write it down before you build anything else, because it is the number that justifies (or kills) the project.

Step 2: Wire the four core pieces

A voice agent has four layers, and each one is a separate cost and a separate decision:

• Telephony. The line that actually carries the call. Providers like Twilio or Telnyx handle this, and Retell passes it through at roughly $0.015/min for US numbers, with no markup for bring-your-own carrier setups.⁵
• Speech-to-text and text-to-speech. Converting the caller's voice to text the LLM can read, and the LLM's response back into voice. TTS voice options run $0.015 to $0.04/min depending on vendor, with premium engines like ElevenLabs at the top of that range.⁵
• The LLM. The reasoning layer. This is where cost varies most. A lightweight model like GPT-4.1 nano runs $0.004/min, while a frontier model like GPT-5.5 runs $0.16/min, a 40x spread for the same call.⁵ Most production deployments land on a mid-tier model, trading a little reasoning depth for a much smaller bill.
• Orchestration and knowledge. The logic that decides what the agent knows, when it transfers to a human, and what it logs. This is where you plug in your own knowledge base instead of paying a SaaS vendor to host it for you, which ties directly into a broader argument we have made about owning your AI training data rather than handing it to a third party's black box.

Step 3: Build the containment logic before the voice

The agent's job is not to sound human. Its job is to resolve or correctly route the call. Design the containment logic first: what can the agent answer outright, what needs a knowledge base lookup, and what must transfer to a human. Everise, running enterprise IT and HR service desks, contains 65% of internal tickets without a human ever picking up, using this kind of triage-first design. Sunshine Loans applied the same logic to loan applications and cut abandonment to 5% while scaling past 700,000 monthly applications.

Billing continues through hold time and silence because the speech-to-text engine stays active listening, so a poorly designed containment flow that loops or stalls costs you real money per minute, not just a bad customer experience.⁵ Test your flows against edge cases before they hit production traffic.

Step 4: Run the five-input cost model

Don't guess at ROI. There is a simple five-input model worth stealing: monthly call volume, average handle time, human cost per minute, AI cost per minute, and current missed-call rate. The two formulas are direct cost savings (volume times time times containment rate times cost differential) and recovered revenue from calls that used to go unanswered (volume times missed-call rate times average revenue per call). Industry data shows 20% to 30% of inbound business calls go unanswered or are abandoned in queue, and every one of those is revenue a rented seat never gets you back because the SaaS seat still can't answer a call nobody staffed for.

Run the model on your own volume before you commit engineering time. If your monthly minutes are in the low hundreds, the fixed cost of building and maintaining the system will outweigh the savings. If your calls require deep human judgment on every turn, such as clinical triage or high-stakes underwriting, containment rates drop to 20% to 30% and the math softens considerably.

Step 5: Own the infrastructure, not just the config

The real ownership question is not whether you use Retell, Twilio, or an open-source stack. It's whether your prompts, call logs, knowledge base, and routing logic live somewhere you control, exportable and auditable, rather than locked inside a vendor's proprietary UI. This is the same asset-ownership argument we have made about self-hosting to escape cloud bugs and usage fee overruns: a usage-based voice platform is still a rental unless you also own the data and logic layered on top of it.

For teams that want the ownership model applied specifically to voice, Remy is built around letting companies run and own their own voice agent stack rather than lease a black-box contact center suite. That is the same shift this whole build is chasing: usage-based cost, but the asset stays yours.

The bill you should expect

Expect setup time measured in days, not months, for a first working agent on a platform like Retell. Expect your per-minute cost to land between $0.11 and $0.25 depending on LLM tier and add-ons.⁵ Expect the savings to be real but back-loaded into containment rate and missed-call recovery rather than the sticker price alone. And expect the seat-based alternative to keep charging you for capacity you don't use, every single month, whether the phone rings or not.