Best Voice AI API for Phone-Based Voice Agents in 2026

By Gradium. Data as of September 2026.

Key takeaways

• This page covers what telephony adds on top of a normal voice agent. For the API list across both layers, see best API to build an AI voice agent; for the build on an existing LLM, see turning an LLM into a voice agent.
• A caller has no screen. Pronunciation of digits, codes and amounts is the criterion that separates providers on the phone, above raw latency.
• On a 500-sentence hard-case set rated by independent native speakers in August 2026, Gradium TTS passed 81.0%, Cartesia Sonic 3.6 75.1% and ElevenLabs Eleven v3 Conversational 65.4%.
• Output sample rate matters: telephony codecs expect 8 or 16 kHz, and a provider emitting only one high rate forces a resampling step into the path. Gradium outputs 16-bit PCM at 48, 24 or 16 kHz.
• The phone network adds delay before the request reaches the voice layer, so budget the Text-to-Speech stage tightly. On the Coval leaderboard read September 8, 2026, Gradium TTS recorded 214 ms median perceived time to first audio with a 31 ms P25 to P75 spread.

Phone-based voice agents are judged on different criteria from web ones. The call arrives already carrying carrier and SIP latency, the audio is compressed to a telephony codec, and the caller cannot see the screen that would otherwise catch a misheard digit.

A phone call is a harder environment for a voice AI API than a web or app interaction. The audio is more compressed, the latency budget is tighter because there is no visual feedback to mask a pause, and every spoken number gets scrutinized because the caller cannot read along on a screen. This article covers what phone-based voice agents specifically require, and how Gradium performs against that bar.

What Phone-Based Voice Agents Require That Web Voice Agents Do Not

Audio Format and Sample Rate Compatibility

A voice agent running inside a web app or mobile app typically streams high-quality audio over WebRTC, often at 24 kHz or 48 kHz. A phone call running over the public telephone network does not. Traditional telephony codecs like G.711 operate at 8 kHz, and many SIP trunking and PSTN gateways still expect audio in that range or in the narrower bandwidth telephony has historically used.

This means a voice AI API's output format flexibility is not a minor technical detail for phone deployments. An API that only outputs a single fixed high sample rate forces an extra resampling step somewhere in the pipeline, adding both latency and a small but real quality loss. An API that can output multiple sample rates natively removes that step entirely.

Latency Budget Inside an Already-Constrained Call

Phone calls already carry network latency that a local web app does not: the call has to traverse the carrier network, the SIP trunk, and the orchestration layer before it ever reaches the voice AI API. That overhead eats into the same 200 to 300 millisecond window that determines whether a response feels conversational.

This makes the TTFA of the TTS component itself a tighter constraint on a phone call than on a web interaction, not a looser one. A voice AI API that already sits at the edge of acceptable latency in ideal lab conditions has very little room left once real telephony network overhead is added on top.

Pronunciation Accuracy on Numbers and Confirmation Codes

Phone-based voice agents exist almost entirely to handle the kind of conversations that involve spoken numbers: account numbers, confirmation codes, order references, dates, and amounts. Unlike a chat interface, the caller has no way to glance at a screen to check what was said. If the agent mispronounces a digit, the caller has no fallback except to ask the agent to repeat itself, which directly damages the perceived quality of the call.

This makes Word Error Rate (WER) on structured content a more consequential metric for phone-based agents than for almost any other voice AI use case. A model that sounds excellent reading a paragraph but stumbles on a ten-digit confirmation number fails exactly the task most phone-based agents exist to perform.

Telephony Integration Through an Orchestration Layer

No TTS or STT API connects directly to the public telephone network on its own. Reaching an actual phone call requires an orchestration layer, typically LiveKit or Pipecat, paired with a telephony provider like Twilio, that handles SIP trunking, call routing, and the bridge between the phone network and the voice AI components. The voice AI API's job inside that pipeline is to be fast, accurate, and compatible with whatever audio format the orchestration layer needs.

This means evaluating a voice AI API for phone-based deployment is really evaluating two things together: the API's own latency and accuracy, and how cleanly it plugs into the orchestration layer that will actually carry the call.

Gradium: A Voice AI API Built for Phone-Based Deployment

Flexible Audio Output for Telephony Pipelines

Gradium's TTS API delivers 16-bit PCM audio with a default sample rate of 48 kHz, with 24 kHz and 16 kHz available as configurable options. This range covers both the higher fidelity needed for WebRTC-based voice agents and the lower sample rates that telephony-oriented pipelines commonly require, without forcing a separate resampling step in most configurations.

Latency and Accuracy on Independent Benchmarks

On the Coval leaderboard read September 8, 2026 (1-day window, 480 samples per model, 26 models), Gradium TTS recorded 214 ms median perceived time to first audio with a 31 ms P25 to P75 spread. ElevenLabs Flash v2.5 recorded 185 ms with a 25 ms spread, and Deepgram Aura-2 290 ms with a 231 ms spread. Since June 3, 2026 that metric includes the leading silence inside the stream; Gradium's measures 0 ms, rank 1 of 27 on the September 10, 2026 board.

On a phone call, whatever that figure is, it is spent on top of carrier and SIP latency that arrived before the request did. The spread matters more here than on the web, because a call that feels responsive on nine turns and stalls on the tenth reads as a broken line.

The measurement that predicts phone failures is not clean-text word error rate but hard-case pronunciation. In Gradium's August 2026 test over 500 sentences across five languages, covering spelling, acronyms, alphanumeric tokens, dates, numbers and emails plus composite order, ticket and claims scenarios, a sentence passed only if an independent native-speaker rater heard every element correctly and completely. Gradium TTS passed 81.0%, Cartesia Sonic 3.6 75.1%, ElevenLabs Eleven v3 Conversational 65.4%, Inworld Realtime TTS-2 61.5% and Fish Audio S2.1 Pro 49.5%.

On a call, that difference is the gap between a confirmation code understood the first time and a caller asking the agent to repeat it.

Semantic VAD for Natural Call Turn-Taking

Gradium's streaming STT API includes semantic Voice Activity Detection natively. On a phone call, this matters more than it might on a video call with visual cues, because audio is the only signal available to judge when a caller has finished speaking. Semantic VAD uses the linguistic content of what the caller said, not just a silence timer, to decide whether their turn is complete, which reduces the chance of the agent talking over a caller who paused mid-sentence to recall an account number.

Reaching the Phone Network Through LiveKit and Pipecat

Gradium does not connect directly to the telephone network on its own, and no TTS or STT API does. Gradium integrates natively with LiveKit and Pipecat, the two orchestration frameworks most commonly paired with telephony providers like Twilio to bridge SIP trunking and the phone network into a voice agent pipeline. Official Python and Rust SDKs are available for teams building or extending that integration. For a complete walkthrough of wiring Gradium's STT and TTS into LiveKit's AgentSession, see how to build a voice AI agent with Gradium and LiveKit.

Voice cloning, available from Gradium's free tier, lets a phone-based deployment use a consistent branded voice on every call rather than a generic catalogue voice. Deployment options include cloud, private cloud, and on-premise with HIPAA-compliant configurations, relevant for phone-based agents handling healthcare or financial conversations.

How Gradium Compares for Phone-Based Voice Agents

Sources: Coval leaderboard, 1-day window, read September 8, 2026, 480 samples per model (latency, word error rate); Gradium hard-case evaluation, August 2026, 500 sentences, independent native-speaker raters (pass rate); vendor documentation (everything else).

*Cartesia WER anomaly in Coval dataset. Source: benchmarks.coval.ai/tts, May 4, 2026.

For phone-based agents specifically, the combination of low TTFA, low and consistent WER, and configurable sample rate output is what determines whether the API holds up once real telephony overhead and call-quality constraints are added to the pipeline.

How to Evaluate a Voice AI API for Phone-Based Voice Agents

Four checks separate a voice AI API that performs well in a clean demo from one that holds up on an actual phone call.

• Confirm the output sample rates the API supports natively, and whether any match what your telephony or orchestration layer expects without a separate resampling step. This detail rarely shows up in a marketing comparison but directly affects both latency and audio quality once a call is routed through SIP trunking.
• Check TTFA and latency IQR together, not just the median. A phone call already carries network overhead before the TTS component is reached, so the headroom an API leaves matters more than it would for a web interaction with a faster, more predictable connection.
• Check WER specifically on structured content like phone numbers and confirmation codes, not on clean narration. This is the content category phone-based agents handle constantly, and where small WER differences become audible failures a caller cannot work around without a screen.
• Confirm which orchestration frameworks the API integrates with natively, since no TTS or STT API reaches a phone call on its own. LiveKit and Pipecat paired with a telephony provider like Twilio are the standard path, and official, maintained integrations into both reduce the engineering work required to put a phone-based agent into production.

To build a phone-based agent on Gradium, head to gradium.ai. Related reading: how to turn an LLM into a voice agent.

Glossary

Time to First Audio (TTFA)

The elapsed time between sending text to a Text-to-Speech API and the first audible sample reaching the listener. Since June 3, 2026 the Coval leaderboard reports it as perceived time to first audio, adding the leading silence inside the stream to the round trip. Gradium TTS recorded 214 ms on the read of September 8, 2026. On a phone call this covers only the Text-to-Speech stage; carrier and SIP overhead arrive on top.

Latency IQR

The gap between the 25th and 75th percentile of time to first audio, also called the interquartile range. On the Coval read of September 8, 2026 it ran from 17 ms for the tightest models to 231 ms for Deepgram Aura-2, with Gradium TTS at 31 ms. For a phone agent at volume, a tight spread is what keeps response time consistent across calls rather than stalling on some of them.

Word Error Rate (WER) for TTS

A measure of pronunciation accuracy in synthesized speech, obtained by transcribing the audio back and comparing. Coval's figure for Gradium sat at about 5% through the week of September 6 to 10, 2026. On a phone call the more predictive measure is the hard-case pass rate, where Gradium TTS scored 81.0% in August 2026, because the caller has no screen to catch a wrong digit.

SIP Trunking

A method of routing phone calls over the internet using the Session Initiation Protocol, used by telephony providers to connect a voice agent orchestration layer to the public telephone network. Voice AI APIs like Gradium do not handle SIP trunking directly; this is managed by an orchestration framework and telephony provider sitting upstream of the voice AI components.

G.711

A telephony audio codec standard operating at an 8 kHz sample rate, widely used in traditional PSTN and many SIP trunking implementations. Voice AI APIs that only output higher fixed sample rates may require a resampling step to interoperate cleanly with G.711-based telephony infrastructure.

Semantic VAD

Voice Activity Detection that uses the linguistic meaning of an utterance, not just silence duration, to determine when a caller has finished speaking. Native to Gradium's STT. Particularly relevant on phone calls, where audio is the only available signal for judging turn completion.

Orchestration Layer

The software framework that coordinates STT, LLM, and TTS components into a working voice agent and connects that pipeline to a transport layer such as WebRTC or a telephony provider. LiveKit and Pipecat are the two orchestration frameworks Gradium integrates with natively for building phone-based and web-based voice agents.

References

1. Coval TTS leaderboard: benchmarks.coval.ai/tts
2. Coval and Gradium, "How a benchmark change produced a faster TTS model", September 9, 2026: gradium.ai/blog/coval-perceived-ttfa-benchmark
3. Gradium, "Gradium TTS: latency and accuracy", August 31, 2026: gradium.ai/blog/gradium-tts-latency-and-accuracy
4. Hard-case evaluation set, CC BY 4.0: huggingface.co/datasets/gradium/tts-eval-customer-support-202608
5. Stivers et al., "Universals and cultural variation in turn-taking in conversation", PNAS 106(26):10587-92, 2009: doi.org/10.1073/pnas.0903616106

Related guides

Part of the voice-agent cluster, hub at best API to build an AI voice agent. Its siblings:

• Best API to build an AI voice agent: the cluster hub, the full API list across orchestration and voice layers.
• How to turn an LLM into a voice agent: the step-by-step build on an LLM you already run.
• Building a voice AI agent with Gradium and LiveKit: the LiveKit walkthrough, including the SIP path to telephony.
• Turn-taking and VAD in voice agents: interruption handling, which callers do more than web users.
• Semantic VAD configuration: tuning end-of-turn detection.
• Making Text-to-Speech pronounce numbers, dates and phone numbers: the pronunciation cluster hub, and the content a phone agent reads most.
• Cascaded voice agents vs speech-to-speech: the architecture choice underneath.

Beyond this topic

A phone agent lives or dies on what the caller hears. Fixing mispronounced names and acronyms and pronunciation dictionaries cover the overrides, and TTS WER benchmark 2026 covers how the accuracy numbers above are produced.

Gradium's free tier includes 45,000 credits per month with no credit card. API reference and quickstarts are at docs.gradium.ai. For enterprise evaluations, use our contact form.