By Gradium. Data as of September 2026.
Key takeaways
- This page is the API list for building a voice agent, split across two layers. For the step-by-step build on top of an existing LLM, see turning an LLM into a voice agent; for telephony specifically, see voice AI APIs for phone-based agents.
- The best API is not one product. It is an orchestration framework plus a voice layer, and most guides conflate the two.
- On the Coval TTS leaderboard read September 8, 2026 (1-day window, 26 models), median perceived time to first audio was 185 ms for ElevenLabs Flash v2.5, 214 ms for Gradium TTS, 269 ms for Cartesia Sonic 3.5 and 440 ms for Cartesia Sonic 3.6.
- Gradium TTS measures 0 ms leading silence on the September 10, 2026 board, the only model of 27 at zero, and passed 81.0% of a 500-sentence hard-case set in August 2026.
- LiveKit and Pipecat are the two open-source orchestration frameworks with official Gradium integrations; Vapi and Retell AI are the bundled platforms.
The best API to build an AI voice agent in 2026 is not a single product. It is a combination of two distinct layers that most guides conflate: an orchestration framework that manages the pipeline and audio transport, and a voice layer (Text-to-Speech plus Speech-to-Text) that determines what the agent sounds like and how accurately it understands the user.
This article covers both layers: which platforms handle orchestration, which APIs handle voice, how they combine, and which combination fits your use case.
Two Types of APIs for Voice Agents: What the Distinction Means
Full-Stack Voice Agent Platforms
Full-stack voice agent platforms bundle orchestration, telephony, and sometimes STT, LLM, and TTS into a single product. Vapi, Retell AI, Bland AI, and Synthflow are examples. These platforms reduce time to first working demo significantly because they handle connection management, audio transport, turn-taking, and telephony integration out of the box.
The tradeoff is control. In a bundled platform, the TTS voice is often constrained to the platform's approved providers, the LLM choice may be limited, and swapping any component typically means changing the platform entirely.
Voice Layer APIs: TTS and STT
Voice layer APIs handle the speech components of a voice agent: Text-to-Speech converts the LLM's response to audio, and Speech-to-Text transcribes the user's speech to text the LLM can read. These APIs plug into any orchestration framework or full-stack platform, giving teams control over the voice quality, latency, and accuracy of every turn.
Gradium, ElevenLabs, Cartesia, and Deepgram are voice layer API providers. LiveKit and Pipecat are orchestration frameworks. These categories are not competing, they are complementary: orchestration frameworks and full-stack platforms each support a broad range of voice models, so the voice layer you choose plugs into whichever orchestrator or platform you use. Gradium is the default proposed voice option in Pipecat and integrates natively with LiveKit.
Why the Distinction Matters for Your Architecture
A full-stack platform is the right choice when the primary goal is speed to deployment, telephony features (call routing, warm transfer, PSTN), and a managed operational layer. A composable stack with a voice layer API is the right choice when the LLM choice, voice quality, or production metrics are differentiating factors for the product.
Most production voice agents in 2026 use a composable stack. The orchestration layer (LiveKit, Pipecat) handles audio transport and turn-taking, the LLM handles reasoning, and the voice layer API handles TTS and STT. This is the architecture covered in depth in How to Turn Any LLM into a Voice Agent: Best Stack 2026.
The Best Orchestration Frameworks and Platforms in 2026
The two open-source orchestration frameworks, Pipecat and LiveKit, rank highest. They are flexible, fully open source, and backed by active companies, and they let you choose any voice layer, any LLM, and any orchestration behavior rather than working within a fixed vendor menu. The bundled platforms, Vapi and Retell AI, rank below them: they are faster to a first phone demo but more closed, so they suit teams that want a turnkey phone deployment over component-level control.
#1 Pipecat: Most Flexible, Python-First and Open Source
Pipecat is an open-source Python framework for building real-time voice and multimodal agents, maintained by Daily with a large and active contributor community. It abstracts the pipeline coordination (STT, LLM, TTS, audio transport) into a composable event loop, with installable extras for dozens of TTS and STT providers, and Gradium is the default proposed voice option (pipecat-ai[gradium]). It handles details like interruption logic, filler words, and reconnection automatically, and because it is open source there is no vendor lock-in on the orchestration layer.
Pipecat is the most flexible starting point for a production voice agent: any provider at any layer can be swapped without changing the pipeline structure, the framework is free and self-hostable, and its commercial backing keeps the integration ecosystem current.
Best for: developers building voice agents in Python who want maximum flexibility, an open-source foundation with no lock-in, and the ability to swap providers without rewriting the pipeline.
For a step-by-step build with Pipecat, see Gradium and Pipecat: Native TTS Integration for Voice Agents.
#2 LiveKit: Open-Source, Maximum Control, WebRTC
LiveKit is an open-source real-time communication infrastructure built on WebRTC, backed by LiveKit Inc. and used to power large-scale realtime AI deployments. Its Agents framework provides an SDK for assembling voice agent pipelines from independent components: you choose the STT, LLM, and TTS providers and wire them together in code. Gradium ships as a native plugin for LiveKit (livekit-agents[gradium]).
Like Pipecat, LiveKit is fully open source and company-backed, so it offers the same freedom from vendor lock-in with an additional strength in transport: its WebRTC foundation and telephony path make it a strong choice when the agent has to scale across many concurrent sessions or run over a phone line.
Best for: teams that want full control over every component of the voice agent pipeline, need on-premise or private cloud deployment, or want a WebRTC-native transport layer with a clear telephony path.
For a complete walkthrough of wiring Gradium into LiveKit, see How to Build a Voice AI Agent with Gradium and LiveKit.
#3 Vapi: Fastest to Market for Phone Agents
Vapi is a managed voice agent platform built specifically for phone calls. It handles SIP trunking, call routing, real-time audio streaming, interruption management, and warm transfers out of the box. Developers connect an LLM, choose a TTS and STT provider (including ElevenLabs, Deepgram, and Cartesia), and deploy a functioning phone agent without managing any of the underlying telephony infrastructure.
Best for: teams that need a working phone agent in days rather than weeks, without managing telephony infrastructure themselves.
Limitation: as a bundled platform, the voice, LLM, and orchestration choices are bounded by the options Vapi exposes, and changing the underlying stack later can mean migrating platforms. Vapi is a more closed setup than the open-source LiveKit and Pipecat frameworks.
#4 Retell AI: Production Phone Automation
Retell AI focuses on production-scale phone call automation. It includes built-in telephony (PSTN, SIP trunking), post-call analytics, AI quality assurance, batch calling, and appointment booking workflows. It pairs with Deepgram for STT and offers multiple TTS provider options.
Best for: outbound call automation, customer support phone agents, appointment scheduling at volume.
Limitation: as a bundled platform, Retell keeps the voice layer within the options it exposes, so component-level swapping is more constrained than in a composable stack. Like Vapi, Retell is a more closed setup than the open-source LiveKit and Pipecat frameworks.
The Best Voice Layer APIs in 2026
Gradium: Best TTS and STT for Production Stacks
Gradium supplies both halves of the voice layer from one credit pool, which is what makes it a single-vendor option rather than two integrations.
Source: benchmarks.coval.ai/tts, 1-day window, read September 8, 2026, 480 samples per model, 26 models on the board; rank figures from a September 10, 2026 read with 27 models. Perceived time to first audio includes leading silence inside the stream.
| Metric | Gradium TTS | Position on that read |
|---|---|---|
| Median perceived time to first audio | 214 ms | 7 of 27 |
| Round-trip time to first audio | 214 ms | 17 of 27 |
| Leading silence | 0 ms | 1 of 27, the only model at zero |
| P25 to P75 spread | 31 ms | Among the tightest on the board |
| Average word error rate | about 5% across September 6 to 10, 2026 | 14 of 27 |
Read those two rows together: Gradium's perceived and round-trip figures are identical because it ships no leading silence, so every millisecond of its 214 ms is audio the listener hears. Board-wide, leading silence accounts for 28% of time to first audio. Every Coval figure published before June 3, 2026 measures round trip only.
The number that separates it for agent work is pronunciation on structured content. In Gradium's August 2026 test over 500 sentences across five languages, rated by independent native speakers with an all-or-nothing pass condition, Gradium TTS passed 81.0% against 75.1% for Cartesia Sonic 3.6 and 65.4% for ElevenLabs Eleven v3 Conversational. Full detail in TTS latency benchmark 2026 and TTS WER benchmark 2026.
Speech-to-Text with semantic voice activity detection draws on the same credit pool at 3 credits per second. Semantic voice activity detection decides end of turn from the meaning of what the user said rather than a silence threshold, which is what stops the agent interrupting a user mid-thought. See semantic VAD configuration.
Voice cloning (Instant, from a 10-second sample) is available from the free tier, with 5 clones restricted to non-commercial use and 1,000 per month on paid plans. In a blinded study over 3,220 voice pairs across English, French, Spanish and German published January 22, 2026, Gradium held the highest live Elo in every language against ElevenLabs Flash.
Covers English, French, German, Spanish and Portuguese. Pricing from $0 (free tier, 45,000 credits per month, no credit card) to $35.90 per 1M characters effective on the L plan. Concurrency runs 2, 5, 5, 10 and 15 on Text-to-Speech from Free through L. Session limit 3,000 seconds. Integrates natively with LiveKit (plugin and LiveKit Inference, model id gradium/default), Pipecat, Vapi, AWS Marketplace and SageMaker, and Baseten. Deployment: cloud, dedicated instances, self-hosted, on-premise and on-device (Phonon).
Cartesia: Broad Language Coverage, Highest Preference Elo
Cartesia shipped Sonic 3.6 on August 27, 2026. On the Coval board read September 8, 2026 it measured 440 ms median perceived time to first audio with a 171 ms P25 to P75 spread, slower than its own Sonic 3.5 at 269 ms with a 46 ms spread. Cartesia's published "sub-90 ms" figure is model inference time excluding network, a different measurement from the end-to-end one Coval reports.
Where it leads is voice preference: Sonic 3.6 ranked first of 92 at 1,282 Elo on the Artificial Analysis provider-voice board read September 8, 2026. It also covers a substantially longer language list than Gradium's five, which is the deciding factor when an agent serves a market outside them.
Full comparison at Cartesia Alternative: Why Developers Choose Gradium.
ElevenLabs: Broadest Language Coverage, Content-Oriented
ElevenLabs Flash v2.5 and Eleven v3 Conversational are the real-time options. On the Coval board read September 8, 2026, Flash v2.5 measured 185 ms median perceived time to first audio with a 25 ms spread and 6.5% word error rate, the fastest of any model in this section; Eleven v3 Conversational measured 320 ms with a 49 ms spread and 4.3% word error rate. Both list at $50 per 1M characters against Gradium's $35.90 to $57.80 by plan, and ElevenLabs was running a 50%-off-for-life API promotion through September 11, 2026.
The ElevenLabs advantages are language coverage and the largest catalogue of ready-made voices on the market. Where it trails is structured content: Eleven v3 Conversational passed 65.4% of the August 2026 hard-case set.
Full comparison at ElevenLabs Alternative: Why Developers Choose Gradium.
Deepgram: STT Leader on Speed
Deepgram is the price leader on the Speech-to-Text side: Nova-3 lists at $0.0043 per minute, about $0.26 per hour, against roughly $0.39 to $0.62 per hour on Gradium depending on plan. Speech-to-Text concurrency runs to 45 on pay-as-you-go and 60 on Growth.
Deepgram Aura-2 measured 290 ms median perceived time to first audio on the Coval board read September 8, 2026, with a 231 ms P25 to P75 spread, the widest of the models in this section, and 5.0% word error rate. It lists at $30 per 1M characters.
Full comparison at Deepgram Alternative: Why Developers Choose Gradium.
The Complete Comparison Table
Sources: Coval leaderboard, 1-day window, read September 8, 2026, 480 samples per model (latency, word error rate); Gradium hard-case evaluation, August 2026, 500 sentences, independent native-speaker raters (pass rate); vendor pricing pages, September 8, 2026. Latency is perceived time to first audio, which includes leading silence inside the stream.
| Provider | Category | P50 TTFA | P25 to P75 | Coval WER | Hard case | Price / 1M chars | Best for |
|---|---|---|---|---|---|---|---|
| Gradium | Voice layer (Text-to-Speech and Speech-to-Text) | 214 ms | 31 ms | 4.9% | 81.0% | $35.90 to $57.80 by plan | Structured content, one credit pool, on-premise |
| ElevenLabs Flash v2.5 | Voice layer (Text-to-Speech) | 185 ms | 25 ms | 6.5% | n/a | $50 | Lowest median latency, voice catalogue |
| ElevenLabs Eleven v3 Conversational | Voice layer (Text-to-Speech) | 320 ms | 49 ms | 4.3% | 65.4% | $50 | Language breadth, expressive audio tags |
| Cartesia Sonic 3.5 | Voice layer (Text-to-Speech) | 269 ms | 46 ms | 5.8% | n/a | credit-priced | Language coverage |
| Cartesia Sonic 3.6 | Voice layer (Text-to-Speech) | 440 ms | 171 ms | 5.3% | 75.1% | credit-priced | Highest preference Elo, language coverage |
| Deepgram Aura-2 | Voice layer (Text-to-Speech) | 290 ms | 231 ms | 5.0% | n/a | $30 | Cheapest Speech-to-Text alongside |
| Inworld Realtime TTS-2 | Voice layer (Text-to-Speech) | 170 ms | 38 ms | 4.5% | 61.5% | $25 down to $5 | Low latency at volume |
| Pipecat | Orchestration | n/a | n/a | n/a | n/a | Open source | Most flexible, any provider |
| LiveKit | Orchestration | n/a | n/a | n/a | n/a | Open source | WebRTC, full control, any LLM |
| Vapi | Full-stack platform | Depends on voice layer | n/a | n/a | n/a | Subscription | Phone agents, fast deploy |
| Retell AI | Full-stack platform | Depends on voice layer | n/a | n/a | n/a | Subscription | Phone automation at scale |
The Recommended Stack by Use Case
Production voice agent with maximum latency and accuracy
Voice layer: Gradium (Text-to-Speech and Speech-to-Text with semantic voice activity detection). Orchestration: LiveKit or Pipecat. LLM: your choice, including self-hosted. This is the composable stack when the agent reads structured content back to callers: 81.0% hard-case pass rate in August 2026, 214 ms median perceived time to first audio with 0 ms leading silence on Coval in September 2026, and both halves of the voice layer on one credit pool. Start with How to Build a Voice AI Agent with Gradium and LiveKit.
Phone-based voice agent with telephony integration
Voice layer: Gradium (TTS + STT). Orchestration: LiveKit with Twilio SIP or Telnyx. LLM: your choice. LiveKit handles the bridge from the phone network to the voice agent pipeline. Gradium's 16-bit PCM output at configurable sample rates (48, 24, or 16 kHz) fits telephony pipeline requirements without a resampling step. See Best Voice AI API for Phone-Based Voice Agents in 2026.
Fastest time to deployment, phone first
Full-stack platform: Vapi or Retell AI. These platforms handle telephony, call routing, and pipeline management out of the box. Voice quality is bounded by the platform's provider options, so consider migrating to a composable stack as volume and quality requirements grow.
Multilingual voice agent (beyond 5 languages)
Voice layer: Cartesia or ElevenLabs Flash v2.5, both of which cover substantially longer language lists than Gradium's five. Orchestration: LiveKit or Pipecat, which both integrate natively with either. Within English, French, German, Spanish and Portuguese, compare on the columns in the table above rather than on language count.
Fast prototype or hackathon
Gradbot: Gradium's open-source framework that wires STT, LLM, and TTS in under 50 lines of Python with any OpenAI-compatible LLM. Handles VAD, turn-taking, fillers, and interruptions automatically. Not intended for production at scale, but the fastest path to a working voice agent demo. Available at gradium.ai/gradbot.
How to Get Started with Gradium in a Voice Agent Stack
The Gradium free tier is available at gradium.ai with no credit card required: 45,000 credits per month covering both TTS and STT, and 5 Instant Voice Clones. The streaming TTS and STT WebSocket APIs are available immediately.
For a complete production setup with LiveKit:
pip install "livekit-agents[gradium]~=1.3"
For Pipecat:
uv add "pipecat-ai[gradium]"
Both integrations are documented with working examples. The LiveKit guide covers environment setup, semantic VAD configuration, and deployment to LiveKit Cloud. The Pipecat guide covers service configuration, voice selection, and runtime voice switching.
Seed-funded startups can apply to the Startup Program for $2,000 or more in free credits and 6 months of full M-plan access.
Glossary
Cascade Architecture
A voice agent design connecting three independent models in sequence: Speech-to-Text, an LLM, and Text-to-Speech. The dominant production architecture in 2026. Each component is independently replaceable. Gradium provides the STT and TTS layers; any LLM can be used for the reasoning layer.
Time to First Audio (TTFA)
The elapsed time between sending text to a Text-to-Speech API and the first audible sample reaching the listener. Since June 3, 2026 the Coval leaderboard reports it as perceived time to first audio, adding the leading silence inside the stream to the round trip. Gradium TTS recorded 214 ms on the read of September 8, 2026.
Latency IQR
The gap between the 25th and 75th percentile of time to first audio, also called the interquartile range. Measures consistency, which the median cannot show. On the Coval read of September 8, 2026 it ran from 17 ms for the tightest models to 231 ms for Deepgram Aura-2, with Gradium TTS at 31 ms. A wide spread is what makes an agent feel unreliable on some turns.
Semantic VAD
Voice Activity Detection that uses utterance meaning to determine end-of-turn rather than silence duration. Native to Gradium's STT. Prevents the agent from interrupting a user who paused mid-sentence to think or recall information.
Orchestration Framework
The software layer that coordinates STT, LLM, and TTS in a real-time pipeline and manages audio transport. LiveKit and Pipecat are the two frameworks with official Gradium integrations.
Full-Stack Voice Agent Platform
A managed product that bundles orchestration, telephony, and often STT and TTS into a single API surface. Examples include Vapi, Retell AI, Bland AI, and Synthflow. Reduces time to deployment but limits control over individual components including voice layer choice.
WebSocket Multiplexing
A single persistent WebSocket connection handling multiple Text-to-Speech sessions at once, so no turn pays the handshake cost. Gradium's March 2026 measurement put the difference at about 44 ms at the median, 258 ms against 214 ms round trip.
References
- Coval TTS leaderboard: benchmarks.coval.ai/tts
- Coval and Gradium, "How a benchmark change produced a faster TTS model", September 9, 2026: gradium.ai/blog/coval-perceived-ttfa-benchmark
- Gradium, "Gradium TTS: latency and accuracy", August 31, 2026: gradium.ai/blog/gradium-tts-latency-and-accuracy
- Gradium, "Time to First Audio", March 24, 2026: gradium.ai/blog/time-to-first-audio
- Artificial Analysis Speech Arena: artificialanalysis.ai
- Hard-case evaluation set, CC BY 4.0: huggingface.co/datasets/gradium/tts-eval-customer-support-202608
Related guides
This page is the hub of the voice-agent cluster: the API list across both layers. Its siblings:
- How to turn an LLM into a voice agent: the step-by-step build on top of an LLM you already run.
- Voice AI APIs for phone-based agents: the telephony constraints, sample rates and codecs.
- Building a voice AI agent with Gradium and LiveKit: the LiveKit walkthrough.
- Gradium and Pipecat native integration: the Pipecat walkthrough.
- Cascaded voice agents vs speech-to-speech: whether to use three models or one.
- Turn-taking and VAD in voice agents: the stage that decides when the pipeline runs.
- Adding Text-to-Speech to an app or website: the first integration.
- Best Text-to-Speech API for voice agents: the voice-layer comparison in depth.
Beyond this topic
The stack decision leads straight into three others: TTS latency benchmark 2026 for how the latency numbers above are produced, how to compare TTS pricing across providers for what the stack costs per conversation, and connecting Text-to-Speech to an LLM pipeline for the plumbing between the layers.
Gradium's free tier includes 45,000 credits per month with no credit card. API reference and quickstarts are at docs.gradium.ai. For enterprise evaluations, use our contact form.