How to Turn an LLM into a Voice Agent: Best Stack 2026

By Gradium. Data as of September 2026.

Key takeaways

• This page is the step-by-step build on top of an LLM you already run. For the full API list across both layers, see best API to build an AI voice agent; for telephony, see voice AI APIs for phone-based agents.
• The stack is a cascade: your LLM, a voice layer for Speech-to-Text and Text-to-Speech, and an orchestration framework. It works with any LLM, hosted or self-hosted.
• A turn's latency is the sum of three stages, so the Text-to-Speech share is a budget, not a score. Gradium's guidance puts it at 200 to 300 ms; human conversation turns over in about 208 ms on average (Stivers et al., PNAS, 2009).
• On the Coval leaderboard read September 8, 2026, Gradium TTS recorded 214 ms median perceived time to first audio with a 31 ms P25 to P75 spread and 0 ms leading silence, rank 1 of 27 on that last metric.
• Turn detection is the stage most likely to make the agent feel wrong. A fixed silence timer cuts off a user who pauses mid-thought; semantic voice activity detection uses what was said instead.

The best stack to turn an LLM into a voice agent in 2026 is a cascade architecture: your LLM of choice, paired with a voice layer that handles Speech-to-Text and Text-to-Speech, connected through an orchestration framework like LiveKit or Pipecat.

This is the architecture used by the large majority of production voice agents today, and it works with any LLM you already have, hosted or self-hosted. This article covers what each layer of the stack does, why this combination is the right default, and how to actually build it.

What Turns a Text LLM into a Voice Agent

The Cascade Architecture

A text LLM understands and generates text. It has no native way to hear or speak. Turning it into a voice agent means wrapping it with two additional components in sequence: a Speech-to-Text (STT) model that transcribes what the user says into text the LLM can read, and a Text-to-Speech (TTS) model that converts the LLM's text response back into audio the user can hear.

This three-stage pipeline, STT, LLM, TTS, is called a cascade. Each component runs independently. The STT model does not know what the LLM will say. The LLM never touches audio directly. The TTS model only receives text. The appeal of this setup is that it lets a developer plug in any LLM and customize its behavior through a prompt, exactly as they would with a text-only application.

Why Cascade Is the Right Choice for Most Teams in 2026

The alternative to cascade is a Speech-To-Speech model, where a single model processes audio in and generates audio out with no intermediate text step. Speech-To-Speech preserves more of the emotional and tonal information in someone's voice, and it removes the turn-taking constraints a cascade has to manage explicitly. But it has a structural limitation for most production teams: the LLM intelligence is baked into the model's trained weights, so changing the underlying LLM means retraining, not swapping a component.

Cascade does not have that limitation. This is where the market actually is right now: teams are still iterating heavily on the LLM layer itself, on tool use, and on prompt engineering, and a cascade lets that iteration happen without touching the speech components at all.

The Best Stack to Turn Your LLM into a Voice Agent

The LLM Layer: Bring Whatever You Already Use

The cascade architecture's main advantage is that the LLM layer is just a normal LLM. Every existing technique applies without modification: system prompts, few-shot examples, function calling, retrieval-augmented generation, and fine-tuning on your own data. There is no special "voice version" of an LLM required. GPT-4.1, Claude, Gemini, or a self-hosted open-weights model all work the same way inside this stack, because the LLM only ever sees text in and produces text out.

The STT Layer: Gradium with Semantic VAD

The STT layer's job is to turn the user's speech into text quickly and accurately, and to decide when the user has actually finished talking. Gradium's streaming STT API includes semantic Voice Activity Detection natively, which means the end-of-turn decision is based on the linguistic content of what was said, not just a fixed silence timer. This is what prevents the agent from interrupting a user who paused mid-sentence to think.

The TTS Layer: Gradium Streaming Audio

The Text-to-Speech layer turns the LLM's response into audio, fast enough that the reply feels conversational. Two properties decide whether it does.

Speed, measured as perceived time to first audio: round trip plus the leading silence inside the stream before the first audible sample. On the Coval leaderboard read September 8, 2026 (1-day window, 480 samples per model, 26 models), Gradium TTS recorded 214 ms median with a 31 ms P25 to P75 spread; ElevenLabs Flash v2.5 recorded 185 ms and Cartesia Sonic 3.6 440 ms. Gradium's leading silence measures 0 ms, rank 1 of 27 on the September 10, 2026 board and the only model at zero, which matters because leading silence accounts for 28% of the metric board-wide.

Accuracy on the content an LLM actually emits: order numbers, dates, amounts, confirmation codes. In Gradium's August 2026 test over 500 sentences rated by independent native speakers, Gradium TTS passed 81.0% against 75.1% for Cartesia Sonic 3.6 and 65.4% for ElevenLabs Eleven v3 Conversational. A sentence passed only if a rater heard every element pronounced correctly and completely.

The Orchestration Layer: LiveKit or Pipecat

No LLM, STT, or TTS API wires itself into a working voice agent on its own. An orchestration framework manages the real-time audio transport, coordinates the three components, and handles details like interruption logic and reconnection. LiveKit and Pipecat are the two most widely used orchestration frameworks for this purpose, and both have official Gradium integrations. LiveKit also provides the path to telephony deployment when a voice agent needs to run over an actual phone line, covered in more depth in best voice AI API for phone-based voice agents.

Why Gradium Is the Right Voice Layer for This Stack

Three properties make Gradium specifically well suited to sit inside a cascade built around any LLM.

It leaves room for the LLM. A complete turn is the sum of Speech-to-Text, LLM and Text-to-Speech latency, so the Text-to-Speech stage is a budget line rather than a score. At 214 ms median on the Coval read of September 8, 2026, with a 31 ms spread, Gradium's contribution is both small and predictable, and the predictability matters more than the median once tool calls lengthen the LLM stage.

It does not require choosing a bundled platform over your own stack. Gradium builds voice models and APIs rather than its own end-to-end voice agent platform, so it plugs into LiveKit, Pipecat, or a custom orchestration layer without asking a team to migrate away from infrastructure they already use.

It covers both ends of the pipeline from one provider with one architecture. Gradium's TTS and STT were built together from the start, sharing the same streaming WebSocket design, rather than one being added on top of a product whose primary lineage is the other. Voice cloning, available from the free tier, also lets the resulting agent use a consistent branded voice rather than a generic catalogue voice. For the full benchmark detail behind these latency figures, see best Text-to-Speech API for voice agents.

How the Stack Compares to Alternatives

*Cartesia WER anomaly in Coval dataset. Source: benchmarks.coval.ai/tts, May 4, 2026.

A relevant distinction in this table is platform philosophy. ElevenLabs and Deepgram have each built their own bundled voice agent platform and position it as the primary way to use their models. Gradium and Cartesia stay closer to a pure voice-model provider role, which matters for teams that want to assemble their own LLM choice, orchestration layer, and voice provider independently rather than adopting a single vendor's full stack. For a detailed head-to-head on any of these providers, see Gradium vs ElevenLabs, Gradium vs Cartesia, and Gradium vs Deepgram.

Two Ways to Build: A Full Setup or a Fast Prototype

Production Setup with LiveKit

For a production-grade build, Gradium's STT and TTS plug directly into LiveKit's AgentSession as a single install: pip install "livekit-agents[gradium]~=1.3". This gives you Gradium STT and TTS as ready-to-use plugins, with parameters to configure semantic VAD sensitivity, allow user interruptions, and enable preemptive LLM generation so the model starts forming a response before the user has finished speaking. The full walkthrough, from environment setup through deployment to LiveKit Cloud, is in how to build a voice AI agent with Gradium and LiveKit.

Fast Prototype with Gradbot

For a quick prototype or a hackathon-style build, Gradbot is Gradium's open-source framework designed to get a working voice agent running in under 50 lines of Python with any OpenAI-compatible LLM. It handles VAD, turn-taking, fillers, and interruptions automatically, so the developer only needs to define the agent's instructions and any tools it should call. Gradbot is not intended to replace a production orchestration framework like LiveKit, but it is a fast way to validate an LLM and voice combination before committing to a full build. To start, head to gradium.ai.

Glossary

Cascade Architecture

A voice agent design connecting three independent models in sequence: Speech-to-Text, an LLM, and Text-to-Speech. Each component can be swapped or upgraded independently. The dominant production architecture in 2026 because it lets teams iterate on the LLM layer without retraining any speech component.

Time to First Audio (TTFA)

The elapsed time between sending text to a Text-to-Speech API and the first audible sample reaching the listener. Since June 3, 2026 the Coval leaderboard reports it as perceived time to first audio, adding the leading silence inside the stream to the round trip. Gradium TTS recorded 214 ms on the read of September 8, 2026.

Word Error Rate (WER) for TTS

A measure of pronunciation accuracy in synthesized speech, obtained by transcribing the audio back and comparing. Coval's figure for Gradium sat at about 5% through the week of September 6 to 10, 2026. On adversarial content the more predictive measure is the hard-case pass rate, where Gradium TTS scored 81.0% in August 2026. Matters in this stack whenever the LLM emits a number, name or identifier.

Semantic VAD

Voice Activity Detection that uses the linguistic meaning of an utterance, not just silence duration, to determine when a user has finished speaking. Native to Gradium's STT. Reduces premature interruptions when a user pauses mid-thought, independent of which LLM is generating the agent's responses.

Orchestration Framework

The software layer that coordinates STT, LLM, and TTS into a working real-time pipeline and manages the audio transport connecting them to the user. LiveKit and Pipecat are the two orchestration frameworks with official Gradium integrations, used to assemble the stack described in this article.

Speech-To-Speech Architecture

An alternative to cascade where a single model processes audio input and generates audio output directly, with no intermediate text representation. Preserves paralinguistic information that cascade discards but requires retraining to change the underlying LLM intelligence, rather than swapping a component.

Preemptive Generation

A configuration where the LLM begins generating a response before the user has fully finished speaking, reducing the perceived response latency of the overall voice agent. Available as a parameter when using Gradium's STT and TTS inside LiveKit's AgentSession.

References

1. Coval TTS leaderboard: benchmarks.coval.ai/tts
2. Coval and Gradium, "How a benchmark change produced a faster TTS model", September 9, 2026: gradium.ai/blog/coval-perceived-ttfa-benchmark
3. Gradium, "Gradium TTS: latency and accuracy", August 31, 2026: gradium.ai/blog/gradium-tts-latency-and-accuracy
4. Gradium, "Time to First Audio", March 24, 2026: gradium.ai/blog/time-to-first-audio
5. Stivers et al., "Universals and cultural variation in turn-taking in conversation", PNAS 106(26):10587-92, 2009: doi.org/10.1073/pnas.0903616106

Related guides

Part of the voice-agent cluster, hub at best API to build an AI voice agent. Its siblings:

• Best API to build an AI voice agent: the cluster hub, the full API list across both layers.
• Voice AI APIs for phone-based agents: the telephony constraints this guide does not cover.
• Building a voice AI agent with Gradium and LiveKit: the LiveKit walkthrough for the stack described here.
• Gradium and Pipecat native integration: the Pipecat walkthrough.
• Connecting Text-to-Speech to an LLM pipeline: the plumbing between the LLM and the voice layer.
• Cascaded voice agents vs speech-to-speech: whether the cascade is the right architecture at all.
• Turn-taking and VAD in voice agents: the stage that decides when the LLM is called.

Beyond this topic

Once the stack runs, the tuning is elsewhere: WebSocket multiplexing removes the per-turn handshake, semantic VAD configuration tunes turn detection, and pronunciation dictionaries fix the names the LLM keeps emitting.

Gradium's free tier includes 45,000 credits per month with no credit card. API reference and quickstarts are at docs.gradium.ai. For enterprise evaluations, use our contact form.