How to Connect TTS to an LLM in a Voice Pipeline (STT → LLM → TTS)

A voice agent is three models connected in sequence. Speech-to-Text (STT) transcribes what the user said. A large language model (LLM) generates a response. Text-to-Speech (TTS) converts that response into audio. Each model adds latency. Each is independently replaceable. The architecture that connects them is called a cascade, and it is the dominant production architecture for voice agents in 2026.

This guide covers how to wire those three components together, how streaming at each stage reduces total pipeline latency, where end-of-turn detection fits in, and what the latency budget looks like with real benchmark numbers.

How does the cascade architecture work?

In a cascade, the three stages run sequentially. The STT model cannot pass anything to the LLM until it has transcribed the user's speech. The LLM cannot produce audio until the TTS receives its output. But with streaming, sequential does not mean blocking: each stage can start sending output to the next before it has finished its own work.

The cascade architecture dominates production because it keeps every component independently swappable. As Gradium CEO Neil Zeghidour states: "at the moment what we do is cascaded systems because that's where the market is right now. People are still iterating a lot on the underlying text models that they want to use, on tool use and so on, and there is so much progress on the text side." Switching from GPT-4 to Claude, upgrading the STT model, or changing the TTS voice does not require touching any other component.

Stage 1: STT transcription

The pipeline begins when the user finishes speaking. The STT model transcribes the audio and passes text to the LLM. Two metrics matter here: accuracy (WER, which determines how often the LLM receives correct input) and latency (TTFT, Time to First Token, which is the delay before the first transcribed tokens are available to the next stage).

On the Coval STT benchmark (May 2026, 2,400 runs per model), median TTFT ranges from 992 ms for Deepgram Nova-3 to 1,560 ms for Gradium STT to 2,080 ms for ElevenLabs Scribe v2. Accuracy shows the inverse pattern: Gradium STT leads at 2.4% WER, followed by ElevenLabs Scribe v2 at 3.1%, AssemblyAI Universal Streaming at 4.2%, and Deepgram Nova-3 at 25.3%.

The STT tradeoff in the cascade: the fastest STT on latency (Deepgram Nova-3 at 992 ms median TTFT) has the lowest accuracy (25.3% WER), meaning roughly one word in four is transcribed incorrectly. The most accurate STT (Gradium at 2.4% WER) is slower. For voice agents handling structured content like account numbers, names, and addresses, accuracy has priority over speed because a mistranscribed input produces a nonsensical LLM response regardless of how fast the rest of the pipeline runs.

Gradium's STT also includes semantic VAD natively, which handles the end-of-turn detection decision inside the same model that does transcription. This is covered separately below.

Stage 2: LLM response generation

The LLM receives the transcribed text and generates a response. For the cascade pipeline, two properties of the LLM matter: first-token latency (how quickly the LLM emits its first token, which determines when TTS can start) and streaming output (whether the LLM emits tokens incrementally as it generates, rather than returning the full response at once).

All major LLM providers support streaming output. With a streaming LLM, the TTS layer does not wait for the full response before synthesizing. As the LLM emits its first sentence, that sentence goes directly to the TTS API. Audio for the first sentence can start arriving at the user's device while the LLM is still generating the second sentence. This LLM-TTS interleaving is what reduces total pipeline latency from the sum of all three stages in series to something significantly lower.

In the cascade, the LLM layer is completely standard. Every existing technique applies without modification: system prompts, few-shot examples, function calling, retrieval-augmented generation, and domain fine-tuning. The voice interface is a wrapper around a standard text LLM interaction. Any OpenAI-compatible LLM works in a cascade pipeline.

Stage 3: TTS synthesis

TTS receives text tokens from the LLM and synthesizes audio. TTFA (Time to First Audio) is the relevant metric: the time from the TTS API receiving the first text to the first audio chunk arriving at the client.

On the independent Coval TTS benchmark (May 4, 2026, 750 runs), Gradium leads at 155 ms TTFA P50 with a 2 ms latency IQR, the lowest and most consistent of all 9 streaming models tested. In Gradium's own published benchmark (Paris, 100 queries, WebSocket, warm): 258 ms P50 end-to-end, 274 ms P95, with multiplexing reducing this to 214 ms P50 and 228 ms P95.

With LLM-TTS interleaving enabled (the LLM streams tokens directly to the TTS as they arrive), the TTS does not wait for the LLM to finish. Gradium's full pipeline (STT + LLM + TTS) achieves 420 to 520 ms total turn latency with streaming LLMs like GPT-4 Turbo or Claude, compared to 2.5 to 5.5 seconds with non-streaming architectures where each stage waits for the previous to complete fully. Both figures are self-reported by Gradium.

What is the latency budget for a voice pipeline?

For a voice agent conversation to feel natural, the complete STT, LLM, and TTS pipeline, from end of user speech to start of agent audio, needs to stay under 800 ms.

With real benchmark numbers:

Two observations from this table. First, STT is the largest single contributor to pipeline latency in most configurations. Reducing TTS by 100 ms gains less than reducing STT by 100 ms, simply because STT's absolute contribution is larger. Second, the 800 ms target is achievable but requires all three stages to be streaming-capable. A non-streaming LLM that waits to finish generating before passing output to TTS will push the total pipeline well above 800 ms regardless of how fast the STT and TTS components are.

The practical 420 to 520 ms total latency Gradium documents assumes STT processes the transcript quickly (not the full 1,560 ms median TTFT, which is the TTFT measured in the Coval benchmark but which can be faster in the actual pipeline depending on utterance length), a streaming LLM emitting its first token in 200 to 400 ms, and Gradium TTS receiving the first token and starting synthesis within 155 ms.

How does end-of-turn detection work, and why is it hard?

Connecting STT to an LLM requires one more decision the pipeline must make before passing the transcript: is the user actually done talking? This is end-of-turn detection, and it is where many voice agents fail in ways that benchmark numbers do not capture.

Rule-based VAD makes this decision based on silence duration. As Neil Zeghidour describes: "you have an algorithm called the voice activity detection algorithm that just says whether it's silent or not. And if it's silent more than x amount of milliseconds, then this rule and this rule and this rule, then it's an interruption. But if this happened, it's not an interruption. And so you have rules on top of rules on top of rules to decide whether the model should talk or not."

The failure mode is consistent: rule-based VAD cannot distinguish a user who has finished speaking from a user who has paused briefly mid-thought. It either interrupts the user too early or waits too long after they finish, and no finite set of rules can cover the variability of natural speech patterns.

Semantic VAD uses the content of what the user is saying to determine whether their turn is complete. A model using semantic VAD can recognize that a question implies an expected response, that a sentence ending mid-clause is likely to continue, and that a pause after a complete thought is a genuine end-of-turn signal. Gradium's streaming STT API includes semantic VAD natively, built into the same forward pass that produces the transcript. Turn detection latency is 0 ms above transcription latency, because the same forward pass produces both. No second model is needed in the loop. The configuration reference is in the turn-taking guide.

In practice, semantic VAD is configured through the vad_threshold and vad_bucket parameters on gradium.STT() in the LiveKit integration. For the full implementation detail including delay_in_frames, horizon selection, and the flushing mechanism, see Semantic VAD: turn detection that uses meaning, not silence.

How do you wire the pipeline in practice?

Option 1: LiveKit with Gradium plugin

The fastest production path. Install the Gradium plugin and declare all three models in a single AgentSession:

pip install "livekit-agents[gradium]~=1.3"

session = AgentSession(
    stt=gradium.STT(vad_threshold=0.6, vad_bucket=1),
    llm=inference.LLM(model="openai/gpt-4.1-mini"),
    tts=gradium.TTS(),
    allow_interruptions=True,
    min_interruption_words=0,
    preemptive_generation=True,
)

The AgentSession handles the full cascade loop: it receives audio from the user, transcribes with Gradium STT including semantic VAD, passes the transcript to the LLM, streams LLM tokens to Gradium TTS as they arrive, and plays back audio. preemptive_generation=True starts LLM response generation before the user has fully finished their sentence, further reducing perceived response latency.

The LLM is declared using inference.LLM(model=...) which accepts any OpenAI-compatible model identifier: OpenAI, Anthropic, and self-hosted models all work. The orchestration layer (LiveKit) handles streaming coordination between components.

For the full setup including environment variables, function tools, and deployment: How to Build a Voice AI Agent with Gradium and LiveKit.

Option 2: Pipecat

Pipecat provides a Python-first event-loop abstraction for the pipeline. Install with:

uv add "pipecat-ai[gradium]"

Pipecat's pipeline framework handles the same cascade loop, with GradiumSTTService and GradiumTTSService as the voice components and any LLM service for the middle layer. For the full Pipecat integration: Gradium and Pipecat: Native TTS Integration for Voice Agents.

Option 3: Gradbot for fast prototyping

Gradbot is Gradium's open-source framework for building a working cascade pipeline in under 50 lines of Python. It handles VAD, turn-taking, fillers, and interruptions automatically. The developer defines instructions and any tool calls the agent should make. Gradbot accepts any OpenAI-compatible LLM:

import gradbot

voice = gradbot.flagship_voice("Emma")
config = gradbot.SessionConfig(
    voice_id=voice.voice_id,
    instructions="You are a helpful assistant.",
    language=voice.language,
)

input_handle, output_handle = await gradbot.run(
    session_config=config,
    input_format=gradbot.AudioFormat.OggOpus,
    output_format=gradbot.AudioFormat.OggOpus,
)

Gradbot is designed for prototyping and validation. For production at scale, use LiveKit or Pipecat. Repository at gradium.ai/gradbot.

What does the cascade not change about the LLM layer?

Because cascade keeps the LLM as a standard text-in, text-out component, every technique that works for text LLM applications works without modification in the cascade pipeline:

System prompts define the agent's persona, instructions, and constraints. They work exactly as in a text chat application.

Function calling and tool use work identically. The LLM can call external APIs, query databases, and trigger backend actions. The TTS layer synthesizes whatever text the LLM returns after tool calls resolve.

Retrieval-augmented generation works without modification. Context chunks retrieved from a vector database are injected into the LLM's context the same way they would be for a text chatbot.

Fine-tuning on domain-specific data works the same way. A fine-tuned LLM for healthcare, finance, or legal use cases slots directly into the cascade without any changes to the voice layer.

This is why the cascade architecture dominates production. In a cascade, the LLM is a standalone component: switching to a new LLM requires only updating that component, whereas in a speech-to-speech model the LLM intelligence is baked into the end-to-end trained weights. In a period of rapid LLM development, cascade keeps the voice application current with LLM improvements at zero additional cost.

Glossary

Cascade architecture

A voice agent design connecting three independent models in sequence: STT, LLM, and TTS. Each component is independently replaceable and upgradable. Dominant production architecture in 2026 because it preserves full LLM flexibility and compatibility with all standard LLM techniques.

TTFT (Time to First Token)

For STT: the elapsed time between receiving audio input and returning the first transcribed token. Gradium STT records 1,560 ms median TTFT on the Coval STT benchmark (May 2026). Deepgram Nova-3 records 992 ms median TTFT on the same benchmark.

TTFA (Time to First Audio)

For TTS: the elapsed time between sending text to the TTS API and receiving the first streamed audio chunk. Gradium records 155 ms TTFA P50 on the independent Coval TTS benchmark (May 4, 2026), and 258 ms P50 end-to-end in Gradium's self-reported benchmark. The lowest of all 9 models on Coval.

LLM-TTS interleaving

A streaming pipeline pattern where TTS synthesis begins on the LLM's first emitted tokens, before the full response is available. Requires a streaming LLM and a TTS API that accepts incremental text. Enables total pipeline latency of 420 to 520 ms (Gradium self-reported) vs 2.5 to 5.5 seconds without streaming.

Semantic VAD

Voice Activity Detection that uses the linguistic meaning of an utterance to determine end-of-turn, rather than silence duration alone. Native to Gradium's STT, running in the same forward pass as transcription with zero additional latency overhead.

Preemptive generation

A parameter in the LiveKit AgentSession (preemptive_generation=True) that starts LLM response generation before the user has fully finished their sentence, reducing perceived response latency in conversation.