How to Stream TTS Audio in Real Time: Bidirectional Streaming Guide

Streaming TTS audio in real time means the first audio chunk reaches the client before synthesis of the full text is complete. This is what enables sub-300 ms time to first audio (TTFA) in voice agents. Without true streaming, the TTS API generates a complete audio file before returning a single byte, which means the user waits for the entire response to be synthesized before hearing anything. At 258 ms P50 end-to-end (Gradium self-reported benchmark, Paris, 100 queries), or 155 ms P50 on the independent Coval production benchmark, this difference determines whether a voice agent feels conversational or broken.

Bidirectional streaming takes this further: the connection stays open in both directions simultaneously, so text can be sent while audio is still arriving. This is the architecture required for voice agents that must remain responsive while speaking, and it is what distinguishes a genuine real-time TTS API from one that only streams in one direction.

What does "real-time streaming" actually mean?

Not all providers that describe themselves as streaming use the same architecture. There are three distinct patterns in the market.

Complete audio file, then return. The API synthesizes the full text and returns the complete audio file when synthesis is done. No streaming at all. TTFA equals total render time. Unsuitable for real-time voice agents. Some batch-focused providers operate this way.

HTTP chunk transfer encoding. The API streams audio chunks within a single HTTP request as they are generated. Audio starts arriving before synthesis is complete. TTFA is lower than full-file mode. However, each request requires a new TCP and TLS connection handshake, adding approximately 40 to 100 ms per turn. In a 10-turn conversation, this accumulates to 400 to 1,000 ms of total overhead that does not appear in single-request TTFA benchmarks but is fully present in production. ElevenLabs Flash v2.5 uses this model.

Persistent WebSocket streaming. A single bidirectional connection handles synthesis across multiple conversation turns. Connection overhead is paid once at session start, not per turn. Audio streams incrementally as synthesis proceeds. The connection can receive text while sending audio simultaneously. This is the architecture required for genuine real-time voice agents. Gradium's TTS API is WebSocket-native.

How does the Gradium WebSocket TTS lifecycle work?

Every message sent or received on a Gradium WebSocket TTS connection is a JSON-serialized object with a type field. The full message reference is in the TTS WebSocket API documentation. As Olivier Teboul, CTO and co-founder of Gradium, explains: "the very first message must be the setup message. In it, we define the voice and the output format. As this is a WebSocket connection, you can send multiple text messages in a sequence and in response the server will send back multiple chunks of audio."

Step 1: open the connection

Open a WebSocket connection to the Gradium TTS endpoint, passing your API key in the request header. The endpoint accepts connections over secure WebSocket (wss://).

Step 2: send the setup message

The first message must be a setup message. It defines the voice, output format, and any optional parameters:

{
  "type": "setup",
  "voice_id": "your-voice-id",
  "output_format": "pcm",
  "json_config": {
    "rewrite_rules": "en"
  }
}

Key parameters in the setup message:

• voice_id: a voice ID from Gradium's catalogue or a cloned voice ID
• output_format: the output audio format. "pcm" returns raw 16-bit PCM frames at the configured sample rate, without a WAV header. "wav" wraps the audio in a WAV container
• json_config: optional object enabling text normalization rules (rewrite_rules) for structured content like phone numbers, dates, and email addresses
• pronunciation_id: optional, attaches a pronunciation dictionary for domain-specific terms

If the setup is valid, the server responds with a ready message confirming the stream is initialized.

Step 3: send text messages

Once the connection is ready, send text messages. You can send multiple text messages sequentially without waiting for audio to complete from a prior message:

{
  "type": "text",
  "text": "Your appointment is confirmed for Tuesday at 10:30 AM."
}

This is where bidirectional streaming becomes operational. The server begins synthesizing audio from each text message as it arrives. Audio chunks start coming back before synthesis of the full text is complete, and before you have finished sending all your text messages.

Step 4: receive audio chunks

Audio arrives as a series of audio messages, each containing audio data encoded as a base64 string. The format of the audio content matches what was specified in the setup message. For PCM output, the server sends raw PCM frames, not wrapped in a WAV header.

Alongside audio chunks, Gradium also sends processed text messages containing the text that was synthesized, with word-level timestamps. These timestamps are useful for synchronized captions, lip-sync applications, and transcript display where visual text should align with spoken audio.

Step 5: send the end-of-stream message

When you have finished sending all the text you want to synthesize, send an end-of-stream message:

{
  "type": "end_of_stream"
}

Step 6: receive the server's end-of-stream and close

After processing any remaining text and sending all remaining audio, the server sends its own end-of-stream message and closes the connection. If something went wrong at any point, such as an invalid API key, a missing setup message, or an unsupported format, the server sends an error message before closing the connection. The error code schema is shared between the TTS and STT endpoints.

How does bidirectional streaming work in practice?

The full power of bidirectional streaming is in the overlap between sending and receiving. In a voice agent pipeline, this enables two important patterns.

LLM-TTS interleaving

With a streaming LLM that emits tokens as they are generated, and a TTS API that accepts text incrementally as it arrives, TTS synthesis can begin before the LLM has produced its full response. The LLM emits the first sentence. That sentence goes to the TTS API. Audio for the first sentence starts arriving while the LLM is still generating the second. By the time the first audio chunk plays back on the user's device, the LLM may still be mid-response.

With LLM-TTS interleaving, TTS synthesis begins as the LLM emits its first tokens, before the full response is available. With a streaming LLM and Gradium TTS, the full pipeline (STT + LLM + TTS) achieves 420 to 520 ms total turn latency, compared to 2.5 to 5.5 seconds with non-streaming architectures (self-reported by Gradium). Implementing this pattern requires sending text to the TTS API incrementally as LLM tokens arrive, rather than buffering the full LLM response before calling TTS.

Multi-turn conversations over a persistent connection

In a multi-turn voice agent, reusing the same WebSocket connection across turns eliminates the per-turn connection handshake. The connection stays open between turns: when the user finishes speaking, the new text goes over the existing connection, and audio starts arriving immediately without the 40 to 100 ms handshake overhead that an HTTP-per-request architecture would add.

Gradium's self-reported benchmark shows the practical impact. With a standard WebSocket (one connection per session, reused across turns within that session): P50 258 ms, P95 274 ms. With WebSocket multiplexing (a single connection reused concurrently across multiple TTS streams): P50 214 ms, P95 228 ms. In a 10-turn conversation at 50 ms per-turn overhead, multiplexing saves approximately 450 ms of total accumulated latency compared to HTTP-per-request. This saving is invisible in single-request benchmarks but material in production agents.

The multiplexing implementation is documented in How to Multiplex TTS Requests Over One WebSocket Connection.

How do you configure the stream for your use case?

Audio format and sample rate

Gradium's TTS API outputs 16-bit PCM audio at 48 kHz by default. 24 kHz and 16 kHz are available as configurable options. The choice matters for the downstream integration:

• 48 kHz: standard for WebRTC-based web and app integrations
• 24 kHz: acceptable quality with lower bandwidth
• 16 kHz: telephony-compatible, avoids a resampling step when the audio goes to a SIP trunk or PSTN gateway

Codebook configuration and the latency/quality tradeoff

Gradium exposes a configurable tradeoff between TTFA and audio quality through codebook depth, set in the setup message via json_config. The three configurations:

These are self-reported figures from Gradium. For the independent Coval benchmark figure (155 ms P50, which reflects the default configuration), see TTS Latency Benchmark 2026.

Structured content and normalization

For voice agents that read back phone numbers, email addresses, dates, or alphanumeric codes, the json_config parameter in the setup message controls text normalization. Enabling a language alias like "rewrite_rules": "en" applies all recommended normalization rules for that language. Individual rules (phone numbers, dates, email expansion, URL handling) can also be enabled selectively. This is covered in full in How to Handle TTS Edge Cases with Text Normalization in Gradium.

HTTP streaming vs WebSocket streaming: what changes when you switch?

Teams migrating from providers that use HTTP chunk transfer encoding to Gradium's WebSocket API will see the same basic streaming principle (audio arrives before synthesis completes) with two meaningful differences.

First, the connection persists. HTTP streaming opens and closes a connection per request. WebSocket keeps the connection open. This changes how you handle connection management in your client code: open once per session, send multiple text messages, keep the connection alive for subsequent turns.

Second, the protocol is message-based rather than byte-stream-based. HTTP streaming delivers raw audio bytes as they arrive. Gradium's WebSocket delivers JSON messages containing base64-encoded audio. Your client decodes the base64 payload from each audio message and passes the raw bytes to your audio playback system.

If you are already using Cartesia's WebSocket TTS or ElevenLabs' WebSocket streaming, switching to Gradium requires updating the endpoint, voice ID, and authentication. The streaming flow is the same. For teams coming from HTTP-based providers, the additional step is adapting to the WebSocket message structure.

Glossary

Time to First Audio (TTFA)

The elapsed time between sending text to a TTS API and receiving the first streamed audio chunk. Gradium records 155 ms TTFA P50 on the Coval independent benchmark (May 4, 2026) and 258 ms P50 end-to-end in Gradium's own self-reported benchmark (Paris, 100 queries, warm). The difference reflects different measurement conditions, not a contradiction.

Bidirectional streaming

A WebSocket communication pattern where both the client and server can send and receive messages simultaneously on the same persistent connection. Enables LLM-TTS interleaving and multi-turn voice agent sessions without per-turn reconnection overhead.

WebSocket multiplexing

Reusing a single persistent WebSocket connection concurrently across multiple TTS streams. Reduces Gradium's effective TTFA from 258 ms to 214 ms P50 by eliminating per-session connection establishment overhead.

LLM-TTS interleaving

A streaming pipeline pattern where TTS synthesis begins as the LLM emits its first tokens, before the full response is available. Requires a streaming LLM and a TTS API that accepts incremental text input. Reduces effective end-to-end turn latency by overlapping LLM generation and TTS synthesis.

HTTP chunk transfer encoding

A streaming transport where audio is delivered within a single HTTP request as bytes become available. Opens and closes a connection per request. Not the same as WebSocket streaming. Adds 40 to 100 ms per-turn connection overhead in multi-turn conversations.

json_config

A parameter in Gradium's TTS WebSocket setup message that controls text normalization rules. Used to configure how structured content (phone numbers, dates, emails, URLs) is synthesized, and to set codebook depth for the latency/quality tradeoff. Pronunciation dictionaries are attached separately via the pronunciation_id setup parameter.