streaming text to speech (tts) returns speech in pieces instead of making a listener wait for the whole recording. playback can begin while later audio is still being made or delivered. the text may be complete before synthesis starts. sending unfinished text to tts is a separate capability, though both are called "streaming."
how does streaming tts work?
the tts service sends audio as it becomes ready, then delivers later chunks in order. the player queues each piece and starts speaking once it has enough sound to play without breaking up.
that small buffer matters. if playback starts too soon, a slow connection can leave a gap mid-sentence. hold too much audio, and the listener waits through silence.
audio format changes the work at the receiving end. raw pcm needs no codec decoder. compressed audio uses less bandwidth but must be decoded. neither guarantees a quick start, because the voice model still has to produce sound and the network still has to deliver it.
rumik silk can stream speech over websocket, so an application can play the opening audio while the rest is still arriving. in mulberry 1.5, speech tokens pass through an audio decoder frame by frame. the client still needs a playback buffer to keep the sound continuous.
what would a caller hear?
a clinic voice agent answers, "your appointment is at 3:30 on thursday. please bring your previous prescription." the full reply is written before tts starts. the caller can hear "your appointment..." while the rest of the audio is still arriving.
when testing streaming tts, log when usable audio reaches the application and when playback begins. a response header or a few container bytes are not speech the caller can hear.
streaming audio is different from streaming text input
the word "streaming" can describe either side of a tts request. a tts api can offer one feature or both.
capability | what arrives in pieces | what it lets the application do |
|---|---|---|
streaming audio output | generated speech | start playback before the full recording is ready. the input text may already be complete. |
streaming text input | text sent to tts | start synthesis while an application is still writing the reply. audio output may also be streamed. |
this difference becomes visible when a voice agent's language model writes a reply one phrase at a time. if tts accepts incremental text, the application can start synthesis before the model finishes writing. google's bidirectional streaming guide shows text and audio moving through the same session. amazon polly's api comparison contrasts that with a request using completed text.
sending every new word straight to tts can cause trouble. "your balance is..." leaves the voice waiting for a number. a later word might also change how the start of a phrase should sound.
an application can wait for a phrase boundary before sending that piece. the tts model has more context, but the caller waits longer.
what streaming changes in a live call
returning audio early can reduce the silence after a voice agent sends its reply to tts. it does not remove the time spent detecting the end of the caller's turn or generating that reply. playback may add another delay.
a tts latency benchmark measures only part of the wait.
interruptions reveal another limit. when a caller talks over the voice agent, the application must stop the old voice and clear any audio it has already queued. stopping generation alone can leave the last few words playing, which is why barge-in also has to clear buffered audio.
a finished file is often fine for a recorded lesson or an announcement that will be reused. a call needs audio that starts promptly, keeps playing, and stops when the caller cuts in. a quick start alone proves little.