a neural audio codec turns sound into a short sequence a model can work with then turns that sequence back into audio. in token-based text to speech (tts), the middle representation contains discrete audio tokens. the voice model predicts those tokens instead of predicting every sample in a waveform.

how does a neural audio codec work?

the codec starts with a waveform. an encoder shrinks short slices of that waveform into vectors. a quantizer matches each vector to the closest entry in a learned codebook and keeps the entry number which is the audio token.

the decoder takes the selected codebook entries and rebuilds the waveform. this encoder, quantizer, and decoder pipeline learns from audio rather than relying only on fixed signal-processing rules.

quantization loses detail. two slightly different sounds may map to the same codebook entry, and the decoder cannot recover information the tokens did not keep.

residual vector quantization handles that loss in stages. the first codebook captures a rough version of the frame. each later codebook records part of the remaining error. more codebooks can retain finer detail, but they also create more tokens for the voice model to predict.

neural audio codec example

rumik's open-source tts model, rumik-oss 1, uses the mimi neural audio codec. the model works with 24 khz audio, which contains 24,000 waveform samples per second.

mimi divides that second into 12.5 frames, or one frame every 80 milliseconds. each frame contains eight codebook tokens.

12.5 frames per second × 8 codebooks = 100 audio tokens per second

12.5 frames per second × 8 codebooks = 100 audio tokens per second

rumik-oss 1 model predicts all eight tokens for one frame before it starts the next frame. mimi decodes those tokens into sound. two seconds of speech require 25 codec frames and 200 audio tokens.

the reduction is the useful part. the voice model works through 100 token positions per second instead of 24,000 individual waveform samples.

neural audio codec vs playback format

audio tokens are not an audio file. a browser cannot play a list of mimi token ids because only the matching mimi decoder knows how to turn them into a waveform.

a playback format controls how the finished audio leaves the api. rumik silk can return wav, mp3, opus, raw pcm, μ-law, or a-law. silk produces these outputs after the voice model and audio decoder have generated the speech.

mp3 and opus compress the final audio stream. raw pcm stores waveform samples directly. μ-law and a-law encode audio for telephony. these formats do not replace the neural codec inside a token-based voice model.

why do neural audio codecs matter for tts?

a transformer works well with sequences of discrete items. neural audio codecs give speech that shape so a model can generate audio tokens with the same next-token training objective used for text.

token rate affects the amount of generation work. a lower frame rate means fewer steps for each second of speech. extra codebooks may preserve more detail, but the model must predict every added token.

the codec can also limit the final sound. if clean speech becomes metallic or muffled after an encode-decode pass, a tts model cannot restore the missing detail. generated speech combines codec reconstruction errors with any mistakes the voice model makes while predicting tokens.

a causal codec can decode one frame without waiting for future frames. that makes progressive playback possible while the voice model continues generating the rest of the utterance.

test the codec on its own first. encode and decode a clean recording, then compare that reconstruction with the original. after that, test generated speech. the first comparison exposes codec loss, while the second also includes pronunciation, timing, and token-prediction errors from the tts model.