from mel spectrograms to neural audio codecs: how modern speech ai changed
from mel spectrograms to neural audio codecs: how modern speech ai changed
published
published
July 8, 2026
July 8, 2026
if you've used a voice ai product in the last few years, you've heard the result of a design decision most people never think about: how do you turn sound into something a computer can actually learn from? for most of the last decade, what we had was mel spectrogram. now the newer gen of speech systems the ones being trained the same way llms are, gets built around something structurally different, called neural audio codec.
this isn't a just a fancy upgrade. it changes what's computationally possible, what a model can be trained to do end-to-end, and why "speech ai" is increasingly becoming the same discipline as "language ai" instead of a separate field next to it. to see why, let’s dig into both representations.
many researchers assume that if they use muP (a method by Yang et al, 2022, for transferring hyperparameters across model sizes), they are safe from LR-tuning headaches.
sound, before it's anything else
sound, before it's anything else
start from the physical object. sound is air pressure moving back and forth. when someone speaks, the air near their mouth is repeatedly squeezed and released, and that pressure traces out a curve over time. a microphone doesn't understand any of that, it's a transducer, a device that converts one form of physical energy into another. it turns pressure changes into a continuous electrical voltage that rises and falls in the same shape as the original sound. at this stage, nothing has been "decoded." words, tone, speaker identity don't even come into the picture right now.
to get that voltage into a computer, it has to be measured at discrete moments in time (called sampling) and each measurement has to be rounded to one of a limited number of storable values (called quantization). when done together, you get pcm (pulse code modulation) which is just a long list of numbers. that list, plotted against time, is what people call a waveform. it's the rawest usable digital representation of speech, and it's also a genuinely difficult thing for a model to work with directly: a single second of audio at a typical sample rate contains tens of thousands of individual numbers, with no obvious structure a model can latch onto. this is the starting problem every voice ai system has had to solve: raw waveform is too dense and too unstructured to predict directly, so you need some intermediate representation that's more compact and more meaningful - and then a way to turn that back into a real, listenable waveform.
for most of the deep learning era, that intermediate representation was the mel spectrogram.
start from the physical object. sound is air pressure moving back and forth. when someone speaks, the air near their mouth is repeatedly squeezed and released, and that pressure traces out a curve over time. a microphone doesn't understand any of that, it's a transducer, a device that converts one form of physical energy into another. it turns pressure changes into a continuous electrical voltage that rises and falls in the same shape as the original sound. at this stage, nothing has been "decoded." words, tone, speaker identity don't even come into the picture right now.
to get that voltage into a computer, it has to be measured at discrete moments in time (called sampling) and each measurement has to be rounded to one of a limited number of storable values (called quantization). when done together, you get pcm (pulse code modulation) which is just a long list of numbers. that list, plotted against time, is what people call a waveform. it's the rawest usable digital representation of speech, and it's also a genuinely difficult thing for a model to work with directly: a single second of audio at a typical sample rate contains tens of thousands of individual numbers, with no obvious structure a model can latch onto. this is the starting problem every voice ai system has had to solve: raw waveform is too dense and too unstructured to predict directly, so you need some intermediate representation that's more compact and more meaningful - and then a way to turn that back into a real, listenable waveform.
for most of the deep learning era, that intermediate representation was the mel spectrogram.
what a mel spectrogram actually is
what a mel spectrogram actually is
a waveform tells you amplitude over time, but it hides a question that matters enormously for speech: which frequencies are present at each moment, and how strong is each one? vowels have concentrated bands of frequency energy shaped by the vocal tract; consonants produce very different frequency patterns; pitch is fundamentally a frequency phenomenon. none of that is easy to read directly off a waveform plot.
the tool for answering "which frequencies, how strong" is the fourier transform, which re expresses a signal as a combination of simple frequency components instead of a sequence of amplitudes over time, it doesn't lose information, it just changes what question you're asking of the same signal because speech changes constantly (a vowel, then a consonant, then a pause, all within a fraction of a second), you can't just run one fourier transform on an entire clip. instead you slide a short window across the audio, run the transform on each window, and stack the results that's a spectrogram: frequency on one axis, time on the other, and energy shown as intensity.
a regular spectrogram still uses a straight, linear frequency axis, and that doesn't match how human hearing actually works, we're far more sensitive to differences at low frequencies than at high ones. the mel scale is a frequency scale that's made to reflect that - more resolution where the ear cares more, less where it doesn't. apply a load of filters spaced on the mel scale to the spectrogram, and you get the mel spectrogram.
a waveform tells you amplitude over time, but it hides a question that matters enormously for speech: which frequencies are present at each moment, and how strong is each one? vowels have concentrated bands of frequency energy shaped by the vocal tract; consonants produce very different frequency patterns; pitch is fundamentally a frequency phenomenon. none of that is easy to read directly off a waveform plot.
the tool for answering "which frequencies, how strong" is the fourier transform, which re expresses a signal as a combination of simple frequency components instead of a sequence of amplitudes over time, it doesn't lose information, it just changes what question you're asking of the same signal because speech changes constantly (a vowel, then a consonant, then a pause, all within a fraction of a second), you can't just run one fourier transform on an entire clip. instead you slide a short window across the audio, run the transform on each window, and stack the results that's a spectrogram: frequency on one axis, time on the other, and energy shown as intensity.
a regular spectrogram still uses a straight, linear frequency axis, and that doesn't match how human hearing actually works, we're far more sensitive to differences at low frequencies than at high ones. the mel scale is a frequency scale that's made to reflect that - more resolution where the ear cares more, less where it doesn't. apply a load of filters spaced on the mel scale to the spectrogram, and you get the mel spectrogram.
a spectrogram - frequency content over time, computed via the Short-Time Fourier Transform (STFT), this is the step between a raw waveform and a mel spectrogram: still linear frequency, not yet perceptually scaled.
the mel scale wasn't adopted only because it resembles human hearing, turned out it also has practical advantages. first, it's dramatically more compact than the waveform it came from - roughly 80 to 100 frames per second, compared to tens of thousands of raw samples per second. second, it organizes sound into a structured time-by-frequency picture that a neural network can learn to predict frame by frame, almost like painting a board for how the sound should unfold, rather than trying to guess every microscopic wiggle of the waveform directly. that's why the classical pipeline - text, normalized and converted into phonemes, encoded and time-aligned, then decoded into a sequence of mel frames became the dominant design. the model's whole job was to predict this compact acoustic plan, it never touched the waveform itself.
the mel scale wasn't adopted only because it resembles human hearing, turned out it also has practical advantages. first, it's dramatically more compact than the waveform it came from - roughly 80 to 100 frames per second, compared to tens of thousands of raw samples per second. second, it organizes sound into a structured time-by-frequency picture that a neural network can learn to predict frame by frame, almost like painting a board for how the sound should unfold, rather than trying to guess every microscopic wiggle of the waveform directly. that's why the classical pipeline - text, normalized and converted into phonemes, encoded and time-aligned, then decoded into a sequence of mel frames became the dominant design. the model's whole job was to predict this compact acoustic plan, it never touched the waveform itself.
the piece mel spectrograms can't do alone: the vocoder
the piece mel spectrograms can't do alone: the vocoder
a mel spectrogram is more like a blueprint, than actual sound. it's not something a speaker can play back, it discards the phase information of the original signal and compresses detail at higher frequencies, which means the exact waveform shape has to be reconstructed by something else. that something else is the vocoder - models like wavenet, waveglow, and hifi-gan, whose entire job is to take the mel spectrogram and generate a real, physical waveform that would produce that same time-frequency pattern if you analyzed it again. it's the rendering stage: the acoustic decoder writes the plan, the vocoder performs it. the scale jump here is large - a mel representation running at roughly 80–100 frames per second has to be expanded into a waveform running at tens of thousands of samples per second.
for years, this two-step process worked extremely well. it kept the main modeling problem tractable and gave the field a stable, well-understood target to build around.
a mel spectrogram is more like a blueprint, than actual sound. it's not something a speaker can play back, it discards the phase information of the original signal and compresses detail at higher frequencies, which means the exact waveform shape has to be reconstructed by something else. that something else is the vocoder - models like wavenet, waveglow, and hifi-gan, whose entire job is to take the mel spectrogram and generate a real, physical waveform that would produce that same time-frequency pattern if you analyzed it again. it's the rendering stage: the acoustic decoder writes the plan, the vocoder performs it. the scale jump here is large - a mel representation running at roughly 80–100 frames per second has to be expanded into a waveform running at tens of thousands of samples per second.
for years, this two-step process worked extremely well. it kept the main modeling problem tractable and gave the field a stable, well-understood target to build around.
why that design has a ceiling
why that design has a ceiling
the limitation isn't that mel spectrograms sound bad. it's that they're the wrong shape of object for where the rest of ai has gone.
modern language models work by predicting the next item in a sequence, chosen from a fixed, finite vocabulary, that's easy with text but a mel spectrogram doesn’t works like that. it's continuous (each vaue can be essentially any real number, not one of a fixed set of options), dense, and entangled, a lot of different acoustic properties are smeared together in a way that isn't naturally symbol-like. you can't cleanly do "pick the correct next mel frame from a vocabulary," the way it chooses next word, because there is no vocabulary there's an infinite space of possible values.
that mismatch is the real reason the field has been moving away from mel as the primary generation target. it's not that mel got worse; it's that the rest of the field moved toward architectures - transformer-based, autoregressive, next-token language models that fundamentally want discrete, tokenized input. if you want to use the same kind of model that predicts text tokens to also predict speech, speech has to be turned into something that looks like text: a sequence of symbols from a finite vocabulary.
the limitation isn't that mel spectrograms sound bad. it's that they're the wrong shape of object for where the rest of ai has gone.
modern language models work by predicting the next item in a sequence, chosen from a fixed, finite vocabulary, that's easy with text but a mel spectrogram doesn’t works like that. it's continuous (each vaue can be essentially any real number, not one of a fixed set of options), dense, and entangled, a lot of different acoustic properties are smeared together in a way that isn't naturally symbol-like. you can't cleanly do "pick the correct next mel frame from a vocabulary," the way it chooses next word, because there is no vocabulary there's an infinite space of possible values.
that mismatch is the real reason the field has been moving away from mel as the primary generation target. it's not that mel got worse; it's that the rest of the field moved toward architectures - transformer-based, autoregressive, next-token language models that fundamentally want discrete, tokenized input. if you want to use the same kind of model that predicts text tokens to also predict speech, speech has to be turned into something that looks like text: a sequence of symbols from a finite vocabulary.
what a neural audio codec actually is
what a neural audio codec actually is
a simple idea would be to just treat raw digital audio samples as tokens, since pcm is already discrete (a fixed number of possible stored values per sample). but this doesn't work in practice, for two separate reasons. first, the rate is unworkable - audio sampled at tens of thousands of samples per second means tens of thousands of tokens every second, and autoregressive generation over sequences that long is computationally prohibitive next to a sentence of text, which might be a few dozen tokens. second, a single raw sample carries almost no meaningful structure by itself - a single phoneme spans thousands of samples, so predicting one sample at a time gives a model no useful shortcut into the actual structure of speech.
what we really want is something that's both small and token like and genuinely discrete, like text tokens, that's what a neural audio codec does: it takes a speech waveform, compresses it into a sequence of tokens, and can later turn those tokens back into natural-sounding audio.
a simple idea would be to just treat raw digital audio samples as tokens, since pcm is already discrete (a fixed number of possible stored values per sample). but this doesn't work in practice, for two separate reasons. first, the rate is unworkable - audio sampled at tens of thousands of samples per second means tens of thousands of tokens every second, and autoregressive generation over sequences that long is computationally prohibitive next to a sentence of text, which might be a few dozen tokens. second, a single raw sample carries almost no meaningful structure by itself - a single phoneme spans thousands of samples, so predicting one sample at a time gives a model no useful shortcut into the actual structure of speech.
what we really want is something that's both small and token like and genuinely discrete, like text tokens, that's what a neural audio codec does: it takes a speech waveform, compresses it into a sequence of tokens, and can later turn those tokens back into natural-sounding audio.
soundstream model architecture. a convolutional encoder produces a latent representation of the input audio, quantized by a residual vector quantizer (RVQ); a convolutional decoder reconstructs the waveform from the quantized embeddings. source: arxiv.org/abs/2107.03312
the system has three parts: an encoder that compresses the audio, a middle step that turns it into tokens, and a decoder that rebuilds the sound, but with one core difference: everything is trained together as a single system, and the bottleneck in the middle is forced to be discrete rather than continuous.
that discreteness comes from a technique called vector quantization (vq). at each timestep, the encoder outputs a vector - some point in a multi-dimensional space. instead of keeping that exact continuous vector, vq replaces it with the closest match from a fixed, learned codebook - a set of reference vectors the model has learned during training. only the index of the closest match needs to be stored, which is just an integer. that integer is the token. a learned "alphabet" of typical sound chunks, and quantization is nearest-neighbor lookup into that alphabet.
a single codebook runs into a real limitation quickly: to capture enough acoustic detail, you'd need an enormous number of codebook entries, and in practice most of them end up barely used while training becomes unstable, a failure mode generally known as codebook collapse. the fix that made modern neural codecs work is residual vector quantization (rvq): quantize the vector against a first, small codebook, then quantize the error that first quantization made against a second codebook, then quantize the remaining error against a third, and so on. each stage is cheap and coarse on its own, but stacked together they reconstruct the original vector far more precisely than one large codebook could, the same idea as residual learning elsewhere in deep learning: solve the easy part first, then hand the leftover error to the next stage.
the system has three parts: an encoder that compresses the audio, a middle step that turns it into tokens, and a decoder that rebuilds the sound, but with one core difference: everything is trained together as a single system, and the bottleneck in the middle is forced to be discrete rather than continuous.
that discreteness comes from a technique called vector quantization (vq). at each timestep, the encoder outputs a vector - some point in a multi-dimensional space. instead of keeping that exact continuous vector, vq replaces it with the closest match from a fixed, learned codebook - a set of reference vectors the model has learned during training. only the index of the closest match needs to be stored, which is just an integer. that integer is the token. a learned "alphabet" of typical sound chunks, and quantization is nearest-neighbor lookup into that alphabet.
a single codebook runs into a real limitation quickly: to capture enough acoustic detail, you'd need an enormous number of codebook entries, and in practice most of them end up barely used while training becomes unstable, a failure mode generally known as codebook collapse. the fix that made modern neural codecs work is residual vector quantization (rvq): quantize the vector against a first, small codebook, then quantize the error that first quantization made against a second codebook, then quantize the remaining error against a third, and so on. each stage is cheap and coarse on its own, but stacked together they reconstruct the original vector far more precisely than one large codebook could, the same idea as residual learning elsewhere in deep learning: solve the easy part first, then hand the leftover error to the next stage.
the systems that made this real
the systems that made this real
this isn't a purely theoretical idea, it's the actual architecture behind the neural codecs used in production speech systems today, and it's worth naming a few, because each one solved a different piece of the problem:
soundstream was the system that introduced residual vector quantization to neural audio codecs. it works well at very low bitrates, around 1 kbps or below and specifically solved the problem of needing one giant, infeasible codebook at higher bitrates by splitting quantization across several residual layers (up to 80 in its original experiments) instead. it also introduced "quantizer dropout" randomly dropping some quantization layers during training, so a single trained model could serve a range of bitrates instead of needing a separate model for each one.
encodec built on the same rvq foundation and became a widely used open baseline; its 24khz model supports a range of bitrates from 1.5 up to 24 kbps depending on how many codebook layers are used at inference.
dac (descript audio codec) improved reconstruction quality further by projecting the encoder's output onto a smaller, lower-dimensional space before quantization, using a different activation function, and adding a multi-scale loss on the mel representation of the reconstructed audio during training.
speechtokenizer targeted a specific weak point: in autoregressive generation, errors in the first rvq codebook compound over the whole sequence, since every later token effectively depends on it. it addressed this by specifically enhancing that first codebook through semantic distillation, training it to also align with a self-supervised speech representation, not just reconstruction quality.
mimi, developed at kyutai, pushed frame rate down aggressively: it represents 24khz audio at just 12.5 frames per second at roughly 1.1 kbps, while running fully in a streaming, causal manner with 80 milliseconds of latency per frame, meaning it doesn't need to see future audio to encode the present. it applies the same semantic-distillation idea as speechtokenizer, but using a different self-supervised teacher model (wavlm), and notably manages to match that non-causal teacher's representation without introducing extra delay, despite being causal itself.
these aren't competing footnotes - they represent a real progression: first make discrete audio compression work at all (soundstream), then make it a widely usable open baseline (encodec), then improve raw reconstruction quality (dac), then make the resulting tokens more semantically meaningful and not just acoustically accurate (speechtokenizer, mimi), and increasingly, push the frame rate itself down, because a lower frame rate directly means a shorter token sequence for a generative model to predict.
this isn't a purely theoretical idea, it's the actual architecture behind the neural codecs used in production speech systems today, and it's worth naming a few, because each one solved a different piece of the problem:
soundstream was the system that introduced residual vector quantization to neural audio codecs. it works well at very low bitrates, around 1 kbps or below and specifically solved the problem of needing one giant, infeasible codebook at higher bitrates by splitting quantization across several residual layers (up to 80 in its original experiments) instead. it also introduced "quantizer dropout" randomly dropping some quantization layers during training, so a single trained model could serve a range of bitrates instead of needing a separate model for each one.
encodec built on the same rvq foundation and became a widely used open baseline; its 24khz model supports a range of bitrates from 1.5 up to 24 kbps depending on how many codebook layers are used at inference.
dac (descript audio codec) improved reconstruction quality further by projecting the encoder's output onto a smaller, lower-dimensional space before quantization, using a different activation function, and adding a multi-scale loss on the mel representation of the reconstructed audio during training.
speechtokenizer targeted a specific weak point: in autoregressive generation, errors in the first rvq codebook compound over the whole sequence, since every later token effectively depends on it. it addressed this by specifically enhancing that first codebook through semantic distillation, training it to also align with a self-supervised speech representation, not just reconstruction quality.
mimi, developed at kyutai, pushed frame rate down aggressively: it represents 24khz audio at just 12.5 frames per second at roughly 1.1 kbps, while running fully in a streaming, causal manner with 80 milliseconds of latency per frame, meaning it doesn't need to see future audio to encode the present. it applies the same semantic-distillation idea as speechtokenizer, but using a different self-supervised teacher model (wavlm), and notably manages to match that non-causal teacher's representation without introducing extra delay, despite being causal itself.
these aren't competing footnotes - they represent a real progression: first make discrete audio compression work at all (soundstream), then make it a widely usable open baseline (encodec), then improve raw reconstruction quality (dac), then make the resulting tokens more semantically meaningful and not just acoustically accurate (speechtokenizer, mimi), and increasingly, push the frame rate itself down, because a lower frame rate directly means a shorter token sequence for a generative model to predict.
why lower frame rate matters more than it sounds like it should
why lower frame rate matters more than it sounds like it should
it's worth being specific about the tradeoff, because it's the crux of why this whole shift matters for real products. a codec's frame rate determines how many tokens per second an autoregressive model has to generate to produce speech. higher frame rate generally preserves more acoustic detail, but it also means longer token sequences, more compute per second of generated speech, and more accumulated error over a generation, since autoregressive models generate one token at a time and errors can compound down the sequence.
this is exactly why later codec designs (mimi, and the research that has followed it) have pushed toward lower frame rates with more residual codebook layers per frame, rather than higher frame rates with fewer layers, research comparing the two approaches at the same overall bitrate has found that the lower-frame-rate, more-layers configuration tends to reconstruct audio better. in other words, frame rate isn't just a compute-efficiency knob; it directly interacts with the quality of what a generative model can produce.
why lower frame rate matters more than it sounds like it shouldit's worth being specific about the tradeoff, because it's the crux of why this whole shift matters for real products. a codec's frame rate determines how many tokens per second an autoregressive model has to generate to produce speech. higher frame rate generally preserves more acoustic detail, but it also means longer token sequences, more compute per second of generated speech, and more accumulated error over a generation, since autoregressive models generate one token at a time and errors can compound down the sequence.
this is exactly why later codec designs (mimi, and the research that has followed it) have pushed toward lower frame rates with more residual codebook layers per frame, rather than higher frame rates with fewer layers — research comparing the two approaches at the same overall bitrate has found that the lower-frame-rate, more-layers configuration tends to reconstruct audio better. in other words, frame rate isn't just a compute-efficiency knob; it directly interacts with the quality of what a generative model can produce.
what token-based speech actually unlocks
what token-based speech actually unlocks
once speech is represented as a sequence of discrete tokens instead of a continuous mel spectrogram, the type of model you can point at it changes. a standard transformer-based language model predicts the next item in a sequence from a fixed vocabulary — that's exactly the shape of problem discrete audio tokens now are. speech generation stops being a regression problem (predict a continuous vector) and becomes a language modeling problem (predict the next symbol), using the same core architecture and training approach that text-based llms already use. systems like vall-e and qwen3-tts take this approach directly predicting audio tokens the same way a language model predicts word tokens.
this has consequences that go beyond "one architecture instead of two." because the whole codec - encoder, discrete bottleneck, and decoder is trained end-to-end rather than as two separately-designed stages (a hand-engineered mel filterbank plus a separately trained vocoder), the system can learn a representation that's actually optimized for the downstream task, rather than relying on a fixed, hand-designed transform. and because tokens are a shared vocabulary-shaped object, speech and text can, in principle, live inside the same modeling framework, a single model reasoning over both text tokens and audio tokens in one sequence, rather than needing entirely separate systems glued together. that's the direction systems like kyutai's moshi take: modeling audio streams and text streams together within one architecture, aimed at real-time spoken dialogue rather than one-shot generation.
streaming, low-latency codecs like mimi matter specifically because of this framing: a causal codec that doesn't need to see future audio to encode the present is a requirement for any system meant to generate or respond to speech in real time, rather than only after an entire utterance has been recorded.
once speech is represented as a sequence of discrete tokens instead of a continuous mel spectrogram, the type of model you can point at it changes. a standard transformer-based language model predicts the next item in a sequence from a fixed vocabulary — that's exactly the shape of problem discrete audio tokens now are. speech generation stops being a regression problem (predict a continuous vector) and becomes a language modeling problem (predict the next symbol), using the same core architecture and training approach that text-based llms already use. systems like vall-e and qwen3-tts take this approach directly predicting audio tokens the same way a language model predicts word tokens.
this has consequences that go beyond "one architecture instead of two." because the whole codec - encoder, discrete bottleneck, and decoder is trained end-to-end rather than as two separately-designed stages (a hand-engineered mel filterbank plus a separately trained vocoder), the system can learn a representation that's actually optimized for the downstream task, rather than relying on a fixed, hand-designed transform. and because tokens are a shared vocabulary-shaped object, speech and text can, in principle, live inside the same modeling framework, a single model reasoning over both text tokens and audio tokens in one sequence, rather than needing entirely separate systems glued together. that's the direction systems like kyutai's moshi take: modeling audio streams and text streams together within one architecture, aimed at real-time spoken dialogue rather than one-shot generation.
streaming, low-latency codecs like mimi matter specifically because of this framing: a causal codec that doesn't need to see future audio to encode the present is a requirement for any system meant to generate or respond to speech in real time, rather than only after an entire utterance has been recorded.
where this leaves the mel-based pipeline
where this leaves the mel-based pipeline
none of this makes the classical mel-and-vocoder pipeline obsolete outright, plenty of systems still use some form of it, and it remains a well-understood, effective design for many use cases. but it's a fundamentally different kind of object from what the rest of modern ai, today's ai models are built to generate sequences of tokens, one step at a time. by turning speech into tokens instead of mel spectrograms, researchers made speech fit the same pattern as text. that means the same powerful models, training methods, and scaling techniques that made language models so successful can now be applied to speech too. the shift isn't because mel spectrograms stopped working - it's because tokenized speech fits much better with how modern ai models are designed.
none of this makes the classical mel-and-vocoder pipeline obsolete outright, plenty of systems still use some form of it, and it remains a well-understood, effective design for many use cases. but it's a fundamentally different kind of object from what the rest of modern ai, today's ai models are built to generate sequences of tokens, one step at a time. by turning speech into tokens instead of mel spectrograms, researchers made speech fit the same pattern as text. that means the same powerful models, training methods, and scaling techniques that made language models so successful can now be applied to speech too. the shift isn't because mel spectrograms stopped working - it's because tokenized speech fits much better with how modern ai models are designed.
none of this makes the classical mel-and-vocoder pipeline obsolete outright, plenty of systems still use some form of it, and it remains a well-understood, effective design for many use cases. but it's a fundamentally different kind of object from what the rest of modern ai, today's ai models are built to generate sequences of tokens, one step at a time. by turning speech into tokens instead of mel spectrograms, researchers made speech fit the same pattern as text. that means the same powerful models, training methods, and scaling techniques that made language models so successful can now be applied to speech too. the shift isn't because mel spectrograms stopped working - it's because tokenized speech fits much better with how modern ai models are designed.
references / further readings: