how does our audio model remember which speaker to speak?

published

aug 15, 2026

rumik research team

the first hypothesis that comes to mind is that tts models may have an attention sink at the speaker token or may have high attention score there and the model keeps looking at speaker token, but looking at attention maps gives you a different story.

we studied mid-training checkpoint of our open source tts model by rumik.ai which extends a base llm to emit neural audio codec tokens. we found that the model doesn’t use speaker tags as attention sink for speaker consistency throughout the utterance, instead the actual attention sink is the <BOS> token taking 62.7% of the mass, meanwhile the speaker token gets 0.6% and colon after speaker token gets 0.53% on average.

speaker token and colon gets comparatively high attention score when generating the first audio token as compared with their attention scores when generating the remaining audio tokens.

token	attention scores at the first audio token	attention scores after 32 audio tokens	attention scores after 64 audio tokens	change
BOS	0.7334	0.6955	0.6059	-1.2x
name	0.0214	0.0108	0.0029	-7.4x
colon	0.0307	0.0143	0.0065	-4.8x
sentence	0.0920	0.0186	0.0046	-20.1x
prior audio	NA	0.1769	0.3383	+1.9x

we also ran a knockout test where we zeroed out the value (V) vector in every layer for chosen prompt positions, without touching the attention weights so that the model still attended to that token but retrieves nothing and measured speaker similarity using ECAPA-TDNN and the WER of the transcript generated by whisper-large-v3

token at which V vector is zeroed	speaker identity	word error rate
nothing (baseline)	1.00	0.00
the colon	0.607	0.00
speaker name	0.486	0.107
<BOS> token	0.028	0.964

the speaker identity scores go down by 39.3% when the value at colon is zeroed and when speaker token is zeroed the speaker identity score goes down by 51.4%. zeroing the value vector at BOS token destroys both the speaker identity and intelligibility since most of the internal calculation happens at attention sinks.

but this explains how speaker identity gets introduced and not how it is sustained.

this makes it clear that speaker tokens doesn’t maintain the consistency over the time, that makes us think, what else does?

the prompt starts the voice and audio keeps it

we found a clue in attention scores throughout the audio token generation… mass on BOS falls while the mass on the model’s own prior audio climbs from 0.18 to 0.34, maybe the speaker identity is not held by the prompt at all, but by the model listening to itself.

to test this we took a simple approach of using only a sliced prefix from model’s own output. we took an audio token sample from model’s output but with a different speaker and attached it after prompt naming another speaker.

the results show a transition with only 2 frames of audio (0.16s), the audio is generated according to speaker specified by the speaker tag, but by 128 frames (~10s) the audio tokens have taken over and the speaker in the audio is now according to prior audio tokens instead of speaker tag. the transition sits somewhere around 8-32 frames (~3s).

the mechanism is auto-regressive self-conditioning with a short handover. the prompt decides the voice and within few seconds the audio stream becomes responsible for it, after that speaker prompt is close to vestigial. swapping it for a different speaker still does not change the voice 92% of the time.

we also have more exciting insights about the model and will be posting soon about how we used these insights to increase performance of our model.