voice activity detection (vad) tells a speech system whether an audio stream contains human speech. it marks the start and end of speech without transcribing the words or deciding whether the speaker has finished a turn.
in a voice agent, that signal separates the caller's speech from silence and background sound. it can also reveal when the caller starts speaking over the generated voice.
how does voice activity detection work?
microphone audio comes in small chunks. a vad model gives each chunk a speech probability and a threshold turns that score into a speech or non-speech decision.
a cough or keyboard tap may look like speech for one chunk. a brief pause may look like the end of a sentence even when the caller is still thinking. minimum speech and silence durations keep the detector from switching states after every sound.
threshold choice creates a tradeoff. lower it and the detector catches quieter speech along with more noise. raise it and background sounds trigger it less often, though soft words may disappear with them.
voice activity detection example
a caller is looking for an order number while speaking to a delivery voice agent. a fan is running in the background. the caller says, "i need to change the address on... give me a second... order four eight two zero."
the fan should remain non-speech. vad should open a speech segment at "i need." pauses shorter than the configured minimum silence duration can stay within that segment. longer pauses can end a segment without ending the caller's conversational turn.
if the detector reacts to the fan, it may open a speech segment before the caller begins. if the application treats every segment ending as a finished turn, it may answer while the caller is still searching for the number. waiting too long to end a segment can also delay the reply after the caller has finished.
vad vs endpointing
vad spots speech while endpointing decides when the voice agent should answer.
difference | vad | endpointing |
|---|---|---|
question | does this audio contain speech? | has the caller finished their turn? |
input | small audio chunks | speech boundaries, pause length, and sometimes transcript context |
output | speech or non-speech | turn open or turn complete |
common error | reacts to noise or misses quiet speech | closes during a hesitation or waits too long |
a silent patch can mean the caller has finished but it can also be a hesitation, a breath, or a pause before an order number. vad marks the patch as non-speech and endpointing can keep the turn open by considering the length of the pause, the partial transcript, or a different turn-detection model.
why does vad matter for voice agents?
vad affects when a voice agent begins listening, when it considers speech to have stopped, and whether it notices an interruption. false positives can open a segment for background noise or stop tts playback when nobody spoke. false negatives can lose a quiet first word or miss a caller trying to interrupt.
a rumik voice agent built with livekit keeps the two jobs separate: silero.VAD.load() detects when the caller starts and stops talking. rumik.TTS() generates the spoken reply, while livekit manages turn-taking and interruptions around them.
clear speech from rumik will not fix a detector reacting to fan noise. accurate vad will not decide whether a pause means the caller has finished. the voice model, speech detector, and endpoints shape different parts of the same call.