what is word error rate?
what is word error rate?
what is word error rate?
word error rate (wer) measures how many words in a transcript differ from a reference text. it counts words that were replaced, removed, or added. a lower wer means the two texts match more closely and a wer of 0 means every word matches.
speech recognition benchmarks use wer to compare a model's transcript with what a person actually said. text to speech (tts) evaluation can use the same metric in reverse. a speech recognizer transcribes the generated audio, then the transcript is compared with the text the voice model was asked to speak.
how is wer calculated?
wer uses word-level edit distance. it finds the smallest number of changes needed to turn the reference into the transcript then divides that count by the number of words in the reference.
WER = (substitutions + deletions + insertions) / words in the reference
WER = (substitutions + deletions + insertions) / words in the reference
a substitution replaces one reference word with another. a deletion means a reference word is missing from the transcript. an insertion is an extra word that does not appear in the reference. this is the standard wer calculation used by nist.
word error rate is often shown as a percentage. it can exceed 100% when a transcript contains more errors than the reference contains words, usually because of many insertions.
wer calculation example
suppose a tts model receives the following reference text:
Your appointment is on Friday
Your appointment is on Friday
a speech recognizer hears the generated audio and returns:
Your appointment was Friday today
Your appointment was Friday today
the transcript substitutes was for is, deletes on, and inserts today. the reference contains five words, so the calculation is:
WER = (1 substitution + 1 deletion + 1 insertion) / 5 reference words WER = 3 / 5 = 60
WER = (1 substitution + 1 deletion + 1 insertion) / 5 reference words WER = 3 / 5 = 60
the score identifies three word-level differences. it does not explain whether the tts model pronounced a word badly, skipped it, or produced audio that confused the speech recognizer.
wer vs cer and gcer
word error rate(wer) treats each word as one unit. character error rate (cer) runs the same edit-distance calculation over characters. grapheme cluster error rate (gcer) uses visible writing units, which may contain more than one unicode code point.
metric | unit | one changed mark can count as |
|---|---|---|
wer | word | one incorrect word |
cer | unicode code point | one character-level edit |
gcer | grapheme cluster | one incorrect visible character |
the unit changes what the score reveals. if a transcript changes one letter inside a long word, wer counts the whole word as wrong while cer records a smaller error.
normalization also affects the result. ₹500, Rs 500, and five hundred rupees may express the same amount but produce different word sequences. a fair comparison needs the same rules for punctuation, casing, numbers, and spelling across every system.
why does wer matter for tts voice models?
wer gives a repeatable way to test whether generated speech preserves the requested words. it can expose dropped phrases, repeated words, changed dates, and unclear numbers across a large test set. those errors matter in voice agents because one wrong time, amount, or account digit can change the meaning of a reply.
rumik uses wer as one part of the rumik-oss 1 training and evaluation process. generated speech is transcribed with indicconformer rnnt, then the normalized transcript is compared with the input text. the published benchmark covers 15 languages and provides its prompts, scoring code, methodology, and per-sample results.
the speech recognizer remains part of the measurement. it may make its own mistake even when the generated word is understandable to a listener. results from different asr judges, text-normalization rules, or test sets should not be compared as if they came from one experiment.
a low wer also says little about timbre, pacing, emotion, speaker identity, or audio quality. use it to measure word fidelity, then listen to the same outputs for the parts of speech quality that a transcript cannot contain. for a voice agent, review errors by field as well as by average score. a mistake in a greeting and a mistake in an otp both add one error, but only one can stop the caller from completing the task.
word error rate (wer) measures how many words in a transcript differ from a reference text. it counts words that were replaced, removed, or added. a lower wer means the two texts match more closely and a wer of 0 means every word matches.
speech recognition benchmarks use wer to compare a model's transcript with what a person actually said. text to speech (tts) evaluation can use the same metric in reverse. a speech recognizer transcribes the generated audio, then the transcript is compared with the text the voice model was asked to speak.
how is wer calculated?
wer uses word-level edit distance. it finds the smallest number of changes needed to turn the reference into the transcript then divides that count by the number of words in the reference.
WER = (substitutions + deletions + insertions) / words in the reference
a substitution replaces one reference word with another. a deletion means a reference word is missing from the transcript. an insertion is an extra word that does not appear in the reference. this is the standard wer calculation used by nist.
word error rate is often shown as a percentage. it can exceed 100% when a transcript contains more errors than the reference contains words, usually because of many insertions.
wer calculation example
suppose a tts model receives the following reference text:
Your appointment is on Friday
a speech recognizer hears the generated audio and returns:
Your appointment was Friday today
the transcript substitutes was for is, deletes on, and inserts today. the reference contains five words, so the calculation is:
WER = (1 substitution + 1 deletion + 1 insertion) / 5 reference words WER = 3 / 5 = 60
the score identifies three word-level differences. it does not explain whether the tts model pronounced a word badly, skipped it, or produced audio that confused the speech recognizer.
wer vs cer and gcer
word error rate(wer) treats each word as one unit. character error rate (cer) runs the same edit-distance calculation over characters. grapheme cluster error rate (gcer) uses visible writing units, which may contain more than one unicode code point.
metric | unit | one changed mark can count as |
|---|---|---|
wer | word | one incorrect word |
cer | unicode code point | one character-level edit |
gcer | grapheme cluster | one incorrect visible character |
the unit changes what the score reveals. if a transcript changes one letter inside a long word, wer counts the whole word as wrong while cer records a smaller error.
normalization also affects the result. ₹500, Rs 500, and five hundred rupees may express the same amount but produce different word sequences. a fair comparison needs the same rules for punctuation, casing, numbers, and spelling across every system.
why does wer matter for tts voice models?
wer gives a repeatable way to test whether generated speech preserves the requested words. it can expose dropped phrases, repeated words, changed dates, and unclear numbers across a large test set. those errors matter in voice agents because one wrong time, amount, or account digit can change the meaning of a reply.
rumik uses wer as one part of the rumik-oss 1 training and evaluation process. generated speech is transcribed with indicconformer rnnt, then the normalized transcript is compared with the input text. the published benchmark covers 15 languages and provides its prompts, scoring code, methodology, and per-sample results.
the speech recognizer remains part of the measurement. it may make its own mistake even when the generated word is understandable to a listener. results from different asr judges, text-normalization rules, or test sets should not be compared as if they came from one experiment.
a low wer also says little about timbre, pacing, emotion, speaker identity, or audio quality. use it to measure word fidelity, then listen to the same outputs for the parts of speech quality that a transcript cannot contain. for a voice agent, review errors by field as well as by average score. a mistake in a greeting and a mistake in an otp both add one error, but only one can stop the caller from completing the task.
related terms
related terms
related terms