On November 12, 2026, researchers released a comprehensive study detailing a four-level failure taxonomy for live streaming dictation automatic speech recognition (ASR), demonstrating that standard word error rate (WER) metrics systematically conceal severe transcription failures. Published as an arXiv preprint by speech researchers at heylemon.ai, the benchmark streamed identical speech across eleven production models, finding that systems with near-identical overall WER diverge dramatically under whispering, acoustic background noise, and multilingual code-switching.
In streaming dictation, speech recognition operates under distinct constraints compared to offline transcription. When users dictate messages, notes, or research logs, audio streams continuously and text inserts directly into applications. While conventional benchmarks evaluate pre-recorded files after stripping punctuation, capitalization, and numbers, live dictation requires users to manually detect and correct every omitted or misplaced word.
What the benchmark uncovered across eleven ASR systems
The study, titled “Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR” (arXiv:2611.16919), was conducted by Harras Mansoor, Muhammad Wisal, Hassan Bhatti, and Masood Shakir. The team evaluated eleven real-time speech systems: two self-hosted open models (Qwen3-ASR 1.7B and Whisper Large V3) alongside nine provider-hosted streaming services, including Meta Muse Voice Transcribe, Google Gemini 3.5 Transcribe (Live API), Deepgram Nova-3, AssemblyAI Universal-3.6 Pro and Universal-3.5 Pro, Zoom Scribe, ElevenLabs, OpenAI GPT-Live-Transcribe, and OpenAI Realtime Whisper.
The primary dataset comprised 106 scripted English dictations from a single speaker under three acoustic conditions: a quiet room, with a fan on, and whispered speech. Holding speaker acoustics constant enabled rigorous cross-model isolation. The authors supplemented this with 20 clips across eleven other language and dialect sets (including Modern Standard, Saudi, and Emirati Arabic; French; German; Hindi; Urdu; Mandarin; and three Spanish dialects), 400 bilingual code-switched recordings, and external validation on 107 speakers from the Meta EARS dataset and 400 clips from the VOiCES corpus.
The four-level failure taxonomy: Moving beyond aggregate WER
Traditional evaluations compress all errors into an aggregate WER, obscuring qualitative differences that determine usability. To resolve this, the authors introduced a deterministic taxonomy sorting every error into four functional tiers:
- Level 1: Recognition Errors: Acoustic decoding mistakes, distinguishing substitutions, deletions, and insertions. It identifies “near-misses” (words differing by 1–2 letters), named-entity errors, “missing stretches” (runs of five or more dropped words), and “cut-off endings” (truncation of the final three or more words).
- Level 2: Output Formatting Errors: Surface formatting flaws, including Inverse Text Normalization (ITN) failures (such as numbers written in words instead of digits), missing punctuation or capitalization, disfluency removal discrepancies, and incorrect scripts (such as Urdu transcribed into Devanagari).
- Level 3: Language Handling in Multilingual Speech: Failures in bilingual or code-switched contexts, including total omission of non-primary languages, unauthorized translation into English, or phonetic romanization.
- Level 4: Audio Handling Prior to the Model: Upstream pipeline flaws, notably voice activity detection (VAD) audio dropping, client chunking issues, and hallucinations produced during silent pauses.
Severe divergence in whispered speech and acoustic stress
On quiet English dictation, ten of thirteen system setups performed within a narrow band between 3.9% and 9.5% WER. Zoom Scribe achieved a 7.4% WER [95% CI: 6.5, 8.3], while OpenAI GPT-Live-Transcribe achieved 7.3% [95% CI: 6.3, 8.3].
Under whispered speech, however, their performance diverged sharply. Zoom Scribe’s error rate rose to 34.7% [95% CI: 19.2, 60.2], while GPT-Live-Transcribe recorded 8.8% [95% CI: 7.0, 10.4]. The gap was driven by word deletions: Zoom omitted 28% of whispered words, and both AssemblyAI models (Universal-3.5 and 3.6 Pro) omitted 26% to 28%. Disabling or relaxing provider-side end-of-turn detection still left 21% to 26% of words missing. In contrast, seven other systems dropped at most 3% of whispered words.
| System | Quiet English WER (%) | Fan Noise WER (%) | Whispered WER (%) | Whispered Deletions (%) | Formatted Error Rate (Quiet) |
|---|---|---|---|---|---|
| OpenAI GPT-Live-Transcribe | 7.3 | 3.6 | 8.8 | 1.2 | 11.2 |
| Zoom Scribe | 7.4 | 3.6 | 34.7 | 28.0 | 13.7 |
| AssemblyAI 3.6 Pro | 8.2 | 4.1 | 30.5 | 26.0 | 13.1 |
| Whisper Large V3 (Self-Hosted) | 8.3 | 6.5 | 10.1 | 2.2 | 16.0 |
| OpenAI Realtime Whisper | 8.2 | 4.6 | 8.9 | 1.2 | 11.9 |
| Google Gemini 3.5 Transcribe | 6.9 | 2.5 | 5.0 | 1.1 | 9.1 |
| Meta Muse Voice Transcribe | 9.5 | 7.4 | 11.3 | 1.1 | 18.3 |
| Deepgram Nova-3 | 13.6 | 11.4 | 19.8 | 2.5 | 22.8 |
| Qwen3-ASR 1.7B (Self-Hosted) | 13.6 | 7.0 | 46.6 | 32.0 | 21.6 |
External validation across 107 speakers from the Meta EARS corpus confirmed this vulnerability: Zoom, AssemblyAI, and gated models lost 7% to 31% of whispered words, while six systems kept whispered omissions under 2%. On Zoom Scribe, 45 of 107 speakers experienced whispered error rates of 20% or greater.
The hidden pipeline penalty: Voice activity detection and silence hallucinations
Evaluating nine Whisper Large V3 configurations (Section 9), the researchers revealed how client-side VAD wrappers alter transcription accuracy. When Whisper was deployed behind a standard Silero VAD dropping detected silence, whispered WER jumped from 10.1% to 51.4%, with word deletions surging to 44.6%. The VAD misidentified low-energy whispered phonemes as background noise.
Lowering Silero’s detection threshold from 0.5 to 0.1 did not remedy omissions (WER remained 45.3%) while admitting unwanted noise. Conversely, Tencent’s TEN VAD preserved whispered speech at 10.7% WER with 2.1% deletions. However, eliminating silence dropping exposed a competing failure mode: silence hallucinations. When exposed to pure silence or unvoiced background sounds, self-hosted Whisper generated invented phrases on 6 of 10 test clips, and OpenAI’s batch Whisper-1 hallucinated on all 10 silent clips.
Language filtering and output formatting friction
In 400 bilingual recordings combining English with a secondary language, single-language settings caused massive data loss. When set to “English only”, Gemini, Deepgram, and self-hosted Whisper dropped the non-English segment in 86% to 99% of recordings. Even with both languages pre-selected, Gemini and Whisper still lost the secondary language in 33% to 46% of cases. In contrast, OpenAI GPT-Live-Transcribe and Realtime Whisper lost the secondary language in only 4% to 12% of recordings.
Formatting evaluations highlighted further discrepancies: Qwen3-ASR wrote 68% of digit numbers as words, Meta 47%, and GPT-Live 45%. Proper names also showed marked degradation: Deepgram had a 9.7-point accuracy drop on names (81.4% correct versus 91.1% for general words), and self-hosted Whisper suffered a 6.5-point gap.
What this means for researchers, universities, and research computing offices
The study highlights practical implications across academic computing, fieldwork, and clinical documentation:
- Auditing Self-Hosted Research ASR: Academic computing teams deploying Whisper Large V3 or Qwen3-ASR to safeguard data privacy in clinical studies or human subjects research must audit VAD configurations. Standard Silero VAD wrappers configured to discard silence will silently purge whispered or hesitant participant statements.
- Protecting Qualitative Fieldwork: Researchers gathering interviews in sensitive settings using qualitative data collection techniques risk silent data omissions that compromise qualitative coding in qualitative analysis platforms.
- Clinical Dictation Procurement: University health systems appraising ambient AI scribes and clinical speech recognition cannot depend solely on published WER. Models dropping 25% of whispered words pose clinical documentation hazards. Procurement teams must require stress tests across vocal dynamics and named entities.
- Refining Benchmarking Standards: Speech researchers establishing synthetic benchmarks or conducting a reproducible AI experiment must report deletions, insertions, and formatting metrics separately rather than relying on aggregated WER.
Next steps and open-source availability
The authors released their complete benchmarking harness on GitHub at heylemon-ai/streaming-dictation-asr, including streaming client scripts, scoring logic, and error classifiers. The underlying multi-condition audio corpus and reference transcripts are hosted on Hugging Face at hey-lemon/streaming-dictation-asr. Future work will extend the taxonomy to spontaneous conversational dictation and broader multilingual cohorts.
Sources
- Mansoor, Harras, Muhammad Wisal, Hassan Bhatti, and Masood Shakir. “Beyond Word Error Rate: A Failure Taxonomy of Streaming Dictation ASR.” arXiv preprint, arXiv:2611.16919, submitted November 12, 2026. https://arxiv.org/abs/2611.16919 (Accessed 2026-11-15).
- HeyLemon AI. “Streaming Dictation ASR Benchmark: Code and Streaming Client Harness.” GitHub repository, November 2026. https://github.com/heylemon-ai/streaming-dictation-asr (Accessed 2026-11-15).
- HeyLemon AI. “Streaming Dictation ASR Dataset.” Hugging Face Datasets repository, November 2026. https://huggingface.co/datasets/hey-lemon/streaming-dictation-asr (Accessed 2026-11-15).
- Meta Research. “EARS: Expressive Anechoic Recordings of Speech.” GitHub repository, 2024. https://github.com/facebookresearch/ears_dataset (Accessed 2026-11-15).
- LabROSA and SRI International. “VOiCES: Voices Obscured in Complex Environmental Settings.” Project repository and documentation, 2018–2020. https://voices18.github.io/ (Accessed 2026-11-15).
- Silero Team. “Silero VAD: Pre-trained Enterprise-Grade Voice Activity Detector.” GitHub repository, 2021–2024. https://github.com/snakers4/silero-vad (Accessed 2026-11-15).
