omni-bench · 13 measurements · 11 engines · august 2026
Every engine here transcribes Russian. Ranked on ordinary Russian speech they are within a factor of 1.4 of each other. Ranked on the English terminology inside that speech they span a factor of 71.
For Russian engineering speech: ElevenLabs Scribe v2. It carries 71 of 94 technical terms and holds a competitive word error rate while doing it — the only engine measured that does not force a choice between the two.
For Russian itself: GigaAM-v3. On spontaneous speech it reaches 6.1% word error where the next best is 10.1% — at a fifth of the parameters. It also carries 19 terms of 94, second worst on the board.
Yandex SpeechKit carries one term of 94 while transcribing Russian perfectly acceptably. The Russian-market default is unusable for this workload, and no general benchmark would tell you.
The thesis
Horizontal: word error on spontaneous Russian carrying no English at all — 59 control recordings, so it measures Russian and nothing else. Vertical: technical terms carried, of 94. A single-metric leaderboard reads only one edge of this field.
GigaAM-v3 sits alone at the left edge: the best Russian on the board by a wide margin, and near the bottom for terminology. Yandex sits at the floor. Nothing in the horizontal position predicts the vertical one.
The board
87 recordings of a Russian software-engineering podcast, 33 minutes, 94 marked term occurrences. Every run complete over the exact sample set.
| Engine | Where | Params | Terms /94 | Term error | WER all | WER Russian-only |
|---|---|---|---|---|---|---|
| ElevenLabs scribe_v2 | hosted | — | 71 | 24.5% | 10.6% | 10.2% |
| OpenAI gpt-4o-transcribe | hosted | — | 65 | 30.9% | 18.4% | 19.6% |
| whisper large-v3 | local | 1.54B | 60 | 36.2% | 11.0% | 10.1% |
| AssemblyAI universal-2 | hosted | — | 52 | 44.7% | 11.5% | 10.3% |
| Deepgram nova-3 · multi | hosted | — | 50 | 46.8% | 11.5% | 10.4% |
| Qwen3-ASR | local | 2.04B | 44 | 53.2% | 12.1% | 11.0% |
| parakeet-tdt-0.6b-v3 | local | 0.63B | 39 | 58.5% | 13.0% | 11.6% |
| Canary 1B v2 | local | 0.96B | 36 | 61.7% | 13.9% | 12.4% |
| Deepgram nova-3 · ru | hosted | — | 34 | 63.8% | 12.8% | — |
| Nemotron-3.5-asr | local | 0.64B | 25 | 73.4% | 16.1% | — |
| GigaAM-v3 | local | 0.22B | 19 | 79.8% | 8.5% | 6.1% |
| Yandex SpeechKit | hosted | — | 1 | 98.9% | 16.4% | 13.3% |
| GigaAM-Multilingual | local | 0.59B | 0 | 100% | 11.9% | — |
Deepgram's Nova-3 offers language=ru and a language=multi mode documented for code switching. Nothing else changes — same model, same audio, same request. The mode is worth 34 terms against 50, a wider gap than separates most different engines on this board. Configuration is not a footnote here; it is the second-largest effect measured.
The control
The obvious control is FLEURS, the usual multilingual read-speech benchmark. It gives a reassuring, useless answer.
| Engine | FLEURS ru (read) | Podlodka controls (spontaneous) | Change |
|---|---|---|---|
| GigaAM-v3 | 8.7% | 6.1% | 0.70× |
| whisper large-v3 | 8.7% | 10.1% | 1.16× |
| ElevenLabs scribe_v2 | 9.0% | 10.2% | 1.14× |
| Yandex SpeechKit | 10.2% | 13.3% | 1.30× |
| OpenAI gpt-4o-transcribe | 7.2% | 19.6% | 2.71× |
On FLEURS, whisper and GigaAM tie to six decimal places — 105 word errors out of 1204 each. Their transcripts differ, so the tie is a coincidence of counts rather than identical output, but the conclusion a reader would draw from that column is that the two are equivalent. On spontaneous speech GigaAM is ahead by four-tenths of the other's error rate.
OpenAI's profile is the sharpest: best on read speech, worst on spontaneous, degrading by a factor of 2.7 where nobody else exceeds 1.3. Two explanations fit — FLEURS is a public benchmark that plausibly sits in training mixes, and read speech is simply easier — and these measurements cannot separate them. Either way the practical lesson is the same: a read-speech benchmark did not rank these engines the way the workload does.
Ruled out
Interventions
| Intervention | Engine | Effect | Cost |
|---|---|---|---|
| code-switch mode | Deepgram nova-3 | 34 → 50 terms | one parameter |
| oracle glossary | whisper large-v3 | 24 → 30 of 44 | knowing the answers |
| oracle glossary | GigaAM-v3 CTC | 5 → 12 of 44 | a custom rescorer |
| realistic glossary | whisper large-v3 | 24 → 24 of 44 | 6,100 mined terms |
| LoRA on synthetic data | GigaAM-v3 CTC | 14 → 14 of 94 | a day, and WER worsened |
An oracle glossary — the actual answer key — lifts whisper by six terms. A realistic glossary of 6,100 terms, mined by frequency from a thousand episodes of a different Russian IT podcast, lifts it by none. Coverage explains it: that glossary contains 58 of the 94 occurrences, and observability appears in none of the thousand episodes. Podlodka, Crew and Datadog are speaker-context proper nouns no external list will ever hold.
The deployable conclusion is narrower than "use a glossary": the glossary has to be your own vocabulary — service names, internal tools, team names out of your wiki and repositories. A generic IT word list is dominated by Google, Apple and iPhone, and shares almost nothing with what an engineering standup actually says.
Fine-tuning was tried properly and failed. Synthetic code-switched audio was built from two monolingual FLEURS corpora by phrase-level splicing, a LoRA was trained on the encoder's feed-forward layers, and the holdout loss fell from 2.33 to 0.60 — the model learned the synthetic task. It transferred nothing: 14 terms before, 14 after, and word error rose from 9.7% to 11.0%. At 772 utterances against the thousand hours per language pair the published recipe uses, this tests whether crumbs transfer, not whether the method works.
Limits
Provenance
Every figure comes from a Result document complete over the exact sample set, scored by the same pinned scorer, with the dataset, protocol, model revision and hardware bound into the identity.