I transcribe a lot of Bulgarian audio, and for the last five months my pipeline has been built around ElevenLabs Scribe v2. Five months is a long time in this field: xAI released Grok Voice Transcribe 2.0 this week, OpenAI now recommends gpt-transcribe, and Speechmatics shipped its multilingual Melia 1 model. So instead of trusting vendor leaderboards (Bulgarian is rarely on them), I re-ran every engine I have access to on my own data and scored them against transcripts I had corrected by hand, word by word.
The test set
23 recordings, 2 hours and 41 minutes in total, taken from my internal archive. The point was variety, not studio quality: most of the audio is exactly the kind of material that makes speech-to-text fail.
| Category | Files | Minutes | What it sounds like |
|---|---|---|---|
| Telephone calls | 8 | 45 | stereo recordings, one party per channel, 2–3 speakers, narrow-band phone audio |
| Voice-recorder and phone memos | 7 | 53 | dictaphones, iPhone and Android voice memos, noisy rooms, shouting, overlapping speech, children |
| Recorded customer-service call | 1 | 6 | operator and caller, clean phone line |
| Phone video | 4 | 32 | video shot outdoors and in stairwells, several people, wind and traffic, a home camera |
| TV broadcast | 3 | 25 | studio interviews and reportage, named speakers, clean audio |
Every file has a reference transcript that I corrected manually against the audio, with speaker labels. Two of the video references cover only part of the recording, so they stay in the per-file results but are excluded from the totals below (21 files, 150 minutes, about 22,700 reference words).
The engines
| Engine | Settings | Price per audio hour |
|---|---|---|
| ElevenLabs Scribe v2 | scribe_v2, language_code=bg, diarization on, word timestamps |
$0.22 |
| Speechmatics enhanced | operating_point=enhanced, language=bg, speaker diarization |
$0.30 |
| Speechmatics Melia 1 | model=melia-1, language=multi |
$0.30 |
| OpenAI gpt-transcribe | language=bg, response_format=json |
$0.27 |
| OpenAI gpt-4o-transcribe-diarize | diarized_json, chunking_strategy=auto |
$0.36 |
| OpenAI whisper-1 | verbose_json, word timestamps |
$0.36 |
| xAI Grok Voice Transcribe 2.0 | diarize=true, language auto (Bulgarian is not in its list of 25 languages) |
$0.10 |
| faster-whisper large-v3 (local) | int8_float16 on an RTX 2080 SUPER, VAD, beam 5, language=bg |
free |
| Combined Speechmatics + ElevenLabs | my existing two-engine reconciliation (ElevenLabs supplies structure, the more confident engine wins each disagreement) | $0.52 |
Video files were fed as their extracted audio track to every engine. Files over 25 MB were re-encoded to 48 kbps mono MP3 for OpenAI only, because of its upload cap. The whole round cost about $6.
How I scored
- WER (word error rate) after normalisation: lowercase, punctuation removed, ѝ → и.
- WERnn: the same, but digits, number words and filler words („ъ-ъ-ъ", „мхм") are ignored on both sides. My references write numbers as digits and drop most fillers, so plain WER punishes engines that transcribe more faithfully.
- Word recall: the share of reference words the engine got right. For my use this is the number that matters most. A dropped sentence is much worse than an extra „ми".
- Speaker purity: hypothesis words are aligned to reference words, and each detected speaker is mapped to the reference speaker it mostly covers. It is a proxy for diarization quality, since my references have no timestamps.
Results
Word-weighted over the 21 files with a complete reference. Lower WER is better, higher recall and purity are better.
| Engine | WER | WERnn | Word recall | Deleted | Inserted | Speaker purity | Files won (of 21) | Seconds per audio minute |
|---|---|---|---|---|---|---|---|---|
| ElevenLabs Scribe v2 | 22.5% | 20.5% | 81.9% | 9.2% | 4.4% | 94.6% | 14 | 3.2 |
| Speechmatics enhanced | 25.2% | 24.4% | 77.2% | 13.6% | 2.4% | 88.9% | 2 | 4.5 |
| Speechmatics Melia 1 | 27.2% | 26.4% | 74.5% | 14.3% | 1.7% | 88.1% | 1 | 0.8 |
| OpenAI gpt-transcribe | 28.3% | 27.6% | 72.6% | 19.8% | 0.9% | — | 3 | 1.4 |
| faster-whisper large-v3 (local) | 32.4% | 31.8% | 69.2% | 17.8% | 1.7% | — | 1 | 9.7 |
| OpenAI whisper-1 | 37.1% | 36.6% | 64.5% | 22.1% | 1.6% | — | 0 | 5.3 |
| xAI Grok Voice Transcribe 2.0 | 37.7% | 36.9% | 63.7% | 25.1% | 1.4% | 88.2% | 0 | 1.4 |
| OpenAI gpt-4o-transcribe-diarize | 43.0% | 42.1% | 63.0% | 12.1% | 6.0% | 89.6% | 0 | 20.2 |
| Combined Speechmatics + ElevenLabs | 22.2% | 20.6% | 81.8% | 9.8% | 4.0% | 93.3% | — | — |
WER by category:
| Engine | Telephone calls | Voice recorder / memos | Customer-service call | Phone video | TV broadcast |
|---|---|---|---|---|---|
| ElevenLabs Scribe v2 | 21.0% | 22.5% | 28.4% | 36.7% | 10.0% |
| Speechmatics enhanced | 21.6% | 27.1% | 20.7% | 39.3% | 12.9% |
| Speechmatics Melia 1 | 21.8% | 29.3% | 21.0% | 43.5% | 15.0% |
| OpenAI gpt-transcribe | 20.6% | 32.0% | 22.5% | 47.7% | 12.8% |
| faster-whisper large-v3 (local) | 27.3% | 36.2% | 23.7% | 49.5% | 15.8% |
| OpenAI whisper-1 | 34.1% | 37.0% | 28.2% | 61.3% | 19.3% |
| xAI Grok Voice Transcribe 2.0 | 27.3% | 45.7% | 37.5% | 58.5% | 13.9% |
| OpenAI gpt-4o-transcribe-diarize | 35.2% | 43.7% | 56.0% | 71.0% | 23.1% |
| Combined Speechmatics + ElevenLabs | 20.3% | 22.6% | 25.7% | 35.2% | 10.9% |
Yes, the absolute numbers are high. Clean studio speech comes in at 7–10% WER for the best engines; the rest of the corpus is people talking over each other in bad acoustics, and a 20% WER there is honestly good. The ranking is what matters, and it is stable: Scribe v2 has the lowest error on 14 of 21 files and is never worse than 6th on any of them.
What each engine gets wrong
- ElevenLabs Scribe v2 is the only engine that stays close to verbatim on the noisy conversations. It keeps the short interjections and the half-sentences that the others swallow, and its speaker turns match my labels on almost every file (purity above 95% on 17 of 21). Its weak spot is the opposite of everyone else's: it transcribes fillers and cross-talk that I leave out, which shows up as insertions on the clean phone calls. It also tends to find 2–3 speakers when there are 4 or more.
- Speechmatics enhanced is second on words and clearly weaker on speakers. It merged several two-person conversations into a single speaker and split one phone call into six. On the worst dictaphone file it produced fragments instead of sentences.
- Speechmatics Melia 1 is two points behind enhanced with the same diarization problems. It is five times faster, which is nice but not a reason to switch.
- OpenAI gpt-transcribe is the best of the newcomers and matches Scribe v2 on the clean phone calls. But it is a summarising transcriber: 0.9% insertions and 19.8% deletions. When people shout over each other, it silently skips whole exchanges. On the hardest recordings its recall fell to 37–50%. It also has no speaker labels, no timestamps and a 25 MB upload limit.
- OpenAI gpt-4o-transcribe-diarize is the worst paid engine here: a quarter of all words substituted, 20 seconds of processing per audio minute, and a hard limit of 1,400 seconds per request, so a 24-minute file had to be split in two.
- OpenAI whisper-1 still has the classic Whisper failure: repetition loops. One 2.5-minute call came back as the same street name repeated with „номер 1“ through „номер 6“ (98% WER), and another ended in the same three words repeated six times. It also rejects Android voice-recorder files, which are 3GP-branded containers with an .m4a extension, unless you remux them first.
- xAI Grok Voice Transcribe 2.0 does not support Bulgarian. It reports the language as English and still emits Bulgarian text, which is surprisingly usable on broadcast speech (13.9%) but loses a quarter of the words overall and half of them on the voice-recorder files. It is the cheapest and one of the fastest, and I will re-test it the day Bulgarian appears in its language list.
- faster-whisper large-v3 on a five-year-old GPU lands at 32% WER with the same "give up on hard passages" profile as gpt-transcribe. Fine as an offline fallback, not as the primary engine.
- The two-engine combination (Speechmatics + ElevenLabs) scores the same as ElevenLabs alone. The second engine no longer buys accuracy. What it still buys is a list of every span where the two engines disagree, which is exactly the list of places a human reviewer should re-listen to.
Verdict
For Bulgarian, in September 2026, ElevenLabs Scribe v2 is still the engine to use: lowest error rate, highest recall, and the only diarization I would trust without checking every turn. Speechmatics enhanced remains a good second engine for cross-checking. gpt-transcribe is worth a look if your audio is clean and you can live without speaker labels. Grok is not an option for Bulgarian yet, and whisper-1 should be retired.
The whole benchmark is scripted, so the next time a new model appears the retest is a 40-minute, $6 job. I will update this post when that happens.