Benchmark
How we choose a local Whisper setup
Summary
We do not ask you to trust a raw data download. We explain the decision instead: a local speech-to-text setup must keep names, numbers, mixed-language phrases, and intent intact; return soon enough to be useful; and still work on examples it has not seen. Our current Apple Silicon route is MLX Audio Whisper Large V3 Turbo ASR FP16 with a small, language-specific configuration. Separately, we check whether speech-to-text changes the answer a frontier model gives to the same intended input.
Reproducibility note
You should be able to understand why the setup was chosen without reading raw transcripts. We publish the reference device, what we compare, the selected model and configuration direction, the aggregate local-model result, and the decision rules. We keep the individual corpus and answers protected so a future model can still be tested on examples it has not seen.
View methodology manifest (JSON)What we tested
There are two complementary checks. The local-model benchmark chooses a Whisper setup by testing transcript quality and completion time on visible and hidden reference sets. The answer-parity check asks a more practical question: does the text produced after speech-to-text lead a frontier model to a meaningfully different answer than the original input? They share the same Auto Research discipline, but they answer different questions and are kept separate.
How we compare local STT setups
Does it keep the important words?
A transcript that loses a name, number, command, or mixed-language phrase is not useful just because it looks fluent.
We check: Score transcript fidelity, anchors, and critical terms against references.
Does it keep the answer useful?
We check whether the text after speech-to-text causes a frontier model to answer differently from the same original input.
We check: Compare direct-input and STT-derived answers with the Auto Research judge rubric.
Does it work on new examples?
A setting must not only look good on the examples used to tune it.
We check: Confirm it against held-back references before a keep decision.
Is the wait acceptable?
A higher score is not enough if you have to wait too long after speaking.
We check: Record average STT completion time with the selected local route.
Can it run on the intended hardware?
A model that only works in a lab is not a product route.
We check: Run the selected Apple Silicon route locally and keep hardware boundaries explicit.
Local Whisper Large V3 Turbo decision record
The selected model table is a decision log, not an industry-wide leaderboard. Earlier runtime checks and the current full configuration are labeled separately where their measurement phase differs.
| Setup | What we observed | Decision |
|---|---|---|
| MLX Audio Whisper Large V3 Turbo ASR FP16 — selected | 88.50 combined score; 1,870.97 ms average STT time. Full visible + hidden Apple Silicon run; deterministic decoding, forced language, narrow prompt allowlist, and a 120-second chunk only for audio at least 180 seconds long. | Keep. Best full visible-and-hidden configuration in the current local model line. |
| MLX Audio Whisper Large V3 Turbo ASR FP16 — prior R10 baseline | 87.98 combined score; 1,861.26 ms average STT time. Full visible + hidden Apple Silicon run before the Traditional Chinese-only prompt refinement. | Keep as regression baseline and fallback. The selected route added 0.52 score points for 9.71 ms additional average time. |
| MLX Whisper Large V3 Turbo Q4 — warmed process | 55.92 hidden score; 5,138.71 ms average time. Earlier hidden-first local runtime recheck. This earlier measurement phase is retained as a decision record, not a direct score ranking against the later full configuration. | Discard. Warming helped, but it remained slower and weaker than the later production-grade Apple Silicon route. |
| whisper.cpp ggml-base | 48.30 hidden score; 26,076 ms hidden average time. Earlier hidden-first local runtime check on multilingual long-form audio. This earlier measurement phase is retained as a decision record, not a direct score ranking against the later full configuration. | Discard. It was both too slow and too weak for this local product path. |
Limitations
This is a first-party methodology report, not third-party verification. It does not establish independent performance, universal accuracy, or a product guarantee.
The 88.50 / 1,870.97 ms result belongs to this Apple Silicon benchmark configuration. It is useful for explaining a selection decision, not for predicting every device, microphone, language, or audio length.
The answer-parity report does not disclose its input corpus, direct and STT-derived answers, judge outputs, rubric threshold, run count, model settings, language mix, microphone, or test date. We deliberately protect those materials so future candidates can still be tested against examples they have not seen.
Frontier models can change between runs. Results depend on the configured answer model, STT model, prompt, sampling settings, speaker, language, vocabulary, and environment. No Windows, iOS, Android, or Intel Mac result is claimed here.
How to choose
If you want to run Whisper Large V3 Turbo locally on an Apple Silicon Mac, start with the MLX Audio FP16 route rather than an older whisper.cpp base or MLX Q4 path. Keep the setup conservative: force the expected language, avoid one prompt for every language, and use a short Traditional Chinese prompt only where it has passed visible-and-hidden testing. This is a selection guide for a tested Apple Silicon route, not a universal recommendation for Windows or every local runtime. For Local AI eligibility, use the hardware requirements page.
Methodology
- Reference hardware: MacBook Pro with Apple M2 Pro and 64 GB memory. The supplied record does not include macOS version, microphone, or a recorded test date.
- For the local model route, we first test a candidate against examples we can inspect, then run the same kind of check against held-back examples. A configuration is kept only when the complete result is useful and does not trade a small gain on familiar examples for a regression on held-back ones.
- For each answer-parity input, Auto Research produces an answer from the original input using the configured frontier answer model (for example, Gemini 3.1 Pro or ChatGPT 5.5).
- The same intended input is spoken and transcribed with STT. That transcript enters the same answer workflow, producing a second answer.
- ChatGPT 5.5, configured in the local research environment as the judge, compares the STT-derived answer with the direct-input answer under the Auto Research rubric. A material difference produces a narrow modification suggestion to test next.
- GEPA is used as a reflection tool: it reads the benchmark evidence and suggests a next experiment. It never decides that a model wins. The visible-and-hidden result decides whether a candidate is kept, reviewed, or discarded.
- A related STT evidence line retains 80 `verbose_json` source artifacts with segment timestamps. Its fixed rewrite evaluation uses separate visible and hidden references and the same hidden-first keep / review / discard discipline.
Results
| Metric | Result | Notes |
|---|---|---|
| Reference device | MacBook Pro — M2 Pro, 64 GB memory | First-party supplied test configuration. |
| Selected local route | MLX Audio Whisper Large V3 Turbo ASR FP16 | Current full visible-and-hidden keep candidate on Apple Silicon. |
| Current selection result | 88.50 combined score; 1,870.97 ms average STT time | First-party local-model benchmark result for the selected configuration; it is not a universal accuracy percentage or a promise for every Mac. |
| Answer-parity metric | Answer-quality parity | Judge comparison of STT-derived and direct-input answers. |
| Configured answer model | Frontier model per run | Examples supplied: Gemini 3.1 Pro or ChatGPT 5.5. |
| Configured judge | ChatGPT 5.5 | Used in the local research environment under the Auto Research rubric. |
| Related STT evidence corpus | 80 retained source artifacts | Metadata-rich STT artifacts; the protected corpus is used to avoid tuning only to public examples. |
| Related fixed evaluation line | 10 cases with visible / hidden splits | Tests transcript rewriting, not the answer-parity metric above. |
FAQ
Evidence and updates
Facts checked on July 24, 2026
Reviewed by use.fo product team. Sources below identify whether a fact comes from an official product page, a first-party test, customer evidence, or an independent source.
These sources record first-party methodology and the internal Auto Research evidence lineage only. They support the disclosed Apple Silicon selection result and method, not a universal percentage-accuracy, cross-platform, or independent-performance claim.
- use.fo Auto Research STT-to-answer evaluation methodology — MacBook Pro M2 Pro / 64 GB reference hardware, selected MLX Audio Whisper Large V3 Turbo ASR FP16 route, visible-and-hidden selection result, answer-parity workflow, configured answer-model and judge roles, reporting limitations; checked July 24, 2026 (first party test).
- use.fo Auto Research STT Evidence workspace — 80 retained Groq verbose STT artifacts, segment-level metadata, fixed 10-case rewrite evaluation line, visible / hidden split, hidden-first decision discipline; checked July 24, 2026 (first party test). Internal verification workspace. Raw transcripts and reference answers are intentionally not public so the evaluation corpus remains protected.