We cut VoiceFox's end-to-end dictation time from about 2.4 seconds down to 1.0 to 1.5 seconds. Here is what changed. VoiceFox is simple when it works well. Hold a hotkey, say a sentence, let go, and the words appear wherever your cursor is. No cloud round trip. No meeting transcript service. Just local dictation on an M-series Mac.
When it feels slow, though, the magic breaks. Try to dictate into a text box and wait: the pause is just long enough for your brain to ask, "Did it catch that?"
It was not one clever optimization; it was small measurements, a few autoresearch runs, and one rule: the benchmark has to protect the thing users actually care about. In dictation that is the words coming out right. Speed was the number we chased; accuracy was the thing we refused to trade for it.
A quick word on autoresearch
Autoresearch is what happens when you point an AI coding agent at a piece of code, hand it a benchmark that prints a number, and let it run a tight loop. Read the code. Try a small patch. Run the benchmark. Keep the patch if the number got better, revert it if it did not. The loop runs inside a fence that says exactly which files the agent may touch.
Andrej Karpathy created and named this pattern in his autoresearch repo, which builds on his nanochat. In Karpathy's setup the agent improves a tiny language model and is judged on a single training metric. We borrowed the loop and pointed it at a different target: instead of bringing a small model's training loss down, the agent shaves milliseconds off a local dictation app and has to keep transcription accuracy intact.
The agent is interchangeable. This round was Codex, because that was what we had wired up. The same loop works with Claude Code. It is the read, patch, evaluate, keep-or-revert pattern, plus the gate, that does the actual work.
What VoiceFox is
VoiceFox is push-to-talk dictation for Apple Silicon Macs. The model is Parakeet, NVIDIA's open-source speech recognition model, running locally through an MLX port. MLX is Apple's open-source machine learning framework built for Apple Silicon, so the model runs on the Mac's neural engine and unified memory rather than via CPU only or a cloud round-trip. Around that model is a practical little workflow: capture audio, transcribe it, clean up custom vocabulary, put the text on the clipboard, paste it into the active app, and then get out of the way.
VoiceFox is also intentionally a small script. That made it a good place to try autoresearch because the loop had one main file to work on, voicefox.py, and we could measure one layer at a time.
Six steps from speech to text
The app feels simple because the working path is short. Every latency fix had to respect this path.
First, measure the waiting
The first pass was not fancy. We added profiling and asked a boring question: where does the time go?
That audit found the kind of delays that hide in small apps. VoiceFox was writing captured audio to a temporary WAV file and reading it back before inference. It had fixed sleeps in the paste path. It had capture settings that were fine for correctness but not tuned for speed. It had decode settings that were conservative because nobody had proved a lower cap was safe.
| Change | Measured effect | Why it mattered |
|---|---|---|
| Direct audio processing | about 330 ms saved | VoiceFox no longer writes every capture to a temporary WAV file before feeding MLX. |
| Low-latency capture | about 50 to 100 ms saved | Recording overhead dropped without changing the interaction model. |
| Paste timing | about 160 ms saved | Fixed sleeps and clipboard sequencing were trimmed after profiling showed the floor. |
| Decode controls | about 50 to 100 ms saved | Parakeet decode caps were tuned only after a real-voice accuracy gate existed. |
Those changes account for roughly 590 to 690 ms of the improvement. The bigger headline drop, from about 2.4 seconds to 1.0 to 1.5, is a live end-to-end range: live dictation moves with clip length, model warm-up, and whatever else the Mac is doing. The table holds only the pieces we could measure one layer at a time and put a name on. Good start. But this is also where you can fool yourself. A latency benchmark will happily reward you for deleting safety. VoiceFox needed to get faster, but it still had to paste reliably and transcribe accurately.
Autoresearch is a loop, not a magic button
TaskFox, our agent skills factory, packages the boundary: what Codex or Claude Code may change, what test to run, and when to keep or revert the patch.
Then we let the loop try small changes
The first autoresearch loop went after post-submit latency: the time after VoiceFox has text and before that text lands in the active app. This was a nice starter problem. It was narrow, fast to measure, and mostly isolated from speech recognition quality.
Autoresearch progress: 8 experiments, 8 kept improvements
Charted in the spirit of Karpathy's autoresearch progress plot. Each green dot is a kept row. The step line is the running best. Lower is better.
workspace/autoresearch-voicefox-post-submit-latency/results.tsv in taskfox-factory. Every row in this run advanced, so there are no discarded points to plot. The accuracy-gated STT loop later in the article had the inverse shape: 50 experiments, only two kept.That run used a Codex backend with gpt-5.4-mini at low effort. The agent kept making tiny changes and measuring them. The baseline row was 202.0 ms. The best row was 66.7 ms after eight rows, a 67.0 percent reduction on that specific metric.
That run's gate only timed things: keep a change if it saved at least 25 ms or 10 percent, with the unit tests still green. Nothing in it ever listened to a voice. All 8 experiments advanced, and the report flagged the perfect score itself: a 100 percent advance ratio is a sign the target was easy. When a benchmark says yes every single time, it is not asking a hard enough question. Before trusting any change that touched the model, we needed a gate that could hear.
The next benchmark was too easy to trust
The next loop targeted the real speech-to-text path. We started with one real audio clip because it was fast and convenient. Codex gpt-5.4-mini at high effort did the first big search. It reached the requested 50 experiment rows and found a promising stack: faster audio handoff, a reused decoding config, a lower symbol cap, and a few small cleanup wins around the model call.
Then we handed the same lane to stronger model runs. gpt-5.5 at medium effort extended the loop past row 50. It did not beat the current best. A final gpt-5.5 xhigh pass tried deeper MLX ideas. None advanced. One crashed and was reverted.
Single-fixture loop: baseline median 233.434 ms, best loop median 178.041 ms, 65 experiment rows plus baseline, 6 advances, 57 discards, 2 crashes.
That looked like a clean win. Then confirmation made it less tidy, which is exactly what confirmation is for. Pre-review reruns showed a 216.994 ms median, about 7.0 percent better than the 233.434 ms baseline. Post-review reruns were noisier at 225.608 ms, while a longer 21-iteration check landed at 190.755 ms. Still useful, but not as simple as "the loop found 23.7 percent and we shipped it."
The single-clip win got smaller when we reran it
The loop found a tempting best row. Confirmation made the result more honest.
Speech-to-text median latency
From the VoiceFox run notes. The lesson was not that the loop was wrong; it was that one clip was too small to trust by itself.
This was the second trap: a single fixture is a leaderboard, not a product test. It can tell you a candidate is interesting. It cannot tell you the app still works for the variety of things a person actually says.
So I recorded my own voice samples
Every gate so far timed things and ran tests. Nothing ever listened, so I recorded something for the loop to listen to, with a small program Codex whipped up: scripts/record_voicefox_corpus.py. Nothing fancy. It showed me a short phrase, waited for me to read it aloud, recorded mono 16 kHz WAV, and wrote a manifest pairing each clip with the words I was supposed to say.
I recorded the whole thing in my car, in a parking lot, on my laptop, tethered to the internet through my phone. So the benchmark was not born in a studio or a lab. It came from the same kind of imperfect setup people actually dictate in: a laptop, a built-in mic, a parking lot, a phone hotspot, and an autoresearch loop grinding through bounded changes in the background.
The prompts were not random sentences. They were small, practical phrases that resemble actual dictation work:
- short commands such as "Open the build log" and "Save this note"
- normal dictation such as "Please summarize the pull request and mention any risky assumptions"
- fast speech, punctuation-like speech, long numbers, proper nouns, dense run-on phrases, quiet voice, and one longer dictation sample
The finished set had 16 clips. The audio stays private because it is my real voice, but the shape of the collection is visible in the repo. The recorder writes audio/*.wav and a manifest.json with id, path, category, and expected fields. That was enough structure for the benchmark to load the same clips every time and compare output against the expected transcript.
The gate made latency honest
The eval surface was bench_real_stt_corpus.py. It loads the local Parakeet model once, runs every WAV fixture, records latency samples, and then applies accuracy gates. The primary loop score is intentionally easy to understand:
gated_score_ms = median_ms + 0.25 * p95_msMedian latency captures the normal case. p95 adds a penalty for spiky behavior. Then comes the big rule: if any fixture fails accuracy, the score becomes 999999.0. A bad transcript cannot hide behind a faster median.
| Gate | Failure condition |
|---|---|
| Empty output | Any blank transcription fails. |
| Character similarity | Below max(0.90, baseline fixture similarity - 0.01) fails. |
| Word error rate | Above min(0.25, baseline fixture WER + 0.03) fails. |
| Output length | Below 85 percent of the baseline output length fails. |
Character similarity is a rough "does this look like the expected sentence?" score. Word error rate is exactly what it sounds like: how many words are wrong, missing, or inserted. Output length catches the especially sneaky failure where the model returns a short, plausible fragment and calls it done.
The loop was free to change decode and inference settings in voicefox.py, and nothing else. It could go fast because the gate could say no.
The run we trusted
The accuracy-gated run produced 50 experiment rows. Only two rows advanced. Forty-eight were discarded. The best loop row was retry_tiny_short_cap_transcripts, which used a lower decode cap for short audio and retried once with a safer cap if the first transcript looked suspiciously short.
Accuracy-gated decode loop
Lower gated_score_ms is better. Any accuracy failure becomes 999999.0.
The loop report looked stronger than the final retained result: 394.7 down to 307.6, a provisional 22.1 percent improvement. This is the point where it would have been easy to stop too early. Instead, we reran the retained candidate against the original baseline and two close challengers, then compared median-of-three confirmation scores. The guarded retry stayed ahead, but the confirmed gap was smaller. That is exactly why the confirmation step mattered.
The accepted change is also easy to state. For short audio, VoiceFox tries a faster decode setting first. If the transcript comes back suspiciously short, it retries once with the safer general setting. In code, that means a short-audio cap of 4, a general cap of 5, a short-audio threshold of 7 seconds, and a retry floor of 10 characters.
The rejected rows mattered too
The rejected rows matter as much as the accepted row. The loop tried lower caps, alternative retry floors, and nearby decode configurations. Some were faster in isolation. They did not beat the gated score after the full set of clips ran, or they looked less convincing once compared against the baseline and close challengers.
The one that survived had a last requirement: the tests had to cover the new short-audio path so the behavior could not disappear later.
The same loop ran against a second metric the same day: how long VoiceFox takes to start. It found a real win. Loop rows fell from 1736.267 ms to 1275.963 ms, 26.5 percent lower, and an apples-to-apples verification rerun confirmed a genuine 661.011 ms improvement: 2766.090 ms on main, 2105.079 ms on the candidate. Then the gate said no. The acceptance rule had been written before anyone saw a number: keep the change only if it saves at least 1000 ms or 5 percent, whichever is larger. The 1000 ms bar was the binding one, and the verified win fell about 339 ms short. The candidate was not integrated. The computer really did get faster, we checked twice, and we still said no, because the rule we wrote with no skin in the game said this win was too small to be worth carrying.
A verified 661 ms win that still did not ship
The acceptance rule was written before anyone saw a number: keep the change only if it saves at least 1000 ms or 5 percent, whichever is larger.
VoiceFox startup time
Verified delta: 661.011 ms (23.9 percent lower), about 339 ms short of the gate. The source artifact's verdict stamp reads "Do not integrate yet".
The reusable pattern
The reusable pattern is small. If you have the following, you can run a useful autoresearch loop:
- A piece of code or a setting you are willing to let an agent change
- A repeatable test that prints a number
- A rule that says when a result is not acceptable
The gate is the part that gets skipped most often. Without a gate, the loop becomes a hill climb that optimizes the visible metric and silently regresses a property the metric did not capture. With a gate, the loop can still move quickly, but it cannot win by cheating.
VoiceFox was not the first target. The same recipe has run against bytes, seconds, and milliseconds in this repo: a CSS bundle from 98,059 bytes to 40,118 across 152 logged experiments, an app startup from 2.413 seconds to 1.040, a pytest suite from 98.23 seconds to 59.08, a frontend eager-load payload from 171,517 bytes to 137,386. The honest rows ride along on purpose: one pytest rerun tried a single idea, made things worse, reverted it, and logged 46.96 seconds to 46.96 seconds, and the startup candidate above was rejected at its gate.
The same loop against bytes, seconds, and milliseconds
Bar length is the measured improvement in each campaign. The zero-length rows are real results too.
Improvement per campaign
Same loop, different targets. The empty rows are kept on purpose: the pytest rerun tried one experiment, made things worse, and reverted it, and the startup campaign's verified 661 ms win was rejected at its pre-registered gate and never integrated.
The 16-clip VoiceFox set is intentionally small. A bigger collection would catch more failure modes, but it would slow the loop and reduce the number of ideas the agent could test. For this problem, the better trade was a small set with sharp gates: enough clips to catch short commands, fast speech, numbers, proper nouns, quiet speech, and dense dictation, but fast enough to run repeatedly.
The pattern also makes incidents auditable. When the symbol-cap value gets bumped a year from now, the loop will run, and the gates will refuse the bump if the short-clip case still degrades. The decision lives in the eval, not in someone's memory.
What we learned
The 590 to 690 ms improvement is real, but the better lesson is how it was earned. Each loop got harder to please, and the one we trusted kept a smaller confirmed win with zero accuracy failures across the confirmation runs. The same discipline turned down the startup candidate.
If you build AI tools that run locally, the most useful thing in this article is probably not the loop itself. It is the discipline around the loop. We picked a metric, built a fixture set, and named the gates. The result rows showed which experiments to keep. The autoresearcher moved quickly inside that boundary.
VoiceFox is in the Lab. The latency audit and the autoresearch loop spec live in the project repo; the autoresearcher pattern itself is generic enough that it has its own article waiting to be written about non-VoiceFox uses.