The engineering, not the pitch.

Two transcription engines, both fully local

Plainsay ships two on-device speech engines through Core ML, and you pick which one runs:

• Whisper, via WhisperKit — OpenAI's model family, broad language coverage, running as a Core ML model on the Neural Engine.
• NVIDIA Parakeet TDT 0.6B v3, via FluidAudio — faster, and in our own benchmark measurably more accurate on English, with real Polish coverage most competitors don't market at all.

The Parakeet integration has a couple of specific choices worth stating plainly:

• The encoder loads at int8 precision (AsrModels.downloadAndLoad(version: .v3, encoderPrecision: .int8, …)) — smaller download, faster inference, negligible accuracy cost for this task.
• Mel-context carry-over is disabled (ASRConfig(melChunkContext: false)), on FluidAudio's own recommendation for multilingual v3 recordings longer than one model window — without it, later chunks of a Polish sentence measurably drift toward English as the model's context window fills with the wrong language's statistics.
• Decoder state is fresh for every single utterance — a new TdtDecoderState per transcribe() call, deliberately never reused across dictations, so nothing you said in one dictation can leak linguistic context into the next.

Which model to default to isn't hardcoded, either. OnDeviceModel.recommended(for:) takes the languages you actually speak (set once in the Setup Assistant) and checks them against FluidAudio's own Language.allCases — Parakeet's real, current language coverage, not a table we maintain by hand and let go stale. Speak a language outside that list and the recommendation falls back to Whisper automatically.

Live typing: words landing as you speak, then correcting themselves

The hard constraint: the only editing primitive available to a program typing into an arbitrary focused text field in someone else's app is Backspace. No cursor movement, no select-and-replace, no diff-aware text replacement API that works in Slack, VS Code, and Safari's address bar alike.

Every 1.5 seconds while you're recording, Plainsay re-transcribes the entire growing audio buffer from scratch — not incrementally — through whichever engine is loaded. That's a real, deliberate choice: streaming/incremental decoding is faster per-step but the model can't revise an earlier guess once it's committed, and ASR models frequently do want to revise an earlier word once more audio gives them more context. A fresh full-buffer decode every tick costs more compute; it buys the model the right to change its mind.

With Live typing on, each of those passes doesn't just update a HUD preview — it gets reconciled onto the actual document:

old = what's already typed on screen (our own tracked model of it)
new = this pass's fresh transcript

find the longest common prefix of old and new
  → delete (old.count − prefix.count) characters via Backspace
  → type (new.count − prefix.count) new characters

liveTypedText = new   // this becomes the baseline for the next pass

A pure longest-common-prefix diff, not a full edit-distance alignment — deliberately. Backspace can only ever erase a suffix, so the only diff worth computing is "how much of the tail changed," and the assumption is that ASR revisions overwhelmingly land near the end of what's been said so far, not the beginning.

The bug this shipped with, and the fix

The reconciliation step above sends every Backspace as a synthetic CGEvent keydown/keyup pair. The first version of this sent them back-to-back with zero delay between keystrokes. It worked perfectly in every manual test — and then a real user reported wrong capitalization and missing spaces showing up "very often."

The actual bug: deletion is fire-and-forget. Nothing confirms a synthetic Backspace was actually consumed by the target app, and a busy app (a web view mid-render, an Electron app servicing its own event loop) can silently drop or coalesce a keystroke sent with no gap after it. Drop one Backspace, and Plainsay's internal model of what's on screen — liveTypedText, the baseline the next diff is computed against — is now permanently one character wrong for the rest of that dictation. Every subsequent correction compounds the error instead of fixing it.

The fix: pace the Backspace events the same way the paste path was already paced — a short, deliberate delay between each one, mirroring a lesson learned earlier for ⌘V itself (a busy app needs time to actually service a synthetic keystroke, full stop). Shipped and verified the same day it was reported.

Live typing is off by default, and turning it on turns off the cleanup pass below — reconciling live-typed raw words against a rewritten sentence would mean visibly rewriting text you just watched yourself type, which defeats the point.

Cleanup: best-effort, and never the reason a dictation is lost

Raw ASR output is grammatically honest but reads like speech: filler words, false starts, no punctuation. The optional cleanup pass turns that into written text via an LLM call — by default Gemini Flash Lite, but switchable to your own key, a different provider, a local model, or off.

The design constraint that actually matters: a failed cleanup call must never cost you the dictation. The call has a 3-second timeout; no key, no network, a timeout, or an HTTP error all fall back to inserting the raw transcript instead, and the HUD says so. TextCleaning is a protocol with one method — the whole pipeline doesn't know or care whether the concrete implementation is Gemini, OpenAI-compatible, Anthropic, or a no-op passthrough when cleanup is disabled.

Getting text into someone else's app

Text lands via the pasteboard and a synthetic ⌘V, not Apple's Accessibility text-insertion API. The Accessibility route is cleaner in theory and fails silently in practice — in Electron apps, web views, and most terminals, which cover a large share of where people actually dictate. Pasting works everywhere a human can paste.

The cost is briefly borrowing the real clipboard: your existing clipboard contents are snapshotted before the write and restored after, so a dictation never permanently clobbers whatever you had copied a moment ago.

Read the rest

Why on-device — the privacy case, stated precisely. Benchmark — WER and latency for both engines, reproducible with one command. Source — all of the above, every line, MIT licensed.