--- title: Local Voice Infrastructure: STT and TTS That Never Leave Your Machine description: Run streaming speech-to-text, speech synthesis, and even image generation on an always-on home machine bound to the private network, behind one small internal API surface — so voice becomes a first-class on-prem capability instead of a cloud API bill. track: contributed url: https://www.chiefofstaff.io/guides/contributed/local-voice-infrastructure published: 2026-09-10 contributor: Carmen Ashley (owner: Frederic Haddad) outcome: run speech-to-text and synthesis on an always-on machine behind one private-network endpoint, under a written per-capability voice allowance and an honest degradation path. source: chiefofstaff.io --- # Local Voice Infrastructure: STT and TTS That Never Leave Your Machine Run streaming speech-to-text, speech synthesis, and even image generation on an always-on home machine bound to the private network, behind one small internal API surface — so voice becomes a first-class on-prem capability instead of a cloud API bill. ## The problem Cloud speech APIs have made voice frictionless: one SDK call and your app transcribes or speaks in a dozen languages at startling quality. The cost isn't the invoice. It's that **every utterance leaves the house.** In a chief-of-staff system, voice is not a novelty channel — it's the principal's highest-bandwidth input. Voice notes captured on the move, dictation inside review surfaces, spoken replies to briefs, even a spoken front-desk conversation with the assistant. What gets said into a microphone is intent, context, names of family members, amounts, frustrations. It is among the most private data in the entire system — more sensitive than the mail mirror, because nobody composes a voice note for an audience. Accepting cloud speech APIs means every one of those utterances transits a third party under someone else's retention policy, and cannot be taken back. There's a second, quieter problem: sprawl. Each voice capability added independently — a transcription API here, a synthesis API there — accumulates its own credentials, its own retention policy, its own failure modes. Voice ends up as five subscriptions instead of one capability you own. ## The philosophy behind the choice **Voice belongs on your own silicon.** Consumer hardware has crossed the threshold: an Apple Silicon machine or a workstation GPU runs both a large speech-recognition model and a competitive synthesis model in real time. The choice, then, is to run streaming speech-to-text and speech synthesis on an **always-on home machine**, bound to the private network only. Audio goes from the phone or laptop to a box you own, over a network you own, and stops there. **One API surface, many capabilities.** Rather than wiring each feature to its own endpoint, every voice capability is served behind one small internal API surface: transcription, streaming recognition, synthesis — and, while you're at it, local image generation, which is the same posture and the same GPU. Each consumer (the messaging capture path, the review web app, a spoken voice front-end) points at the shared endpoint. Model upgrades, language configuration, and quality fixes land in exactly one place. **The network bind is the auth.** Internal services like these are exposed only on the private overlay network, never publicly. For internal model servers that don't offer real authentication, the perimeter *is* the security posture — and that's acceptable precisely because nothing is reachable from outside. Where an upstream component has an insecure default (a hardcoded shared token, say), don't pretend it's a lock: document that the network bind carries the trust, and keep the bind private. **Every capability gets an explicit allowance.** Not all outputs are equal. Synthesis voices, in particular, deserve per-capability rules: in this design, voices are reference-audio clones built from a small set of locally stored recordings rather than a library of stock voices — better (the assistant sounds like a person you chose) and heavier (licence and provenance obligations attach to the reference material). The rule: each capability that can *speak or render* has a written allowance stating which voice it uses, where its reference audio lives, and what licence consequence follows. A non-commercial synthesis licence, for example, can never front a commercial product — write that down when you install the model, not when a lawyer asks. **Own the failure mode.** The always-on machine becomes a dependency of capture itself. That trade is made explicitly: when transcription is unreachable, the capture path degrades to an honest placeholder (the note is stored with a marker, never silently dropped), and dictation buttons fail with a visible error over a working typed fallback. ## How it's wired at a high level **The services.** On the always-on machine: - **STT (authoritative).** A Whisper large-v3-class model served behind a small proxy, auto-detecting language or pinning it per config. In a two-engine setup, a fast streaming recogniser renders provisional words live (shown greyed, never forwarded downstream), while the authoritative model transcribes each completed utterance and supplies the final text — low latency and high accuracy together, at a measured cost of tens of milliseconds per turn. - **TTS.** A synthesis model chosen on measured first-audio latency and human judgement of voice quality, streaming natively; the previous engine's service definition is kept as a one-line rollback. Voices are reference-audio clones, with recordings stored on the same machine and the licence consequence recorded. - **Image generation.** A diffusion model driven by a small script over SSH — free, local, private-network-only — with outputs landing in the vault's assets folder and a rate discipline (a single image is a probe; batches need the principal's go-ahead) so it never starves the model servers that matter more. **Binding and supervision.** Each model server runs as a supervised service (launchd or systemd, keep-alive enabled) with a log file and a crash path defined from day one — three new resident processes means three new things to monitor. Everything binds to the private-network interface only. Memory is budgeted honestly: a resident LLM plus the voice models must fit together, with a memory-pressure check deciding what stays resident versus loads on demand. **The unified surface.** A thin proxy presents the OpenAI-compatible endpoints for STT and exposes synthesis and image capabilities alongside it. Consumers read their key from a config directory at request time. If the authoritative STT doesn't answer within a small budget, the consumer falls back to the live text and marks the turn accordingly — degradation is per-request and visible, never a crash. **The honest trade-offs.** Setup is real work: building model servers for your platform, tuning streaming behaviour, wiring the fallbacks. Model quality trails the best cloud offerings at the frontier — hybrid setups exist precisely to buy back accuracy where it matters. There's GPU contention to manage, one box whose uptime the capture path now assumes, and licences (some synthesis weights are non-commercial) that constrain future reuse. The counterweight: zero per-utterance cost, zero retention exposure, and voice that keeps working when the internet doesn't. ## What your chief of staff should be able to do afterwards - **Transcribe anything, on-prem.** Voice notes, dictation, meeting audio — recognized by your own models, on your own network, with no third party in the loop. - **Speak with a voice you chose.** The assistant's replies are synthesized locally from reference-audio voices, under a written per-capability allowance covering voice, provenance, and licence. - **Point every feature at one surface.** Capture, review dictation, a spoken front-end, image generation — all consumers of one internal API, upgraded and debugged in one place. - **Degrade honestly.** When the voice box is down, captures become marked placeholders and dictation fails visibly — nothing is lost silently, nothing pretends to work. The measurable outcome: voice becomes a first-class, on-prem capability of the system — measured in first-audio latency and transcription accuracy on *your* voice, owned outright, and priced at zero marginal cost per utterance.