--- title: Running Your Chief of Staff on Local Models: Privacy Without a Datacenter description: A local GLM/Qwen/Whisper stack behind one LiteLLM proxy on an always-on box — the four model roles a chief-of-staff pipeline needs, tiered by the hardware you actually have. track: contributed url: https://www.chiefofstaff.io/guides/contributed/running-on-local-models published: 2026-09-10 contributor: Carmen Ashley (owner: Frederic Haddad) outcome: serve every model role — reasoning, embedding, speech — from hardware you own behind one OpenAI-compatible proxy on a private network, sized to the RAM budget you actually have. source: chiefofstaff.io --- # Running Your Chief of Staff on Local Models: Privacy Without a Datacenter A local GLM/Qwen/Whisper stack behind one LiteLLM proxy on an always-on box — the four model roles a chief-of-staff pipeline needs, tiered by the hardware you actually have. Most "AI assistant" setups ship your entire life to a third party: every message you send, every email it summarizes, every calendar entry it reads. If the assistant is a chief of staff — a system that holds your ventures, your family log, your finances — that's a deal you should think hard about. The good news: you don't need to accept it. Every heavyweight component of a personal chief-of-staff system can run on hardware you own, and the pipeline is actually *simpler* without vendor round-trips. This guide abstracts how the reference system did it — the method, not the memoir — with concrete model picks for two very different budgets. ## The components you need locally A chief-of-staff pipeline has four model roles: 1. **Reasoning/agent model** — reads your inbox, decides what matters, drafts replies. This is the big one. 2. **Embedding model** — turns your notes into vectors so the assistant can *find* what you wrote months ago. Small, cheap, runs constantly. 3. **Speech-to-text** — voice notes in, transcripts out. 4. **(Optional) TTS** — for a voice front-end. All four have excellent open-weight options. None need to leave your machine. ## The pattern: a proxy in front of everything The single most useful architectural trick: put **one OpenAI-compatible proxy** (the reference system uses LiteLLM) in front of all your local models, on a box that is always on, and reach it over a private network (Tailscale). Every consumer — a Telegram listener, a cron job, a web UI — talks to one endpoint with one API shape. Swap a model behind the proxy and nothing downstream changes. The reference always-on box is a Mac Studio. Yours can be anything that stays powered: a Mini-PC, a gaming PC, a home server. What matters is: **always on, one endpoint, private network only** (bind the proxy to the tailnet interface, not 0.0.0.0). ## Example stack 1: GLM as the reasoning model The reference system runs **GLM** as the workhorse for triage — the high-frequency job where an assistant reads new incoming messages and decides: answer now, create a task, ask a clarifying question, or ignore. Why GLM fits this role well: - **Cost per token when self-hosted is electricity.** Triage is bursty — many small sessions per day — which is exactly where API bills hurt most and where a local model with a warm cache shines. - **Prompt caching matters more than raw speed.** The reference triage prompt carries thousands of tokens of stable context (procedures, digest files, standing facts) that only change daily. A local deployment that keeps the model resident and supports KV/prefix caching makes the volatile tail the only expensive part. Schedule a **cache warm-up** at 00:05 and on stable-context change, so a real session never eats a full cold prefill. - **Agentic reliability** — tool calls, exact-format replies — is what to bench before trusting a model with triage. Test with your actual procedure text, not a generic benchmark. Serving: GLM's open weights run under vLLM, LM Studio, or llama.cpp depending on hardware. Below is how to size it. ## Example stack 2: Qwen, tiered by hardware The Qwen family is the best example of "there's a size for every wallet," because the same lineage spans 4B to 400B+. Pick by RAM/GPU budget: **Tier A — modest hardware (a laptop with 16–32 GB RAM, no big GPU):** - **Qwen3 4B–8B class, quantized (Q4/Q5 GGUF)** runs on CPU or a small GPU through Ollama or llama.cpp. Expect a capable assistant at single-digit tokens per second — fine for batch jobs (nightly digests, weekly reviews), slow for interactive chat. - **Embeddings:** a small embedding model (100M–1B class, e.g. a Qwen3-Embedding small variant or bge-family) runs comfortably on CPU. Embedding is where the 80/20 of "feels smart" lives: retrieval quality over model size. - **STT:** whisper.cpp with a small/medium Whisper runs real-time on CPU. **Tier B — one good GPU (24–48 GB VRAM, e.g. RTX 3090/4090/5090):** - **Qwen3 14B–32B quantized** (AWQ/GPTQ/GGUF Q4) — a genuinely strong reasoning model at 30–80 tok/s. The 32B dense class is the sweet spot for an agentic chief of staff on a single consumer GPU. - **Embeddings:** the 4B–8B embedding tiers run alongside with room to spare; they're fast enough to re-index a personal vault continuously. - **STT:** Whisper large-v3 at faster-than-real-time. **Tier C — the "enough RAM to be ridiculous" case (unified memory boxes, 128–512 GB):** This is where the reference system sits: an M3 Ultra Mac Studio with 512 GB unified memory. Unified memory is the quiet superpower here — the GPU can address all of it, so "fits in RAM" = "fits in VRAM": - **Frontier-scale MoE models run locally.** A ~235B-parameter mixture-of-experts (Qwen3-235B-A22B class) needs ~130 GB at Q4 — trivially resident — and MoE sparsity keeps decode speed usable (~20 tok/s on the M3 Ultra). You get near-frontier reasoning without any network egress. - **Everything runs simultaneously:** the big reasoner, the embedding server (8B-class, continuously indexing the vault), Whisper large-v3 for every voice note, plus image generation — all resident, all behind the one proxy. - The same logic scales down: a 128 GB Mac mini comfortably holds a 70B-class dense or a mid-size MoE plus the small models. The rule of thumb across tiers: **biggest model that stays fully resident, quantized to Q4–Q6, with the small specialist models (embeddings, STT) always co-resident.** ## The privacy math What actually leaves your machine in a fully local setup: nothing. Voice notes transcribe on your box, notes embed on your box, the reasoning model never sees a network route it didn't make to *your* proxy. Two honest caveats: 1. **Don't tunnel your proxy to the public internet.** Private overlay network (Tailscale/WireGuard) or LAN only. The convenience of "reach my models from anywhere" must not become "anyone can reach my models." 2. **Secrets stay out of the repo.** API keys for your own proxy live outside the codebase, in config with restrictive permissions. Your own infrastructure deserves the same hygiene you'd demand from a vendor. ## Where to be pragmatic Local-first does not mean local-only. The reference system deliberately keeps a hosted frontier model as a **last resort** for the rare task where local quality isn't enough — and routes to it explicitly, per task, so the default path stays on-prem. The architecture (one proxy, one endpoint) makes that a routing decision, not a redesign. ## Checklist to replicate - [ ] One always-on box; install Ollama / LM Studio / vLLM - [ ] LiteLLM (or similar) proxy on top, bound to the private network - [ ] Reasoning model sized to your tier (GLM or Qwen class), kept resident - [ ] Embedding model + a vector index over your notes; re-index on change - [ ] Whisper for voice; wire it into your capture channel - [ ] Cache warm-up job for your stable system prompt - [ ] Secrets in `~/.config//` with 600 perms, never in git - [ ] Backups: snapshot the notes and the config; models are re-downloadable The result: an assistant that knows your life, runs while you sleep, and costs you nothing per message — and tells no one anything.