Running Your Chief of Staff on Local Models: Privacy Without a Datacenter

Most “AI assistant” setups ship your entire life to a third party: every message you send, every email it summarizes, every calendar entry it reads. If the assistant is a chief of staff — a system that holds your ventures, your family log, your finances — that’s a deal you should think hard about. The good news: you don’t need to accept it. Every heavyweight component of a personal chief-of-staff system can run on hardware you own, and the pipeline is actually simpler without vendor round-trips.

This guide abstracts how the reference system did it — the method, not the memoir — with concrete model picks for two very different budgets.

The components you need locally

A chief-of-staff pipeline has four model roles:

1. Reasoning/agent model — reads your inbox, decides what matters, drafts replies. This is the big one.
2. Embedding model — turns your notes into vectors so the assistant can find what you wrote months ago. Small, cheap, runs constantly.
3. Speech-to-text — voice notes in, transcripts out.
4. (Optional) TTS — for a voice front-end.

All four have excellent open-weight options. None need to leave your machine.

The pattern: a proxy in front of everything

The single most useful architectural trick: put one OpenAI-compatible proxy (the reference system uses LiteLLM) in front of all your local models, on a box that is always on, and reach it over a private network (Tailscale). Every consumer — a Telegram listener, a cron job, a web UI — talks to one endpoint with one API shape. Swap a model behind the proxy and nothing downstream changes.

The reference always-on box is a Mac Studio. Yours can be anything that stays powered: a Mini-PC, a gaming PC, a home server. What matters is: always on, one endpoint, private network only (bind the proxy to the tailnet interface, not 0.0.0.0).

Example stack 1: GLM as the reasoning model

The reference system runs GLM as the workhorse for triage — the high-frequency job where an assistant reads new incoming messages and decides: answer now, create a task, ask a clarifying question, or ignore.

Why GLM fits this role well:

• Cost per token when self-hosted is electricity. Triage is bursty — many small sessions per day — which is exactly where API bills hurt most and where a local model with a warm cache shines.
• Prompt caching matters more than raw speed. The reference triage prompt carries thousands of tokens of stable context (procedures, digest files, standing facts) that only change daily. A local deployment that keeps the model resident and supports KV/prefix caching makes the volatile tail the only expensive part. Schedule a cache warm-up at 00:05 and on stable-context change, so a real session never eats a full cold prefill.
• Agentic reliability — tool calls, exact-format replies — is what to bench before trusting a model with triage. Test with your actual procedure text, not a generic benchmark.

Serving: GLM’s open weights run under vLLM, LM Studio, or llama.cpp depending on hardware. Below is how to size it.

Example stack 2: Qwen, tiered by hardware

The Qwen family is the best example of “there’s a size for every wallet,” because the same lineage spans 4B to 400B+. Pick by RAM/GPU budget:

Tier A — modest hardware (a laptop with 16–32 GB RAM, no big GPU):

• Qwen3 4B–8B class, quantized (Q4/Q5 GGUF) runs on CPU or a small GPU through Ollama or llama.cpp. Expect a capable assistant at single-digit tokens per second — fine for batch jobs (nightly digests, weekly reviews), slow for interactive chat.
• Embeddings: a small embedding model (100M–1B class, e.g. a Qwen3-Embedding small variant or bge-family) runs comfortably on CPU. Embedding is where the 80/20 of “feels smart” lives: retrieval quality over model size.
• STT: whisper.cpp with a small/medium Whisper runs real-time on CPU.

Tier B — one good GPU (24–48 GB VRAM, e.g. RTX 3090/4090/5090):

• Qwen3 14B–32B quantized (AWQ/GPTQ/GGUF Q4) — a genuinely strong reasoning model at 30–80 tok/s. The 32B dense class is the sweet spot for an agentic chief of staff on a single consumer GPU.
• Embeddings: the 4B–8B embedding tiers run alongside with room to spare; they’re fast enough to re-index a personal vault continuously.
• STT: Whisper large-v3 at faster-than-real-time.

Tier C — the “enough RAM to be ridiculous” case (unified memory boxes, 128–512 GB):

This is where the reference system sits: an M3 Ultra Mac Studio with 512 GB unified memory. Unified memory is the quiet superpower here — the GPU can address all of it, so “fits in RAM” = “fits in VRAM”:

• Frontier-scale MoE models run locally. A ~235B-parameter mixture-of-experts (Qwen3-235B-A22B class) needs ~130 GB at Q4 — trivially resident — and MoE sparsity keeps decode speed usable (~20 tok/s on the M3 Ultra). You get near-frontier reasoning without any network egress.
• Everything runs simultaneously: the big reasoner, the embedding server (8B-class, continuously indexing the vault), Whisper large-v3 for every voice note, plus image generation — all resident, all behind the one proxy.
• The same logic scales down: a 128 GB Mac mini comfortably holds a 70B-class dense or a mid-size MoE plus the small models.

The rule of thumb across tiers: biggest model that stays fully resident, quantized to Q4–Q6, with the small specialist models (embeddings, STT) always co-resident.

The privacy math

What actually leaves your machine in a fully local setup: nothing. Voice notes transcribe on your box, notes embed on your box, the reasoning model never sees a network route it didn’t make to your proxy. Two honest caveats:

1. Don’t tunnel your proxy to the public internet. Private overlay network (Tailscale/WireGuard) or LAN only. The convenience of “reach my models from anywhere” must not become “anyone can reach my models.”
2. Secrets stay out of the repo. API keys for your own proxy live outside the codebase, in config with restrictive permissions. Your own infrastructure deserves the same hygiene you’d demand from a vendor.

Where to be pragmatic

Local-first does not mean local-only. The reference system deliberately keeps a hosted frontier model as a last resort for the rare task where local quality isn’t enough — and routes to it explicitly, per task, so the default path stays on-prem. The architecture (one proxy, one endpoint) makes that a routing decision, not a redesign.

Checklist to replicate

• One always-on box; install Ollama / LM Studio / vLLM
• LiteLLM (or similar) proxy on top, bound to the private network
• Reasoning model sized to your tier (GLM or Qwen class), kept resident
• Embedding model + a vector index over your notes; re-index on change
• Whisper for voice; wire it into your capture channel
• Cache warm-up job for your stable system prompt
• Secrets in ~/.config/<app>/ with 600 perms, never in git
• Backups: snapshot the notes and the config; models are re-downloadable

The result: an assistant that knows your life, runs while you sleep, and costs you nothing per message — and tells no one anything.

#local-models · #privacy · #self-hosted · #glm · #qwen