Spend Tokens Like Money: The Cost Redesign of a Pipeline

The problem

An always-on assistant bills by the token, around the clock. It doesn’t feel that way at first — a single session costs pocket change, and the novelty of watching it work hides the meter. Then you do a usage audit and discover the truth: over a couple of days of normal use, the pipeline had run a little over a hundred spawns and burned an amount of money that, annualized, put it in the same category as real line items — one of the largest recurring costs of the entire system.

Worse, most of it bought nothing. Roughly 40% of the spend was pure ceremony: startup rituals, re-reading context the previous session had just read, re-deriving state that already lived in a file. Another 30% was chain-compounding — the cost of a session that has been resumed several times in a row, where the context grows with every hop and each additional turn re-pays the full accumulated bill for identical work. The actual judgment — classifying a message, filing a fact, deciding what needs a reply — was a small fraction of the total.

The default design that produces this is seductive: take your strongest model and run it on every step of the pipeline. It’s simpler to wire, and every step is as smart as it can possibly be. But “every step as smart as possible” is not what the work demands, and it prices routine traffic like frontier research.

The philosophy behind the choice

Three principles carried the redesign.

Route by judgment need, not by step. The pipeline’s steps are not equal. Triaging an inbox capture — classifying it, routing it to the right place, writing a plain acknowledgement — is a pattern-recognition task with rails around it: provenance rules, an allowlist, git as the audit trail. That task does not need the strongest model; it needs a cheap fast model that follows instructions reliably, operating inside guardrails designed for a weaker worker. Reasoning-heavy work — implementing something new, untangling a subtle failure — gets a stronger model in a fresh session. The routing question is not “which step is this?” but “does this step require judgment that pays for itself?”

Determinism belongs in scripts. Anything that is a fixed procedure — capturing a prompt shape, hashing a context prefix, scheduling a job, moving an archived file — should be a plain script, not a model turn. Every operation you move from a model call to a script is a line item that disappears from the bill and a failure mode that disappears from the system. Models are for the parts where language meets ambiguity; everything else is code.

Repeated tokens should become a rounding error. Every session in this kind of system starts with a long, nearly identical head: the system prompt, the tool list, the standing instructions. Inference providers offer prefix caching for exactly this — if the stored prompt is an exact prefix of the new request, the server reuses the computed cache and the prefill becomes nearly free. But the cache only pays off if the tail of the prefix is stable. If your prompt ends with volatile context — today’s dashboard, a git log, a task list — every run differs at the very end and the cache misses entirely. The fixed head and the volatile tail must be deliberately separated.

How it’s wired at a high level

The redesign landed as three moves.

Move one: model routing by task class. The routine triage worker — the thing that reads incoming captures and files them — runs on a cheap fast model at low effort. The standing rails do the safety work: writes only land in the vault through git commits, anything outbound needs per-item approval, and the worker’s authority is explicitly limited. Build work — anything that implements something new — is never executed inside the triage session. The triage worker writes a short brief and spawns a fresh session with a stronger model and a clean context for the actual implementation. That split also caps context growth: the triage session never becomes a 150k-token ballast of half-finished build state.

Two structural tweaks came with it. Rapid-fire messages get debounced into a single worker run instead of one session per message, and the resume chain is capped — after a few chained bursts, state is carried in files, not in a growing conversation context. Conversational memory fading after three hops sounds like a loss; in practice files carry state better than context does, because files are auditable.

Move two: mechanical steps in plain scripts. Each worker run starts with one collapsed startup command instead of a ceremony of re-reads. The stable part of the session context is built once by a shared script. Capture files are written per batch and archived once complete, so no worker re-reads the day’s whole history. None of this involves a model call.

Move three: cache warm-up for the fixed head. Measured on a local inference setup, every fresh session paid a full cold prefill — tens of thousands of tokens, half a minute of wall time — because the prompt ended in volatile context and the exact-prefix cache never hit. The fix: a script captures the live system prompt and tool list (a throwaway one-token reply through a temporary logging proxy, leaving global config untouched), then sends that same prompt shape straight to the inference endpoint with a one-token reply, so the server stores a cache entry ending exactly at the stable boundary. The volatile tail — today’s state — is appended after it. The warm-up runs shortly after local midnight (the prompt contains a date line that flips then) and before any fresh session when the stable prefix hash changes, under a lock so it never overlaps a real worker. The runner logs each session’s first-step cache read and warns if a fresh session reads zero.

The result: the same user-facing behavior at a fraction of the cost — the reduction was projected at 80% before the caching work, and the caching then removed the largest remaining fixed cost per session.

What your chief of staff should be able to do afterwards

Hand this guide to your system, and afterwards it should be able to:

• Audit its own spend: enumerate pipeline spawns over a recent window, classify where tokens went (ceremony, chain-compounding, real judgment), and report the breakdown with numbers.
• Route models by task class: maintain an explicit mapping — which steps run on the cheap fast model, which spawn a stronger model in a fresh session — and re-propose the mapping when the work changes.
• Cap context growth: debounce rapid inputs into single worker runs, cap resume chains, and carry state in files rather than an ever-growing conversation.
• Separate stable from volatile prompt content: build the session prompt so the fixed head is byte-identical across runs, with volatile state appended after it.
• Warm the prefix cache: capture the live prompt shape, issue a one-token warm-up call before fresh sessions (at midnight and on any change to the stable prefix), under a lock, and alert when a fresh session’s cache read is zero — the tell that the cache semantics changed underneath you.

The system does exactly what it did before. The invoice is an order of magnitude smaller.

#cost · #model-routing · #caching · #pipelines · #architecture · #ai-ops