--- title: Digests at Scale: Keep the Digest Read, the Raw File Append-Only description: How to stop an ever-growing rules log from decaying into noise: a short digest that is actually re-read every session, plus an append-only raw file holding full entries, cross-linked by stable line pointers. track: contributed url: https://www.chiefofstaff.io/guides/contributed/digests-at-scale published: 2026-09-06 contributor: Carmen Ashley (owner: Frederic Haddad) outcome: split any append-mostly record into a short digest it re-reads every session and an append-only raw file it can follow a stable line pointer into, and write both halves in the same commit. source: chiefofstaff.io --- # Digests at Scale: Keep the Digest Read, the Raw File Append-Only How to stop an ever-growing rules log from decaying into noise: a short digest that is actually re-read every session, plus an append-only raw file holding full entries, cross-linked by stable line pointers. ## The problem A well-run vault has rules that are read at every session start. Two of the most important ones are the **feedback log** — every correction the principal has ever given, so the same mistake is never made twice — and the **decisions record** — one entry per architectural decision, with context and a "reverse if" clause. Both have a growth problem. They are append-mostly by nature: corrections and decisions only accumulate. In a live vault, the feedback log went from a few hundred bytes to ~24 KB (over 500 lines) in about ten days; the decisions file grew to 30+ entries at roughly one new entry every two days. Both files were read in full into every session and every background worker's context. The failure mode is subtle: **an unread rule is a broken rule.** The moment the reading list gets long enough that the assistant skims it — or the prompt builder grips it with a fragile grep — lessons stop being applied. Worse, the reading cost is paid on every single worker run and every conversational round-trip, so the log's success (it's growing because the system is learning) becomes the system's tax. The two options usually on offer — keep everything (slow, unread) or trim entries (lossy, and you've just deleted provenance) — are both wrong. ## The philosophy behind the choice Split every accumulating record into two files with two different jobs: - A **digest**: short, human-readable, grouped by theme, sized to be *actually re-read in full every session*. Target orders-of-magnitude: a few dozen lines, not hundreds. - A **raw file**: append-only at the end, holding every full entry forever. Nothing is ever edited or deleted there. The two are cross-linked with stable pointers: each digest line carries a `[[raw-file#heading]]` wiki-link plus a `raw-file.md:` pointer into the raw file. The constitution (or whatever session-start instruction file drives the system) reads **only the digest**, with the standing rule that the full entry must be read via its pointer whenever the nuance matters — and the digest says so explicitly. Two properties make the split durable: - **Append-only raw keeps pointers stable.** Because new entries are added only at the end and existing text is never edited, every `raw:` pointer written into the digest stays valid forever. The moment you allow edits or reordering, line pointers rot and the cross-linking silently breaks. - **Digest freshness is a discipline, not a batch job.** Whoever records a new entry appends the full version to the raw file **and** adds or refreshes one digest line, **in the same commit**. Digest and raw can then never drift by more than one edit — and any drift is a spot-check away from detection. The deeper principle: a rule compounds only if it's loaded into context. Anything that makes the loading cheaper (short digest) while keeping the depth reachable (one pointer away) makes the whole system's judgment improve over time instead of decaying into noise. ## How it's wired at a high level Both real records in the vault adopted this pattern independently, each after outgrowing its single file — which is good evidence it's the general shape, not a one-off fix. **The feedback log.** `Efforts/feedback.md` became a compact do/don't digest (target ≤60 lines), grouped by theme, one line per lesson. Each line links to the full entry in `Efforts/feedback-raw.md` — an append-only file where new corrections land at the bottom. The write path is a single habit: record the correction fully in raw, add the one-line digest version, commit once. Compaction — merging near-duplicate lessons, promoting ones that recur three or more times into firmer rules — happens only inside a periodic review session, never on the hot path, so recording stays cheap and instant. **The decisions record.** `Atlas/decisions.md` became an ADR digest: one short block per decision — title, date and status, a one-sentence statement — with a link to `Atlas/decisions-raw.md` plus a `raw:` pointer. Full ADRs (context, rationale, the reverse-if clause) live in the raw file, append-only. Two wiring details are worth copying: - The worker prompt **inlines the digest verbatim**, replacing a fragile grep-based skim. Because the digest is short and deterministic, any change to it automatically invalidates resumed sessions via the prompt hash — so a new decision reaches every worker on its next run without anyone remembering to redistribute it. - The digest carries its own usage rule ("read the full ADR when the nuance matters"), which is what stops sessions from acting on one-liners and ignoring the reverse-if clauses. **Mechanics that make it cheap.** Nothing exotic is required: two Markdown files, one git commit per entry, and pointers that are plain text. The raw file's append-only property is enforced by convention ("correct the record with a new entry, never an edit") rather than tooling. If the system already has a search layer over the vault, the raw files are fully indexed — so full depth is one pointed read away even when the digest line wasn't enough. ## What a reader's chief of staff should be able to do afterwards After adopting this pattern, the chief of staff should be able to: 1. **Keep every lesson loaded.** Session-start reading stays bounded — a fixed, small digest — no matter how long the system has been running, so corrections keep applying instead of getting skimmed past. 2. **Write new entries correctly in one commit:** full entry appended to the raw file, digest line added or refreshed, both cross-linked with a stable `raw:` pointer. 3. **Navigate depth on demand:** when a digest one-liner isn't enough — a decision's reverse-if clause, the full story behind a correction — follow the pointer and read the exact range, then cite it. 4. **Compact safely on a schedule:** merge duplicate lessons and promote recurring ones during periodic review, never during triage, and spot-check digest-vs-raw consistency so the two can't drift unnoticed. The result is that the system's rules keep compounding — each correction making every future run slightly better — instead of the log becoming the very noise it was created to prevent.