---
title: Pointers, Not Copies: Semantic Search Inside the Vault
description: How to wire retrieval over a plain-file vault so the search layer returns file-and-line pointers, never prose answers — and why that distinction keeps the assistant honest.
track: contributed
url: https://www.chiefofstaff.io/guides/contributed/pointers-not-copies
published: 2026-09-10
contributor: Carmen Ashley (owner: Frederic Haddad)
outcome: run retrieval over its own vault that returns file-and-line pointers instead of prose answers, read the pointed range before stating a fact, and cite the location it actually read.
source: chiefofstaff.io
---

# Pointers, Not Copies: Semantic Search Inside the Vault

How to wire retrieval over a plain-file vault so the search layer returns file-and-line pointers, never prose answers — and why that distinction keeps the assistant honest.

## The problem

A small vault of plain Markdown files needs no search layer. At session start, the
chief of staff reads the few files that matter — a decisions digest, a feedback log,
a dashboard — and everything else is a few `grep` calls away. That works until it
doesn't. Notes accumulate: project files, meeting notes, daily logs, a growing
decisions record. Past a few dozen files, nobody can honestly say "I read everything
that's relevant." Retrieval stops being optional.

The standard fix is RAG-style: chunk the corpus, embed it, and let the model answer
questions directly from retrieved snippets. This is exactly the wrong shape for a
vault whose entire value is trustworthiness. Snippet-answering recreates a bug family
we already learned to fear with digests: **text that reads complete but isn't**. A
snippet blends sources, drops the caveat that appeared two paragraphs after the
excerpt, and misses an amendment made later in the file. The answer comes back
fluent, confident, and wrong — a silent lie, which is the most expensive kind,
because nobody knows to challenge it.

A vault run on plain files + git lives or dies by provenance: every fact carries a
source. An answer generated from retrieval snippets has no source. It has a query.

## The philosophy behind the choice

The rule is one sentence: **a pointer is not a source.**

The search layer's job is not to answer questions. Its job is to say, within a
second, "the passage most likely relevant to this is at `Atlas/decisions-raw.md:701-775`"
— a path plus a line range — and get out of the way. The assistant must then open the
file and read the range before stating any fact, and cite the file-and-line location
when it does.

Three sub-rules make this work in practice:

- **Capped output.** The tool returns at most ~8 pointers. This is a budget, not a
  limit: eight reads of ±20-line windows is a few kilobytes of context, versus the
  tens of kilobytes of full-file reads it replaces. Scarcity forces ranking quality.
- **Read the range, then extend it.** A matched passage can be cut off mid-sentence
  or sit near a note that was amended later. The assistant reads the pointed range,
  widens it when the passage is incomplete or the surrounding note may carry later
  edits, and only then claims the fact.
- **Never answer from the snippet.** The snippet text shown by search exists to
  decide *whether to read*, not *what to say*. The provenance rule — every fact gets
  a `` comment before it's written down — extends naturally to
  retrieval: every fact spoken from a search must name the file and lines it came from.

This costs one extra read per question. That cost is the point: the friction is
where honesty lives.

## How it's wired at a high level

The implementation (`tools/vault-search/`) is deliberately small — one Python file,
a thin CLI wrapper, an eval harness — sitting inside the vault repo it searches.

**Hybrid retrieval.** Every Markdown file is hashed and chunked. Chunks get two
representations: dense vectors from a self-hosted embedding model on an always-on
home machine (reachable through a local proxy; the API key lives outside the vault),
and a keyword index (SQLite FTS5, i.e. BM25). By default, ranking is vector-only,
because the principal's captures are paraphrase-shaped — spoken, casual questions
that share few keywords with the notes that answer them. When the query contains an
identifier-like token — a bead id, a reference number, a date, a full name — BM25
fuses into the ranking (reciprocal-rank fusion), because exact tokens should beat
semantic similarity for lookups. An offline eval over 38 real-style questions
justified the gating: paraphrased queries hit in the top-5 about 82% of the time
with embeddings versus roughly half with keyword search alone, and the gated hybrid
was best on every metric.

**Gated, with a retry queue — never a fallback.** Indexing runs in two places: a
post-commit hook (so a triage commit updates the index instantly) and a small
scheduled job every few minutes (so edits made directly in an editor, which never
touch git, get picked up too). Embedding requests are deliberately tiny — a few
chunks each, strictly sequential, with a per-request abort guard and bounded retries
— because one oversized batch can push the home machine into memory pressure. If the
embedding backend is unreachable, nothing fails: un-indexed files stay in a `pending`
state and remain findable by keyword, and the tool **says so** in its output
("vector search unavailable — N files pending"). There is no local fallback model,
on purpose: vectors from two different models cannot share one index, so a fallback
would quietly corrupt search quality rather than visibly degrade it.

**The index lives outside the repo.** Content hashes, chunks, FTS5 tables and
vectors sit in a cache directory, not in git. The chunk store is a plain-text copy
of the vault; the vectors are tens of megabytes and growing. Rebuilding is cheap
and scripted: hash every file, re-chunk and re-embed only what changed, drop deleted
ones.

**Scope is curated.** Atlas, Calendar, and Efforts are searched by default.
Delivery infrastructure and reports that hold other people's work data are indexed
but opt-in per query — they add noise, and noise in the pointer list is how the
assistant ends up citing the wrong file with total confidence.

A note for operators: the eval harness is part of the tool, not an afterthought.
After any change to chunking, the embedding model, or the fusion logic, a fixed
question set must stay at or above its measured hit-rate floor. Retrieval changes
should be measured, not vibes-reviewed.

## What a reader's chief of staff should be able to do afterwards

After handing this guide to their own system, the chief of staff should be able to:

1. Answer "what do we know about X" with **file-and-line citations** instead of
   confident paraphrase — every factual claim traceable to a range the assistant
   actually read, not a snippet it skimmed.
2. Use retrieval as **context discovery** — replacing multi-round `grep` and
   whole-file `cat` with ≤8 pointers plus targeted reads, cutting context cost per
   question substantially while improving the hit rate on paraphrased questions.
3. Run the same posture the production system runs: keyword-only degradation that
   *announces itself*, a retry queue that never silently substitutes a different
   model, and an index refreshed from the git-committed vault plus scheduled sweeps
   for out-of-band edits.
4. Say "I don't know" when the pointers don't actually contain the answer — the
   failure mode of snippet-answering is that it never leaves room for that sentence.