---
title: Giving Your Chief of Staff a Phone: The Outbound Voice Agent
description: How to stand up a voice agent that places outbound phone calls on your behalf — the credential, the persona and disclosure rules, the system tools it actually needs, and the dial script that refuses to run without your approval.
track: contributed
url: https://www.chiefofstaff.io/guides/contributed/the-outbound-voice-agent
published: 2026-09-10
contributor: Carmen Ashley (owner: Frederic Haddad)
outcome: stand up an outbound voice agent that cannot dial without a per-call approval line — with an enumerated tool list, dashboard-owned guardrails and a one-action kill switch.
source: chiefofstaff.io
---

# Giving Your Chief of Staff a Phone: The Outbound Voice Agent

How to stand up a voice agent that places outbound phone calls on your behalf — the credential, the persona and disclosure rules, the system tools it actually needs, and the dial script that refuses to run without your approval.

Most of what a chief-of-staff system does is text. Mail, tasks, notes, briefs — all of it composable, reviewable, revocable. Then you hit the class of errands that only moves on the phone: confirming a part is in stock, chasing a vendor who doesn't answer email, asking a restaurant whether the terrace is open on Tuesday. These are low-judgement, high-friction calls. They eat an afternoon and produce four facts.

They are also the single most exposed thing your assistant can do. A voice call speaks live, in your name, to a stranger, with no draft to approve mid-sentence. So the design problem isn't "how do I get an LLM onto a phone line" — that part is nearly a solved product. The design problem is **how to make an outbound call impossible without your explicit, per-call approval**, and how to make everything the agent says on the line traceable to something you wrote down.

This guide covers the setup half: the credential, the agent configuration, the persona and disclosure rules, and the dial script. A companion guide, [The Call Brief](/guides/contributed/the-call-brief), covers the brief format and the operating rules that govern each individual call.

## What you need

- **A conversational-voice platform with an outbound-call API.** The reference system uses ElevenLabs Agents; the shape below is generic enough to port. What you need from any candidate: per-call prompt override, a system-tool set (hang up, voicemail detection, DTMF), and a fetchable transcript keyed by conversation id.
- **A telephony number to dial from.** Typically imported from a telephony provider (Twilio and equivalents) into the voice platform. Note the caller-ID consequence up front: a US number calling a Gulf landline looks like spam, and some callees won't pick up.
- **A restricted API key, stored outside the repository.** Product scope limited to the agent product only, plus a monthly credit cap set in the provider dashboard. In the reference system it lives at a path under `~/.config/<system>/`, `chmod 600`, directory `700`. It never appears in chat, in the vault, or in git.
- **A small script, standard library only**, that talks to exactly one host: the voice platform's API. No SDK, no webhook server, no inbound listener.
- **A directory for call briefs**, one markdown file per call, inside the repo — because the brief is the auditable artifact.
- **A directory for transcripts**, in the immutable/log side of your vault.
- **A messaging channel back to you** for approvals and outcome reports.

## 1. Treat the phone as a credentialed operator, not a channel

Before any code: write a charter. In the reference system, outbound voice was the *first* credentialed operator ever created, and the charter that resulted is still the strictest in the roster. The charter states, in plain prose:

- **What it may do.** Place individually-approved outbound calls, and return each transcript. Nothing else. No inbound handling at all — an agent that can answer your phone is a different, larger risk surface with a different approval story.
- **What it may never do.** Negotiate terms, commit money, book, sign, or share any information not in the brief.
- **The one-way valve.** The operator executes approved briefs and reports. It never reads the knowledge base, never decides whom to call, never edits anything. The chief of staff composes; the human approves; the operator dials. That separation is what keeps a prompt-injected assistant from inventing a call target.
- **The kill switch, named explicitly.** One action, doable by you alone, in under a minute: disable the API key in the provider dashboard. Write down which single action it is, in the charter, before the first call.
- **What the charter knowingly does not satisfy.** If your provider's terms carry a clause you're choosing not to meet, quote it verbatim to the principal, get an explicit acceptance, and record it as *knowingly unmet* — with whose risk it is. A guide that pretends its compliance posture is spotless teaches nothing. (See §3.)

## 2. The agent's standing prompt: persona, conduct, mechanics

The agent has one standing system prompt into which each call's brief is injected. It has four blocks, and every block exists because something went wrong once.

**Persona.** Who the caller says they are, in one line — an assistant in the principal's office, calling on their behalf. Give the persona a name and stick to it. Details (a surname, the exact role) are answered *only if asked*, not volunteered; a caller who front-loads their CV sounds like a script.

**Disclosure — the part that deserves real thought.** You are choosing a position on a spectrum, and you should choose it consciously rather than by default:

- The agent must **never claim or imply it is human.** This is not negotiable and should be a hard line in your charter, not a prompt suggestion. If a system operator asks you to build a persona that denies being a machine, that's the point to refuse.
- Asked directly — "am I talking to a bot?" — it answers **truthfully in one short sentence** and continues with the task.
- Whether it *proactively volunteers* being an AI is a genuine trade-off. Volunteering derails a two-minute errand call into a conversation about AI. Not volunteering may conflict with your provider's terms (ElevenLabs' agent terms carry a proactive-disclosure clause) and, in some jurisdictions, with law. The reference system's principal was shown the clause verbatim, chose no-volunteer, and the gap is recorded in the decision record as knowingly unmet with the risk sitting with him. **Whatever you choose, document the choice and who owns the consequence.**
- **Recording disclosure is a separate obligation and is not optional** where you or the callee are. Many jurisdictions require all-party consent for recorded calls; if you archive transcripts and audio, you are recording. The reference system weaves it mid-sentence into the fixed opening — "…calling from [principal]'s office on a recorded line — " — rather than as a standalone announcement, which satisfies the obligation without the call feeling like a compliance script. Check your own jurisdiction; this is one of the few places where "we'll add it later" is a legal exposure rather than a backlog item.

**Conduct.** Stay strictly inside the brief. Never negotiate, commit money, book, sign, or share information beyond it. If asked for a commitment, say plainly that you can't decide that and the office will follow up, then close politely. This is the sentence that turns a runaway call into a logged escalation.

**Task completion.** Non-obvious, and the source of the most expensive early failure: models are agreeable. A callee who starts wrapping up will get a warm goodbye from an agent that never delivered the message. So the prompt states that the brief's task line and every listed question are *the reason the call exists*, that if the callee starts closing early the agent politely says there's one more thing, and that the call must not end with the task undelivered unless the callee refuses or hangs up.

**Mechanics.** Three rules learned live, each worth stealing:

- **A phone menu is not voicemail.** Listen to the options fully, press the one that serves the brief, stay silent while navigating, speak only when a human answers. If a menu loops twice, try `0` or `#`; if still stuck, hang up and report the exact options heard.
- **"Hold on, let me check" means wait silently.** Use the skip-turn tool; don't fill the pause. An agent that chatters over a callee looking something up reads as unbearable.
- **Language rules are explicit.** Open in one default language always, switch only when the brief says so or the callee clearly prefers otherwise — and if you support a second language, specify the *register*: spoken regional dialect versus formal standard is the difference between sounding like a person and sounding like a news broadcast.

## 3. System tools: the enumerated verb list

Grant tools deliberately; each is a capability, and the list *is* your exposure surface. The working set:

- **End call.** Sounds trivial. The very first test call couldn't hang up, and the agent stayed on a dead line after goodbyes. Grant it first.
- **Voicemail detection**, configured to leave **nothing** — detect, hang up, report. A machine leaving messages on strangers' voicemail is a different consent conversation than a machine having one.
- **DTMF / keypad tones**, for phone menus.
- **Skip turn**, for the silent wait.
- **Language detection**, if you run more than one language preset.
- **Warm transfer to the principal**, optional and worth it: when a callee explicitly asks to speak to you directly, the agent conferences you in — and, in the setup used here, first reads you a one-sentence summary of the call so far so you don't join blind. Gate it with a condition string: *only* when the person explicitly asks and it suits this brief.

Conspicuously absent from that list: anything that decides *whom* to call or *why*. Those don't exist as tools, so no amount of in-call persuasion can reach them.

## 4. Guardrails and turn-taking live in the dashboard, and your script must not stomp them

Two categories of configuration belong to the human, set in the provider's UI:

**Guardrails.** Topic focus, prompt-injection protection, and content moderation across the usual categories, with the trigger action set to **end the call**. Accept the consequence honestly: a guardrail trip hangs up on a real person mid-sentence. That is the preferred failure — better an abrupt goodbye than an agent steered off-brief by whoever is on the line.

**Turn-taking and speech recognition.** These get tuned against real calls, not guessed. The settings that mattered in practice: enable background-voice detection (a colleague talking near the callee was being mistaken for the callee's turn), a ~7-second turn timeout, high-quality streaming recognition, and an explicit list of backchannel words — *mm-hmm, uh-huh, okay, right, go on*, plus equivalents in your second language — that the agent does **not** treat as an interruption. Check whether your platform's ignore-list *replaces* or *merges with* its defaults; the reference system's replaces.

Now the operational trap. Your setup script sends section-replacing PATCHes. A PATCH that omits a field the human set in the dashboard can silently reset it. Two defences: (a) always send the *full* block you're touching, including fields you didn't change, and (b) keep dashboard-owned sections entirely out of the script's payloads, then verify after every setup run that they survived. Write in your notes which settings are dashboard-owned, and re-read them live from the API before assuming — a note about someone else's UI drifts.

One more platform quirk with a general lesson: per-call prompt injection requires the agent to *explicitly allow* overrides for those fields. The reference system discovered this from a live failure mid-call — *"Override for field 'first_message' is not allowed by config."* Enable the override allowances in setup, and treat your first real call as the integration test it actually is.

## 5. The dial script: four commands and a set of refusals

The whole surface is four verbs:

- **`setup`** — verify the key, resolve the phone number id, create the agent if missing, enforce the tool and override configuration, cache ids in a small state file.
- **`place <brief>`** — dial an approved brief.
- **`fetch <brief>`** — poll the conversation, archive transcript and audio, print a summary.
- **`watch <brief>`** — the detached follow-through (§6).

What makes `place` trustworthy is that it is mostly a wall of refusals, each one a scar:

1. **No approval line, no dial.** The brief carries an `approved:` field that starts as `PENDING`; a `PENDING` brief is refused outright. Approval is per-call and never standing.
2. **Already placed, no dial.** If the brief already carries a conversation id, it can never be placed again. A redial needs a *fresh* brief with a *fresh* approval — because from the callee's side, a second ring is a second contact.
3. **Malformed number, no dial.** Strict E.164 match, no cleverness.
4. **Opener that re-greets, no dial.** The system prepends a fixed intro, so an opener written as its own greeting plays doubled: *"Hi, this is [assistant] from [principal]'s office on a recorded line — Hi, it's [assistant]…"*. The script pattern-matches greetings in the opener and refuses.
5. **No task line, no dial.** Learned from a call where the payload lived only in the facts section and the call ended in five polite turns without it ever being spoken. If the call is pointless without a particular sentence, that sentence must be in the task line.

Refusals 4 and 5 are the ones worth copying verbatim in spirit: **the failures that cost you real calls should become mechanical refusals in the tool, not warnings in a document.** A document is read by a careful human at composition time. The script is read every single time.

`place` then dials with the brief injected as the per-call prompt and the opener injected into the fixed first message, writes the conversation id and a UTC timestamp back into the brief file, and spawns the watcher.

## 6. After the call: archive, then report

No webhook server. The dial spawns a **detached watcher** that polls the conversation until it ends, and:

- archives the transcript and the audio recording into the immutable side of the vault, one dated file per call;
- classifies an `outcome:` on every call — voicemail / no answer / transferred / completed — with loose string matching so a provider schema change degrades to "completed" rather than crashing;
- sends the audio straight to the principal's own chat as soon as it's archived, *before* any summarization runs;
- drops a note into the inbox so the normal triage path updates the linked task, captures learned facts with provenance, and summarizes the outcome.

Four details that make this reliable rather than merely working:

**Absolute paths and a real log file.** The first watcher died silently — relative path, stderr to `/dev/null`. An unattended process without a log is an unattended process you will one day be unable to explain.

**Stop early on a call that never connected.** Detect the never-connected state (non-terminal status, zero duration, no accepted timestamp), and after a few consecutive stuck polls, give up. Otherwise a call that never rang burns the entire watcher budget before reporting.

**A wall-clock timeout that reports honestly.** The reference watcher caps at 60 minutes and *says so* when it hits it. A silent hang must become a logged failure.

**"Watcher alive" ≠ "call in progress."** This one produced a real mistake: the system told the principal a call was ongoing, twice, based only on the process still appearing in `ps`. The process was stuck. For any is-X-happening question, query the actual source of truth — the conversation status from the API — never a proxy for it.

## 7. What you should be able to do afterwards

You should be able to say "call the appliance service and ask whether the part came in," see a written brief come back within a minute, reply *approved*, and get an audio recording plus a summary a few minutes later — with the task updated and the answer captured as a fact with provenance. You should also be able to say nothing at all, and be confident that nothing dialled.

That last property is the whole design. Everything above — the enumerated verbs, the dashboard-owned guardrails, the wall of refusals, the one-line kill switch — exists to protect one principle:

**The agent never dials without the human's explicit approval line for that specific call.**

Not a standing permission. Not a policy the model was asked to follow. A field in a file that a script refuses to run without.