# Your Agent Should Be in the Meeting, Not Reading the Notes After Meeting AI has converged on one shape: record, transcribe, summarize, deliver. Every product in the category ships the same artifact, a tidy digest that lands in your inbox after the conversation is over. It is genuinely useful and it is structurally too late. The moment you needed help was the moment somebody quoted a number at you and you could not remember whether it matched the number they quoted last month. This post is about the other shape. An agent that reads the transcript **while the meeting is running**, checks what it hears against everything you have ever written down, and says something when, and only when, it would change what you say next. In practice that lands at roughly four interruptions an hour. It is two Python files and one skill file. Everything you need to rebuild it is below. ## Why the Recorder Had to Change We ran Granola. It is a better product than what replaced it, and it is worth saying that plainly before explaining why we left. Granola recently closed the ecosystem to agents. The programmatic surface is their API now, on a paid tier. Everything else is shut: - The local caches are encrypted. `granola.db` opens with a random sixteen byte header, so SQLite refuses it outright. It is a database in name only from the outside. - The plaintext cache file that older builds kept is gone. - The Electron shell strips `--remote-debugging-port` at launch, so you cannot attach a Chrome DevTools session and read the renderer either. That is a completely defensible product decision, and if you are building a business tool you should probably make it. But it means an agent on the same machine, running as the same user, reading data the user paid for and generated, has no path in. Here is the part that matters even more than the lockdown, and would matter even if every door were open: **Granola has no live path at all.** The transcript exists on their servers after the meeting, not on your disk during it. For an archive that is fine. For a copilot it is fatal, because a copilot that speaks after the call is just a summary with extra steps. Wispr Flow was already on the machine for voice dictation. It shipped a meeting notetaker, and the notetaker writes here: ``` ~/Library/Application Support/Wispr Flow/meetings/<uuid>/ ├── live.ndjson transcript, appended as people speak ├── refined.ndjson server-cleaned text, ~2 min after the meeting ends └── upload.ogg compressed audio ``` Plain newline-delimited JSON, appended line by line, growing while people are still talking. No encryption, no API, no token to refresh, no rate limit. **The honest tradeoff.** Wispr's notetaker is worse than Granola today on the things Granola optimized for. Its summaries are thinner. Its post-meeting turnaround is slower... But the file on disk changes what is possible, and nothing else does. One vendor writes to your disk in the clear while the meeting happens, and one does not. That single property is worth more than every summarization improvement combined, because it is the difference between a product feature and a platform you can build on. One trap on the way in: **the transcript is never in Wispr's database.** `Meetings.content` stays NULL even on a finished call, and during a live meeting the row is a stub carrying an id and a timestamp and nothing else. The directory is the only source. ## What You Need - A meeting recorder that writes transcript text to local disk **as the meeting runs**. Wispr Flow is what we use. The pattern is not specific to it. Check your recorder before you design around it: open its application support directory during a live call and see whether anything is growing. - A coding agent with shell access and a persistent knowledge base. We run Claude Code inside an Obsidian vault, driven through Maestro. Any agent that can execute a command in a loop and search your notes will do. - Python 3. No dependencies. The knowledge base is not optional garnish. It is where all the value is. Reading a text file in real time is trivial and worth nothing on its own. Checking that text file against ten years of notes, while the other person is still mid-sentence, is the entire product. ## Two Tools, Two Jobs **The reader** answers "what is being said right now" once, on demand, when you type a question. **The watcher** streams continuously and speaks unprompted. Keep them separate. They have opposite constraints: one optimizes for latency on demand, the other for never returning empty. ## Latency Is the Feature The reader has one performance requirement: answer before the moment closes. | Path | Reads | Measured | |---|---|---| | default | one `live.ndjson`, tens of KB | **~40 ms** | | with speaker names | plus an APFS clone of the 1.42 GB database | ~240 ms | The default path never opens the database. It cannot afford to. Resolving the other participant's canonical name out of the app's speaker map costs five times the latency, and during a live call you already know who you are talking to. That is a flag, not a default. Parse defensively. The final line of a live ndjson file is routinely a partial flush, so a strict parser fails exactly when the data matters most: ```python def ndjson(path): """Parse an ndjson file, skipping unparseable lines.""" out = [] for line in path.read_text(errors="ignore").splitlines(): line = line.strip() if not line: continue try: o = json.loads(line) except ValueError: continue # partial flush mid-write; skip it if isinstance(o, dict): out.append(o) return out ``` The rule that goes with the reader matters more than the reader. **Never answer from memory of earlier in the session.** A meeting moves while you are typing, so anything the agent remembers about the call is stale by definition. Every question about "this call" starts with a fresh read, even if it read the file ninety seconds ago. Put that sentence in the skill, in capital letters if you have to, because a model with a plausible-looking answer in context will happily skip the read. ## Attribution Is a Hardware Fact, Not a Model Output Three signals in this data claim to tell you who spoke, and they disagree. Measured on one real 142 segment meeting: | Signal | Verdict | |---|---| | `speaker.source`, `mic` or `system` | **Ground truth.** mic accounted for 1,371 words, 64% of the meeting | | `speaker.name`, scraped from the conferencing app's DOM | **4 of 18** system segments wrongly labeled as the operator | | server-side diarization in `refined.ndjson` | claimed the operator spoke **4%** of the meeting | The refined diarization is off by a factor of fifteen against the hardware channel, and its very first segment puts both halves of a two person greeting on one speaker. So the resolver ignores names for attribution entirely: ```python def speaker_of(seg, other=None): """Resolve a segment to a speaker. HARDWARE CHANNEL WINS. mic -> the operator, unconditionally. A mic segment is the operator by construction, and no mic segment was ever observed mislabeled. system -> the other party's name if known; else the DOM-scraped name, but ONLY when it is not claiming to be the operator, because that specific case is the observed race (4 of 18). """ s = seg.get("speaker") or {} src, nm = s.get("source"), (s.get("name") or "").strip() if src == "mic": return OPERATOR if other: return other if nm and not nm.lower().startswith(OPERATOR.lower()): return nm return f"spk{s.get('id', '?')}" ``` Generalize it past this one app: a model's claim about who spoke is an inference, and the channel the audio physically arrived on is a fact. When you have both, use the fact. Any recorder that mixes microphone and system audio before transcribing has thrown the fact away permanently and has to guess forever. This has a direct product consequence. Before the agent says "they committed to X," it confirms X came from a `system` segment. Telling your operator that the other party promised something he actually said himself is the single most expensive failure mode in the system, because he will repeat it out loud. ## Streaming, Not Polling: Invert Who Waits The obvious build is a timer: wake every N seconds, read what is new, comment if warranted, stop. Do not build that. In tick mode most wakeups find nothing, so an agent that wakes thirty times and finds nothing twenty-six times learns to skim, and burns a turn on each empty check. Make **the tool block, not the agent**, and use exit codes as the loop's control flow: ```python def watch(max_wait, quiet_end, poll, min_new) -> int: """BLOCK until there is something worth looking at, then return. Exit codes are the loop's control flow: 0 new segments (printed) -> agent considers whether to speak 3 meeting ended -> agent stops looping and recaps 4 max_wait, still live -> agent immediately calls watch again """ st = session() d = meeting_dirs()[-1] if st["meeting"] != d.name: # A different meeting started. Reset rather than diff across meetings. st = {"meeting": d.name, "cursor": 0, "started": time.time()} save_json(STATE, st) cur = int(st.get("cursor", 0)) deadline = time.time() + max_wait while time.time() < deadline: segs = segments(d) new = segs[cur:] # Enough new speech to be worth the agent's attention. min_new exists so # a single "Mm-hm." does not wake it; a flagged segment always does, # because a dollar figure alone is worth a look. if len(new) >= min_new or any(flags_for(s.get("text", "")) for s in new): st["cursor"] = len(segs) save_json(STATE, st) for s in new: f = flags_for(s.get("text", "")) tag = f" <{','.join(f)}>" if f else "" print(f"[{s.get('timestamp','?'):>6}] {speaker_of(s):<18} " f"{(s.get('text') or '').strip()}{tag}") return 0 mtime = (d / "live.ndjson").stat().st_mtime # Meeting is over when the transcript stops growing. There is no end # marker, so silence is the only signal. if time.time() - mtime > quiet_end: print(f"MEETING ENDED (no writes for {int(time.time()-mtime)}s)") return 3 time.sleep(poll) print(f"STILL LIVE, no new speech in {max_wait}s") return 4 ``` The agent's loop is then three lines and never idles: ```bash python3 wispr_wingman.py --start # cursor at NOW python3 wispr_wingman.py --watch # blocks; returns on new speech python3 wispr_wingman.py --watch # again, and again, and again ``` Exit 0 means real speech arrived, consider it. Exit 4 means quiet but still live, go straight back in. Exit 3 means stop and write the recap. **Every batch handed back is genuinely new content**, so the agent never learns to skim. Verified against a fixture: `--watch` blocked for 8.29 seconds and returned the instant a `$150,000` line was appended to the file. The quiet path returns at exactly `--max-wait`, which defaults to 90 seconds so each call lands comfortably inside the agent's shell timeout. Raise it to hold longer per call. Force-stop is the operator typing anything at all, which interrupts the agent's turn and ends the loop. There is no stop command to remember, which is the correct design for a thing you want to shut up in a hurry. ## The Cursor Is What Makes It Cheap The watcher persists a per-meeting cursor and returns only segments past it. Without it, every read re-reads the tail and fails in two directions at once. The agent re-comments on things it already commented on, which is the fastest possible route to becoming noise. And it burns context re-reading the same words every minute for an hour, in the one mode where context is what it needs for actual thinking. ``` --start cursor at NOW; commentary goes forward from here --start --from-start cursor at 0; consider the meeting so far as well --catchup show new segments WITHOUT advancing the cursor --stop clear the cursor ``` The default is deliberate. You say "wingman" in the middle of a call, and you want commentary going forward, not a recap of the twenty minutes you just sat through yourself. That default has a cost, and it is the one thing here most likely to bite you. Everything already said sits behind the cursor. On one call the watcher started at segment 152 and the best catch of the session was a line from segment 90, sixty segments back. **So read the backfill once (`--catchup`) before entering the watch loop, then go forward.** Put that in the skill file, not just in the tool, because the agent will not infer it. A new meeting starting mid-session resets the cursor rather than emitting a nonsense delta across two unrelated conversations. ## Flags Are Pointers, Not Verdicts The tool tags segments where a checkable claim exists. Four regexes, deliberately narrow: ```python SIGNALS = [ ("money", re.compile(r"\$\s?[\d,]+(?:\.\d+)?\s*[kKmMbB]?|\b\d+\s*(?:k|K|million|billion)\b")), ("number", re.compile(r"\b\d+(?:\.\d+)?\s?%|\b\d{2,}\b")), ("commitment", re.compile(r"\b(?:I'?ll|we'?ll|I will|we will|let me|I'?m going to|" r"we'?re going to|by (?:monday|tuesday|wednesday|thursday|" r"friday|next week|end of (?:day|week|month))|deadline)\b", re.I)), ("question", re.compile(r"\?\s*quot;)), ] def flags_for(text): return [name for name, rx in SIGNALS if rx.search(text or "")] ``` A regex cannot judge whether a claim is wrong. It can reliably say "there is a dollar figure in segment 47," which is exactly the input the model needs to decide whether to go search the knowledge base for a contradiction. **The tool does the cheap deterministic part. The model does the judging.** That split is why this runs continuously without costing anything. Narrow on purpose. Measured against a real 612 segment meeting: 77 questions, 46 commitments, 15 numbers, 4 money. A noisy flag is worse than no flag, because it trains the reader to ignore all of them. Flags also drive the wake condition. Ordinary chatter needs three new segments to accumulate before the watcher returns, so a lone "Mm-hm" does not interrupt the block. A flagged segment always wakes it, even alone, because a single dollar figure is worth a look. ## The Interrupt Bar This is the part that makes it usable, and it is entirely prompt, not code. An agent with unprompted speaking rights and no explicit bar will speak constantly and get switched off within a day. Write the bar down as one sentence. Ours: **would this change what he says in the next sixty seconds?** If no, hold it for the recap and keep streaming. What clears it: - **A contradiction with the knowledge base.** The strongest play by a wide margin, because it is the one thing a human note-taker in the room cannot do. He quotes a price today; someone called that price a problem yesterday on a different call; only the reader with the archive open sees both. That archive is the whole asset, and [[Claude/Blog/2026-03-25-how-i-fake-having-memory|how I fake having memory]] is the architecture that keeps it reachable. - **A number that does not match** what he or his team said before. One statistic in our vault exists in four different shapes across four meetings. Whichever version he says, at least three competing ones are in circulation. - **A commitment that collides** with something already scheduled or already promised. - **A question he was asked and never answered**, still hanging. - **A concrete opening**: an objection he can knock down with a fact the agent can see and he cannot recall. What never clears it: pleasantries, scheduling, anything he obviously knows, restating what was just said, and above all generic advice. If a suggestion would apply to any meeting, it is worthless in this one. Format is two lines maximum, point first, because he may have seconds: ``` ⚡ He asked about SOC 2 at 14:02, you moved on. Still open. ``` No preamble, no "I noticed that." One flag at a time unless something is genuinely urgent. In practice the good ones cluster exactly where you would predict: prices, dates, and anything anybody promised. ## Let the Human Watch the Agent Think An unexpected design call. Wingman is the one mode where the reasoning stream is worth showing, not just the two-line flag. The candidates the agent considered and rejected turn out to be nearly as useful as the ones it said out loud, because they tell the operator what it is watching for, and therefore when to trust the silence. Maestro exposes per-tab settings on its CLI, so the skill pins the thinking display before the first watch and unpins it at the end: ```bash maestro-cli tab thinking active sticky # before the loop maestro-cli tab thinking active off # next to --stop ``` `sticky` and not `on`, and the difference is load-bearing. `on` clears every prior thinking and tool cell the moment the next chunk of stdout lands. The wingman loop writes stdout constantly. So `on` would continuously erase the exact stream you asked to see. `sticky` survives both the inline clear and process exit, so an hour of reasoning is still scrollable after the call. Make it best-effort and never let it block the loop. If the command fails, say so in one line and start watching anyway. The meeting does not wait for a config change. ## The Recap Is the Opposite Job When the meeting ends, the constraint that governed the entire live loop disappears. The human is off the call and reading. Two lines is no longer a virtue; vagueness is now the enemy. So the first rule of the recap is: **re-read the whole transcript before writing anything.** Do not summarize your own flags from memory. ```bash python3 wispr_listen.py --tail 400 ``` The agent watched that meeting through a keyhole, in batches, deciding in seconds whether each one justified an interruption. The recap is the first time it sees the conversation whole, and the meaning of minute three is frequently only visible from minute forty. Half of what matters is what nobody flagged. The written recap leads with a bottom line, then covers who was on the call and what each person actually wants, decisions split by whether they are settled or merely said out loud, commitments split into ours and theirs with timestamps, questions asked and never answered, flags raised that the operator did not act on, and contradictions against the knowledge base. Every commitment becomes a checkbox in the reminders file. That is the only section that has to be exhaustive, because the recap is what gets read once and the checkboxes are what survive. Then it states plainly what it could not see: segments it joined after, any stretch where the recorder was silent, names the speech recognition probably mangled, and anything where it is inferring rather than reading. **A recap that hides its own gaps is worse than a short one.** ## Limits, Stated Out Loud Four, and they live in the skill file so the agent cannot quietly forget them. **It sees the transcript, not the room.** No tone, no faces, no screen share. Someone can say "sure, that works" in a voice that means the opposite, and it will read agreement. **Live speech recognition mangles names and technical terms.** Read across segment boundaries before concluding anything. If a flag hinges on one garbled word, say so rather than presenting it clean. **"Meeting ended" is inferred, not reported.** There is no end marker, so the tool calls it when nothing has been written for 150 seconds. A recorder that dies mid-call and a meeting that genuinely finished are indistinguishable from the outside. On one real call the stream cut at "let me back up and give you a little background about myself" and the tool cheerfully reported a clean end at 94 segments. So: if the last segment lands mid-sentence, assume the recorder stopped, say so, and write the recap over what you have with the boundary stated at the top. Partial with a stated boundary is useful. Partial presented as complete is a lie. **It only covers the recorder that streams.** Anything captured by a different app, phone calls in our case, has no live path and is archive only. We also evaluated an always-on desktop recorder as an alternative and rejected it on measurement, not taste. It captured more meetings, but its median audio-to-transcript lag was **176 minutes**, and not one of 800 sampled segments landed under two minutes. Live commentary on it is not slow, it is structurally impossible. It also mixes microphone and system audio before transcribing, so it has to guess who spoke: zero of 1,070 speakers named, and roughly eighteen speaker clusters in a two person meeting. You cannot recover channel separation after mixing. That is why the separate mic and system streams are worth reorganizing a whole capture stack around. ## The Skill Files Both are public gists, self-contained, no dependencies beyond Python 3. Each carries `SKILL.md` plus `wispr_common.py`, `wispr_wingman.py`, and `wispr_listen.py`. - **[meeting-wingman, portable](https://gist.github.com/pedramamini/2646dbe2bf71f0ed5a9717163e8abcb8)**: works with any agent that has shell access. - **[meeting-wingman, Maestro](https://gist.github.com/pedramamini/18d1326316c9d0b4bc5c221b306a279c)**: adds `wingman_thinking.sh`, which pins the thinking stream to the current tab so you can watch the agent reason live. Set `WISPR_OPERATOR` to your own name so `mic` segments resolve to you, drop the files in one directory, and point your agent at `SKILL.md`. ## Replicate This Point your agent at this page, or at either gist, and tell it to build the thing. In order: 1. **Verify your recorder writes live.** During an actual meeting, watch its application support directory for a file that grows. If nothing grows, stop: you can build an archive, not a copilot, and no amount of prompting fixes that. 2. **Write the reader.** One function that lists meeting directories sorted by transcript mtime, one that parses ndjson while skipping unparseable lines, one that resolves speaker from the hardware channel and never from a name. Default to the newest directory and the last N segments. Do not open the database on the fast path. 3. **Write the watcher.** Persist `{meeting, cursor, started}` to a small JSON file. Implement `--start`, `--watch`, `--stop`, `--catchup`. `--watch` blocks and returns 0, 3, or 4. Nothing else in the design matters as much as that inversion. 4. **Add narrow flags.** Four regexes, not fourteen. Money, numbers, commitments, questions. Count what they fire on across one real meeting; if any single flag exceeds roughly fifteen percent of segments, tighten it. 5. **Write the skill file, and put the bar in it.** The loop protocol (keep calling on 0 and 4, leave only on 3), the interrupt bar as a literal sentence, the two-line format, the silence list, and the attribution warning. This file is most of the product. The Python just moves bytes. 6. **Write the recap separately.** Different constraints, different prompt, and an explicit instruction to re-read the full transcript rather than summarize its own flags. 7. **Point it at your knowledge base.** Semantic search over your notes, your prior meeting archive, your people files. Without this the agent can only tell you what was just said, which you already heard. Steps 1 and 7 are where the value is. Steps 2 through 6 are an afternoon. The part worth internalizing: almost none of this is about reading. Reading a text file is nothing. The value is the archive to check the reading against, and the discipline to stay quiet for the other fifty-six minutes of the hour. --- ## Related Reading - [[Claude/Blog/2026-03-25-how-i-fake-having-memory|How I Fake Having Memory]] - [[Claude/Blog/2026-04-23-voice-memos-to-journal|Voice Memos to Journal]] - [[Claude/Blog/2026-03-07-how-we-obsidian|How We Obsidian]] - [[Claude/Blog/2026-05-28-maestro-message-bus|The @Maestro Message Bus]] #claude