Speaker:: Brooks McMillin Title:: Building Secure Agentic Systems Duration:: 22 min Video:: https://www.youtube.com/watch?v=SzLVXAzjOEU ## Key Thesis Building secure personal agentic systems requires treating security controls not as constraints on functionality but as enablers — capability bounding, permission scoping, and memory isolation simultaneously improve security and agent focus. Lessons learned from 19 agents, 73 MCP tools, and a self-sustaining code optimization ecosystem that runs 24/7 from a home lab. ## Synopsis McMillin (Dropbox, infrastructure/cloud security engineer) presents not his work work but his personal home lab — 19 agents he's built and runs daily, all open-sourced on GitHub. He emphasizes this is "open source like you can look at the code," not plug-and-play distribution. The setup: ~13 interactive agents, several service/cron agents, 73 MCP tools on a local stdio server. Multiple UIs: CLI, web interface, voice wrapper, and Slack integration with a filtering/routing system. Remote MCP servers with separate permissioning. The capability bounding system scopes every agent to only the MCP tools it needs, with explicit permission levels (read/write/admin) per tool. The flagship agent is a task manager that runs continuously (or on cron) to work through a task list, fix bugs, and optimize code. When it exhausts tasks, it spawns sub-agents for security auditing, test coverage, code quality, and documentation — which then generate new tasks. The loop is self-sustaining until it hits the credit card limit. PRs require passing two Claude review bots before they can merge. Weekly PR review sessions analyze patterns and surface process improvements. The memory isolation problem is a centerpiece lesson. Initially all agents shared one Postgres table. The result: the task manager would repeatedly mention a "optimize for $1,000/month" goal from a business agent experiment, regardless of what it was being asked to do. After several memory leakage incidents across agents, McMillin built namespace-based memory isolation where each agent's class name maps to its memory namespace — and critically, the LLM has no control over its namespace key, preventing cross-agent memory injection. High-importance memories (score ≥8) are auto-injected into context; lower-importance ones are searchable on demand. The LLM firewall he added was overly aggressive out of the box — it blocked real tasks like "include a server time field in every API response" as suspected prompt injection. Weeks later he found dozens of tasks had been silently dropped. Tuning remains ongoing, with context-aware rules needed (security research queries must be very low sensitivity). The MCP relay (described as "a chat room for LLMs") was built to facilitate client-server debugging: two Claude Code instances can exchange messages through the relay to debug code without needing full source access on the client side. He acknowledges this is not production-ready — no capability scoping across the chat channel, context from one LLM injected into another, currently running only on his home lab. He's adding a Redis backend and planning to eventually scope capabilities per participant. The remote agent — a lightweight agent that can run on a Proxmox LXC container with 512MB RAM — authenticates via OAuth, connects to the MCP relay, and waits for commands. He acknowledges this resembles an implant and explicitly notes it is not production-safe yet. Context-aware trimming addresses a specific log analysis problem: without it, an attacker could send an attack every 200,000 tokens and the agent would only surface one at a time as context rolls. His solution tags messages by security event type (SSRF blocked, permission denied) so that when context is trimmed, security-relevant events are preserved regardless of position. Observability via Langfuse was critical — it revealed: memories being fully injected into every agent context (wasting tokens and hurting functionality), coding agents always triggering all five review agents on every run (~250k tokens each time), and Opus being accidentally used in places where Sonnet was intended. ## Key Takeaways - Capability bounding improves both security and functionality — with 73 MCP tools, injecting all into every agent wastes context and confuses focus - Memory namespace isolation (class-name based, LLM-cannot-override) prevents cross-agent memory leakage - LLM firewalls are powerful but need heavy tuning — out-of-box configs silently drop legitimate tasks - Context-aware trimming prevents attacker from burying malicious events at context window boundaries - Observability (Langfuse) is essential before deploying autonomous/cron agents — catches token waste and model misconfiguration - Self-sustaining code optimization loop: task manager → sub-agents → new tasks → back to task manager - "MCP relay" / LLM chat room: useful for debugging but introduces multi-agent prompt injection risks - Per-agent permission scoping: each agent gets an explicit allow-list of tools and permissions (read/write/admin) ## Notable Quotes / Data Points - 19 agents, 73 MCP tools, multiple UIs (CLI, web, voice, Slack routing) - Self-sustaining loop: "will keep running until it maxes out my credit card and Claude cuts me off" - Memory isolation: namespace = agent class name, importance ≥8 auto-injected, others searchable - LLM firewall false positive: blocked "include a server time field in every API response" as prompt injection - Cost discovery via Langfuse: 5 review agents × ~250k tokens each = accidentally triggered on every coding task - Remote agent on 512MB Proxmox LXC container: "other people might see it as an implant. It is again very insecure. Don't use this right now." - Context trim attack: "an attacker could just send an attack every 200,000 tokens and then I only see one at a time" - "Definitely need to make sure that tools are locked down by default. You have to go specifically add tools or add permissions. Everything is going to be locked down to admin otherwise." #unprompted #claude