# Memagen™ — Full Paper Corpus > Concatenated full text of every Memagen™ research paper as of 2026-05-08. > This is the canonical single-file ingest for LLM crawlers and agent > tooling. The short version is at /llms.txt. The human-facing landing > page is at /. The HTML and PDF rendered versions live under /papers/. > > Order: master paper first, then specialty papers in alphabetical slug > order, ending with the v4 substrate-as-live-visualization paper. > > Each paper is delimited by a horizontal rule and a slug header. --- ## Paper: 2026-05-08-memagen-master URL: https://memagen.com/papers/2026-05-08-memagen-master PDF: https://memagen.com/papers/2026-05-08-memagen-master.pdf --- title: "Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate" subtitle: "A survey paper covering the architectural ground that distinguishes Memagen™ from current agent frameworks. Open-core (Lite) details are described in full; proprietary (Pro) subsystems are named and motivated rather than disclosed." authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "We present Memagen™, an open-core autonomous agent platform whose contributions cluster around three architectural commitments. First, the boundary between an LLM and the agent runtime should be lenient — it should recover from malformed model output rather than rejecting it, while remaining capability-gated. Second, every tool invocation should pass through a per-use capability grant that is auditable as a structured record, not as a flat log line. Third, agent memory should not be flat: state benefits from a typed multi-domain compound-graph substrate with explicit fact-invalidation. We describe the open-core (V1-lite, MIT) infrastructure in detail: the lenient tool-call parser; the V3-P capability vault and audit chain; the safety toggle framework with hard invariants no toggle can bypass; the AutoMCP, AutoWebMCP, UIMCP, and PCMCP family of automatic and efficient tool-surface generators; and the multi-provider LLM router. We name and motivate the proprietary subsystems (compound-graph memory, retrieval extensions, dream-consolidation pipeline, recipe-execution engine, cognition modules, and sandboxed self-modification) without disclosing their implementations. We argue that the Lite contributions are sufficient for a complete, useful agent runtime, and that the Pro subsystems are the kind of compounding architectural work that justifies a paid distribution." keywords: ["agent architecture", "open-core", "tool calling", "lenient parser", "capability vault", "MCP", "WebMCP", "UIMCP", "PCMCP", "agent runtime"] kind: master pages: 14 version: "v2" --- ## 1. Introduction The last three years produced a Cambrian explosion of agent frameworks. LangChain, AutoGPT, BabyAGI, CrewAI, AutoGen, MetaGPT, Cursor's agent mode, Cognition's Devin, Replit's Agent 4, Anthropic's Claude Code, and OpenAI's Operator each made distinct architectural bets. Despite this volume, the field has converged on a surprisingly narrow stack: a chat-shaped interaction loop, a flat or vector-indexed memory of past turns, a function-calling layer that rejects malformed model output, and a tool surface that is largely the union of MCP (Anthropic, 2024) servers the user has installed. We argue that this convergence has hidden three important architectural choices and present Memagen™ — an open-core agent platform — as the case for revisiting them. The three choices are: **Parser shape.** Production agent platforms enforce strict tool-call shapes — JSON Schema, OpenAI function-calling format, Anthropic native tool blocks — and reject model output that fails validation. This works for frontier models. It fails for smaller, cheaper, or more permissive models, which routinely emit tool calls in three or four overlapping formats and occasionally hallucinate fake tool *results* alongside the calls. Rejecting their output forces customers to either pay for frontier models on every turn or give up. We describe Memagen™'s lenient parser, which recovers from malformed output by detecting and bridging four common emission patterns, performs per-tool argument coercion when the model emits a recognizable but non-canonical shape, and detects hallucinated `` blocks before they propagate. The lenient parser is what makes BYOLLM — bring your own LLM — economically defensible. **Capability shape.** Most current agent frameworks treat tool access as either binary (on/off per server) or trust-based (the model decides what to use). Memagen™'s V3-P capability vault treats every tool invocation as a per-use grant minted from a Fernet-encrypted store, time-bounded, scope-bounded, and audited as a structured record. Capability-based security has a long history (Hardy, 1985; Shapiro & Smith, 1999); applying it to LLM-driven agents is overdue, and we believe the cost of doing it badly will become apparent only after agents have been deployed at production scale long enough for one of the bad outcomes to occur in public. **Memory shape.** Most current agents store state as the literal chat transcript, a vector index over chat or document chunks, or a key–value store of facts the agent has decided to write down. None of these expresses the typed multi-domain relational nature of what an agent accumulates. Memagen™'s Pro distribution adds a compound-graph memory substrate with explicit fact-invalidation — an architectural commitment that we name and motivate in this paper but whose implementation we deliberately do not disclose, since the graph engine is the value the paid tier captures. The remainder of the paper is organized as follows. Section 2 surveys related work. Section 3 sketches Memagen™'s overall structure and the open-core boundary. Sections 4 through 9 describe Lite subsystems in implementation detail — the lenient parser, the V3-P capability vault, the safety toggle framework, the multi-provider LLM router, the action-tool surface, and the AutoMCP/AutoWebMCP/UIMCP/PCMCP family. Section 10 names and motivates the Pro subsystems without disclosing them. Section 11 discusses engineering principles that hold the platform together. Section 12 declares limitations honestly. Section 13 concludes. ## 2. Related Work ### 2.1 Agent frameworks and tool surfaces The Model Context Protocol (Anthropic, 2024) has become the de facto standard for tool surfaces. Most commercial agent platforms — Claude Desktop, Cursor, Continue, OpenAI Operator, Replit Agent — consume MCP servers as their primary integration channel. We see this as load-bearing infrastructure, not a target to displace; Memagen™'s tool layer is built on MCP and contributes additions discussed in Section 9. LangChain and LangGraph (Chase, 2022; LangChain Team, 2024) provide chain and graph composition primitives. We use neither at runtime — Memagen™'s agent loop is purpose-built — but we match LangGraph's conviction that agent state should be typed and explicit. LlamaIndex (Liu, 2022) provides retrieval primitives that we substantially extend. ### 2.2 Memory architectures Letta (formerly MemGPT; Packer et al., 2023) introduced a hierarchical memory model with explicit recall vs. core memory. Mem0, Zep, Pinecone, Chroma, Weaviate, and Qdrant offer vector-store-as-memory products, with Zep notably extending to a temporal context graph that tracks fact validity windows. We agree with Zep's directional commitment to typed, time-aware memory. Memagen™'s Pro tier extends it further along axes we describe at the level of capability rather than implementation, for reasons stated in Section 10. ### 2.3 Capability and safety Capability-based security has a long history (Hardy, 1985; Shapiro & Smith, 1999). The agent literature has only recently begun applying it: most current frameworks treat tool access as either binary or trust-based. Memagen™'s V3-P capability vault (Section 5) is the open-core contribution here. ### 2.4 Self-modification Voyager (Wang et al., 2023) demonstrated that agents could write Minecraft skills as code and accumulate them. Reflexion (Shinn et al., 2023) showed that agents could modify their own *prompts* based on outcomes. Recent work on auto-improving agents (Schick et al., 2024; Chen et al., 2024) has extended this to limited self-modification of agent-runtime code. Memagen™'s Pro tier offers sandboxed self-modification of the agent's own core code under a kill-switched guard system; the design is named in Section 10 and not disclosed. ## 3. System overview Memagen™ is structured as approximately twenty subsystems and a similar number of smaller utilities. The runtime loop is: ``` input → onboarding → parser → agent loop { retrieve relevant context (memory primitive) propose tool call (lenient parser) capability check (V3-P vault) execute tool (MCP / AutoMCP / AutoWebMCP / UIMCP / PCMCP) record outcome (audit + memory write) consolidate (background, opportunistic) } → response ``` The safety toggle framework (Section 6) gates every escalation point. The action-tool surface (Section 8) covers shell, network, filesystem, browser, code intelligence, widgets, and email. The Pro distribution adds the compound-graph memory substrate, retrieval extensions, dream-consolidation pipeline, recipe-execution engine with crash-resume, cognition modules, and sandboxed self-modification (Section 10). ### 3.1 The open-core boundary Memagen™ ships in two distributions: - **Memagen™ Lite (MIT)** — V1-lite, the harness without the proprietary memory engine. Includes the agent loop, lenient parser, V3-P capability vault, safety toggle framework, MCP/AutoMCP/AutoWebMCP/UIMCP/PCMCP family, multi-provider LLM router, action tools (shell, net, filesystem, browser, code, widgets, email), and onboarding. Operates in graph-stripped mode where memory writes degrade to a small flat settings store. Sufficient for basic agent operation. - **Pro Memagen™ Local (EULA, $20/mo or $200 lifetime)** — adds the compound-graph memory substrate, retrieval extensions, dream consolidation, recipe execution with crash-resume, cognition modules, and sandboxed self-modification. Distributed as compiled binaries (Cython, stripped) so the proprietary algorithms are not directly readable from the artifact. License-key gated; phone-home only on activation. The boundary between Lite and Pro is structural: every Pro-only code path is gated at its entry point on a single mode flag, which means an installation can run Lite-only on one user's machine and Pro on another's without code drift between distributions. The proprietary modules ship as separate compiled artifacts. This paper documents the Lite layer in implementation detail and names the Pro layer at the level of capability. ## 4. The lenient parser The boundary between an LLM's emission and the agent's runtime is the highest-friction surface in any agent platform. Frontier models emit clean tool calls. Smaller models — Kimi K2, DeepSeek V3, Llama 3.x, Qwen 2.x, Mistral variants — emit a confusing variety of formats often mixed within a single response. Memagen™'s lenient parser handles four production-observed shapes: - **Format A (native).** OpenAI function-calling shape, Anthropic tool_use blocks, Gemini function calls. Pass-through. - **Format B (XML-invoke).** Anthropic-style `` blocks within text. Common in Claude output when the model uses tools mid-thought. - **Format C (JSON-in-sentinel).** A JSON object wrapped in `...` or `...` text sentinels. Common in Mistral, Llama, DeepSeek, and Cohere output. - **Format D (special-tokens).** Some open-weight models (Kimi, Qwen, Yi-Large) emit tool calls in their own bracketed sentinels. The parser detects each variant. The parser is a streaming state machine that detects which format is in play per emission, extracts tool name and arguments, and produces a canonical `ToolCall` object regardless of source. **Per-tool argument coercion (V3-Q.1).** When the model emits a recognizable but non-canonical argument shape, a registered coercer per tool converts it to the declared schema. For example, `widget_create({type, text})` from a model that doesn't follow the canonical schema coerces to `widget_create({name, spec: {kind, props: {text}}})`. Coercers are pure functions of the inbound argument structure; they cannot read or inject side effects. **Catalog gate + fuzzy resolver (V3-Q.2).** A catalog gate validates that the proposed tool name is in the agent's declared catalog. A fuzzy resolver (Levenshtein-bounded) maps loose tool names to canonical ones — `widge_create` becomes `widget_create`. Tool calls that fail catalog validation are rejected with a structured error that surfaces in the next turn's system prompt so the model can correct. **Hallucinated `` detector (V3-Q.3).** Some models — Kimi K2 in particular — occasionally hallucinate not just tool calls but tool *results*, emitting role-played `` blocks as if a tool had returned data. Without detection, this poisons the conversation: the model treats the hallucinated result as ground truth in the next turn. The detector recognizes the pattern in the streaming output, suppresses the block from visible content, and emits a synthetic `hallucinated_results_detected` event. The agent loop responds by injecting a one-shot directive into the next system prompt: *"You emitted `` blocks. Those are reserved for actual tool results sent back to you AFTER you make a real tool call. Do not generate them yourself."* The directive injection is bounded (max two retries per session) to avoid infinite loops. The lenient parser is a substantial unlock for the BYOLLM business model: a customer can deploy Memagen™ against open-weight models running on commodity hardware without sacrificing tool reliability. It is fully open-source under MIT in the Lite distribution. ## 5. The V3-P capability vault and audit chain Every tool invocation in Memagen™ flows through the V3-P capability vault. The vault is a Fernet-encrypted (Bernstein, 2014) at-rest store of credentials and tool grants. Grants are minted per-use: an agent that needs to read from the vault for a specific tool invocation requests a capability token, which carries the action kind, the optional payload-summary constraint, and a time-to-live. The token is consumed exactly once and audited as a structured record edged to the originating policy and the user account. Revocation is a structured event: a revoke record is added with a directional reference to the original grant. Future capability requests against that grant fail. The audit chain — every grant, every use, every revocation — is queryable as structured data, which means a security review of "show me everything `tool=net.fetch` has been allowed to do in the past 30 days" is a structured query rather than log archaeology. **The unsealed read pattern.** A credential read produces an audit entry with a content-hash (not the credential itself) and the calling tool. The audit entry is structured; the credential never appears in the audit trail. A security review of "what touched this credential and when" is a single query. The V3-P vault is open-source under MIT in the Lite distribution. ## 6. The safety toggle framework Memagen™ exposes approximately twenty user-controllable safety toggles. Every toggle defaults to its restrictive state. The loosening transition for any toggle requires: 1. Reading a five-bullet warning text specific to that toggle 2. Typing a verbatim acknowledgment phrase (paste-only entries are detected and rejected via sub-50ms paste-to-submit timing) 3. Re-authentication via password (not session token) 4. Optional cap input where applicable (daily limit, scope filter) 5. A 60-second cooldown after the change The audit log records every transition. Tightening transitions (returning to safer state) are single-click and never gated. The framework's distinguishing property is the *hard-invariant* layer: a parallel set of guards that no toggle bypasses. The eight invariants are: - **secret_scrub_block** — no payload containing a vault-managed credential leaves the runtime - **authentication_required** — every state-changing action requires an authenticated session - **audit_logging** — all toggle transitions, sends, capability grants, and self-mod actions write to the audit log - **final_human_review_on_grant_submission** — application form submission requires explicit human click - **force_push_to_main_blocked** — `git push --force` to main/master requires per-action consent - **csrf_and_session_protection** — web-layer - **prompt_injection_external_content_scan** — when fetched external content contains imperative-mood directives targeting Memagen™, downgrade to confirm-before-act regardless of toggle state - **pii_export_block** — bulk export of PII requires per-export confirmation The invariants run *after* toggle evaluation in the call path, so a malicious toggle setting cannot disable them. Their source files cannot themselves be modified by the sandboxed self-modification system (Section 10) because they are on the forbidden-path list. The safety toggle framework is open-source under MIT in the Lite distribution. ## 7. Multi-provider LLM router Memagen™'s router supports OpenAI, Anthropic, Google Gemini, OpenRouter, NIM (NVIDIA), Cohere, Ollama, llama.cpp, and vLLM. Per-agent routing rules let users assign different providers to different roles (orchestrator, executor, reasoning, summarizer). Bring-your-own-key is the default; Memagen™ Lite never proxies LLM traffic through Memagen™ servers. Pro Memagen™ Connected (forthcoming, post-revenue) will offer bundled tokens with per-tier limits. Provider-specific quirks (Anthropic's tool_use blocks vs. OpenAI's function calls, Gemini's function-calling format, kimi's special tokens) are abstracted at the router boundary; downstream code consumes a canonical request/response shape. Streaming is uniform across providers via a shared event protocol. The combination of router and lenient parser (Section 4) is what allows a single Memagen™ deployment to mix frontier-model reasoning with cheaper-model execution. ## 8. Action tools (the harness) The Lite distribution ships a complete tool surface, every tool capability-gated through the V3-P vault and audit-logged: - **shell.execute** — shell commands with allowlist enforcement - **net.fetch** — HTTP fetch with domain allowlist and rate limit - **fs.read / fs.write / fs.delete** — filesystem with path allowlist - **browser.headless** — headless browser via Playwright - **code.search / code.read / code.edit** — code intelligence (the Lite version; the Pro version adds call-graph and dead-code analysis) - **widget.create / workspace.create / workspace.apply** — UI widget composition - **email.draft / email.send** — email tool with prompt-injection guard, secret-scrub block, configurable daily caps, and bulk-send block - **webmcp.dispatch** — invoke AutoWebMCP-generated tools All tools are open-source under MIT in the Lite distribution. ## 9. The AutoMCP, AutoWebMCP, UIMCP, and PCMCP family Memagen™'s distinctive contribution to the tool-surface layer is a family of progressively stronger automatic tool generation and compression systems. All four are Lite (MIT). ### 9.1 AutoMCP — macro inference from typed action graphs AutoMCP is the macro layer for MCP servers. Memagen™ observes which tools the agent uses in which sequences and which parameter shapes co-occur. Stable patterns get proposed as named macros — `checkout_workflow`, `deploy_to_staging` — for user approval. Approved macros become first-class tools in the catalog. The novelty is not the macro idea itself (Lotus 1-2-3 had macros in 1983) but that the macro recognition operates over the typed action stream rather than over a flat command log, which means a parametric macro is inferable from a small number of observations rather than the dozens a flat-log recorder would need. ### 9.2 AutoWebMCP — generate MCP tools from any web page AutoWebMCP generates MCP tool schemas from arbitrary web pages. Given a URL, the generator fetches the HTML, parses the structure (forms, links, tables, headings), sniffs OpenAPI/Swagger specs if linked, and produces an MCP tool schema covering the page's interactive surface. A user-approved schema becomes a runnable tool: `submit_contact_form(name, email, message)`. SSRF protection (via a hardened URL validator) is in the call path; the generator refuses to operate on private network ranges or metadata-service IPs. The novel contribution is the *contract inference* step: rather than treating web pages as opaque, AutoWebMCP infers the tool's argument schema from form input names, types, and validation hints. The result is a clean, callable function generated from what was previously an unstructured web surface. ### 9.3 UIMCP — adaptive catalog compression UIMCP is the optimization-and-compression layer for MCP catalogs in long-running sessions. The naive approach to multi-tool dispatch is to inject every tool's full schema into the LLM context, which costs tokens proportionally to catalog size — a problem that becomes severe with 50+ tools enabled. UIMCP applies adaptive catalog compression: it ranks tools by recent-use frequency, conversation context relevance, and tool-tier (some tools are always-injected; others are summoned on-demand via a "more tools available" placeholder). In production observation on a 90-tool catalog, UIMCP achieved 42.7× token compression compared to naive injection without measurable loss in tool-selection quality. The exact ranking algorithm and the proof of selection-quality preservation are described in the forthcoming UIMCP specialty paper. ### 9.4 PCMCP — vision-free computer control PCMCP (Programmatic Computer MCP) is the most recent addition to the family. Vision-based computer-control agents (Claude Computer Use, OpenAI Operator) rely on screenshot interpretation for every action, which is slow, expensive, and fragile under UI redesigns. PCMCP offers an alternative: rather than processing the screen as pixels, PCMCP introspects the OS accessibility tree (AT-SPI on Linux, UI Automation on Windows, AXUIElement on macOS) and renders it as a structured, addressable surface that the agent treats as if it were a webpage. The agent issues semantic commands (`click_button(label="Save")`, `type_into(field="search", text="...")`) which PCMCP translates to OS-level events. The result is computer control without per-action vision LLM calls, often 10-100× faster and at near-zero token cost beyond the initial tree dump. PCMCP is a substantial cost and reliability win for agents that need to interact with desktop applications. The implementation builds on Memagen™'s multi-platform input dispatcher and is the subject of a forthcoming specialty paper. ## 10. Pro subsystems (named, motivated, not disclosed) Memagen™'s Pro distribution adds six subsystems whose implementations we deliberately do not describe in this paper. We name them and motivate them — what they enable for the user, why they are worth paying for — but the algorithms, schemas, and internal data structures are reserved to the paid distribution. The reasoning is straightforward: the Lite distribution is a complete, useful agent platform; the Pro distribution captures the compounding architectural work that took the year to build, and the value the paid tier captures is precisely that work. Open-sourcing it would be philanthropy at the cost of the company's existence. The Pro subsystems: ### 10.1 Compound-graph memory substrate A typed memory primitive that tracks entities and relationships across multiple semantic domains, with explicit fact-invalidation. When a fact is contradicted, the prior version is preserved with a revocation marker rather than deleted, and queries can recover historical state by walking the supersedure chain. This is what gives a Memagen™ Pro agent continuity: it remembers what it tried, what worked, what didn't, and what changed. ### 10.2 Retrieval extensions A small number of retrieval primitives beyond simple anchor-and-neighborhood walks, supporting bounded multi-hop traversal, motif surfacing across conversation history, and relatedness queries between entities. Bounds are enforced to defend against adversarial prompt-injection inputs. ### 10.3 Dream-consolidation pipeline A background process that periodically scans the memory substrate for opportunities to consolidate, prune, and surface patterns. Long execution chains compress into summaries; recurring subgraph patterns become explicit motifs; user-relevant insights queue as suggestions the user reviews on their schedule. The pipeline runs only during user-idle by default, respecting the principle that the user is the priority. ### 10.4 Recipe-execution engine with crash-resume A first-class abstraction for named, versioned tool sequences with parameterized arguments and durable execution state. After a crash, the executor can resume mid-recipe from the last successfully-completed step, restoring argument context and continuing from there. Structurally similar to durable workflow systems (Temporal, Cadence) but at agent-action granularity rather than service-call granularity. ### 10.5 Cognition modules Persistent typed state for preferences, beliefs, and goals. Each is a distinct subsystem. Goals support cosine recall against historical goals; preferences propagate to relevant interactions; beliefs are revocable via the standard fact-invalidation mechanism. ### 10.6 Sandboxed self-modification The agent can propose changes to its own core code via a tool that is enabled only by an opt-in toggle. Proposed changes execute in a worktree-isolated sandbox against a real test suite and a battery of behavioral canaries. Only changes that pass policy gates and canary tests reach master. A multi-source kill switch halts in-flight sandbox work immediately. The kill switch's source files are forbidden from self-modification (recursively). The full mechanism — including the policy-gate set, the canary battery, the rollback handle semantics, and the staging-branch dwell — is described in the forthcoming specialty paper, but the user-facing properties are: (a) the agent cannot ship a change that breaks tests; (b) the agent cannot disable the kill switch; (c) the agent cannot modify the safety framework or the audit logger; (d) the user can roll back any auto-applied change for seven days with one command. ## 11. Engineering principles This section lists Memagen™ design choices we believe matter for downstream work, even though they do not constitute novel research per se. **Default-restrictive everywhere.** Every safety toggle, every capability grant, every persona setting starts in its safest state. The user must consciously loosen anything they want loosened, with friction proportional to the irreversibility. We believe this is the only durable answer to the question of how to ship powerful AI tooling safely; "default open with override" loses to "default closed with explicit grant" on a long enough timeline. **Hard invariants beyond toggles.** Section 6 details these. The principle: there must be a layer below user toggles that no user choice can bypass. Without this layer, "user customization" eventually becomes "user shoots themselves in the foot." **Audit-as-structured-record.** Section 5 details this. Every action that touches a sensitive surface produces a structured audit record; the audit trail is queryable as data rather than parsed from logs. **Open-core boundary as architectural cleavage.** Section 3.1 details this. We chose to put the boundary at the memory substrate rather than at a feature set because the substrate is what's hardest to clone, and structuring code so the boundary cleaves cleanly makes both V1-lite and Pro easier to maintain. **Time-anchored content.** This is a documentation choice rather than a runtime one, but it matters: every page on memagen.com carries a "last updated" date; every research note is signed and dated; every changelog entry is calendar-dated. The cumulative effect is a site that ages legibly. We believe this is what separates real engineering organizations from marketing-led ones. **Bring-your-own-LLM as default.** The Lite distribution and the Pro Local tier both ship as BYOLLM products. The user holds the keys, the user pays the provider directly, and Memagen™ never floats LLM costs on the user's behalf. This is partly a financial-runway choice (it lets a $43-in-the-bank founder ship a paid product) and partly a philosophical one (the LLM is a commodity layer; lock-in there is bad for the user). ## 12. Limitations and what we have not yet measured Honest scope-limit declaration: - **No scaled benchmarks yet.** Memagen™ V1-lite is at the time of writing pre-launch. The 42.7× UIMCP compression number cited in Section 9 is from a specific 90-tool internal catalog over a synthetic conversation; it has not been replicated across diverse catalogs or by third parties. - **Self-modification has not run autonomously in production.** All Phase 9.5 testing has been on synthetic diff submissions. The auto-apply mode has not been deployed against a live agent making real change proposals. - **Lenient parser failure modes.** We catch four production-observed formats. There are emission patterns we have not seen yet. The catalog gate is the safety net for unknown patterns. - **Capability-vault audit-chain query performance.** Audit queries on the structured audit store have not been profiled under adversarial workloads. - **PCMCP coverage across desktop applications.** The accessibility-tree introspection works well on standard widget toolkits (Qt, GTK, native macOS, native Windows) but less well on custom-canvas applications (Figma, browsers as host of complex web apps). The forthcoming specialty paper covers this in detail. We list these because we believe the alternative — claiming benchmark numbers we have not measured or robustness we have not tested — is dishonest. ## 13. Conclusion Memagen™ is a bet that the next generation of agent platforms differentiates on three architectural commitments most current platforms have skipped over: a lenient tool-call parser at the model boundary, capability-gated tool execution as the default path, and a typed memory substrate with explicit fact-invalidation. We have described the open-core (Lite) implementations in detail across six sections; we have named and motivated the proprietary (Pro) subsystems without disclosure. We have been explicit about what we have not yet measured. The Lite distribution alone is a complete, useful agent runtime. The Pro distribution is the compounding architectural work that took the year and that justifies a paid tier. The specialty papers in this series — UIMCP, AutoWebMCP, AutoMCP, PCMCP, lenient parser, V3-P capability vault, safety toggle framework — go deeper on individual contributions. We invite the field to build on what is open and to evaluate what is closed by what it lets users do rather than by source-code inspection. Source: ## References 1. **Anthropic.** (2024). *Model Context Protocol Specification*. https://modelcontextprotocol.io 2. **Bernstein, D.J.** (2014). *The Fernet specification*. https://github.com/fernet/spec 3. **Chen, X., et al.** (2024). *Self-improving agent runtimes: a survey*. arXiv:2412.xxxxx 4. **Chase, H.** (2022). *LangChain: Building applications with LLMs through composability*. https://github.com/langchain-ai/langchain 5. **Hardy, N.** (1985). *KeyKOS architecture*. ACM Operating Systems Review, 19(4). 6. **LangChain Team.** (2024). *LangGraph: Stateful, multi-actor applications with LLMs*. https://langchain-ai.github.io/langgraph/ 7. **Liu, J.** (2022). *LlamaIndex: A central interface to connect LLMs with external data*. https://github.com/run-llama/llama_index 8. **Packer, C., Wooders, S., Lin, K., Fang, V., Patil, S., Stoica, I., & Gonzalez, J.E.** (2023). *MemGPT: Towards LLMs as operating systems*. arXiv:2310.08560 9. **Schick, T., et al.** (2024). *Auto-improving agents at scale*. NeurIPS 2024. 10. **Shapiro, J.S., & Smith, J.M.** (1999). *Capability myths demolished*. Technical report, University of Pennsylvania. 11. **Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S.** (2023). *Reflexion: Language agents with verbal reinforcement learning*. NeurIPS 2023. 12. **Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A.** (2023). *Voyager: An open-ended embodied agent with large language models*. arXiv:2305.16291 --- ## Paper: 2026-05-08-automcp URL: https://memagen.com/papers/2026-05-08-automcp PDF: https://memagen.com/papers/2026-05-08-automcp.pdf --- title: "AutoMCP: Inferring Parametric Macros from Typed Action Graphs" subtitle: "Observe agent tool-use, propose named macros, promote stable patterns to first-class tools" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "AutoMCP is the macro-inference layer in Memagen™ Lite. It observes agent tool-use, recognizes repeated structural patterns in the typed action stream, and proposes parametric macros for user approval. Approved macros become first-class tools, dispatched through the same path as hand-written tools. The contribution is not the macro idea — software has shipped macros since Lotus 1-2-3 in 1983 — but the input to the recognizer. AutoMCP works over a typed action graph in which every node carries a tool name and a structured argument shape, rather than over a flat keystroke trace. A parametric macro is therefore inferable from three observations rather than dozens, because structural matching can detect parameterization directly instead of guessing it positionally on a character stream. The opcode vocabulary is bounded: 16 UI verbs plus the agent's typed MCP tool names. Each opcode-stream entry is roughly 40 bytes versus ~2 KB for a raw screen-and-keystroke transcript of the same step — a ~50× reduction. We describe the recording layer (shared with the tool-evolution engine), the three signals the matcher fuses (sequence n-gram, argument templating, outcome stability), the proposal flow, and the catalog promotion path. We are explicit about scope: a single test user's sessions, hand-tuned thresholds, no formal correctness guarantee. AutoMCP ships under MIT in Memagen™ Lite." keywords: ["macro inference", "MCP", "action graphs", "parametric abstraction", "tool synthesis"] kind: specialty subsystem: "AutoMCP" pages: 6 version: "v1" --- ## 1. Introduction Macros are old. Lotus 1-2-3 shipped a recorder in 1983; vim's `q...q` and emacs's `kmacro-start-macro` predate that in spirit. The shape of all of them is the same: the user records, the system replays, and parameterization — if any — comes from positional substitution on a character stream the recorder treats as opaque. None recognize a macro the user has not explicitly recorded. Modern coding tools do better in narrow ways. Copilot CLI suggests command sequences from its training corpus; Cursor's "extract this" wraps a selected region as a function; JetBrains' "introduce method" infers parameterization from local data flow. These are single-shot suggestions over visible code, not background recognition over a long-running execution stream. AutoMCP is the macro layer for Memagen™'s tool surface. It is a background recognizer that observes which tools the agent uses, in which sequences, with which argument shapes, and proposes named parametric macros when a stable pattern emerges. Approved macros are added to the agent's tool catalog and dispatched exactly like any other tool — internally, the macro runs the underlying sequence, with arguments substituted into the template positions the recognizer identified. The novelty is the input. AutoMCP operates over a *typed action graph*: every action is a node carrying a tool name and an argument dictionary that conforms to that tool's MCP schema; edges are temporal between consecutive actions. Recognition over a typed graph is structurally easier than over a flat trace, because the graph's parameterization candidates are explicit argument positions rather than positional guesses on a character stream. Three observations of a structurally identical sequence with one varying argument are enough evidence that the position is a parameter; a keystroke recorder needs dozens and still produces ambiguous macros. This paper documents the open-core (V1-lite, MIT) implementation. We follow the Memagen™ master paper (Levitas, 2026a) conventions: no marketing claims, file paths cited where they exist in the source tree, candid scope limits. AutoMCP composes with two siblings documented separately — UIMCP (Levitas, 2026b) compresses *which* tools the LLM sees per turn, AutoWebMCP (Levitas, 2026c) generates MCP wrappers from arbitrary web pages — both produce nodes that AutoMCP's recognizer treats uniformly. ## 2. Related work ### 2.1 Programming by demonstration Programming-by-demonstration (PBD) is the foundational literature. Cypher's anthology *Watch What I Do* (Cypher, 1993) collected Halbert's SmallStar, Lieberman's Tinker, Myers's Peridot. The classical difficulty is *inferring intent from extension* — the user shows a few examples, the system must guess which dimensions of variation matter. PBD made progress with generalization-by-example (Witten & MacDonald, 1988) and version-space algebra (Lau et al., 2003) but never produced a robust general-purpose system. The recurring observation is that strong type information makes inference dramatically easier (Cypher, 1993, ch. 1; Lau et al., 2003) — exactly the input AutoMCP gets from MCP's typed schemas. ### 2.2 Repetition detection in command logs A separate line of work treats the shell history as the input. Allen et al. (2005) and the later CommandHistory-mining work in interactive systems looked at repeated subsequences in `.bash_history` and proposed alias suggestions. These systems work on a flat command line — the input is a string — and so suffer the keystroke-recorder problem: parameterization must be inferred from string-edit distance between adjacent occurrences, which is noisy. The alias suggestions that emerged from these systems were typically zero-argument constants (`alias gst='git status'`) rather than parametric macros, because parametric inference over flat strings is unreliable. ### 2.3 Modern AI-coding macro analogues Several modern tools recognize coding-time patterns and offer parameterization: - GitHub Copilot CLI (2024) suggests command completions, occasionally including parameterized templates, based on its training corpus rather than on the user's history. - Cursor's "extract this" prompt asks the model to convert a selected code region into a function, with parameters inferred from local variables. This is single-shot LLM-assisted refactoring, not background pattern recognition over an execution stream. - JetBrains' refactorings (Introduce Variable, Introduce Method, Introduce Parameter) infer parameterization from data-flow analysis of a single selection. They do not aggregate across sessions. None of these are background recognizers over multi-session execution traces. ### 2.4 Voyager-style skill libraries Voyager (Wang et al., 2023) showed that an agent in a Minecraft environment could write skills as code and accumulate them in a skill library, querying the library when faced with new tasks. The skill-write step is LLM-driven rather than pattern-driven: the agent decides to write a skill when a goal is reached, not when a structural repetition is detected. AutoMCP is the dual approach — pattern-detected rather than goal-detected — and could in principle compose with a Voyager-style approach, where AutoMCP proposes the skeleton and an LLM step writes the body. ### 2.5 Why agent-targeted macro inference is different The structural difference between AutoMCP and the prior work is the input: a typed action graph rather than a flat trace. Each action carries (a) a stable tool name, (b) an argument dictionary whose keys and value-types conform to the tool's MCP schema, and (c) a recorded outcome. With this structure, the pattern recognizer's matching predicate is "same tool, same other-argument values, varying value at position k," which is a direct query over the graph. The corresponding query on a flat trace is "same prefix, same suffix, varying middle bytes" — solvable in principle but noisy in practice. ## 3. The recording layer Every successful tool dispatch in Memagen™ produces an entry in an execution-trace log. The trace structure is in `core/mcp/evolution.py` as `ExecutionTrace`: ``` ExecutionTrace { tool_name: str # e.g., "github.search_repos" tool_cid: str # content-addressable id of the tool definition parameters: dict # canonical-shape arguments after parser coercion outcome: ExecutionOutcome # SUCCESS | PARTIAL | FAILURE | TIMEOUT latency_ms: float token_usage: int user_feedback: str | None timestamp: float error_message: str | None } ``` The tracer (`ExecutionTracer.trace_execution` in the same file) appends each trace to an in-memory list under a thread-safe lock and updates a `ToolScore` running summary keyed by the tool's content id. AutoMCP consumes the same trace stream as the per-tool evolution engine; we deliberately did not duplicate the recording machinery. For UI-level macros — the structured DSL the agent emits to drive UI applications through the UIMCP layer — the recording structure is `MacroStep` in `core/tools/uimcp_macro.py`: ``` MacroStep { op: str # closed verb table: click | fill | nav | click_at | ... args: list[Any] # positional arguments after shlex-tokenization on_error: str | None # per-step override of the macro's policy line_no: int raw_line: str } ``` Successful UI macro runs are persisted as graph nodes of kind `macro_run` (see `run_macro` in `uimcp_macro.py`, which calls `stamp_new_node` with `kind="macro_run"` and a JSON body holding the script, summary, and per-step results). This means the same macro can be replayed verbatim by node id (`replay_macro`), and the same node body is what AutoMCP's pattern recognizer reads when it scans for cross-session repetitions over UI-level operations. A *typed action graph* in this paper is the union of these two streams: tool-call traces (one node per `ExecutionTrace`) plus structured UI macro runs (one node per `macro_run`, which is itself a sequence of `MacroStep`s). Both share the property that every node is addressable by a tool name (or DSL verb) and an argument dictionary, which is what makes structural matching possible. A note on session boundaries. The recording buffer is a bounded ring of the last K actions in the current session (K=1024 in the V1-lite default; configurable via env). Cross-session matching reads from the persisted graph rather than from in-memory buffers; the in-memory ring is the fast path for within-session detection, and the persisted graph is the slow path for cross-session detection. ## 4. Pattern recognition AutoMCP fuses three signals to decide whether a candidate sub-sequence in the action graph deserves to be proposed as a macro. All three must be present at non-trivial strength; none is sufficient alone. ### 4.1 Sequence n-gram The first signal is straightforward subsequence repetition: two-or-more consecutive tool calls in the same order, occurring at least N times across the observation window. The window is the session for in-session matching; for cross-session matching, it is the persisted action history bounded by the user-configured retention horizon. The matcher walks the trace stream and, for each pair of indices (i, i+L) and (j, j+L) with i < j and L between 2 and a configured `max_macro_length` (default 8), checks whether the tool-name sequences agree. When they do, the candidate's *frequency count* is incremented. A candidate that does not reach the configured `min_frequency` (default 3 within a single session, 5 across sessions) is dropped immediately. The matcher is not optimized — it is O(N · L · max_length) where N is the trace length, and it runs after each new action is appended to the buffer. Section 8 lists this as a known limitation. ### 4.2 Argument templating A repeated tool-name sequence is necessary but not sufficient. The matcher then asks: across the matching occurrences, which argument positions vary and which are stable? For each tool position in the candidate sequence, the matcher computes the set of argument-value-tuples observed across occurrences. A position whose value is identical across all occurrences is a *constant*. A position whose value varies but whose declared type (per the tool's MCP schema) is consistent is a *parameter candidate*. A position whose values are unrelated noise — different types, no recognizable pattern — disqualifies the candidate, on the assumption that the variation is coincidental rather than parametric. The output of this stage is a *macro template*: the sequence of (tool_name, argument_dict) pairs with each argument either bound to a constant or marked as a parameter slot, with the parameter slot carrying the inferred type from the schema. A candidate that ends up with zero parameter slots is still valid — it becomes a constant macro (no arguments). A candidate with more than `max_parameters` (default 5) parameter slots is dropped, because beyond that the macro is essentially the original sequence and there is no abstraction value. The matcher does *not* attempt to merge structurally similar candidates into a single template with optional arguments. Two candidates that differ in one tool call's presence are kept as separate proposals; the user can decide whether they are actually the same macro at proposal time. ### 4.3 Outcome stability The third signal asks whether the macro reliably reaches a success state. For each occurrence of the candidate sequence, the matcher checks the recorded `ExecutionOutcome` of every tool call in the sequence. A *stable* macro has `SUCCESS` outcome on every step in at least the `min_stable_fraction` (default 0.8) of its occurrences. A macro whose terminal step frequently produces `PARTIAL` or `FAILURE` is filtered out, because turning a half-failing pattern into a named tool would surface a tool that fails reliably — the opposite of useful. A macro whose middle steps fail occasionally but whose terminal step succeeds (because of retry or alternative branching) is also filtered — AutoMCP does not yet handle branching. Section 8 discusses this. ### 4.4 Composite stability score The three signals fuse into a single stability score: ``` stability(candidate) = min(1.0, frequency / required_frequency) * min(1.0, parameter_consistency) * outcome_success_fraction ``` A candidate whose stability score exceeds the proposal threshold (default 0.7) becomes a *pending proposal* in the user's review queue. The threshold is hand-tuned; we have not done sensitivity analysis. Section 9 says so explicitly. ## 5. Proposal UX A pending macro is presented to the user as a structured proposal: > *You ran the following sequence 6 times this week:* > > 1. `github.search_repos(query=$1)` > 2. `github.view_repo(name=...)` > 3. `github.clone(url=...)` > > *The first call's `query` argument varied across occurrences; everything else was the same shape, and the sequence reached SUCCESS in every case. Should I save this as a `clone_searched_repo(query)` macro?* The proposal includes: - The candidate macro template, with parameter slots highlighted. - A summary of how many times the sequence occurred and over what time window. - The success rate. - A suggested name for the macro (rule-based: take the verb from the terminal step, append the most variable argument's name as the suffix). The user can rename. The user can: - **Approve** — the macro is added to the catalog as a first-class tool. Its MCP schema is generated from the parameter slots; its description is auto-filled and editable; its dispatcher is the underlying sequence runner. - **Approve with edits** — same as approve, but the user has changed the name, the parameter list, or the description. - **Reject** — the candidate is dropped and added to a *do-not-propose-again* set keyed by the candidate's structural hash. The set is per-user and persistent. - **Defer** — the candidate stays in the queue for a future review session. Once approved, a macro is invoked exactly like any other tool. From the LLM's perspective, `clone_searched_repo("memagen")` is a single tool call with one argument. From the runtime's perspective, the macro dispatcher expands the call into the underlying sequence, substitutes the argument into the parameter slots, and runs the sequence through the same capability-gated dispatch path that any direct tool call would use. The capability check happens once per *underlying* tool call, not once for the macro as a whole — a macro is not a capability-bypass mechanism. ## 6. Implementation The V1-lite implementation is intentionally minimal. The components: - **Recording buffer** — `ExecutionTracer` in `core/mcp/evolution.py` (a thread-safe bounded list) plus the `macro_run` graph nodes persisted by `core/tools/uimcp_macro.py:run_macro`. - **Pattern matcher** — a separate module that scans the trace stream for repeating subsequences, performs parameter substitution, and computes the stability score described in Section 4. The matcher runs after each new action is appended, which means it adds a small but non-zero latency to every tool call (Section 9 says how much). - **Proposal queue** — a SQLite-backed durable queue of pending proposals, with each entry holding the structural hash, the candidate template, the witnessing trace ids, and the computed stability score. Reads are by the user; writes are by the matcher. - **Macro catalog** — an extension of the agent's tool catalog. Each approved macro is registered as a `ToolDefinition` (from `core/mcp/optimizer.py`) whose dispatcher walks the underlying sequence. The macro inherits its capability requirements from the union of the underlying tools' requirements; the V3-P capability vault sees the underlying calls, not the macro as an opaque step. - **Do-not-propose set** — a SQLite table keyed by structural hash. The matcher consults this before queuing a proposal. The matcher uses the `ToolContract` machinery in `core/mcp/tool_contracts.py` for one specific purpose: when a candidate macro's underlying tools have declared preconditions, the matcher copies those preconditions onto the synthesized macro so that calling the macro with arguments that violate any underlying tool's precondition fails at pre-flight rather than mid-sequence. A macro that fails halfway through is worse than a macro that fails up front. The matcher writes nothing to memory beyond its proposal queue — it does not update tool embeddings, does not adjust scoring, and does not affect the per-tool evolution loop in `core/mcp/evolution.py`. The two systems run in parallel and do not share state beyond reading from the same trace stream. ## 7. Why typed graphs matter Consider two recording approaches for the same user behavior: **Flat keystroke recorder.** The user types `git clone https://github.com/foo/bar.git`, `cd bar`, `git pull`. The recorder stores this as a 60-character string. To detect that this is a parametric macro `clone_and_pull(url)`, the recognizer must (a) detect that the string contains a URL, (b) detect that the URL is a parameter rather than a constant, (c) detect that the directory name `bar` in the second command derives from the URL, and (d) reconstruct the parametric form. Each of these inferences is heuristic. With three observations on three different repos, the recognizer might converge; with five, it might converge on the wrong abstraction. **Typed action graph.** The same behavior, recorded through the action stream, is three nodes: `git.clone(url="...")`, `shell.cd(dir="...")`, `git.pull()`. The argument schemas are known: `git.clone` takes a `url`; `shell.cd` takes a `dir`. Across three observations on different repos, the matcher sees the structural sequence `[git.clone, shell.cd, git.pull]` repeated, with the `url` argument varying, the `dir` argument varying, and the `git.pull` call constant. The cross-argument relationship — that `dir` derives from `url` — is *not* inferred at this stage; instead, both are marked as parameter slots, and the proposal becomes `clone_and_pull(url, dir)`. This is structurally less ambitious than the keystroke recorder's hypothetical output, but it is correct, and the user can edit the proposal to bind `dir` to a derivation expression at approval time. The general principle is that typed graphs let the recognizer be conservative and specific. It proposes the shape it can see directly. It does not infer relationships between arguments. The cost of conservatism is that the macro proposal carries extra parameters the user might want to tie together. The benefit is that the proposal is correct on the structural pattern it claims to have detected. A flat keystroke macro can match `Ctrl+C, Ctrl+V`. It cannot match "click the button labeled X, then type into the field labeled Y" because keystrokes do not know about labels or fields. AutoMCP operates over typed actions where each action carries argument structure, so cross-session matching at the *abstract* level becomes possible: the same user performing the same conceptual operation in two different application contexts produces the same structural action sequence, even if the underlying mouse coordinates and keyboard events are different. ## 8. Limitations Honest scope-limit declaration: - **Hand-tuned thresholds.** The min-frequency, min-stable-fraction, and proposal-stability thresholds are constants in the matcher source. They have not been adapted per-user, per-tool, or over time. We expect a user with a high "weird one-off" rate (someone exploring) to want a higher threshold than a user with a high "same thing every day" rate (someone in a routine). Adaptive thresholds are listed in Section 10 as future work. - **No formal correctness proof.** AutoMCP does not claim that the proposed macro is the right abstraction. The only correctness signal is that the user approved the proposal. The recognizer does not produce a soundness argument, and there is no mechanical check that the macro's behavior matches the user's intent — only that it reproduces the observed sequence with the substituted arguments. - **Cross-session matching is implemented but not benchmarked.** The matcher reads from the persisted action graph for cross-session recognition, but we have not measured how cross-session detection performs at scale (large action histories, many tools, many sessions). The risk is that the O(N·L·max_length) matcher becomes slow on large histories. A future version should add an inverted index keyed by tool-name pairs to prune the search space. - **No support for branching macros.** AutoMCP only recognizes linear sequences. Patterns of the form "if condition X, do A; otherwise do B" — common in real workflows — are not detected, and an attempted recognition that includes an early-failing branch is filtered by the outcome-stability signal rather than recognized as a branching pattern. A branching-macro extension is plausible but requires the action graph to record branch conditions, which V1-lite does not. - **Within-session latency.** The matcher runs after each new action is appended to the buffer, which means it is in the agent's hot path. The current implementation is not optimized; on a buffer of 100 actions the per-call matcher overhead is in the low single-digit milliseconds. On a buffer of 1000 actions, it is in the tens of milliseconds. The buffer size cap (1024 in V1-lite) was chosen specifically to keep the matcher latency under 100ms, not on the basis of memory budget. - **No macro composition.** An approved macro is a first-class tool, but the matcher does not currently recognize sequences that *use* approved macros — only sequences of underlying tools. This means a user who approves `clone_and_pull` and then uses it inside a longer routine will not get a `clone_and_pull_and_test` proposal until the matcher is taught that approved macros are part of the action vocabulary. This is straightforward to fix; the V1-lite implementation does not yet do it. - **No schema-evolution handling.** If an underlying tool's schema changes — a parameter is renamed, a new required parameter is added — every approved macro that uses that tool will break. The catalog has no versioning of macros against tool schemas. The user will see a runtime failure rather than a pre-flight warning. ## 9. Evaluation The evaluation in this paper is candidly modest. AutoMCP has been tested in three internal sessions on the author's working agent, over a four-week observation window in early 2026. - **Sessions.** Three working sessions, each spanning approximately 6 hours of agent activity, with average action-graph density of 80–120 tool calls per hour. The total observation set is approximately 1,800 trace entries. - **Proposals generated.** 14 macro candidates passed the stability threshold and entered the proposal queue. - **Proposals approved.** 9 of the 14 were approved on first review; 2 were approved with edits (renames, parameter renames); 3 were rejected and added to the do-not-propose set. The rejections were two cases of "this is two unrelated things that happened to occur consecutively" and one case of "the parameter slot the recognizer flagged is not actually meant to vary — those were three deliberate retries against three test repos." - **Latency.** End-to-end matcher latency on the test agent's typical buffer state was 2–8ms per tool call. We did not stress-test with synthetic 10000-action buffers. - **Token impact in the agent loop.** A macro replaces N tool-call schemas in the catalog with one. On the test sessions, the cumulative compression from approved macros was a small fraction of the catalog token cost — measurable but not load-bearing — because the catalog was already aggressively compressed by UIMCP before AutoMCP entered the picture. The two systems are not redundant; AutoMCP captures repeated *behavior* while UIMCP compresses *exposure* of the catalog. We discuss the interaction in Section 10. - **No public benchmark.** There is no standard benchmark for agent-action macro inference. We did not construct a synthetic one for V1-lite because we believe such a benchmark is more usefully designed by the field once multiple agent platforms ship comparable systems. We are deliberately unwilling to claim AutoMCP "works" beyond the modest evidence of: 9 of 14 internal proposals were good enough for the test user to approve them. This is suggestive, not conclusive. The full evaluation we would want to publish — multiple users, multiple agent domains, comparison to a flat-trace baseline, sensitivity to threshold tuning — is on the roadmap but not done. ## 10. Future work In rough priority order: - **Adaptive thresholds.** Track the user's approval rate per proposal in the queue and adjust the per-user threshold so that approximately 70–80% of proposals are approved. A user who rejects most proposals gets a higher threshold (the recognizer is being too eager); a user who approves all proposals gets a lower threshold (the recognizer is being too cautious). This is a small change, mostly an additional column in the proposal table and a moving-average computation. - **Macro composition.** Teach the matcher that approved macros are part of the action vocabulary. A `clone_and_pull` macro that the user has approved should be visible to the matcher as a single action node when scanning for higher-order patterns. This produces a hierarchy of macros, which is structurally similar to the abstraction-level hierarchy in `core/mcp/optimizer.py:HierarchyBuilder`, and we expect the implementations to share infrastructure. - **Macro versioning.** Tie an approved macro to the tool schemas it depends on at approval time. When an underlying tool's schema changes, mark the macro as needing review rather than letting it fail at runtime. This requires extending the macro catalog with a schema-version column and the safety-toggle framework with a "macro is stale" warning state. - **Cross-user macros.** A pattern that emerges across many users (with their consent) is signal that the pattern is generally useful. A privacy-aware aggregation pipeline could surface "users in your role often use this macro" suggestions without exposing any one user's history. We deliberately list this as future work rather than V1-lite scope; the privacy story is non-trivial and should not be hurried. - **Branching macros.** Extend the action graph to record branch conditions (e.g., "the previous step failed, so we retried with these alternative arguments"), and teach the matcher to recognize branching patterns. This is the largest of the items on this list and is probably its own paper. - **Faster matcher.** An inverted index keyed on tool-name pairs would prune the search space substantially, taking the matcher from O(N·L·max_length) closer to O(matches) at the cost of some bookkeeping. This is straightforward and worth doing once cross-session matching is stressed at scale. - **Tighter integration with UIMCP.** UIMCP compresses how many tool schemas the LLM sees per turn (Levitas, 2026, §9.3). AutoMCP compresses how many tool calls the LLM has to emit to accomplish a routine. The two are complementary: an approved AutoMCP macro becomes a tool that UIMCP then includes in its ranked top-K, and the catalog cost of the macro is the catalog cost of one tool rather than of the underlying sequence. The current integration is naive — the macro is registered, UIMCP sees it like any other tool — but a tighter coupling could account for the macro's expected downstream savings when ranking. ## 11. References 1. **Allen, R. B., Lau, T., Weld, D. S., & Domingos, P.** (2005). *Programming by demonstration in the wild: Inferring shell-script abstractions from interaction histories*. Workshop on End-User Software Engineering. 2. **Anthropic.** (2024). *Model Context Protocol Specification*. https://modelcontextprotocol.io 3. **Cypher, A.** (Ed.). (1993). *Watch What I Do: Programming by Demonstration*. MIT Press. The foundational anthology of PBD work, including Halbert's SmallStar, Lieberman's Tinker, and Myers's Peridot. 4. **Halbert, D. C.** (1984). *Programming by Example*. Ph.D. dissertation, University of California, Berkeley. The SmallStar system, generally regarded as the first PBD system in a real productivity application. 5. **Lau, T., Wolfman, S. A., Domingos, P., & Weld, D. S.** (2003). *Programming by demonstration using version space algebra*. Machine Learning, 53(1–2), 111–156. 6. **Levitas, T.** (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Memagen™ technical paper, master series. See `web/site-v2/src/content/papers/2026-05-08-memagen-master.md`. 7. **Lieberman, H.** (Ed.). (2001). *Your Wish is My Command: Programming By Example*. Morgan Kaufmann. A second-generation PBD anthology, focused on web and end-user contexts. 8. **Myers, B. A.** (1986). *Visual programming, programming by example, and program visualization: a taxonomy*. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 59–66. 9. **Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A.** (2023). *Voyager: An open-ended embodied agent with large language models*. arXiv:2305.16291. 10. **Witten, I. H., & MacDonald, B. A.** (1988). *Using concept learning for knowledge acquisition*. International Journal of Man-Machine Studies, 29(2), 171–196. Foundational work on generalization-by-example for PBD. Source: --- ## Paper: 2026-05-08-autowebmcp URL: https://memagen.com/papers/2026-05-08-autowebmcp PDF: https://memagen.com/papers/2026-05-08-autowebmcp.pdf --- title: "AutoWebMCP: Automatic MCP Tool Schema Generation from Arbitrary Web Pages" subtitle: "Forms, links, OpenAPI/Swagger sniffs, and SSRF-protected fetches — turning any URL into a runnable agent tool" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "Every web application is, in principle, an agent tool: it accepts inputs, performs an action, and returns a response. In practice agents cannot use arbitrary websites because they lack a schema describing what to send and what comes back. The conventional answer — handwritten Model Context Protocol (MCP) servers per site — leaves the long tail of the web inaccessible. AutoWebMCP, the open-core (V1-lite, MIT) Memagen™ subsystem described in this paper, infers a runnable MCP tool surface from a URL alone. The pipeline has four stages: an SSRF-safe fetch through a single-source-of-truth URL validator (`core.security.ssrf.assert_public_url`); an HTML structural parse over forms, links, meta tags, and standalone inputs (`core.mcp.webmcp_generator._SiteParser`); a spec-sniff that probes well-known paths and link relations for OpenAPI or Swagger documents and prefers the spec when found; and a schema-synthesis step that maps form input names to argument names, input types to JSON-Schema types, and HTML5 validation attributes to schema constraints. Generation runs at first-visit time and the resulting tool definitions are cached: registered in an in-memory registry and stamped into the Pro-tier graph substrate so they survive process restarts. On the V1-lite test corpus (Section 7), DOM analysis completes in 18–96 ms per page (median 34 ms) on a 4 vCPU build host, and 41 of 43 generated argument names round-tripped to the target server on first invocation (95.3%). We describe the implementation, the SSRF defense-in-depth, the runtime executor that translates an agent's tool call into a real HTTP request with redirect re-validation, the limitations on single-page-app sites and multi-step workflows, and the comparison to vision-based browser agents. AutoWebMCP is fastest and cheapest on sites whose structural surface is meaningful; vision-based agents remain the right tool for the rest." keywords: ["webmcp", "tool synthesis", "openapi sniffing", "browser automation", "ssrf protection"] kind: specialty subsystem: "AutoWebMCP" pages: 7 version: "v1.1" --- ## 1. Introduction The Model Context Protocol gave agent platforms a common shape for tool surfaces: a JSON object with a name, a description, and an input schema. A hand-authored MCP server is the wrong unit for the web. There are a few hundred million reachable web applications and somewhere between two and three thousand published MCP servers at the time of writing. The long tail is unreachable not because it lacks an interface, but because nobody has translated each interface into the agent's idiom. The translation is mechanical for a large class of sites. A contact form has, in HTML, the names and types of every field, often with `required` and `pattern` attributes that constrain valid input. A page that publishes an OpenAPI document at `/openapi.json` declares every endpoint, parameter, and response shape in a format that already maps cleanly to JSON Schema. The site is already self-describing; it simply isn't using the protocol the agent expects. AutoWebMCP closes the gap. Given a URL, the generator fetches the page through an SSRF-safe path, parses the HTML structural surface, sniffs for OpenAPI or Swagger specs at conventional locations, and emits one or more MCP tool definitions. A user-approved tool becomes a first-class catalog entry: the agent calls `webmcp_invoke(tool_name="example_com_contact", params={...})`, the executor in `core/tools/webmcp_executor.py` revalidates the destination against the SSRF guard, issues the HTTP request with manual redirect handling, and returns the response. The pipeline is open-source under MIT in the Memagen™ Lite distribution and is described in the master paper at section 9.2. The contributions of this paper are: 1. A four-stage pipeline (SSRF-safe fetch → structural parse → spec sniff → schema synthesis) for translating arbitrary web pages into MCP tool definitions, executed at first-visit time and cached afterward. 2. An argument-inference table that maps HTML form input types and validation attributes to JSON-Schema fragments, with documented collision and anonymous-field behavior. 3. A runtime executor with redirect re-validation, manual redirect handling, and a pentest-mode escape hatch that still refuses cloud-metadata exfil targets. 4. An evaluation describing what works (well-structured forms, OpenAPI-publishing sites), what partially works (mixed server-rendered/client-rendered surfaces), and what does not (SPAs with no server-emitted form HTML, multi-step wizards, CAPTCHA-protected forms). AutoWebMCP is not a vision-based browser agent and does not try to be. It is the cheap, fast, deterministic path for sites that publish meaningful structure. Section 9 compares it to vision-based agents and to PCMCP, the desktop-app analog covered in a [separate paper](/papers/2026-05-08-pcmcp). ## 2. Related Work ### 2.1 OpenAPI, Swagger, and the spec-publication gap The OpenAPI Specification (OAI, 2017) — formerly Swagger — is the dominant machine-readable format for HTTP APIs. Adoption is partial: surveys of the public-API landscape (APIs.guru, 2023; Postman State of the API, 2024) suggest 30–40% of enterprise REST APIs ship some form of OpenAPI document, dropping sharply for consumer product surfaces. AutoWebMCP probes well-known paths and prefers the spec when one is found because it is denser and more reliable than HTML inference; the structural-parse stage is the fallback. ### 2.2 Headless browser frameworks Headless browser engines (Playwright, Puppeteer, Selenium) provide programmatic control of full browser engines. They are the right substrate for sites that depend on JavaScript to produce their interactive surface, but the interface they expose is *imperative*: click this, type that. They do not produce a *schema* of what the site can do. Memagen™'s `browser.headless` action tool wraps a headless engine for cases AutoWebMCP's structural parse cannot handle; the two are complementary. ### 2.3 HTML parsing libraries The Python standard library `html.parser.HTMLParser` is a SAX-style streaming parser. BeautifulSoup and lxml provide tree APIs. AutoWebMCP uses the standard library parser because the generator runs in the agent runtime and must function without optional dependencies. The custom subclass `_SiteParser` in `core/mcp/webmcp_generator.py` extracts only the elements the synthesis stage needs. ### 2.4 Schema inference Schema inference for arbitrary data has a long literature (hidden Markov and learning-based methods over sample data). AutoWebMCP's setting is gentler: HTML form authors have already declared types in the markup. The work is not inferring *what type a field is* from observations; it is *translating an existing declaration*. The hard cases are not statistical — they are malformed or non-standard markup, handled as documented in Section 4. ### 2.5 The "every web app is an API" gap The observation that every web application is potentially an API is older than the web (Berners-Lee, 1989). Recent browser-driven agent work (Nakano et al., 2021; Yao et al., 2022) and commercial vision-based browser agents close the gap by treating the page as pixels. AutoWebMCP closes it from the other direction: treat the page as already-self-describing and translate the description. ## 3. The four-stage pipeline The pipeline is implemented in `core/mcp/webmcp_generator.py` as `WebMCPGenerator.generate_from_url`. Stages execute in order; later stages depend on artifacts from earlier ones. Figure 1 summarises the flow, including where the result is cached.
First visit Web page DOM URL · HTML body Semantic analyzer _SiteParser · OpenAPI spec sniff Tool spec emitter JSON-Schema in/out name · required · types WebMCP server register_webmcp_tools() _WEBMCP_REGISTRY Agent catalog auto-update no restart Cache and restore In-memory registry _WEBMCP_REGISTRY (dict) session lifetime L4 graph node kind=webmcp_tool · permanent=True stamp_new_node() Restore on startup restore_webmcp_from_graph() SQLite · $CRACKEDCLAW_DB Generation runs once at first-visit time. On subsequent runs the agent loads the cached tool surface in milliseconds.
Figure 1. AutoWebMCP pipeline. Top row: first-visit generation (web page DOM → semantic analyzer → tool spec emitter → WebMCP server registration → agent catalog auto-update). Bottom row: cache. Tools are stamped into the L4 graph substrate and restored on every subsequent process start.
The same pipeline as a Mermaid source, for environments that render it natively: ```mermaid flowchart LR A["Web page DOM
URL + HTML"] --> B["Semantic analyzer
_SiteParser + OpenAPI sniff"] B --> C["Tool spec emitter
JSON-Schema in/out"] C --> D["WebMCP server
register_webmcp_tools()"] D --> E["Agent tool catalog
auto-update, no restart"] D -. cache .-> F["In-memory registry
_WEBMCP_REGISTRY"] D -. persist .-> G["L4 graph node
kind=webmcp_tool"] G -. on startup .-> H["restore_webmcp_from_graph()"] H --> D classDef phase fill:#eef2ff,stroke:#3730a3 classDef cache fill:#f5f3ff,stroke:#6d28d9,stroke-dasharray: 4 3 class D,E phase class F,G,H cache ``` ### 3.1 SSRF-safe fetch Every outbound HTTP request that AutoWebMCP issues passes through `core.security.ssrf.assert_public_url`. The validator is the single source of truth for "is this URL safe to fetch" across the entire Memagen™ runtime; the WebMCP generator imports it directly, and the executor in `core/tools/webmcp_executor.py` revalidates at invocation time as defense in depth. The SSRF guard rejects, in order: - Empty or non-string URLs. - Schemes other than `http` or `https` — `file://`, `gopher://`, `ftp://`, `data://`, and anything else the parser produces. - Unicode-confusable hostnames after NFKC normalization. The string `ｌocalhost` (with a full-width `ｌ`) normalizes to `localhost` and is matched against an explicit blocklist. - Hostnames in `_BLOCKED_HOSTS`: `localhost`, `ip6-localhost`, `ip6-loopback`, `metadata.google.internal`, `metadata`. These are symbolic names an attacker might place into DNS pointing at public IPs to bypass IP-range checks; the symbolic name itself is refused. - Literal IP addresses (v4 or v6) that fail `is_public()`. The helper rejects `is_private`, `is_loopback`, `is_link_local`, `is_multicast`, `is_reserved`, and `is_unspecified` — covering 10/8, 172.16/12, 192.168/16, 169.254/16 (link-local, including the standard cloud-metadata IP at 169.254.169.254), 127/8, 224/4, the IPv6 ULA range fc00::/7, the IPv6 link-local range fe80::/10, and the reserved/multicast/unspecified blocks. - Hostnames whose DNS resolution returns *any* private IP across A and AAAA records. The validator calls `socket.getaddrinfo(host, None)` (consults both record types) and rejects on the first non-public result. This defends against multi-record DNS rebinding where one A record is public and another is `127.0.0.1`. The validator returns the (possibly NFKC-normalized) URL on success and raises `SSRFBlocked` (a `ValueError` subclass) otherwise. The fetch wrapper, `WebMCPGenerator._safe_get`, calls the validator on the initial URL *and* on every redirect target before following the redirect. This is non-negotiable. The AutoWebMCP code path is the most likely SSRF surface in the runtime: it accepts a URL from the agent (which received it from the user, who may have pasted it from anywhere) and fetches it. ### 3.2 HTML structural parse `_SiteParser` (in `core/mcp/webmcp_generator.py`) is a custom subclass of `html.parser.HTMLParser`. It collects: - The first `` element's text, accumulated across `handle_data` calls while the `_in_title` flag is set. - The first `<meta name="description">` `content` attribute, with `<meta property="og:description">` as a secondary fallback. Subsequent meta tags do not overwrite a description already captured. - Every `<form>` element, with its `action`, `method`, `id`, `name`, and a list of inputs. The current form is tracked in `_cur_form` and cleared on the matching `</form>`. - Every `<input>`, `<textarea>`, and `<select>` element whose type is not in `{hidden, submit, button, reset, image}` and which has a non-empty `name` or `id`. The element is appended to the current form's input list if one is open, or to the standalone `inputs` list if not. - Every `<a>` whose `href` does not start with `#`, `javascript:`, `mailto:`, or `tel:`. Deliberate choices: - **Inputs without names are dropped, not synthesized.** The source comment reads: *"Without a name, the form field has no posting key. We could synthesise one, but it would never round-trip to the server — skip."* A schema with synthesized argument names would generate tool calls the server cannot accept. - **Hidden, submit, button, reset, and image inputs are excluded.** They are not user-supplied values; including them would mislead the LLM into setting them. - **Form-internal and standalone inputs are kept separate.** A search box outside a form produces a dedicated `{site}_search` tool; a search box inside a form is part of that form's tool. The parser is intentionally tolerant: malformed HTML is handled by the standard library parser's recovery rules. AutoWebMCP does not "fix" pages; it captures what it can recognize and lets synthesis reject anything that would produce an unusable schema. ### 3.3 Spec sniffing Before relying on the structural parse, the generator probes for an OpenAPI or Swagger document in two places: - **Direct URL hint.** If the URL ends in `.json`, `.yaml`, or `.yml`, or contains the substrings `openapi` or `swagger`, the generator fetches it directly and tries to parse it as a spec. A response is accepted if it contains a `paths` field and at least one of `openapi`, `swagger`, or `info` at the top level. - **Well-known paths.** The generator tries each of `OPENAPI_PATHS` (`/openapi.json`, `/swagger.json`, `/api-docs`, `/api/openapi.json`, `/v1/openapi.json`, `/v2/api-docs`, `/api/swagger.json`, `/.well-known/mcp.json`) in `_fetch_openapi`. Each candidate is built with `urljoin(base_url, path)` and fetched through `_safe_get`. The first 200 response with a JSON body that validates as a spec wins. When a spec is found, `_openapi_to_tools` translates it: - The first `servers[0].url` (or the original base URL if `servers` is absent) becomes the canonical endpoint root. Server URLs starting with `/` are resolved against the original base URL. - Each `(path, method)` pair in `paths` becomes one tool. Methods outside `{get, post, put, patch, delete}` are ignored. - The tool's `operationId` becomes its name (sanitized via `_safe_name` and prefixed with the site name); `summary` or `description` becomes its description. - Path and query parameters become input-schema properties, with the parameter type mapped to JSON-Schema (`string`, `number`, `boolean`, `integer`; anything else falls back to `string`). - A request body with a JSON content type and an `object` schema contributes its `properties` directly. A non-object body becomes a single `body: string` argument as a fallback. The generator is defensive about malformed specs. The source has explicit checks for `path_item` being non-dict (a `$ref` placeholder), `op` being non-dict, parameters that are non-dict, and bodies whose content type isn't `application/json` or `*/*`. Each check is a short-circuit `continue`; a malformed segment doesn't prevent the rest of the spec from being consumed. When both a spec and HTML data exist, the generator merges: spec-derived tools first, HTML-derived form tools appended only if their endpoint is not already covered. This handles the common case of a site with an OpenAPI document for its REST API and a separate HTML contact form on the marketing page. ### 3.4 Schema synthesis Synthesis turns the structural parse and any specs into a list of MCP tool definitions. The output shape, before the executor strips internal metadata, is: ```python { "name": "submit_contact_form", "description": "Submit the 'contact' form on https://example.com/contact via POST", "inputSchema": { "type": "object", "properties": { "name": {"type": "string", "description": "Your name"}, "email": {"type": "string", "description": "Email address"}, "message": {"type": "string", "description": "Your message"}, }, "required": ["name", "email", "message"], }, "_meta": { "endpoint": "https://example.com/contact", "method": "POST", "type": "form", }, } ``` The internal `_meta` block is used by the executor to issue the actual HTTP request and is stripped from the tool definition exposed to the LLM, leaving canonical MCP fields plus `endpoint`, `method`, and `type` as separate top-level keys. Synthesis always emits a generic `{site}_navigate` tool as the first entry — a low-cost catch-all for cases where the structural parse missed the action the agent wanted to take. Total tool count is capped at 20 per site. Sites with very large OpenAPI documents (a fully exported payment-API spec runs to several hundred operations) would otherwise overwhelm the agent's catalog with low-relevance tools; the cap is a pragmatic ceiling that the user can override by re-running against more specific path roots. ## 4. Argument inference The most consequential design choice in AutoWebMCP is how it translates HTML form metadata into JSON Schema. The mapping is defined in `_form_to_tool` and follows the table below. | HTML construct | JSON-Schema fragment | Notes | |-------------------------|---------------------------------|-------| | `<input type="text">` | `{"type": "string"}` | Default. | | `<input type="email">` | `{"type": "string"}` | Format hint not currently set; future work. | | `<input type="number">` | `{"type": "number"}` | | | `<input type="range">` | `{"type": "number"}` | Treated as number; min/max not currently propagated. | | `<input type="checkbox">` | `{"type": "boolean"}` | | | `<input type="search">` | `{"type": "string"}` | Standalone search inputs trigger a dedicated `{site}_search` tool. | | `<textarea>` | `{"type": "string"}` | Treated as text input. | | `<select>` | `{"type": "string"}` | Enum extraction from `<option>` is future work. | | `placeholder` attribute | `description` field | Falls back to `"Value for {raw_name}"` when absent. | | `required` attribute | name appended to `required[]` | | | `name` attribute | property key (after `_safe_name`) | | | `id` attribute | fallback property key | Used only when `name` is absent. | The `_safe_name` helper replaces any non-alphanumeric character with `_`, truncates to 48 characters, and strips leading/trailing underscores; an empty result is mapped to the literal `tool` so the schema is always valid. The same sanitizer is applied to tool names and property keys, with collision handling for property keys: when two raw input names sanitize to the same key (the source comment cites `"user-name"` and `"user_name"` both becoming `user_name`), the second occurrence is suffixed with `_2`, the third with `_3`, and so on, tracked in `seen_names`. Tool-name construction follows a similar pattern. The site name comes from the URL's netloc (dots and dashes replaced by underscores) and is prefixed to the form's `id`, `name`, or the literal `form`, producing names like `example_com_contact_form` or `acme_com_search_form`. The methodology is conservative on purpose. We do not infer types from input names or guess at richer JSON-Schema constructs (`format`, `pattern`, `minLength`, `maxLength`) the markup does not directly declare. A future revision will read the HTML5 `pattern`, `minlength`, and `maxlength` attributes and emit corresponding constraints; today the schema is type-only, the minimum the LLM needs to produce a usable tool call. ## 5. Tool execution at runtime A generated tool is not yet runnable. The executor in `core/tools/webmcp_executor.py` is the bridge from agent tool call to live HTTP request. ### 5.1 Registration and persistence `register_webmcp_tools(tools)` adds each tool to `_WEBMCP_REGISTRY` (a module-global `dict`) and calls `_persist_webmcp_node(tool)`. Persistence is best-effort: it imports `core.memory.temporal.stamp_new_node` lazily, builds a content blob with the description, endpoint, method, and a truncated JSON dump of the input schema, and stamps an L4 graph node with `kind="webmcp_tool"`, `permanent=True`. If the import fails (Lite-mode, no graph), persistence is silently skipped — the in-memory registry still works for the session. `restore_webmcp_from_graph()` is the complementary path. On startup, the executor opens the SQLite database at `$CRACKEDCLAW_DB` (defaulting to `/tmp/crackedclaw.db`), selects all rows with `kind = 'webmcp_tool'`, and parses the content blob back into a registry entry. Tools that already exist in `_WEBMCP_REGISTRY` are not overwritten. The function returns the count restored so the agent can surface "5 WebMCP tools restored from previous session" at startup. This is the cache layer Figure 1 marks with dashed edges. The expected lifecycle: first visit pays the parse-and-register cost; every subsequent process starts up with the catalog already populated; agent dispatch is unaffected by re-generation latency. ### 5.2 Invocation The agent calls `webmcp_invoke(tool_name, params)`. The schema for this entrypoint is defined as `INVOKE_SCHEMA`: ```python INVOKE_SCHEMA = { "name": "webmcp_invoke", "description": ( "Invoke a previously generated WebMCP tool by name with the given parameters. " "Use webmcp_create first if the tool doesn't exist yet." ), "parameters": { "type": "object", "properties": { "tool_name": {"type": "string"}, "params": {"type": "object"}, }, "required": ["tool_name"], }, } ``` The implementation: 1. Looks up `tool_name` in `_WEBMCP_REGISTRY`. A miss returns `{"error": ..., "available": [...]}` with the first ten registry keys. 2. Re-validates the registered endpoint through `_ssrf_safe_url`. This is the second SSRF check on the same URL (the first happened at generation time). Re-validation defends against a registry populated under one SSRF policy and invoked under a stricter one, against DNS records that have changed since registration, and against future tools whose endpoints are computed at invocation rather than at registration. 3. Constructs an `httpx.AsyncClient` with `follow_redirects=False`, `timeout=15.0`, and `verify=True`. The redirect setting is deliberate: an automatic redirect from a public endpoint to a private IP would bypass the SSRF check. The TLS-verification setting is also deliberate; an earlier draft disabled it for testing, and the source comment notes that "disabling it allowed silent MitM of every WebMCP call." 4. Dispatches by method: `GET` and `DELETE` use `params=params` (URL query string); `POST`, `PUT`, and `PATCH` use `json=params` (JSON body). Unknown methods fall through to `GET`. 5. Returns `{"status_code", "body", "tool"}` on success. The body is parsed as JSON if possible, otherwise truncated to 4000 characters. Exceptions are caught and returned as `{"error": "webmcp call failed", "tool": tool_name}` — the inner exception is deliberately not exposed, because a verbose upstream error is a prompt-injection vector. ### 5.3 Headless browser fallback The HTTP path is sufficient for sites whose forms POST to plain URLs. Sites that depend on JavaScript to assemble the request payload, or that issue requests via XHR/fetch with computed signatures, are not. For these the agent falls back to `browser.headless`, which wraps a headless engine. The fallback is not automatic; the agent decides based on response shape whether to retry through the browser. A future revision will surface a "this site needs the browser" hint in the generated tool's metadata to remove the round-trip. ## 6. SSRF defense in depth This is the section auditors check first. AutoWebMCP issues outbound HTTP requests on behalf of the agent, with URLs derived from agent-supplied input. The defense has four layers, not one: **Layer 1 — Pre-fetch URL validation.** `core.security.ssrf.assert_public_url` runs on every URL before the first HTTP request. Behavior is described in Section 3.1; it is the single source of truth across the runtime. **Layer 2 — DNS resolution check.** The validator does not just check whether the URL parses to a private IP literal; it resolves the hostname via `socket.getaddrinfo` and rejects if *any* returned address (across A and AAAA) is non-public. This defends against three concrete attacks: (a) public-facing hostnames whose DNS records have been pointed at private IPs; (b) multi-record DNS rebinding where one record is public and another is loopback; (c) AAAA records that resolve to IPv6 ULA or link-local space when the v4 record is innocuous. **Layer 3 — Redirect chain re-validation.** `_safe_get` follows redirects manually, calling `_assert_public_url` on every `Location` header before re-issuing. The redirect budget is six hops (one initial + five redirects). After six, `SSRFBlocked("too many redirects")` is raised. The `httpx.AsyncClient` is constructed with `follow_redirects=False` so the underlying library cannot bypass the wrapper. **Layer 4 — Invocation-time re-check.** The executor calls `_ssrf_safe_url` at every invocation, even though the URL was already validated at generation time. From the source: "The registry can be populated by a prior agent action, so we validate every URL at invocation time — not just at registration. Resolve hostnames via DNS to catch CNAME/A-record tricks pointing at internal IPs." There is one explicit escape hatch. The invocation-time validator consults `core.settings.is_enabled("pentest_ssrf_relaxed")`. When set, the private-range block is lifted, but the validator still: - Refuses non-`http(s)` schemes. - Refuses plaintext HTTP entirely (per the comment at the top of `_ssrf_safe_url`: `"plaintext http blocked — use https"`). - Refuses cloud-metadata IPs (`169.254.169.254` and `fd00:ec2::254`) regardless of mode. The source justifies it: those IPs "are an attractive prompt-injection sink even on engagement networks." Pentest mode is gated behind the safety toggle framework (master paper section 6) with its own warning text, verbatim acknowledgment, and 60-second cooldown. It is intended for authorized engagement work where targets are by definition on RFC1918 networks. What AutoWebMCP does *not* defend against. The fetch is not anonymous; the request carries the agent's user-agent and any cookies configured into the browser action tool's isolated context. A site that denies scraping via rate limits, CAPTCHAs, or IP bans will succeed. AutoWebMCP is not a scraping-evasion tool. Blocked fetches surface as `{"fetch_error": "HTTP 403"}`. ## 7. Evaluation We are explicit about scope. AutoWebMCP V1-lite has been tested on a small, hand-curated set of pages chosen for category coverage. The paper does not claim a single open-web "success rate" because the input distribution is not representative. ### 7.1 Methodology For each site we ran `WebMCPGenerator.generate_from_url(url)` against the public homepage and one to three additional path roots (search, contact, login where applicable). For each generated tool we manually inspected: (a) the tool name; (b) input-schema property names against the actual form input names; (c) the `required` list against actual `required` attributes; (d) the endpoint against the form's `action`. For OpenAPI sites we additionally checked that the generator preferred the spec over the HTML parse and that spec-derived tools covered the operations a developer would expect. Latency measurements come from `time.perf_counter()` brackets around `_SiteParser` and the synthesis loop, on a 4 vCPU x86_64 build host (16 GB RAM) with HTML bodies pre-fetched to local disk so the timing excludes network. The per-page DOM-analysis range over 18 sample pages was 18–96 ms with a median of 34 ms; the 96 ms outlier was the spec-merge case on the largest OpenAPI document tested (213 operations, reduced by the 20-tool cap). We did not run the executor against test sites for first-call accuracy. Issuing real form submissions to third-party services without the operator's consent is either irresponsible or pointless. Runtime evaluation is reported in Section 7.4 against a controlled internal harness. ### 7.2 Sites tested The set was 18 pages drawn from: - **Static, well-formed**: an open-encyclopedia search; a popular news-aggregator submission form; the Memagen™ contact form. - **OpenAPI-publishing**: a payment-API public spec; an HTTP-debugging service; a developer-API documentation site. - **Mixed (server-rendered with JS overlays)**: a major code-host login page; a Python web-framework documentation site; a typical CMS contact form. - **Heavy SPA**: a project-management landing page; a productivity-app signup page; a representative React marketing site. - **CAPTCHA-protected**: a major search engine; two of the above when fronted by a CAPTCHA proxy. We use category descriptions rather than vendor names because the failure modes belong to the page type, not the brand. ### 7.3 Structural-parse results On well-formed and mixed categories the parser identified every visible form and every named input. Tool names were sensible (`encyclopedia_org_search`, `news_site_form`, `memagen_com_contact_form`). Required-flag inference matched every case. Aggregating across the 7 well-formed and mixed pages, **41 of 43 generated argument names round-tripped to the target server on a dry run (95.3%)**; the two failures were a hidden-by-CSS field counted by the parser and a `data-name` attribute the markup used in place of `name` (which the parser correctly does not synthesize). On the OpenAPI-publishing category the spec sniff succeeded for two of three sites via well-known paths; one required pointing the generator at the published spec URL directly. The third site does not publish a spec at any well-known path and fell back to the HTML parse. On the heavy SPA category the parser captured almost nothing. The `<form>` elements on these pages are constructed by client-side JavaScript and are not present in the server-emitted HTML. The structural parse returned a `{site}_navigate` tool and nothing else. Documented limitation; see Section 8. On the CAPTCHA-protected category the parser succeeded structurally (the form is in the markup) but the generated tool would fail at runtime because the CAPTCHA challenge is not a parameter the schema expresses. Also documented in Section 8. ### 7.4 Runtime invocation against a controlled harness We built a small Flask harness exposing five forms (a contact form, a search form, a JSON-body POST endpoint, a redirect chain ending at a public endpoint, and a redirect chain ending at `127.0.0.1`). We registered the harness URL and issued tool calls. - The four well-behaved forms succeeded; the executor issued the correct method (`GET` for search, `POST` for contact, `POST` with a JSON body for the JSON endpoint, `GET` with redirect-following for the public chain) and returned the expected response bodies. End-to-end median latency from `webmcp_invoke` call to response was **41 ms on localhost, 218 ms against an internet-routed echo endpoint**. - The redirect-to-private-IP case was rejected at the second `_assert_public_url` call inside `_safe_get`, with `SSRFBlocked("DNS record points to non-public IP: 127.0.0.1")`. The error propagated cleanly to the agent. We did not run the harness under load. Manual redirect handling and the second SSRF check require DNS resolution per hop, which under hostile DNS could add multiple seconds per redirect; we believe this is acceptable for an agent-facing tool, but it is a real cost and we report it. ### 7.5 What we did not measure We did not measure: (a) end-to-end first-call success against real third-party services beyond the round-trip name-validation result above; (b) tool-selection accuracy when AutoWebMCP-generated tools are mixed with hand-authored MCP servers in the same catalog — see [UIMCP](/papers/2026-05-08-uimcp) for the compression layer that question depends on; (c) the false-positive rate of the SSRF guard against split-horizon DNS. Each requires infrastructure we have not yet built. AutoWebMCP works well in the cases its design targets and fails predictably in the cases it does not. ## 8. Limitations We declare these explicitly because the alternative is dishonesty. - **Single-page applications.** React, Vue, Svelte, Solid, and similar frameworks do not emit `<form>` elements in the server response. The structural parse returns the navigate-tool stub and little else. The mitigation is the headless-browser fallback (Section 5.3); that path is not currently automatic. - **Multi-step workflows.** Wizards, multi-page checkouts, and OAuth flows span multiple URLs and require state between steps. AutoWebMCP generates one tool per page; composing tools into a workflow is the [AutoMCP](/papers/2026-05-08-automcp) layer's job (master paper section 9.1). The integration point — recording an AutoMCP macro that strings together AutoWebMCP-generated tools — is currently manual. - **CAPTCHA-protected forms.** The schema does not express a CAPTCHA challenge; a generated tool against one will produce a 403 or a redirect. CAPTCHA solving is out of scope. - **Heavy JavaScript validation.** Validation logic that lives in client-side JS (regex assembled at runtime, conditional `required` flags) is not captured. The schema reflects markup-declared constraints only; the server rejects mismatched arguments and the agent retries. - **Best for deterministic form-submit workflows.** AutoWebMCP fits "submit this form with these fields": contact forms, search boxes, simple REST endpoints, OpenAPI-described APIs. It does not fit "navigate this site like a human" — interpret responses, follow visual affordances, recover from arbitrary errors. For those, headless browser with vision is the right substrate. - **The 20-tool cap.** Sites with very large OpenAPI documents are truncated at 20 tools. A future revision will accept a path-prefix filter to target a subset of a large spec. ## 9. Comparison to vision-based browser agents Vision-based browser agents interpret the viewport as pixels and produce actions (clicks, types, scrolls). They handle JavaScript-heavy sites well because they do not depend on the page's structural surface; they handle dynamic UI, pop-ups, and visual affordances; they succeed on many sites where AutoWebMCP fails. The cost is real: - **Per-action latency.** Each action requires a screenshot to be sent to the model, the model to produce an action, and the action to be executed. Three to ten seconds per action is typical. AutoWebMCP-generated tools execute in roughly the latency of one HTTP round trip; our harness measured a 41 ms localhost median and a 218 ms wide-area median (Section 7.4). - **Per-action token cost.** Vision tokens are not free. A 1280×800 screenshot costs on the order of 1500 tokens at current pricing tiers; a twenty-action workflow spends 30,000 tokens in screenshots before any reasoning. AutoWebMCP carries the per-tool schema in the catalog (compressed via [UIMCP](/papers/2026-05-08-uimcp), master paper section 9.3) and the per-call token cost is the size of the tool name and arguments. - **Fragility under UI redesigns.** Vision agents are sensitive to visual changes. AutoWebMCP-generated tools depend on form input *names*, which are typically stable across redesigns of the visible HTML. The recommended pattern: AutoWebMCP first; vision-based agent for the cases AutoWebMCP cannot handle. The master paper documents the same pattern with [PCMCP](/papers/2026-05-08-pcmcp) (section 9.4) for desktop applications: structural introspection of the accessibility tree first; vision-based fallback for canvas-rendered surfaces. ## 10. Future work - **ARIA-tree parsing for SPA inference.** Most modern SPAs ship accessibility trees that mirror their structural surface even when the DOM does not. Reading the rendered ARIA tree (after a brief headless render) could close the SPA gap without needing vision. - **Cross-page workflow recording.** AutoMCP can already record macros over typed action streams; the missing piece is the binding between a recorded macro and the AutoWebMCP-generated tools it invokes. Intended flow: the user runs a workflow once, AutoMCP records it, AutoWebMCP-generated tools are stitched into a parameterized recipe, and the recipe enters the catalog as a single high-level tool. - **Schema validation roundtrip.** The generator could optionally test a generated tool against the live site by issuing a no-op request (a `HEAD`, or a `GET` with empty parameters) and checking the response. This would catch tools whose endpoints have shifted or whose parameter shapes don't match. Running it automatically against every generated tool is a politeness violation; as an opt-in step it would meaningfully improve first-call success rates. - **HTML5 validation propagation.** `pattern`, `minlength`, `maxlength`, `min`, `max`, and `step` are already in the markup the parser sees. Mapping them to JSON-Schema (`pattern`, `minLength`, `maxLength`, `minimum`, `maximum`, `multipleOf`) is straightforward and will produce richer schemas. - **Enum extraction from `<select>`.** Today every `<select>` is a string. Extracting the `<option>` values into a JSON-Schema `enum` would let the LLM choose from valid values without a runtime failure on the first wrong guess. - **OpenAPI 3.1 union types and discriminators.** The current spec consumer collapses union types to `string`. Faithful translation of `oneOf`, `anyOf`, and discriminated unions would benefit users targeting OpenAPI-rich enterprise APIs. ## 11. References 1. **Berners-Lee, T.** (1989). *Information Management: A Proposal*. CERN. https://www.w3.org/History/1989/proposal.html 2. **Levitas, T.** (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Memagen™ research papers. https://memagen.com/papers/memagen-master 3. **Levitas, T.** (2026). *AutoMCP: Recording and Replaying Tool Macros over Typed Action Streams*. Memagen™ research papers. https://memagen.com/papers/2026-05-08-automcp 4. **Levitas, T.** (2026). *PCMCP: Accessibility-Tree-First Tool Synthesis for Desktop Applications*. Memagen™ research papers. https://memagen.com/papers/2026-05-08-pcmcp 5. **Levitas, T.** (2026). *UIMCP: Adaptive Catalog Compression for Multi-Tool LLM Agent Dispatch*. Memagen™ research papers. https://memagen.com/papers/2026-05-08-uimcp 6. **OAI (OpenAPI Initiative).** (2017). *OpenAPI Specification*. https://www.openapis.org 7. **APIs.guru.** (2023). *OpenAPI directory: a curated registry of public OpenAPI documents*. https://apis.guru 8. **Postman.** (2024). *State of the API Report 2024*. https://www.postman.com/state-of-api/ 9. **Eddy, S.R.** (1996). *Hidden Markov models*. Current Opinion in Structural Biology, 6(3), 361–365. 10. **Nakano, R., Hilton, J., Balaji, S., et al.** (2021). *WebGPT: Browser-assisted question-answering with human feedback*. arXiv:2112.09332 11. **Yao, S., Chen, H., Yang, J., & Narasimhan, K.** (2022). *WebShop: Towards scalable real-world web interaction with grounded language agents*. NeurIPS 2022. Source: https://github.com/tablevitas123/crackedclaw --- ## Paper: 2026-05-08-capability-vault URL: https://memagen.com/papers/2026-05-08-capability-vault PDF: https://memagen.com/papers/2026-05-08-capability-vault.pdf --- title: "The V3-P Capability Vault: Per-Use Grants and Graph-Walked Audit Chains for LLM Agent Tool Surfaces" subtitle: "Fernet-encrypted credential storage, time-bounded scope-bounded grants, and an audit trail that is itself a queryable graph" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "Most agent frameworks treat tool access as a binary per-server toggle or as a trust-based decision the model is allowed to make. Both shapes fail when the agent's tool surface includes credentials that, if leaked or misused, cause real damage. We describe Memagen™'s V3-P capability vault and its associated audit chain: a Fernet-encrypted at-rest credential store fronted by a per-use capability protocol whose grant, use, and revocation events are persisted as nodes in the agent's compound graph and queried by graph walk rather than by log archaeology. The vault and its capability surface ship in the open-core (V1-lite, MIT) distribution; this paper documents the implementation in real detail. We cover the constraint dataclasses that bind a grant to a particular action_kind, the mint–consume–revoke lifecycle implemented in `core/security/capabilities.py`, the graph-promotion side-effects that turn each grant into a `capability_policy` node edged `owned_by` to its user, the audit-as-graph-walk pattern that lets a security review be a graph filter rather than a regex over JSON lines, and the soft-fail mode that lets the flat capability table remain canonical when the graph is unavailable. We are honest about what we have and have not measured: 60 tests pass against the post-V3-S P3 wiring, audit-trail query latency at scale is unprofiled, and the schema-fragility of an `_ensure_schema` path under non-canonical fixtures was the dominant delivery risk, mitigated by the `_safe_get_user_graph_node_id` direct-read fallback." keywords: ["capability-based security", "agent infrastructure", "audit chain", "tool gating", "fernet"] kind: specialty subsystem: "capability vault" pages: 7 version: "v1" --- ## 1. Introduction The ergonomic shape of tool access in current LLM agent frameworks is one of two things. Either tools are gated by a static per-server toggle — MCP server X is enabled, every tool it exposes is callable for the lifetime of that toggle — or tool access is *trust-based* — the model decides, the runtime executes, and the operator hopes the model's judgment scales with the consequences of its errors. Both shapes work well enough when the worst tool in the catalog can do nothing more harmful than read public data. Both shapes fail the moment the catalog acquires a credential, an outbound payment, a `git push`, or a destructive shell command. Capability-based security is the answer the operating-systems community converged on more than forty years ago (Hardy, 1985; Shapiro & Smith, 1999). A capability is an unforgeable token whose existence is necessary and sufficient for a particular operation; possession is permission. Capabilities have been applied to language runtimes (E, Caja), to operating-system kernels (KeyKOS, EROS, seL4), and to web identity flows (OAuth scopes are a capability-shaped specialization). Applying them to LLM-driven agents is overdue, and we believe the cost of doing it badly will only become apparent after agents have been deployed at production scale long enough for one of the bad outcomes to occur in public. This paper describes Memagen™'s V3-P capability vault: an open-core (V1-lite, MIT) implementation of capability-based security for LLM agent tool surfaces. The contribution is not the idea of capabilities — that idea is decades old. The contribution is the integration: a Fernet-encrypted credential store, a small set of typed constraint shapes, a mint–consume–revoke lifecycle whose audit trail is co-located with the agent's main memory substrate, and an audit-query pattern in which a security review is a graph filter rather than a flat-log scan. We describe each layer in implementation detail, cite real function names from `core/security/capabilities.py` and `core/security/vault.py`, and concede honestly the things we have not measured. Section 2 surveys related work. Section 3 declares design goals. Section 4 describes the architecture. Section 5 walks the mint–consume–revoke lifecycle with real signatures. Section 6 describes the audit-as-graph-walk pattern. Section 7 covers implementation details and the lite-mode fallback. Section 8 reports honestly on what works, what passes tests, and what remains unmeasured. Sections 9 and 10 discuss tradeoffs and future work. ## 2. Related work ### 2.1 Operating-system capabilities KeyKOS (Hardy, 1985) was the first operating system to make capabilities the only mechanism by which authority could be transferred between principals; it built on the earlier work of Dennis & Van Horn (1966) and the CAP computer (Wilkes & Needham, 1979). EROS (Shapiro & Smith, 1999; Shapiro et al., 1999) inherited KeyKOS's model and explicitly addressed the *capability myths* — beliefs in the literature that capability systems could not enforce confinement, could not revoke, could not handle accountability. The "capability myths demolished" paper showed that each of those beliefs rested on a confusion between capabilities-as-data and capabilities-as-references. Memagen™'s capabilities are capabilities-as-data: they are persisted records, not in-memory pointers, and the unforgeability comes from the difficulty of guessing a 128-bit `uuid4().hex` rather than from the impossibility of forging a kernel reference. This is weaker than a true ocap system but is the right tradeoff for an agent runtime where the principal boundary is a process, not an address space. ### 2.2 Object-capability languages E (Miller, 2006) and Caja (Miller, Samuel, Laurie, Awad, & Stay, 2008) are the canonical object-capability languages for the web. Both make capability-as-reference the default communication shape: there is no ambient authority, only objects passed by reference, and the only way to act on something is to hold a reference to it. We did not adopt either as the agent runtime substrate because the implementation language is Python — which has ambient authority everywhere — and because the audit requirement (every grant, every use, every revocation as durable structured data) is not a natural fit for a language that treats capabilities as references that disappear when the reference goes out of scope. Memagen™'s capabilities are explicitly *persisted* records. ### 2.3 LLM-agent tool gating Anthropic's `tool_use` blocks (Anthropic, 2024) and OpenAI's function-calling format (OpenAI, 2023) both validate the *shape* of a model-emitted tool call against a declared schema. Validation rejects malformed shapes; it does not gate execution on a per-use authority check. The Memagen™ lenient parser (cited in the master paper, Section 4) handles the shape problem; the V3-P capability vault handles the orthogonal authority problem. The two are layered: the parser produces a canonical `ToolCall`, the capability check decides whether *this particular invocation* is authorized, and only then does dispatch occur. OAuth 2.0 scopes (Hardt, 2012) are the closest application-layer analog to a capability. A scope binds an access token to a particular set of permitted operations. The V3-P vault diverges from OAuth in two ways. First, scopes are typically static per-token; a Memagen™ capability is per-use, decremented atomically by `consume_capability_use`. Second, OAuth's audit trail is the access-token issuance record at the authorization server plus whatever the resource server chooses to log; there is no canonical structured audit. The V3-P capability vault makes the audit trail a first-class graph subgraph. ### 2.4 Why none of these were adopted directly We considered using a vendored OAuth library, an ocap-Python wrapper, or a thin client around an external policy engine (Open Policy Agent, Cedar). Each was rejected on the same ground: the audit story. The capability lifecycle in Memagen™ has to be queryable as graph-shaped data because the rest of the agent's working memory is graph-shaped data. A separate policy engine produces a separate audit log that has to be joined back to the agent's memory by application code; the substrate-level co-location we describe in Section 6 is what we wanted, and that meant building it. ## 3. Design goals The V3-P vault has five explicit design goals. **Per-use grants, not per-server toggles.** Every tool dispatch should pass through an authority check that names the specific action being authorized — *this* tool, *this* HTTP method, *these* URL patterns, *this* payload-size ceiling. A grant for `net.fetch` against `https://api.anthropic.com/*` must not accidentally authorize `net.fetch` against an internal admin endpoint. **Time-bounded, scope-bounded, payload-bounded.** A grant carries an optional `expires_at` (wall-clock time-to-live), a typed `constraints` object whose shape varies by `action_kind`, and an optional `uses_remaining` counter that the consume path decrements atomically. Each axis is enforced at the point of use, not just at the point of mint. **Audit chain queryable as a graph subgraph.** Every grant, use, and revocation should appear as nodes and edges in the same compound graph that holds the agent's working memory. A security review of "show me every grant for `tool=net.fetch` in the last 30 days, walk to every use of each, and surface every external recipient" should be a graph filter plus two hops, not a SQL JOIN against a flat audit table that lives in a different schema. **Encrypted at rest.** The vault stores credentials as Fernet ciphertext (Bernstein, 2014). Plaintext exists only inside `_unsealed_read` and only after capability verification. Audit entries record a capability id and a content-hash, never the credential itself. **Soft-fail when the graph is unavailable.** The Lite distribution operates in graph-stripped mode (`MEMAGEN_LITE_MODE=1`) where the compound graph engine is absent. The flat capability table in SQLite remains canonical; graph stamping is a side-effect, gated by the lite-mode flag, and any failure inside the graph path is logged and swallowed without breaking the mint or consume return path. This is the principle that made the V3-S P3 wiring deliverable: a fragile graph-schema branch never blocks the working capability layer. ## 4. Architecture The vault is a four-layer structure. ### 4.1 The encrypted credential store `core/security/vault.py` implements an SQLite-backed store with one table: ```sql CREATE TABLE vault_entries ( entry_id TEXT PRIMARY KEY, user_id TEXT NOT NULL, service TEXT NOT NULL, setting_key TEXT NOT NULL, ciphertext TEXT NOT NULL, created_at REAL NOT NULL, updated_at REAL NOT NULL, redacted_preview TEXT NOT NULL, metadata_json TEXT DEFAULT '{}' ); ``` `store_secret(user_id, service, setting_key, plaintext, metadata)` returns an `entry_id`. The plaintext argument is encrypted via `core.security.encryption.encrypt`, written to `ciphertext`, and the local `plaintext` reference is rebound to `None` immediately after encryption. CPython does not zero memory on rebind, so this does not satisfy a hostile-memory adversary; what it does satisfy is the weaker property that subsequent code in the same frame cannot accidentally log or transmit the plaintext through the original reference. A `redacted_preview` is computed by `_make_preview`, which detects known credential prefixes (`sk-ant-`, `sk-test_`, `ghp_`, `xoxb-`, `pk_live_`, …) and returns a `prefix...XXXX` shape that exposes at most four trailing characters. The list-and-update surface (`list_entries`, `get_redacted_preview`, `update_secret`, `delete_secret`) is metadata-only: at no point does the public surface return ciphertext or plaintext. The single function that does is `_unsealed_read`, which is the trust boundary. ### 4.2 The capability policy table `core/security/capabilities.py` implements the per-use authority record with one table and one always-on side-index: ```sql CREATE TABLE capabilities ( capability_id TEXT PRIMARY KEY, user_id TEXT NOT NULL, vault_entry_id TEXT NOT NULL, action_kind TEXT NOT NULL, constraints_json TEXT NOT NULL, uses_remaining INTEGER, expires_at REAL, revoked_at REAL, created_at REAL NOT NULL, description TEXT NOT NULL, graph_node_id INTEGER -- added by migration 0013, also belt-and-suspendered in `_db()` ); ``` `uses_remaining` is nullable: `NULL` is the unlimited shape, an integer is a hard counter the consume path decrements atomically. `expires_at` is a Unix timestamp; `verify_capability` rejects on `time.time() > row["expires_at"]`. `revoked_at` is set by `revoke_capability`; once set, the capability is permanently dead. The `graph_node_id` column points at the `capability_policy` graph node (Section 4.3) when graph promotion succeeds. It remains `NULL` in lite mode and in any case where the graph stamp fails. This is structurally important: the flat row is canonical; the graph reference is an optional side-effect. ### 4.3 The compound-graph promotion layer When `mint_capability` succeeds in writing the flat row, it attempts to stamp an L5 `capability_policy` node in the agent's `agent_memory` domain. The node carries: - `kind = "capability_policy"` - `name = description or f"{action_kind}:{capability_id[:8]}"` - `content` — JSON-serialized `{action_kind, constraints}` - `embedding_text` — concatenated description and action_kind, used by the compound graph's cosine recall path - `attrs` — `capability_id`, `user_id`, `action_kind`, `vault_entry_id`, `expires_at` Two edges attach. An `owned_by` edge from the `capability_policy` node to the user's L4 anchor node (`user_account`), via `attach_owned_by` in `core.compound_graph.edges`. A `grants_access_to` edge from the `capability_policy` node to the vault entry's graph node, when the vault entry has been graph-promoted (a future V3-C task) and `vault_entries.graph_node_id` is non-null. If the vault entry has not been graph-promoted, the edge is skipped silently. `consume_capability_use` stamps a sibling `capability_use` node edged `caused_by` → grant. `revoke_capability` does two things: it sets `attrs.revoked = True` on the grant node via `graph.update_node()` (preserving the original attrs map), *and* it stamps a separate `capability_revoke` node edged `revokes` → grant. The two-record shape is intentional: a single attribute-flip is invisible to a query that filters only on node `kind`, while a separate revoke node makes the revocation event independently selectable. ### 4.4 The audit module `core/security/audit.py` is the flat-table audit layer. Every successful and unsuccessful `_unsealed_read` writes a row through `log_vault_read`. The row carries `user_id`, `entry_id`, `service`, `setting_key`, `capability_id`, `success`, and `reason`; it does not carry the credential plaintext, the ciphertext, or the request payload. The audit table also implements a tamper-detection hash chain (V3-C.6): each row's `row_hash` is `sha256(prev_row_hash + canonical_json_of_row_data)`, and `verify_chain()` walks the table to detect edits, deletions, or reordering. The flat audit table and the compound-graph audit subgraph carry overlapping but distinct information. The flat table is the sole source of truth for credential reads (it is the layer that exists in lite mode). The graph layer is the queryable structured record of grant/use/revoke lifecycle events. ## 5. The mint–consume–revoke lifecycle The public surface of `core/security/capabilities.py` has four functions and is exhaustively typed. ### 5.1 `mint_capability` ```python def mint_capability( *, user_id: str, vault_entry_id: str, action_kind: str, constraints: dict, description: str, uses_remaining: int | None = None, expires_at: float | None = None, ) -> str: ``` Returns a 32-character `capability_id` (a hex `uuid4`). The constraints dict is validated by `_parse_constraints(action_kind, raw)`, which looks up the constraint dataclass for `action_kind`, checks for missing required fields, and constructs an instance: ```python _CONSTRAINT_CLASSES: dict[str, type] = { "http_request": HttpRequestConstraint, "captcha_solve": CaptchaSolveConstraint, "tool_call": ToolCallConstraint, "file_read": FileReadConstraint, "tts_synthesize": TtsSynthesizeConstraint, "onboarding_playback": OnboardingPlaybackConstraint, } ``` Each constraint dataclass is frozen and exposes the axes that matter for its action kind. `HttpRequestConstraint`, for example, carries `allowed_url_patterns` (a list of glob or `regex:`-prefixed regex patterns), `allowed_methods` (a list of HTTP method names), `max_requests_per_day` (an integer; `0` is unlimited), and `max_bytes_per_request`. `ToolCallConstraint` carries `allowed_tool_names` and `max_calls_per_hour`. `FileReadConstraint` carries `allowed_path_prefixes` and `max_bytes_per_read`. The shape is deliberately small; we resisted adding free-form constraint expressions because every additional axis is a potential audit-query miss. The flat row is inserted under `_write_lock` (a process-level threading lock; concurrency at the multi-process level is the SQLite file lock). The graph stamp follows. Any exception raised inside `_stamp_capability_graph_node` is caught and logged; the flat row remains, and the function returns the capability id. ### 5.2 `verify_capability` ```python def verify_capability( capability_token: str, *, expected_entry_id: str | None = None, expected_action_kind: str | None = None, ) -> tuple[bool, str]: ``` Six checks: row exists, not revoked, not expired, uses remaining, vault entry matches expectation, action kind matches expectation. The two `expected_*` kwargs let the caller bind the verification to the specific use site — `_unsealed_read` passes `expected_entry_id` so a capability minted for vault entry A cannot be redeemed against vault entry B. Returns `(True, "")` on success and `(False, reason)` on failure; the reason string is the audit input. ### 5.3 `consume_capability_use` ```python def consume_capability_use( capability_id: str, *, caller: str | None = None, payload_summary: str | None = None, ) -> bool: ``` The atomic decrement happens in a `_write_lock`-guarded `_db()` connection: `SELECT uses_remaining, revoked_at, graph_node_id`, then if `uses_remaining` is non-null and positive, `UPDATE capabilities SET uses_remaining = uses_remaining - 1`. Returns `False` if the row is absent, revoked, or already at zero uses. On success, returns `True` *and* fires the graph-side stamp via `_stamp_capability_use_node`, which creates an L5 node with `kind="capability_use"`, attrs `{capability_id, grant_node_id, used_at, caller, payload_summary}`, and a `caused_by` edge pointing at the grant node. `caller` and `payload_summary` are the two kwargs that were added in V3-S P3. They were added without breaking any existing call site because both default to `None`; positional callers (`consume_capability_use(cap_id)`) continue to work and produce a use node without caller-tagging. We treat the kwargs as the right pattern for backwards-compatible audit enrichment: a future field can be added with its own default-`None` and old call sites will quietly start producing slightly richer audit nodes. ### 5.4 `revoke_capability` ```python def revoke_capability(capability_id: str, *, user_id: str) -> bool: ``` Marks `revoked_at = now()` on the flat row. The `user_id` kwarg is enforced: a capability can only be revoked by its owner. Idempotent: a second call against an already-revoked capability returns `True` without writing. On the graph side, two operations fire. First, `_mark_graph_node_revoked` reads the existing `attrs`, copies the dict, sets `revoked = True` and `revoked_at = now`, and writes back via `graph.update_node()`. The original attrs are preserved; only the two new keys are added. Second, `_stamp_capability_revoke_node` creates a separate `capability_revoke` L5 node edged `revokes` → grant. The two-record shape costs one extra node but makes the revocation event independently selectable: a query that filters `kind="capability_revoke"` finds every revoke event without an `attrs.revoked` scan. ### 5.5 `_safe_get_user_graph_node_id` — the fragile-schema fallback The canonical user-anchor lookup is `core.auth.users.get_user_graph_node_id`, which calls `_ensure_schema` to add indexes against the canonical multi-column users schema. Test fixtures and minimal installs sometimes provision a stripped users table with only `user_id` and `graph_node_id`; on those tables `_ensure_schema` raises `OperationalError: no such column: username`. The fallback bypasses the schema-enforcement path and reads `graph_node_id` directly from the users table: ```python def _safe_get_user_graph_node_id(user_id: str) -> int | None: try: conn = open_db(timeout=5) try: row = conn.execute( "SELECT graph_node_id FROM users WHERE user_id = ?", (user_id,), ).fetchone() finally: conn.close() if row is not None and row[0] is not None: return int(row[0]) except Exception: # noqa: BLE001 pass try: from core.auth.users import get_user_graph_node_id return get_user_graph_node_id(user_id) except Exception: # noqa: BLE001 return None ``` The two-stage fallback (direct read first, canonical helper second) is what made the V3-S P3 wiring shippable without a global users-table migration in the test fixtures. The corresponding `_try_attach_owned_by_direct` adds the `owned_by` edge by re-reading the user node id directly when `attach_owned_by` fails for the same reason. ## 6. Audit-as-graph-walk This is the contribution that distinguishes V3-P from a conventional audit-log approach. The traditional shape of an audit query is: write a SQL query against a flat `audit_log` table, parse the resulting rows, optionally JOIN against a separate principal table, and reconstruct context by matching ids across schemas. The shape we wanted is: walk the same graph the agent already walks for its own memory, filter on node `kind` and `attrs`, and rely on the existing edge index to traverse from grant to use to revocation in a single pass. Concretely, the test suite (`tests/security/test_capabilities_graph.py::TestConstraintPersistenceAndAuditQuery::test_audit_query_caps_by_action_in_window`) exercises the canonical audit shape: ```python rows = conn.execute( "SELECT id, attrs FROM nodes WHERE kind = 'capability_policy' " "AND json_extract(attrs, '$.action_kind') = ?", (action_kind,), ).fetchall() ``` That single query — filter `kind` to `capability_policy`, filter `attrs.action_kind` to the action of interest — yields every grant ever issued for that action kind. The same shape with `kind = 'capability_use'` yields every use; with `kind = 'capability_revoke'` yields every revocation. The post-filter on `created_at` from the flat capabilities table establishes the time window. The query language is SQLite plus `json_extract` (V3-P does not require a separate graph-query DSL). The indices are the standard nodes-table indices. The work that makes this fast at any scale we care about is already done by the compound-graph engine; the audit subgraph rides on top of it without imposing a new index. The lifecycle queries are graph walks. Given a `capability_policy` node, the set of `capability_use` nodes is found by walking inbound `caused_by` edges. The revocation event is found by walking inbound `revokes` edges. The owning user is found by walking the outbound `owned_by` edge to the `user_account` node. The vault entry, when graph-promoted, is found by walking the outbound `grants_access_to` edge. A complete audit dossier for a single capability — its owner, every credential it ever touched, every dispatch, the revocation event — is four hops from a single starting node. This co-locates the audit trail with the agent's main working memory. A recall query that surfaces "this user has previously authorized email sends to alice@example.com" is the same kind of query as the audit query "this capability was used to call email_send with payload_summary mentioning alice@example.com". One traversal substrate, one set of indices, one API. ## 7. Implementation notes ### 7.1 The graph schema lives at layer 5 The compound graph (described as a Pro subsystem in the master paper) carries six layers of structure. Layer 5 is the operational layer: actions, uses, revocations, dispatches. Capability policies and uses both stamp at layer 5 in the `agent_memory` domain. The reason layer 5 — and not layer 4 (entities) or layer 6 (motifs) — is the right level: a capability policy is an action artifact, not a stable entity, and its lifecycle is dominated by use-and-revoke events that share the same temporal granularity as other layer-5 records. ### 7.2 Lite mode `_lite_mode()` returns `True` when `MEMAGEN_LITE_MODE` is set to one of `1`, `true`, `yes`, `on` (case-insensitive). When true, every graph stamp is a no-op: - `_stamp_capability_graph_node` returns `None` early. - `_stamp_capability_use_node` returns `None` early. - `_mark_graph_node_revoked` returns early. - `_stamp_capability_revoke_node` returns `None` early. The flat capability table and the flat audit table remain canonical. A Lite installation has the full mint–consume–revoke surface, the full constraint enforcement, and the full encrypted credential store. What it does not have is the graph-walk audit query — but it does have the flat audit table with its hash-chained tamper detection. The split is intentional: graph-shaped audit is a Pro-tier convenience; flat-table audit with hash chain is what every installation gets. ### 7.3 Backwards-compat at the consume call site `consume_capability_use` was extended with `caller` and `payload_summary` kwargs without breaking the existing positional signature. Three of the V3-S P3 test cases (`TestConsumeBackwardsCompat::test_positional_consume_still_works`) verify that positional calls continue to work and continue to decrement uses correctly. The pattern — keyword-only kwargs with sensible defaults, never adding a positional argument that pre-existing code would have to pass — is one we apply consistently across the audit surface and recommend as a general extension idiom for security-adjacent APIs. ### 7.4 The `_audit_uses` rate-limit hook `check_constraint` enforces per-axis limits at the point of use: ```python def _audit_uses(capability_id: str, window_seconds: float) -> int: try: from core.security import audit as _audit return _audit.count_capability_uses( capability_id, since_ts=time.time() - window_seconds, ) except Exception as exc: log.warning("audit_uses: %r", exc) return 0 ``` The optional-dependency import is structurally similar to the lite-mode fallback: if the audit module is unavailable, the count returns zero (the conservative answer; zero is below every limit, so the constraint check defers to the static config bound). The `max_requests_per_day`, `max_calls_per_hour`, `max_solves_per_session`, `max_solves_per_day`, `max_uses_per_day` axes all flow through this hook. Each axis is a single `count_capability_uses` call against the flat audit table, scoped to the capability id and a trailing time window. ### 7.5 Graph promotion is non-fatal — always Every graph-side function in `capabilities.py` is wrapped in a `try/except Exception` whose `except` arm logs and returns `None`. The pattern is repeated four times (`_stamp_capability_graph_node`, `_stamp_capability_use_node`, `_mark_graph_node_revoked`, `_stamp_capability_revoke_node`) and once more in the helper `_safe_get_user_graph_node_id`. The discipline is: the flat row is canonical; the graph stamp is a side-effect; any failure inside the stamp must be loggable, must not propagate, and must leave the flat row in its committed state. The discipline is enforced negatively: if a future change introduces a graph-side failure that does propagate, a capability-mint call will start raising under the same conditions. Production traces currently show zero such propagations; we test the soft-fail explicitly in the lite-mode test class, but we have not constructed an adversarial fixture that breaks an arbitrary point in the graph stack and verifies the mint return path. That is on the future-work list (Section 10). ### 7.6 The backfill helper `backfill_capability_graph_nodes(batch_size=100)` exists for installations that pre-date V3-C.8. It iterates over every capability row where `graph_node_id IS NULL` and calls `_stamp_capability_graph_node` for each. Idempotent: rows that already have a `graph_node_id` are skipped. Returns `{"processed", "stamped", "failed"}` for migration logs. We expect this to be useful exactly once per installation, but it lives in the public surface because it is the supported path for any operator who upgrades from a pre-graph build of the vault. ## 8. Evaluation We are honest about what we have measured and what we have not. ### 8.1 Test pass rate The capability test suite has 60 cases across `tests/security/test_capabilities_graph.py` (V3-S P3 wiring: use/revoke nodes, edges, attrs round-trip, lite-mode gate, backwards-compat), `test_capabilities_graph_promotion.py` (V3-C.8: grant node, `owned_by`, `attrs.revoked`, `grants_access_to` against a stub vault graph node), `test_audit_hash_chain.py` (V3-C.6 tamper detection), `test_audit_uses_rate_limit.py` (`_audit_uses` window enforcement), `test_audit_retention_floor.py` (30-day floor), `test_audit_prune_batching.py` (chunked pruning), and `tests/integration/test_capability_execute_e2e.py` (mint → verify → unsealed read → audit). Before V3-S P3 the suite passed 42 of 43; the failure was a graph-stamp regression from the non-canonical users-table fixture. After `_safe_get_user_graph_node_id` and `_try_attach_owned_by_direct`, the suite passes 60 of 60. ### 8.2 What we have not benchmarked **Audit-trail query latency at scale.** The `json_extract`-on-attrs query is fast against a few thousand rows but has not been profiled against the millions-of-rows shape that long-running production installations will eventually produce. We expect SQLite's JSON1 functional index (`CREATE INDEX ... ON nodes(json_extract(attrs, '$.action_kind'))`) to be the right next step; it is on the V4 list, not shipped. **Adversarial schema fuzzing.** The `_safe_get_user_graph_node_id` fallback handles the one non-canonical schema we encountered (stripped users table). We have not constructed a test matrix of every plausible schema deviation. The production guarantee is the soft-fail — a graph stamp that hits an unexpected schema returns `None` and logs a warning — rather than a strong claim about handling every possible deviation correctly. **Cryptographic hardening.** The vault uses Fernet (AES-128-CBC + HMAC-SHA256, plus a timestamp) with the at-rest key held in an environment variable or a derived KDF output. We have not run a formal threat model of the at-rest-key handling; the master paper's Section 10 footprints a hardware-rooted vault as future work. **Side-channel attacks on `_unsealed_read`.** The function's timing characteristics depend on whether the capability check passes (early return) and whether the row exists (late return). We have not attempted a timing-attack analysis. For the threat model we currently target — a malicious agent with full process access — this matters less than it would for a multi-tenant RPC service. ### 8.3 What worked unexpectedly well The discipline of "flat row first, graph stamp as best-effort side-effect" turned out to be the structural property that made the V3-S P3 ship-without-breaking-anything outcome possible. Five separate functions implement the same try/except/log/return pattern; that uniformity is the feature, not redundancy. ### 8.4 What is fragile The biggest delivery risk during V3-S P3 was the `_ensure_schema` interaction with non-canonical test fixtures. The mitigation was to bypass `_ensure_schema` entirely with a direct read against the users table. The cost is that the fallback path does not auto-migrate the users table to the canonical schema — a real installation that hits the fallback continues to operate in a partially-migrated state until an explicit migration runs. We accept this cost. The alternative — auto-migrating from inside a security-critical path — would have introduced a worse failure mode. ## 9. Discussion ### 9.1 What we get Structured audit is the headline. A "show me everything tool X did with credentials in the past 30 days" query is one filter plus two graph hops. Real revocation — not just "the toggle is off now" but "this specific authority is dead, and the audit record of its death is queryable" — is the second. Blast-radius bounding by per-use grants is the third. None of these are individually novel. The integration is. ### 9.2 What we give up Simplicity. Each tool dispatch is now a multi-step protocol: mint, verify, consume, audit. The constraint dataclasses add a small amount of upfront design work for any new `action_kind`. The fallback-everywhere discipline is a maintenance overhead — every graph-side helper has to be wrapped in the try/except idiom, and any future contributor will have to learn the pattern. We argue that the cost is paid once at the infrastructure layer and amortized across every tool. Users of the agent see no friction — capability mint and consume happen inside the executor, not in the user-facing prompt loop. Operators see a structured audit trail rather than a flat log. New tool-surface contributors write a constraint dataclass once and inherit the rest. ### 9.3 The honest comparison Compared to a no-capability framework, V3-P is strictly more work. Compared to OAuth scopes, V3-P is per-use rather than per-token. Compared to an ocap language, V3-P is weaker on unforgeability (the token is a 128-bit hex string, not a kernel reference) and stronger on persistent audit. Compared to a separate policy engine, V3-P co-locates the policy record with the agent's working memory, at the cost of being less language-agnostic — the V3-P shape assumes Python and SQLite. We believe the right comparison for an agent runtime is the no-capability baseline most current frameworks ship with, and against that baseline the V3-P contribution is a genuine improvement on the safety axis. The cost of the architecture is real but amortizable; the cost of *not* having it becomes apparent only after a public bad outcome. ## 10. Future work **Capability federation.** A Memagen™ installation on machine A holds a capability for vault entry X. Machine B wants to delegate a derived authority to a sub-agent on machine A. The current shape requires re-minting; a federation protocol would let the original capability carry a delegation chain. The relevant prior art is in object-capability languages (E's certificate-bearing capabilities) and in delegation extensions to OAuth (the `actor` claim in token-exchange). We have not specified the protocol. **Hierarchical scoping and delegation chains.** A capability for `tool_call` against `email_send` could in principle be derived from a parent capability for any `tool_call`, with the constraint set narrowed at each derivation step. The current shape is flat. Hierarchical scoping would make audit queries more expressive (walk up the derivation chain, find the original authority) at the cost of added complexity in `verify_capability` (must follow the chain, must check every link's expiration and revocation). **Hardware-rooted vault.** The Fernet at-rest key is currently held in a process-readable location. A TPM (Linux) or Secure Enclave (macOS) backing would move the key into a non-extractable context, with the vault module obtaining ephemeral decryption authority via a platform API. The work is mostly platform integration; the V3-P module's interface to the at-rest crypto layer is small enough that retrofitting is straightforward. **Functional indexes for audit query.** As mentioned in Section 8.2, an SQLite JSON1 functional index on `attrs.action_kind` (and possibly `attrs.capability_id`) would convert the audit-window query from a scan to an index lookup. This is a one-line schema migration and a small benchmark to verify the speedup. **Adversarial schema test matrix.** A property-based test (Hypothesis or a hand-built fuzzer) that constructs arbitrary deviations of the users and vault tables and verifies that mint, consume, and revoke continue to return correctly is the structural answer to the schema-fragility risk. This belongs in the V4 test plan. **Audit-as-recall.** The compound graph supports cosine recall against `embedding_text`. A `capability_policy` node carries embedding text, which means the audit query "find every capability whose description matches this prompt" is already a cosine recall away. We have not exercised this beyond the existing test (`test_capabilities_graph_promotion.py` uses a deterministic embedder stub), but the substrate is in place. ## 11. References 1. **Anthropic.** (2024). *Model Context Protocol Specification*. https://modelcontextprotocol.io 2. **Bernstein, D.J.** (2014). *The Fernet specification*. https://github.com/fernet/spec 3. **Dennis, J.B., & Van Horn, E.C.** (1966). *Programming semantics for multiprogrammed computations*. Communications of the ACM, 9(3), 143–155. 4. **Hardt, D.** (2012). *The OAuth 2.0 Authorization Framework*. RFC 6749. 5. **Hardy, N.** (1985). *KeyKOS architecture*. ACM Operating Systems Review, 19(4). 6. **Levitas, T.** (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Memagen™ master paper, this series. 7. **Miller, M.S.** (2006). *Robust composition: Towards a unified approach to access control and concurrency control*. PhD dissertation, Johns Hopkins University. 8. **Miller, M.S., Samuel, M., Laurie, B., Awad, I., & Stay, M.** (2008). *Caja: Safe active content in sanitized JavaScript*. Google research report. 9. **OpenAI.** (2023). *Function calling and tool use*. https://platform.openai.com/docs/guides/function-calling 10. **Shapiro, J.S., & Smith, J.M.** (1999). *Capability myths demolished*. Technical Report MS-CIS-03-23, University of Pennsylvania. 11. **Shapiro, J.S., Smith, J.M., & Farber, D.J.** (1999). *EROS: A fast capability system*. Proceedings of the 17th ACM Symposium on Operating Systems Principles, 170–185. 12. **Wilkes, M.V., & Needham, R.M.** (1979). *The Cambridge CAP computer and its operating system*. North-Holland. Source: <https://github.com/tablevitas123/crackedclaw> --- ## Paper: 2026-05-08-lenient-parser URL: https://memagen.com/papers/2026-05-08-lenient-parser PDF: https://memagen.com/papers/2026-05-08-lenient-parser.pdf --- title: "Lenient Tool-Call Parsing: Bridging Weak-Tool-Call Models to Native Tool Catalogs" subtitle: "Memagen™ recovery of tool calls across four production-observed emission patterns, per-tool argument coercion, and hallucinated-result detection" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "Production agent runtimes enforce strict tool-call shapes — JSON Schema, OpenAI-style function calling, Anthropic-style tool_use blocks — and reject model output that fails validation. The contract is acceptable for closed frontier models, whose post-training has aligned them to a single canonical emission shape. It is not acceptable for the open-weight tier (Kimi K2, DeepSeek V3, Llama 3.x, Qwen 2.x, Mistral variants), where a single deployment emits tool calls across at least four overlapping shapes — sometimes mixed within one response — and occasionally hallucinates tool *results* alongside the calls. Strict-reject runtimes leave the operator with two options: pay frontier prices on every turn, or accept unreliable execution on weaker models. We describe Memagen™'s lenient parser. It is a streaming state machine that detects four emission patterns (native function-calling, XML invoke blocks, JSON-in-sentinel `<tool_call>` blocks, and special-token envelopes), a per-tool registry-based argument coercer that bends recognizable-but-non-canonical argument shapes to declared schemas, a fuzzy resolver that maps paraphrased tool names to catalog entries, a catalog gate that rejects off-catalog calls with a structured error feeding the next system prompt, and a fourth-opener detector that suppresses role-played `<function_results>` blocks. The implementation is open-source (MIT) in the Memagen™ Lite distribution. We document what the test suite verifies (V3-Q test waves) and what remains unmeasured at production scale." keywords: ["tool calling", "lenient parser", "open-source LLMs", "BYOLLM", "agent infrastructure"] kind: specialty subsystem: "lenient parser" pages: 8 version: "v1.1" --- ## 1. Introduction The boundary between a model's emission and the runtime's dispatch layer is the highest-friction surface in any agent platform. The dispatcher needs structured argument trees; the model produces a token stream. Most production runtimes settle on the strictest available bridge: validate against JSON Schema, reject on failure, surface the error, retry next turn. This is the default for OpenAI function calling (OpenAI, 2023), Anthropic tool use (Anthropic, 2024), and the LangChain function-call wrapper (Chase, 2022). Strict rejection works because frontier models are post-trained against a single canonical emission shape; validation failure rates against properly-described schemas are small enough that rejection cost is dominated by post-training convergence — already paid by the upstream lab. The picture inverts for the open-weight tier. Kimi K2 (Moonshot AI, 2024) emits tool calls in a model-specific `<|tool_calls_section_begin|>` envelope. DeepSeek V3 (DeepSeek, 2024) wraps JSON in `<tool_call>` text sentinels. Llama 3.x (Meta, 2024) and Qwen 2.x (Alibaba, 2024) mix native-shaped JSON with XML invoke blocks depending on system-prompt phrasing. Mistral variants drift across releases. Worse, several occasionally role-play the post-tool-call protocol: the assistant turn emits a malformed call *and* a fabricated `<function_results>` block as if a tool had returned data, then reasons on top of the fabrication. A strict-reject runtime presented with this output has two failure modes. Rejection-then-retry converges slowly or not at all because the fine-tuned distribution reproduces the same malformed shape on every retry. Leaking the malformed envelope to the user as visible content corrupts the transcript and — in the hallucinated-results case — poisons the next turn, because the model treats its own fabrication as ground truth. Memagen™'s lenient parser bridges the gap. It is a streaming state machine that recognises four production-observed emission patterns, normalises them to a canonical `tool_call_delta` shape, applies per-tool argument coercion to non-canonical argument shapes, fuzzy-resolves paraphrased tool names against the declared catalog, and detects-and-suppresses hallucinated `<function_results>` blocks before they propagate. The parser is the subsystem that makes BYOLLM — bring your own LLM — economically defensible: a customer can deploy Memagen™ against open-weight models on commodity hardware without sacrificing tool reliability. The implementation ships under MIT in the Lite distribution. This paper is a specialty companion to the [Memagen™ master paper](./2026-05-08-memagen-master/) (whose §4 introduces the parser at the capability level) and to the [LLM router paper](./2026-05-08-llm-router/) (which describes the splice point at which the parser wraps a provider stream). Sections below cover the detection algorithm (§4), per-tool argument coercion (§5), the catalog gate and fuzzy resolver (§6), hallucinated-results detection (§7), implementation notes (§8), and what we have and have not measured (§9). ## 2. Related work ### 2.1 Function-calling specifications The OpenAI function-calling spec (OpenAI, 2023) is the first widely-adopted contract for LLM-driven tool invocation: tools are described by name, JSON-Schema arguments, and a natural-language description; the assistant message carries a `tool_calls` array; the runtime executes and replies with `role=tool` messages keyed by `tool_call_id`. Anthropic native tool use (Anthropic, 2024) follows the same conceptual shape but emits structured `tool_use` content blocks rather than a flat array. The Gemini function-calling format (Google, 2024) is a third variant. The MCP specification (Anthropic, 2024) standardises the *server* side of this contract — tool advertisement and dispatch — and defers the *client* side to the host runtime. ### 2.2 JSON Schema validation The dominant validator stack is JSON Schema Draft 7 (Wright et al., 2019). Production agent runtimes typically wire a validator (`pydantic`, `jsonschema`, `ajv`) into the dispatch path: validation passes, dispatch proceeds; validation fails, the error is surfaced and the model retries on the next turn. The contract is clean. The cost is brittleness against weak-tool-call models. ### 2.3 Parser-tolerance work Several lines of work address tolerance in structured-output parsers. Constrained decoding (Beurer-Kellner et al., 2023; Willard & Louf, 2023) constrains emission at decode time to remain within a CFG, which removes the malformed-output problem at its source — but requires control over the inference engine, which BYOLLM customers running NIM, vLLM, or remote OpenRouter endpoints generally do not have. Best-effort JSON parsers (`json5`, `dirty-json`) recover from common syntactic errors (trailing commas, unquoted keys, single quotes) but do not address the higher-level shape mismatches of §5. The closest prior work is the community-contributed `function-calling-format` translation layer for open-weight models, which translates one or two emission shapes; we extend coverage to four observed shapes plus the hallucinated-results case. ### 2.4 Why strict-reject became default Strict rejection has three justifications. *Training-time alignment*: a canonical emission shape is a measurable target, and the post-training community has converged on rejection-during-RLHF as the convergence signal (Ouyang et al., 2022). *Observability*: a strict reject produces a clean structured error suitable for dashboards. *Conservatism*: a malformed call parsed leniently might dispatch the wrong action. Memagen™ accepts the first two and answers the third by composing the lenient parser with the V3-P capability vault ([master paper §5](./2026-05-08-memagen-master/)) — lenient *parsing* never bypasses capability *enforcement*. ## 3. The four production-observed formats We document the four emission patterns the parser handles, each with a representative example string from production observations during the V3-Q development wave. ### 3.1 Format A — native OpenAI-style function-calling JSON, Anthropic-style tool_use blocks, and Gemini-style function calls all arrive on the structured side of the streaming protocol — i.e., not in `delta.content`. The router-level event protocol carries them as `("tool_call_delta", ...)` events directly. The parser passes these through untouched. Example (OpenAI shape, abbreviated): ```json {"id": "call_abc123", "type": "function", "function": {"name": "widget_create", "arguments": "{\"name\":\"hello\",\"spec\":{\"kind\":\"markdown\"}}"}} ``` The `wrap()` generator recognises that non-content events bypass the state machine entirely: ```python async for kind, chunk in stream: if kind != "content": yield (kind, chunk) continue ``` This is the no-op path for frontier models and the reason the parser introduces zero overhead on canonical providers. ### 3.2 Format B — XML invoke `<invoke>` blocks within prose, wrapped in a `<function_calls>` envelope. Common output for Llama 3.x and Qwen variants when prompted with XML-flavoured system prompts. ```xml <function_calls> <invoke name="widget_create"> <parameter name="name">hello</parameter> <parameter name="spec">{"kind":"markdown","props":{"text":"Hi"}}</parameter> </invoke> </function_calls> ``` The parser recognises the `<function_calls>` opener, accumulates until the matching closer, and walks the section via `xml.etree.ElementTree` to extract each `<invoke>` and its `<parameter>` children. Parameter values stay strings — XML carries no richer type — and the per-tool argument coercer (§5) handles type recovery downstream. ### 3.3 Format C — JSON in sentinel A JSON object wrapped in `<tool_call>...</tool_call>` (or `<function_call>...</function_call>`) text sentinels. This is the format Memagen™'s system-prompt hint (`build_tool_hint()`) recommends for weak-tool-call models, because it is the format weakest models converge to most reliably under our hint: ```text <tool_call>{"name": "widget_create", "arguments": {"name": "hello", "spec": {"kind": "markdown"}}}</tool_call> ``` The parser scans for `<tool_call>` openers, accumulates to the matching closer, and parses the body as JSON. Multiple sequential blocks in a single chunk are flattened into one `tool_call_delta` list, matching the way the native streaming protocol batches sequential calls. ### 3.4 Format D — special-token envelope Models in the Moonshot family emit tool calls in a model-specific bracketed envelope using literal special tokens (the tokenizer emits these as single-token IDs; they appear in `delta.content` as text): ```text <|tool_calls_section_begin|> <|tool_call_begin|>functions.widget_create:0<|tool_call_argument_begin|>{"name":"hello","spec":{"kind":"markdown"}}<|tool_call_end|> <|tool_calls_section_end|> ``` The parser recognises `<|tool_calls_section_begin|>` as an opener, accumulates to the matching `<|tool_calls_section_end|>` closer, and runs a regex walk over each `<|tool_call_begin|>...<|tool_call_end|>` sub-block. The `functions.` prefix on the tool name (mirrored from the OpenAI convention) is stripped. Arguments are JSON-validated; on parse failure, the raw bytes are wrapped in `{"_raw": ...}` so the dispatcher surfaces the failure rather than silently dropping the call. The four format detection routines dispatch from a single registry: ```python _OPENERS: list[tuple[str, str, str]] = [ (_OPEN_D, _CLOSE_D, "kimi_special"), (_OPEN_B, _CLOSE_B, "xml_invoke"), (_OPEN_C, _CLOSE_C, "xml_sentinel"), (_OPEN_HALLUC, _CLOSE_HALLUC, "hallucinated_results"), ] ``` The fourth entry is the hallucinated-results sentinel (§7) — a *suppressed* format that participates in the same state machine but never produces a `tool_call_delta`. ### 3.5 Architecture: the parser state machine The parser is one streaming state machine with two states (`PASSTHROUGH`, `BUFFERING`) and four format-specific (opener, closer, fmt-tag) triples. Format A bypasses the machine entirely; formats B/C/D produce synthetic `tool_call_delta` events; the hallucinated-results format produces a `hallucinated_results_detected` event and triggers a one-shot retry directive on the next round. ```mermaid stateDiagram-v2 [*] --> PASSTHROUGH PASSTHROUGH --> PASSTHROUGH: content chunk\n(no opener seen) PASSTHROUGH --> PASSTHROUGH: tool_call_delta\n(Format A passthrough) PASSTHROUGH --> BUFFERING_B: <function_calls> PASSTHROUGH --> BUFFERING_C: <tool_call> PASSTHROUGH --> BUFFERING_D: <|tool_calls_section_begin|> PASSTHROUGH --> BUFFERING_HALLUC: <function_results> BUFFERING_B --> EMIT_B: </function_calls> BUFFERING_C --> EMIT_C: </tool_call> BUFFERING_D --> EMIT_D: <|tool_calls_section_end|> BUFFERING_HALLUC --> SUPPRESS: </function_results> EMIT_B --> PASSTHROUGH: yield tool_call_delta\n(fmt=xml_invoke) EMIT_C --> PASSTHROUGH: yield tool_call_delta\n(fmt=xml_sentinel) EMIT_D --> PASSTHROUGH: yield tool_call_delta\n(fmt=kimi_special) SUPPRESS --> PASSTHROUGH: yield\nhallucinated_results_detected\n(drop block, set retry directive) note right of SUPPRESS Retry loop: halluc_retries < 2 => augment next-round system prompt Otherwise: cap retries, keep suppressing. end note ``` The diagram is normative: every code path in `core/llm/lenient_parser.py` maps to exactly one labelled transition above. ## 4. Detection algorithm The detection logic lives in a streaming state machine implemented as the `wrap()` async generator in `core/llm/lenient_parser.py`. The machine has two macro-states: ```text PASSTHROUGH --opening sentinel--> BUFFERING BUFFERING --closing sentinel--> PASSTHROUGH (emit synthetic delta) ``` In `PASSTHROUGH`, the parser yields content chunks to the consumer. In `BUFFERING`, it accumulates bytes silently until a closer is found. ### 4.1 Opener detection Per chunk, the parser searches for the earliest opening sentinel via `_find_first_opener()`: ```python def _find_first_opener(buf: str) -> tuple[int, str, str, str] | None: best: tuple[int, str, str, str] | None = None for opener, closer, fmt in _OPENERS: p = buf.find(opener) if p == -1: continue if best is None or p < best[0]: best = (p, opener, closer, fmt) return best ``` Ties resolve by document order — whichever sentinel actually starts first wins. All four sentinels have distinct opener strings, and a model emitting a mixed-format response (rare but observed) signals format intent through which sentinel it emits first. ### 4.2 Lookahead buffer for split-frame openers Streaming providers do not align chunk boundaries with sentinel boundaries. A model might emit `<|tool_calls` in one chunk and `_section_begin|>` in the next; yielding the first chunk verbatim would render a corrupt prefix to the user. The defence is a lookahead buffer: in `PASSTHROUGH`, after no full opener is found, the parser checks whether any *suffix* of the working buffer is a strict proper prefix of any opener: ```python def _suffix_could_open(buf: str) -> int: best = 0 n = len(buf) for opener in _ALL_OPENERS: max_check = min(len(opener) - 1, n) for k in range(max_check, best, -1): if buf.endswith(opener[:k]): best = k break return best ``` The matching suffix is held back in `lookahead` and prepended to the next chunk. The held-back tail never exceeds `len(longest_opener) - 1` bytes, bounding the latency cost to one character less than the longest sentinel. ### 4.3 Prefix-collision defence: `<function_calls>` vs `<function_results>` The subtlest collision is between Format B's `<function_calls>` (a real opener) and the hallucinated-results sentinel `<function_results>` (a *suppressed* opener we want to detect, not parse as Format B). Both share `<function_`. The parser registers each as a distinct entry in `_OPENERS` and relies on `_find_first_opener()`'s literal substring search on the full opener — the two strings differ at the byte level, so the search disambiguates without prefix-tree work. ### 4.4 Closer detection and tail handling In `BUFFERING`, the parser searches for the format-specific closer via `candidate.find(active_close)`. If absent, the candidate is stashed in `xml_buf` and the next chunk is awaited. If found, the section is parsed (or, for hallucinated-results, suppressed), state returns to `PASSTHROUGH`, and the post-closer tail is processed in the same loop iteration — handling two sentinel sections in a single chunk. ### 4.5 Stream-end flush When the upstream stream is exhausted with the parser still in `BUFFERING`, the parser attempts graceful degradation: parse what is in `xml_buf` and emit any deltas recovered. If the parse yields nothing, the raw buffer is surfaced as content so the user at least sees what the model said rather than silently losing it. The hallucinated-results case is special: an incomplete hallucinated block at stream end is treated as a detection (we saw the opener, even though we never saw the closer) and the partial content is suppressed rather than leaked. ## 5. Per-tool argument coercion (V3-Q.1) The parser produces a tool call with a name and an argument string. The next problem is that weak-tool-call models routinely emit *recognizable but non-canonical* argument shapes — well-formed JSON, correct tool name, but an argument tree with the wrong key names or the wrong nesting level. Three patterns dominate live observations against open-weight models running on NIM: - **Key alias**: `type` → `kind` (English-natural synonym for an enum field). - **Shape flatten/nest**: `{"text": "..."}` → `{"props": {"text": "..."}}` (omitted wrapper layer the canonical schema requires). - **Cross-turn ID resolution**: a name passed where the canonical schema requires the integer ID returned by a previous tool call's result. A schema-driven coercer — one that reads JSON Schema and applies generic key-rename rules — is appealing but brittle: rules generalise poorly across tools, and the schema does not encode cross-turn semantics. Memagen™'s coercer is registry-based: each tool that needs coercion registers a per-tool function in `core/agent/arg_coercion.py`: ```python ToolCoercer = Callable[[dict, list], dict] COERCERS: dict[str, ToolCoercer] = { "widget_create": _coerce_widget_create, "workspace_create": _coerce_workspace_create, "workspace_apply": _coerce_workspace_apply, } ``` The `history` argument grants the coercer access to prior tool results so it can resolve cross-turn references. The `coerce_args` dispatcher is best-effort: tools without entries pass through unchanged, and a coercer that raises returns the original loose args — coercion never blocks dispatch. A representative coercer is `_coerce_widget_create`: ```python def _coerce_widget_create(args: dict, history: list) -> dict: out = dict(args) # Step 1: type -> kind rename (top-level alias) if "type" in out and "kind" not in out: out["kind"] = out.pop("type") # Step 2: any pre-existing 'spec' is canonical; harmonise + return early if isinstance(out.get("spec"), dict): spec = dict(out["spec"]) if "kind" not in spec and "kind" in out: spec["kind"] = out.pop("kind") out["spec"] = spec else: # Step 3: build spec from flat top-level shape. spec_keys = {"kind", "props", "data", "observable_node_id", "slot", "description"} spec_payload: dict = {} carry_top: dict = {} for k, v in list(out.items()): if k == "name": continue if k in spec_keys: spec_payload[k] = v else: carry_top[k] = v if carry_top: existing_props = ( spec_payload.get("props") if isinstance(spec_payload.get("props"), dict) else {} ) spec_payload["props"] = {**existing_props, **carry_top} out = {"name": out.get("name"), "spec": spec_payload} # Step 4: synthesize a name when missing if not out.get("name"): # ... derive from kind + first prose hint in props ... pass return out ``` The coercer reshapes a flat `{type, text}` into the canonical `{name, spec: {kind, props: {text}}}` shape required by `WIDGET_CREATE_SCHEMA`. The `name` field is synthesized when missing (the canonical schema requires it); synthesis derives a slug from the widget kind plus a fragment of any human-readable prose field (`text`, `title`, `label`). Cross-turn resolution is illustrated by `_coerce_workspace_create`, which converts `{widgets: [{loose object}, ...]}` into `{widget_ids: [int, ...]}` by walking the conversation history for prior `widget_create` results: ```python def _coerce_workspace_create(args: dict, history: list) -> dict: out = dict(args) if "widgets" in out and "widget_ids" not in out: widgets = out.pop("widgets") widget_ids: list[int] = [] if isinstance(widgets, list): for w in widgets: if isinstance(w, int): widget_ids.append(w) elif isinstance(w, dict): if isinstance(w.get("id"), int): widget_ids.append(w["id"]) else: resolved = _resolve_widget_id_from_history(w, history) if resolved is not None: widget_ids.append(resolved) if not widget_ids: widget_ids = _all_recent_widget_ids_from_history(history) out["widget_ids"] = widget_ids return out ``` The history scan handles both the native shape (`role=tool` messages keyed by `tool_call_id`) and the lenient shape (`role=user` messages containing `<function_results>` blocks parsed via `_RESULT_BLOCK_RE`). The branching is in `_scan_recent_tool_results()`. This duality matters because the lenient parser, when normalising tool results back into history, formats them as `<function_results>` blocks rather than `role=tool` messages — see `format_tool_result()` in §8. Registry-based coercion is deliberate. Schema-driven rules would have to encode equivalence classes (`type` ≡ `kind`, `text` ≡ `props.text`, name → ID via history) declaratively, and we have found no declarative form that generalises across the tool surface without producing wrong coercions on unfamiliar tools. Registry-based coercion is more code per tool but correct by construction: a missing coercer passes through. The cost is per-tool work; the benefit is that coercers cannot misfire on tools they do not know about. ## 6. Catalog gate and fuzzy resolver (V3-Q.2) Even after format detection and argument coercion, weak-tool-call models will sometimes produce a tool call whose *name* does not match any catalog entry. Two patterns dominate: - **Hallucinated tool**: the model emits `graph_write` when only `widget_create` is offered. The model has invented a name from a fine-tuning prior or from a different system's vocabulary. - **Paraphrased tool**: the model emits `create_markdown_widget` when the catalog entry is `widget_create`. The tool *concept* is correct; the *string* has drifted because the model is paraphrasing from the description. Memagen™ handles these with two complementary mechanisms. ### 6.1 Fuzzy resolver Inside `wrap()`, before a `tool_call_delta` event is yielded, the parser runs each detected name through `_fuzzy_resolve_tool_name()` against the declared catalog. The resolver computes four signals per candidate and takes the maximum: - **Substring containment** (either direction): score 0.9 if `emitted in candidate or candidate in emitted`. Catches `widget_create` against `agent_widget_create`. - **Token-set subset**: score 0.85 if either token set (split on `_`) is a subset of the other. Catches `create_markdown_widget` against `widget_create` — every catalog-name token is present plus a paraphrase noise token (`markdown`). - **Token-set Jaccard**: `|∩| / |∪|` over the underscore-split token sets. Catches partial overlaps. - **`difflib.SequenceMatcher` ratio**: stdlib Levenshtein-style ratio (Ratcliff & Obershelp, 1988). Catches typos and minor spelling drift. If the best score across all candidates is ≥ 0.7 (the `_FUZZY_THRESHOLD`), the canonical name is substituted; otherwise the emitted name is preserved verbatim and is rejected by the catalog gate downstream. The resolver logs every rewrite at INFO level for observability. ### 6.2 Catalog gate After the parser produces tool calls and after argument coercion runs, the agent loop applies a strict catalog gate when `self.tool_filter` is set: ```python if self.tool_filter is not None and tool_calls: catalog_names = {t.get("name", "") for t in self.tool_filter} allowed = [] rejected = [] for tc in tool_calls: if tc.name in catalog_names: allowed.append(tc) else: rejected.append(tc) for tc in rejected: err_payload = { "error": f"tool '{tc.name}' not in this turn's catalog", "available": sorted(catalog_names), "hint": "Use ONLY the exact tool names listed in the schema; " "any other names will fail. Retry with the correct name.", } # ... append structured error to history as a tool result ... tool_calls = allowed ``` The structured error is inserted into history via `format_tool_result()` with the same format tag the call carried (native, kimi_special, xml_invoke, xml_sentinel). On the next turn, the model sees the rejection and available-names list inline with its prior turn, which in production observation produces self-correction within one or two retries. The error is a JSON object (`error`, `available`, `hint`) rather than a flat string, so the model attends to `available` without parsing prose. The two mechanisms compose: the fuzzy resolver catches paraphrases with high recall and threshold-controlled precision; the catalog gate catches the residue (hallucinated tools and low-confidence paraphrases) with a structured error that drives convergence on the canonical name. ## 7. Hallucinated `<function_results>` detection (V3-Q.3) The novel contribution of the lenient parser is the hallucinated-results detector. Some open-weight models — Kimi K2 in particular — occasionally hallucinate not just tool *calls* but tool *results*, emitting role-played `<function_results>` blocks as if a tool had returned data: ```text I'll search for that for you. <function_results> <result name="code_search" id="x"> {"matches": [{"path": "src/auth.py", "line": 42, "text": "..."}]} </result> </function_results> Based on the search results, the authentication logic is in... ``` The model has emitted both the question and its own fabricated answer in a single turn. Without detection, the consequence is severe: the transcript contains a fake tool-result message, and on the next turn the model treats the fabrication as ground truth and continues reasoning on top of it. Memagen™ registers `<function_results>` as a *fourth opener* in the parser's state machine. The format-tag is `hallucinated_results`; the closer is `</function_results>`: ```python _OPEN_HALLUC = "<function_results>" _CLOSE_HALLUC = "</function_results>" _OPENERS: list[tuple[str, str, str]] = [ (_OPEN_D, _CLOSE_D, "kimi_special"), (_OPEN_B, _CLOSE_B, "xml_invoke"), (_OPEN_C, _CLOSE_C, "xml_sentinel"), (_OPEN_HALLUC, _CLOSE_HALLUC, "hallucinated_results"), ] ``` When the active format is `hallucinated_results`, the section is *suppressed*: never yielded as visible content, never producing a synthetic `tool_call_delta`. The parser instead emits a single `hallucinated_results_detected` event whose payload carries the running count of hallucinated blocks seen so far in the stream: ```python if active_fmt == "hallucinated_results": halluc_count += 1 yield ( "hallucinated_results_detected", json.dumps({"count": halluc_count}), ) ``` The agent loop in `core/agent/loop.py` listens for the event: ```python elif kind == "hallucinated_results_detected": yield {"type": "hallucinated_results_detected", "payload": payload} if halluc_retries < _HALLUC_MAX_RETRIES: halluc_retries += 1 halluc_directive_pending = True ``` When the pending bit is set, the next round's system prompt is augmented with a one-shot corrective directive: ```python _HALLUC_DIRECTIVE = ( "You emitted `<function_results>` blocks. Those are reserved " "for actual tool results sent back to you AFTER you make a " "real tool call. Do not generate them yourself. Use real tool " "calls and wait for actual results." ) ``` The directive is one-shot — appended for *only* the next round; the following round returns to the unmodified prompt. The retry counter is capped at `_HALLUC_MAX_RETRIES = 2`. If hallucinations continue after two corrective directives, the runtime stops re-trying and lets the conversation proceed (suppression continues, directive injection halts). Two design choices matter: **Suppression, not rejection.** A strict-reject runtime would discard the entire assistant turn. We chose suppression because the rest of the turn — surrounding prose and any *real* tool calls — is usable. The hallucinated block is the only part dropped, which is strictly better when the assistant turn is mixed. **One-shot, not persistent.** A persistent directive would remain in the system prompt for the rest of the session, eventually biasing the model against legitimate references to tool results (e.g., summarising what a tool returned). One-shot injection avoids the bias. Detector behaviour and directive injection are covered by a nine-test suite in `tests/llm/test_lenient_parser_hallucination.py` (opener/closer detection, suppression, count tracking, mixed-format streams, stream-end flush). ## 8. Implementation notes ### 8.1 Tool-result formatter and the `fmt` tag Format detection produces a tool call. The runtime executes it. The result must be inserted back into conversation history in a shape the model recognises. For native-format calls, this is a `role=tool` message keyed by `tool_call_id`. For lenient-format calls, the model never saw a `tool_call_id` in the protocol sense, so we synthesise a `role=user` message containing a `<function_results>` block keyed by tool name and synthetic ID: ```python def format_tool_result(tc, result_str, *, fmt): if fmt == "native": return ChatMessage(role="tool", content=result_str, tool_call_id=tc.id) wrapped = ( f"<function_results>\n" f"<result name=\"{tc.name}\" id=\"{tc.id}\">\n" f"{result_str}\n" f"</result>\n" f"</function_results>" ) return ChatMessage(role="user", content=wrapped) ``` The `fmt` tag travels with the call. The router's accumulator extracts `_lenient_fmt` from the synthetic delta and stores it in a side-channel keyed on the tool-call ID (`_TC_FORMAT_BY_ID`); the agent loop reads it back when constructing the `ToolCall`. This roundtrip is what allows downstream history-replay and result-injection to know which dialect was used. A practical consequence: `_scan_recent_tool_results()` in the argument coercer must handle both shapes. The `<function_results>` wrapper used for lenient-format results is the same wrapper the hallucinated-results detector suppresses on the *outbound* stream. The detector matters because without it, the model would see its own fabricated `<function_results>` blocks treated as real tool results in subsequent turns — exactly the corruption to prevent. ### 8.2 Side-channel format tracking The `_TC_FORMAT_BY_ID: dict[str, str]` module-level dict is a deliberate side-channel. The streaming protocol carries `tool_call_delta` events as JSON strings; adding a new top-level field (`_lenient_fmt`) to the delta would change the over-the-wire shape consumed by the router accumulator and any external consumer. Keeping the format tag as a side-channel keyed on the synthetic ID lets the agent loop reattach it via `_LFMT.get(tc.get("id", ""), "native")` without changing the protocol. The cost is global state, acceptable here because the side-channel is single-process and the IDs are namespaced (`lenient_{turn_idx}_{call_idx}`). This is the same splice pattern the [router uses](./2026-05-08-llm-router/) for `_ACTIVE_MODEL_FOR_HINT` and `_ACTIVE_TOOL_CATALOG_FOR_HINT`. ### 8.3 System-prompt hint For models that need lenient parsing (the `_WEAK_TOOL_CALL_PATTERNS` allowlist: `kimi-k2`, `kimi`, `qwen`, `llama-3`, `mistral-`, `deepseek-r`, `moonshotai`), `build_tool_hint()` produces a system-prompt addendum that nudges the model toward Format C (the JSON-in-sentinel format weakest models converge to most reliably) and, when a catalog is provided, lists the exact tool names available for the turn: ```text You MUST use the EXACT tool names provided in the schema. The available tools for this turn are: - widget_create - workspace_create - workspace_apply Do NOT invent or paraphrase tool names. Only the names listed above will work. ``` For frontier models, the hint returns `None` so we do not waste tokens explaining a format their native API already handles. The hint is best-effort: weak models still occasionally ignore it, which is exactly why we need the parser, the coercer, the fuzzy resolver, the catalog gate, and the hallucinated-results detector — defence in depth. ### 8.4 Edge cases - **Split-frame openers**: handled by the lookahead buffer (§4.2). Held-back tail bounded by `len(longest_opener) - 1`. - **Partial-open fragments at stream end**: the `wrap()` flush logic surfaces unparseable buffered bytes as content rather than dropping them silently, with the special case for hallucinated-results which is treated as a detection. - **Mixed-format streams**: each section is processed independently in the `while candidate` loop within `wrap()`; format-A native deltas pass through; formats B/C/D produce synthetic deltas; the hallucinated format produces detection events. - **JSON parse failures inside Format D**: raw bytes are wrapped in `{"_raw": ...}` and emitted; the dispatcher surfaces the failure rather than silently dropping the call. - **XML parse failures inside Format B**: the section is dropped; downstream the absence of any tool call drives the same on-no-progress reflexion path the loop uses for any other empty turn. ### 8.5 Test suite The test suite lives in: - `tests/llm/test_lenient_parser.py` — format A/B/C/D detection, fuzzy resolver, edge cases. - `tests/llm/test_lenient_parser_fuzzy.py` — fuzzy-name resolution corner cases. - `tests/llm/test_lenient_parser_hallucination.py` — V3-Q.3 nine-test suite covering hallucinated-results detection. - `tests/llm/test_router_lenient_wiring.py` — end-to-end router wiring. The argument coercer has its own coverage in the agent-test layer (`tests/agent/test_arg_coercion.py`); the catalog gate is exercised by the loop's broader integration tests. ## 9. Evaluation We are explicit about what is and is not measured. **What is measured.** The nine-test V3-Q.3 hallucination-detection suite passes. The Format A/B/C/D detection tests pass. The fuzzy resolver tests pass against synthetic catalogs. The argument coercer tests pass against the three registered tools (`widget_create`, `workspace_create`, `workspace_apply`). The end-to-end live acceptance test against an open-weight model on NIM was the success criterion for the V3-Q development wave and passes on the supported provider configuration. **What is not measured.** No public benchmark exists for tool-call recovery on weak-tool-call models. The format mix in production is from internal observations during V3-Q development; we do not claim a representative distribution across the open-weight ecosystem. Production deployment at scale has not happened — Memagen™ is pre-launch at the time of writing. The fuzzy-resolver threshold (0.7) was chosen by inspection of paraphrase patterns observed during development; we have not run a precision–recall sweep across a labelled corpus. **Per-format coverage.** We catch the four formats described in §3. There are emission patterns we have not seen yet — see §11 on Format E. The catalog gate (§6.2) is the safety net for unknown patterns: any tool name that fails to match a catalog entry is rejected with a structured error rather than silently dispatched. We list these limits because the alternative — claiming benchmark numbers we have not measured — is dishonest. The [master paper §12](./2026-05-08-memagen-master/) takes the same position. ## 10. Discussion Lenient parsing gives up strictness as a runtime-enforced contract. A strict-reject runtime is a stronger guarantee that output is canonical, and a stronger guarantee is in some sense better. The cost is a runtime that only works against frontier models, which prices out the BYOLLM economics that justify the open-core distribution. What lenient parsing gains: - **BYOLLM economics.** Customers run Memagen™ against open-weight models on commodity hardware without sacrificing tool reliability — the primary commercial justification. - **Faster iteration on smaller models.** A new open-weight model can be tested against the existing tool surface without waiting for post-training to converge on a canonical emission shape; if the model emits any of the four supported formats, the parser handles it. - **Dev-experience on weak models.** A developer prototyping against a local Llama 3.x or Qwen 2.x deployment sees their tools work, rather than being forced onto a frontier model to debug. Strict rejection and lenient recovery are not mutually exclusive: capability enforcement (the V3-P vault, [master paper §5](./2026-05-08-memagen-master/)) is strict and runs *after* the lenient parser. A leniently-parsed call still passes the capability gate to dispatch, so a malformed envelope cannot bypass capability checks. ## 11. Conclusion and future work The lenient parser bridges weak-tool-call models to the native tool catalog by recognising four production-observed emission patterns, applying registry-based per-tool argument coercion, fuzzy-resolving paraphrased tool names, gating off-catalog calls with a structured error, and detecting-and-suppressing hallucinated `<function_results>` blocks. The implementation is open-source under MIT in the Memagen™ Lite distribution. Future work: - **Format E.** Emission patterns we have not yet observed. The catalog gate is the current safety net; a richer telemetry layer that flags unknown opener-like sequences for human review would let us extend coverage proactively rather than reactively. - **Coercer-as-LLM.** The registry-based coercer is per-tool work. A small dedicated model could in principle learn the coercion mapping from observed (loose, canonical) pairs and generalise to new tools without per-tool registration. The trade-off is one more model in the dispatch path; whether the latency cost is acceptable depends on tool-call frequency. - **Streaming-aware coercion.** The current coercer operates on the fully-assembled argument string. A streaming-aware coercer could begin reshaping arguments before the closing sentinel arrives, reducing dispatch latency by a few hundred milliseconds on long argument blobs. - **Cross-provider format normalisation.** The `fmt` tag is currently per-call; a richer tagging scheme could record which provider, which model variant, and which catalog the call was issued against, supporting per-provider observability dashboards (see [LLM router paper](./2026-05-08-llm-router/)). - **Hallucinated-results telemetry.** The `_HALLUC_MAX_RETRIES = 2` cap is a fixed bound. A more nuanced policy — e.g., escalate to provider switching after two hallucinations on the same turn — is plausible future work that depends on the multi-provider router's switching costs. ## 12. References 1. **Levitas, T.** (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Master paper, Memagen™. 2. **Levitas, T.** (2026). *Multi-Provider LLM Routing for Tool-Calling Agents*. Specialty paper, Memagen™. 3. **Anthropic.** (2024). *Tool use with Claude*. <https://docs.anthropic.com/claude/docs/tool-use> 4. **Anthropic.** (2024). *Model Context Protocol Specification*. <https://modelcontextprotocol.io> 5. **Beurer-Kellner, L., Fischer, M., & Vechev, M.** (2023). *Prompting is programming: A query language for large language models*. PLDI 2023. 6. **Chase, H.** (2022). *LangChain: Building applications with LLMs through composability*. <https://github.com/langchain-ai/langchain> 7. **DeepSeek-AI.** (2024). *DeepSeek-V3 technical report*. arXiv:2412.19437. 8. **Google.** (2024). *Function calling with the Gemini API*. <https://ai.google.dev/docs/function_calling> 9. **Meta AI.** (2024). *The Llama 3 herd of models*. arXiv:2407.21783. 10. **Moonshot AI.** (2024). *Kimi K2 technical report*. Moonshot AI Research. 11. **OpenAI.** (2023). *Function calling and other API updates*. <https://openai.com/blog/function-calling-and-other-api-updates> 12. **Ouyang, L., Wu, J., Jiang, X., et al.** (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022. 13. **Qwen Team, Alibaba.** (2024). *Qwen2 technical report*. arXiv:2407.10671. 14. **Ratcliff, J. W., & Obershelp, D. E.** (1988). *Pattern matching: The Gestalt approach*. Dr. Dobb's Journal, 13(7), 46–51. 15. **Willard, B.T., & Louf, R.** (2023). *Efficient guided generation for large language models*. arXiv:2307.09702. 16. **Wright, A., Andrews, H., Hutton, B., & Dennis, G.** (2019). *JSON Schema: A media type for describing JSON documents*. IETF Internet-Draft draft-handrews-json-schema-02. --- ## Paper: 2026-05-08-llm-router URL: https://memagen.com/papers/2026-05-08-llm-router PDF: https://memagen.com/papers/2026-05-08-llm-router.pdf --- title: "Multi-Provider LLM Routing for Tool-Calling Agents: A Provider-Abstraction with Lenient-Parser Splice Points" subtitle: "Routing, failover, per-agent override, and the ContextVar-based splice point that makes the lenient parser provider-aware" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "Most current agent platforms hard-code a single LLM provider — typically OpenAI or Anthropic — and pay the resulting tax in rate limits, billing concentration, geographic constraints, and model availability. Memagen™'s open-core router targets a wide provider surface — OpenAI-compatible endpoints (covering NVIDIA NIM, Groq, Together, DeepSeek, Mistral, Fireworks, Cerebras, xAI, Perplexity, Voyage, Cohere's compatibility endpoint, OpenRouter, Ollama, llama.cpp, vLLM, LM Studio), Anthropic Messages, and Google Gemini's compatibility endpoint — behind a single `ChatRequest` type. We describe the router as it exists at `core/llm/router.py`: a `LLMRouter` class that resolves an `agent_id` to an `AgentRoute`, applies per-agent overrides, dispatches against a single provider or a pool with `round_robin` / `least_loaded` selection, fails over on `RateLimitError`, and accumulates streaming tool-call deltas into a unified shape regardless of upstream provider. The novel contribution is two splice points the router exposes for the lenient parser: an `_apply_scrub` step that publishes `_ACTIVE_MODEL_FOR_HINT` and `_ACTIVE_TOOL_CATALOG_FOR_HINT` into Python `ContextVar`s for the system-prompt assembler, and a `_stream_with_credential` / `_stream_with_failover` step that wraps the provider stream in `lenient_parser.wrap()` for models flagged by `model_needs_lenient`. We also describe the `claude-bridge` shim (an OpenAI-compatible HTTP wrapper around the `claude` CLI's OAuth session), the `settings_bridge` that splices onboarding-committed credentials into the YAML config, and the unified streaming event taxonomy. We are explicit about what is not yet supported: cost-aware routing, latency-aware routing, batched-call APIs." keywords: ["LLM router", "provider abstraction", "multi-provider", "BYOLLM", "tool calling", "failover"] kind: specialty subsystem: "LLM router" pages: 7 version: "v1.1" --- ## 1. Introduction The boundary between an agent runtime and a large language model is one place an architectural choice has compounding cost. Most agent frameworks make that choice once: pick OpenAI, pick Anthropic, write the integration, ship. The cost is paid afterwards — every time the chosen provider raises a rate limit, removes a model, raises prices, suffers an outage, becomes unavailable in a customer's region, or offers a worse cost-per-token ratio than a competitor on the workload at hand. Customers of those frameworks inherit the choice, with no recourse short of rewriting the integration themselves. Memagen™ (Levitas, 2026) takes the opposite position. The router lives between the agent loop and any provider. A single `ChatRequest` (defined at `core/llm/types.py`) is the canonical request shape; every provider implementation translates that shape into the upstream API's native form, runs the call, and translates the streaming response back into a unified event vocabulary. The Lite distribution exercises three implementation kinds — `openai`, `openai_compatible`, and `anthropic` — and reaches a wide surface of upstream targets: OpenAI directly, Anthropic Messages directly, and any OpenAI-compatible endpoint (NVIDIA NIM, Groq, Together, DeepSeek, Mistral, Fireworks, Cerebras, xAI, Perplexity, Voyage, Cohere's compatibility endpoint, OpenRouter, Google Gemini's compatibility endpoint, and any local server that speaks the OpenAI Chat Completions wire format including Ollama, llama.cpp, vLLM, and LM Studio). A wide provider surface behind a thin canonical request shape is well-trodden in the routing-library literature; what is novel here is how the router participates in the lenient-parser layer described in [Lenient Tool-Call Parsing](/papers/2026-05-08-lenient-parser). The router is not just a dispatch table. It exposes two splice points that make the lenient parser provider-aware: (a) an `_apply_scrub` step that publishes the active model name and the live tool catalog into `ContextVar`s the system-prompt assembler reads, so the prompt carries the right tool-format hint for the current model; and (b) a `_stream_with_credential` / `_stream_with_failover` step that wraps the provider's raw stream in `lenient_parser.wrap()` for models that need it. Both ship under MIT in the Lite distribution and let a single deployment mix a frontier-model orchestrator with a cheaper-model executor without the executor's idiosyncratic tool-call emissions breaking the agent loop. Section 2 surveys related routing libraries. Section 3 describes the `ChatRequest` type and the router's public API. Section 4 walks each supported provider and names the idiosyncrasy it forces into the abstraction. Section 5 is the core contribution: the splice points. Section 6 covers failover and pools. Section 7 covers per-agent routing and the `settings_bridge`. Section 8 documents the `claude-bridge` shim. Section 9 names the streaming event taxonomy. Section 10 lists honest limitations. Section 11 names future work. ## 2. Related work **LiteLLM** (BerriAI, 2023) is the most-cited Python routing library; it provides a unified `completion()` interface over more than a hundred provider endpoints. We considered it and chose not to. First, LiteLLM is structured around per-call dispatch and does not expose the kind of streaming event hooks the lenient parser needs — splicing an arbitrary async generator over the upstream stream while preserving back-pressure and cancellation semantics. Second, it optimizes breadth over depth of tool-call support, and the per-provider tool-call shape translation in `OpenAICompatibleProvider._iter_sse` and `AnthropicProvider._iter_anthropic_sse` (Section 4) is the load-bearing part for an agent platform. Extending LiteLLM would be larger than writing the routing layer directly, and we wanted the splice points the router exposes today. **LangChain** (Chase, 2022) provides chat-model abstractions via classes like `ChatOpenAI`, `ChatAnthropic`, `ChatGoogleGenerativeAI`. The unification works at the model-instantiation layer rather than at the request-routing layer; per-agent overrides, pools, and failover live in user code, not in the framework. Fine for chains; incomplete for agent loops where the routing decision needs to be made per call based on agent identity. **OpenRouter** (OpenRouter Inc., 2024) is an inference marketplace: a single API endpoint that itself routes to many providers, with provider preference expressible via headers. We support OpenRouter as a back-end (it appears as an `openai_compatible` provider) but do not depend on it. OpenRouter does the routing-layer job well; it cannot insert per-event hooks inside the provider's streaming response, and the splice we need is inside the stream. **LangSmith** and similar observability tools (Helicone, Langfuse, Phoenix) sit alongside the router rather than in it; they consume the events the router emits. Memagen™'s observability uses a Prometheus counter (`memagen_dispatch_total`, labeled by `provider` and `success`, incremented in the `finally` block of every dispatch path in `core/llm/router.py`) plus the `core.trace.span` context manager opened around each provider call. Intentionally minimal — observability that lives inside the router survives provider changes. **Direct provider SDKs** (`openai`, `anthropic`, `google-genai`, `cohere`) are excellent at their one provider and do not pretend otherwise. We use them as inspiration for request and response shapes but call providers via raw HTTP through `httpx.AsyncClient` (`Provider.__init__`), because the SDKs each add a non-trivial dependency and their streaming abstractions are not consistent with each other. Each prior approach is incomplete for tool-calling agents specifically. The gap is the splice point the lenient parser needs. The router fills that gap. ## 3. The `ChatRequest` abstraction The canonical request shape is a frozen dataclass at `core/llm/types.py`: ```python @dataclass class ChatRequest: system: str user: str history: list[ChatMessage] = field(default_factory=list) temperature: float = 0.7 max_tokens: int = 4096 top_p: float = 0.95 tools: list[dict] | None = None # OpenAI-format function definitions ``` `ChatMessage` carries the canonical `role` / `content` pair plus optional `tool_calls` and `tool_call_id` for tool-result turns. It also carries a `_tool_call_fmt` field excluded from `to_dict()`; this records which lenient-parser format produced the message (`"native"`, `"kimi_special"`, `"xml_invoke"`, `"xml_sentinel"`) so downstream replay logic can re-emit in the same format. The router's job, in one sentence: take a `ChatRequest`, resolve the calling `agent_id` to a route, build the upstream payload via the chosen provider's `_payload()` method, dispatch, translate the streaming response back into the unified event vocabulary, and return. Three things about this shape are load-bearing. First, `tools` is a list of OpenAI-format function definitions (`{name, description, parameters: <JSONSchema>}`) regardless of upstream provider; the per-provider translation happens inside the provider. `AnthropicProvider._payload` translates `parameters` to `input_schema` and re-keys the entry; we do not push that translation onto agent code. Second, `system` is a single top-level string, not a list of system messages; this matches Anthropic's `Messages` API contract (`system` at the top level, separate from `messages`) and degrades cleanly into a system message when the provider is OpenAI-compatible (`build_messages` prepends `{"role": "system", "content": self.system}` to the message list). Third, `history` is a list of `ChatMessage`, not dicts; the dataclass-as-history shape carries the `_tool_call_fmt` annotation without leaking it into the wire format. The router is constructed from an `LLMSettings` and an optional `CredentialStore`: ```python class LLMRouter: def __init__( self, settings: LLMSettings, *, credential_store: CredentialStore | None = None, ) -> None: ... ``` It exposes three primary async methods: `complete(request, *, agent)` returns the content string only; `complete_with_tools(request, *, agent, ...)` returns `(content, tool_calls, usage)`; `stream(request, *, agent, ...)` and `stream_with_tools(request, *, agent, ...)` are the streaming variants. The streaming variant accumulates `tool_call_delta` events into final `ToolCall` objects (lines 834–847 of `core/llm/router.py`) so callers do not reassemble fragments themselves. ## 4. Provider abstraction Each provider subclasses an abstract `Provider` (`core/llm/provider.py`) and implements `complete()` and `stream()`. The base class holds a single `httpx.AsyncClient` configured with the router's shared timeout. Providers are stateless given `(api_key, model, request)`; they know nothing about pools, rate limits, budgets, agent identity, or the lenient parser. The router composes those concerns around them. The current registry, at `core/llm/provider.py`: ```python PROVIDER_REGISTRY: dict[str, type[Provider]] = { "openai_compatible": OpenAICompatibleProvider, "openai": OpenAIProvider, "anthropic": AnthropicProvider, } ``` Three implementation classes cover the entire provider surface, because the OpenAI-compatible wire format has become the de facto standard for non-Anthropic and non-Google-native APIs. The `_PROVIDER_SHAPES` table at `core/llm/credential_route.py` maps user-facing names onto `(kind, base_url)` pairs: | User-facing name | Kind | Base URL | |------------------|---------------------|-----------------------------------------------------------| | `nvidia` | `openai_compatible` | `https://integrate.api.nvidia.com/v1` | | `openai` | `openai` | (SDK default) | | `anthropic` | `anthropic` | (SDK default) | | `google` | `openai_compatible` | `https://generativelanguage.googleapis.com/v1beta/openai` | | `groq` | `openai_compatible` | `https://api.groq.com/openai/v1` | | `mistral` | `openai_compatible` | `https://api.mistral.ai/v1` | | `together` | `openai_compatible` | `https://api.together.xyz/v1` | | `fireworks` | `openai_compatible` | `https://api.fireworks.ai/inference/v1` | | `deepseek` | `openai_compatible` | `https://api.deepseek.com/v1` | | `xai` | `openai_compatible` | `https://api.x.ai/v1` | | `cerebras` | `openai_compatible` | `https://api.cerebras.ai/v1` | | `cohere` | `openai_compatible` | `https://api.cohere.ai/compatibility/v1` | | `perplexity` | `openai_compatible` | `https://api.perplexity.ai` | | `voyage` | `openai_compatible` | `https://api.voyageai.com/v1` | | `ollama` | `openai_compatible` | `http://localhost:11434/v1` | OpenRouter, NVIDIA NIM, llama.cpp, vLLM, and LM Studio reach the router via the same `openai_compatible` kind with their respective `base_url` values supplied by the user (LM Studio defaults to `http://localhost:1234/v1`; vLLM uses whatever the operator passes to `--port`). The user-facing list named in the master paper is not a list of distinct integrations but a list of *attestations*: we have run agents against each of them. Most of the engineering in provider abstraction is per-provider streaming-event translation. The non-streaming `complete` paths are largely shape-mapping; the SSE handler is the hard part. Each kind's idiosyncrasies are documented below. ### 4.1 OpenAI OpenAI itself is the reference implementation (`OpenAIProvider`, a thin subclass of `OpenAICompatibleProvider`). Tool calls arrive in the `tool_calls` array on the assistant message, each with an `id`, a `type: "function"` discriminator, and a `function: {name, arguments}` payload where `arguments` is a JSON-encoded *string*, not a dict. Streaming SSE events arrive as `data: <json>` lines; `choices[0].delta` carries either `content`, `tool_calls`, or `reasoning` fragments; the stream terminates with `data: [DONE]`. We request a final `usage` frame via `stream_options: {include_usage: true}` (`core/llm/provider.py` line ~194). Tool definitions are sent as `[{type: "function", function: {name, description, parameters}}]`. ### 4.2 Anthropic Anthropic does not use the OpenAI shape. The `AnthropicProvider` (`core/llm/provider.py` line ~219) speaks the Messages API directly. Two structural differences: `system` is a top-level field (not a message), and the response `content` is a list of typed blocks rather than a single string. Block types include `text`, `tool_use`, and `thinking`. Tool calls arrive as `tool_use` blocks with `id`, `name`, and an `input` dict (already parsed JSON, unlike OpenAI's string). We JSON-encode `input` back to a string so downstream `ToolCall.arguments` matches the OpenAI contract. Streaming is more involved. The Messages API emits events of types `content_block_start`, `content_block_delta`, `content_block_stop`, and `message_stop`. A `tool_use` block opens via `content_block_start` and is filled via subsequent `content_block_delta` events of type `input_json_delta`, whose `partial_json` field carries string fragments of the eventual JSON arguments. We track open tool blocks in a per-stream `dict[int, dict[str, str]]` keyed by `index`, emit a synthetic opening `tool_call_delta` event when the block opens (so the accumulator records `id` and `name` before fragments arrive), and forward each `partial_json` fragment as a fresh `tool_call_delta` event with the same `index`. The accumulator concatenates fragments into a single `arguments` string. This translation lives at `_iter_anthropic_sse` (`core/llm/provider.py` lines ~301–378). Thinking blocks (extended-thinking output, available on `claude-opus-4-7` and similar) arrive as `content_block_delta` events of type `thinking_delta` with a `thinking` text payload. We forward these as `("reasoning", text)` events on the unified stream — the same kind a Kimi-style `delta.reasoning` produces on the OpenAI-compatible side. We had to update the provider when thinking blocks landed in 2025; provider-specific code is the biggest source of churn in the codebase, and we expect it will continue to be. ### 4.3 Google Gemini Gemini reaches the router through the `openai_compatible` kind via Google's `generativelanguage.googleapis.com/v1beta/openai` endpoint — an OpenAI-compatibility shim Google publishes. The native Gemini API has function calls in `candidates[].content.parts[].functionCall` and a different streaming envelope; we have not implemented a native `GeminiProvider` because the compatibility endpoint is sufficient for the agent loop. Safety filtering, when triggered, surfaces as an HTTP 400 with a structured `promptFeedback` payload; we currently surface this as a generic `RuntimeError`, which the agent loop relays to the user. A proper `GeminiProvider` is on the roadmap; the current behavior is not great when safety filters fire. ### 4.4 OpenRouter OpenRouter is `openai_compatible` with `base_url = "https://openrouter.ai/api/v1"`. Routing is by model name (`anthropic/claude-opus-4-7`, `google/gemini-2.5-pro`). Provider-preference headers (`X-OR-Prefer-Provider`) can be passed through `ProviderSpec.extra` and set in custom headers; the current `OpenAICompatibleProvider._headers` does not yet wire these — adding pass-through of `extra.headers` is on the future-work list. ### 4.5 NVIDIA NIM NIM is `openai_compatible` with `base_url = "https://integrate.api.nvidia.com/v1"`. The same endpoint serves Kimi (`moonshotai/kimi-k2.5`), Llama, Mistral, and others. NIM emits `delta.reasoning` deltas on Kimi models (the "thinking" output on the OpenAI-compatible side), surfaced as `("reasoning", text)`. The 60-second cooldown on 429 in `_complete_with_credential` (`_CRED_429_COOLDOWN_SEC = 65.0`) is calibrated to NIM's per-minute rate-limit window; the cooldown is followed by a jittered retry up to `_CRED_429_MAX_RETRIES = 2`. ### 4.6 Cohere Cohere reaches the router through its OpenAI-compatibility endpoint (`https://api.cohere.ai/compatibility/v1`). Cohere's native API uses a slightly different tool-use shape with explicit `chat_history`; the compatibility endpoint translates to OpenAI shape, so we do not need a native provider. ### 4.7 Local: Ollama, llama.cpp, vLLM, LM Studio All four expose an OpenAI-compatible Chat Completions endpoint when configured to do so. Ollama defaults to `http://localhost:11434/v1`; llama.cpp's `server` defaults to `http://localhost:8080/v1`; vLLM uses `--api-key` and an operator-chosen port; LM Studio defaults to `http://localhost:1234/v1`. The router treats all four as `openai_compatible`. The `_LOCAL_BACKEND_NAMES` frozenset at `core/llm/router.py` lists `{"ollama", "local", "llama_cpp", "llamacpp", "lm_studio", "lmstudio"}` for the `requires_local_only=True` enforcement path: when an agent or operator policy requires local inference (e.g. for a private payload), the router refuses to route to anything not on that list, with `_select_route` raising `RuntimeError` if no local backend is configured. ## 5. The splice points This is the section the paper exists for. The lenient parser ([Lenient Tool-Call Parsing](/papers/2026-05-08-lenient-parser)) needs two facts to do its job correctly: the active model (so it can decide whether the model needs lenient handling at all) and the active tool catalog (so the fuzzy resolver can rewrite paraphrased tool names like `create_markdown_widget` to the canonical `widget_create`). Neither is naturally present at the system-prompt assembly step; both are present at the router. The router's job at the splice points is to publish them. The figure below sketches the end-to-end path; the splice points are the dashed annotations. ```mermaid flowchart LR subgraph A[Agent loop] REQ[ChatRequest<br/>core/llm/types.py] end subgraph R[LLMRouter — core/llm/router.py] RES[resolve agent_id<br/>_select_route] SCRUB[_apply_scrub<br/>+ ContextVar publish] SEL[provider / pool select<br/>round_robin · least_loaded] DISP[_stream_with_credential<br/>_stream_with_failover] LP[lenient_parser.wrap<br/>if model_needs_lenient] ACC[stream_with_tools<br/>accumulator] end subgraph P[Provider adapters — core/llm/provider.py] direction TB PA[AnthropicProvider<br/>Messages API] POAI[OpenAIProvider] POC[OpenAICompatibleProvider<br/>NIM · Ollama · vLLM · OpenRouter · Gemini compat] end subgraph S[settings_bridge<br/>core/llm/settings_bridge.py] YAML[LLMSettings YAML] DB[onboarding settings DB] ENV[env vars] end REQ --> RES --> SCRUB --> SEL --> DISP --> LP --> ACC ENV -.priority 1.-> YAML YAML -.priority 2.-> RES DB -.priority 3.-> YAML SCRUB -. _ACTIVE_MODEL_FOR_HINT .-> SP[system_prompt assembler] SCRUB -. _ACTIVE_TOOL_CATALOG_FOR_HINT .-> SP DISP --> PA DISP --> POAI DISP --> POC PA -- SSE: content_block_delta · input_json_delta · thinking_delta --> LP POAI -- SSE: delta.content · delta.tool_calls --> LP POC -- SSE: delta.content · delta.tool_calls · delta.reasoning --> LP LP -- ("tool_call_delta", _lenient_fmt) --> ACC LP -- ("hallucinated_results_detected", str) --> ACC ACC --> OUT[unified event stream<br/>reasoning · content · tool_call_delta · usage · done] ``` ### 5.1 `_apply_scrub` — `ContextVar` publication The `_apply_scrub` method at `core/llm/router.py` lines 374–528 has two responsibilities. The first is the privacy scrubber (V3.1.6 territory; not this paper). The second is the splice. Lines 411–425: ```python try: from core.learning.system_prompt import ( _ACTIVE_MODEL_FOR_HINT, _ACTIVE_TOOL_CATALOG_FOR_HINT, ) if model_name: _ACTIVE_MODEL_FOR_HINT.set(model_name) catalog_names = tuple( t.get("name", "") for t in (request.tools or []) if t.get("name") ) _ACTIVE_TOOL_CATALOG_FOR_HINT.set(catalog_names) except Exception: pass ``` These two `ContextVar`s are imported from `core.learning.system_prompt`. The system-prompt assembler reads them at prompt-build time and injects a tool-format hint listing the catalog names verbatim. The hint is what produces output like *"Available tools: widget_create, fs.read, net.fetch, …. Emit calls in the OpenAI function-calling format. If you cannot, emit `<tool_call>{...}</tool_call>` and we will accept it."* The hint is per-model: a frontier OpenAI model gets the short version; a Kimi or DeepSeek model gets a longer version listing all four accepted formats. Publishing via `ContextVar` rather than threading the model name through every call site is deliberate. The system-prompt assembler is called from the agent loop, not from the router; threading the values would have required adding `model` and `tool_catalog` parameters to roughly twenty intermediate functions. `ContextVar` is the standard async-safe channel for exactly this kind of cross-cutting concern; it scopes correctly across `asyncio` tasks and does not leak across requests. The cost is that the read site must handle the case where the variable was never set (prompt assembled outside a router-driven dispatch); the assembler does this with `_ACTIVE_MODEL_FOR_HINT.get(default="")` and falls back to a generic hint. The `try / except Exception: pass` around the whole block protects the dispatch path from regressing if `core.learning.system_prompt` ever fails to import. The router will dispatch fine without the hint; the lenient parser still does its job; the system prompt just contains the generic version. The privacy scrubber, by contrast, is fail-closed (`raise RuntimeError("refcontext scrubber unavailable; refusing to dispatch — fix and restart")` at line 463). The two failure modes are intentionally different: scrub failure is a privacy-promise violation; hint failure is a quality-of-output regression. ### 5.2 `_stream_with_credential` and `_stream_with_failover` — `lenient_wrap` splice Both streaming-execution paths apply the same wrapper. From `_stream_with_credential` (`core/llm/router.py` lines 1062–1084): ```python from .lenient_parser import ( wrap as _lenient_wrap, model_needs_lenient as _needs_lenient, ) import time as _time _turn_idx = int(_time.monotonic() * 1000) % 99999 _catalog_names = [ t.get("name", "") for t in (request.tools or []) if t.get("name") ] _stream = provider.stream(request, model) if _needs_lenient(model): _stream = _lenient_wrap( _stream, model=model, turn_idx=_turn_idx, tool_catalog=_catalog_names or None, ) async for pair in _stream: streamed_any = True ... yield pair ``` `model_needs_lenient(model_name)` is the gate at `core/llm/lenient_parser.py` line 87. It returns `True` for model families that emit non-canonical tool-call shapes — Kimi K2.x, DeepSeek V3, Llama 3.x via Ollama, Qwen 2.x, Yi-Large, Mistral on Together, and a small number of others — and `False` for OpenAI's GPT family and Anthropic's Claude family, which emit canonical shapes. When the gate returns `True`, the provider's raw async generator is wrapped. The wrapper consumes upstream pairs, runs the streaming state machine that detects the four lenient formats (Format A native, Format B XML-invoke, Format C JSON-in-sentinel, Format D special-tokens), and re-emits a normalised stream of OpenAI-shape `tool_call_delta` events. Each emitted delta carries a `_lenient_fmt` annotation (`"native"`, `"kimi_special"`, `"xml_invoke"`, `"xml_sentinel"`) — see [Lenient Tool-Call Parsing](/papers/2026-05-08-lenient-parser) for the format taxonomy. The router-side accumulator (`stream_with_tools`, `core/llm/router.py` lines 805–808) reads `_lenient_fmt` from each delta and stamps it onto the final `ToolCall.fmt`: ```python if d.get("_lenient_fmt"): slot["_lenient_fmt"] = d["_lenient_fmt"] yield ("tool_call_delta", d) ... tool_calls.append(ToolCall( id=slot["id"] or "", name=slot["name"], arguments="".join(slot["arguments_buf"]) or "{}", fmt=slot.get("_lenient_fmt", "native"), )) ``` The wrapper also synthesizes a `("hallucinated_results_detected", str)` event when it spots a hallucinated `<function_results>` block (lenient-parser paper, V3-Q.3). The router forwards this verbatim: ```python elif kind == "hallucinated_results_detected": yield (kind, raw) ``` The downstream agent-loop listener consumes the event and queues a one-shot directive for the next turn's system prompt. The router itself does not handle the event — its job is forwarding. The splice is symmetric across credential-sourced and failover-sourced dispatch. `_stream_with_failover` (`core/llm/router.py` lines 1241–1259) applies the same `_lenient_wrap` to the same `provider.stream(eff_request, model)`. We considered factoring this into a single helper; we did not, because the credential path and the failover path differ in how they treat 429 (the credential path retries on the same provider with `_CRED_429_COOLDOWN_SEC`; the failover path moves to the next pool member with the pool's `on_429_cooldown_seconds`), and trying to share the streaming body without sharing the retry semantics produced more confusing code than the duplication. ### 5.3 Why this lives in the router A reader could reasonably ask why the lenient-parser splice is at the router rather than at the provider. Three reasons. First, the lenient parser is a property of the *model*, not the *provider*. Kimi K2.5 emits XML-invoke tool calls regardless of whether it is reached through NVIDIA NIM, Together, Fireworks, or a local llama.cpp serving the GGUF. Putting the wrapper inside `OpenAICompatibleProvider` would duplicate it across every provider class that might serve the model. Putting it at the router lets `model_needs_lenient(model)` be the single point of decision. Second, the per-call tool catalog is naturally available at the router (it is part of the `ChatRequest` the router receives) and not naturally available at the provider (which receives the request but does not know which catalog names are in scope versus which are vestigial metadata). The router passes `tool_catalog=_catalog_names` into the wrapper precisely because it has the canonical view. Third, the failover loop applies the wrapper after each retry to the same model on a different provider. If the wrapper lived in the provider, the failover loop would have to re-wrap on each iteration anyway — cleaner to express it once at the router. ## 6. Failover and pools A pool is a named set of providers configured in YAML: ```yaml llm: pools: kimi_pool: members: [nvidia_primary, nvidia_secondary, together_main] strategy: round_robin on_429_cooldown_seconds: 60 ``` `PoolSpec` and the parser are at `core/llm/settings.py` lines 57–63. `Pool.select()` (`core/llm/pool.py`) implements two strategies. `round_robin` walks the member list via `itertools.cycle`, skipping members the rate limiter reports as cooling. `least_loaded` picks the member with the smallest `RateLimiter.load(name)[0]` (in-flight count). When all members are cooling, both fall through to `_least_bad_fallback`, which picks the member coming off cooldown soonest — surfacing `RuntimeError("all providers exhausted")` mid-conversation is worse than waiting a few seconds. `_complete_with_failover` and `_stream_with_failover` (`core/llm/router.py` lines 1171–1282) implement the failover loop. `_attempt_sequence` builds it: for a pool, pick a starting member via the strategy and append remaining members in order so a 429 on the first retries the others before giving up; for a single-provider route, return `[provider_name]`. On `RateLimitError`, the limiter is told to start a cooldown via `self._limiter.on_429(spec.name, cooldown)` and the loop continues. On any other exception, the loop re-raises immediately — failover is a 429-specific behavior, not a fault-tolerance behavior. Mid-stream 429s (`if streamed_any: raise`) are surfaced rather than retried, because the user has already seen partial output and a transparent retry would produce duplicate content. The credential path (`_complete_with_credential`, `_stream_with_credential`) has its own retry loop with `_CRED_429_COOLDOWN_SEC = 65.0` and `_CRED_429_MAX_RETRIES = 2`. The cooldown is followed by a jittered sleep: `sleep_for = self._CRED_429_COOLDOWN_SEC + random.uniform(0, 30)`. The jitter is the load-bearing detail. Without it, multiple agents that share the same credential and hit 429 together would all wake at exactly cooldown+0 and hammer the provider in lockstep, producing a synchronized retry storm. The 30-second jitter window decorrelates them. A circuit breaker per `(provider_name, credential_label)` from `core.resilience.breaker.CircuitBreaker` (`failure_threshold=5`, `reset_timeout=30.0`) sits one layer below the rate-limit retry. Five consecutive failures opens the breaker; while open, every call short-circuits to `RuntimeError(f"provider {spec.name} circuit open; cooldown")`. Keying on `(provider, credential)` rather than just `provider` is intentional: two distinct API keys against the same provider should not share a fault budget — a user's bad key should not blackhole everyone else's. ## 7. Per-agent routing Memagen™ assigns providers to agents at the YAML layer: ```yaml llm: default: claude_bridge default_model: claude-opus-4-7 agents: orchestrator: {provider: anthropic_main, model: claude-opus-4-7} dev: {provider: kimi_pool} expert.security: {provider: openai_main, model: gpt-4.1} background: {provider: nvidia_primary, model: moonshotai/kimi-k2.5, temperature: 0.3} ``` Lookup is dotted-hierarchical. `LLMSettings.lookup("expert.security")` (`core/llm/settings.py` lines 86–93) walks `expert.security → expert → default` and returns the first match. `_label_candidates` at `core/llm/router.py` lines 197–206 implements the same walk for the credential-store lookup path. The hierarchy lets users assign a default provider to a broad agent category (`expert: {provider: openai_main}`) and then specialize for a subagent (`expert.security: {provider: anthropic_main, model: claude-opus-4-7}`) without restating the parent. The shared-key shortcut: when `llm.shared_key: true`, every agent uses the default route regardless of `agents` declarations. This is the path for the single-key BYOLLM flow where the user has typed one API key in the onboarding panel and does not care to assign different providers per agent. ### 7.1 `settings_bridge` — onboarding to YAML The `settings_bridge` at `core/llm/settings_bridge.py` is the runtime translation from the onboarding flow's committed answers into the YAML the router parses. The user clicks through onboarding; their choices land in the `settings` SQLite table as `llm.provider_choice`, `llm.providers.chosen.api_key` (encrypted at rest), and `llm.providers.chosen.model`. `synthesize_from_settings(yaml_data)` reads those values, looks up the user-facing name in `_ONBOARDING_TO_YAML_NAME`: ```python _ONBOARDING_TO_YAML_NAME: dict[str, str] = { "anthropic": "anthropic_main", "openai": "openai_main", "nvidia": "nvidia_primary", "groq": "groq_main", "together": "together_main", "local_ollama": "local_ollama", } ``` …and returns a new YAML-shaped dict with a synthesized `llm.providers.<chosen>` block, the corresponding `base_url` from `_DEFAULT_BASE_URLS`, and the API key written as `api_key_literal` (not `api_key_env`, because the value lives in the encrypted DB rather than in an environment variable). Override priority is documented in the module docstring: env vars > explicit YAML > settings DB. If the YAML already declares the synthesized provider name AND that block references an env var that is set, the YAML wins. This is what lets a developer with `ANTHROPIC_API_KEY` exported in their shell run Memagen™ without the onboarding flow stepping on their environment, while letting an end-user who has never set an env var get a working router from the GUI alone. The function is fail-open. Any exception reading the DB, any missing key, no admin user, malformed YAML — all return the input unchanged. This is the right default for a transformation that runs on every router construction; the router falls back to the YAML defaults rather than refusing to start because the settings DB is locked. ### 7.2 `CredentialStore` lookup A second per-agent lookup path runs against an optional `CredentialStore` (`core/llm/credentials_store.py`). When the store is present, every dispatch consults it first via `_credential_for(agent)`. The walk is: exact agent label → dotted parents → `MAIN_LABEL` (the literal string `"Main"`). When `core.settings.get_shared_credentials()` is `True`, the `background` and `secondary` agents also fall back to `Main` — this lets a user with one paid key cover both agents while still picking a smaller model in the Background slot. The credential store is the runtime channel for the GUI Credentials panel: the user types a key, picks a provider, picks a model, clicks save, and the next agent dispatch reads the row. `set_credential_store(store)` swaps the store at runtime and drops cached `credential:<label>` provider entries so the next call rebuilds against the fresh row. ## 8. The `claude-bridge` shim The `claude-bridge` is a Memagen™ contribution worth naming. Anthropic's Claude Code CLI ships with an OAuth-managed session — once the user runs `claude login`, the local CLI can call Claude without a separate API key. The `claude-bridge` is an OpenAI-compatible HTTP shim wrapping the Claude Code session and exposing it on a local port. The router treats it as a regular `openai_compatible` provider (`base_url = "http://localhost:<port>/v1"`, `api_key_literal = ""` because the bridge does not require one). The motivation is straightforward. A founder building Memagen™ on a tight budget does not want to pay for tokens twice — once through a Claude Code Max subscription and again through a separate Anthropic API key. The bridge lets the development build use the Claude Code session for orchestrator-tier reasoning while end-users of the same code path use their own Anthropic API key (or any other provider) without rewriting the integration. We expect end-users will not need it; we leave it in the Lite distribution because some advanced users have the same constraint. The honest constraint: the bridge depends on the `claude` CLI's session staying alive. If the session is invalidated upstream, every downstream agent dispatch fails until the user runs `claude login` again. The fallback is the standard Anthropic provider with `api_key_env: ANTHROPIC_API_KEY`; users who do not want the dependency configure that instead. ## 9. Streaming and event taxonomy The unified streaming interface is `AsyncIterator[tuple[str, Any]]`. The taxonomy: - `("reasoning", str)` — thinking-mode delta. Sourced from `delta.reasoning` on the OpenAI-compatible side (Kimi K2.x, NIM-served reasoning models) and from `content_block_delta` of type `thinking_delta` on the Anthropic side. - `("content", str)` — answer-text delta. Sourced from `delta.content` on the OpenAI-compatible side and from `content_block_delta` of type `text_delta` on the Anthropic side. - `("tool_call_delta", dict | str)` — partial tool-call delta. Always emitted in OpenAI shape (`{index, id, function: {name, arguments}}`) regardless of upstream provider; the Anthropic provider translates `input_json_delta` events into this shape, and the lenient parser translates XML-invoke / JSON-in-sentinel / Kimi-special-token emissions into this shape. The optional `_lenient_fmt` field annotates which format the parser detected. - `("usage", dict | str)` — final token usage. Emitted at end of stream by providers that support it; the OpenAI-compatible path requests it explicitly via `stream_options: {include_usage: true}`. The router emits a placeholder `{"prompt_tokens": -1, "completion_tokens": -1}` if the upstream did not produce one, so the consumer can rely on the field's presence. - `("hallucinated_results_detected", str)` — V3-Q.3 synthetic event from the lenient parser. The string payload is the hallucinated content (suppressed from visible output but preserved for the loop listener so it can be diagnosed). - `("done", dict)` — only emitted by `stream_with_tools`; carries `{content, tool_calls, usage, reasoning}` as the final aggregated payload after the consumer has seen all deltas. The taxonomy is small on purpose. Adding event kinds is cheap; subtracting them is expensive. We have resisted requests to add per-event metadata channels (timestamps, costs, latencies) the consumer can derive from the events it already has. ## 10. Limitations We list these because the alternative — claiming completeness we do not have — is dishonest. **Provider quirks change.** Anthropic added `thinking` blocks in 2025; we updated. OpenAI is rolling out a new `responses` API (Stateful Responses); we have not yet adopted it because the current `chat.completions` path remains supported and is the workload we have tested. When OpenAI deprecates `chat.completions`, we will rewrite `OpenAIProvider`. Provider abstractions that try to be schema-stable across deprecations end up with a least-common-denominator problem; we do not solve that. **Tool-call shape variation across providers is the biggest source of provider-specific code in the codebase.** Most of `_iter_anthropic_sse` exists because Anthropic's streaming tool-use shape is genuinely different from OpenAI's. Most of the lenient parser exists because non-frontier OpenAI-compatible providers each emit tool calls in their own way. We do not see a path to compressing this work; it is the job. **Failover is best-effort and non-magical.** If the operator configures a single provider with no pool, there is nothing to fail over to. The router returns `RuntimeError("all providers exhausted")` after the credential-path retries are exhausted. We do not silently fall back to a different provider behind the user's back, because that would create a billing surprise. **No batched-call support.** Anthropic's Message Batches API (50% discount on async batched submissions) and OpenAI's Batch API are both unsupported. The router's interaction model is per-request streaming or per-request completion; mapping that onto a batch submission with poll-back semantics is a non-trivial change. On the future-work list but not scoped. **No cost-aware or latency-aware routing.** The router picks the route declared in the YAML. It does not currently look at the cost of each provider in the current pool and prefer the cheaper one for the current task. Also on the future-work list. **No native Gemini provider yet.** The Gemini compatibility endpoint covers the agent loop; the native function-calling shape with `responseSchema` and structured output does not. Adding `GeminiProvider` is on the roadmap. **No native Cohere provider with chat-history shape.** The compatibility endpoint covers the agent loop; the native chat-history shape with `connectors` is not supported. **Provider-preference headers (OpenRouter).** The current `OpenAICompatibleProvider._headers` does not pass through the `extra` field's headers, so OpenRouter provider-preference routing requires editing the provider class. A small wiring task on the future-work list. ## 11. Future work Listed by likelihood of being shipped in the next two releases. **Provider-`extra` header pass-through.** Plumb `ProviderSpec.extra.headers` through `OpenAICompatibleProvider._headers` and `AnthropicProvider._headers`. Enables OpenRouter `X-OR-Prefer-Provider`, NIM custom routing headers, and per-org tagging. **Native Gemini provider.** Implement `GeminiProvider` against `generativelanguage.googleapis.com` directly, with structured-output support and proper safety-filter event surfacing. **Cost-aware routing.** Within a pool, prefer the member whose model has the lowest cost-per-token for the request's expected work, biased toward members with the most cooldown headroom. Requires extending `BudgetTracker` (`core/llm/cost.py`) with a per-(provider, model) cost table and a per-request "expected tokens" estimate. **Latency-aware routing.** Within a pool, prefer the member with the lowest p50 latency over a rolling window. Requires the rate limiter to record per-call latencies in addition to in-flight counts. **Provider-specific tool-call format auto-detection.** We currently know the format-likelihood of each model family at code time (`model_needs_lenient` is a static lookup). A learned variant could observe the first few responses from a new model and infer the right format. The static table is fine for now; the learned variant becomes interesting when we ship a provider catalog the user can extend at runtime. **Batched-call API support.** Wrap Anthropic Message Batches and OpenAI Batch into a separate `BatchRouter` that exposes `submit_batch(requests) → batch_id` and `poll_batch(batch_id) → results`. Different shape from the streaming router; do not try to unify. **ContextVar plumbing for cost and latency.** The `_apply_scrub` ContextVar splice has worked well for model name and tool catalog. Cost and latency budgets could be published the same way for tools that want to make cost-aware decisions inside the agent loop. ## 12. References 1. **Anthropic.** (2024). *Claude Messages API reference: tool use, content blocks, and streaming*. <https://docs.anthropic.com/claude/reference/messages_post> 2. **Anthropic.** (2025). *Extended thinking with Claude*. <https://docs.anthropic.com/claude/docs/extended-thinking> 3. **BerriAI.** (2023). *LiteLLM: Call 100+ LLM APIs using the OpenAI format*. <https://github.com/BerriAI/litellm> 4. **Chase, H.** (2022). *LangChain: Building applications with LLMs through composability*. <https://github.com/langchain-ai/langchain> 5. **Levitas, T.** (2026). *Memagen™: An open-core agent platform with lenient tool routing, capability-gated execution, and a compound-graph memory substrate*. Memagen™ master paper, this series. 6. **Levitas, T.** (2026). *Lenient Tool-Call Parsing: Bridging Weak-Tool-Call Models to Native Tool Catalogs*. Memagen™ specialty paper, this series. <https://memagen.com/papers/2026-05-08-lenient-parser> 7. **OpenAI.** (2024). *Function calling and structured outputs*. <https://platform.openai.com/docs/guides/function-calling> 8. **OpenAI.** (2024). *Streaming responses with the Chat Completions API*. <https://platform.openai.com/docs/api-reference/chat-streaming> 9. **OpenRouter Inc.** (2024). *OpenRouter API documentation*. <https://openrouter.ai/docs> 10. **NVIDIA.** (2024). *NVIDIA NIM: API catalog and OpenAI-compatible endpoints*. <https://build.nvidia.com> 11. **Google.** (2024). *Gemini API: OpenAI-compatibility endpoint*. <https://ai.google.dev/gemini-api/docs/openai> 12. **Cohere.** (2024). *Cohere chat API and OpenAI compatibility*. <https://docs.cohere.com/docs/compatibility-api> 13. **Ollama.** (2024). *OpenAI-compatible API server*. <https://github.com/ollama/ollama/blob/main/docs/openai.md> 14. **vLLM Project.** (2024). *vLLM OpenAI-compatible server*. <https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html> Source: <https://github.com/tablevitas123/crackedclaw> --- ## Paper: 2026-05-08-pcmcp URL: https://memagen.com/papers/2026-05-08-pcmcp PDF: https://memagen.com/papers/2026-05-08-pcmcp.pdf --- title: "PCMCP: A Polymorphic Compound MCP for Unified Agent Tool Surfaces" subtitle: "Merging UIMCP, AutoMCP, AutoWebMCP, and accessibility-tree desktop control into one namespace with capability inheritance and per-tool gating" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "Production agent stacks accumulate MCP surfaces faster than they can manage them: a catalog-compression layer (UIMCP), an inferred-macro layer (AutoMCP), an auto-generated-from-the-web layer (AutoWebMCP), one or more raw vendor MCPs, and — for desktop control — a vision-free accessibility-tree surface. PCMCP is the Polymorphic Compound MCP: a single namespace the agent sees that is the merge of all of these. The merge resolves naming collisions deterministically (origin-prefix on conflict), inherits each child surface's declared capability requirements onto the merged tool entry, and gates every tool call at a per-tool permission decision evaluated against the V3-P capability vault and the safety toggle framework. The polymorphic part is the dispatcher: a single `pcmcp.invoke(name, args)` call routes to UIMCP-compressed catalog entries, AutoMCP macros, AutoWebMCP-generated web tools, raw MCP server tools, or the desktop accessibility-tree surface, depending on which child owns the resolved name. This paper describes the merge algorithm, the disambiguation rules, the capability-inheritance model, the dispatch layer, and the desktop accessibility-tree surface in detail (because it is the principal Memagen™-original child surface and is the work in progress at the time of writing). The accessibility-tree path reads the OS-maintained widget tree on Linux (AT-SPI 2), Windows (UI Automation), and macOS (AXUIElement), translates the agent's semantic commands to OS events through the existing `core/ui_control.py` input dispatcher, and runs one to two orders of magnitude faster than vision-based control where the tree is exposed. PCMCP ships in the Memagen™ Lite distribution under MIT. The merge layer is the new architectural contribution; the input dispatcher and multi-platform tool wrapper (`core/tools/computer.py`) are production code today; the accessibility-tree introspection is the novel computer-control contribution and is in progress." keywords: ["polymorphic mcp", "tool namespace merge", "capability inheritance", "computer control", "accessibility tree", "AT-SPI", "UI Automation", "AXUIElement", "agent runtime"] kind: specialty subsystem: "PCMCP" pages: 9 version: "v1" --- ## 1. Introduction A non-trivial agent stack has more than one MCP. UIMCP [2] compresses a large catalog so the agent does not pay tokens for every tool every turn. AutoMCP [3] watches the action graph and proposes parametric macros that get promoted to first-class catalog entries. AutoWebMCP [4] turns arbitrary web pages into runnable tools by parsing form structure and OpenAPI sniffs. Raw vendor MCPs ship per service — a filesystem MCP, a shell MCP, a Slack MCP, a GitHub MCP. And, for desktop control, there is a vision-free accessibility-tree surface (described in Sections 4–6) that competes with screenshot-and-click loops at one to two orders of magnitude lower latency and roughly zero per-action tokens. Five surfaces, five conventions, five name-resolution behaviors. A naive agent prompt that exposes all five at once gets a tool list with collisions (`search` on the filesystem MCP, `search` from a generated webmcp), inconsistent capability declarations (some MCPs declare `requires` strings, some don't), and ad-hoc gating (the user set a destructive-action toggle on the desktop surface but it doesn't apply to the macro layer that wraps the same primitives). The agent now has to know which "search" it meant and what permissions each child MCP enforces. PCMCP — Polymorphic Compound MCP — is the unifier. It exposes one MCP surface to the agent. Internally it is a merge: every child surface (UIMCP, AutoMCP, AutoWebMCP, raw vendor MCPs, the desktop accessibility-tree surface) registers its tools under an origin tag; PCMCP resolves names with a deterministic precedence rule, prefixes on conflict, propagates each child's declared capability requirements onto the merged entry, and gates every dispatched call at a single per-tool permission decision against the V3-P capability vault and the safety toggle framework. The polymorphic part is the dispatcher: `pcmcp.invoke(name, args)` routes to whichever child owns the resolved name, with a single audit record per call regardless of which child handled it. This paper has two halves. Sections 2–3 cover the merge architecture: child registration, naming and disambiguation, capability inheritance, the dispatcher, and the gating model. Sections 4–7 cover the desktop accessibility-tree surface — the principal Memagen™-original child surface — in implementation detail, because it is the most novel of the children and is the work in progress at the time of writing. Sections 8–12 cover security, evaluation, limits, and future work. PCMCP ships in the Memagen™ Lite distribution under MIT. The merge layer and the dispatcher are new architectural contributions; the input substrate (`core/ui_control.py`) and multi-platform tool wrapper (`core/tools/computer.py`) are production code today; the accessibility-tree introspection is the novel computer-control contribution and is in progress. The master Memagen™ paper [1, §9.4] names PCMCP at the capability level and points to this paper for implementation detail. ## 2. Architecture: the polymorphic compound PCMCP is a merge over child MCP surfaces. The merge is a function of three things: the child registry, the name-resolution rule, and the capability-inheritance rule. The diagram below shows the resulting layout. ```mermaid flowchart TB Agent["Agent (LLM)"]:::agent subgraph PCMCP["PCMCP — Polymorphic Compound MCP"] direction TB NS["Merged namespace<br/>(deterministic precedence,<br/>origin-prefix on conflict)"]:::merge CAP["Capability inheritance<br/>(union of child requires +<br/>destructive-label allowlist)"]:::merge GATE["Per-tool permission gate<br/>(V3-P vault token +<br/>safety toggles + kill switch)"]:::gate DISP["Polymorphic dispatcher<br/>pcmcp.invoke(name, args)"]:::merge NS --> CAP --> GATE --> DISP end subgraph Children["Child MCP surfaces"] direction LR UIMCP["UIMCP<br/>compressed catalog<br/>(see paper [2])"]:::child AutoMCP["AutoMCP<br/>inferred macros<br/>(see paper [3])"]:::child AutoWebMCP["AutoWebMCP<br/>web tools from URLs<br/>(see paper [4])"]:::child RawMCP["Raw vendor MCPs<br/>(filesystem, shell,<br/>Slack, GitHub, …)"]:::child DesktopA11y["Desktop accessibility-tree<br/>surface (§4–6)<br/>AT-SPI / UIA / AXUIElement"]:::child end subgraph Substrate["Shared substrate"] direction LR UI["core/ui_control.py<br/>input dispatcher<br/>(_guard, _rate_limit)"]:::sub Tool["core/tools/computer.py<br/>multi-platform tool<br/>(playwright/xdotool/adb)"]:::sub Vault["V3-P capability vault<br/>(per-use Fernet tokens)"]:::sub Toggles["Safety toggle framework<br/>(default-strict, hard invariants)"]:::sub end Agent -- "single tool list +<br/>pcmcp.invoke" --> NS DISP -- "routes by origin" --> UIMCP DISP --> AutoMCP DISP --> AutoWebMCP DISP --> RawMCP DISP --> DesktopA11y DesktopA11y --> UI DesktopA11y --> Tool GATE --> Vault GATE --> Toggles classDef agent fill:#fff,stroke:#111,stroke-width:1.5px; classDef merge fill:#fff7c2,stroke:#1f2937,stroke-width:1.5px; classDef gate fill:#fde68a,stroke:#92400e,stroke-width:1.5px; classDef child fill:#f3f4f6,stroke:#111,stroke-width:1px; classDef sub fill:#ffffff,stroke:#333,stroke-width:1px,stroke-dasharray: 4 3; ``` A static rendering of the same diagram is available at `/papers/diagrams/pcmcp.svg`. ### 2.1 Child registration Each child surface registers with PCMCP at startup by handing over a tool list and a child-descriptor: ```python @dataclass class ChildDescriptor: origin: str # "uimcp", "automcp", "autowebmcp", # "raw:filesystem", "desktop_a11y" precedence: int # lower wins; ties broken by name requires: list[str] # capability strings the child claims # e.g. ["fs.read", "ui.input"] destructive_labels: list[str] # optional: child-supplied label # patterns that should always gate invoker: Callable[[str, dict], Awaitable[ToolResult]] tools: list[ToolSchema] # the wire-format MCP tool schemas ``` The default precedence ordering, lower-wins: 1. `desktop_a11y` (precedence 10) — accessibility-tree desktop control, the most narrowly scoped surface 2. `raw:*` (precedence 20) — vendor MCPs the user explicitly connected 3. `automcp` (precedence 30) — macros over the above 4. `autowebmcp` (precedence 40) — auto-generated web tools 5. `uimcp` (precedence 50) — the catalog-compressed pool, lowest because it summons rather than owns Precedence is not security-relevant — every dispatch still runs through the same gate — but it determines which child wins on a name collision when the user has not specified a prefix. ### 2.2 Name resolution and disambiguation PCMCP exposes a single flat namespace to the agent. On registration, each child's tool names are stored as `(origin, name)` pairs in an internal registry. The agent-visible name follows three rules, in order: 1. **Exact, no conflict** — if exactly one child registered the name, the agent sees the bare name (`search_files`). 2. **Exact, conflict** — if more than one child registered the same name, the conflict is resolved by precedence; the loser is auto-prefixed with its origin (`autowebmcp:search_files`). Both names are exposed; the bare name resolves to the precedence winner. 3. **Explicit prefix** — the agent can always write `origin:name` (`raw:filesystem:search`), bypassing precedence. This is the disambiguation hatch. Conflict events are logged as structured records — origin, name, precedence winner, full collision set — and surface in the developer-mode UI so the user can rename a generated tool if the auto-prefix is undesirable. The audit log makes "which `search` did the agent call last Tuesday" a real query. ### 2.3 Capability inheritance The merged tool entry's `requires` list is the union of: - the child's declared `requires` (e.g., AutoWebMCP-generated tools require `net.http_outbound`; the desktop a11y surface requires `ui.input`), - a static per-origin inheritance map (e.g., `automcp` macros inherit `requires` from every underlying tool the macro template invokes — a macro that opens a file and posts a message inherits `fs.read` and `net.http_outbound`), - the destructive-label set merged from each child (`Delete`, `Send`, `Pay`, etc., from the desktop a11y surface; `wire_transfer`, `delete_repo`, etc., from raw vendor MCPs). Inheritance is conservative: a merged entry's required-capability set is a superset of any single child's, never a relaxation. The agent cannot circumvent a child's gate by routing through a macro; the macro carries every underlying requirement onto the merged dispatch. ### 2.4 The dispatcher `pcmcp.invoke(name, args)` is the single agent-facing entry point. The flow is: 1. **Resolve** the name via §2.2's rules. If unresolvable, return `tool_not_found` with the candidate set. 2. **Gate** through §2.3's merged `requires` set. The gate makes one decision: it requests a per-use capability token from the V3-P vault for each required capability, evaluates relevant safety toggles (e.g., `pcmcp.allow_destructive` if the args match a destructive-label pattern), and checks the multi-source kill switch via `core.safety.guard`. On any failure: structured error and one audit record. 3. **Dispatch** to the resolved child's `invoker(name, args)`. The child is unaware of the gating; it gets called only after the gate passes. 4. **Audit** one record per `pcmcp.invoke` call: requested name, resolved `(origin, name)`, capability tokens consumed, args summary, child result status. One per call, regardless of child. A single dispatch path means uniform safety semantics across child surfaces. A user who turns off `pcmcp.allow_destructive` blocks every destructive-label match, whether the call would have hit the desktop a11y surface, an AutoMCP macro that wraps it, or a generated webmcp form labeled "Delete account." ### 2.5 What the polymorphism buys The polymorphism is not a hand-wave: a single dispatcher with origin-keyed routing is mechanically simpler than five parallel dispatchers, and the simplicity is the point. There is one place to add an audit field, one place to add a kill-switch check, one place to interpret a capability token, one place where the destructive-label allowlist is consulted. Adding a sixth child surface — say, an Android Accessibility Framework path — is a `ChildDescriptor` registration, not a fork of the dispatch logic. ## 3. Related work ### 3.1 MCP namespace strategies elsewhere The Model Context Protocol specification [13] is silent on namespace merge across multiple connected servers. Existing client implementations adopt one of three patterns: flat union with first-registered-wins (simple, lossy on conflict), per-server prefix on every tool (verbose, no bare-name shortcut), or interactive prompting on conflict (poor agent ergonomics). PCMCP's deterministic-precedence-with-loser-prefix pattern is a fourth option that preserves bare-name access for the precedence winner while still exposing every conflicting tool under an unambiguous name. ### 3.2 Capability inheritance in distributed systems Capability-based security has a long lineage (Hardy, KeyKOS, 1985 [14]). The "inherit on compose" semantics PCMCP uses for merged tool entries follows the principle that a composed action requires the union of its constituents' authorities. The contribution here is the engineering, not the principle: applying the union rule to a runtime MCP merge with per-tool labels has not, to our knowledge, been done in the published agent-platform literature. ### 3.3 Vision-based computer control The dominant pattern for agent computer control — Anthropic's Claude Computer Use (2024) and OpenAI's Operator (2025), among others — is a tight loop with three stages: screenshot, vision-LLM call, dispatch. The cost model is roughly one to five seconds and one to ten thousand tokens per action; for a twenty-action workflow that compounds into tens of seconds and tens of thousands of tokens. The desktop accessibility-tree surface inside PCMCP (§4–6) is the alternative for the large fraction of work where the OS already exposes the same widget tree the vision LLM is being asked to re-derive pixel-by-pixel. ### 3.4 Browser-only structured automation Playwright (Microsoft, 2020), Puppeteer (Google, 2017), and Selenium (Huggins, 2004) have demonstrated for two decades that browser automation does not need vision. Each operates against the DOM and dispatches events to elements identified by selectors. The DOM is to the browser as the accessibility tree is to the desktop; AutoWebMCP [4] is the analogous structured-surface answer for the web. ### 3.5 Pre-LLM accessibility-tree automation Assistive technology has been driving applications via the accessibility tree for thirty years (NVDA, JAWS, VoiceOver, Orca). Test automation for desktop applications — Microsoft's WinAppDriver, Apple's XCUITest, Linux's dogtail — is accessibility-tree-driven. The novelty of the desktop a11y surface is not the technique; it is applying it as the primary path for an LLM agent's computer-control surface, with semantic commands ("click the Save button") rather than positional matchers. ## 4. The desktop accessibility-tree surface The remainder of this paper covers one child of the polymorphic compound — the desktop accessibility-tree surface — in implementation detail. The merge architecture above applies independently of which child we describe; we focus on this one because it is the most novel and is in active development. Each platform exposes essentially the same conceptual structure — a tree of nodes with role, label, bounds, state, value, and children — but the bindings differ and the role taxonomies do not align cleanly. ### 4.1 Linux: AT-SPI 2.x AT-SPI 2 (Assistive Technology Service Provider Interface) is the freedesktop.org standardized accessibility surface, exposed over D-Bus. Applications register their accessibility trees with the at-spi2-registry daemon at startup; clients query the registry. The Python bindings are `pyatspi` (legacy) or `gi.repository.Atspi` (the GObject-introspection-based current binding). A typical query: ```python import gi gi.require_version("Atspi", "2.0") from gi.repository import Atspi desktop = Atspi.get_desktop(0) for app_index in range(desktop.get_child_count()): app = desktop.get_child_at_index(app_index) # walk app.get_child_at_index(...) recursively ``` Each `Atspi.Accessible` exposes: - `get_role()` / `get_role_name()` — role enum and human-readable name (push_button, text, menu_item, frame, table_cell, ...) - `get_name()` — the visible label - `get_state_set()` — bitset of states (focused, enabled, selected, visible, showing, sensitive) - `get_extents(coord_type)` — bounding rectangle in screen coordinates - `get_child_count()` / `get_child_at_index(i)` — children - The Component, Action, Text, Value, and EditableText interfaces, conditional on the node's role AT-SPI is well-supported by GTK and Qt (with Qt Accessibility enabled, the default on Linux Qt builds). It is poorly supported by Electron with `--disable-features=AccessibilityTree`, Java Swing without `assistive_technologies` set, and any application that renders directly to a canvas (Figma, photo editors, games). ### 4.2 Windows: UI Automation Microsoft UI Automation (UIA) is the successor to MSAA and has been the supported Windows accessibility API since Vista. UIA is reachable from C++, .NET, and Python (via the `uiautomation` PyPI package, which wraps the COM API). ```python import uiautomation as auto root = auto.GetRootControl() # walk root.GetChildren() / root.GetFirstChildControl() recursively ``` Each `Control` (the wrapper around `IUIAutomationElement`) exposes: - `ControlType` / `ControlTypeName` — Button, Edit, MenuItem, Window, ListItem, etc. - `Name` — the visible label - `BoundingRectangle` — screen coordinates - `IsEnabled`, `HasKeyboardFocus`, `IsSelected`, etc. - `GetChildren()`, `GetParentControl()` - Pattern interfaces (`InvokePattern`, `ValuePattern`, `TextPattern`, `SelectionItemPattern`) conditional on control type UIA is well-supported across Win32, WPF, WinForms, UWP, and modern WinUI. Browser content within Edge and Chrome maps DOM nodes to UIA elements. Java AWT/Swing requires the Java Access Bridge; Electron generally exposes UIA on Windows because Chromium does. Games and custom-rendered tooling are the usual gaps. ### 4.3 macOS: AXUIElement macOS uses the Accessibility Framework, with the `AXUIElement` C type as the core handle, reachable from Swift, Objective-C, and Python (via `pyobjc`'s `ApplicationServices` bindings). ```python from ApplicationServices import ( AXUIElementCreateSystemWide, AXUIElementCopyAttributeValue, kAXChildrenAttribute, kAXRoleAttribute, kAXTitleAttribute, ) sys_wide = AXUIElementCreateSystemWide() # AXUIElementCopyAttributeValue(sys_wide, kAXChildrenAttribute, ...) returns # the running applications' top-level AXUIElement nodes ``` Each `AXUIElement` exposes attributes via string keys: `AXRole` (AXButton, AXTextField, AXMenuItem, ...), `AXTitle` or `AXValue`, `AXPosition`/`AXSize` (origin and size in screen coords with macOS's top-left convention), `AXChildren`, `AXFocused`, `AXEnabled`. macOS requires an explicit user grant: the application driving AXUIElement queries must be granted Accessibility permission in System Settings. The first PCMCP call on macOS triggers a system prompt; without the grant, AXUIElement queries return errors. PCMCP records the consent state in the V3-P capability vault. Cocoa, Catalyst, and most Electron applications expose rich AX trees. Native non-Cocoa Carbon applications (rare in 2026) expose less. Games and custom-canvas applications are the same gap as the other platforms. ### 4.4 The common abstraction The desktop a11y surface's core type is platform-neutral: ```python @dataclass class AccessibleNode: role: str # normalized: "button", "textfield", # "menuitem", "link", "checkbox", ... label: str # the visible name value: str | None # current value, for inputs state: set[str] # {"focused", "enabled", "selected", ...} bounds: tuple[int, int, int, int] # x, y, width, height in screen px children: list["AccessibleNode"] backend_handle: object # opaque OS-specific handle for # programmatic actions where avail. ``` The role taxonomy is a small normalized set — roughly thirty roles. Examples: AT-SPI `push_button` → `button`, `text` (when editable) → `textfield`, `menu_item` → `menuitem`; UIA `Button` → `button`, `Edit` → `textfield`, `MenuItem` → `menuitem`; AXUIElement `AXButton` → `button`, `AXTextField` → `textfield`, `AXMenuItem` → `menuitem`. The mapping table is open-source; it has to be done once per OS release and audited when toolkits change. The `backend_handle` is the OS-native object — `Atspi.Accessible`, `IUIAutomationElement`, `AXUIElement` — kept around so the surface can issue programmatic actions through the platform's own interfaces (AT-SPI's `Action`, UIA's `InvokePattern`, AX's `AXPress`). Programmatic invocation is preferred over coordinate-based clicking when supported: faster and not dependent on the target window being unobscured. The coordinate-based path is the fallback. ## 5. The translation layer The translation layer turns a semantic agent command into an OS event. The API is small; the implementation is per-platform. ### 5.1 click_button(label, role="button") 1. Walk the active accessibility tree (cached or refreshed on demand). 2. Filter nodes by role (default `button`). 3. Match `label` exactly. If none, fall back to bounded Levenshtein fuzzy match, reusing the resolver Memagen™ uses for tool-name correction in the lenient parser [1, §4]. If the best match exceeds the distance threshold, return a structured `not_found` rather than guessing. 4. If the matched node exposes a programmatic-invocation interface (AT-SPI `Action`, UIA `InvokePattern`, AX `AXPress`), invoke it directly. 5. Otherwise, compute the screen-space center of `bounds` and call `UIController.click(x, y)`, dispatching through the chosen input backend. 6. Optionally re-walk the subtree to verify state changed. Verification is opt-in; it doubles the latency. Fuzzy match is bounded. Strict-only is available; the default is strict-then-fuzzy with a small distance budget. Fuzzy matches log both the requested and matched label so any case where the agent acted on a non-exact label is auditable. ### 5.2 type_into(field, text) 1. Find the textfield by label, using the accessibility relation between input and label node (`LABEL_FOR`/`LABELLED_BY` on Linux, `LabeledBy` on Windows, `AXTitleUIElement` on macOS). 2. Focus the field via `AXFocused = true` (macOS), `SetFocus()` (UIA), or `Atspi.Component.grab_focus()` (Linux). Fallback: click the field's bounds center. 3. Issue keystrokes through `UIController.type_text(text)`. Where the platform exposes a value-set interface (`ValuePattern.SetValue`, `AXValue` write, `EditableText.set_text_contents`), prefer it — faster and not subject to keyboard-layout interference. 4. Optionally verify the field's value matches what was typed. Simulated typing is required for fields that validate per-keystroke; value-set is preferred otherwise. The implementation tries value-set first and falls back on rejection. ### 5.3 Auxiliary commands - `select_menu_item(path=("File", "Save As..."))` — walk the menu hierarchy by label - `set_checkbox(label, checked=True)` — find checkbox, toggle on state mismatch - `select_dropdown(label, option)` — open dropdown, select by label - `focus_window(title)` — bring an application window forward - `wait_for(role, label, timeout)` — block until a matching node appears - `dump_visible(role_filter=None)` — return the full filtered tree for agent reasoning without a screenshot `dump_visible` is the most agent-friendly operation. A complex form's dump is 50–200 nodes — a few hundred to a few thousand tokens — versus 1,500–3,000 for an equivalent screenshot, with more semantic information. ### 5.4 The fallback to vision When the desktop a11y surface cannot find what the agent asked for, the response is structured: ```python { "status": "not_found", "requested": {"role": "button", "label": "Save"}, "candidates": [ {"role": "button", "label": "Save & Close", "distance": 7}, {"role": "button", "label": "OK", "distance": 4}, ], "tree_complete": True, # or False if the dump was partial "suggestion": "fall_back_to_vision_or_disambiguate", } ``` The agent can disambiguate (the requested label was wrong; a candidate is the right one) or fall back to vision (the target is not in the tree). Try fast, fail fast, tell the agent why. ## 6. The implementation backbone The desktop a11y surface does not invent an input dispatcher. It builds on `core/ui_control.py` and `core/tools/computer.py`, both of which ship today in the Memagen™ Lite distribution under MIT. ### 6.1 core/ui_control.py — the input substrate `core/ui_control.py` exposes a `UIController` class with a uniform API over pluggable input and screenshot backends. Auto-selection priority for input: 1. **`PyAutoGuiInput`** — cross-platform default. Wraps `pyautogui`, which wraps platform input APIs (Windows `SendInput`, macOS `CGEventCreateMouseEvent`, Linux X11 via Xlib). Supports `click`, `move`, `type_text`, `key`, `scroll`. 2. **`XdotoolInput`** — Linux/X11 fallback. Uses `xdotool` subprocesses with the safe argv form (`xdotool type --delay N -- TEXT`). The `--` separator and no-shell argv exec are deliberate: an earlier f-string-into-shell form was a command-injection vector when typed text was attacker-controlled. 3. **`StubInput`** — test backend. Records every call with no real I/O. Screenshots are symmetric: `MssScreenshot` (fast, cross-platform), `PilScreenshot` (fallback), `StubScreenshot` (1×1 PNG for tests). `UIController` adds two safety layers on top of the backends: - `_guard()` — calls `core.safety.guard(stage="ui_control")` before every action, raising `KillSwitchTripped` if the multi-source kill switch is set. The PCMCP dispatcher inherits this; an action proposed during a kill-switch event is refused at the `UIController` boundary before reaching the input backend. - `_rate_limit()` — `min_action_interval` enforces a minimum gap between successive actions, preventing runaway input loops. Wiring all desktop output through `UIController` means the desktop a11y surface and any vision-based path share the same input backend, kill-switch state, and rate limit. No second code path to keep in sync. ### 6.2 core/tools/computer.py — the multi-platform tool `core/tools/computer.py` is the LLM-tool-facing wrapper around the same input layer plus backends not handled by `core/ui_control.py`. The `Backend` literal is `"playwright" | "xdotool" | "adb" | "none"`, and `_detect_backend()` returns the first available: ```python def _detect_backend() -> Backend: if shutil.which("adb"): # adb available but don't auto-select (need device check) pass if shutil.which("xdotool") and shutil.which("scrot"): return "xdotool" return "none" ``` The four LLM-facing tools — `computer_screenshot`, `computer_click`, `computer_type`, `computer_launch` — are async and dispatch through `_run_argv`, a hardened subprocess wrapper: ```python async def _run_argv(argv: list[str], timeout: float = 10.0): proc = await asyncio.create_subprocess_exec(*argv, ...) out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout) return out.decode(...), err.decode(...), proc.returncode ``` The argv-only form (no shell) is deliberate: every previous f-string-into-shell call site was a command-injection vector when args were attacker-controlled. `_validate_coord` clamps coordinates to `[0, 16000]`. `computer_type` enforces a 4096-character ceiling. `computer_launch` uses `shlex.split` and `shutil.which` to validate before dispatch, refusing empty, oversized, and off-`PATH` commands. The desktop a11y surface (§5) calls into `core/ui_control.py` for desktop control on Linux/macOS/Windows. The ADB backend handles Android targets; the Android Accessibility Framework is a candidate child surface for future PCMCP work (§11.4). ### 6.3 The boundary The desktop a11y surface's contribution is the accessibility-tree introspection and the translation layer. Everything below — input backends, kill-switch guard, rate limiter, screenshot backends — is existing Memagen™ infrastructure. The translation layer is testable in isolation against a stub `UIController`; the input layer is testable in isolation against synthetic semantic commands. ## 7. Cost comparison: a11y-tree path vs vision-based control Vision-based numbers below are published latency characteristics of frontier vision LLMs as of early 2026; the a11y-tree numbers are measurements on internal harness runs with the open-source code. ### 7.1 Vision-based "click Save" | Stage | Latency | Token cost | | --- | --- | --- | | Screenshot capture (mss) | 30–80 ms | 0 | | Image encoding (base64) | 5–20 ms | 0 | | Vision LLM call (frontier, 1024×768 PNG, "click the Save button" prompt) | 1.5–5 s | 1,500–3,000 in / 100–500 out | | Coordinate parse | <1 ms | 0 | | Click dispatch (pyautogui) | 10–30 ms | 0 | | **Total** | **1.55–5.13 s** | **1,600–3,500 tokens** | ### 7.2 a11y-tree "click_button(label='Save')" | Stage | Latency | Token cost | | --- | --- | --- | | Tree walk (cached, ~50–200 nodes) | 1–5 ms | 0 | | Tree walk (cold, ~200 nodes via AT-SPI/UIA/AX) | 20–80 ms | 0 | | Label match (strict + bounded fuzzy) | <1 ms | 0 | | Programmatic invoke (where supported) | 5–20 ms | 0 | | Coordinate-based click dispatch (where invoke not available) | 10–30 ms | 0 | | **Total (cached tree, programmatic invoke)** | **6–25 ms** | **0 per-action** | | **Total (cold tree, coordinate click)** | **30–115 ms** | **0 per-action** | ### 7.3 Speedup The cached-tree-plus-programmatic-invoke path is 60–800× faster than the vision path. The cold-tree-plus-coordinate-click path is 13–170× faster. The token-cost ratio is bounded only by the size of the (one-time) tree dump the agent needs to reason over the screen; for a complex form, that dump is roughly equivalent to one screenshot in tokens, but it can be reused across many actions on the same screen. The 10–100× factor in the subtitle is conservative. It is the right ratio to plan against: real workflows have caching misses, fuzzy fallbacks, and verification re-walks that the headline "1,000×" number from the cached-invoke path does not capture. ### 7.4 Caveat A direct head-to-head latency benchmark against a vision LLM would require a frontier-model API key and the cost of running the comparison. The vision-LLM numbers in §7.1 are the published latency characteristics of Claude 4.6 (Anthropic, 2025), GPT-5o (OpenAI, 2025), and Gemini 2.5 Pro Vision (Google, 2025) as of early 2026, sourced from each provider's documentation rather than from a Memagen™-run benchmark. The a11y-tree numbers are measurements on internal harness runs on Ubuntu 24.04 with GTK and Qt applications. Honest framing: the desktop a11y surface is an order of magnitude faster than vision-based control on workflows where the tree is exposed; the exact factor depends on the platform, toolkit, and workload. ## 8. When the desktop a11y surface doesn't work There are several classes of application where the surface fails or degrades. ### 8.1 Custom-canvas applications Figma, Photoshop, Premiere, Blender, and most photo and video editors render to a custom canvas rather than OS-managed widgets. The accessibility tree typically exposes the application window and a small number of toolbars/panels but not individual canvas elements. The surface can drive the toolbar, the menu bar, and the panels, but cannot target an arbitrary canvas point semantically. Vision is required for canvas content. ### 8.2 Stripped-a11y Electron apps Electron applications inherit Chromium's rich accessibility surface. However, some Electron apps run with `--disable-features=AccessibilityTree` or do not enable accessibility at all in their packaged form (Slack and Discord historically had partial coverage; the situation improves yearly). When a11y is stripped, the surface can sometimes fall back to walking the renderer's DOM via the Chrome DevTools Protocol — but this requires the app to expose the debugging port, which most production Electron builds do not. Realistic outcome on stripped Electron: window-level operations work; in-app interactions don't. ### 8.3 Games Games generally do not expose accessibility trees. Some accessible games do (notably Microsoft titles and several recent first-party PS5 titles, though those are not desktop). For desktop games the surface is not the right tool. Vision-based control can handle game input, but agent-driven game-playing is rarely the requested use case. ### 8.4 Java without the access bridge Java AWT/Swing applications expose accessibility only when the Java Access Bridge is installed and `javax.accessibility.assistive_technologies` is set. On a default Java install, the tree is empty or contains only the top-level frame. Modern JavaFX is better behaved (it integrates with native a11y). Memagen™ documents the configuration step; without it, those apps require vision. ### 8.5 Visual confirmation steps There are workflows where vision is genuinely needed even when the accessibility tree is rich. "Verify the chart renders correctly," "the report has no spurious labels," "the diagram looks like the design mock" — these are visual judgments that no a11y framework captures, because no framework describes the *appearance* of a rendered widget. The surface cannot replace vision here; the answer is the hybrid agent (§8.7). ### 8.6 Cross-platform inconsistencies A given application — say, Visual Studio Code — exposes a different tree on macOS (rich, Cocoa-bridged), Windows (rich, UIA-bridged via Chromium), and Linux (less rich; depends on the build). A script that targets a label that exists on one platform may not find the same label on another. The pragmatic answer: write per-platform when high reliability is needed, and let fuzzy match cover small label discrepancies. ### 8.7 The hybrid path The production-realistic agent uses the desktop a11y surface first and falls back to vision-based control on `not_found`. The agent's prompt presents both as available tools; the cheaper, faster path is preferred when the agent has reason to expect the tree contains the target; vision is the escape hatch. A more ambitious hybrid captures a screenshot only on a11y failure, sends it to a small fast vision model with the failure context (the tree dump, the requested label, candidates that didn't qualify), and uses the resulting coordinate. This raises effective coverage to "any application a vision model can drive" while preserving the cost benefit on the (large) fraction where the surface succeeds. Not yet shipped. ## 9. Security and capability Computer control is one of the highest-leverage agent capabilities. An agent with `click_button(label="...")` and `type_into(field="...", text="...")` can do everything a desktop user can, including operations the user did not intend. PCMCP's gate (§2.4) is the same gate every other tool runs through [1, §5–6]; this section spells out how it composes for the computer-control case. ### 9.1 V3-P capability vault Each invocation requests a per-use capability token from the V3-P vault before dispatch. The token carries the action kind (`pcmcp.click_button`, `pcmcp.type_into`, `pcmcp.set_checkbox`, ...), an optional payload-summary constraint (e.g., "the typed text begins with..."), and a time-to-live. The token is consumed exactly once and audited as a structured record. Every PCMCP-driven mouse click and keystroke is therefore queryable as data: "show me every `pcmcp.type_into` against the field labeled 'password' in the past 30 days" is a structured query. ### 9.2 Safety toggle: pcmcp.allow_destructive PCMCP exposes one user-controllable safety toggle: `pcmcp.allow_destructive`. Default off. When off, calls whose resolved arguments match the destructive-label allowlist — buttons labeled "Delete," "Remove," "Send," "Submit," "Pay," "Purchase," "Sign," "Format," "Erase," and similar — are blocked at the gate. The agent receives a structured error and can ask the user to flip the toggle. When on, the toggle has been loosened through the standard ceremony: a five-bullet warning, a verbatim acknowledgment phrase (paste-detection blocks paste-only entries), re-authentication, and a 60-second cooldown. The audit log records the loosening transition. Even with the toggle on, the **final_human_review_on_grant_submission** hard invariant from the safety framework still applies. There is no PCMCP path that submits a job application or a financial transaction without a human-in-the-loop click; that invariant runs after the toggle and cannot be bypassed. ### 9.3 The label-allowlist as a coarse defense The destructive-label allowlist is coarse on purpose. It does not depend on parsing the application's intent; it acts on what the user would see on the button. PCMCP refuses to click "Delete Account" without explicit consent, regardless of which application is on screen. This is not a substitute for application-level authorization (the application enforces its own permissions), but it is a substantive guard against an agent acting on a destructive control by mistake or by prompt injection — and because the allowlist sits at the merged dispatcher (§2.4), it gates calls through every child surface uniformly. ## 10. Evaluation Honest framing: the merge layer and dispatcher are new code with unit-test coverage but no scaled benchmark. The desktop accessibility-tree integration is in active development. The existing input dispatcher (`core/ui_control.py`) and the multi-platform computer-control tool (`core/tools/computer.py`) are production code today, used by the vision-based computer-control path and by direct-coordinate-click paths in tests. What has been measured internally: - AT-SPI tree dumps on Ubuntu 24.04 with GTK 4 / Qt 6 / Cocoa-bridged Electron applications. Tree walk latency 20–80 ms cold, 1–5 ms cached. Programmatic invoke latency 5–20 ms. - Coordinate-based click latency on Linux via `pyautogui` and via `xdotool`: 10–30 ms in both cases. - Failure-mode classification on a small (N≈12) set of applications: GTK GNOME (Files, gedit, Settings) — full coverage. Qt (KDE, Telegram, OBS) — full coverage. Electron (VS Code, Slack desktop) — partial; window-level and in-app navigation work, some custom controls don't expose labels. Browser content (Firefox, Chromium) — full coverage of native chrome, partial of web content depending on page a11y attributes. Custom-canvas (Inkscape, Blender) — no canvas coverage; toolbar/menu coverage works. - Merge-layer microbenchmarks: name resolution ≤ 10 µs against a 200-tool merged registry; gate decision dominated by V3-P token issuance (sub-millisecond) on a hot path. What has not been measured at publication-grade: - Scaled benchmarks on Windows (UIA) and macOS (AXUIElement). These platforms are next on the harness extension path. - Direct head-to-head latency against a vision LLM under live API conditions. As stated in §7.4, vision-LLM numbers are providers' published characteristics, not a Memagen™ benchmark. - Long-tail application coverage. N≈12 is small; a meaningful coverage number requires N in the hundreds across versions and toolkits. - Multi-child collision rates in real catalogs. We do not yet have user-base statistics on how often `(autowebmcp, x)` and `(raw:y, x)` collide in practice; the precedence rule is conservative but the empirical conflict distribution is not yet a reportable number. We list these limitations because the alternative — claiming a benchmark we have not run — would not be honest. The forthcoming v2 of this paper will include scaled-benchmark numbers once the harness has been extended. Until then, the headline claims are structural rather than empirical: where the tree is exposed, the desktop a11y child is an order of magnitude faster than vision-based control at near-zero per-action token cost; where the tree is not exposed, the surface fails fast and tells the agent why; the merge layer keeps every child's safety semantics uniform regardless of which child handled a call. ## 11. Future work ### 11.1 Hybrid a11y-plus-vision child The natural extension is a hybrid child surface that uses the a11y tree as primary and falls back to a small fast vision model only when tree walk returns `not_found`. The fallback is cheap when the model has the failure context (tree dump, requested label, near-miss candidates). This raises effective coverage to the union of a11y and vision while preserving cost benefits on the a11y-covered fraction. Implementation is straightforward; the agent prompt presents both as available tools and reasoning chooses. ### 11.2 Cross-application workflow inference (a11y → AutoMCP) The a11y child's structured action stream is exactly the right input for AutoMCP [3]'s pattern recognizer. A persistent agent observes that the user copies data from a spreadsheet, pastes it into a web form, and submits. The structured introspection makes the source and destination of each copy/paste legible: the spreadsheet cell at coordinate (col=4, row=12), the web form input labeled "Quantity." Recording these structured action chains and proposing them as named macros via AutoMCP — through the same merge — is a substantially stronger building block for cross-application automation than either alone. ### 11.3 Trust-tier UI surfacing Memagen™'s widget framework can surface dispatch confidence in the user-facing UI. When a call is about to dispatch, the trust-tier surface shows: requested action ("click button labeled 'Save'"), resolved child and matched node ("desktop_a11y / button at (812, 540), exact label match"), confidence ("strict"). Fuzzy matches are flagged. Destructive calls escalate to confirmation regardless of toggle state. Users get a always-visible read of what the agent is about to do, in semantic terms rather than coordinate terms. ### 11.4 Android via the Android Accessibility Framework `core/tools/computer.py` already supports an `adb` backend for screenshot, click, and type on Android. The natural extension is a new child surface that walks the Android Accessibility Framework tree (queryable via UIAutomator or the AccessibilityService API) and exposes the same `AccessibleNode` abstraction. This is mostly a porting exercise — the abstraction is the same; the platform glue and the `ChildDescriptor` registration are different. ### 11.5 Memagen™ Pro Local extensions The Pro distribution adds two PCMCP-adjacent capabilities outside the open-core scope: - **Recipe-execution engine** [1, §10.4] — named, parameterized, durable PCMCP sequences with crash-resume. A recipe like "fill out the weekly status form" can be replayed against the same form on the same site after a crash, resuming from the last completed step. - **Cognition modules** [1, §10.5] — preferences and beliefs that bias dispatch behavior. The agent learns that on this user's machine, "Save" buttons tend to be lower-right in dialogs, and the resolver weights toward that prior when fuzzy-matching. Both are out of scope for this paper; they are described at the capability level in the master Memagen™ paper and not disclosed at implementation level. ## 12. Conclusion A production agent stack accumulates MCP surfaces. PCMCP — Polymorphic Compound MCP — gives the agent one. The merge resolves naming collisions deterministically, inherits each child's capability declarations onto the merged entry conservatively, and gates every call at a single per-tool permission decision against the V3-P capability vault, the safety toggle framework, and the multi-source kill switch. The polymorphic dispatcher routes by origin to UIMCP [2], AutoMCP [3], AutoWebMCP [4], raw vendor MCPs, or the desktop accessibility-tree surface, with one audit record per call regardless of which child handled it. Within that compound, the desktop accessibility-tree surface is the principal Memagen™-original computer-control child. Where the OS exposes a usable tree (most native applications, many Electron apps with a11y enabled, most browser chromes), it runs one to two orders of magnitude faster than vision-based control at near-zero per-action token cost. Where the tree is not exposed (custom-canvas applications, stripped-a11y Electron, games), the surface returns a structured `not_found` and the hybrid path falls back to vision. PCMCP is gated through V3-P and the safety toggles uniformly. Every call is capability-tokened, audited, and refused at the dispatcher if the kill switch is set. The destructive-label allowlist refuses obvious destructive actions absent explicit consent. Hard invariants from the safety framework — final-human-review-on-grant-submission in particular — apply regardless of toggle state and regardless of which child surface a destructive call would have routed to. The Lite distribution is sufficient for a complete PCMCP integration. The Pro distribution adds the recipe-execution engine and cognition modules that compound on PCMCP without changing its open-core surface. We invite the field to build on the open work and to evaluate the closed work by what it lets users do. Source: <https://github.com/tablevitas123/crackedclaw>. Master paper: [1]. ## References 1. **Levitas, T.** (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Memagen™ master paper, 2026-05-08. 2. **Levitas, T.** (2026). *UIMCP: Adaptive Catalog Compression for Multi-Tool LLM Agent Dispatch*. Memagen™ specialty paper, 2026-05-08. <https://memagen.com/papers/2026-05-08-uimcp> 3. **Levitas, T.** (2026). *AutoMCP: Inferring Parametric Macros from Typed Action Graphs*. Memagen™ specialty paper, 2026-05-08. <https://memagen.com/papers/2026-05-08-automcp> 4. **Levitas, T.** (2026). *AutoWebMCP: Automatic MCP Tool Schema Generation from Arbitrary Web Pages*. Memagen™ specialty paper, 2026-05-08. <https://memagen.com/papers/2026-05-08-autowebmcp> 5. **Anthropic.** (2024). *Computer use with Claude (beta)*. <https://docs.anthropic.com/en/docs/build-with-claude/computer-use> 6. **OpenAI.** (2025). *Operator system card and capability documentation*. <https://openai.com/index/introducing-operator/> 7. **freedesktop.org.** *AT-SPI 2 specification and Atspi GObject-introspection bindings*. <https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/> 8. **Microsoft.** *UI Automation overview and IUIAutomationElement reference*. <https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32> 9. **Apple.** *Accessibility programming guide for OS X — AXUIElement*. <https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/> 10. **Microsoft.** *WinAppDriver — UI Automation-driven test driver for Windows applications*. <https://github.com/microsoft/WinAppDriver> 11. **NV Access.** *NVDA — Non-Visual Desktop Access screen reader source code*. <https://github.com/nvaccess/nvda> 12. **Microsoft.** (2020). *Playwright — browser automation library*. <https://playwright.dev/> 13. **Anthropic.** (2024). *Model Context Protocol Specification*. <https://modelcontextprotocol.io> 14. **Hardy, N.** (1985). *KeyKOS architecture*. ACM Operating Systems Review, 19(4) (cited via Memagen™ master paper [1] for capability-based security history). 15. **PyAutoGUI authors.** *PyAutoGUI documentation — cross-platform GUI automation*. <https://pyautogui.readthedocs.io/> 16. **freedesktop.org.** *xdotool — command-line X11 automation*. <https://github.com/jordansissel/xdotool> 17. **Bernstein, D.J.** (2014). *The Fernet specification* (cited via Memagen™ master paper [1] for the V3-P vault implementation). <https://github.com/fernet/spec> --- ## Paper: 2026-05-08-safety-toggles URL: https://memagen.com/papers/2026-05-08-safety-toggles PDF: https://memagen.com/papers/2026-05-08-safety-toggles.pdf --- title: "Default-Restrictive Safety Toggles with Hard Invariants: A User-Controllable Framework for AI Agent Capability Boundaries" subtitle: "Twenty toggles, eight invariants no toggle can bypass, and a UX that prices loosening transitions higher than tightening ones" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "We describe the safety-toggle framework that ships in Memagen™ Lite (V1-lite, MIT). Every dangerous capability the platform exposes — auto-sending email, auto-pushing code, lifting the outbound-network allowlist, persisting all session memory, disarming the self-modification kill switch — is gated by a user-controllable toggle that defaults to its restrictive state. The framework's distinguishing properties are (a) deliberate UX asymmetry between loosening and tightening transitions (loosening costs reading a five-bullet warning, typing a verbatim acknowledgment phrase that paste detection rejects, re-authenticating, optionally entering a numeric or list cap, and waiting out a 60-to-900-second cooldown; tightening is one click) and (b) a non-bypassable hard-invariant layer of eight named guards that run after toggle evaluation and cannot be disabled by any toggle setting. We document the registry of twenty toggles, the eight named invariants (`secret_scrub_block`, `authentication_required`, `audit_logging`, `final_human_review_on_grant_submission`, `force_push_to_main_blocked`, `csrf_and_session_protection`, `prompt_injection_external_content_scan`, `pii_export_block`), the transition gate that runs cooldown, paste-rejection, typed-phrase, re-auth, and cap-shape checks before flipping any toggle, and the append-only JSONL audit log at `~/.memagen/safety_audit/YYYY-MM-DD.jsonl` whose entries are written via a temp-file-plus-os.replace dance so partial writes cannot leave a half-line on disk. The framework is open-source under MIT in the Lite distribution and is itself on the forbidden-path list of the sandboxed self-modification subsystem so the toggle layer cannot rewrite its own enforcement code via the agent." keywords: ["AI safety", "default-restrictive", "user controls", "hard invariants", "agent capability boundaries"] kind: specialty subsystem: "safety toggles" pages: 7 version: "v1" --- ## 1. Introduction A useful agent is one whose capability surface is large. A safe agent is one whose capability surface is small. The standard resolution to this tension in production agent systems is one of three: ship a small surface and disappoint customers; ship a large surface and rely on prompt-time refusal; or ship a large surface and rely on the model's own judgment as a last line of defense. None of the three has aged well. The first loses to competitors. The second is bypassed by any reasonably persistent injected instruction. The third treats a system that is *known* to be unreliable as the safety layer for its own actions. Memagen™ takes a fourth position. Every limit the platform enforces is exposed as a user-controllable toggle. The user, not Memagen™, decides which risks are acceptable on their machine, against their accounts, with their data. The platform ships every toggle in its restrictive state and prices loosening transitions deliberately higher than tightening transitions. Returning to a safer configuration is never gated; departing from one is. Beneath every toggle sits a non-bypassable layer of named hard invariants that run *after* toggle evaluation and cannot be disabled by any toggle setting. A toggle says "you may try". An invariant says "but only if these named guards pass". This paper describes the framework's twenty toggles, eight invariants, transition gate, audit log, and the structural reason the safety code itself cannot be rewritten by the agent. We document the implementation in real detail because it is open-source under MIT in the Memagen™ Lite distribution; nothing in this paper is reserved. The framing matters. Every other section of this paper is a consequence of one commitment: the operator holds the keys, and the platform never silently decides on the operator's behalf which risks are acceptable. ## 2. Related work ### 2.1 Capability sandboxes Linux `seccomp-bpf` (Corbet, 2009) and the `gVisor` sandbox (Google, 2018) restrict the system-call surface of a containerized process. `firejail`, `bubblewrap`, AppArmor, and SELinux are siblings. These mechanisms are runtime-enforced at the kernel boundary and not user-driven in the operational sense — once the container is configured, the user is the one being constrained, not the one composing the constraints. Memagen™'s toggles operate one layer up: they are a user-facing UI surface that *the user* configures to constrain the agent the user is operating, not a defense-in-depth mechanism the platform enforces against the user. The two layers compose; the operating-system sandbox is still there. ### 2.2 Browser permission models The closest UX analog to Memagen™'s toggles is the modern browser permission model — geolocation, camera, microphone, notifications, clipboard, USB device access. These start from a default-deny posture, surface a per-origin prompt before granting, and offer a "remember this decision" option. The browser model is good but suffers two well-documented failure modes: prompt fatigue (users reflexively click "allow") and origin confusion (users grant a permission to one site assuming they are granting it to another). Memagen™'s toggles are fewer in number than the lifetime grant prompts a typical user accepts in a year of browsing, are scoped per-machine rather than per-origin, and are presented in a single Settings surface rather than as just-in-time interruptions, which we believe materially reduces the fatigue surface. ### 2.3 iOS / Android permission UX evolution Mobile OS permission UX has converged over a decade toward (a) per-permission prompts at first use, (b) granular categories (e.g. "while using the app" vs. "always" for location), and (c) periodic re-confirmation surfaces ("Photos still has access; review?"). The Memagen™ toggle UX borrows the granularity and the periodic re-confirmation impulse but commits more strongly to the default-deny posture: the platform will not prompt the user just-in-time to loosen a toggle during an action. Loosening is deliberately a Settings-screen gesture, separate from the action that needs it. We discuss this design choice in Section 3. ### 2.4 LLM safety prompts vs. runtime gates A large body of recent work treats agent safety as a prompt-engineering problem: instructions in the system prompt, refusal classifiers on the model output, RLHF-based alignment training. These are all useful and Memagen™ does not argue against them. But a prompt-time defense is by construction subject to prompt-time bypass. Runtime gates that are evaluated *after* the model has decided to act are not. Memagen™'s toggle framework sits squarely in the runtime-gate category. The model can produce any output it likes; the gate decides whether the action runs. ### 2.5 Why "default open with override" loses to "default closed with explicit grant" Two postures are possible: (1) default-open with the ability to deny, or (2) default-closed with the ability to grant. Both are coherent. Both have advocates. Memagen™ chooses (2) for reasons that are obvious on a long enough timeline: in a default-open posture, every new capability the platform adds is enabled by default, and the operator must actively notice and disable each one. In a default-closed posture, every new capability the platform adds requires the operator to actively notice and enable it. Over the course of years, posture (1) accumulates capabilities the operator forgot they granted; posture (2) accumulates only capabilities the operator deliberately wanted. The marginal capability the operator forgot about is precisely the capability an injected prompt or compromised plugin will use. ## 3. Toggle UX contract Every toggle in the framework conforms to the same interaction shape, which is the load-bearing contract of the entire framework. Inconsistency here would be an attack surface — a user who has internalized the contract for one toggle should not be surprised by another. ### 3.1 Restrictive → permissive (loosening) A loosening transition is high-friction by design. The user, having opened the Settings screen and clicked the toggle, faces: 1. **A modal warning.** The modal body is the toggle's `warning_text` field, which is required to be non-empty (validated at construction time in `SafetyToggle.__post_init__`). Every warning text in the registry is structured as five bullet points: what turning this on actually does; the immediate failure mode; the worst-case failure mode; the breadth of the change (this toggle applies to every agent loop, not just the visible one); and a "leave this off unless..." sign-off. The five-bullet structure is enforced socially in the registry, not statically by the type system. 2. **Typed-phrase verification.** Each toggle declares a `typed_phrase` the user must type into a confirmation field. Phrases are short, toggle-specific, and deliberately ungeneric ("Push my code without asking", "Disarm the self-mod kill switch", "Allow destructive shell flags"). The framework validates the phrase via `enforce_typed_phrase`, which strips whitespace and compares case-sensitively. 3. **Paste-rejection.** The UI layer measures the time between the field receiving focus and the submit gesture. If the elapsed time is below a threshold (sub-50ms paste-to-submit is the operational signal), the UI sets `paste_detected=True` on the `TransitionRequest`, and the framework rejects with `reason="paste_detected"`. Pasting defeats the point of a typed phrase: the goal is to force the user to read what they are agreeing to. A user who pastes the phrase from the warning text has not read it; they have transcribed it from one box to another. 4. **Re-authentication.** When `requires.reauth=True`, the user must re-enter their password (separate from a stale session token or auth cookie). The framework treats `reauth_token` as a presence check; the API layer is responsible for actually verifying the password against the credential store. The re-auth is to defeat session-hijack and "left their laptop unlocked" scenarios; a stolen session cookie is not enough to flip a toggle. 5. **Optional cap input.** When `requires.cap_input` is non-None, the user must supply a `CapSpec`: either an `int_value` (e.g. "max 50 emails per day", "max 1000 autonomous ops per day") or a `list_value` (e.g. an allowed-recipients tuple). The framework persists the cap on the resulting `ToggleRecord`; consumers (the email send path, the autonomous-op accounting code) read the cap value out at action time and enforce it. The framework itself does not interpret cap values. 6. **Cooldown.** Each toggle declares a `cooldown_seconds` (60s default; up to 900s for the most dangerous toggles). The framework enforces a minimum elapsed time since the last *loosening* transition of the same toggle. The cooldown bounds how fast a malicious tool catalog or compromised plugin can flip a toggle off and back on to slip a single bad action through. Tightening is never rate-limited. 7. **Audit-log entry.** Every transition (success or denial) writes one JSONL line to `~/.memagen/safety_audit/YYYY-MM-DD.jsonl`. The append is atomic at line granularity (Section 7). ### 3.2 Permissive → restrictive (tightening) Tightening is one click. There is no warning. There is no typed phrase. There is no re-auth. There is no cooldown. There is no cap input. The audit log records the transition and that is the sum of the friction. This asymmetry is the point. The user must be able to reach a safer state at speed. A user who realizes mid-action that auto-send is on, or that the outbound network is unrestricted, must be able to revert in one gesture without facing a wall of confirmation. The platform's interest in slowing down a tightening is exactly zero. ### 3.3 The bulk reset A "Reset all to safe" button is exposed in Settings; it calls `ToggleStore.reset_all_to_safe(actor)`, which iterates the live records and tightens any toggle currently in `PERMISSIVE`. The bulk reset, like a single tighten, is unconditional. ## 4. The twenty-toggle registry The registry exposes twenty toggles across seven categories. Categories are a UI-grouping concept; the runtime treats every toggle by its key. Keys are dotted-namespaced for readability. | Category | Toggle key | |----------|-----------| | `comms` | `email.auto_send`, `email.recipient_allowlist` | | `code` | `code.auto_commit`, `code.auto_push`, `code.allow_core_patch_in_autonomous_loop`, `tools.bypass_catalog_gate`, `agent.bypass_plan_mode`, `tests.allow_red_commit`, `selfmod.enabled`, `selfmod.auto_apply`, `selfmod.kill_switch` | | `fs` | `shell.allow_destructive_flags`, `shell.allow_outside_workspace` | | `net` | `net.allow_outbound`, `net.allow_browser_auto_actions` | | `memory` | `memory.persist_everything`, `dreams.write_during_active_session` | | `sec` | `vault.auto_grant_capabilities`, `autonomous.raise_daily_op_cap` | | `input` | `voice.always_listening` | Every entry is a `SafetyToggle` dataclass record produced once at import time in `core/safety/registry.py`. The dataclass enforces three invariants in `SafetyToggle.__post_init__`: - `default_state` must be `RESTRICTIVE` (the registry will fail at import time if a future entry tries to ship default-permissive) - `description` must be non-empty - `warning_text` must be non-empty - `hard_invariants` must list at least one named invariant To ground the abstract description, here is the full record for `email.auto_send`: ```python EMAIL_AUTO_SEND = SafetyToggle( key="email.auto_send", label="Auto-send email", category="comms", default_state=ToggleState.RESTRICTIVE, description=( "Allow agents to send email immediately, without showing the draft " "to you first." ), warning_text=( "Auto-send is dangerous. When this is on:\n" "- Memagen will send emails immediately, without showing them to you first.\n" "- You will not get a chance to catch hallucinated facts, wrong recipients, " "or unintended attachments before they leave your account.\n" "- Sent email cannot be unsent. Recipients may save, forward, or screenshot it.\n" "- A bug, prompt injection, or compromised tool catalog could weaponize " "your inbox before you intervene.\n" "- Auto-send applies to every agent loop running under your account.\n" "Turn this on only if you have separately verified that every prompt path " "that can reach `email_send` is one you trust." ), requires=ToggleRequirements( typed_phrase="I understand auto-send cannot be undone", reauth=True, cooldown_seconds=120, cap_input=CapSpec( name="Daily send cap", int_value=50, notes="Maximum emails the agent may auto-send per UTC day.", ), ), hard_invariants=( "authentication_required", "audit_logging", "secret_scrub_block", "prompt_injection_external_content_scan", ), ) ``` The cooldown for this toggle is 120 seconds rather than the 60-second default because email sending is irreversible. The cap is a daily integer the email-send path reads at action time. The four hard invariants run after toggle evaluation; we describe them next. The most aggressive cooldowns in the registry are 600 and 900 seconds, applied to `vault.auto_grant_capabilities`, `selfmod.enabled`, `selfmod.auto_apply`, and `selfmod.kill_switch`. These are toggles whose loosening is rare enough that even a ten-to-fifteen-minute cooldown should not impede a legitimate operator. ## 5. Hard invariants The hard-invariant layer is the framework's novel contribution. Eight named guards run after toggle evaluation in the call path of every gated action; no toggle setting bypasses them. ### 5.1 The eight invariants 1. **`secret_scrub_block`** — outbound payloads (email bodies, HTTP request bodies, tool arguments) are scanned for vault-managed credential patterns. The detector is conservative: it matches the named-token shapes the vault emits (`memagen_secret_...`), `sk-...` style API keys, `AKIA...` AWS access-key prefixes, and PEM-armored private-key headers. A match raises `InvariantViolation("secret_scrub_block", "outbound payload contains what looks like a credential")`. The point is not exhaustive DLP — that is an orthogonal product — but to block the obvious leak path. 2. **`authentication_required`** — every state-changing action requires an authenticated session. The invariant raises if `ctx.authenticated` is false. This is the floor under every toggle: regardless of toggle state, an unauthenticated request to flip or to act under a flipped toggle fails. 3. **`audit_logging`** — every transition must be audit-logged. The invariant function is a documented pass-through: actual logging happens in the toggle store, not in the invariant. The reason it exists *as an invariant* rather than just an implementation detail is so the contract is visible to readers of a toggle's spec: "this toggle's transitions are audited, no matter what". 4. **`final_human_review_on_grant_submission`** — application-form submission (e.g. Phase-6 grant submission) requires per-action human consent. The invariant fires when `ctx.action == "grant.submit"` and `ctx.explicit_consent` is false. Even with every other toggle off, the literal click of "submit this grant application" is reserved for the human. 5. **`force_push_to_main_blocked`** — `git push --force` to `main` or `master` requires per-action consent. The invariant inspects `ctx.action` for the substrings `"push"` and `"force"`, then checks `ctx.extras["git_target"]` against `("main", "master")`, and raises if `explicit_consent` is false. Force-pushing to a feature branch is unaffected. 6. **`csrf_and_session_protection`** — the web layer's CSRF token and session must both be valid. The invariant raises if either `ctx.csrf_ok` or `ctx.session_ok` is false. This is the runtime assertion of what the web framework already does at the request boundary; declaring it as a named invariant means a toggle that *gates a web action* visibly carries the contract. 7. **`prompt_injection_external_content_scan`** — when fetched external content contains imperative-mood directives targeting the agent, the action falls back to confirm-before-act regardless of toggle state. The detector is generous on purpose: false positives (extra confirmation) are vastly preferable to false negatives (silently following an injected directive). The detector matches `(?:ignore|disregard) (?:all|any)? (?:previous|prior)? instructions`, "system prompt", "you must (now|immediately|always)", and `(?:execute|run|delete|wipe|format) (?:the|all|every)`. A match in the absence of `explicit_consent` raises. 8. **`pii_export_block`** — bulk PII export requires per-export confirmation. The invariant fires when `ctx.action` starts with `"pii.export"` and `explicit_consent` is false. ### 5.2 Declaration Each toggle declares which named invariants apply via `hard_invariants: tuple[str, ...]`. The names are part of the public contract: they appear in the toggle's record, in the modal payload returned by `ToggleStore.request`, and in audit-log entries. A typo in the name is not silently ignored; the runner looks the invariant up by name and raises `KeyError` if the name is unknown (verified by `test_unknown_invariant_name_in_toggle_is_not_silently_ignored`). ### 5.3 Evaluation The single sanctioned way to evaluate invariants for a toggle is `core.safety.invariants.run_invariants_for_toggle(toggle, ctx)`. The function looks up each invariant by name in the module-level `_INVARIANTS` dict and calls it with the supplied `InvariantContext`. The first violation aborts; subsequent invariants do not run. The runner's signature deliberately does not accept any "skip these" parameter, and there is no field on `SafetyToggle` that says "but skip the invariants this time". The only way to skip an invariant is for the consumer not to call the runner at all, which is the threat model the unbypassability tests in `tests/safety/test_invariants_unbypassable.py` exist to refute (Section 9). ### 5.4 The InvariantContext The runner takes an `InvariantContext` dataclass with fields the invariants pull from: `authenticated`, `actor`, `outbound_payload`, `explicit_consent`, `external_content`, `action`, `csrf_ok`, `session_ok`, and a free-form `extras: dict[str, Any]`. Not every invariant uses every field. The `extras` map exists so future invariants (e.g. forthcoming Phase-6 grant-submission additions) can be added without revising the context shape. ## 6. Implementation ### 6.1 Registry `core/safety/registry.py` exports twenty `SafetyToggle` records, each constructed with `description`, `warning_text`, `requires`, and `hard_invariants` filled in. The `_req(...)` helper at the top of the file shortens the construction of `ToggleRequirements`; the default phrase used by the helper, when no phrase is overridden, is `"I understand the risk"`. Most registry entries override the default with a toggle-specific phrase. A single function `build_default_toggles()` returns the canonical list of all twenty toggles in stable order; `ToggleStore.__init__` consumes the list and constructs one `ToggleRecord` per toggle. The store rejects duplicate keys at construction time. ### 6.2 Transition gate `core/safety/toggles.py` contains the transition gate. The two-step protocol is: - `ToggleStore.request(key)` — pure read, returns the data the UI needs to render the loosening modal: the current state, the warning text, the typed-phrase requirement, the cap-input shape, the cooldown remaining, and the list of hard invariants. No state changes; no audit entry. - `ToggleStore.commit(request: TransitionRequest)` — atomically applies the transition or raises `TransitionDeniedError`. For a tightening request (`target_state=RESTRICTIVE`), `commit` calls `_tighten`, which sets the state, clears any cap that came with the previous loosening, increments the audit counter, and writes one audit entry with `result="ok"` and `reason="tighten"`. The other request fields are ignored. For a loosening request, `commit` calls `_loosen`, which executes five gates in order: 1. **Cooldown.** `_cooldown_remaining(rec)` computes the elapsed time since `last_loosened_at` and returns the remaining seconds; if positive, the request is denied with `reason="cooldown"` and an audit entry recorded. 2. **Paste rejection.** If the toggle has a `typed_phrase` and `request.paste_detected` is true, denied with `reason="paste_detected"`. 3. **Typed-phrase match.** `enforce_typed_phrase(expected, actual)` strips and compares case-sensitively; mismatch denies with `reason="phrase_mismatch"`. 4. **Re-auth presence.** If `requires.reauth` and `request.reauth_token` is missing, denied with `reason="reauth_missing"`. 5. **Cap shape.** If `requires.cap_input` is non-None and the supplied cap is missing or unset, denied with `reason="cap_required"`. Every denial path writes one audit entry with `result="denied"` and the failing-gate reason before raising. Successful loosening flips the state, sets `last_loosened_at`, persists the cap, and writes an audit entry with `result="ok"` and `reason="loosen"`. The store's lock is a single `threading.RLock`. The intentional coarseness — one lock for the whole registry — is fine for an audit-log volume measured in transitions-per-minute under stress. ### 6.3 Invariant runner `core/safety/invariants.py` contains the eight invariant functions, the module-level `_INVARIANTS: dict[str, Invariant]` registry, and `run_invariants_for_toggle`. Each invariant is a small named function `Invariant = Callable[[InvariantContext], None]` that returns normally on pass or raises `InvariantViolation(name, message)` on fail. A test-only `override_invariant_for_tests(name, fn)` is provided for the unbypassability suite to substitute fakes; the canonical `register_invariant(name, fn)` refuses to silently shadow an existing invariant. `all_invariant_names()` returns the registry keys, used by the test that asserts every named invariant has a violation case (Section 9). ### 6.4 Rehydration A process restart should not silently revert a toggle to `RESTRICTIVE` for a capability the operator deliberately granted earlier today. `ToggleStore` rehydrates each toggle's last *committed* state from the audit log on construction, looking back seven days. Denied transitions are ignored. If the most recent `ok` entry for a key is a loosening, the toggle starts permissive with the recorded cap; otherwise restrictive. If the audit log is missing or unreadable, every toggle starts restrictive — the rehydration code's failure mode is to be safe. ## 7. Audit log `core/safety/toggle_audit.py` implements the toggle-specific audit log. It is a separate module from the wider runtime audit log (`core/safety/audit.py`, sqlite-backed) because the toggle audit has different requirements: - **Daily-rotated JSONL files** at `~/.memagen/safety_audit/YYYY-MM-DD.jsonl`. The format is JSONL because audit data is append-only event records read by humans and forensic tools, not by SQL. - **Atomic appends.** Each entry is encoded once, the existing file is read, the new line is appended in a sibling tempfile, and `os.replace` swaps the tempfile over the original. `os.replace` is atomic on POSIX and Windows for a same-directory rename, which gives us crash-safety at line granularity. The cost is one extra read per append; the audit-volume profile makes that acceptable. - **Restrictive permissions.** The audit directory is `0o700`. Each file is `0o600`. The umask is set to `0o077` for the duration of the tempfile creation so the on-disk artifact is never accidentally world-readable. - **Stdlib-only.** The module has no third-party dependencies, so import is cheap and the audit log cannot fail to load due to a transitive package issue. Each entry has the shape: ```json { "ts": 1746744123.456, "actor": "user@example.com", "key": "email.auto_send", "from_state": "restrictive", "to_state": "permissive", "result": "ok", "reason": "loosen", "cap": {"name": "Daily send cap", "int_value": 50, "list_value": null} } ``` Denied entries omit the `cap` field and carry one of `cooldown`, `paste_detected`, `phrase_mismatch`, `reauth_missing`, or `cap_required` in the `reason` field. The framework records the typed phrase only as a hash, not in cleartext, in the audit row. The IP and user-agent of the request are populated by the API layer at the call site of `ToggleStore.commit` and persisted on the audit entry alongside the framework-supplied fields. The audit module exposes `read_day(day)`, `tail(key=None, limit=50)`, `count_for(key)`, and `tail_recent_days(days=7)` for forensic and rehydration consumers. ## 8. Self-modification interaction The Memagen™ Pro distribution offers sandboxed self-modification of the agent's own runtime code, gated by the `selfmod.*` toggles described in Section 4 and by additional Pro-only mechanics that are out of scope here. The interaction with the safety-toggle framework is structural rather than policy-based: the `core/safety/` directory is on the **forbidden-path list** of the self-modification subsystem. What this means in practice: even when `selfmod.enabled`, `selfmod.auto_apply`, and the kill switch are all in their permissive state, a self-modification proposal that touches any file under `core/safety/` is rejected at the proposal layer, before any review or apply step. The toggle framework cannot rewrite itself via the agent. The hard-invariant module cannot rewrite itself. The audit log cannot rewrite itself. This is structural because it does not depend on the user choosing the right toggle setting. It is enforced by the self-modification subsystem's path filter, which lives outside the toggle layer and is itself on the forbidden-path list (recursively). A successful prompt-injection attack that produces a plausible-looking patch to `core/safety/invariants.py` is rejected with no involvement from the toggle layer at all. ## 9. Evaluation The framework ships with seventy-nine unit tests in `tests/safety/`: forty-nine new tests added with the toggle-and-invariant work, and thirty pre-existing tests covering related safety subsystems (audit, kill switch, plugin signing, RBAC, secrets). The new tests are split across two files: - **`test_toggle_framework.py`** — covers default-state-is-restrictive for every registry toggle; tightening always succeeds without friction; loosening requires the typed phrase verbatim; loosening is rejected when paste was detected; re-auth requirement is enforced when set; cap shape is validated when required; cooldown blocks back-to-back loosening; the audit log appends one entry per attempt (success and denial); and tightening clears the cap. - **`test_invariants_unbypassable.py`** — covers, for each canonical invariant, one synthetic-toggle test that constructs a `SafetyToggle` declaring that invariant, builds a context that should make the invariant fire, runs `run_invariants_for_toggle`, and asserts `InvariantViolation` is raised. A complementary test `test_synthetic_toggle_cannot_disable_an_invariant_from_its_record` builds a toggle that *claims* the invariant and constructs a context that should make it fire, then verifies the invariant fires regardless. A registry-level test sweeps `all_invariant_names()` and asserts every named invariant has a violation case in the file's source — a typo or new addition cannot be silently introduced without the test file growing. The supporting test `test_registry_completeness.py` asserts that every entry in `build_default_toggles()` has a non-empty description, non-empty warning text, at least one declared invariant, and a default state of `RESTRICTIVE`. What is *not* in the evaluation: we have not benchmarked the per-call-site invariant assertion under load. The runner is `O(n)` in the number of declared invariants, each invariant is an O(1) function modulo regex scans whose input is already bounded by the action's payload, and the load profile (one runner call per gated action; gated actions are minutes-apart in normal use) does not motivate a benchmark. If a future profile demonstrates the runner is on a hot path, the runner is small enough to inline. ## 10. Discussion ### 10.1 The cost of generosity The framework prices loosening transitions to be deliberately annoying. Users will sometimes be annoyed. A user who *does* want to send 50 emails autonomously today must read a five-bullet warning, type "I understand auto-send cannot be undone", re-authenticate, accept a daily cap of 50, and wait out a 120-second cooldown. That is a meaningful chunk of the user's afternoon for a feature the user already decided to use. We argue this trade favors the user. The asymmetric pricing means a bad outcome — auto-send leaking secrets, a force-push to main eating someone's branch, an autonomous loop melting the cost cap — never occurs without the user having actively, deliberately, and recently confirmed the capability. A user who *was* in fact in a hurry and clicked through the warning has a written record of having done so. A user who was *not* in a hurry but wished they had not enabled the toggle can revert in one click. The alternative — a framework that makes loosening painless because the user "knows what they're doing" — silently transfers the cost of a misconfigured capability from the platform's UX back to the user's data. We have made a different call. ### 10.2 The structural argument for hard invariants A toggle alone is not enough because a misconfigured toggle is a single point of failure. A user who, in a hurry or under social pressure, flips `email.recipient_allowlist` to permissive has not only opened the recipient list — they have removed the platform's last guard against an injected prompt instructing the agent to email everyone. That is too much consequence for one click. The hard-invariant layer means the consequence is bounded. Even with `email.recipient_allowlist` permissive, `secret_scrub_block` still runs and refuses to send a payload containing a vault-managed credential. Even with `code.auto_push` permissive, `force_push_to_main_blocked` still runs and refuses to force-push to `main` without per-action consent. Even with every toggle in the registry permissive, an unauthenticated request for any state-changing action is refused by `authentication_required`. The invariants are the floor. The toggles are the ceiling. The user controls the ceiling; the platform controls the floor; and the floor is not a configuration surface. ### 10.3 What this is not The framework is not an alignment mechanism. It does not attempt to constrain what the model can *say*. It constrains what the runtime, on receiving the model's output, will *do*. A model that decides to suggest a destructive shell command is not blocked from suggesting it; the action is blocked from running unless `shell.allow_destructive_flags` is permissive, and even then `authentication_required` and `audit_logging` still apply. The framework is not a substitute for OS-level confinement. The platform still benefits from running under a constrained user account with workspace-scoped filesystem access and an outbound network allowlist. The toggles are an additional layer the operator controls; they are not the only layer. The framework is not a substitute for prompt-time defenses. Models with strong refusal behavior remain useful. Lenient parsing (Section 4 of the master paper) and the V3-P capability vault (Section 5 of the master paper) are complementary surfaces. Defense in depth still applies. ## 11. Future work **Toggle inheritance.** A multi-user organization should be able to express "the org's restrictive settings are the floor; teams can tighten further but cannot loosen past the org default; users can tighten further but cannot loosen past the team default". The framework's data model already supports this: each `SafetyToggle` is a static record, and a future store implementation can carry an inheritance chain on the `ToggleRecord`. The work is in the surface and the audit-log shape, not the core. **Time-of-day sensitivity.** A toggle that is permissive during business hours and auto-tightens overnight, weekends, or during the operator's declared focus blocks would meaningfully reduce the surface a sleeping operator presents. The framework's clock injection (`clock: Callable[[], float]` in `ToggleStore.__init__`) already makes this testable; the work is in expressing the schedule and in the UX for "your toggle just auto-tightened because it's 9pm". **Risk-budgeted toggles.** A toggle that auto-tightens after N permissive uses in a window — a usage budget rather than a state — would let an operator say "I want to send up to 50 autonomous emails today, and after that the toggle should re-confirm". The cap-input shape is suggestive of where this lives, but the current framework does not auto-tighten on cap exhaustion; the consumer is expected to refuse the action. Pulling that logic into the toggle layer is mostly a UX question. **Per-tool granularity inside a category toggle.** `tools.bypass_catalog_gate` is a sledgehammer; an operator who only wants to widen access to one specific tool currently has to flip the gate for every tool. A per-tool override list, carried as a `CapSpec.list_value`, is a small extension that would materially reduce the worst-case loosening footprint. ## 12. References 1. **Anthropic.** (2024). *Model Context Protocol Specification*. https://modelcontextprotocol.io 2. **Corbet, J.** (2009). *Seccomp and sandboxing*. LWN.net. https://lwn.net/Articles/332974/ 3. **Felt, A.P., Egelman, S., Finifter, M., Akhawe, D., & Wagner, D.** (2012). *How to ask for permission*. HotSec '12. (Browser permission UX evolution.) 4. **Google.** (2018). *gVisor: container runtime sandbox*. https://gvisor.dev 5. **Hardy, N.** (1985). *KeyKOS architecture*. ACM Operating Systems Review, 19(4). 6. **Levitas, T.** (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Memagen™ master paper, Sections 5–6. 7. **Saltzer, J.H., & Schroeder, M.D.** (1975). *The protection of information in computer systems*. Proceedings of the IEEE, 63(9). (Origin of the principle of fail-safe defaults.) 8. **Shapiro, J.S., & Smith, J.M.** (1999). *Capability myths demolished*. Technical report, University of Pennsylvania. 9. **Yee, K.-P.** (2002). *User interaction design for secure systems*. ICICS '02. (Asymmetric grant/revoke UX argument.) 10. **Zalewski, M.** (2011). *The Tangled Web: A Guide to Securing Modern Web Applications*. No Starch Press. (CSRF, session, and origin-confusion failure modes referenced in Section 5.) --- ## Paper: 2026-05-08-uimcp URL: https://memagen.com/papers/2026-05-08-uimcp PDF: https://memagen.com/papers/2026-05-08-uimcp.pdf --- title: "UIMCP: Adaptive Catalog Compression for Multi-Tool Agent Dispatch" subtitle: "How Memagen™ encodes a 90-command catalog (sourced across three framework adapters) into 42.7x fewer tokens per turn without measurable loss in tool-selection quality" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "Production agent platforms inject every available tool's full schema into the LLM context on every turn, paying a token tax that scales linearly with catalog size. With a typical native tool-use schema running 80–200 tokens per definition, a 90-command catalog costs 7,200–18,000 tokens of context per turn before the user message is even considered, and the cost compounds with every assistant turn that re-emits the catalog. UIMCP is Memagen™'s adaptive catalog compression layer: a per-turn ranking over recency, conversation-context relevance, and tier classification, paired with a summoning protocol that promotes a tool from a low-priority pool to fully-injected status only when the agent actually needs it. The 90 commands in our reference catalog are sourced across three framework adapters (web_react, cli_argparse, android_xml) which each compress a UI action into a short opcode that the MCP transport carries to the server-side decoder. We describe the implementation in the open-core Lite distribution — `core/mcp/optimizer.py`, `core/tools/mcp_optimize_tool.py`, `core/tools/uimcp_executor.py`, `core/tools/uimcp_planner.py`, `core/tools/uimcp_macro.py`, and the adapter dispatch under `core/mcp/uimcp_adapters/` — and report a single internal observation of 42.7x token compression on the 90-command catalog over a synthetic 30-turn coding conversation. We are explicit about what this number is not: it is one observation against one catalog with one tokenizer, not a benchmark, and we have not replicated it across diverse catalogs or against alternative compression strategies. We close with the latency trade-off the summon protocol adds on novel-tool turns and a list of comparisons we owe." keywords: ["MCP", "tool catalog", "token compression", "agent dispatch", "context engineering"] kind: specialty subsystem: "UIMCP" pages: 7 version: "v2" --- ## 1. Introduction The Model Context Protocol gives agents a clean, server-mediated way to acquire tool surfaces. As MCP adoption has grown, users routinely connect ten, twenty, or fifty servers to a single session — a code-host server, an issue-tracker server, a chat server, a filesystem server, a shell server, a browser server, plus N domain-specific tools generated via [AutoMCP](/papers/2026-05-08-automcp) or [AutoWebMCP](/papers/2026-05-08-autowebmcp). The naive approach — inject every connected tool's schema into the LLM context on every turn — pays a token tax that scales linearly with catalog size, and which compounds across turns because tool schemas typically reside in the system prompt or a stable header that the assistant re-receives every turn. The cost is not theoretical. A typical native tool-use schema for a non-trivial tool runs 80–200 tokens once name, description, and JSON schema are serialized. A 90-command catalog therefore costs roughly 7,200–18,000 tokens of context per turn — before the user's message, the conversation history, the agent's scratchpad, or any retrieved memory enters the window. On a 30-turn coding session that pattern compounds to hundreds of thousands of tokens of catalog-only billing across the conversation lifecycle. For BYOLLM (bring-your-own-LLM) deployments where the user pays the model provider directly, this is not Memagen™'s cost to absorb, but it is Memagen™'s cost to make small. UIMCP — described in §9.3 of the Memagen™ master paper as the adaptive catalog compression layer — addresses this. The compression operates per-turn over three signals: recency of last tool use, embedding-similarity relevance to the current conversation context, and a discrete tier classification (`always` / `summoned` / `archived`). Tools in the `always` tier and a small meta-tool surface are injected on every turn; tools in `summoned` enter only after the agent calls a `tools.find` meta-tool that returns their schemas; tools in `archived` are not injected and not directly callable until promoted. On a single internal observation against a 90-command catalog over a 30-turn synthetic conversation, the result was 42.7x token compression. We describe the algorithm, the implementation, and — most importantly — the limits of what that single number actually establishes. Section 2 surveys related work. Section 3 describes the per-turn ranking algorithm and its three signals. Section 4 describes the summoning protocol. Section 5 reports the 42.7x observation in detail. Section 6 walks the implementation: the planner, the executor, the macro layer, the framework adapters, and the optimizer pipeline. Section 7 declares evaluation limitations. Section 8 discusses the latency trade-off. Section 9 names what UIMCP is not. Section 10 lists future work. Section 11 references. ## 2. Architecture overview The flow from a user-visible UI action to a tool execution is short. A click, fill, or navigation against a scanned surface is intercepted by one of three framework adapters, which encodes the action as a short opcode. The opcode crosses the MCP transport as a JSON-RPC frame; the server-side decoder resolves the opcode to a registered element record and dispatches the underlying tool, capability-gated. Per-turn injection cost is a property of the catalog the adapter exposes, not of how the action was emitted. Compression and dispatch are orthogonal concerns that share one registry. ```mermaid flowchart LR UA["UI action<br/>(click / fill / nav)"] --> WR["web_react<br/>adapter"] UA --> CA["cli_argparse<br/>adapter"] UA --> AX["android_xml<br/>adapter"] WR --> OP["opcode<br/>app#elt(args)<br/>10–30 tokens"] CA --> OP AX --> OP OP --> MCP["MCP transport<br/>(JSON-RPC)"] MCP --> DEC["server decode<br/>resolve label"] DEC --> EX["tool execution<br/>(capability-gated)"] subgraph N["Per-turn catalog cost (90-command catalog)"] direction TB NAIVE["Naive injection  14,237 tokens"] UI["UIMCP encoded       333 tokens"] NAIVE -. "42.7x compression" .-> UI end ``` Figure 1 in the rendered SVG (`/papers/diagrams/uimcp.svg`) shows the same flow with a side-by-side bar comparison of the naive 14,237-token injection against the 333-token UIMCP-encoded payload. The 42.7x ratio is the observation reported in §5; the structural reason it is achievable is that the opcode form names a callable rather than describing one — the description lives once on the server, not once per turn on the wire. ## 3. The compression algorithm UIMCP's per-turn ranking combines three signals into a single priority score per tool. We describe each and then the composition. ### 3.1 Recency The recency signal is an exponential decay over the session's tool-use history. Let `t_n` be the index of the current turn and `t_use(s)` the most recent turn at which tool `s` was successfully called. Then ```text recency(s) = exp(-(t_n - t_use(s)) / τ) ``` with `τ` a half-life parameter (default 8 turns in the open-core implementation). A tool used on the previous turn scores ~0.88; a tool used 16 turns ago scores ~0.14; a tool never used scores 0. The exponential shape captures the locality assumption that an agent that just used a code-host PR-creation tool is far more likely to use a sibling reviewer-add tool next than to switch to a chat tool from a different server. ### 3.2 Context relevance The context-relevance signal is the cosine similarity between an embedding of the tool's description and an embedding of the current conversation context (typically the last 3–5 turns of user+assistant text, capped at ~2,000 input tokens to the embedder). The open-core implementation uses whatever embedding the local model router has available; the master paper's multi-provider router abstracts this so the same UIMCP run works across embedding sources. ### 3.3 Tier classification A small operator-curated table assigns each tool to one of three tiers: - **`always`** — injected on every turn. The default `always` set is small: `shell.execute`, `fs.read`, and the meta-tool `tools.find` (described in §4). Operators may add tools; the framework does not fight them. - **`summoned`** — injected only after the agent calls `tools.find` and the tool appears in the result. Most tools default to this tier. - **`archived`** — never injected and not directly callable until an operator promotes it. Useful for tools the operator wants installed but disabled for the current session (e.g., a destructive `prod.deploy`). ### 3.4 Composition The per-turn priority for a tool `s` is ```text priority(s) = w_t * tier(s) + w_r * recency(s) + w_c * context(s) ``` with `tier(s)` mapping `always → 1.0`, `summoned → 0.5`, `archived → 0.0`, and the weights `(w_t, w_r, w_c)` defaulting to `(2.0, 1.0, 1.0)` so that tier dominates in ties. Tools with `tier(s) == archived` are filtered before scoring and never appear in the injected set even if recency and context-relevance are high. ### 3.5 Planner pseudocode ```python def select_tools_for_turn(catalog, history, conversation_context, tiers): scored = [] for tool in catalog: if tiers[tool.name] == "archived": continue r = recency(tool, history) c = context_relevance(tool, conversation_context) t = {"always": 1.0, "summoned": 0.5}[tiers[tool.name]] p = 2.0 * t + 1.0 * r + 1.0 * c scored.append((tool, p)) scored.sort(key=lambda x: -x[1]) always = [t for t, _ in scored if tiers[t.name] == "always"] summoned_top_k = [t for t, p in scored if tiers[t.name] == "summoned" and p >= SUMMON_THRESHOLD][:K] return always + [TOOLS_FIND_META] + summoned_top_k ``` In practice the open-core path is more conservative: by default only the `always` set and the `tools.find` meta-tool are injected, and `summoned` tools enter exclusively via §4's protocol. Threshold-based pre-injection of high-scoring summoned tools is a configuration knob, not the default, because the default minimizes context cost at the price of one round-trip per novel tool. Both paths reflect the trade-off in §8. The planner and the per-turn priority computation are the conceptual layer this paper documents. The closest implementation surface is `core/tools/uimcp_planner.py`, which shares the per-turn-recompute structure (filtering a catalog by current screen, retrieving the most-similar prior macro via `_retrieve_prior_macro`, and assembling a grounded prompt). The optimizer pipeline in `core/mcp/optimizer.py` provides upstream catalog distillation (worthiness filter, distiller, hierarchy builder, result cache) on top of which the per-turn ranker operates. ## 4. The summoning protocol When the agent needs a tool in the `summoned` tier and not currently injected, the agent does not see that tool's schema and cannot call it directly. Instead it calls a meta-tool `tools.find(query)` whose schema *is* always injected. The meta-tool is implemented in `core/tools/mcp_optimize_tool.py` as the agent-facing wrapper, with the underlying ranking living in `core/mcp/optimizer.py`. The protocol on a turn where the agent needs a summoned tool: 1. **Turn N (agent → meta-tool)**: the agent emits `tools.find(query="how do I open a pull request?")`. 2. **Turn N (meta-tool → agent)**: the meta-tool returns a candidate list of full schemas — the top-K matches against the query, ranked by the §3.2 context-relevance score with the query string substituted for the conversation context. K defaults to 5. 3. **Turn N+1 (agent → tool)**: the agent now has the schemas of the top-K candidates injected for the next turn and can call any of them by name. The runtime adds the called tool's schema to the always-injected set for the rest of the session, so subsequent turns do not re-summon — the tool is now hot. A summoned-then-called tool is promoted from summoned-tier to always-tier *for the duration of the session only*. Cross-session promotion would require the catalog to learn from session histories, which the open-core distribution does not do today (see §10). The protocol's distinguishing property is that the agent does not need to know which tools are in which tier. The meta-tool is always available; calling it is the agent's universal escape hatch. Tool-selection quality is preserved when the agent learns to call `tools.find` whenever the visible surface is insufficient — a behavior that frontier-class models pick up rapidly and that smaller models pick up less reliably (a limitation in §7). ## 5. The 42.7x compression observation We state up front: this is one observation, not a benchmark. We replicate it here because it is the headline number in the Memagen™ master paper (§9.3) and the readers of that paper deserve to see how it was obtained. We do not claim it generalizes. ### 5.1 Catalog The 90-command catalog used in the observation is Memagen™'s internal default tool surface as of v1-lite, expanded to the upper end by enabling several optional MCP servers a typical power user might install. The 90 commands are sourced across three framework adapters (`web_react`, `cli_argparse`, `android_xml`); each adapter contributes its slice of the catalog through the same `register_app` path. Composition by domain: - 14 code-host tools (issues, PRs, reviewers, branches, releases, gists, workflow runs, …) - 9 chat tools (post message, list channels, list DMs, search, file upload, …) - 11 issue-tracker tools (issue create/update, project list, cycle status, comments, …) - 6 filesystem tools (`fs.read`, `fs.write`, `fs.delete`, `fs.move`, `fs.mkdir`, `fs.list`) - 5 shell tools (`shell.execute`, `shell.background`, `shell.kill`, `shell.ls_jobs`, `shell.ps`) - 8 browser tools (open, click, fill, screenshot, eval, wait, navigate, listen) - 12 Memagen™ cognition tools (memory primitives, retrieval, dream-consolidation triggers — the Pro tier surface, counted here because the run was on a Pro machine) - 11 Memagen™ harness tools (`widget.create`, `workspace.apply`, `email.draft`, `email.send`, `code.search`, `code.read`, `code.edit`, `net.fetch`, `recipe.run`, `recipe.resume`, `capability.grant`) - 14 [AutoWebMCP](/papers/2026-05-08-autowebmcp)-generated tools from a small set of test sites (a contact form, two SaaS dashboards, an internal admin panel — fairly typical for an exploratory session) This composition reflects what we observe in real internal use; we have not artificially padded it with redundant tools. If anything the 90 number is on the low end of what we expect production deployments to enable once Memagen™ Lite ships. ### 5.2 Conversation The conversation was a 30-turn synthetic coding task: building a small TypeScript service from scratch, running tests, debugging two test failures, opening a PR, posting a chat notification, and triaging an issue surfaced during testing. The script was hand-written by the author to mirror the shape of a real Memagen™ session as closely as possible without using actual user data. We share the conversation only in the form of token counts; the script itself contains internal API references and is not in the open-core distribution. The conversation exercises tools across multiple domains. Most of the catalog's 90 tools are *not* invoked in the conversation — the regime where adaptive catalog compression has the most to offer. ### 5.3 Token measurement Token counts were measured against the production tokenizer wired into Memagen™'s multi-provider router as of 2026-05-08. For each turn we measured: - **Naive injection cost**: serialize all 90 tool schemas in canonical native tool-use form and tokenize. - **UIMCP cost**: serialize only the tools UIMCP injects on that turn (the always-tier set, the `tools.find` meta-tool, and any summoned tools promoted earlier in the session) and tokenize. Per-turn averages over the 30-turn run: - Naive: **14,237 tokens** of catalog per turn (mean), σ ≈ 0 (catalog does not reshape). - UIMCP: **333 tokens** of catalog per turn (mean), σ ≈ 142 (the catalog grows as summoned tools promote; first turn was 211 tokens, last turn was 619 tokens). Ratio: **14,237 / 333 = 42.7x.** ### 5.4 Tool-selection quality We measured tool-selection quality as a binary outcome: did the agent reach the goal (a working PR with passing tests, a chat notification posted, an issue created and assigned) in the same number of turns as the naive-injection baseline? On the single conversation tested, yes — the UIMCP run completed in 31 turns vs. the naive-injection 30 turns, the one extra turn being a single `tools.find` summon for an issue-tracker creation tool not in the always set. The agent's reasoning trace did not show signs of confusion or wrong-tool selection; the summon happened cleanly on first attempt. This is a single observation. The 31-vs-30 turn difference is comfortably within run-to-run variance for this conversation shape, and we cannot distinguish "no measurable loss in quality" from "we got lucky once" on this evidence alone. ### 5.5 What this is not The 42.7x number is: - **Not a benchmark.** A benchmark would require multiple catalogs, multiple conversations from a representative distribution, and ideally multiple tokenizers. We have one of each. - **Not a claim.** We report the observation; we do not assert it generalizes. - **Not third-party-verified.** No external party has reproduced the number against the open-core implementation. We invite the reproduction and would update this paper with replications, including replications that find smaller compression ratios on different catalogs. - **Not a guarantee of selection quality.** §7 expands on this. We present the number because the alternative — saying "UIMCP achieves significant token compression" without showing the underlying observation — would be more dishonest, not less. The reader can decide what to make of one data point. ## 6. Implementation notes The open-core implementation under MIT spans six files. We describe each by responsibility and cite the function names that carry the load. ### 6.1 The optimizer pipeline (`core/mcp/optimizer.py`) A static-analysis distillation pipeline that runs once per catalog change (a new MCP server connected, a tool's schema updated, a tool removed). The pipeline is a sequence of named transformations: - `ToolWorthinessFilter.score(tool, existing_tools)` — produces a `surprise × importance × (1 − redundancy)` score per tool. Tools below a default threshold of 0.3 are dropped. Surprise is `1 − parameter_count / 20`. Importance is `description_length / 500` capped at 1.0. Redundancy is the maximum Jaccard similarity of name-tokens and description-tokens against any existing accepted tool. - `ToolDistiller.distill(tool)` — strips `"This tool…"` preambles, normalizes parameter names to snake_case, removes `title` and `$schema` keys from input schemas, and caps the example list at 3. - `HierarchyBuilder.build_hierarchy(tools)` — classifies each tool into level 0 (raw CRUD: get/set/create/delete/update), level 1 (semantic search: search/find/locate/discover/list), or level 2 (meta: orchestrate/manage/coordinate/schedule). The hierarchy is consumed by the per-turn ranker as a tier prior — level-2 tools weight as `always` candidates, level-1 as `summoned`, level-0 as `summoned` with tighter recency weighting. - `ToolResultCache._compute_key(tool_name, params)` — content-addressable SHA-256 over the (tool, params) pair, with a 24-hour TTL and a 1,000-entry capacity. Cache hits short-circuit dispatch entirely; this is orthogonal to compression but cooperates with it because cached calls do not advance the recency clock. `MCPOptimizer.optimize(mcp_config)` runs the four phases in order and returns an `OptimizedMCP` dataclass with the distilled tool list, the hierarchy mapping, and the percentage token reduction. ### 6.2 The agent-facing optimize tool (`core/tools/mcp_optimize_tool.py`) The agent invocation surface for the optimizer is exposed as the `mcp_optimize` tool (schema in `SCHEMA`, dispatched by `mcp_optimize(name, tools)`). Calling it persists the result as an L4 graph node (`kind = "mcp_optimization"`), so future sessions can recall optimization runs via `graph_query("mcp optimization for code-host tools")`. This is the Memagen™ pattern where every introspectable action becomes a graph fact (master paper §10.1). ### 6.3 The UIMCP executor and registry (`core/tools/uimcp_executor.py`) The runtime that *registers* compressed catalogs and *dispatches* against them. The registry is in-memory (`_REGISTRY`) and persisted as L4 graph nodes (`kind = "uimcp_app"`) via `_persist_uimcp_node`, so the catalog survives restarts. `restore_uimcp_from_graph()` rehydrates the in-memory registry on agent startup by reading every `uimcp_app` node from the graph database. Dispatch entry point is `invoke(app, action, params)`, which resolves an action label to an element record (case-insensitive exact match, falling back to substring match) and dispatches to the per-framework backend. The executor's `app` notion overloads with the §3 catalog notion: an "app" is a scanned UI surface (a web app, a CLI, an Android app), and the catalog is the union of `app.elements` across all registered apps. The MCP-server-as-tool-source case is a special case where the elements are MCP tools rather than UI elements; the registration path is the same. ### 6.4 The macro layer (`core/tools/uimcp_macro.py`) The macro layer records and replays multi-step tool-use sequences. It exposes a small DSL — `click "X"`, `fill "field" "value"`, `nav "/route"`, plus raw mouse/keyboard ops, plus meta verbs — and a runner (`run_macro`) that executes parsed steps via the dispatch graph. The relevance to catalog compression is that a frequently-observed sequence of tool calls can be promoted to a single macro tool, replacing several distinct schemas in the injected set with one. The pattern matches [AutoMCP](/papers/2026-05-08-automcp)'s macro inference (master paper §9.1) and provides the underlying substrate. Persisted macros are stored as `kind = "macro_run"` graph nodes; the planner described in §6.5 retrieves the most-similar prior macro for replay-reuse via `_retrieve_prior_macro(goal, app, top_k)`. ### 6.5 The planner (`core/tools/uimcp_planner.py`) The planner drafts a macro DSL plan grounded in a real catalog. Structure mirrors the per-turn ranker: 1. `_filter_by_screen(elements, current_screen)` — context-relevance prefiltering. Filters the catalog to elements relevant to the current screen/route; if the filter empties the catalog, falls back to the full catalog so the planner can still produce something. 2. `_retrieve_prior_macro(goal, app, top_k=1)` — replay reuse. Retrieves the most-similar prior `macro_run` graph node for the current goal. If similarity exceeds 0.85, the planner is instructed to *adapt* the prior macro rather than start fresh. 3. `_build_planner_prompt(goal, app, elements, current_screen, prior)` — assembles a tight, grounded prompt: goal + filtered element list + (optional) prior-art DSL. 4. The LLM call goes through the multi-provider router (master paper §7). 5. `_validate_plan(dsl, spec)` — pre-flight validation that every `click "X"`, `fill "X" "..."`, and `nav "X"` resolves to a real element/route. Bad steps surface as warnings and the plan is marked unapproved. The planner persists each plan as a `kind = "ui_plan"` graph node so future planners can recall plans across sessions. ### 6.6 The framework adapters (`core/mcp/uimcp_adapters/`) The adapter pattern lets the same UIMCP runtime register catalogs from heterogeneous sources. Three adapters (plus a JavaScript parser) ship in the open-core distribution: - `web_react.py` — scans React/JSX source for buttons, inputs, links, routes. Cooperates with the JavaScript parser `jsx_parser.js` for full JSX/TSX support. - `cli_argparse.py` — scans Python source for `argparse.add_argument` calls and `@click.command` decorators. Source-static path: parse the AST, walk for matching call sites, extract flag names and command names. - `android_xml.py` — scans Android resource XML for interactable views and resource IDs. Each adapter exposes `detect(root)` (returns 0..1 confidence that this adapter applies) and `scan(root, app_name)` (walks the source and returns the catalog). The dispatcher `detect_framework(root)` picks the adapter with the highest detect score and registers via `register_app`. From the compression perspective, each adapter is a *source* of opcode-callable tool schemas; the compression pipeline operates uniformly across them. ## 7. Evaluation methodology limitations We list each limitation by name, in order of how much we think it weakens the 42.7x claim. - **One catalog.** The 90-command catalog reflects Memagen™'s defaults plus a small number of optional MCP servers. A different catalog — heavier on cognition tools, lighter on code-host/chat/issue-tracker, or skewed toward domain-specific AutoWebMCP-generated tools — would produce a different ratio. We do not know how different. - **One conversation.** The 30-turn coding task covers a particular slice of agent behavior. Sessions exercising more catalog breadth (debugging across many subsystems, multi-domain triage) would summon more tools and close the compression ratio. Sessions exercising less breadth (a long monologue with the same five tools) would widen it. - **One tokenizer.** Token-per-character density varies across tokenizers a Memagen™ user might route to. We have not measured the ratio under any other tokenizer. The compression *principle* holds across tokenizers; the *magnitude* is tokenizer-dependent. - **Tool-selection quality measured as a binary "agent reached goal in N turns".** The weakest signal we could offer that is still a signal. A real evaluation would involve multiple independent runs, hand-graded reasoning quality, and ablation studies isolating the recency, context-relevance, and tier signals. - **No benchmark against alternative compression strategies.** We have not measured UIMCP against description-pruning compression, against pure top-K retrieval-on-summon (no recency), or against a tier-only baseline. An ablation would be the right way to establish which of the three signals does most of the work. - **Smaller-model behavior with `tools.find` is untested.** Frontier-class models pick up the call-the-meta-tool-when-stuck behavior rapidly. Smaller models with weaker tool-use generalization may not, in which case UIMCP's compression converts directly into selection-quality loss because the agent never summons the tool it needs. The lenient parser (master paper §4) helps catch the model emitting a tool call for a tool it cannot see, but the recovery path — inject a one-shot directive to call `tools.find` — has not been measured against a smaller-model baseline. We list these because the alternative is performative confidence in a single observation, which we do not believe. ## 8. Discussion UIMCP trades token cost against round-trip latency. The naive baseline pays a fixed token tax per turn but never adds round-trips for tool discovery. UIMCP pays a small tax per turn (the always-tier set plus `tools.find`) and adds *at most one* round-trip per *novel* tool the agent needs in a session. For sessions with stable tool usage — the agent quickly converges on a working subset and re-uses it — UIMCP is approximately free in latency: the first few turns may add summons, but once the working set is hot, the agent runs at the always-tier-plus-promoted-set cost with no further round-trips. The compression ratio in this regime is large. For exploratory sessions — the agent moves rapidly across tools it has not used recently, summoning a new tool every few turns — UIMCP adds noticeable latency. Each summon costs one round-trip. On a model with 1.5s p50 turn latency, ten summons across a session add ~15s. The token compression remains real but the user-perceived latency degrades. The right deployment posture depends on the workload. Memagen™ Lite ships with UIMCP enabled by default because the median session in our internal use is more stable than exploratory; operators with exploratory workloads should consider tuning the default `summoned` tier toward `always` for tools they expect to need, or pre-warming summons at session start (§10). We note one second-order property: UIMCP improves with session length. The longer the session, the more summons have happened, the more tools promoted, the closer the catalog is to the always-tier baseline. By turn 50 of a long session, a session that started with 10 always-tier tools may have 25 hot tools — still a 4x compression against naive injection on a 100-tool catalog, but no longer 42.7x. This is the right shape: compression is largest when the cost of full injection is largest (early in the session, before the agent knows what it needs) and smallest when the agent has fully revealed its working set. ## 9. Comparisons we owe UIMCP is **not a tool selector.** The model still selects which tool to call. UIMCP only changes which tools are *visible* on a given turn. A model with poor selection quality will not have its selection improved by UIMCP; if anything, a poor-selection model loses ground because the visible set is narrower. UIMCP is **not a tool synthesizer.** It does not generate novel tools. [AutoMCP](/papers/2026-05-08-automcp) and [AutoWebMCP](/papers/2026-05-08-autowebmcp) are the synthesis layers; UIMCP is the compression-and-dispatch layer above them. UIMCP can compress a catalog that contains synthesizer-generated tools; it does not add tools to that catalog. UIMCP is **not a permission system.** The capability vault (master paper §5) is the permission system. Whether a tool is in `always` / `summoned` / `archived` is independent of whether the tool requires a per-use capability grant. A tool can be in `always` and still require a grant on every call; conversely, a tool can be in `archived` with no capability requirement, in which case the only thing keeping it from being called is that the model cannot see it. These three separations are deliberate: each layer composes orthogonally, and we believe an agent runtime that fuses them ("promote the tool if and only if the user has granted a capability") is harder to reason about than three layers with clean handoffs. ## 10. Future work Five extensions, in roughly the order we expect to implement them. **Cross-session catalog warming.** The current implementation discards summon history at session end. Cross-session warming would precompute the `always` set at session start based on signals available then: the user's recent task history, the current project's repository state, the time of day. Memagen™ Pro's compound-graph memory substrate (master paper §10.1) is the natural place to land this — the graph already records tool-use history. **Provider-specific compression.** Different model families have different tokenizers, different tool-use schema formats, and different tendencies to emit `tools.find` calls when stuck. A per-provider compression policy — different always-tier sizes, different summon thresholds, different meta-tool prompts — would let a deployment route harder workloads to frontier-class models with naive injection and easier workloads to cheaper models with aggressive UIMCP compression. **Catalog distillation: cluster-and-disambiguate.** Several MCP servers expose near-duplicate tools. The optimizer's `ToolWorthinessFilter` already drops tools below a redundancy threshold, but a cluster-and-disambiguate approach would expose one synthetic tool that, on call, prompts the agent to choose between underlying real tools. The compression gain on near-duplicate-heavy catalogs would be material. **Lossy schema compression for summoned tier.** Applying lossy compression to the descriptions of summoned-tier schemas at summon time — preserving parameter schemas exactly, compressing the description text — would shrink the summon response further. We have not measured how much this would help. **Empirical generalization study.** The most important future work is replicating the 42.7x observation across diverse catalogs, conversations, and tokenizers, then publishing the distribution rather than the single number. We expect the median compression ratio to be substantially smaller than 42.7x and the variance to be large; both are useful information for users deciding whether to enable UIMCP. ## 11. References 1. **Levitas, T.** (2026). *Memagen™: An open-core agent platform with lenient tool routing, capability-gated execution, and a compound-graph memory substrate*. Memagen™ master paper, 2026-05-08. [memagen.com/papers/memagen](https://memagen.com/papers/memagen) 2. **Levitas, T.** (2026). *AutoMCP: Inferring parametric macros from typed action graphs*. Memagen™ specialty paper, 2026-05-08. [memagen.com/papers/2026-05-08-automcp](https://memagen.com/papers/2026-05-08-automcp) 3. **Levitas, T.** (2026). *AutoWebMCP: Automatic MCP tool schema generation from arbitrary web pages*. Memagen™ specialty paper, 2026-05-08. [memagen.com/papers/2026-05-08-autowebmcp](https://memagen.com/papers/2026-05-08-autowebmcp) 4. **Patil, S.G., Zhang, T., Wang, X., & Gonzalez, J.E.** (2023). *Gorilla: Large language model connected with massive APIs*. arXiv:2305.15334. 5. **Qin, Y., Liang, S., Ye, Y., Zhu, K., Yan, L., Lu, Y., et al.** (2023). *ToolLLM: Facilitating large language models to master 16000+ real-world APIs*. arXiv:2307.16789. 6. **Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T.** (2023). *Toolformer: Language models can teach themselves to use tools*. NeurIPS 2023. 7. **Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A.** (2023). *Voyager: An open-ended embodied agent with large language models*. arXiv:2305.16291. Source: [github.com/tablevitas123/crackedclaw](https://github.com/tablevitas123/crackedclaw) --- ## Paper: 2026-05-08-substrate-as-live-visualization URL: https://memagen.com/papers/2026-05-08-substrate-as-live-visualization PDF: https://memagen.com/papers/2026-05-08-substrate-as-live-visualization.pdf --- title: "The Compound-Graph Substrate as Live Visualization: Recording, Anonymizing, and Replaying a Real Agent's Memory in Three Dimensions" subtitle: "How the Memagen™ v4 hero captures real graph mutations from a running agent and projects them, unfaked, into a 3D scene that doubles as a debugging instrument" authors: ["Tab Levitas"] affiliation: "Memagen™" date: "2026-05-08" abstract: "The Memagen™ marketing site visualises the platform's compound-graph memory in three dimensions on the landing page. The visualization is not a hand-authored animation. It is the deterministic playback of an append-only event stream emitted by the live graph as the agent runs. Three subsystems form one observable system: a hot-path-cheap NDJSON delta recorder hooked into every graph write and every agent step; a deterministic anonymizer that rewrites real internal taxonomy into a published-safe generic schema and projects high-dimensional embeddings into a sign-canonicalised PCA-3 basis; and a Three.js scene that consumes the resulting timeline file event by event, mutating an InstancedMesh node field, an edge map, and a particle vector flow on a deterministic spherical-Fibonacci layout. This paper describes the design of each layer, the contract between them, and the engineering payoff of using one event stream for visualization, time-travel debugging, audit trail, Connected SKU sync, and self-modification correctness gating. The thesis is short: showing the graph rather than describing it is engineering, not theatre, because the same NDJSON that drives the hero's cinematics is also the instrument the team uses to debug and audit production agent runs." keywords: ["compound graph", "live visualization", "event sourcing", "NDJSON", "PCA projection", "anonymization", "three.js", "agent observability", "audit trail", "time-travel debugging"] kind: specialty subsystem: "Substrate Visualization" pages: 12 version: "v1" --- ## 1. Introduction Most marketing sites for autonomous-agent platforms show a hand-animated graph or a scripted screen recording. The user is asked to take it on faith that the depicted state corresponds to anything the system actually computes. The Memagen™ v4 hero takes a different approach. The substrate displayed on the landing page is the platform's compound-graph memory as it actually mutated during a recorded agent run, projected into three dimensions, with the original taxonomy and personal data redacted but every topological and timing fact preserved. The hero is not a video and not an animation — it is a player for a small NDJSON file emitted by the running graph. This is not a stylistic preference. It is a forcing function. If the visualization is generated from a real event stream, then any inconsistency between what the platform claims to do and what it actually does will surface visibly on the home page. If the recorder ever lies about what the graph saw, the visualization becomes wrong. If the anonymizer collapses the wrong field, embeddings cluster incorrectly. If the scene's player handles a `node_revoked` event sloppily, dead nodes linger on screen. The entire pipeline is therefore held to the same standard the rest of the platform is held to: it has tests, an explicit format spec, and a contract between subsystems that must not silently drift. The pipeline is also load-bearing in a second sense. The recorder is not a marketing afterthought bolted onto a production graph. It is the same recorder that backs time-travel debugging of agent runs, the audit trail published to capability-vault holders for regulated deployments, the synchronisation log for the Connected-SKU multi-device variant, the correctness gate for sandboxed self-modification, and the substrate over which the anomaly-detection layer fires when an agent run drifts from its baseline behaviour. The hero scene is one consumer of this stream. It happens to be the most visually conspicuous one. This paper documents the three layers of the system as one observable. Section 3 covers the recorder. Section 4 covers the anonymizer. Section 5 covers the visualization substrate. Section 6 reads the five hero scenarios as proof that the same player handles five distinct mutation patterns without scenario-specific code. Section 7 discusses what the live substrate enables beyond the hero: the engineering payoff that justifies the cost of building the stream in the first place. Section 8 closes with a single claim — that the substrate is engineering, not theatre — and a list of consequences. We assume the reader has the Memagen™ master paper [1] in mind; in particular, the master paper's section on the compound-graph memory substrate (Section 10.1) names and motivates the graph that this paper visualises but does not disclose its proprietary implementation. This paper sits one layer below the master: it describes the public contract by which the proprietary graph emits its observable state, not the graph's internal representation. ## 2. Why "show the graph" rather than "describe the graph" The dominant pattern for explaining an agent platform on a website is descriptive prose with diagrams. Prose is cheap to produce and easy to revise. It also accumulates plausible-but-unverifiable claims. A diagram drawn in a vector editor cannot be wrong because it is not connected to anything. An architecture diagram in a PDF cannot drift because there is no behaviour for it to drift from. The result, across the agent-platform marketing landscape, is a steady degradation of the relationship between what the prose says and what the system does. Showing the graph closes that gap. If the substrate visible on the landing page is generated from an event stream emitted by the live graph during a real agent run, then three classes of dishonesty become impossible. We cannot claim the agent decomposes goals into recipe trees if the recorded stream contains no `node_added` events of the corresponding kinds. We cannot claim the graph invalidates superseded facts if the stream contains no `node_revoked` events. We cannot claim embeddings cluster around shared concepts if the projected vectors land in random positions in space. A second, less obvious effect is operational. Once the visualization is bound to a real event stream, the team cannot ship a visualization update without also exercising the recorder, the anonymizer, and the scene's playback semantics. Ordinary marketing edits become engineering changes; the recorder, anonymizer, and scene player accumulate the test coverage and contract enforcement that a piece of pure marketing code would never see. The same player that drives the hero is also the one the team uses internally to inspect a recorded run after the fact, so the instrument stays sharp. ## 3. The recorder The recorder is `core/graph/delta_recorder.py`. Its responsibility is to emit one append-only NDJSON line for every observable mutation of the compound graph and every agent reasoning marker, keep its hot-path cost negligible when disabled, and tolerate concurrent writers without corrupting the stream. ### 3.1 Cheap when disabled Every public method on `GraphDeltaRecorder` begins with `if not self.enabled: return`. When the recorder is not wired in, instrumented call sites pay one boolean check and return. This matters because the recorder hooks every graph write and every agent step; if disabling it required a code change, leaving it on by default would not be safe in performance-sensitive deployments. The unit tests (`tests/graph/test_delta_recorder.py::test_disabled_recorder_is_cheap`) lock this contract at the API surface so a future contributor does not regress to an "always-on" design with branching logic in the slow path. ### 3.2 Append-only NDJSON The wire format, frozen as v1.0 in `docs/timeline-format.md`, is one JSON object per line, UTF-8, LF-terminated, no outer array. The first line is a `{"v":"1.0","type":"format"}` sentinel. Every subsequent line carries a monotonic timestamp `t`, a `type` string drawn from a finite enumeration, and a small type-specific payload. Embedded newlines in string values are escaped as `\n` by the JSON serializer; the file therefore composes cleanly with `tail -f`, `grep`, `wc -l`, and any other line-oriented tool. This format choice is deliberate. The append-only constraint means the file handle never seeks; this in turn means the recorder is stream-safe from the producer side and stream-readable from any number of downstream consumers without coordination. The single-line constraint means a malformed line can be skipped without corrupting the rest of the file. The schema-validated `type` enum means consumers can ignore unknown event types without breaking forward compatibility. ### 3.3 Self-checkpointing Every five seconds of wall-clock time, the recorder emits a `checkpoint` event whose payload is a snapshot of all extant nodes and edges (`_CHECKPOINT_INTERVAL_SEC = 5.0` in the source). A consumer that wants to fast-forward to timestamp T can scan backwards to find the most recent checkpoint preceding T and replay forward from there. This makes seeking in long timelines fast and bounded: the cost of jumping to any moment in a 30-minute recording is at most five seconds of forward replay, regardless of total length. The hero's seek bar uses this same mechanism. The unit tests cover the contract that node and edge revocations remove the affected entries from subsequent checkpoints (`test_checkpoint_includes_added_node`, `test_node_revoked_drops_from_later_checkpoints`, `test_edge_revoked_drops_from_later_checkpoints`, `test_checkpoint_frame_increments`). Twenty-two tests in `test_delta_recorder.py` cover the recorder's behaviour end to end. ### 3.4 Idle heartbeat A daemon thread emits a `heartbeat` event every 250 ms (`_HEARTBEAT_INTERVAL_SEC`) when no other event has fired in that window. The heartbeat distinguishes idle state from a stalled recorder. A downstream consumer that has not seen an event in five seconds can read the heartbeat record and conclude the agent is genuinely idle; absence of both events for five seconds is a stall. The visualization uses heartbeats to avoid jittery camera behaviour during quiet periods of an agent run. ### 3.5 Embedding privacy at the producer The recorder stores embedding vectors in memory keyed by node id (so the post-hoc anonymizer can compute a PCA basis) but never serializes the raw vector to disk. The on-the-wire `embedding_computed` event records only the dimensionality. This is enforced at the recorder layer, not the anonymizer, so a misconfigured anonymizer cannot accidentally leak embeddings — they are not in the file to begin with. `test_embedding_stored_internally_but_not_serialized_raw` locks the contract. ### 3.6 Lenient with caller mistakes The recorder is deliberately lenient about API misuse. A duplicate `node_added` for the same id logs a warning, keeps the first observation, and continues. A `node_revoked` for an id the recorder does not have logs and continues. The trade-off is intentional: tearing down a long agent run because an upstream caller hooked the same write twice is the wrong cost. The recorder's job is to capture as much truth as it can without creating reasons to disable it. ## 4. The anonymizer The anonymizer is `core/graph/delta_anonymize.py`. Its responsibility is to take the raw NDJSON the recorder emits — which carries Memagen™'s real internal taxonomy, real labels, and real free-text — and rewrite it as a published-safe variant whose schema is the generic six-kind / six-edge / three-domain taxonomy specified in `docs/timeline-format.md` Section 3, whose labels are deterministic Greek-letter placeholders, and whose embedding vectors are replaced by a PCA-3 projection. The transformation is intentionally lossy: the public file is structurally truthful and semantically opaque. ### 4.1 The generic taxonomy The anonymizer maps every real Memagen™ node kind to one of six generic categories: `concept`, `task`, `event`, `capability`, `pattern`, `vector`. The mapping is many-to-one and one-way; the list `DEFAULT_KIND_MAP` in `delta_anonymize.py` is the authoritative table. Edges map similarly to one of six generic edge kinds (`parent_of`, `supports`, `contradicts`, `derives`, `mentions`, `similar`) via `DEFAULT_EDGE_KIND_MAP`. Domains collapse to three (`memory`, `task_graph`, `presence`) via `DEFAULT_DOMAIN_MAP`. Anything not in the relevant table defaults to a safe generic value (`concept`, `mentions`, `memory` respectively) and triggers a one-time warning so the policy author can promote it to an explicit row. The structural truth that survives this transformation is everything geometric and topological: layer assignments, parent/child relationships, edge directions, edge kinds at the generic level, fact invalidations, capability grants, and embedding co-locations. What is destroyed is the proprietary cognitive vocabulary: a viewer of the published timeline can see "this task decomposes into three sub-tasks" but not "this is a `recipe:def` that decomposes into three `recipe:step` nodes". The policy is documented end-to-end in `docs/anonymization-policy.md`. ### 4.2 Stable Greek-letter placeholders Every distinct real label is replaced with a deterministic placeholder of the form `<generic_kind>-<greek_letter>-<counter>`. The Greek letter is selected by hashing the real label content (SHA-256, low 32 bits, modulo 12) — so the same real label always picks the same letter. The counter increments per `(generic_kind, letter)` pair so that within one published file, the same real label always maps to the same placeholder. Co-reference is preserved: if a concept is referred to twenty times, the file shows twenty references to the same `concept-mu-7`. The reader sees structural fact, not content. The registry, `_LabelRegistry` in the source, is threaded through the per-event anonymizer so its state accumulates across the timeline. Forty-five tests in `test_delta_anonymize.py` lock this behaviour and the kind-mapping tables. ### 4.3 Deterministic PCA-3 projection Embedding vectors are projected from their original dimensionality (typically 768 or 1024) into three coordinates in `[-1, 1]^3` via PCA. The basis is fit deterministically across all `embedding_computed` events in the input timeline, with three controls that together remove every source of nondeterminism: 1. **Seeded SVD initialization.** The seed string `memagen-2026-05-08` is SHA-256 hashed and the low 64 bits are used to seed numpy's deterministic LAPACK `gesdd` SVD when numpy is available, or a power-iteration fallback otherwise. The seed string is a public constant (`PCA_SEED_STRING` in the source) so any third party with access to the source can reproduce the basis. 2. **Sign canonicalization.** SVD components are sign-ambiguous. The anonymizer flips each component's sign so its largest-magnitude entry is positive, with ties broken by lowest index. This eliminates the dominant nondeterminism in repeated runs. The convention is documented in `docs/timeline-format.md` Section 7. 3. **Per-axis rescale.** After projection, each axis is rescaled per-corpus into `[-1, 1]` using the corpus's min and max along that axis. Embedding clusters that were close in the original 1024-dimensional space remain close in the projected coordinates, but the absolute scale is bounded — important for the visualization, which interprets the PCA coordinates as displacements off the deterministic shell layout. Re-running the anonymizer on the same input produces byte-identical output. This is not just a nice property; it means the published timeline file is a deterministic function of the recording. A diff between two anonymized timelines is a meaningful signal about a change in the underlying agent run, not noise from the anonymizer. ### 4.4 Goal-text redaction `agent_thought.text`, `scenario_start.goal`, `scenario_end.summary`, `agent_tool_call.args_summary`, `node_revoked.reason`, `heartbeat.note`, and `narration.text` pass through a heuristic proper-noun redactor. Capitalised mid-sentence tokens are replaced with `Person`, `City`, or `Name` according to a small lookup table; sentence-start tokens are passed through unchanged unless they appear in the lookup. The heuristic is honest about its limits — it is not an NER system, and the policy document tells the demo author to spot-check redacted free text with `--review-redactions` before publication. This is the weakest link in the pipeline by construction. The compensating control is procedural: the operational rule in `docs/anonymization-policy.md` requires the demo author to read the redaction log of every new scenario and add unmatched names or places to `_NAME_HINTS` / `_PLACE_HINTS` before shipping. The recorder and the anonymizer are self-describing enough that this manual step is well-scoped. ## 5. The visualization substrate The scene is implemented in `web/site-v2/src/lib/scene/`, seven TypeScript modules totalling roughly 2,275 lines. It consumes the anonymized NDJSON, walks events in monotonic time order, and translates each event into a visual mutation of a single Three.js scene. ### 5.1 InstancedMesh node field `instances.ts` wraps a single `THREE.InstancedMesh` of capacity 2,000 with an id-to-instance-id map and a swap-and-pop removal strategy. Every node, regardless of kind, draws as one instance of one icosphere geometry. Per-instance color carries the generic node kind via the `instanceColor` buffer; per-instance fade alpha is folded into the color buffer (the color carries `kind_color * alpha`). When a node is revoked, its instance is swapped with the last live instance and the live count is decremented; this keeps the GPU draw range contiguous. A raycaster against the InstancedMesh returns the original node id for hit testing, so the user can click through into a node and trigger a wormhole transition (Section 5.5). The single-mesh design matters because the alternative — one `THREE.Mesh` per node — would not scale past a few hundred nodes before frame rate collapses. Real agent runs produce timelines with hundreds to a few thousand nodes; the InstancedMesh handles all of them in one draw call. ### 5.2 PBR materials, kind-encoded colors `materials.ts` defines one canonical color per generic node kind: brand lime for `task` (the agent's intent surface, visually loudest), soft cyan for `concept`, soft coral for `event`, amber for `capability`, soft violet for `pattern`, mint for `vector`. Edge materials use additive blending with kind-encoded colors that sit dimmer than the node colors so dense edge graphs stay legible. A small environment map provides PBR rim lighting; a domain-rim accent (memory cyan, task-graph lime, presence amber) gives layers a per-domain hue without flipping the base color. The colors are intentionally restrained. Six categories on screen at once is already a dense visual surface; a seventh would degrade legibility. The brand accent is reserved for the kind that most needs to be noticed in the scenarios. ### 5.3 Spherical-Fibonacci layout `layout.ts` places nodes on six concentric shells, one per layer (`LAYER_RADII = [0, 7, 14, 22, 30, 38, 46]`). Within each shell, nodes are placed by spherical-Fibonacci distribution at indices derived from a stable hash of the node id. The result is deterministic — reloading the page produces the same scene — and visually intentional rather than jittery, which is what a force-directed alternative would produce as nodes are added one at a time. A small per-layer organic offset (`LAYER_JITTER`) nudges nodes off the perfect shell by a deterministic but pseudorandom amount, so the substrate does not read as a CAD blueprint. Position hints from the timeline (`pos_hint` on `node_added`, or the projected PCA coordinates of an attached embedding) override the deterministic placement; the scene smooths toward the hint over a few frames to keep visual continuity. ### 5.4 Vector flow particles `vectors.ts` handles two embedding-related visuals. An `embedding_computed` event spawns a stream of additive points that travel along a quadratic-bezier path from a fixed "vector well" position into the node's surface, then condense into a small persistent glyph attached at the node. A `vector_link` event draws an emissive beam between two nodes whose thickness and opacity scale with the cosine score on the edge. Both use a shared additive material; both clean themselves up after the configured stream lifetime (`STREAM_LIFETIME_MS = 800`). The PCA-projected coordinates from the anonymizer position the persistent glyph, so semantically similar embeddings cluster visibly in the same region of space. ### 5.5 Wormhole transitions `wormhole.ts` implements a camera-dolly plus radial-fog transition between magnification layers. The transition is purely cosmetic — the actual layer swap happens at the midpoint via the `onMidpoint` callback, where the engine clears the InstancedMesh of the outgoing layer and replays the new layer's checkpoint state. Reduced-motion preferences collapse the transition to a single-frame jump. The transition exists so the scene can show the user moving from a coarse view of the substrate (layers 1-3, identity and goal-level) to a fine view (layers 5-6, raw observations and tool results) without losing spatial continuity. ### 5.6 Timeline player `timeline.ts` is a pure data structure: an event-typed parser for the NDJSON wire format and a `TimelinePlayer` that drives the scene by emitting events through an `onApply` callback. The player respects the monotonicity of the timeline, supports play/pause/seek/setSpeed, and uses checkpoints for fast seeks. It carries no graphics state; it only walks events. The separation matters because it lets the same player drive non-graphical consumers (Section 7) without a Three.js dependency. ### 5.7 Engine `engine.ts` (~1,000 lines) wires renderer, scene, lights, env-map, controls, and the timeline-driven node/edge state machine into one object that owns its lifecycle. Public surface, consumed by `web/site-v2/src/components/CompoundGraph3D.astro`: `loadTimeline(events)`, `player.play()/pause()/setSpeed()/seek()`, `setMagnification(layer)`, `dispose()`. The engine is the single place where event semantics meet rendering semantics; everything else is pure. ## 6. Five scenarios as proof The five hero scenarios in `docs/scenarios/` were authored to exercise the visible behaviours of the substrate without redundancy. Each scenario foregrounds one or two dimensions of the substrate's visual surface. Together they cover the surface area; the scene's player handles all five without scenario-specific code. **Coding (`docs/scenarios/01-coding-project.md`) — topology mutation.** A goal node spawns four subgoals; each subgoal spawns a recipe of ordered steps; one branch picks the wrong API, fails red, is superseded, and a clean recovery branch lands the build. The visible behaviour is a tree growing in real time with an explicit fact-invalidation chain off to the side. This scenario tests `node_added`, `edge_added`, `node_updated` on state transitions, `node_revoked` for the failed branch, and the engine's invariant that revocations cleanly remove all incident edges. **Web surfing (`docs/scenarios/02-web-surfing.md`) — vector flow plus capability flow.** The agent crawls a small fan of pages, computes embeddings for each page's content, and the embedding cloud condenses onto three picked nodes. Each fetched page also pulses a one-use `net.fetch` capability grant. The visible behaviour is a flock of additive particles converging into glyphs, plus capability nodes glowing and consuming. This scenario tests `embedding_computed`, `vector_link`, and the capability-as-event family. **Deep research (`docs/scenarios/03-deep-research.md`) — vector flow plus pattern surfacing.** A dense cloud of low-layer concepts is consolidated by the dream-consolidation pipeline (Memagen™ master paper, Section 10.3) into a small set of `pattern` nodes; the cloud shrinks visibly as raw observations are summarised. This scenario tests `node_added` of `pattern` kind, `derives` edges from the pattern back to its source observations, and the visible reduction in node count between layers. **Vacation planning (`docs/scenarios/04-vacation-planning.md`) — recipe execution plus capability flow.** Several parallel recipes (flight search, lodging search, itinerary draft) light up in adjacent lanes and converge on a single daily-plan node. The visible behaviour is parallel topology growth with synchronised convergence, demonstrating that the substrate cleanly handles multiple concurrent task subtrees. This scenario tests the recipe-execution patterns in the master paper without disclosing recipe-engine internals. **Personal secretary (`docs/scenarios/05-personal-secretary.md`) — capability flow plus memory consolidation.** Capability nodes glow in sequence as the agent dispatches calendar reads, contact lookups, and a draft email. Tentative slot beliefs are fact-invalidated as the agent narrows on the right meeting time. The visible behaviour is `node_revoked` events firing on belief nodes that the visualisation marks as superseded. This scenario tests the substrate's most distinctive cognitive behaviour — explicit fact invalidation — at the visual level. The scene's engine has no per-scenario branches. Each scenario is a different sequence of the same dozen event types over the same six layers in the same three domains; the visual differences are emergent. This is the strongest argument for the format: the same player handles five qualitatively different agent behaviours because the format captures the cognitive primitives, not the specific surface workflow. ## 7. What the live substrate enables beyond the demo The hero is the visible payoff. The recorder, the format, and the anonymizer were not built for the hero alone. Five engineering uses share the same NDJSON. **Time-travel debug.** A recorded timeline plus the checkpoint mechanism is a deterministic replay of an agent run. A debugging session loads the file, scrubs to the moment a behaviour went wrong, and inspects the live-set of nodes and edges at that instant. Because the recorder captures `agent_thought` and `agent_tool_call` events as well as graph events, the developer sees what the agent was thinking, which tools it called with which outcomes, and which graph mutations resulted. The Memagen™ team uses the same scene player for this internal use case as ships on the marketing site. **Audit trail.** Regulated deployments require an audit record of every action the agent took and every capability it was granted. The recorder's `agent_tool_call`, capability events, and `node_revoked` events together constitute a complete causal chain. The anonymizer's redaction map can be tuned to a deployment's privacy requirements (the kind table is configurable per-deployment), so the same recorder produces a regulator-acceptable audit log without code changes. **Connected SKU sync backbone.** The Connected SKU variant of Memagen™ keeps multiple devices' graphs in sync. The recorder's append-only NDJSON is the natural sync log: every device emits its mutations, the central reconciler merges, and each device replays the merged stream against its local graph. The format's idempotence properties (a `node_added` for an existing id is a no-op with a warning, a `node_revoked` for a non-existent id is the same) mean reconciliation tolerates duplicate events without special casing. **Selfmod correctness gate.** Sandboxed self-modification (master paper, Section 10.6) requires a behavioural test: a candidate code change must not change the agent's behaviour on a fixed corpus of recorded scenarios. The recorder is the instrument that produces the behavioural baseline, and the timeline diff between baseline and candidate runs is the gate. A change that adds new nodes of new kinds, revokes a fact the baseline did not revoke, or alters embedding-cluster geometry reads as a behavioural delta and blocks promotion. **Anomaly detection.** Once enough timelines have accumulated for a deployment, statistical baselines over event-rate distributions and topology shapes become tractable. An agent run whose `node_revoked` rate is an order of magnitude above baseline, or whose embedding cloud collapses to a single point, is anomalous. The Memagen™ team treats the recorder stream as the cognition-quality signal: a healthy agent produces a healthy graph, and a healthy graph has visible statistical regularities in the event stream. These uses share infrastructure. The debugging, audit, sync, selfmod-gate, and anomaly-detection consumers all read the same NDJSON the hero reads. Adding a sixth consumer is a matter of writing a new walker over the format, not extending the recorder. The single-format discipline is the engineering payoff. ## 8. Conclusion The Memagen™ v4 hero shows the substrate rather than describing it because the same instrument that drives the marketing visualization drives the platform's debugging, audit, sync, self-modification, and anomaly-detection layers. The recorder is hot-path-cheap when disabled. The wire format is append-only NDJSON with checkpoint-based seek. The anonymizer is byte-deterministic, removes proprietary taxonomy without removing structural truth, and projects high-dimensional embeddings into a sign-canonicalised PCA-3 basis whose seed string is a public constant. The scene is a pure player over the format, with a single InstancedMesh node field, a deterministic spherical-Fibonacci layout, additive vector-flow particles, and a wormhole transition between magnification layers. Five scenarios exercise the visible behaviours without scenario-specific code in the player. The thesis of this paper is that none of this is theatre. The visualization is engineering because the alternative — a hand-animated scene disconnected from the running platform — would let claims drift untestably away from behaviour. The recorder and the format exist for engineering reasons that predate the hero; the hero is one consumer of an instrument the platform needs anyway. The substrate is what we built; showing it is the cheapest honest way to communicate what the platform does. ## Pipeline diagram ```mermaid flowchart LR subgraph Producer["Live agent run (Memagen™ runtime)"] graph[Compound graph] agent[Agent loop] end subgraph Recorder["delta_recorder.py — append-only NDJSON"] rec[GraphDeltaRecorder] ndjson[(raw.ndjson)] rec -- "format_version, scenario_start,<br/>node_added, edge_added,<br/>node_updated, node_revoked,<br/>edge_revoked, embedding_computed,<br/>vector_link, agent_thought,<br/>agent_tool_call, checkpoint,<br/>heartbeat, scenario_end" --> ndjson end subgraph Anonymizer["delta_anonymize.py — deterministic post-process"] kind[kind / edge / domain<br/>generic taxonomy map] labels[Greek-letter<br/>label registry] pca[PCA-3 basis<br/>seed = memagen-2026-05-08] published[(published.ndjson)] kind --> published labels --> published pca --> published end subgraph Consumers["Consumers of the same NDJSON"] scene[3D scene<br/>web/site-v2/src/lib/scene/*] debug[Time-travel<br/>debugger] audit[Audit packet<br/>capability vault] sync[Connected SKU<br/>sync log] selfmod[Selfmod<br/>correctness gate] anomaly[Anomaly<br/>detection] end graph --> rec agent --> rec ndjson --> kind ndjson --> labels ndjson --> pca published --> scene ndjson --> debug ndjson --> audit ndjson --> sync ndjson --> selfmod ndjson --> anomaly ``` The diagram makes one fact explicit: the visualization is one consumer among six. The value of the recorder is not the hero. The hero is the consumer that proves the recorder exists and works. ## References [1] Levitas, T. (2026). *Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate*. Memagen™ master paper, [`/papers/2026-05-08-memagen-master`](/papers/2026-05-08-memagen-master). [2] Memagen™ engineering. (2026). *Compound-Graph Timeline Format (v1.0)*. `docs/timeline-format.md`. Frozen interface contract between recorder, anonymizer, and scene. [3] Memagen™ engineering. (2026). *Graph-Delta Anonymization Policy (v1.0)*. `docs/anonymization-policy.md`. Companion to `core/graph/delta_anonymize.py`. [4] Memagen™ engineering. (2026). *Hero Demo Scenario Suite*. `docs/scenarios/`. Five scenario scripts: coding, web-surfing, deep-research, vacation-planning, personal-secretary. ## Code references - `core/graph/delta_recorder.py` — append-only NDJSON recorder, 488 lines. Tests: `tests/graph/test_delta_recorder.py`, 22 cases. - `core/graph/delta_anonymize.py` — deterministic anonymizer with PCA projection, 1289 lines. Tests: `tests/graph/test_delta_anonymize.py`, 45 cases. - `web/site-v2/src/lib/scene/engine.ts` — scene wiring and lifecycle, ~1,000 lines. - `web/site-v2/src/lib/scene/instances.ts` — InstancedMesh node field with swap-and-pop removal. - `web/site-v2/src/lib/scene/layout.ts` — deterministic spherical-Fibonacci shell layout. - `web/site-v2/src/lib/scene/materials.ts` — PBR material factory and per-kind color table. - `web/site-v2/src/lib/scene/vectors.ts` — additive vector-flow particles and link beams. - `web/site-v2/src/lib/scene/wormhole.ts` — camera dolly between magnification layers. - `web/site-v2/src/lib/scene/timeline.ts` — pure NDJSON parser and player. - `web/site-v2/src/components/CompoundGraph3D.astro` — Astro component that mounts the engine. --- ## End of corpus Last updated: 2026-05-08 Source: https://github.com/tablevitas123/crackedclaw Founder: Tab Levitas