PCMCP: A Polymorphic Compound MCP for Unified Agent Tool Surfaces

1. Introduction

A non-trivial agent stack has more than one MCP. UIMCP [2] compresses a large catalog so the agent does not pay tokens for every tool every turn. AutoMCP [3] watches the action graph and proposes parametric macros that get promoted to first-class catalog entries. AutoWebMCP [4] turns arbitrary web pages into runnable tools by parsing form structure and OpenAPI sniffs. Raw vendor MCPs ship per service — a filesystem MCP, a shell MCP, a Slack MCP, a GitHub MCP. And, for desktop control, there is a vision-free accessibility-tree surface (described in Sections 4–6) that competes with screenshot-and-click loops at one to two orders of magnitude lower latency and roughly zero per-action tokens.

Five surfaces, five conventions, five name-resolution behaviors. A naive agent prompt that exposes all five at once gets a tool list with collisions (search on the filesystem MCP, search from a generated webmcp), inconsistent capability declarations (some MCPs declare requires strings, some don’t), and ad-hoc gating (the user set a destructive-action toggle on the desktop surface but it doesn’t apply to the macro layer that wraps the same primitives). The agent now has to know which “search” it meant and what permissions each child MCP enforces.

PCMCP — Polymorphic Compound MCP — is the unifier. It exposes one MCP surface to the agent. Internally it is a merge: every child surface (UIMCP, AutoMCP, AutoWebMCP, raw vendor MCPs, the desktop accessibility-tree surface) registers its tools under an origin tag; PCMCP resolves names with a deterministic precedence rule, prefixes on conflict, propagates each child’s declared capability requirements onto the merged entry, and gates every dispatched call at a single per-tool permission decision against the V3-P capability vault and the safety toggle framework. The polymorphic part is the dispatcher: pcmcp.invoke(name, args) routes to whichever child owns the resolved name, with a single audit record per call regardless of which child handled it.

This paper has two halves. Sections 2–3 cover the merge architecture: child registration, naming and disambiguation, capability inheritance, the dispatcher, and the gating model. Sections 4–7 cover the desktop accessibility-tree surface — the principal Memagen™-original child surface — in implementation detail, because it is the most novel of the children and is the work in progress at the time of writing. Sections 8–12 cover security, evaluation, limits, and future work.

PCMCP ships in the Memagen™ Lite distribution under MIT. The merge layer and the dispatcher are new architectural contributions; the input substrate (core/ui_control.py) and multi-platform tool wrapper (core/tools/computer.py) are production code today; the accessibility-tree introspection is the novel computer-control contribution and is in progress. The master Memagen™ paper [1, §9.4] names PCMCP at the capability level and points to this paper for implementation detail.

2. Architecture: the polymorphic compound

PCMCP is a merge over child MCP surfaces. The merge is a function of three things: the child registry, the name-resolution rule, and the capability-inheritance rule. The diagram below shows the resulting layout.

flowchart TB
    Agent["Agent (LLM)"]:::agent
    subgraph PCMCP["PCMCP — Polymorphic Compound MCP"]
        direction TB
        NS["Merged namespace<br/>(deterministic precedence,<br/>origin-prefix on conflict)"]:::merge
        CAP["Capability inheritance<br/>(union of child requires +<br/>destructive-label allowlist)"]:::merge
        GATE["Per-tool permission gate<br/>(V3-P vault token +<br/>safety toggles + kill switch)"]:::gate
        DISP["Polymorphic dispatcher<br/>pcmcp.invoke(name, args)"]:::merge
        NS --> CAP --> GATE --> DISP
    end
    subgraph Children["Child MCP surfaces"]
        direction LR
        UIMCP["UIMCP<br/>compressed catalog<br/>(see paper [2])"]:::child
        AutoMCP["AutoMCP<br/>inferred macros<br/>(see paper [3])"]:::child
        AutoWebMCP["AutoWebMCP<br/>web tools from URLs<br/>(see paper [4])"]:::child
        RawMCP["Raw vendor MCPs<br/>(filesystem, shell,<br/>Slack, GitHub, …)"]:::child
        DesktopA11y["Desktop accessibility-tree<br/>surface (§4–6)<br/>AT-SPI / UIA / AXUIElement"]:::child
    end
    subgraph Substrate["Shared substrate"]
        direction LR
        UI["core/ui_control.py<br/>input dispatcher<br/>(_guard, _rate_limit)"]:::sub
        Tool["core/tools/computer.py<br/>multi-platform tool<br/>(playwright/xdotool/adb)"]:::sub
        Vault["V3-P capability vault<br/>(per-use Fernet tokens)"]:::sub
        Toggles["Safety toggle framework<br/>(default-strict, hard invariants)"]:::sub
    end
    Agent -- "single tool list +<br/>pcmcp.invoke" --> NS
    DISP -- "routes by origin" --> UIMCP
    DISP --> AutoMCP
    DISP --> AutoWebMCP
    DISP --> RawMCP
    DISP --> DesktopA11y
    DesktopA11y --> UI
    DesktopA11y --> Tool
    GATE --> Vault
    GATE --> Toggles
    classDef agent fill:#fff,stroke:#111,stroke-width:1.5px;
    classDef merge fill:#fff7c2,stroke:#1f2937,stroke-width:1.5px;
    classDef gate fill:#fde68a,stroke:#92400e,stroke-width:1.5px;
    classDef child fill:#f3f4f6,stroke:#111,stroke-width:1px;
    classDef sub fill:#ffffff,stroke:#333,stroke-width:1px,stroke-dasharray: 4 3;

A static rendering of the same diagram is available at /papers/diagrams/pcmcp.svg.

2.1 Child registration

Each child surface registers with PCMCP at startup by handing over a tool list and a child-descriptor:

@dataclass
class ChildDescriptor:
    origin: str                        # "uimcp", "automcp", "autowebmcp",
                                       # "raw:filesystem", "desktop_a11y"
    precedence: int                    # lower wins; ties broken by name
    requires: list[str]                # capability strings the child claims
                                       # e.g. ["fs.read", "ui.input"]
    destructive_labels: list[str]      # optional: child-supplied label
                                       # patterns that should always gate
    invoker: Callable[[str, dict], Awaitable[ToolResult]]
    tools: list[ToolSchema]            # the wire-format MCP tool schemas

The default precedence ordering, lower-wins:

1. desktop_a11y (precedence 10) — accessibility-tree desktop control, the most narrowly scoped surface
2. raw:* (precedence 20) — vendor MCPs the user explicitly connected
3. automcp (precedence 30) — macros over the above
4. autowebmcp (precedence 40) — auto-generated web tools
5. uimcp (precedence 50) — the catalog-compressed pool, lowest because it summons rather than owns

Precedence is not security-relevant — every dispatch still runs through the same gate — but it determines which child wins on a name collision when the user has not specified a prefix.

2.2 Name resolution and disambiguation

PCMCP exposes a single flat namespace to the agent. On registration, each child’s tool names are stored as (origin, name) pairs in an internal registry. The agent-visible name follows three rules, in order:

1. Exact, no conflict — if exactly one child registered the name, the agent sees the bare name (search_files).
2. Exact, conflict — if more than one child registered the same name, the conflict is resolved by precedence; the loser is auto-prefixed with its origin (autowebmcp:search_files). Both names are exposed; the bare name resolves to the precedence winner.
3. Explicit prefix — the agent can always write origin:name (raw:filesystem:search), bypassing precedence. This is the disambiguation hatch.

Conflict events are logged as structured records — origin, name, precedence winner, full collision set — and surface in the developer-mode UI so the user can rename a generated tool if the auto-prefix is undesirable. The audit log makes “which search did the agent call last Tuesday” a real query.

2.3 Capability inheritance

The merged tool entry’s requires list is the union of:

• the child’s declared requires (e.g., AutoWebMCP-generated tools require net.http_outbound; the desktop a11y surface requires ui.input),
• a static per-origin inheritance map (e.g., automcp macros inherit requires from every underlying tool the macro template invokes — a macro that opens a file and posts a message inherits fs.read and net.http_outbound),
• the destructive-label set merged from each child (Delete, Send, Pay, etc., from the desktop a11y surface; wire_transfer, delete_repo, etc., from raw vendor MCPs).

Inheritance is conservative: a merged entry’s required-capability set is a superset of any single child’s, never a relaxation. The agent cannot circumvent a child’s gate by routing through a macro; the macro carries every underlying requirement onto the merged dispatch.

2.4 The dispatcher

pcmcp.invoke(name, args) is the single agent-facing entry point. The flow is:

1. Resolve the name via §2.2’s rules. If unresolvable, return tool_not_found with the candidate set.
2. Gate through §2.3’s merged requires set. The gate makes one decision: it requests a per-use capability token from the V3-P vault for each required capability, evaluates relevant safety toggles (e.g., pcmcp.allow_destructive if the args match a destructive-label pattern), and checks the multi-source kill switch via core.safety.guard. On any failure: structured error and one audit record.
3. Dispatch to the resolved child’s invoker(name, args). The child is unaware of the gating; it gets called only after the gate passes.
4. Audit one record per pcmcp.invoke call: requested name, resolved (origin, name), capability tokens consumed, args summary, child result status. One per call, regardless of child.

A single dispatch path means uniform safety semantics across child surfaces. A user who turns off pcmcp.allow_destructive blocks every destructive-label match, whether the call would have hit the desktop a11y surface, an AutoMCP macro that wraps it, or a generated webmcp form labeled “Delete account.”

2.5 What the polymorphism buys

The polymorphism is not a hand-wave: a single dispatcher with origin-keyed routing is mechanically simpler than five parallel dispatchers, and the simplicity is the point. There is one place to add an audit field, one place to add a kill-switch check, one place to interpret a capability token, one place where the destructive-label allowlist is consulted. Adding a sixth child surface — say, an Android Accessibility Framework path — is a ChildDescriptor registration, not a fork of the dispatch logic.

3. Related work

3.1 MCP namespace strategies elsewhere

The Model Context Protocol specification [13] is silent on namespace merge across multiple connected servers. Existing client implementations adopt one of three patterns: flat union with first-registered-wins (simple, lossy on conflict), per-server prefix on every tool (verbose, no bare-name shortcut), or interactive prompting on conflict (poor agent ergonomics). PCMCP’s deterministic-precedence-with-loser-prefix pattern is a fourth option that preserves bare-name access for the precedence winner while still exposing every conflicting tool under an unambiguous name.

3.2 Capability inheritance in distributed systems

Capability-based security has a long lineage (Hardy, KeyKOS, 1985 [14]). The “inherit on compose” semantics PCMCP uses for merged tool entries follows the principle that a composed action requires the union of its constituents’ authorities. The contribution here is the engineering, not the principle: applying the union rule to a runtime MCP merge with per-tool labels has not, to our knowledge, been done in the published agent-platform literature.

3.3 Vision-based computer control

The dominant pattern for agent computer control — Anthropic’s Claude Computer Use (2024) and OpenAI’s Operator (2025), among others — is a tight loop with three stages: screenshot, vision-LLM call, dispatch. The cost model is roughly one to five seconds and one to ten thousand tokens per action; for a twenty-action workflow that compounds into tens of seconds and tens of thousands of tokens. The desktop accessibility-tree surface inside PCMCP (§4–6) is the alternative for the large fraction of work where the OS already exposes the same widget tree the vision LLM is being asked to re-derive pixel-by-pixel.

3.4 Browser-only structured automation

Playwright (Microsoft, 2020), Puppeteer (Google, 2017), and Selenium (Huggins, 2004) have demonstrated for two decades that browser automation does not need vision. Each operates against the DOM and dispatches events to elements identified by selectors. The DOM is to the browser as the accessibility tree is to the desktop; AutoWebMCP [4] is the analogous structured-surface answer for the web.

3.5 Pre-LLM accessibility-tree automation

Assistive technology has been driving applications via the accessibility tree for thirty years (NVDA, JAWS, VoiceOver, Orca). Test automation for desktop applications — Microsoft’s WinAppDriver, Apple’s XCUITest, Linux’s dogtail — is accessibility-tree-driven. The novelty of the desktop a11y surface is not the technique; it is applying it as the primary path for an LLM agent’s computer-control surface, with semantic commands (“click the Save button”) rather than positional matchers.

4. The desktop accessibility-tree surface

The remainder of this paper covers one child of the polymorphic compound — the desktop accessibility-tree surface — in implementation detail. The merge architecture above applies independently of which child we describe; we focus on this one because it is the most novel and is in active development.

Each platform exposes essentially the same conceptual structure — a tree of nodes with role, label, bounds, state, value, and children — but the bindings differ and the role taxonomies do not align cleanly.

4.1 Linux: AT-SPI 2.x

AT-SPI 2 (Assistive Technology Service Provider Interface) is the freedesktop.org standardized accessibility surface, exposed over D-Bus. Applications register their accessibility trees with the at-spi2-registry daemon at startup; clients query the registry. The Python bindings are pyatspi (legacy) or gi.repository.Atspi (the GObject-introspection-based current binding).

A typical query:

import gi
gi.require_version("Atspi", "2.0")
from gi.repository import Atspi

desktop = Atspi.get_desktop(0)
for app_index in range(desktop.get_child_count()):
    app = desktop.get_child_at_index(app_index)
    # walk app.get_child_at_index(...) recursively

Each Atspi.Accessible exposes:

• get_role() / get_role_name() — role enum and human-readable name (push_button, text, menu_item, frame, table_cell, …)
• get_name() — the visible label
• get_state_set() — bitset of states (focused, enabled, selected, visible, showing, sensitive)
• get_extents(coord_type) — bounding rectangle in screen coordinates
• get_child_count() / get_child_at_index(i) — children
• The Component, Action, Text, Value, and EditableText interfaces, conditional on the node’s role

AT-SPI is well-supported by GTK and Qt (with Qt Accessibility enabled, the default on Linux Qt builds). It is poorly supported by Electron with --disable-features=AccessibilityTree, Java Swing without assistive_technologies set, and any application that renders directly to a canvas (Figma, photo editors, games).

4.2 Windows: UI Automation

Microsoft UI Automation (UIA) is the successor to MSAA and has been the supported Windows accessibility API since Vista. UIA is reachable from C++, .NET, and Python (via the uiautomation PyPI package, which wraps the COM API).

import uiautomation as auto
root = auto.GetRootControl()
# walk root.GetChildren() / root.GetFirstChildControl() recursively

Each Control (the wrapper around IUIAutomationElement) exposes:

• ControlType / ControlTypeName — Button, Edit, MenuItem, Window, ListItem, etc.
• Name — the visible label
• BoundingRectangle — screen coordinates
• IsEnabled, HasKeyboardFocus, IsSelected, etc.
• GetChildren(), GetParentControl()
• Pattern interfaces (InvokePattern, ValuePattern, TextPattern, SelectionItemPattern) conditional on control type

UIA is well-supported across Win32, WPF, WinForms, UWP, and modern WinUI. Browser content within Edge and Chrome maps DOM nodes to UIA elements. Java AWT/Swing requires the Java Access Bridge; Electron generally exposes UIA on Windows because Chromium does. Games and custom-rendered tooling are the usual gaps.

4.3 macOS: AXUIElement

macOS uses the Accessibility Framework, with the AXUIElement C type as the core handle, reachable from Swift, Objective-C, and Python (via pyobjc’s ApplicationServices bindings).

from ApplicationServices import (
    AXUIElementCreateSystemWide,
    AXUIElementCopyAttributeValue,
    kAXChildrenAttribute,
    kAXRoleAttribute,
    kAXTitleAttribute,
)
sys_wide = AXUIElementCreateSystemWide()
# AXUIElementCopyAttributeValue(sys_wide, kAXChildrenAttribute, ...) returns
# the running applications' top-level AXUIElement nodes

Each AXUIElement exposes attributes via string keys: AXRole (AXButton, AXTextField, AXMenuItem, …), AXTitle or AXValue, AXPosition/AXSize (origin and size in screen coords with macOS’s top-left convention), AXChildren, AXFocused, AXEnabled.

macOS requires an explicit user grant: the application driving AXUIElement queries must be granted Accessibility permission in System Settings. The first PCMCP call on macOS triggers a system prompt; without the grant, AXUIElement queries return errors. PCMCP records the consent state in the V3-P capability vault.

Cocoa, Catalyst, and most Electron applications expose rich AX trees. Native non-Cocoa Carbon applications (rare in 2026) expose less. Games and custom-canvas applications are the same gap as the other platforms.

4.4 The common abstraction

The desktop a11y surface’s core type is platform-neutral:

@dataclass
class AccessibleNode:
    role: str                            # normalized: "button", "textfield",
                                         # "menuitem", "link", "checkbox", ...
    label: str                           # the visible name
    value: str | None                    # current value, for inputs
    state: set[str]                      # {"focused", "enabled", "selected", ...}
    bounds: tuple[int, int, int, int]    # x, y, width, height in screen px
    children: list["AccessibleNode"]
    backend_handle: object               # opaque OS-specific handle for
                                         # programmatic actions where avail.

The role taxonomy is a small normalized set — roughly thirty roles. Examples: AT-SPI push_button → button, text (when editable) → textfield, menu_item → menuitem; UIA Button → button, Edit → textfield, MenuItem → menuitem; AXUIElement AXButton → button, AXTextField → textfield, AXMenuItem → menuitem. The mapping table is open-source; it has to be done once per OS release and audited when toolkits change.

The backend_handle is the OS-native object — Atspi.Accessible, IUIAutomationElement, AXUIElement — kept around so the surface can issue programmatic actions through the platform’s own interfaces (AT-SPI’s Action, UIA’s InvokePattern, AX’s AXPress). Programmatic invocation is preferred over coordinate-based clicking when supported: faster and not dependent on the target window being unobscured. The coordinate-based path is the fallback.

5. The translation layer

The translation layer turns a semantic agent command into an OS event. The API is small; the implementation is per-platform.

5.1 click_button(label, role=“button”)

1. Walk the active accessibility tree (cached or refreshed on demand).
2. Filter nodes by role (default button).
3. Match label exactly. If none, fall back to bounded Levenshtein fuzzy match, reusing the resolver Memagen™ uses for tool-name correction in the lenient parser [1, §4]. If the best match exceeds the distance threshold, return a structured not_found rather than guessing.
4. If the matched node exposes a programmatic-invocation interface (AT-SPI Action, UIA InvokePattern, AX AXPress), invoke it directly.
5. Otherwise, compute the screen-space center of bounds and call UIController.click(x, y), dispatching through the chosen input backend.
6. Optionally re-walk the subtree to verify state changed. Verification is opt-in; it doubles the latency.

Fuzzy match is bounded. Strict-only is available; the default is strict-then-fuzzy with a small distance budget. Fuzzy matches log both the requested and matched label so any case where the agent acted on a non-exact label is auditable.

5.2 type_into(field, text)

1. Find the textfield by label, using the accessibility relation between input and label node (LABEL_FOR/LABELLED_BY on Linux, LabeledBy on Windows, AXTitleUIElement on macOS).
2. Focus the field via AXFocused = true (macOS), SetFocus() (UIA), or Atspi.Component.grab_focus() (Linux). Fallback: click the field’s bounds center.
3. Issue keystrokes through UIController.type_text(text). Where the platform exposes a value-set interface (ValuePattern.SetValue, AXValue write, EditableText.set_text_contents), prefer it — faster and not subject to keyboard-layout interference.
4. Optionally verify the field’s value matches what was typed.

Simulated typing is required for fields that validate per-keystroke; value-set is preferred otherwise. The implementation tries value-set first and falls back on rejection.

5.3 Auxiliary commands

• select_menu_item(path=("File", "Save As...")) — walk the menu hierarchy by label
• set_checkbox(label, checked=True) — find checkbox, toggle on state mismatch
• select_dropdown(label, option) — open dropdown, select by label
• focus_window(title) — bring an application window forward
• wait_for(role, label, timeout) — block until a matching node appears
• dump_visible(role_filter=None) — return the full filtered tree for agent reasoning without a screenshot

dump_visible is the most agent-friendly operation. A complex form’s dump is 50–200 nodes — a few hundred to a few thousand tokens — versus 1,500–3,000 for an equivalent screenshot, with more semantic information.

5.4 The fallback to vision

When the desktop a11y surface cannot find what the agent asked for, the response is structured:

{
  "status": "not_found",
  "requested": {"role": "button", "label": "Save"},
  "candidates": [
      {"role": "button", "label": "Save & Close", "distance": 7},
      {"role": "button", "label": "OK", "distance": 4},
  ],
  "tree_complete": True,   # or False if the dump was partial
  "suggestion": "fall_back_to_vision_or_disambiguate",
}

The agent can disambiguate (the requested label was wrong; a candidate is the right one) or fall back to vision (the target is not in the tree). Try fast, fail fast, tell the agent why.

6. The implementation backbone

The desktop a11y surface does not invent an input dispatcher. It builds on core/ui_control.py and core/tools/computer.py, both of which ship today in the Memagen™ Lite distribution under MIT.

6.1 core/ui_control.py — the input substrate

core/ui_control.py exposes a UIController class with a uniform API over pluggable input and screenshot backends. Auto-selection priority for input:

1. PyAutoGuiInput — cross-platform default. Wraps pyautogui, which wraps platform input APIs (Windows SendInput, macOS CGEventCreateMouseEvent, Linux X11 via Xlib). Supports click, move, type_text, key, scroll.
2. XdotoolInput — Linux/X11 fallback. Uses xdotool subprocesses with the safe argv form (xdotool type --delay N -- TEXT). The -- separator and no-shell argv exec are deliberate: an earlier f-string-into-shell form was a command-injection vector when typed text was attacker-controlled.
3. StubInput — test backend. Records every call with no real I/O.

Screenshots are symmetric: MssScreenshot (fast, cross-platform), PilScreenshot (fallback), StubScreenshot (1×1 PNG for tests).

UIController adds two safety layers on top of the backends:

• _guard() — calls core.safety.guard(stage="ui_control") before every action, raising KillSwitchTripped if the multi-source kill switch is set. The PCMCP dispatcher inherits this; an action proposed during a kill-switch event is refused at the UIController boundary before reaching the input backend.
• _rate_limit() — min_action_interval enforces a minimum gap between successive actions, preventing runaway input loops.

Wiring all desktop output through UIController means the desktop a11y surface and any vision-based path share the same input backend, kill-switch state, and rate limit. No second code path to keep in sync.

6.2 core/tools/computer.py — the multi-platform tool

core/tools/computer.py is the LLM-tool-facing wrapper around the same input layer plus backends not handled by core/ui_control.py. The Backend literal is "playwright" | "xdotool" | "adb" | "none", and _detect_backend() returns the first available:

def _detect_backend() -> Backend:
    if shutil.which("adb"):
        # adb available but don't auto-select (need device check)
        pass
    if shutil.which("xdotool") and shutil.which("scrot"):
        return "xdotool"
    return "none"

The four LLM-facing tools — computer_screenshot, computer_click, computer_type, computer_launch — are async and dispatch through _run_argv, a hardened subprocess wrapper:

async def _run_argv(argv: list[str], timeout: float = 10.0):
    proc = await asyncio.create_subprocess_exec(*argv, ...)
    out, err = await asyncio.wait_for(proc.communicate(), timeout=timeout)
    return out.decode(...), err.decode(...), proc.returncode

The argv-only form (no shell) is deliberate: every previous f-string-into-shell call site was a command-injection vector when args were attacker-controlled. _validate_coord clamps coordinates to [0, 16000]. computer_type enforces a 4096-character ceiling. computer_launch uses shlex.split and shutil.which to validate before dispatch, refusing empty, oversized, and off-PATH commands.

The desktop a11y surface (§5) calls into core/ui_control.py for desktop control on Linux/macOS/Windows. The ADB backend handles Android targets; the Android Accessibility Framework is a candidate child surface for future PCMCP work (§11.4).

6.3 The boundary

The desktop a11y surface’s contribution is the accessibility-tree introspection and the translation layer. Everything below — input backends, kill-switch guard, rate limiter, screenshot backends — is existing Memagen™ infrastructure. The translation layer is testable in isolation against a stub UIController; the input layer is testable in isolation against synthetic semantic commands.

7. Cost comparison: a11y-tree path vs vision-based control

Vision-based numbers below are published latency characteristics of frontier vision LLMs as of early 2026; the a11y-tree numbers are measurements on internal harness runs with the open-source code.

7.1 Vision-based “click Save”

Stage	Latency	Token cost
Screenshot capture (mss)	30–80 ms	0
Image encoding (base64)	5–20 ms	0
Vision LLM call (frontier, 1024×768 PNG, “click the Save button” prompt)	1.5–5 s	1,500–3,000 in / 100–500 out
Coordinate parse	<1 ms	0
Click dispatch (pyautogui)	10–30 ms	0
Total	1.55–5.13 s	1,600–3,500 tokens

7.2 a11y-tree “click_button(label=‘Save’)“

Stage	Latency	Token cost
Tree walk (cached, ~50–200 nodes)	1–5 ms	0
Tree walk (cold, ~200 nodes via AT-SPI/UIA/AX)	20–80 ms	0
Label match (strict + bounded fuzzy)	<1 ms	0
Programmatic invoke (where supported)	5–20 ms	0
Coordinate-based click dispatch (where invoke not available)	10–30 ms	0
Total (cached tree, programmatic invoke)	6–25 ms	0 per-action
Total (cold tree, coordinate click)	30–115 ms	0 per-action

7.3 Speedup

The cached-tree-plus-programmatic-invoke path is 60–800× faster than the vision path. The cold-tree-plus-coordinate-click path is 13–170× faster. The token-cost ratio is bounded only by the size of the (one-time) tree dump the agent needs to reason over the screen; for a complex form, that dump is roughly equivalent to one screenshot in tokens, but it can be reused across many actions on the same screen.

The 10–100× factor in the subtitle is conservative. It is the right ratio to plan against: real workflows have caching misses, fuzzy fallbacks, and verification re-walks that the headline “1,000×” number from the cached-invoke path does not capture.

7.4 Caveat

A direct head-to-head latency benchmark against a vision LLM would require a frontier-model API key and the cost of running the comparison. The vision-LLM numbers in §7.1 are the published latency characteristics of Claude 4.6 (Anthropic, 2025), GPT-5o (OpenAI, 2025), and Gemini 2.5 Pro Vision (Google, 2025) as of early 2026, sourced from each provider’s documentation rather than from a Memagen™-run benchmark. The a11y-tree numbers are measurements on internal harness runs on Ubuntu 24.04 with GTK and Qt applications. Honest framing: the desktop a11y surface is an order of magnitude faster than vision-based control on workflows where the tree is exposed; the exact factor depends on the platform, toolkit, and workload.

8. When the desktop a11y surface doesn’t work

There are several classes of application where the surface fails or degrades.

8.1 Custom-canvas applications

Figma, Photoshop, Premiere, Blender, and most photo and video editors render to a custom canvas rather than OS-managed widgets. The accessibility tree typically exposes the application window and a small number of toolbars/panels but not individual canvas elements. The surface can drive the toolbar, the menu bar, and the panels, but cannot target an arbitrary canvas point semantically. Vision is required for canvas content.

8.2 Stripped-a11y Electron apps

Electron applications inherit Chromium’s rich accessibility surface. However, some Electron apps run with --disable-features=AccessibilityTree or do not enable accessibility at all in their packaged form (Slack and Discord historically had partial coverage; the situation improves yearly). When a11y is stripped, the surface can sometimes fall back to walking the renderer’s DOM via the Chrome DevTools Protocol — but this requires the app to expose the debugging port, which most production Electron builds do not. Realistic outcome on stripped Electron: window-level operations work; in-app interactions don’t.

8.3 Games

Games generally do not expose accessibility trees. Some accessible games do (notably Microsoft titles and several recent first-party PS5 titles, though those are not desktop). For desktop games the surface is not the right tool. Vision-based control can handle game input, but agent-driven game-playing is rarely the requested use case.

8.4 Java without the access bridge

Java AWT/Swing applications expose accessibility only when the Java Access Bridge is installed and javax.accessibility.assistive_technologies is set. On a default Java install, the tree is empty or contains only the top-level frame. Modern JavaFX is better behaved (it integrates with native a11y). Memagen™ documents the configuration step; without it, those apps require vision.

8.5 Visual confirmation steps

There are workflows where vision is genuinely needed even when the accessibility tree is rich. “Verify the chart renders correctly,” “the report has no spurious labels,” “the diagram looks like the design mock” — these are visual judgments that no a11y framework captures, because no framework describes the appearance of a rendered widget. The surface cannot replace vision here; the answer is the hybrid agent (§8.7).

8.6 Cross-platform inconsistencies

A given application — say, Visual Studio Code — exposes a different tree on macOS (rich, Cocoa-bridged), Windows (rich, UIA-bridged via Chromium), and Linux (less rich; depends on the build). A script that targets a label that exists on one platform may not find the same label on another. The pragmatic answer: write per-platform when high reliability is needed, and let fuzzy match cover small label discrepancies.

8.7 The hybrid path

The production-realistic agent uses the desktop a11y surface first and falls back to vision-based control on not_found. The agent’s prompt presents both as available tools; the cheaper, faster path is preferred when the agent has reason to expect the tree contains the target; vision is the escape hatch.

A more ambitious hybrid captures a screenshot only on a11y failure, sends it to a small fast vision model with the failure context (the tree dump, the requested label, candidates that didn’t qualify), and uses the resulting coordinate. This raises effective coverage to “any application a vision model can drive” while preserving the cost benefit on the (large) fraction where the surface succeeds. Not yet shipped.

9. Security and capability

Computer control is one of the highest-leverage agent capabilities. An agent with click_button(label="...") and type_into(field="...", text="...") can do everything a desktop user can, including operations the user did not intend. PCMCP’s gate (§2.4) is the same gate every other tool runs through [1, §5–6]; this section spells out how it composes for the computer-control case.

9.1 V3-P capability vault

Each invocation requests a per-use capability token from the V3-P vault before dispatch. The token carries the action kind (pcmcp.click_button, pcmcp.type_into, pcmcp.set_checkbox, …), an optional payload-summary constraint (e.g., “the typed text begins with…”), and a time-to-live. The token is consumed exactly once and audited as a structured record. Every PCMCP-driven mouse click and keystroke is therefore queryable as data: “show me every pcmcp.type_into against the field labeled ‘password’ in the past 30 days” is a structured query.

9.2 Safety toggle: pcmcp.allow_destructive

PCMCP exposes one user-controllable safety toggle: pcmcp.allow_destructive. Default off. When off, calls whose resolved arguments match the destructive-label allowlist — buttons labeled “Delete,” “Remove,” “Send,” “Submit,” “Pay,” “Purchase,” “Sign,” “Format,” “Erase,” and similar — are blocked at the gate. The agent receives a structured error and can ask the user to flip the toggle.

When on, the toggle has been loosened through the standard ceremony: a five-bullet warning, a verbatim acknowledgment phrase (paste-detection blocks paste-only entries), re-authentication, and a 60-second cooldown. The audit log records the loosening transition. Even with the toggle on, the final_human_review_on_grant_submission hard invariant from the safety framework still applies. There is no PCMCP path that submits a job application or a financial transaction without a human-in-the-loop click; that invariant runs after the toggle and cannot be bypassed.

9.3 The label-allowlist as a coarse defense

The destructive-label allowlist is coarse on purpose. It does not depend on parsing the application’s intent; it acts on what the user would see on the button. PCMCP refuses to click “Delete Account” without explicit consent, regardless of which application is on screen. This is not a substitute for application-level authorization (the application enforces its own permissions), but it is a substantive guard against an agent acting on a destructive control by mistake or by prompt injection — and because the allowlist sits at the merged dispatcher (§2.4), it gates calls through every child surface uniformly.

10. Evaluation

Honest framing: the merge layer and dispatcher are new code with unit-test coverage but no scaled benchmark. The desktop accessibility-tree integration is in active development. The existing input dispatcher (core/ui_control.py) and the multi-platform computer-control tool (core/tools/computer.py) are production code today, used by the vision-based computer-control path and by direct-coordinate-click paths in tests.

What has been measured internally:

• AT-SPI tree dumps on Ubuntu 24.04 with GTK 4 / Qt 6 / Cocoa-bridged Electron applications. Tree walk latency 20–80 ms cold, 1–5 ms cached. Programmatic invoke latency 5–20 ms.
• Coordinate-based click latency on Linux via pyautogui and via xdotool: 10–30 ms in both cases.
• Failure-mode classification on a small (N≈12) set of applications: GTK GNOME (Files, gedit, Settings) — full coverage. Qt (KDE, Telegram, OBS) — full coverage. Electron (VS Code, Slack desktop) — partial; window-level and in-app navigation work, some custom controls don’t expose labels. Browser content (Firefox, Chromium) — full coverage of native chrome, partial of web content depending on page a11y attributes. Custom-canvas (Inkscape, Blender) — no canvas coverage; toolbar/menu coverage works.
• Merge-layer microbenchmarks: name resolution ≤ 10 µs against a 200-tool merged registry; gate decision dominated by V3-P token issuance (sub-millisecond) on a hot path.

What has not been measured at publication-grade:

• Scaled benchmarks on Windows (UIA) and macOS (AXUIElement). These platforms are next on the harness extension path.
• Direct head-to-head latency against a vision LLM under live API conditions. As stated in §7.4, vision-LLM numbers are providers’ published characteristics, not a Memagen™ benchmark.
• Long-tail application coverage. N≈12 is small; a meaningful coverage number requires N in the hundreds across versions and toolkits.
• Multi-child collision rates in real catalogs. We do not yet have user-base statistics on how often (autowebmcp, x) and (raw:y, x) collide in practice; the precedence rule is conservative but the empirical conflict distribution is not yet a reportable number.

We list these limitations because the alternative — claiming a benchmark we have not run — would not be honest. The forthcoming v2 of this paper will include scaled-benchmark numbers once the harness has been extended. Until then, the headline claims are structural rather than empirical: where the tree is exposed, the desktop a11y child is an order of magnitude faster than vision-based control at near-zero per-action token cost; where the tree is not exposed, the surface fails fast and tells the agent why; the merge layer keeps every child’s safety semantics uniform regardless of which child handled a call.

11. Future work

11.1 Hybrid a11y-plus-vision child

The natural extension is a hybrid child surface that uses the a11y tree as primary and falls back to a small fast vision model only when tree walk returns not_found. The fallback is cheap when the model has the failure context (tree dump, requested label, near-miss candidates). This raises effective coverage to the union of a11y and vision while preserving cost benefits on the a11y-covered fraction. Implementation is straightforward; the agent prompt presents both as available tools and reasoning chooses.

11.2 Cross-application workflow inference (a11y → AutoMCP)

The a11y child’s structured action stream is exactly the right input for AutoMCP [3]‘s pattern recognizer. A persistent agent observes that the user copies data from a spreadsheet, pastes it into a web form, and submits. The structured introspection makes the source and destination of each copy/paste legible: the spreadsheet cell at coordinate (col=4, row=12), the web form input labeled “Quantity.” Recording these structured action chains and proposing them as named macros via AutoMCP — through the same merge — is a substantially stronger building block for cross-application automation than either alone.

11.3 Trust-tier UI surfacing

Memagen™‘s widget framework can surface dispatch confidence in the user-facing UI. When a call is about to dispatch, the trust-tier surface shows: requested action (“click button labeled ‘Save’”), resolved child and matched node (“desktop_a11y / button at (812, 540), exact label match”), confidence (“strict”). Fuzzy matches are flagged. Destructive calls escalate to confirmation regardless of toggle state. Users get a always-visible read of what the agent is about to do, in semantic terms rather than coordinate terms.

11.4 Android via the Android Accessibility Framework

core/tools/computer.py already supports an adb backend for screenshot, click, and type on Android. The natural extension is a new child surface that walks the Android Accessibility Framework tree (queryable via UIAutomator or the AccessibilityService API) and exposes the same AccessibleNode abstraction. This is mostly a porting exercise — the abstraction is the same; the platform glue and the ChildDescriptor registration are different.

11.5 Memagen™ Pro Local extensions

The Pro distribution adds two PCMCP-adjacent capabilities outside the open-core scope:

• Recipe-execution engine [1, §10.4] — named, parameterized, durable PCMCP sequences with crash-resume. A recipe like “fill out the weekly status form” can be replayed against the same form on the same site after a crash, resuming from the last completed step.
• Cognition modules [1, §10.5] — preferences and beliefs that bias dispatch behavior. The agent learns that on this user’s machine, “Save” buttons tend to be lower-right in dialogs, and the resolver weights toward that prior when fuzzy-matching.

Both are out of scope for this paper; they are described at the capability level in the master Memagen™ paper and not disclosed at implementation level.

12. Conclusion

A production agent stack accumulates MCP surfaces. PCMCP — Polymorphic Compound MCP — gives the agent one. The merge resolves naming collisions deterministically, inherits each child’s capability declarations onto the merged entry conservatively, and gates every call at a single per-tool permission decision against the V3-P capability vault, the safety toggle framework, and the multi-source kill switch. The polymorphic dispatcher routes by origin to UIMCP [2], AutoMCP [3], AutoWebMCP [4], raw vendor MCPs, or the desktop accessibility-tree surface, with one audit record per call regardless of which child handled it.

Within that compound, the desktop accessibility-tree surface is the principal Memagen™-original computer-control child. Where the OS exposes a usable tree (most native applications, many Electron apps with a11y enabled, most browser chromes), it runs one to two orders of magnitude faster than vision-based control at near-zero per-action token cost. Where the tree is not exposed (custom-canvas applications, stripped-a11y Electron, games), the surface returns a structured not_found and the hybrid path falls back to vision.

PCMCP is gated through V3-P and the safety toggles uniformly. Every call is capability-tokened, audited, and refused at the dispatcher if the kill switch is set. The destructive-label allowlist refuses obvious destructive actions absent explicit consent. Hard invariants from the safety framework — final-human-review-on-grant-submission in particular — apply regardless of toggle state and regardless of which child surface a destructive call would have routed to.

The Lite distribution is sufficient for a complete PCMCP integration. The Pro distribution adds the recipe-execution engine and cognition modules that compound on PCMCP without changing its open-core surface. We invite the field to build on the open work and to evaluate the closed work by what it lets users do.

Source: https://github.com/tablevitas123/crackedclaw. Master paper: [1].

References

1. Levitas, T. (2026). Memagen™: An Open-Core Agent Platform with Lenient Tool Routing, Capability-Gated Execution, and a Compound-Graph Memory Substrate. Memagen™ master paper, 2026-05-08.
2. Levitas, T. (2026). UIMCP: Adaptive Catalog Compression for Multi-Tool LLM Agent Dispatch. Memagen™ specialty paper, 2026-05-08. https://memagen.com/papers/2026-05-08-uimcp
3. Levitas, T. (2026). AutoMCP: Inferring Parametric Macros from Typed Action Graphs. Memagen™ specialty paper, 2026-05-08. https://memagen.com/papers/2026-05-08-automcp
4. Levitas, T. (2026). AutoWebMCP: Automatic MCP Tool Schema Generation from Arbitrary Web Pages. Memagen™ specialty paper, 2026-05-08. https://memagen.com/papers/2026-05-08-autowebmcp
5. Anthropic. (2024). Computer use with Claude (beta). https://docs.anthropic.com/en/docs/build-with-claude/computer-use
6. OpenAI. (2025). Operator system card and capability documentation. https://openai.com/index/introducing-operator/
7. freedesktop.org. AT-SPI 2 specification and Atspi GObject-introspection bindings. https://www.freedesktop.org/wiki/Accessibility/AT-SPI2/
8. Microsoft. UI Automation overview and IUIAutomationElement reference. https://learn.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32
9. Apple. Accessibility programming guide for OS X — AXUIElement. https://developer.apple.com/library/archive/documentation/Accessibility/Conceptual/AccessibilityMacOSX/
10. Microsoft. WinAppDriver — UI Automation-driven test driver for Windows applications. https://github.com/microsoft/WinAppDriver
11. NV Access. NVDA — Non-Visual Desktop Access screen reader source code. https://github.com/nvaccess/nvda
12. Microsoft. (2020). Playwright — browser automation library. https://playwright.dev/
13. Anthropic. (2024). Model Context Protocol Specification. https://modelcontextprotocol.io
14. Hardy, N. (1985). KeyKOS architecture. ACM Operating Systems Review, 19(4) (cited via Memagen™ master paper [1] for capability-based security history).
15. PyAutoGUI authors. PyAutoGUI documentation — cross-platform GUI automation. https://pyautogui.readthedocs.io/
16. freedesktop.org. xdotool — command-line X11 automation. https://github.com/jordansissel/xdotool
17. Bernstein, D.J. (2014). The Fernet specification (cited via Memagen™ master paper [1] for the V3-P vault implementation). https://github.com/fernet/spec