Multi-Provider LLM Routing for Tool-Calling Agents: A Provider-Abstraction with Lenient-Parser Splice Points

1. Introduction

The boundary between an agent runtime and a large language model is one place an architectural choice has compounding cost. Most agent frameworks make that choice once: pick OpenAI, pick Anthropic, write the integration, ship. The cost is paid afterwards — every time the chosen provider raises a rate limit, removes a model, raises prices, suffers an outage, becomes unavailable in a customer’s region, or offers a worse cost-per-token ratio than a competitor on the workload at hand. Customers of those frameworks inherit the choice, with no recourse short of rewriting the integration themselves.

Memagen™ (Levitas, 2026) takes the opposite position. The router lives between the agent loop and any provider. A single ChatRequest (defined at core/llm/types.py) is the canonical request shape; every provider implementation translates that shape into the upstream API’s native form, runs the call, and translates the streaming response back into a unified event vocabulary. The Lite distribution exercises three implementation kinds — openai, openai_compatible, and anthropic — and reaches a wide surface of upstream targets: OpenAI directly, Anthropic Messages directly, and any OpenAI-compatible endpoint (NVIDIA NIM, Groq, Together, DeepSeek, Mistral, Fireworks, Cerebras, xAI, Perplexity, Voyage, Cohere’s compatibility endpoint, OpenRouter, Google Gemini’s compatibility endpoint, and any local server that speaks the OpenAI Chat Completions wire format including Ollama, llama.cpp, vLLM, and LM Studio).

A wide provider surface behind a thin canonical request shape is well-trodden in the routing-library literature; what is novel here is how the router participates in the lenient-parser layer described in Lenient Tool-Call Parsing. The router is not just a dispatch table. It exposes two splice points that make the lenient parser provider-aware: (a) an _apply_scrub step that publishes the active model name and the live tool catalog into ContextVars the system-prompt assembler reads, so the prompt carries the right tool-format hint for the current model; and (b) a _stream_with_credential / _stream_with_failover step that wraps the provider’s raw stream in lenient_parser.wrap() for models that need it. Both ship under MIT in the Lite distribution and let a single deployment mix a frontier-model orchestrator with a cheaper-model executor without the executor’s idiosyncratic tool-call emissions breaking the agent loop.

Section 2 surveys related routing libraries. Section 3 describes the ChatRequest type and the router’s public API. Section 4 walks each supported provider and names the idiosyncrasy it forces into the abstraction. Section 5 is the core contribution: the splice points. Section 6 covers failover and pools. Section 7 covers per-agent routing and the settings_bridge. Section 8 documents the claude-bridge shim. Section 9 names the streaming event taxonomy. Section 10 lists honest limitations. Section 11 names future work.

2. Related work

LiteLLM (BerriAI, 2023) is the most-cited Python routing library; it provides a unified completion() interface over more than a hundred provider endpoints. We considered it and chose not to. First, LiteLLM is structured around per-call dispatch and does not expose the kind of streaming event hooks the lenient parser needs — splicing an arbitrary async generator over the upstream stream while preserving back-pressure and cancellation semantics. Second, it optimizes breadth over depth of tool-call support, and the per-provider tool-call shape translation in OpenAICompatibleProvider._iter_sse and AnthropicProvider._iter_anthropic_sse (Section 4) is the load-bearing part for an agent platform. Extending LiteLLM would be larger than writing the routing layer directly, and we wanted the splice points the router exposes today.

LangChain (Chase, 2022) provides chat-model abstractions via classes like ChatOpenAI, ChatAnthropic, ChatGoogleGenerativeAI. The unification works at the model-instantiation layer rather than at the request-routing layer; per-agent overrides, pools, and failover live in user code, not in the framework. Fine for chains; incomplete for agent loops where the routing decision needs to be made per call based on agent identity.

OpenRouter (OpenRouter Inc., 2024) is an inference marketplace: a single API endpoint that itself routes to many providers, with provider preference expressible via headers. We support OpenRouter as a back-end (it appears as an openai_compatible provider) but do not depend on it. OpenRouter does the routing-layer job well; it cannot insert per-event hooks inside the provider’s streaming response, and the splice we need is inside the stream.

LangSmith and similar observability tools (Helicone, Langfuse, Phoenix) sit alongside the router rather than in it; they consume the events the router emits. Memagen™‘s observability uses a Prometheus counter (memagen_dispatch_total, labeled by provider and success, incremented in the finally block of every dispatch path in core/llm/router.py) plus the core.trace.span context manager opened around each provider call. Intentionally minimal — observability inside the router survives provider changes.

Direct provider SDKs (openai, anthropic, google-genai, cohere) are excellent at their one provider and do not pretend otherwise. We use them as inspiration for request and response shapes but call providers via raw HTTP through httpx.AsyncClient (Provider.__init__); the SDKs each add a non-trivial dependency and their streaming abstractions are not consistent with each other.

Each prior approach is incomplete for tool-calling agents specifically. The gap is the splice point the lenient parser needs. The router fills that gap.

3. The ChatRequest abstraction

The canonical request shape is a frozen dataclass at core/llm/types.py:

@dataclass
class ChatRequest:
    system: str
    user: str
    history: list[ChatMessage] = field(default_factory=list)
    temperature: float = 0.7
    max_tokens: int = 4096
    top_p: float = 0.95
    tools: list[dict] | None = None  # OpenAI-format function definitions

ChatMessage carries the canonical role / content pair plus optional tool_calls and tool_call_id for tool-result turns. It also carries a _tool_call_fmt field excluded from to_dict(); this records which lenient-parser format produced the message ("native", "kimi_special", "xml_invoke", "xml_sentinel") so downstream replay logic can re-emit in the same format.

The router’s job, in one sentence: take a ChatRequest, resolve the calling agent_id to a route, build the upstream payload via the chosen provider’s _payload() method, dispatch, translate the streaming response back into the unified event vocabulary, and return.

Three things about this shape are load-bearing. First, tools is a list of OpenAI-format function definitions ({name, description, parameters: <JSONSchema>}) regardless of upstream provider; the per-provider translation happens inside the provider. AnthropicProvider._payload translates parameters to input_schema and re-keys the entry; we do not push that translation onto agent code. Second, system is a single top-level string, not a list of system messages; this matches Anthropic’s Messages API contract (system at the top level, separate from messages) and degrades cleanly into a system message when the provider is OpenAI-compatible (build_messages prepends {"role": "system", "content": self.system} to the message list). Third, history is a list of ChatMessage, not dicts; the dataclass-as-history shape carries the _tool_call_fmt annotation without leaking it into the wire format.

The router is constructed from an LLMSettings and an optional CredentialStore:

class LLMRouter:
    def __init__(
        self,
        settings: LLMSettings,
        *,
        credential_store: CredentialStore | None = None,
    ) -> None:
        ...

It exposes three primary async methods: complete(request, *, agent) returns the content string only; complete_with_tools(request, *, agent, ...) returns (content, tool_calls, usage); stream(request, *, agent, ...) and stream_with_tools(request, *, agent, ...) are the streaming variants. The streaming variant accumulates tool_call_delta events into final ToolCall objects (lines 834–847 of core/llm/router.py) so callers do not reassemble fragments themselves.

4. Provider abstraction

Each provider subclasses an abstract Provider (core/llm/provider.py) and implements complete() and stream(). The base class holds a single httpx.AsyncClient configured with the router’s shared timeout. Providers are stateless given (api_key, model, request); they know nothing about pools, rate limits, budgets, agent identity, or the lenient parser. The router composes those concerns around them.

The current registry, at core/llm/provider.py:

PROVIDER_REGISTRY: dict[str, type[Provider]] = {
    "openai_compatible": OpenAICompatibleProvider,
    "openai": OpenAIProvider,
    "anthropic": AnthropicProvider,
}

Three implementation classes cover the entire provider surface, because the OpenAI-compatible wire format has become the de facto standard for non-Anthropic and non-Google-native APIs. The _PROVIDER_SHAPES table at core/llm/credential_route.py maps user-facing names onto (kind, base_url) pairs:

User-facing name	Kind	Base URL
nvidia	openai_compatible	https://integrate.api.nvidia.com/v1
openai	openai	(SDK default)
anthropic	anthropic	(SDK default)
google	openai_compatible	https://generativelanguage.googleapis.com/v1beta/openai
groq	openai_compatible	https://api.groq.com/openai/v1
mistral	openai_compatible	https://api.mistral.ai/v1
together	openai_compatible	https://api.together.xyz/v1
fireworks	openai_compatible	https://api.fireworks.ai/inference/v1
deepseek	openai_compatible	https://api.deepseek.com/v1
xai	openai_compatible	https://api.x.ai/v1
cerebras	openai_compatible	https://api.cerebras.ai/v1
cohere	openai_compatible	https://api.cohere.ai/compatibility/v1
perplexity	openai_compatible	https://api.perplexity.ai
voyage	openai_compatible	https://api.voyageai.com/v1
ollama	openai_compatible	http://localhost:11434/v1

OpenRouter, NVIDIA NIM, llama.cpp, vLLM, and LM Studio reach the router via the same openai_compatible kind with their respective base_url values supplied by the user (LM Studio defaults to http://localhost:1234/v1; vLLM uses whatever the operator passes to --port). The user-facing list named in the master paper is not a list of distinct integrations but a list of attestations: we have run agents against each of them.

Most of the engineering in provider abstraction is per-provider streaming-event translation. The non-streaming complete paths are largely shape-mapping; the SSE handler is the hard part. Each kind’s idiosyncrasies are documented below.

4.1 OpenAI

OpenAI itself is the reference implementation (OpenAIProvider, a thin subclass of OpenAICompatibleProvider). Tool calls arrive in the tool_calls array on the assistant message, each with an id, a type: "function" discriminator, and a function: {name, arguments} payload where arguments is a JSON-encoded string, not a dict. Streaming SSE events arrive as data: <json> lines; choices[0].delta carries either content, tool_calls, or reasoning fragments; the stream terminates with data: [DONE]. We request a final usage frame via stream_options: {include_usage: true} (core/llm/provider.py line ~194). Tool definitions are sent as [{type: "function", function: {name, description, parameters}}].

4.2 Anthropic

Anthropic does not use the OpenAI shape. The AnthropicProvider (core/llm/provider.py line ~219) speaks the Messages API directly. Two structural differences: system is a top-level field (not a message), and the response content is a list of typed blocks rather than a single string. Block types include text, tool_use, and thinking. Tool calls arrive as tool_use blocks with id, name, and an input dict (already parsed JSON, unlike OpenAI’s string). We JSON-encode input back to a string so downstream ToolCall.arguments matches the OpenAI contract.

Streaming is more involved. The Messages API emits events of types content_block_start, content_block_delta, content_block_stop, and message_stop. A tool_use block opens via content_block_start and is filled via subsequent content_block_delta events of type input_json_delta, whose partial_json field carries string fragments of the eventual JSON arguments. We track open tool blocks in a per-stream dict[int, dict[str, str]] keyed by index, emit a synthetic opening tool_call_delta event when the block opens (so the accumulator records id and name before fragments arrive), and forward each partial_json fragment as a fresh tool_call_delta event with the same index. The accumulator concatenates fragments into a single arguments string. This translation lives at _iter_anthropic_sse (core/llm/provider.py lines ~301–378).

Thinking blocks (extended-thinking output, available on claude-opus-4-7 and similar) arrive as content_block_delta events of type thinking_delta with a thinking text payload. We forward these as ("reasoning", text) events on the unified stream — the same kind a Kimi-style delta.reasoning produces on the OpenAI-compatible side. We had to update the provider when thinking blocks landed in 2025; provider-specific code is the biggest source of churn in the codebase, and we expect it will continue to be.

4.3 Google Gemini

Gemini reaches the router through the openai_compatible kind via Google’s generativelanguage.googleapis.com/v1beta/openai endpoint — an OpenAI-compatibility shim Google publishes. The native Gemini API has function calls in candidates[].content.parts[].functionCall and a different streaming envelope; we have not implemented a native GeminiProvider because the compatibility endpoint is sufficient for the agent loop. Safety filtering, when triggered, surfaces as an HTTP 400 with a structured promptFeedback payload; we currently surface this as a generic RuntimeError, which the agent loop relays to the user. A proper GeminiProvider is on the roadmap; the current behavior is not great when safety filters fire.

4.4 OpenRouter

OpenRouter is openai_compatible with base_url = "https://openrouter.ai/api/v1". Routing is by model name (anthropic/claude-opus-4-7, google/gemini-2.5-pro). Provider-preference headers (X-OR-Prefer-Provider) can be passed through ProviderSpec.extra and set in custom headers; the current OpenAICompatibleProvider._headers does not yet wire these — adding pass-through of extra.headers is on the future-work list.

4.5 NVIDIA NIM

NIM is openai_compatible with base_url = "https://integrate.api.nvidia.com/v1". The same endpoint serves Kimi (moonshotai/kimi-k2.5), Llama, Mistral, and others. NIM emits delta.reasoning deltas on Kimi models (the “thinking” output on the OpenAI-compatible side), surfaced as ("reasoning", text). The 60-second cooldown on 429 in _complete_with_credential (_CRED_429_COOLDOWN_SEC = 65.0) is calibrated to NIM’s per-minute rate-limit window; the cooldown is followed by a jittered retry up to _CRED_429_MAX_RETRIES = 2.

4.6 Cohere

Cohere reaches the router through its OpenAI-compatibility endpoint (https://api.cohere.ai/compatibility/v1). Cohere’s native API uses a slightly different tool-use shape with explicit chat_history; the compatibility endpoint translates to OpenAI shape, so we do not need a native provider.

4.7 Local: Ollama, llama.cpp, vLLM, LM Studio

All four expose an OpenAI-compatible Chat Completions endpoint when configured to do so. Ollama defaults to http://localhost:11434/v1; llama.cpp’s server defaults to http://localhost:8080/v1; vLLM uses --api-key and an operator-chosen port; LM Studio defaults to http://localhost:1234/v1. The router treats all four as openai_compatible. The _LOCAL_BACKEND_NAMES frozenset at core/llm/router.py lists {"ollama", "local", "llama_cpp", "llamacpp", "lm_studio", "lmstudio"} for the requires_local_only=True enforcement path: when an agent or operator policy requires local inference (e.g. for a private payload), the router refuses to route to anything not on that list, with _select_route raising RuntimeError if no local backend is configured.

5. The splice points

This is the section the paper exists for. The lenient parser (Lenient Tool-Call Parsing) needs two facts to do its job correctly: the active model (so it can decide whether the model needs lenient handling at all) and the active tool catalog (so the fuzzy resolver can rewrite paraphrased tool names like create_markdown_widget to the canonical widget_create). Neither is naturally present at the system-prompt assembly step; both are present at the router. The router’s job at the splice points is to publish them.

The figure below sketches the end-to-end path; the splice points are the dashed annotations.

flowchart LR
  subgraph A[Agent loop]
    REQ[ChatRequest<br/>core/llm/types.py]
  end

  subgraph R[LLMRouter — core/llm/router.py]
    RES[resolve agent_id<br/>_select_route]
    SCRUB[_apply_scrub<br/>+ ContextVar publish]
    SEL[provider / pool select<br/>round_robin · least_loaded]
    DISP[_stream_with_credential<br/>_stream_with_failover]
    LP[lenient_parser.wrap<br/>if model_needs_lenient]
    ACC[stream_with_tools<br/>accumulator]
  end

  subgraph P[Provider adapters — core/llm/provider.py]
    direction TB
    PA[AnthropicProvider<br/>Messages API]
    POAI[OpenAIProvider]
    POC[OpenAICompatibleProvider<br/>NIM · Ollama · vLLM · OpenRouter · Gemini compat]
  end

  subgraph S[settings_bridge<br/>core/llm/settings_bridge.py]
    YAML[LLMSettings YAML]
    DB[onboarding settings DB]
    ENV[env vars]
  end

  REQ --> RES --> SCRUB --> SEL --> DISP --> LP --> ACC
  ENV -.priority 1.-> YAML
  YAML -.priority 2.-> RES
  DB -.priority 3.-> YAML
  SCRUB -. _ACTIVE_MODEL_FOR_HINT .-> SP[system_prompt assembler]
  SCRUB -. _ACTIVE_TOOL_CATALOG_FOR_HINT .-> SP
  DISP --> PA
  DISP --> POAI
  DISP --> POC
  PA -- SSE: content_block_delta · input_json_delta · thinking_delta --> LP
  POAI -- SSE: delta.content · delta.tool_calls --> LP
  POC -- SSE: delta.content · delta.tool_calls · delta.reasoning --> LP
  LP -- ("tool_call_delta", _lenient_fmt) --> ACC
  LP -- ("hallucinated_results_detected", str) --> ACC
  ACC --> OUT[unified event stream<br/>reasoning · content · tool_call_delta · usage · done]

5.1 _apply_scrub — ContextVar publication

The _apply_scrub method at core/llm/router.py lines 374–528 has two responsibilities. The first is the privacy scrubber (V3.1.6 territory; not this paper). The second is the splice. Lines 411–425:

try:
    from core.learning.system_prompt import (
        _ACTIVE_MODEL_FOR_HINT,
        _ACTIVE_TOOL_CATALOG_FOR_HINT,
    )
    if model_name:
        _ACTIVE_MODEL_FOR_HINT.set(model_name)
    catalog_names = tuple(
        t.get("name", "")
        for t in (request.tools or [])
        if t.get("name")
    )
    _ACTIVE_TOOL_CATALOG_FOR_HINT.set(catalog_names)
except Exception:
    pass

These two ContextVars are imported from core.learning.system_prompt. The system-prompt assembler reads them at prompt-build time and injects a tool-format hint listing the catalog names verbatim. The hint is what produces output like “Available tools: widget_create, fs.read, net.fetch, …. Emit calls in the OpenAI function-calling format. If you cannot, emit <tool_call>{...}</tool_call> and we will accept it.” The hint is per-model: a frontier OpenAI model gets the short version; a Kimi or DeepSeek model gets a longer version listing all four accepted formats.

Publishing via ContextVar rather than threading the model name through every call site is deliberate. The system-prompt assembler is called from the agent loop, not from the router; threading the values would have required adding model and tool_catalog parameters to roughly twenty intermediate functions. ContextVar is the standard async-safe channel for exactly this kind of cross-cutting concern; it scopes correctly across asyncio tasks and does not leak across requests. The cost is that the read site must handle the case where the variable was never set (prompt assembled outside a router-driven dispatch); the assembler does this with _ACTIVE_MODEL_FOR_HINT.get(default="") and falls back to a generic hint.

The try / except Exception: pass around the whole block protects the dispatch path from regressing if core.learning.system_prompt ever fails to import. The router will dispatch fine without the hint; the lenient parser still does its job; the system prompt just contains the generic version. The privacy scrubber, by contrast, is fail-closed (raise RuntimeError("refcontext scrubber unavailable; refusing to dispatch — fix and restart") at line 463). The two failure modes are intentionally different: scrub failure is a privacy-promise violation; hint failure is a quality-of-output regression.

5.2 _stream_with_credential and _stream_with_failover — lenient_wrap splice

Both streaming-execution paths apply the same wrapper. From _stream_with_credential (core/llm/router.py lines 1062–1084):

from .lenient_parser import (
    wrap as _lenient_wrap,
    model_needs_lenient as _needs_lenient,
)
import time as _time
_turn_idx = int(_time.monotonic() * 1000) % 99999
_catalog_names = [
    t.get("name", "")
    for t in (request.tools or [])
    if t.get("name")
]
_stream = provider.stream(request, model)
if _needs_lenient(model):
    _stream = _lenient_wrap(
        _stream, model=model, turn_idx=_turn_idx,
        tool_catalog=_catalog_names or None,
    )
async for pair in _stream:
    streamed_any = True
    ...
    yield pair

model_needs_lenient(model_name) is the gate at core/llm/lenient_parser.py line 87. It returns True for model families that emit non-canonical tool-call shapes — Kimi K2.x, DeepSeek V3, Llama 3.x via Ollama, Qwen 2.x, Yi-Large, Mistral on Together, and a small number of others — and False for OpenAI’s GPT family and Anthropic’s Claude family, which emit canonical shapes.

When the gate returns True, the provider’s raw async generator is wrapped. The wrapper consumes upstream pairs, runs the streaming state machine that detects the four lenient formats (Format A native, Format B XML-invoke, Format C JSON-in-sentinel, Format D special-tokens), and re-emits a normalised stream of OpenAI-shape tool_call_delta events. Each emitted delta carries a _lenient_fmt annotation ("native", "kimi_special", "xml_invoke", "xml_sentinel") — see Lenient Tool-Call Parsing for the format taxonomy. The router-side accumulator (stream_with_tools, core/llm/router.py lines 805–808) reads _lenient_fmt from each delta and stamps it onto the final ToolCall.fmt:

if d.get("_lenient_fmt"):
    slot["_lenient_fmt"] = d["_lenient_fmt"]
yield ("tool_call_delta", d)
...
tool_calls.append(ToolCall(
    id=slot["id"] or "",
    name=slot["name"],
    arguments="".join(slot["arguments_buf"]) or "{}",
    fmt=slot.get("_lenient_fmt", "native"),
))

The wrapper also synthesizes a ("hallucinated_results_detected", str) event when it spots a hallucinated <function_results> block (lenient-parser paper, V3-Q.3). The router forwards this verbatim:

elif kind == "hallucinated_results_detected":
    yield (kind, raw)

The downstream agent-loop listener consumes the event and queues a one-shot directive for the next turn’s system prompt. The router itself does not handle the event — its job is forwarding.

The splice is symmetric across credential-sourced and failover-sourced dispatch. _stream_with_failover (core/llm/router.py lines 1241–1259) applies the same _lenient_wrap to the same provider.stream(eff_request, model). We considered factoring this into a single helper and did not, because the two paths treat 429 differently (the credential path retries on the same provider with _CRED_429_COOLDOWN_SEC; the failover path moves to the next pool member with on_429_cooldown_seconds), and trying to share the streaming body without sharing the retry semantics produced more confusing code than the duplication.

5.3 Why this lives in the router

A reader could reasonably ask why the lenient-parser splice is at the router rather than at the provider. Three reasons.

First, the lenient parser is a property of the model, not the provider. Kimi K2.5 emits XML-invoke tool calls regardless of whether it is reached through NVIDIA NIM, Together, Fireworks, or a local llama.cpp serving the GGUF. Putting the wrapper inside OpenAICompatibleProvider would duplicate it across every provider class that might serve the model. Putting it at the router lets model_needs_lenient(model) be the single point of decision.

Second, the per-call tool catalog is naturally available at the router (it is part of the ChatRequest the router receives) and not naturally available at the provider (which receives the request but does not know which catalog names are in scope versus which are vestigial metadata). The router passes tool_catalog=_catalog_names into the wrapper precisely because it has the canonical view.

Third, the failover loop applies the wrapper after each retry to the same model on a different provider. If the wrapper lived in the provider, the failover loop would have to re-wrap on each iteration anyway — cleaner to express it once at the router.

6. Failover and pools

A pool is a named set of providers configured in YAML:

llm:
  pools:
    kimi_pool:
      members: [nvidia_primary, nvidia_secondary, together_main]
      strategy: round_robin
      on_429_cooldown_seconds: 60

PoolSpec and the parser are at core/llm/settings.py lines 57–63. Pool.select() (core/llm/pool.py) implements two strategies. round_robin walks the member list via itertools.cycle, skipping members the rate limiter reports as cooling. least_loaded picks the member with the smallest RateLimiter.load(name)[0] (in-flight count). When all members are cooling, both fall through to _least_bad_fallback, which picks the member coming off cooldown soonest — surfacing RuntimeError("all providers exhausted") mid-conversation is worse than waiting a few seconds.

_complete_with_failover and _stream_with_failover (core/llm/router.py lines 1171–1282) implement the failover loop. _attempt_sequence builds it: for a pool, pick a starting member via the strategy and append remaining members in order so a 429 on the first retries the others before giving up; for a single-provider route, return [provider_name]. On RateLimitError, the limiter is told to start a cooldown via self._limiter.on_429(spec.name, cooldown) and the loop continues. On any other exception, the loop re-raises immediately — failover is a 429-specific behavior, not fault-tolerance. Mid-stream 429s (if streamed_any: raise) are surfaced rather than retried, because the user has already seen partial output and a transparent retry would produce duplicate content.

The credential path (_complete_with_credential, _stream_with_credential) has its own retry loop with _CRED_429_COOLDOWN_SEC = 65.0 and _CRED_429_MAX_RETRIES = 2, followed by a jittered sleep: sleep_for = self._CRED_429_COOLDOWN_SEC + random.uniform(0, 30). The jitter is the load-bearing detail. Without it, multiple agents sharing the same credential and hitting 429 together would all wake at exactly cooldown+0 and hammer the provider in lockstep, producing a synchronized retry storm. The 30-second jitter window decorrelates them.

A circuit breaker per (provider_name, credential_label) from core.resilience.breaker.CircuitBreaker (failure_threshold=5, reset_timeout=30.0) sits one layer below the rate-limit retry. Five consecutive failures opens the breaker; while open, every call short-circuits to RuntimeError(f"provider {spec.name} circuit open; cooldown"). Keying on (provider, credential) rather than just provider is intentional: a user’s bad key should not blackhole everyone else’s against the same provider.

7. Per-agent routing

Memagen™ assigns providers to agents at the YAML layer:

llm:
  default: claude_bridge
  default_model: claude-opus-4-7
  agents:
    orchestrator: {provider: anthropic_main, model: claude-opus-4-7}
    dev:          {provider: kimi_pool}
    expert.security: {provider: openai_main, model: gpt-4.1}
    background:   {provider: nvidia_primary, model: moonshotai/kimi-k2.5, temperature: 0.3}

Lookup is dotted-hierarchical. LLMSettings.lookup("expert.security") (core/llm/settings.py lines 86–93) walks expert.security → expert → default and returns the first match. _label_candidates at core/llm/router.py lines 197–206 implements the same walk for the credential-store lookup path. The hierarchy lets users assign a default provider to a broad agent category (expert: {provider: openai_main}) and then specialize for a subagent (expert.security: {provider: anthropic_main, model: claude-opus-4-7}) without restating the parent.

The shared-key shortcut: when llm.shared_key: true, every agent uses the default route regardless of agents declarations. This is the path for the single-key BYOLLM flow where the user has typed one API key in the onboarding panel and does not care to assign different providers per agent.

7.1 settings_bridge — onboarding to YAML

The settings_bridge at core/llm/settings_bridge.py is the runtime translation from the onboarding flow’s committed answers into the YAML the router parses. The user clicks through onboarding; their choices land in the settings SQLite table as llm.provider_choice, llm.providers.chosen.api_key (encrypted at rest), and llm.providers.chosen.model. synthesize_from_settings(yaml_data) reads those values, looks up the user-facing name in _ONBOARDING_TO_YAML_NAME:

_ONBOARDING_TO_YAML_NAME: dict[str, str] = {
    "anthropic":    "anthropic_main",
    "openai":       "openai_main",
    "nvidia":       "nvidia_primary",
    "groq":         "groq_main",
    "together":     "together_main",
    "local_ollama": "local_ollama",
}

…and returns a new YAML-shaped dict with a synthesized llm.providers.<chosen> block, the corresponding base_url from _DEFAULT_BASE_URLS, and the API key written as api_key_literal (not api_key_env, because the value lives in the encrypted DB rather than in an environment variable).

Override priority (documented in the module docstring): env vars > explicit YAML > settings DB. If the YAML already declares the synthesized provider name AND that block references an env var that is set, the YAML wins. This lets a developer with ANTHROPIC_API_KEY exported run Memagen™ without the onboarding flow stepping on their environment, while letting an end-user who has never set an env var get a working router from the GUI alone.

The function is fail-open. Any exception reading the DB, any missing key, no admin user, malformed YAML — all return the input unchanged. This is the right default for a transformation that runs on every router construction; the router falls back to YAML defaults rather than refusing to start because the settings DB is locked.

7.2 CredentialStore lookup

A second per-agent lookup path runs against an optional CredentialStore (core/llm/credentials_store.py). When the store is present, every dispatch consults it first via _credential_for(agent). The walk is: exact agent label → dotted parents → MAIN_LABEL (the literal string "Main"). When core.settings.get_shared_credentials() is True, the background and secondary agents also fall back to Main — this lets a user with one paid key cover both agents while still picking a smaller model in the Background slot.

The credential store is the runtime channel for the GUI Credentials panel: the user types a key, picks a provider, picks a model, clicks save, and the next agent dispatch reads the row. set_credential_store(store) swaps the store at runtime and drops cached credential:<label> provider entries so the next call rebuilds against the fresh row.

8. The claude-bridge shim

The claude-bridge is a Memagen™ contribution worth naming. Anthropic’s Claude Code CLI ships with an OAuth-managed session — once the user runs claude login, the local CLI can call Claude without a separate API key. The claude-bridge is an OpenAI-compatible HTTP shim wrapping the Claude Code session and exposing it on a local port. The router treats it as a regular openai_compatible provider (base_url = "http://localhost:<port>/v1", api_key_literal = "" because the bridge does not require one).

The motivation is straightforward. A founder building Memagen™ on a tight budget does not want to pay for tokens twice — once through a Claude Code Max subscription and again through a separate Anthropic API key. The bridge lets the development build use the Claude Code session for orchestrator-tier reasoning while end-users of the same code path use their own Anthropic API key (or any other provider) without rewriting the integration. We expect end-users will not need it; we leave it in the Lite distribution because some advanced users have the same constraint.

The honest constraint: the bridge depends on the claude CLI’s session staying alive. If the session is invalidated upstream, every downstream agent dispatch fails until the user runs claude login again. The fallback is the standard Anthropic provider with api_key_env: ANTHROPIC_API_KEY; users who do not want the dependency configure that instead.

9. Streaming and event taxonomy

The unified streaming interface is AsyncIterator[tuple[str, Any]]. The taxonomy:

• ("reasoning", str) — thinking-mode delta. Sourced from delta.reasoning on the OpenAI-compatible side (Kimi K2.x, NIM-served reasoning models) and from content_block_delta of type thinking_delta on the Anthropic side.
• ("content", str) — answer-text delta. Sourced from delta.content on the OpenAI-compatible side and from content_block_delta of type text_delta on the Anthropic side.
• ("tool_call_delta", dict | str) — partial tool-call delta. Always emitted in OpenAI shape ({index, id, function: {name, arguments}}) regardless of upstream provider; the Anthropic provider translates input_json_delta events into this shape, and the lenient parser translates XML-invoke / JSON-in-sentinel / Kimi-special-token emissions into this shape. The optional _lenient_fmt field annotates which format the parser detected.
• ("usage", dict | str) — final token usage. Emitted at end of stream by providers that support it; the OpenAI-compatible path requests it explicitly via stream_options: {include_usage: true}. The router emits a placeholder {"prompt_tokens": -1, "completion_tokens": -1} if the upstream did not produce one, so the consumer can rely on the field’s presence.
• ("hallucinated_results_detected", str) — V3-Q.3 synthetic event from the lenient parser. The string payload is the hallucinated content (suppressed from visible output but preserved for the loop listener so it can be diagnosed).
• ("done", dict) — only emitted by stream_with_tools; carries {content, tool_calls, usage, reasoning} as the final aggregated payload after the consumer has seen all deltas.

The taxonomy is small on purpose. Adding event kinds is cheap; subtracting them is expensive. We have resisted requests to add per-event metadata channels (timestamps, costs, latencies) the consumer can derive from the events it already has.

10. Limitations

We list these because the alternative — claiming completeness we do not have — is dishonest.

Provider quirks change. Anthropic added thinking blocks in 2025; we updated. OpenAI is rolling out a new responses API (Stateful Responses); we have not yet adopted it because the current chat.completions path remains supported and is the workload we have tested. When OpenAI deprecates chat.completions, we will rewrite OpenAIProvider. Provider abstractions that try to be schema-stable across deprecations end up with a least-common-denominator problem; we do not solve that.

Tool-call shape variation across providers is the biggest source of provider-specific code in the codebase. Most of _iter_anthropic_sse exists because Anthropic’s streaming tool-use shape is genuinely different from OpenAI’s. Most of the lenient parser exists because non-frontier OpenAI-compatible providers each emit tool calls in their own way. We do not see a path to compressing this work; it is the job.

Failover is best-effort and non-magical. With a single provider and no pool, there is nothing to fail over to. The router returns RuntimeError("all providers exhausted") after credential-path retries are exhausted. We do not silently fall back to a different provider behind the user’s back — that would create a billing surprise.

No batched-call support. Anthropic’s Message Batches API (50% discount on async batched submissions) and OpenAI’s Batch API are both unsupported. The router’s interaction model is per-request streaming or per-request completion; mapping that onto a batch submission with poll-back semantics is a non-trivial change. On the future-work list but not scoped.

No cost-aware or latency-aware routing. The router picks the route declared in the YAML. It does not currently look at the cost of each provider in the current pool and prefer the cheaper one for the current task. Also on the future-work list.

No native Gemini provider yet. The Gemini compatibility endpoint covers the agent loop; the native function-calling shape with responseSchema and structured output does not. Adding GeminiProvider is on the roadmap.

No native Cohere provider with chat-history shape. The compatibility endpoint covers the agent loop; the native chat-history shape with connectors is not supported.

Provider-preference headers (OpenRouter). The current OpenAICompatibleProvider._headers does not pass through the extra field’s headers, so OpenRouter provider-preference routing requires editing the provider class. A small wiring task on the future-work list.

11. Future work

Listed by likelihood of being shipped in the next two releases.

Provider-extra header pass-through. Plumb ProviderSpec.extra.headers through OpenAICompatibleProvider._headers and AnthropicProvider._headers. Enables OpenRouter X-OR-Prefer-Provider, NIM custom routing headers, and per-org tagging.

Native Gemini provider. Implement GeminiProvider against generativelanguage.googleapis.com directly, with structured-output support and proper safety-filter event surfacing.

Cost-aware routing. Within a pool, prefer the member whose model has the lowest cost-per-token for the request’s expected work, biased toward members with the most cooldown headroom. Requires extending BudgetTracker (core/llm/cost.py) with a per-(provider, model) cost table and a per-request “expected tokens” estimate.

Latency-aware routing. Within a pool, prefer the member with the lowest p50 latency over a rolling window. Requires the rate limiter to record per-call latencies in addition to in-flight counts.

Provider-specific tool-call format auto-detection. We currently know the format-likelihood of each model family at code time (model_needs_lenient is a static lookup). A learned variant could observe the first few responses from a new model and infer the right format. The static table is fine for now; the learned variant becomes interesting when we ship a provider catalog the user can extend at runtime.

Batched-call API support. Wrap Anthropic Message Batches and OpenAI Batch into a separate BatchRouter that exposes submit_batch(requests) → batch_id and poll_batch(batch_id) → results. Different shape from the streaming router; do not try to unify.

ContextVar plumbing for cost and latency. The _apply_scrub ContextVar splice has worked well for model name and tool catalog. Cost and latency budgets could be published the same way for tools that want to make cost-aware decisions inside the agent loop.

12. References

1. Anthropic. (2024). Claude Messages API reference: tool use, content blocks, and streaming. https://docs.anthropic.com/claude/reference/messages_post
2. Anthropic. (2025). Extended thinking with Claude. https://docs.anthropic.com/claude/docs/extended-thinking
3. BerriAI. (2023). LiteLLM: Call 100+ LLM APIs using the OpenAI format. https://github.com/BerriAI/litellm
4. Chase, H. (2022). LangChain: Building applications with LLMs through composability. https://github.com/langchain-ai/langchain
5. Levitas, T. (2026). Memagen™: An open-core agent platform with lenient tool routing, capability-gated execution, and a compound-graph memory substrate. Memagen™ master paper, this series.
6. Levitas, T. (2026). Lenient Tool-Call Parsing: Bridging Weak-Tool-Call Models to Native Tool Catalogs. Memagen™ specialty paper, this series. https://memagen.com/papers/2026-05-08-lenient-parser
7. OpenAI. (2024). Function calling and structured outputs. https://platform.openai.com/docs/guides/function-calling
8. OpenAI. (2024). Streaming responses with the Chat Completions API. https://platform.openai.com/docs/api-reference/chat-streaming
9. OpenRouter Inc. (2024). OpenRouter API documentation. https://openrouter.ai/docs
10. NVIDIA. (2024). NVIDIA NIM: API catalog and OpenAI-compatible endpoints. https://build.nvidia.com
11. Google. (2024). Gemini API: OpenAI-compatibility endpoint. https://ai.google.dev/gemini-api/docs/openai
12. Cohere. (2024). Cohere chat API and OpenAI compatibility. https://docs.cohere.com/docs/compatibility-api
13. Ollama. (2024). OpenAI-compatible API server. https://github.com/ollama/ollama/blob/main/docs/openai.md
14. vLLM Project. (2024). vLLM OpenAI-compatible server. https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html

Source: https://github.com/tablevitas123/crackedclaw