Llm Routing

• Why install this
• Quick Start
• Works With
• How It Works
• Features
• Reference
• Savings
• Trust and Security
• Documentation
• Enterprise, Contributing, License

Why install this

You are on a Claude Pro or Max plan, have not spent a cent beyond it, and still hit the five-hour limit — not because you asked too much, but because every prompt went to the premium model: "what does this error mean", "reformat this JSON", "is the service up" drew the same quota as the question that needed it.

Why a proxy cannot do this

Most routers here are a proxy: your agent forwards requests using your API keys. That has a hard limit — a proxy cannot intercept a session authenticated by a subscription, because there is no key to forward.

Honesty as a feature

llm-router status reports verified and unverified savings separately, each with its own n — not a blended percentage. This project does not publish a general savings percentage; any number depends on your own workload. See Savings below.

Quick Start

pip install llm-routing        # installs the `llm-router` command
llm-router install             # wire up Claude Code (default host)
llm-router doctor               # check provider connectivity and setup
llm-router status               # verify it's routing — est. savings vs baseline

Works with no API keys on a Claude Pro/Max subscription — routing goes through MCP tools and local models. Adding provider keys widens the pool; nothing requires them. See guide/PROVIDERS.md. To install into another host, see Works With. The daily KPI job is installed only with consent (install --schedule-daily, or Y when asked; --no-schedule skips); uninstall removes it.

Python agents route in-process, no server: from llm_router import Router; r = Router(budget_usd=0.50).run("...") (await .arun(...) in async code; BudgetExceeded when the cap is spent).

Works With

llm-router install --host all installs or prints every host config in one pass. Full per-host detail, including what each host genuinely cannot do: guide/HOST_SUPPORT_MATRIX.md.

How It Works

Hooks intercept the prompt before your coding tool's own model sees it. A free regex heuristic classifies it instantly, scoring about half of real prompts with confidence — 788 of 1,571 measured (scripts/measure_low_signal_rate.py, run 2026-09-23); the rest fall back to a default. A routable prompt gets a draft from a free/local model first, then walks up the chain toward paid models only if needed.

The draft is advisory — handed to Claude as an unverified hint, not a turn replacement, unless you opt in to zero-Claude mode. Enforcement acts at the tool-call level: smart (default) holds selected tool calls until the prompt is routed; off disables that. Full modes and per-host overrides: guide/GETTING_STARTED.md.

Features

• A prompt with a secret goes to a local model or nowhere. When a routed call (the llm_* text tools, the gateway, streaming) contains an API key, token or private key, llm-router sends it only to Ollama models on a loopback daemon — direct, not through an HTTP proxy, with no semantic cache, judge queue or cloud fallback — or refuses it with the secret's category and how to proceed (cloud: prefix to override one request). Placeholders such as AKIA...EXAMPLE or postgres:postgres@ are not refused. Not covered: the Claude Code proxy and hooks (logs only), agent delegation, and what a host already sent to its own model. Details and limits.
• Automatic fallback with circuit breakers. A provider that fails or rate-limits is skipped, not retried into the ground.
• You can see it working. A status line, terminal title and OS notification show the last model routed, savings and health.
• Session-end summary. Savings vs baseline, tier mix, per-provider cost, latency p50/p95/p99 and top routes.
• Media and pipelines too. llm_image / llm_video / llm_audio, and llm_orchestrate for multi-step research.

Reference

The default MCP surface shows 12 front-door tools; an opt-out shows everything registered (see docs/configuration.md for user settings).

llm-router gain today           # savings so far today
llm-router okf status           # knowledge base freshness
llm-router sessions status      # per-session summary

Savings

Savings figures are a counterfactual — what the same tokens would have cost at API list price, against what was actually spent — not money saved on a flat subscription. llm-router status and llm-router savings-report split verified from unverified savings, each with its own n. Methodology, assumptions and limitations: docs/MEASUREMENT.md.

Trust and Security

llm-router runs entirely on your machine — no hosted proxy, nothing sent to an llm-router service, no account required. A grounding check (src/llm_router/grounding.py) discards any draft citing a file or function absent from its context and the indexed repo, falling through to Claude.

LLM_ROUTER_DIRECT_EXECUTION is default on — a local model gets write_file, edit_file and run_command under a confined, allowlisted path, and none of that stops targeted deletes, git push --force, or exfiltration. Shell redirects and copies that write outside the project root are refused by a static tripwire, not a sandbox; an OS sandbox is 16.1. A command the tripwire cannot judge (an interpreter one-liner, sudo, a $(...)) returns needs_approval, but approval is not wired anywhere yet, so both command paths treat it as refused and do not run it. Turn it off with LLM_ROUTER_DIRECT_EXECUTION=false. Full analysis: SECURITY.md.

Self-audits are published in audit/ and docs/repo_goals/AUDIT-2026-09-25.md (13 met, 6 partial, 1 not met), including negative results.

Ground Truth accumulation (opt-in) builds a corpus of replayable routing decisions instead of unlinked telemetry; off by default since it's the only part that writes prompt text to disk. Details: guide/GROUND_TRUTH.md.

Documentation

Full index: guide/README.md

llm-router is scored on the RouterArena benchmark — full split, 8,400 queries, graded locally with this repo's harness, not yet independently verified. Current Arena Score: 72.35. Full methodology and negative results (including a proxy split that misled tuning by 4.25 points): docs/ROUTERARENA.md.

Enterprise, Contributing, License

llm-router is built for individual developers and small teams: local cost savings, zero ops overhead, no hosted anything. For team-wide policy enforcement, audit export, SSO or per-org budgets, see Chuzom — a separate, sibling product.

Contributions welcome — see CONTRIBUTING.md.

git clone https://github.com/ypollak2/llm-router.git
cd llm-router
uv sync --extra dev
uv run pytest tests/ -q         # Run tests (10,000+)
uv run ruff check src/ tests/   # Lint

MIT License. Issues · Discussions · PyPI · Changelog