84% fewer tokens on JSON tool results. 53% fewer tokens on tool-heavy requests. Sub-300ms semantic cache hits. Zero code changes.
Self-hosted LLM gateway and tier-routing proxy for Claude Code, Cursor, and Codex. Routes across Ollama, AWS Bedrock, OpenRouter, Databricks, Azure OpenAI, llama.cpp, and LM Studio with prompt caching, MCP tools, and 60-80% cost savings.
2026-11-15
ดาว
—
เครื่องมือ
—
อัปไทม์
—
Numbers from the bundled benchmark against LiteLLM on identical free local backends — run it yourself: node benchmark-tier-routing.js. It doubles as a 19-scenario routing regression harness (currently 12/12 correctness checks), and MODE=routing runs a routing-only head-to-head that judges both proxies on the same acceptable-tier sets — including LiteLLM's Auto Router v2. How it works →
Agentic benchmark (Terminal-Bench core, 80 tasks, Oct 2026): Lynkr 42/80 for $1.54 vs openrouter/auto 44/80 for $4.72 on the same VM and evening — tied on accuracy within single-run noise (Lynkr's three runs: 39, 39, 42) at ~30% of the cost; vLLM Semantic Router on the same models/hosts: 25/80 (34 agent timeouts from unbounded upstream stalls and per-turn CPU classification). Full comparison, fairness notes and what it changed in Lynkr →
Third-party benchmark: on RouterArena (ICLR 2026, 8,400 queries) Lynkr's routing scores 67.65 arena / 68.41% accuracy at $0.29 per 1K queries with 92.38 robustness — above GPT-5's built-in router and NotDiamond at a fraction of their cost. Methodology & caveats →
🚀 New: Wrap Mode for AI Coding Tools
Use Lynkr's routing with your AI coding assistant — maximize your subscription value:
npm install -g lynkr
# Claude Code Pro/Max
lynkr wrap claude
Wrapping gives you:
✅ Tier routing (send simple tasks to free Ollama, complex to your subscription/API)
✅ Sticky sessions: one routing decision per conversation via content fingerprinting, with automatic escalation when the task outgrows the model
✅ TOON/RTK compression (84% token reduction on large JSON tool outputs)
✅ Your subscription is used only where it counts — simple requests route to your own local/API models, so subscription capacity goes to the work that needs it
✅ Works with OAuth (Claude, Copilot, Cursor) or API keys (Aider, Codex)
It asks four questions — usage mode (Claude Pro/Max via wrap, or direct API keys), tier picks for SIMPLE/MEDIUM/COMPLEX/REASONING across the 14 supported providers, credentials for what you chose, and a few routing-intelligence knobs — then writes a fully-populated .env with sensible production defaults for everything else (caching, compression, policy budgets, MCP sandbox, rate limiting).
✅ 84% compression on large JSON tool results (grep, file reads, test output)
✅ Semantic cache serves repeated queries in under 300ms with 0 tokens billed
✅ Automatic tier routing — simple questions go to cheap models, complex ones escalate; sessions stick to one model until the task genuinely outgrows it
✅ A closed learning loop — every outcome trains a kNN router and bandit, and tier thresholds re-calibrate nightly from your own traffic
✅ Route through your company's infrastructure (Databricks, Azure, Bedrock)
✅ Zero code changes — just change one environment variable
Supported Providers
Provider
Type
Example Models
Cost
Ollama
Local
qwen2.5-coder, deepseek-coder, llama3
Free
llama.cpp
Local
Any GGUF model
Free
LM Studio
Local
Local models with GUI
Free
OpenRouter
Cloud
GPT-4o, Claude 3.5, Llama 3, Gemini
$
OrcaRouter
Cloud
Chat, reasoning, image, video (one endpoint, live catalog)
$ (zero-markup)
AWS Bedrock
Cloud
Claude, Llama, Mistral, Titan
$$
Databricks
Cloud
Claude Sonnet 4.5, Opus 4.6
$$$
Azure OpenAI
Cloud
GPT-4o, o1, o3
$$$
Azure Anthropic
Cloud
Claude Sonnet, Opus
$$$
OpenAI
Cloud
GPT-4o, o3-mini
$$$
Atlas Cloud
Cloud
Qwen, DeepSeek, and other OpenAI-compatible models
$-$$$
Cheaper Inference
Cloud
GPT, Claude, Gemini, DeepSeek, and other OpenAI-compatible models
$-$$$
AtmoRouter
Cloud
Claude, GPT, Gemini, GLM, Kimi, Qwen, DeepSeek, and other OpenAI-compatible models
$-$$
DeepSeek
Cloud
DeepSeek R1, Reasoner
$
Z.ai
Cloud
GLM-4.7, GLM-4.5-Air
$
Moonshot AI
Cloud
Kimi K2.6, Kimi K3
$
Baidu Qianfan
Cloud
ERNIE 4.5 Turbo, ERNIE X1.1
$ (unverified — not yet probed against a live key)
Fireworks AI
Cloud
Kimi K2, GLM-5, DeepSeek V3 (serverless)
$ (unverified — not yet probed against a live key)
4 local providers for 100% offline, free usage. 14+ cloud providers for scale.
Advanced: Tier Routing (Save Even More)
Route different request types to different models automatically:
# .env file
MODEL_PROVIDER=ollama
FALLBACK_ENABLED=false
# Use small/fast models for simple tasks
TIER_SIMPLE=ollama:qwen2.5:3b
# Use medium models for normal coding
TIER_MEDIUM=ollama:qwen2.5:7b
# Use powerful models for complex architecture
TIER_COMPLEX=ollama:deepseek-r1:14b
TIER_REASONING=ollama:deepseek-r1:14b
# Optional: Limits (remove for unlimited) for long conversations
POLICY_MAX_STEPS=50
POLICY_MAX_TOOL_CALLS=100
Lynkr analyzes each request and routes it to the appropriate tier. Simple questions use fast models. Complex refactoring uses powerful models. The scorer combines anchor-embedding classification (WS7, payload-invariant) with the Jev routing judge (an optional typed second opinion over the four tiers, enabled by TYPESAFE_API_KEY; it replaced the earlier local qwen2.5:3b classifier, so no local Ollama is required) that catches topic-vs-difficulty confounding — list the exports from this file correctly routes to MEDIUM instead of getting boosted to REASONING by "technical vocabulary" alone.
Result: 70-90% of requests use cheaper/faster models. Only hard problems hit expensive models. 15× reduction in expensive-tier over-routing vs the anchor-only baseline on the 381-prompt eval set (0.6% vs ~15%).
Tier configuration is strictly authoritative — bandit exploration is constrained to the models you've listed in TIER_*, and multi-turn conversations score with a recency-weighted sliding window so context isn't lost on short follow-ups. Conversations get a content-fingerprint session id (clients like Claude Code send none), the decision pins for the session, and a guarded escape ladder (risk keywords, force phrases, score drift, context overflow) re-escalates the moment a task outgrows its model. Full pipeline: docs/routing-intelligence.md · intent scorer: docs/intent-window-routing.md · verify any change: docs/benchmarking.md.
Common Issues & Fixes
Issue
Solution
"Service temporarily overloaded"
Ollama model too large for RAM. Use smaller model or increase --max-old-space-size
"Route not found: HEAD /"
Ignore - harmless health check from Claude Code
"Hallucinated tool calls"
Normal - Lynkr automatically filters invalid tools
"Safe Command DSL blocked"
Add POLICY_SAFE_COMMANDS_ENABLED=false to .env
"spawn graphify ENOENT"
Optional feature. Set CODE_GRAPH_ENABLED=false in .env (see Advanced Features section for installation)
Slow first request (20+ sec)
Ollama loading model into memory. Add OLLAMA_KEEP_ALIVE=30m in Ollama config
No response after N turns
Remove POLICY_MAX_STEPS and POLICY_MAX_TOOL_CALLS from .env (unlimited by default in v9.3.0+)
Advanced Features
Real-time SSE streaming (all tiers)
Responses stream token-by-token instead of arriving all at once — including
through the tier router. Two mechanisms, both on by default:
Native passthrough: when the upstream already speaks Anthropic SSE
(Anthropic endpoints, Z.AI, Ollama v0.14+), Lynkr pipes the bytes straight
through with backpressure. Kill switch: LYNKR_NATIVE_PASSTHROUGH=false.
Cross-format transform: OpenAI-format upstreams (OpenAI, Azure OpenAI,
OpenRouter, Databricks, llama.cpp, LM Studio) are reshaped into Anthropic
events in flight — including reassembling tool-call argument fragments into
clean tool_use blocks. Kill switch: LYNKR_STREAM_TRANSFORM=false.
The LYNKR_VISIBLE_ROUTING badge streams too (injected as the first content
block), and telemetry is recorded on stream close with real token counts.
Fallback safety: if an upstream fails before the first byte, the request
falls back to the buffered path automatically.
# Ollama thinking models (MiniMax): streaming skips the <think>-leak repair.
# If you see raw <think> text in responses, buffer that provider instead:
LYNKR_OLLAMA_BUFFER_RESPONSES=true # default true; false = stream Ollama
Token Optimization (60-80% savings)
# Enable all optimizations
PROMPT_CACHE_ENABLED=true
SEMANTIC_CACHE_ENABLED=true
Always-on (no config): RTK tool-result compression
(test/git/grep/lint/build/JSON output), MCP tool dedup (drops
built-in WebSearch/WebFetch when an Exa/Tavily MCP tool is present), and
request bypass (Claude CLI Warmup / title-extraction calls are answered
locally, never hitting a provider).
Optional terse-output mode to cut output tokens:
CAVEMAN_ENABLED=true # off by default — nudges the model to be concise
CAVEMAN_LEVEL=lite # lite | full | ultra
Built-in dashboard
Open http://localhost:8081/dashboard while Lynkr is running. No setup, reads
the local telemetry store:
Spend & savings — actual cost vs what the same traffic would have cost
on the flagship model (the counterfactual no pass-through gateway can show,
since they neither pick your model nor see local-model traffic)
Tier mix — daily request breakdown across SIMPLE/MEDIUM/COMPLEX/REASONING
Routing accuracy — over-/under-provisioned request counts, a self-audit
of the tier router's decisions
Request logs — filterable by provider, tier, and errors, with latency,
tokens, and cost per request; every session id links to a drill-down
Provider health — configured providers, credential warnings, circuit
breaker states
Insights — analyzer findings ranked by past overspend (auditable
against spend already recorded, never projected savings): dead tool
schemas paying rent, harness side-traffic served on expensive tiers,
weak prompt-cache hit ratios, and Wilson-bounded tier-downsize verdicts.
Each finding ships its evidence rows, the fix, the caveat, and how the
figure was computed. "Not measured yet" is rendered distinctly from
zero — an un-run scan is not an all-clear.
Explore — a Metric × Dimension (× Stack) pivot over routing telemetry
(spend/tokens/requests/sessions by provider, model, tier, request type,
day…) with preset views, period-over-period deltas, and CSV export
Session drill-down — per-model cost mix and an input-context-per-call
chart where distillation and cache behavior are directly visible
Cache receipts — projected switch/hold economics from cache-aware
routing next to the measured companion (dollars saved by cache reads
actually served)
JSON APIs behind it
(/dashboard/api/overview|usage|routing|logs|recommendations|analytics|sessions/:id|statusline)
if you want the raw numbers.
Status line for Claude Code
One line after every turn — routed tier, served model, today's spend, and
cache re-read share — with zero token cost (out-of-band, never enters model
context). Add to ~/.claude/settings.json:
◆ Opus 4.8 · MEDIUM → minimax-m3:cloud (ollama) · today $0.06 · cache 67%
Always prints exactly one line and always exits 0 — a down proxy can never
break the harness. Set LYNKR_PORT if Lynkr isn't on 8081.
Cost tracking & model pricing
Per-request cost is computed from a model-pricing registry (LiteLLM → models.dev,
cached 24h) and recorded in telemetry. Models the registry doesn't know record
cost_usd=null (logged once) rather than a fabricated price. Pin prices for
unknown models:
# Per-1M-token USD prices, JSON keyed by model name
MODEL_PRICE_OVERRIDES={"my-model":{"input":0.5,"output":1.5}}
Head-to-head against LiteLLM on the same backends (Ollama minimax-m2.5, Moonshot, Azure OpenAI), 9 scenarios across 4 feature categories. Apples-to-apples comparison is Lynkr vs LiteLLM billed tokens on the same scenario. Run with node benchmark-tier-routing.js.
Runs: token benchmarks — June 5, 2026 (Lynkr v9.3.2 · LiteLLM v1.87.1). Tier routing head-to-head — re-run July 19, 2026 (Lynkr working tree w/ Phase-6 classifier + config B · LiteLLM v1.94.0.dev1 Auto Router v2). macOS, Apple Silicon.
Token reduction (vs LiteLLM, same model & prompt)
Mechanism
Lynkr
LiteLLM
Result
TOON compression (60-item grep JSON)
427 tokens · $0.009
3,458 tokens · $0.018
87.6% fewer tokens, 50% cheaper
Lynkr binary-compresses large JSON tool results (TOON) in-process, with no added latency.
Semantic cache
Tokens billed
Response time
First call (cold)
2,857
1,891ms
Second call — paraphrased, cache hit
0 (served from cache)
171ms (11× faster)
Near-identical prompts return cached responses in 171ms. Zero model tokens billed on a cache hit.
Tier routing — vs LiteLLM Auto Router v2 (re-run July 19, 2026)
LiteLLM v1.94 shipped a native complexity router (auto_router/complexity_router) with the same four tier names Lynkr uses. Head-to-head on the same backends with identical prompts, both proxies live, both judged against the same acceptable-tier sets (11 routing scenarios, MODE=routing node benchmark-tier-routing.js, LiteLLM config in litellm-autorouter-v2.yaml):
Anchor-embedding classifier over 4 difficulty classes + FRONTIER_MIN_SIM floor
Jev routing judge (optional, TYPESAFE_API_KEY) reconciles borderline anchor calls — no local Ollama required; the earlier qwen2.5:3b classifier is kept dark
Envelope invariance — scores cleaned user text only, ignoring tool schemas / history / system-reminders
LiteLLM's router reads the raw last message with no envelope-stripping, no verify-then-escalate cascade, no risk classifier, no agentic detection — its fallbacks trigger only on HTTP errors, never on a bad answer.
Fairness notes: the 11 scenarios derive from Lynkr's own regression suite, so Lynkr has home-field advantage — the transferable finding is the direction of LiteLLM's failures (systematic under-routing to cheap on hard prompts), not the exact scores. Lynkr's top tiers used Claude Opus 4.8 / z.ai GLM-5.2; LiteLLM was given the identical tier targets.
Cost projection (100,000 requests/month, same backend)
Direct cost (raw meter):
Monthly cost
vs LiteLLM
LiteLLM (TOON tool-heavy scenario)
~$818
baseline
Lynkr (TOON tool-heavy)
~$409
~50% cheaper via token optimization
Effective cost per correct-tier answer (July 19 head-to-head, all 11 routing scenarios):
Direct $/mo
Effective $/mo if you re-issue misroutes at correct tier
Lynkr
~$5.50 (spends where it should, doesn't where it shouldn't)
~$5.50
LiteLLM v2 heuristic
$0 (under-routes everything to ollama)
~$89 (7 misroutes need re-issue on Moonshot/COMPLEX)
The story isn't "Lynkr always spends less." It's Lynkr spends where it should and doesn't where it shouldn't. LiteLLM's zero-dollar cost on complex prompts is bought with wrong-tier answers to hard work — a 7B model answering "prove this rate limiter is fair under concurrent refill" is silent quality collapse, not a savings.
✅ evidence-backed findings (dead tool schemas, side-traffic, cache weak spots, Wilson-proven downsizes) ranked by past overspend
❌
❌
⚠️ alerts, no findings
Long-term memory
✅ SQLite, per-session
❌
❌
❌
MCP integration
✅
❌
❌
❌
Self-hosted
✅ Node.js only
✅ Python stack
❌ SaaS
✅ Docker
Dependencies
Node.js 20+
Python, Prisma, PostgreSQL
None
Docker, Python
Lynkr's edge: Purpose-built for AI coding tools. Compresses tokens before they reach the model — not just after. Zero-config for Claude Code, Cursor, and Codex. Installs in one command.
Self-hosted LLM gateway and tier-routing proxy for Claude Code, Cursor, and Codex. Routes across Ollama, AWS Bedrock, OpenRouter, Databricks, Azure OpenAI, llama.cpp, and LM Studio with prompt caching
Self-hosted LLM gateway and tier-routing proxy for Claude Code, Cursor, and Codex. Routes across Ollama, AWS Bedrock, OpenRouter, Databricks, Azure OpenAI, llama.cpp, and LM Studio with prompt caching, MCP tools, and 60-80% cost savings.
ตรวจสอบล่าสุดเมื่อ 2026-11-15 Self-hosted LLM gateway and tier-routing proxy for Claude Code, Cursor, and Codex. Routes across Ollama, AWS Bedrock, OpenRouter, Databricks, Azure OpenAI, llama.cpp, and LM Studio with prompt caching, MCP tools, and 60-80% cost savings.