CascadeGateway

CascadeGateway ⚡

Hardware-Adaptive Local LLM & Cloud Cascading Architecture

CascadeGateway is an intelligent, hardware-adaptive cascading proxy that routes AI queries between local GPUs (via Ollama at $0 token cost) and frontier cloud models (Google Gemini, OpenAI, etc.).

It automatically detects your GPU hardware and physical VRAM on startup—whether running a flagship RTX 5090 (32GB), RTX 4090 (24GB), mainstream RTX 3080 (10GB), or Apple Silicon—and dynamically selects and sizes the optimal models without requiring manual reconfiguration.

flowchart TD
    Client["AI Client: Cursor / Continue / Antigravity"] --> Gateway["CascadeGateway :8000/v1"]

    Gateway --> Phase1{"Phase 1: Structural Guard"}
    Phase1 -->|"Context > 32k or Tools"| CloudFallback["Cloud Fallback: Gemini"]
    Phase1 -->|"Passed"| Phase2{"Phase 2: Lexical Automaton <1ms"}

    Phase2 -->|"Architecture / Reasoning"| LocalArchitect["Local Architect: DeepSeek-R1 / Gemma 4 (RTX 5090)"]
    Phase2 -->|"Coding / Tests / Routine"| LocalBuilder["Local Builder: Qwen 2.5 Coder 32B (RTX 5090)"]

    LocalArchitect --> Phase3{"Phase 3: 3-Token Lookahead"}
    LocalBuilder --> Phase3
    Phase3 -->|"Stalled / Timeout"| CloudFallback
    Phase3 -->|"Healthy Stream"| LoopBreaker["Sliding-Window Loop Breaker"]
    LoopBreaker --> ClientStream["SSE Stream to Client"]

    CloudFallback --> ClientStream

    VerifyRoute["Asymmetric Verification: /verify"] --> LocalDraft["Local Generator: RTX 5090"]
    LocalDraft --> CloudCritic["Cloud Critic: Gemini Flash"]
    CloudCritic --> ClientStream

    Gateway -. Telemetry .-> HUD["🖥️ Always-On-Top Desktop HUD"]

🚀 Key Features

• 🖥️ Always-On-Top Desktop HUD: Sleek, frameless, draggable obsidian overlay (#0b0f19) that sits above your IDEs next to the clock or anywhere on screen. Displays live routing decisions (🟢 Local Builder, 🟣 Local Architect, 🔵 Asymmetric Verify, 🟠 Gemini Cloud), real-time VRAM gauge (GB, %, °C, W), decision latency (sub-5ms), prompt snippet, dollar savings counter, pin toggle, compact mini-mode, and 1-click mode switcher buttons.
• 🧠 Local GPU Architectural Reasoning: True multi-model orchestration on local silicon. System design, race condition, thread safety, and algorithmic queries are dispatched directly to the local reasoning model (deepseek-r1:14b or gemma4:26b) on the RTX 5090 rather than escaping to cloud tokens. Routine code generation is routed to qwen2.5-coder:32b.
• Hardware-Adaptive VRAM Sizing: Automatically profiles your GPU (nvidia-smi / Metal / CPU) and assigns the optimal model tier.
• Autonomous $0 Offload: Resolves 85–95% of routine coding, unit tests, refactoring, and QA directly on local VRAM.
• Dynamic Token Biasing Engine:
  • Adaptive Mode: Automatically increases local GPU bias as cloud token budgets get consumed.
  • Manual Slider: Fine-tune local preference from 0% (Quality First) to 100% (Maximum Local Offload).
  • Strict Local: 100% execution on local VRAM with automatic cloud overflow only if context exceeds physical limit.
• 🧠 Multi-Task Workflow Modes & Interactive Review Gate:
  • 🧠 Architect & Builder: Reasoning model drafts architecture, data contracts, and edge cases → Pauses at a Human Review Gate → You approve or refine → Builder synthesizes production code.
  • 🚀 Solo Sprint: Instant local coding via qwen2.5-coder:32b for quick functions, tests, and scripts.
  • 🌐 Deep Context: Gemini 2.5 Flash ingests massive repository files (1M context) → modular execution on local 5090.
  • 🔬 Math & Algo Proof: Deep Chain-of-Thought formal verification for cryptography and concurrency algorithms.
  • 🛡️ Asymmetric Verification (/verify): High-speed local draft synthesis on RTX 5090 ($0) combined with an anonymous strict cloud auditor (Gemini) providing production-grade critique and code enhancement without leaking full history.
• 🛡️ Sub-5ms Intelligent Routing & 3-Token Lookahead Failover: Three-phase classification pipeline: Phase 1 Structural & VRAM Budget Validation (<1ms, prevents swapping to system RAM), Phase 2 Single-Pass Lexical Scan (<1ms), and Phase 3 Resilient Streaming with an in-memory 3-token lookahead buffer that transparently replays stalled local requests to Gemini Cloud without dropping client connections or throwing IDE error popups.
• 🔁 Anti-Hallucination Sliding-Window Loop Breaker: In-flight ring buffer tracking token emission sequences (lengths 2, 3, 4 repeated ≥ 4×), terminating runaway generative loops immediately.
• ⚡ 1-Click IDE Auto-Configuration: Automated zero-friction setup endpoint (/api/ide/auto-config) and UI button for VSCode Continue (~/.continue/config.json) and Cursor.
• 💰 Zero-Surprise Dollar Savings Telemetry: Hardware-native token accounting (eval_count) calculating real-time dollar savings based on commercial frontier rates ($3.00/1M tokens).
• ⚡ Persistent GPU VRAM Residency & Zero-Cold-Start: Models remain permanently resident in GPU VRAM (keep_alive: -1) across idle periods and tasks, ensuring instant zero-delay responses. Includes manual 1-Click "Pause GPU" (instant VRAM purge for AAA gaming) and "Warm GPU" (preloads weights into VRAM).
• Native Model Context Protocol (MCP): Exposes a high-performance Streamable HTTP and Stdio MCP endpoint for Google Antigravity and Claude Desktop.
• 🎛️ Dynamic Model Selection: Select any installed local model for both the Architect role (deepseek-r1:14b, gemma4:26b, etc.) and the Builder role (qwen2.5-coder:32b, etc.) directly from the Web UI toolbar or Windows Tray submenus. Automatically detects newly pulled models from Ollama without restarting the gateway.
• Zero-Window Background Tray App: Runs silently in the system tray, boots with Windows/Linux, and includes single-instance mutex protection.
• 🎮 1-Click Pause & Free GPU (Instant VRAM Purge): Evicts loaded models from VRAM in <1s via a dedicated button on the Web UI, Windows Tray, or POST /api/models/unload. Frees 20–30+ GB of VRAM immediately for AAA gaming, Blender, or video editing without terminating the server. Models reload automatically on demand when coding.
• Live Hardware Telemetry: In-browser control center showing real-time VRAM allocation, GPU power draw (W), temperature (°C), lifetime token savings, and an interactive prompt runner.
• Standard OpenAI-Compatible API: Seamless drop-in replacement (/v1/chat/completions) for Cursor, VSCode (Continue.dev), Aider, Claude Dev, and custom scripts.

🧠 The Interactive Review Gate

sequenceDiagram
    autonumber
    actor Dev as You (Developer)
    participant Arch as 🧠 Architect - DeepSeek-R1 / Gemma 4
    participant Gate as 🛑 Review Gate - You in the Loop
    participant Build as ⚡ Builder - Qwen 2.5 Coder 32B

    Dev->>Arch: "Add rate-limiting and burst protection to API routes"
    Note over Arch: Deep CoT: Identifies race conditions,<br/>evaluates Redis vs in-memory,<br/>drafts interface contracts.
    Arch->>Gate: Presents Architectural Blueprint + Edge Cases
    Note over Gate: PAUSE: No code written yet.<br/>You review the proposed interfaces & strategy.

    alt If you want adjustments
        Dev->>Gate: "Use Redis, and add IPv6 CIDR subnet matching"
        Gate->>Arch: Quick amendment - 100 tokens
        Arch->>Gate: Updated spec
    end

    Dev->>Gate: "Approve & Build"
    Gate->>Build: Sends final structured specification
    Note over Build: Zero ambiguity.<br/>High-speed code synthesis - 70 t/s.
    Build-->>Dev: Delivers complete implementation + unit tests - $0 Cost

🏛️ Hybrid Architect-Builder Execution (Google Antigravity & CascadeGateway)

When running agentic coding assistants like Google Antigravity on a Pro plan, repeatedly shipping 20k to 50k tokens of repo file dumps, compiler logs, and multi-turn message history quickly exhausts 5-hour quota refill blocks.

CascadeGateway implements the native Hybrid Architect-Builder Pattern:

flowchart TD
    subgraph Cloud ["☁️ Cloud Tier (Pro Plan Quota)"]
        Architect["Cloud Architect: Gemini Pro / Claude<br/>• High-level system decomposition<br/>• Milestone roadmap planning<br/>• Concise diff reviews"]
    end

    subgraph Gateway ["⚡ CascadeGateway :8000/v1"]
        Router["Context Pruning & Intent Router"]
        Clamp["Terminal Clamping (&gt;60 lines &rarr; 15 head + 30 tail)"]
        Dedup["Repeat File Buffer Deduplication"]
    end

    subgraph Local ["🖥️ Local Workstation Tier (NVIDIA RTX 5090 - 32GB VRAM)"]
        Builder["Local Builder (vLLM / Ollama): Qwen 2.5 Coder 32B<br/>• Code generation, unit tests, refactoring<br/>• Iterative bug fixes & lint diagnostics<br/>• $0 Cloud Token Cost"]
        LocalMCP["Local MCP Tools<br/>• AST Symbol Lookup (@llmtooling/code-search-mcp)<br/>• Targeted File Slicing"]
    end

    Architect -->|"Milestone Specifications"| Router
    Router --> Clamp
    Clamp --> Dedup
    Dedup -->|"Iterative Coding Tasks"| Builder
    Builder <-->|"Precise Symbol Queries"| LocalMCP
    Dedup -. Escalation on repeated test failures .-> Architect

📉 Token Efficiency & Telemetry Audit Benchmark

Benchmarked using python -m cascadegateway.audit --compare baseline.jsonl optimized.jsonl:

🚀 Cross-Platform Local Engine Runners

• Windows Native (Zero-Window Flash):

  powershell -File scripts\start_local_engine.ps1
  

  Auto-detects Ollama or vLLM, warms up qwen2.5-coder:32b in 32GB VRAM with keep_alive: -1.

• Linux / WSL (vLLM with Automatic Prefix Caching):

  ./scripts/start_local_vllm.sh
  

  Launches vllm with --enable-prefix-caching, --enable-chunked-prefill, and --tool-call-parser hermes.

• Antigravity SDK & Standalone Orchestration:

  python scripts/run_architect_builder.py --task "Draft and build target module"
  

📊 VRAM Hardware Sizing Matrix

⚡ Quick Start

Prerequisites

1. Ollama installed and running on your machine.
2. Python 3.10+ installed.
3. (Optional) A free API key from Google AI Studio for cloud fallback.

Windows (One-Click)

git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
scripts\setup.bat

Creates .venv, installs dependencies, auto-generates your desktop shortcut, and places an auto-start shortcut in your Windows Startup menu.

Linux / macOS

git clone https://github.com/jadberro/CascadeGateway.git
cd CascadeGateway
chmod +x scripts/setup.sh
./scripts/setup.sh

🛠️ Connecting Your IDEs & Tools

Once running, the control center is available at http://127.0.0.1:8000/.

1. Google Antigravity

Add to ~/.gemini/config/mcp_config.json:

{
  "mcpServers": {
    "rtx5090-cascade": {
      "serverUrl": "http://127.0.0.1:8000/mcp"
    }
  }
}

Antigravity automatically detects the toolset (query_local_5090, cascade_llm, get_cascade_metrics) and dynamically dispatches code generation directly to your GPU.

2. Cursor

• Navigate to Settings > Models > OpenAI API Key.
• Set Base URL: http://127.0.0.1:8000/v1
• Set API Key: cascading-local (any string)
• Model: cascade-auto

3. Continue.dev (config.json)

{
  "models": [
    {
      "title": "Cascade Gateway (Local + Cloud)",
      "provider": "openai",
      "model": "cascade-auto",
      "apiBase": "http://127.0.0.1:8000/v1",
      "apiKey": "cascading-local"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Local Autocomplete",
    "provider": "openai",
    "model": "cascade-auto",
    "apiBase": "http://127.0.0.1:8000/v1",
    "apiKey": "cascading-local"
  }
}

4. Claude Desktop (claude_desktop_config.json)

{
  "mcpServers": {
    "local-cascade": {
      "serverUrl": "http://127.0.0.1:8000/mcp"
    }
  }
}

5. Python OpenAI SDK

from openai import OpenAI

client = OpenAI(
    base_url="http://127.0.0.1:8000/v1",
    api_key="cascading-local"
)

response = client.chat.completions.create(
    model="cascade-auto",
    messages=[
        {"role": "user", "content": "Write an optimized LRU cache in Python."}
    ]
)
print(response.choices[0].message.content)

📁 Repository Architecture

CascadeGateway is structured as a modern, modular Python package:

CascadeGateway/
├── src/
│   └── cascadegateway/
│       ├── core/                  # Hardware profiler, model sizing & routing engine
│       │   ├── config.py          # YAML config & environment loader
│       │   ├── hardware.py        # GPU VRAM auto-profiling (5090/4090/etc.)
│       │   ├── classifier.py      # Phase 1 structural validation & Phase 2 regex automaton (<1ms)
│       │   ├── streaming.py       # Phase 3 lookahead failover, loop breaker & asymmetric audit
│       │   └── router.py          # Biasing engine, metrics & model resolution
│       ├── api/                   # Modular FastAPI endpoints
│       │   ├── server.py          # App initialization, SSE streaming & /v1/chat/completions
│       │   ├── pipeline.py        # Architect, Review Gate & Builder endpoints
│       │   └── models.py          # Pause & Free GPU (VRAM purge) & 1-click IDE auto-config
│       ├── web/                   # Clean decoupled web assets & responsive UI
│       │   ├── templates/
│       │   │   └── dashboard.html # Responsive HTML5 dashboard & real-time telemetry
│       │   └── static/
│       │       ├── css/dashboard.css
│       │       └── js/dashboard.js
│       ├── hud/                   # Always-On-Top Desktop HUD & Telemetry Overlay
│       │   ├── __init__.py
│       │   └── overlay.py         # Tkinter floating dark obsidian window
│       ├── tray/                  # Windows system tray controller
│       │   └── app.py             # Pystray background service & menus
│       ├── mcp/                   # Model Context Protocol endpoints
│       │   ├── __init__.py
│       │   └── server.py          # Antigravity & Claude Desktop integration
├── scripts/                       # Platform launchers & utilities
│   ├── start_cascade.bat          # Windows batch launcher
│   ├── start_tray.vbs             # Silent windowless tray runner
│   ├── start_hud.bat              # Desktop HUD launcher
│   ├── start_hud.vbs              # Silent windowless HUD launcher
│   ├── setup.bat / setup.sh       # One-click installers
│   └── create_shortcuts.ps1       # Desktop & startup shortcut generator
├── tests/                         # Comprehensive automated test suites
│   ├── test_gateway.py            # End-to-end API integration tests
│   ├── test_resilience.py         # Sub-5ms SLA, structural validation & lookahead tests
│   └── test_new_features.py       # Loop breaker, IDE auto-config, dollar savings & /verify tests
├── assets/                        # Icons & diagrams
├── config.yaml                    # Gateway configuration
├── pyproject.toml                 # Modern pip/uv packaging metadata
└── requirements.txt

📡 API Reference

⚙️ Configuration (config.yaml)

server:
  host: "127.0.0.1"
  port: 8000

local:
  provider: "ollama"
  base_url: "http://127.0.0.1:11434"
  # Set empty to let the hardware profiler auto-select the best model for your GPU
  primary_model: ""
  max_context_tokens: 32768
  timeout_seconds: 120

cloud:
  provider: "gemini"
  default_model: "gemini-2.5-flash"
  pro_model: "gemini-2.5-pro"
  max_context_tokens: 1048576

biasing:
  mode: "adaptive"        # "adaptive", "manual", or "local_only"
  bias_factor: 0.35       # Baseline local bias (0.0 to 1.0)
  cloud_token_budget: 500000
  allow_context_overflow_to_cloud: true

📜 License

This project is licensed under the MIT License.