aigate
A self-hosted AI platform. One docker-compose up.
Everything an AI-powered workflow needs, inference, tool use, browser automation, image generation, speech synthesis, transcription, object storage, agentic code execution, web search, an email gateway, a Telegram client, time-series forecasting, typed decisions, an async job queue, and a web UI, behind a single OpenAI-compatible endpoint at http://localhost:4000. Point any existing client at it and it works.
Models and routing
Models across multiple providers. Five providers offer free tiers (Groq, OpenRouter, HuggingFace, Mistral, Cohere). Free means rate-limited and capped, not unlimited. Cerebras needs a paid plan as of the last audit. See free-tier reality check for exact RPM/RPD/monthly caps before relying on a tier. Local providers run on your own hardware, CPU or NVIDIA GPU, with no network calls, rate limits, or usage costs. They include Ollama, talkies for ASR and TTS, vllm-wrap for text LLMs and embeddings, llama.cpp for Surya OCR 2 and Qwen3-8B embeddings, stable-diffusion.cpp, audiolla, predictalot, and decidealot. The gateway tries providers in priority order and falls back when one rate-limits or fails.
Tools and capabilities
MCP tools across multiple servers. Any model with function calling can invoke them autonomously — the model orchestrates, you just prompt.
- Stealth browser cluster — 5 Camoufox replicas behind HAProxy, real OS-level mouse and keyboard input, zero CDP exposure. Passes Cloudflare, CreepJS, BrowserScan, and every other bot detector we've thrown at it.
- Two agentic coding agents — full shell access, persistent workspaces, file I/O, tool use. Claude Code (your subscription or API key) and pi-coding-agent pointed at z.ai for GLM models. Both expose REST API, OpenAI-compatible endpoint, and MCP server.
- Sandboxed code execution — piston at
/piston/. Multi-language code execution in nsjail-isolated sandboxes (own user namespace, chroot, seccomp, cgroups, no network). Default install: Python + Node. Add Bash / Deno / Go / Rust / TypeScript / 40+ other languages viaPISTON_LANGUAGES+ rebuild. REST API +execute_codeMCP tool any function-calling LLM can call to compute deterministic results (hashes, parses, transforms) instead of guessing. - Object storage — S3-compatible, public-read uploads, presigned URLs, auto-expiry. Plain HTTP and boto3.
- Image generation — FLUX, DALL-E, Stable Diffusion (cloud + local CPU/CUDA via stable-diffusion.cpp) via MCP tools that return persistent URLs, not base64 blobs. Local CPU: sd-turbo, sdxl-turbo. Local CUDA adds: sdxl-lightning, flux-schnell, juggernaut-xi.
- Text-to-speech — Kokoro (CPU + CUDA-ONNX, English+Chinese, voice presets), Qwen3-TTS (CUDA, voice cloning + voice-design + emotion control via the
instructionsfield), OpenAI TTS. - Transcription — Whisper (cloud + local CPU/CUDA), Parakeet (~3400× real-time on CPU).
- Web search — SearXNG self-hosted meta-search (Google, Bing, DuckDuckGo, Wikipedia) via MCP
search_webtool. - Telegram client — Telethon at
/telethon/. Send/read/edit/delete messages, list dialogs, forward, send files from URL, manage group membership. REST API + MCP tools. - Email gateway, mailbox at
/mailbox/. IMAP+SMTP across N accounts from one YAML config, unified inbox, per-account list/search/CRUD, SMTP send. REST API + flat MCP tool set (mailboxparameter selects account). - Time-series forecasting + tabular ML — predictalot at
/predictalot/. Five foundation forecasters (chronos-2, timesfm-2.5, moirai-2, toto-1, sundial-base-128m) across six forecast types (univariate, multivariate, past/future covariates, samples) with per-type weighted ensembles at/v1/timeseries/<type>/…+ a sibling/v1/tabular/*family in v1.0.0 — 9 supervised backends (lightgbm, xgboost, hist-gbt, random-forest, logistic, mlp, svm-rbf, knn, naive-bayes) and 3 meta-learners (calibrated / stacking / diversified). REST API for both families + 27 MCP tools (foundation-model forecasting plusunload_models, tabular is REST-only). CPU or CUDA. - Typed decisions: decidealot at
/decidealot/. Local Laya, Von, and CLM are enabled by default. Hosted TypeSafe Jev enables when its key is configured. Send astateplus namedchoice,score, ornoulquestions and get typed answers with model probabilities, so the caller sets its own threshold. Laya routes between English and multilingual checkpoints and has a checkpoint tuned for policy, routing, and triage decisions. Von is English-only, for short questions with clear criteria. CLM calls AIGate's internal Qwen3-8B embeddings route;make run-bgstarts its CUDA encoder automatically. SetDECIDEALOT_CLM_ENABLED=falseto remove that GPU dependency. REST API, batch decisions, and 4 MCP tools. CPU or CUDA. - Audio production — audiolla at
/audiolla/. Stem separation (Demucs / UVR), restoration (UVR de-reverb / de-echo / de-noise), mastering (matchering / pedalboard chains + curated presets likemaster-for-spotify,podcast-cleanup,vocal-cleanup), MIR analysis (BPM / key / LUFS / beats / onsets / melody / chords / segments), DSP transforms (sox + ffmpeg), loudness normalization, speech enhancement (DeepFilterNet), VAD (silero), diarization (pyannote), CLAP embeddings + zero-shot classification, AudioSet tagging (AST), audio→MIDI (basic-pitch), MIDI compose / inspect / transform / render via fluidsynth, text-to-audio generation (stable-audio-open / musicgen / riffusion / audioldm2 — CUDA only), ad-hoc op-chain pipelines, async jobs + webhooks. REST API + MCP tools. v1.0+ contract: JSON body on every audio endpoint, raw bytes only atPUT /v1/files/{path},output_pathxoroutput_urlmandatory for any audio-producing call. - Video toolkit — flickies at
/flickies/. Lipsync via LatentSync 1.5 (ByteDance, Apache-2.0, ~8 GB VRAM, default on CUDA) and Wav2Lip / Wav2Lip-GAN (LRS2 non-commercial, gated). Face restore via GFPGAN v1.4. ffmpeg ops — trim, concat, transcode (incl. gif + fps + codec change), scale, mux audio, extract audio, thumbnail grid. ffprobe info. Async jobs + webhooks. REST API + 11 MCP tools. Same JSON-body +output_pathxoroutput_urlcontract as audiolla. GFPGAN + LatentSync 1.5 are CUDA-only; CPU image runs ffmpeg ops + slow Wav2Lip-CPU.
Ask a Groq model to research something and it opens a browser, reads pages, screenshots them, uploads to storage, and comes back with a summary and links. The model decides what tools to use and in what order.
Web UI
LibreChat at /librechat/ — pre-configured with all models and MCP tools, conversation history, file uploads, WebSocket streaming. Email/password auth, first user becomes admin.
Infrastructure
Multiple containers. Nginx reverse proxy. PostgreSQL + MongoDB for persistence. Two Redis instances (cache + browser session sync). Async job queue for long-running inference. Cloudflare Tunnel for public exposure with zero open ports.
Security
Network-isolated by default — internal services have no host ports. Every endpoint requires auth. Application containers carry no-new-privileges:true (genuine carve-outs: piston needs privileged:true for nsjail; tailscale needs NET_ADMIN for the userspace tun device). File path pre-flight validation on startup.
Everything is opt-in
Enable what you want in .env, ignore what you don't. The core stack (nginx, LiteLLM, PostgreSQL, Redis, proxq) is always on. Every provider, local model, MCP server, and extra service is a flag flip away.
make run-bg. That's the whole install.
Architecture
client
▼
cloudflared (CLOUDFLARED=1) │ tailscale (TAILSCALE=1, tailnet-only)
▼
nginx :4000 ┌──────────── always on ────────────┐
├─► /claudebox/ → claudebox │ nginx, LiteLLM, PostgreSQL, Redis │
├─► /pibox-zai/ → pibox-zai │ proxq — everything else is opt-in │
├─► /pibox/ → pibox │ │
├─► /stealthy-auto-browse/ → HAProxy → [browser ×5]└───────────────────────────────────┘
├─► /storage/ → hybrids3
├─► /q/ → proxq → LiteLLM (async, returns job ID)
├─► /librechat/ → LibreChat (web UI, LIBRECHAT=1)
├─► /searxng/ → SearXNG (meta-search, SEARXNG=1)
├─► /telethon/ → Telethon (Telegram client, TELETHON=1)
├─► /predictalot/ → predictalot (time-series forecasting, CPU, PREDICTALOT=1)
├─► /predictalot-cuda/ → predictalot (time-series forecasting, NVIDIA GPU, PREDICTALOT_CUDA=1)
├─► /decidealot/ → decidealot (typed decisions REST + MCP, CPU, DECIDEALOT=1)
├─► /decidealot-cuda/ → decidealot (typed decisions REST + MCP, NVIDIA GPU, DECIDEALOT_CUDA=1)
├─► /audiolla/ → audiolla (audio-production REST + MCP, CPU, AUDIOLLA=1)
├─► /audiolla-cuda/ → audiolla (audio-production REST + MCP, NVIDIA GPU, AUDIOLLA_CUDA=1)
├─► /flickies/ → flickies (video toolkit REST + MCP, CPU, FLICKIES=1)
├─► /flickies-cuda/ → flickies (video toolkit REST + MCP, NVIDIA GPU, FLICKIES_CUDA=1)
├─► /mailbox/ → mailbox (IMAP+SMTP gateway, MAILBOX=1)
├─► /piston/ → piston (sandboxed multi-language code execution, PISTON=1)
└─► / → LiteLLM (sync)
├─ Groq (free: 30 RPM, 1K-14.4K RPD per model, GROQ=1)
├─ Cerebras (paid plan required, CEREBRAS=1)
├─ OpenRouter (free: 50 RPD $0 / 1K RPD with $10+, OPENROUTER=1)
├─ HuggingFace (free: $0.10/mo credits — eval only, HUGGINGFACE=1)
├─ Mistral (free "Experiment" tier — exact limits not published, MISTRAL=1)
├─ Cohere (trial: 1K calls/MONTH — runs out fast, COHERE=1)
├─ Ollama CPU (local, OLLAMA=1)
├─ Ollama CUDA (local, NVIDIA, OLLAMA_CUDA=1)
├─ Talkies CPU (local, unified ASR + Kokoro TTS, TALKIES=1)
├─ Talkies CUDA (local, NVIDIA, full ASR + Kokoro + Qwen3-TTS voice cloning, TALKIES_CUDA=1)
├─ sd.cpp CPU (local, image gen, SDCPP=1)
├─ sd.cpp CUDA (local, image gen, SDCPP_CUDA=1)
├─ vLLM CPU (local, text LLM + embeddings, VLLM=1)
├─ vLLM CUDA (local, NVIDIA, text LLM + embeddings, VLLM_CUDA=1)
├─ llama.cpp CPU (local, GGUF + vision-VLM incl. Surya OCR 2, LLAMACPP=1)
├─ llama.cpp CUDA (local, NVIDIA, Surya OCR 2 + Qwen3-8B embeddings, LLAMACPP_CUDA=1)
├─ claudebox (subscription, CLAUDEBOX=1)
├─ pibox-zai (subscription, PIBOX_ZAI=1)
├─ pibox (this stack's own models, PIBOX=1)
├─ Anthropic (pay-per-token, ANTHROPIC=1)
└─ OpenAI (pay-per-token, OPENAI=1)
MCP servers (all optional):
├─ stealthy_auto_browse — run_script: multi-step browser automation (BROWSER=1)
├─ hybrids3 — file upload, download, list, delete, presign (HYBRIDS3=1)
├─ claudebox — agentic Claude Code via OAuth or API key (CLAUDEBOX=1)
├─ pibox_zai — agentic pi-coding-agent via z.ai/GLM (PIBOX_ZAI=1)
├─ mcp_tools — generate_image + generate_tts + search_web + execute_code (auto-enabled when HuggingFace, OpenAI, talkies[-cuda], sd.cpp[-cuda], or SearXNG is active — PISTON=1 alone does NOT enable it; execute_code only appears once mcp_tools is already active for one of those other reasons)
├─ telethon — Telegram send/read/edit messages, dialogs, files, group management (TELETHON=1)
├─ predictalot — time-series forecasting via 5 foundation models × 6 type-routed endpoints + per-type ensemble (PREDICTALOT=1 and/or PREDICTALOT_CUDA=1)
├─ decidealot typed decisions with Laya, Von, CLM, and hosted Jev via system_one, system_one_batch, list_models, unload_models (DECIDEALOT=1 and/or DECIDEALOT_CUDA=1)
├─ audiolla — audio-production: stem separation, restoration, mastering, MIR, DSP, loudness, speech enhancement, diarization, MIDI compose + render, workflow presets + ad-hoc pipelines (AUDIOLLA=1 / AUDIOLLA_CUDA=1)
├─ flickies — video toolkit: lipsync (LatentSync 1.5 + Wav2Lip), face restore (GFPGAN), ffmpeg ops — trim/concat/transcode/scale/mux_audio/extract_audio/thumbnail_grid (FLICKIES=1 / FLICKIES_CUDA=1)
├─ mailbox — IMAP+SMTP gateway: inbox, list/search/send across N accounts (MAILBOX=1)
└─ piston (via mcp_tools) — execute_code: sandboxed multi-language code execution via nsjail (requires PISTON=1 AND mcp_tools already active per above — hosted by the mcp_tools aggregator, not a standalone MCP server)
Most persistent data lives under .data/ by default (bind mounts). Override the base with DATA_DIR or per-service with DATA_DIR_* env vars (e.g. DATA_DIR_OLLAMA=/mnt/nas/ollama), see .env.example for the full list. The default directory structure is tracked in git via .gitkeep files so the right directories exist on a fresh clone, contents are gitignored.
Default writable locations:
Mailbox also uses the mailbox-state named volume for its SQLite audit log, send limits, checkpoints and webhook delivery state. Back up that volume alongside the account config. Read the Mailbox migration notes and browser migration notes before upgrading older deployments.
Services
Security and Exposure
Network isolation — internal services live on the private aigate-internal Docker network with no host port bindings. PostgreSQL, Redis, and LiteLLM are always internal. Optional services (hybrids3, HAProxy, Ollama, talkies, vllm-wrap, llamacpp-wrap, sdcpp, piston, audiolla, predictalot, decidealot, etc.) join the same private network when enabled. Services that need to reach the public internet (e.g. cloud LLM providers, HuggingFace model pulls) also join aigate-public; the rest are internal-only. Only nginx is exposed.
Auth on everything. Every service requires a bearer token. LiteLLM needs LITELLM_MASTER_KEY. Claudebox instances each have their own token. Hybrids3 uses per-bucket keys. The stealthy browser cluster has an AUTH_TOKEN (defaults to lulz-4-security if unset). The MCP tools server validates MCP_TOOLS_AUTH_TOKEN. LibreChat has its own email/password auth, and the first registered user becomes admin; set LIBRECHAT_ALLOW_REGISTRATION=false in .env and restart after creating your account. The admin UI supports HTTP basic auth.
No new privileges — application containers carry security_opt: no-new-privileges:true by default. Two unavoidable exceptions: piston requires privileged: true to set up nsjail user namespaces + chroots inside the sandbox (the code-execution isolation lives one level deeper, inside nsjail itself), and tailscale requires NET_ADMIN for the userspace tun device. Both are documented in their service pages.
Pre-flight validation — make run and make run-bg validate that any file paths set in .env (e.g. CLOUDFLARED_CONFIG, CLOUDFLARED_CREDS) actually exist before starting Docker. If a path is set but missing, the stack refuses to start with a clear error rather than silently creating a broken directory mount.
Public exposure — if you want to reach the gateway from outside, use Cloudflare Tunnel instead of opening ports. Set CLOUDFLARED=1 in .env for a quick *.trycloudflare.com URL (no account needed), or configure a named tunnel for a fixed custom domain. Traffic goes through Cloudflare's network before it reaches nginx — DDoS protection and TLS termination included.
Tailnet-only access. Set TAILSCALE=1 for an HTTP proxy through tailscale serve, accessible only from the tailnet. It preserves each caller's IP. Headscale users set TS_SERVE_FQDN to the node's actual MagicDNS name. Access http://<node-hostname>/ from your tailnet; the connection is encrypted by WireGuard.
MCP OAuth. Optional browser authorization needs PROXY_BASE_URL set to your browser-facing HTTPS origin. It can be private; it just has to be reachable from the browser. Bearer-key clients keep working without OAuth setup. See OAuth, trusted proxies and upgrade instructions.
MCP Tools
Tools across optional servers. Any model that supports function calling can invoke them — the model decides when and how to use them based on the prompt.
→ Full MCP tool reference with parameters
Providers and Models
Models across multiple providers. Six offer free tiers with no credit card required — but every "free" tier has a hard cap (RPM, RPD, monthly tokens, or monthly request count). Read free-tier reality check for exact numbers. Five providers run locally on your own hardware — CPU or NVIDIA GPU — with no rate limits. Per-model fallback chains route automatically through alternative providers when one fails or rate-limits.
Routing philosophy
Fallback chains
Every model has a fallback chain defined in litellm/config/fallbacks.json. When a provider fails or rate-limits, LiteLLM tries the next one automatically. You request one model — the gateway figures out who can actually serve it.
Example: you request groq-gpt-oss-120b. Groq returns 429 (rate limited). LiteLLM silently retries with cerebras-gpt-oss-120b. Cerebras is down. It tries mistral-large. Mistral responds. You get the response — same format, same schema, no error. The model field in the response tells you which provider actually served it.
groq-gpt-oss-120b → 429 rate limited
↓ fallback
cerebras-gpt-oss-120b → 503 unavailable
↓ fallback
mistral-large → 200 ✓
For LLM chat models, chains follow the priority tiers: free cloud first, then subscription, then pay-per-token, then local. For image, TTS, and STT models, local models are preferred over paid cloud (they're free and have no rate limits). Small models fall back to other small models. Code models fall back to other code models. Local CUDA models fall back to local CPU models.
Chains are filtered at startup — make run regenerates the LiteLLM config and strips out any provider you haven't enabled. If you only have GROQ=1 and OLLAMA=1, the chain skips everything in between.
Local models (Ollama, CPU)
All local CPU models are last in fallback chains — used when cloud providers are rate-limited or unavailable. Ollama unloads models after 5 minutes of inactivity.
Local models (Ollama CUDA — OLLAMA_CUDA=1)
CUDA models run with flash attention and quantized KV cache. See Resource management below for how VRAM is shared across services.
Dolphin Phi runs only through local-ollama-cpu-dolphin-phi. Use the CUDA Qwen or Gemma abliterated aliases for uncensored GPU inference.
Local transcription + TTS (talkies, CPU — TALKIES=1)
Local transcription + TTS (talkies, CUDA — TALKIES_CUDA=1)
Adds the heavier ASR models that need a GPU to be useful. Kokoro TTS still runs on CPU even inside the CUDA image.
Local image generation (sd.cpp, CPU — SDCPP=1)
Local image generation (sd.cpp, CUDA — SDCPP_CUDA=1)
Models auto-download on first use and cache in .data/sdcpp/models/.
Local text LLM + embeddings (vllm, CPU — VLLM=1)
Supervised single-model wrapper around vllm serve (CPU). Only one model resident in RAM at a time. Same wrapper and surface as the CUDA variant; just slower. Add more by editing vllm/models.cpu.json.
Local text LLM + embeddings (vllm-cuda — VLLM_CUDA=1)
Supervised single-model wrapper around vllm serve (NVIDIA). Only one model resident in VRAM at a time — the wrapper restarts the subprocess when a different model is requested. Add more by editing vllm/models.cuda.json. Both variants share the same ${DATA_DIR_VLLM}/models/ weight store.
→ Full provider and model list
Resource management
Local services share limited hardware — a single GPU can't run an LLM, an image generator, and a TTS model simultaneously. The platform handles this automatically so you never have to think about it.
Automatic unloading — every local service unloads idle models after a configurable timeout. Ollama unloads after 5 minutes by default. sd.cpp unloads after 5 minutes (SDCPP_IDLE_TIMEOUT / SDCPP_CUDA_IDLE_TIMEOUT). Talkies unloads ASR models after TALKIES_MODEL_TTL (default 10 min) and Qwen3-TTS unloads on demand. predictalot lazy-loads each forecaster on first call and unloads after PREDICTALOT_MODEL_IDLE_TIMEOUT (default 30m) — only the models you've actually asked for occupy memory. decidealot keeps one local model resident by default, swaps Laya and Von on demand, and releases it after DECIDEALOT_PROVIDER_IDLE_UNLOAD_SECONDS (default 600). This means VRAM and RAM are only held while a model is actively serving or within its idle window.
Hardware locks: a LiteLLM callback (resource_manager.py) runs one CUDA job at a time across its local groups and Predictalot MCP calls, and the same on CPU. Aigate's Decidealot launcher shares that Redis lock for Laya and Von through REST, MCP, and batch items. Before CUDA inference it unloads Aigate's local llama.cpp encoder. CLM takes no outer lock because its nested LiteLLM embeddings request owns admission. Remote encoder lifecycles remain the remote service's responsibility. A crashed worker's lock expires after RESOURCE_LOCK_TTL_SECONDS. If Redis is down, these local requests fail instead of running unlocked. The thin launcher inherits Decidealot's image without reinstalling Torch or CUDA. Build it with make build-decidealot; run focused regression tests with make test-decidealot-coordination.
Competing-group unload: before each LiteLLM-routed local request, other groups on the same hardware are asked to free resources. Ollama uses keep_alive: 0, sd.cpp uses /sdcpp/v1/unload, talkies / vllm-cuda / llamacpp-cuda use DELETE /api/ps/{model}, audiolla uses POST /v1/unload, flickies uses DELETE /v1/engines/{slug}, and predictalot / decidealot use POST /v1/models/unload. Busy services can reject eviction. Predictalot MCP inference also requests competing-group eviction. Decidealot's launcher specifically evicts the local llama.cpp encoder before CUDA Laya or Von; it does not evict every other resident GPU service. Direct audiolla, flickies, and predictalot HTTP requests still bypass this admission path. See resource management for the exact boundaries.
Operator-facing unload — POST /v1/unload/cuda and POST /v1/unload/cpu (and POST /v1/unload for both classes) fan out the eviction across every service on the given hardware class, predictalot and decidealot included, for manual VRAM cleanup outside the LiteLLM path. Same bearer as the rest of aigate. See docs/resource-management.md for response shape.
sd.cpp pre-load (image generation only). sd.cpp returns 503 another load or generation in progress while a model is mid-load (~5-20s for typical SDXL weights). LiteLLM's image-gen path retries internally + walks the fallback chain on 5xx, every retry hits the same lock, every retry 503s, and the whole chain ends with HTTP 200 + EMPTY data (a router-side bug: image-gen fallback exhaustion looks like success). The resource manager pre-empts this by issuing an explicit POST /sdcpp/v1/load?model=<key> while it still holds the CUDA or CPU hardware lock, blocking until the backend is fully warm. By the time LiteLLM dispatches the actual /v1/images/generations call, attempt 1 succeeds. No 503 storm, no fallback amplification, no empty-data response. No-op when the model is already loaded.
Auto-load on demand — models load automatically when needed. Send a request to any model and its service loads it on the fly. No pre-loading, no manual model management. The sd.cpp wrapper accepts /v1/images/generations requests even when no model is loaded — it starts the sd-server subprocess with the right model automatically.
Non-blocking rejection. The sd.cpp wrapper uses TryLock instead of blocking. If a generation or model swap is already in progress, new requests get an immediate 503 instead of queuing indefinitely. The resource manager's hardware lock handles scheduling at a higher level. Requests wait at the LiteLLM layer, not inside individual services.
The net effect: you can freely mix LLM chat, image generation, TTS, and STT requests across local services. The platform queues, unloads, loads, and routes automatically. The only constraint is throughput — one local job per hardware at a time.
Setup
1. Clone
git clone https://github.com/psyb0t/aigate
cd aigate
2. Configure
make bootstrap
This creates .env from .env.example. Any make target does it for you, so you can skip straight to make run if you prefer. .env is gitignored, so your keys and flags survive every update.
Reach for .env first for anything you want to change. Profile flags, API keys, data directories, and timeouts are all env vars already, and every service's memory/CPU limit is an env var with a fixed default (see make limits below), so most changes need no compose edit at all.
For the rest, do not edit docker-compose.yml. It is tracked and moves with the repository, because its 46 service definitions and nginx routes have to stay in step with the provider configs, the Makefile profiles, and the LiteLLM config builder. An update overwrites your changes there, and a stale copy silently breaks new services.
Put your changes in docker-compose.override.yml instead. It is gitignored, Compose merges it last so it wins, and nothing upstream ever touches it. Write only the keys you are changing:
services:
claudebox:
mem_limit: 8g
ports:
- "127.0.0.1:8080:8080"
Everything else about that service still comes from the base file. make bootstrap prints the active compose file chain if you want to check what is merged.
Fill in the values — every variable is documented with comments in .env.example.
Everything is opt-in via flags in .env. API keys are stored separately and never activate anything on their own — set the flag to 1 to enable:
make run regenerates litellm/config.yaml before starting — only enabled providers are included, fallback chains are filtered to match.
If CLOUDFLARED_CONFIG or CLOUDFLARED_CREDS are set, make run/make run-bg will verify those files exist before starting. A missing file causes an immediate error — better than Docker silently mounting a directory instead.
3. Set resource limits
make limits
Every service has a fixed memory and CPU limit: the default in docker-compose.yml (for example TALKIES_CUDA_MEM_LIMIT defaults to 10g). A model needs the same RAM on any host, so limits never scale with the machine. Override one by setting the variable in .env, such as TALKIES_CUDA_MEM_LIMIT=16g.
make limits checks the enabled services against this machine:
- It prints each enabled service with its limit and totals the worst-case RAM. The LiteLLM resource manager runs one job per hardware class, so the CPU group (
ollama,talkies,sdcpp,vllm,llamacpp,audiolla,flickies,predictalot,decidealot) counts at its largest member, and the CUDA group likewise. It warns when the total does not fit in RAM minus an OS reserve (2 GB or 5%, whichever is larger). It never shrinks a limit to fit, because a limit below a model's load size gets the container OOM-killed. - It writes
.env.limitswith CPU caps only, for services whose defaultcpusexceeds the host core count. Docker refuses acpusvalue above the core count.
Re-run it after enabling or disabling a service or moving to a different machine. .env.limits is gitignored. Each server maintains its own.
4. Start
make run-bg # detached (background)
make run # foreground with logs
Gateway is at http://localhost:4000. Admin UI at http://localhost:4000/ui/. LibreChat at http://localhost:4000/librechat/ (if LIBRECHAT=1). SearXNG at http://localhost:4000/searxng/ (if SEARXNG=1).
On first start, Ollama will pull all local models in the background. talkies pre-downloads its enabled ASR + TTS models (whisper, canary, parakeet, nemotron, kokoro, qwen3-tts on CUDA) on boot, and vllm-pull / llamacpp-pull populate the wrapper model dirs from HuggingFace before their containers start. Everything caches to .data/ and won't re-download on restart.
Usage
# cloud provider (free tier — capped, see docs/providers.md#free-tier-reality-check; auto-fallback on 429)
curl http://localhost:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "cerebras-gpt-oss-120b", "messages": [{"role": "user", "content": "hello"}]}'
# local model (no network, no rate limits)
curl http://localhost:4000/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "local-ollama-cpu-llama3.2-3b", "messages": [{"role": "user", "content": "hello"}]}'
# image generation
curl http://localhost:4000/images/generations \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "hf-flux-schnell", "prompt": "a cat riding a skateboard"}'
# local image generation (CUDA — sd-turbo, fast)
curl http://localhost:4000/images/generations \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "local-sdcpp-cuda-sd-turbo", "prompt": "a red panda in a forest", "size": "512x512"}'
# transcription (cloud — Groq Whisper, fast)
curl http://localhost:4000/audio/transcriptions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-F "model=groq-whisper-large-v3" -F "[email protected]"
# local transcription (talkies — parakeet, English, very fast on CUDA)
curl http://localhost:4000/audio/transcriptions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-F "model=local-talkies-cuda-parakeet-tdt-0.6b-v3" -F "[email protected]"
# stereo speaker diarization (left/right channel split)
curl http://localhost:4000/audio/transcriptions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-F "model=local-talkies-cuda-parakeet-tdt-0.6b-v3" \
-F "[email protected]" \
-F "diarization=true" \
-F "response_format=verbose_json"
# text-to-speech (local CPU Kokoro via talkies — many voices)
curl http://localhost:4000/audio/speech \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "local-talkies-kokoro-tts", "input": "Hello world", "voice": "af_heart"}' \
-o speech.mp3
# text-to-speech (local CUDA — Qwen3-TTS, voice cloning)
curl http://localhost:4000/audio/speech \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "local-talkies-cuda-qwen3-tts", "input": "Hello world", "voice": "alloy"}' \
-o speech.mp3
# text embeddings (local)
curl http://localhost:4000/embeddings \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "local-ollama-cpu-bge-m3", "input": "your text here"}'
# time-series forecasting (PREDICTALOT=1 / PREDICTALOT_CUDA=1 — direct route, not via LiteLLM)
# v1.0.0: type-routed under /v1/timeseries/<type>/. Pick a type: univariate / multivariate / covariates/{past,future} / covariates / samples
# v1.0.0 also ships a sibling /v1/tabular/* family (9 supervised backends + 3 meta-learners) — REST-only.
curl http://localhost:4000/predictalot/v1/timeseries/univariate/forecast \
-H "Authorization: Bearer $PREDICTALOT_AUTH_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "chronos-2",
"context": [[10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]],
"config": {"horizon": 5}
}'
# typed decision (DECIDEALOT=1 / DECIDEALOT_CUDA=1, direct route, not via LiteLLM)
# answers.handling.choice is the picked key, with a probability per criteria key
curl http://localhost:4000/decidealot/v1/systemone \
-H "Authorization: Bearer $AIGATE_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"model": "laya",
"state": "A proposed action would permanently delete protected data.",
"questions": {
"handling": {
"type": "choice",
"instructions": "Choose the required handling for this action.",
"criteria": {
"allow": "The action is reversible and does not affect protected data.",
"require_review": "The action is irreversible or affects protected data."
}
}
}
}'
# mailbox — unified inbox across configured accounts (MAILBOX=1, direct route)
curl "http://localhost:4000/mailbox/inbox?limit=5" \
-H "Authorization: Bearer $MAILBOX_AUTH_TOKEN"
# mailbox — send mail through a configured account
curl -X POST http://localhost:4000/mailbox/mailboxes/work/send \
-H "Authorization: Bearer $MAILBOX_AUTH_TOKEN" \
-H "Content-Type: application/json" \
-d '{"to": ["[email protected]"], "subject": "hello", "body_text": "from aigate"}'
# telethon — send a Telegram message (TELETHON=1, direct route)
curl -X POST http://localhost:4000/telethon/api/messages \
-H "Authorization: Bearer $TELETHON_AUTH_KEY" \
-H "Content-Type: application/json" \
-d '{"chat": "@username", "text": "hello from aigate"}'
Async (via proxq)
Any request sent through /q/ is queued and processed in the background — useful for long-running inference that would otherwise time out.
# submit — returns instantly with a job ID
curl http://localhost:4000/q/v1/chat/completions \
-H "Authorization: Bearer $LITELLM_MASTER_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "cerebras-gpt-oss-120b", "messages": [{"role": "user", "content": "write a novel"}]}'
# → 202 {"jobId": "550e8400-e29b-41d4-a716-446655440000"}
# check status
curl http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000 \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
# → {"id": "...", "status": "running"}
# get the response (replays upstream response exactly)
curl http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000/content \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
# → {"choices": [...], "usage": {...}}
# cancel
curl -X DELETE http://localhost:4000/q/__jobs/550e8400-e29b-41d4-a716-446655440000 \
-H "Authorization: Bearer $LITELLM_MASTER_KEY"
Only OpenAI API paths (/v1/chat/completions, /v1/embeddings, /v1/audio/*, /v1/images/*, /v1/responses) are queued. Health checks, model lists, key management, and admin UI requests pass through to LiteLLM directly.
Configurable via .env:
→ Per-service usage — browser automation, object storage, agentic claudebox tasks, vision, streaming, web search, time-series forecasting, email gateway, Telegram client, Python SDK examples
Services Reference
All endpoints, auth requirements, request/response formats, and config options.