Why Tomoshibi
If you're paying Firecrawl/Exa by the token or spawning a Playwright per request, you're overpaying.
接入你的智能体
發布者自己的指令。不是 PlugMCP 端點。
用我們的金鑰與日誌託管一份副本。
npx -y [email protected]Claude Code 沒有本機指令的安裝連結。複製下方指令並在終端機中執行。
claude mcp add michael-obele-tomoshibi -- npx -y tomoshi@1.1.12在終端機中執行。
{ "mcpServers": { "michael-obele-tomoshibi": { "command": "npx", "args": [ "-y", "[email protected]" ] } }}claude_desktop_config.json
遠端 URL 還是本機指令? 兩種安裝方式的差異
來自發布者
If you're paying Firecrawl/Exa by the token or spawning a Playwright per request, you're overpaying.
Tomoshibi is a self-hosted search & scraping server for AI agents, built with Go. It turns any site into LLM-ready markdown in 200ms. One binary, $5/mo. API-compatible with Firecrawl's schema. Formerly Cinder.
| What hurts with hosted APIs | What Tomoshibi gives you | Outcome |
|---|
| $0.01–0.10 per scrape + rate limits | $0 self-hosted — one binary, hobby-tier RAM | Ship RAG without a cloud bill |
| 500ms Chrome spawn per request | One shared allocator + recycled tabs | ~200ms static, parallel pools |
| JS SPAs return empty HTML | Smart mode — static first, fallback to Chromedp | Works on React/Vue without guessing |
| Noisy HTML (nav/ads/footer) | Readability main-content + ad block | Clean markdown your LLM wants |
| Crawl needs a worker fleet | Monolith — Gin + Asynq in one process | Pay per container, not per service |
Proof: SearXNG search benchmark (scripts/search-bench.py, 10 workers/30s) — Tomoshibi 560 req/s p50 11ms vs Firecrawl self-hosted 1.9 req/s p50 5.4s — ~300× throughput, same $0. See docs/SEARCH_COMPARISON.md.
git clone https://github.com/Michael-Obele/tomoshibi.git && cd tomoshibi
docker compose up -d # api (7431) + redis (7434) — searxng opt-in: --profile searxng
curl http://localhost:7431/health # → {"status":"ok","service":"tomoshibi"}
# optional front doors to the same API:
docker compose -f packages/web/docker-compose.yml up -d --build # web UI (7432)
docker compose -f packages/mcp/docker-compose.yml up -d # MCP (7433)
Ports 7431–7435 (avoids clashing with 3000/8000/8080). Every feature works out of the box — scrape, crawl, batch, monitor, search.
# Pull from Docker Hub or GHCR — pick one image or all
# Docker Hub
docker pull michaelobele/tomoshibi-api:latest # api only (7431)
docker pull michaelobele/tomoshibi-mcp:latest # mcp only (7433)
docker pull michaelobele/tomoshibi-web:latest # web only (7432)
docker pull michaelobele/tomoshibi:latest # all-in-one
# GHCR (same images)
docker pull ghcr.io/michael-obele/tomoshibi-api:latest
docker pull ghcr.io/michael-obele/tomoshibi-mcp:latest
docker pull ghcr.io/michael-obele/tomoshibi-web:latest
docker pull ghcr.io/michael-obele/tomoshibi:latest
# Run one service standalone (needs a running API on 7431)
docker run --rm -p 7431:7431 michaelobele/tomoshibi-api
docker run --rm -p 7433:7433 -e TOMOSHI_API_URL=http://host.docker.internal:7431 michaelobele/tomoshibi-mcp
# web bakes its API URL at image build time (http://api:7431), so run it via its
# compose file instead — that overrides the build arg: packages/web/docker-compose.yml
# Or via compose per-package
docker compose -f packages/api/docker-compose.yml up -d # api + redis (searxng: --profile searxng)
docker compose -f packages/mcp/docker-compose.yml up -d # mcp (needs api)
docker compose -f packages/web/docker-compose.yml up -d # web (needs api)
git clone https://github.com/Michael-Obele/tomoshibi.git && cd tomoshibi
go run ./packages/api/cmd/api # needs Go 1.25+ and Chromium on PATH (7431)
REDIS_URL=redis://localhost:7434 go run ./packages/api/cmd/api # with async
curl -X POST http://localhost:7431/v1/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com"}'
{
"url": "https://example.com",
"markdown": "# Example Domain\n\n...",
"metadata": { "title": "Example Domain" }
}
No mode needed — smart is default. static skips JS; dynamic always renders.
# All packages
bun install # root (if using workspaces)
# Or per-package
bun --cwd packages/mcp install && bun --cwd packages/mcp dev
bun --cwd packages/web install && bun --cwd packages/web dev
tomoshibi/
├── packages/
│ ├── api/ # Go scraping API — Gin + Chromedp + Colly + embedded Asynq worker (7431)
│ ├── mcp/ # MCP server — 3 tools over the API, npm package `tomoshi` (7433)
│ └── web/ # Svelte 5 playground — scrape/crawl/search in the browser (7432)
├── docs/ # API reference, search comparison, SearXNG Fly guide (the rest is local-only)
├── skills/ # agent skills published to skills.sh (mirrored to .agents/, .agent/)
├── plan/ # local design plans (gitignored)
├── test_reports/ # historical test runs (gitignored)
├── .github/workflows/ # CI — ci.yml, docker.yml (images), npm.yml (publish)
├── docker-compose.yml # full stack: api 7431 + redis 7434 (+ searxng 7435 opt-in)
├── package.json # bun workspaces — dev:api · dev:mcp · dev:web · check
├── AGENTS.md # agent context source of truth (CLAUDE.md symlinks to it)
├── README.md # this file
└── LICENSE # MIT
Every search records one event (who answered, fallbacks, latency, trace id) and every native engine attempt one more (ok/empty/blocked/timeout + reason) into daily JSONL files that never leave the machine:
data/telemetry/search-2026-09-27.jsonl # one line per search
data/telemetry/engine-2026-09-27.jsonl # one line per engine attempt
curl -s 'http://localhost:7431/v1/insights?hours=24' | jq → per-engine scorecard (attempts, ok/empty/blocked, last_error, latency), chain stats (fallbacks, weak-result gates), p50/p95 latency, and recent_errors carrying trace_ids you can grep in docker logs tomoshibi-api-1.cat data/telemetry/engine-*.jsonl | jq -s 'group_by(.engine) | map({engine: .[0].engine, n: length})'TELEMETRY_ENABLED (true) · TELEMETRY_DIR (data/telemetry) · TELEMETRY_RETAIN_DAYS (14, 0 = keep forever) — documented in packages/api/.env.example../data:/app/data, so files survive recreation; /data/ is gitignored.TELEMETRY_ENABLED=false → /v1/insights returns {"enabled": false}.Release checklist: make check in packages/api (or bun run check in packages/mcp) → commit → push → watch gh run list → rebuild locally → verify:
docker logs tomoshibi-api-1 2>&1 | grep -E "Native search engines loaded|Telemetry recording"
curl -s localhost:7431/health
Self-hosters pull the same images CI pushed: docker pull ghcr.io/michael-obele/tomoshibi-api:latest.
curl -s 'localhost:7431/v1/insights?hours=24' | jq '.insights.engines' → blocked / last_error. Confirm with a live probe (CANARY_ENGINES=ddg go test -tags=canary ./internal/search/engines/ -run TestCanary -v from packages/api), log the verdict in plan/tomoshi-search/research.md §3b, then fix internal/search/engines/registry.default.yaml (better block_markers, proxy: webshare, or enabled: false) and rebuild. Roster rule: no edit without a live probe.recent_errors[].trace_id in /v1/insights and grep that id in docker logs. The chain explains itself: search: backend failed, trying next, search: results concentrated on one domain (weak-result gate firing), proxy budget exhausted — engines degraded to direct egress.environment: list (no env_file), and compose interpolation reads only the repo-root .env (gitignored). Add the var in both places, then docker compose up -d — use --build only when the code changed./v1/*. New builds honor APP_API_KEYS/API_KEYS; Docker passes neither, so auth is off there. If you enable it, also set TOMOSHI_API_KEY in your MCP config — otherwise every MCP call 401s.SERPER_API_KEY (Google, 2,500 free), TAVILY_API_KEY (1,000 free/mo), BRAVE_SEARCH_API_KEY ($5/mo free credit). Set them in packages/api/.env or just in your MCP config; the MCP forwards whatever the backend asks for (GET /v1/env) on every request via X-Tomoshi-Env, so the key lives in one place (TOMOSHI_FORWARD_ENV=false opts out).docker ps). Experiments use SERVER_PORT=7452./v1/crawl, /v1/batch, /v1/monitor return 503. Redis is optional by design — scrape/search keep working; docker compose up -d redis.{} or a raw Invalid arguments … dump. You're on tomoshi < 1.1.6, which published a root-level oneOf schema clients rendered as empty properties. Upgrade/restart the MCP — 1.1.6+ ships flat schemas and readable usage errors.backends (who answered), weak_gates (how often native was single-domain and got merged with SearXNG/Brave hits), and engines.<name>.empty (corpus mismatches) in /v1/insights.Base: http://localhost:7431 — every endpoint under /v1.
Client → Gin Router → Scraper (Colly / Chromedp + Readability)
↘ Asynq (Redis) → Worker → same Scraper
↘ Browser Pool (one allocator, recycled every 100 scrapes)
3 tools, not 8 — resource-oriented (action enum, ≤7 philosophy).
// .vscode/mcp.json or Claude config
{
"mcpServers": {
"tomoshi": {
"command": "npx",
"args": ["-y", "tomoshi"],
"env": {
"TOMOSHI_API_URL": "http://localhost:7431",
// optional search keys — forwarded to the backend on every request
// as X-Tomoshi-Env, so configure them HERE, not in the backend's .env
"BRAVE_SEARCH_API_KEY": "BSB-...", // or SERPER_API_KEY / TAVILY_API_KEY
},
},
},
}
The MCP asks the backend which key names it accepts (GET /v1/env) once, then forwards the matching ones from its own env on every call. TOMOSHI_FORWARD_ENV tunes it: "" = all requested (default), "false" = off, or a comma list to restrict.
Also: npx tomoshibi (alias, same package).
Svelte 5 playground for scrape/crawl/search — same API, browser UI.
bun --cwd packages/web dev # http://localhost:5173 (dev) / 7432 (docker)
Remote Functions proxy to the Go backend; API keys stay server-side.
Reusable instructions for AI coding agents — discoverable via skills.sh and the skills CLI (supports 75+ agents: Claude Code, Cursor, OpenCode, Codex, etc.).
This repo exposes skills from skills/ (canonical, indexed by skills.sh) — mirrored to .agents/skills/ and .agent/skills/ for OpenCode/Cursor compatibility.
Verify discovery locally:
npx skills add ./ --list # → tomoshibi
# from GitHub (recommended) — installs to ~/<agent>/skills/ (e.g. ~/.claude/skills/, ~/.agents/skills/)
npx skills add Michael-Obele/tomoshibi -g
# only this skill (same when repo has one skill)
npx skills add Michael-Obele/tomoshibi --skill tomoshibi -g
# all skills, all detected agents, no prompts (CI-friendly)
npx skills add Michael-Obele/tomoshibi --all -g -y
# verify
npx skills list -g
# installs to ./.claude/skills/, ./.agents/skills/, etc. — commit to share with team
npx skills add Michael-Obele/tomoshibi
# specific agents
npx skills add Michael-Obele/tomoshibi -a claude-code -a opencode -a cursor
# from local checkout
npx skills add ./ --skill tomoshibi
npx skills use Michael-Obele/tomoshibi --skill tomoshibi | claude
npx skills use Michael-Obele/tomoshibi --skill tomoshibi --agent claude-code
Browse on the web: skills.sh/Michael-Obele/tomoshibi (after first push/index).
Adding a new skill: create
skills/<name>/SKILL.mdwithname+descriptionfrontmatter (seetomoshibi/SKILL.md), thennpx skills add ./ --listto verify. Keepskills/canonical — mirror to.agents/skills//.agent/skills/if you need OpenCode compat.
Measured 2026-08-31 (docker images + docker stats). 4× lighter, 13× less RAM. Fits on 512 MB Fly.io / $5 Hetzner — Firecrawl doesn't.
Tomoshibi (灯火 — ともしび) — Japanese for lamp light, the glow of a paper lantern. Search as illumination: lighting up the web for AI agents. Short alias tomoshi (灯) for daily use. Formerly Cinder (extinct fire — wrong metaphor for discovery; cinder also collided with github.com/cinder/Cinder 5.5k★ and npm/cinder).
MIT — see LICENSE.
Migrating from Cinder?
github.com/Michael-Obele/cindernow redirects totomoshibi. Docker imageghcr.io/michael-obele/cinder→ghcr.io/michael-obele/tomoshibi. No code changes needed beyond the import path (github.com/Michael-Obele/tomoshibi).
Tomoshibi is a self-hosted search & scraping server for AI agents, built with Go. It turns any site into LLM-ready markdown in 200ms. One binary, $5/mo. API-compatible with Firecrawl's schema. Formerl
{ "mcpServers": { "michael-obele-tomoshibi": { "command": "npx", "args": [ "-y", "[email protected]" ] } }}總覽
Tomoshibi is a self-hosted search & scraping server for AI agents, built with Go. It turns any site into LLM-ready markdown in 200ms. One binary, $5/mo. API-compatible with Firecrawl's schema. Formerly Cinder.
工具
尚無工具清單。遠端伺服器會在首次健康檢查後取得工具清單。
常見問題
在本頁複製適用於 Cursor、Claude Desktop、Claude Code 或 VS Code 的 stdio 設定。它使用發布者的安裝方式,而不是 PlugMCP 代理伺服器。
這筆記錄未列出必要的環境變數。
Tomoshibi 透過 stdio 在本機執行。
上次檢查時間:2026-11-16。 Tomoshibi is a self-hosted search & scraping server for AI agents, built with Go. It turns any site into LLM-ready markdown in 200ms. One binary, $5/mo. API-compatible with Firecrawl's schema. Formerly Cinder.