Rust-native universal retrieval layer for humans and AI agents — search, extract, rank, and synthesize the web. CLI + REST API + MCP server, with adaptive extraction (CEP), token-budgeted packing, multi-signal ranking, and cited research.
星數
4
工具
—
可用率
—
The universal retrieval layer for humans and AI agents
Rust-native search, extraction, ranking, and synthesis — delivered as a CLI, a REST API, and an MCP server.
Most "search" tools hand an LLM a wall of raw HTML and hope for the best. Fetchium is built for the
opposite: it finds, fetches, extracts, ranks, validates, and packages information so that the
output is clean, cited, and token-efficient — whether the consumer is a human at a terminal or an
AI agent over the network.
It runs as a single Rust binary with no required runtime dependencies and exposes one engine three ways:
CLI — fetchium search, fetch, summarize, compare, research, and more.
REST API (fetchium-api) — the engine over HTTP for any language.
MCP server (fetchium-mcp) — a first-class tool for AI agents (Codex, Claude, any MCP client).
Why Fetchium
Token-efficient by design — extraction and packaging are query-aware, so agents spend context on signal, not boilerplate.
Evidence-first — results carry sources and citations, with a validation pass before output.
Adaptive — extraction and ranking escalate only as far as a query needs, keeping latency and cost down.
Agent-native — MCP + REST + LangChain/CrewAI adapters, not an afterthought.
Resilient — per-backend circuit breakers and bulkheads keep one slow/broken source from sinking a query.
How it compares
Naive retrieval (raw HTML → LLM)
Fetchium
Extraction
dump full HTML, hope the model copes
adaptive 5-layer CEP cascade + QADD DOM distillation
Extraction escalates layer by layer and stops as soon as the content is good enough, so the cheap
path handles the common case and expensive paths run only when needed.
Layer
Technique
Handles
1
HTML + CSS selectors (scraper)
~85% of pages
2
Streaming HTML rewriter (lol_html)
enhanced boilerplate removal
3
Headless JS rendering
SPAs / dynamic content (headless feature)
4
PDF / document extraction
PDF, DOCX, RTF
5
Screenshot OCR
image-heavy / canvas content (headless feature)
A learned predictor (cep_predictor) chooses where to start instead of always running Layer 1.
QADD — Query-Aware DOM Distillation
Prunes the DOM against the query before extraction, dropping nav/ads/chrome to cut tokens
dramatically (design target ~10–20×) while keeping query-relevant content.
SCS splits content into typed semantic segments; QATBE (Query-Aware Token-Budgeted
Extraction) scores segments with BM25 and packs the best ones into a fixed token budget
(greedy knapsack) — you get the most relevant content that fits, not an arbitrary truncation.
PDS — Progressive Detail Streaming
The same result can be served at four tiers — key_facts (~200 tok) → summary (~1k) →
detailed (~5k) → complete — so callers request exactly the depth they need.
HyperFusion — 8-signal ranking
Final ranking fuses eight independent signals rather than a single relevance score:
BM25, semantic, temporal, authority/trust, evidence, diversity, quality, and consensus/cluster
(rank/{bm25,semantic,temporal,trust,evidence,diversity,quality,cluster}.rs).
AMRS — Adaptive Multi-Agent Research Swarm
fetchium research decomposes a question and runs a swarm of cooperating agents
(research/amrs/) — decompose → search → synthesize → verify — to produce a cited report.
Evidence, validation & citations
Results flow through a validation stage and a citation layer (validate/, citation/,
rank/evidence.rs) so claims are backed by sources before they reach the output.
PIE — Persistent Intelligence Engine
Cross-session learning (source trust, failure patterns, query prediction) persisted locally,
so the engine improves with use instead of starting cold each time.
RAR — Retry-and-Refine
A multi-checkpoint self-correction loop that detects weak results and refines the query/extraction
rather than returning a poor answer.
Resilience
Every backend call is wrapped in a circuit breaker and bulkhead (resilience/), with
SimHash-based de-duplication across federated backends — one degraded source can't stall a query.
Configuration lives in ~/.fetchium/config.toml (with env-var overrides). API keys you provide are
stored locally and never committed. Run fetchium doctor to verify provider/tool setup. Optional
integrations: an AI provider (e.g. Ollama) for ai/research/deep, and
Chromium for CEP Layers 3/5. Details:
docs/guide/configuration.md.
This dual license is the Rust ecosystem standard (Rust itself and most crates use it). Two files are
included because each license has its own canonical text: MIT is short and maximally permissive,
while Apache-2.0 adds an explicit patent grant that some organizations require. "At your
option" means any user may choose whichever terms suit them — maximizing compatibility.
Unless you state otherwise, any contribution you submit shall be dual-licensed as above, without
additional terms.
F
Fetchium
zuhabul·4 顆星·
Rust-native universal retrieval layer for humans and AI agents — search, extract, rank, and synthesize the web. CLI + REST API + MCP server, with adaptive extraction (CEP), token-budgeted packing, mul
Rust-native universal retrieval layer for humans and AI agents — search, extract, rank, and synthesize the web. CLI + REST API + MCP server, with adaptive extraction (CEP), token-budgeted packing, multi-signal ranking, and cited research.
上次檢查時間:2026-10-28。 Rust-native universal retrieval layer for humans and AI agents — search, extract, rank, and synthesize the web. CLI + REST API + MCP server, with adaptive extraction (CEP), token-budgeted packing, multi-signal ranking, and cited research.