web-retrieval-mcp — reliable MCP web search and web fetch for AI agents
An open-source Model Context Protocol (MCP) server for source-grounded web research. Connect Codex CLI and the Codex IDE extension, Claude Code, Claude Desktop, Cursor, ChatGPT desktop, or any other compatible MCP client to six read-only tools for Exa or Tavily search, resilient Exa → Camoufox → Tavily → Firecrawl page retrieval, AI/ML paper discovery, and developer-source search—with explicit provenance, optional local caching, and SSRF guards.
MCP web search + tiered web fetch for AI agents (Exa, Firecrawl), SSRF-guarded, cross-platform.
Web research is more reliable when an agent can see where every result came from,
choose the right retrieval strategy, and recover when one provider or page-access
method fails. web-retrieval-mcp makes those controls part of the tool contract:
Keep sources separate. Default search returns one block per result with its own
title, URL, highlights, and text, followed by a Sources trailer. Exa deep modes
additionally return a clearly labeled synthesized answer, with grounding when Exa
provides it, before the source-separated result blocks.
Use more than one retrieval path. Search can use Exa or Tavily and fall back to
Firecrawl. Full-page fetches can escalate from indexed content to a local Camoufox
browser, optional Tavily Extract, and Firecrawl.
Know what actually served the answer. Fetch responses identify the serving tier,
cache state, semantic mode, and truncation instead of hiding the route.
Ask for the right shape. Fetch a readable page body, a concise summary, or a
grounded answer to a question. Control freshness, rendering, domains, dates, result
count, and per-result text size.
Search specialist corpora. Discover AI/ML papers, inspect query-relevant
full-text passages, expand to related papers, and search developer documentation,
READMEs, issues, and merged pull requests.
Own the deployment boundary. Run locally over stdio or host a loopback HTTP
service; select providers and secret storage yourself; optionally cache completed
fetches in a private Valkey sidecar.
Fail explicitly. Unsupported provider choices, missing optional packages, and
unavailable credentials fail in the tool result. Tavily mappings, Firecrawl fallback
filter gaps, and Exa category migrations are caller-visible; other Exa filter drops
are written to server diagnostics.
How it compares with built-in agent web tools
Built-in capabilities vary by agent and can improve over time. The useful comparison
is therefore not “all defaults are bad”; it is whether you need a portable,
inspectable retrieval layer with controls your current client does not expose.
Need
Typical bundled search/fetch surface
web-retrieval-mcp
Search output
Client-defined result or synthesis format
Default search returns source-separated blocks plus Sources; Exa deep modes add a labeled synthesis
Backend choice
Provider and routing are usually managed by the client
Exa or Tavily search, with a disclosed Firecrawl fallback
Difficult pages
One client-specific access path
Indexed content, guarded local browser rendering, Tavily Extract, and Firecrawl tiers
Retrieval intent
Usually search or page text
Full body, concise summary, or query-grounded page answer
Research discovery
General web index
Dedicated paper and developer-source tools in addition to web search
Freshness and filters
Whatever the client exposes
Date windows, hour-level recency, domains, categories, result limits, and forced freshness
Auditability
Client-specific
Fetch tier/cache/truncation disclosures; Tavily and Firecrawl search-fallback labels; explicit filter notices where supported
Operations
Client-defined; often hosted or opaque
Local stdio or self-hosted loopback HTTP, optional private UDS cache
Security policy
Client-specific
Initial URL validation plus guarded browser requests and redirects
Use it as an independent complementary retrieval lane by default. If you want it
to replace Claude Code's built-in WebSearch and WebFetch, the package also includes
an optional, previewable hook installer.
Quick start
Python 3.10 or newer is required.
1. Install
For the complete feature set in an isolated environment:
--include-deps exposes Camoufox's own command from the isolated pipx environment;
the second command downloads its managed browser. If you do not want local rendering,
install the base package or selected non-render extras and skip the browser download.
For a lean Exa + Firecrawl installation:
pipx install web-retrieval-mcp
With uv installed, you can run the base
package in a temporary isolated environment without a permanent install:
You do not need all three keys to start the server. Tools resolve credentials lazily
and report what a selected route is missing. See Credential storage
for a key file and OS secret-store alternatives.
All three providers advertised no-card entry allocations when checked on 2026-09-01;
plans and unit costs can change, so check their live pricing before estimating a
workload:
That configuration assumes the executable is on the client's PATH. For an uvx
launch, use "command": "uvx" and "args": ["web-retrieval-mcp"].
If a desktop client does not inherit your shell environment, use the private key file
described below or place literal key values in the client's protected environment
configuration. Do not rely on ${VARIABLE} interpolation unless your client documents
that behavior.
ChatGPT desktop and clients with different setup surfaces can connect through the MCP
transport their current version supports. This server uses stdio by default and can
also expose Streamable HTTP; consult the client's documentation for its configuration
format and supported transports.
4. Give the agent a real task
Once the MCP server is connected, prompts can stay natural:
Search the web for the latest primary documentation about Python package metadata.
Keep each source separate and preserve its URL.
Fetch https://example.com/report, force a fresh retrieval, and answer only:
What methodology did the authors use?
Find AI/ML papers about retrieval reranking, inspect the strongest paper for its
reported benchmark result, then find newer work that cites it.
Search merged pull requests and documentation for the origin of this exact error:
"transport closed".
Six read-only tools
All tools carry MCP's read-only annotation. MCP clients receive the complete generated
input schema and descriptions when they connect.
Developer documentation, repository READMEs, issues, and merged pull requests
web_search supports relevance or date-oriented filters, domain inclusion/exclusion,
publication windows, generated per-result summaries, and Exa's fast/deep search modes.
Tavily receives equivalent controls where its API supports them; approximations and
dropped controls are disclosed in the output.
The research-paper index is arXiv-oriented and best suited to AI/ML. For scholarly
work outside that scope, use web_search(category="publication").
How retrieval works
Search routing
Exa is the default search provider. Select Tavily for one call with
provider="tavily", or globally:
export WEB_SEARCH_PROVIDER=tavily
Only exa and tavily are accepted. If the selected provider fails, the server tries
Firecrawl and names both the failed primary provider and serving fallback in the
result.
Fetch routing
For an ordinary full-body request in automatic mode:
Exa indexed contents
↓ unusable or unavailable
guarded local Camoufox browser
↓ unusable or unavailable
optional Tavily Extract
↓ unusable or unavailable
Firecrawl
render="auto" uses that adaptive cascade.
render="never" forbids the local browser.
render="always" starts with the local browser and skips Exa; optional Tavily and
Firecrawl remain backstops.
mode="concise" asks for a compact generated summary.
question="…" asks for a grounded answer rather than the whole page.
max_age_hours=0 forces fresh provider work; -1 requests Exa's always-use-cache
behavior; positive values set a freshness window.
Explicit full-body requests above Exa's body-size ceiling start with a tier capable
of satisfying the requested size rather than silently returning a short body.
Enable Tavily Extract per call with tavily=true, or globally:
export WEB_FETCH_TAVILY_TIER=1
Tavily Extract is a full-body tier. The server skips it when it cannot honor a
concise, question-answer, or explicit freshness contract.
Credential storage
The server resolves each API key lazily in this order:
EXA_API_KEY, FIRECRAWL_API_KEY, or TAVILY_API_KEY in the process environment.
A dotenv-style private key file.
The optional Python keyring package.
macOS Keychain or Linux Secret Service command-line clients.
Override the file with WEB_RETRIEVAL_MCP_ENV_FILE, or its directory with
WEB_RETRIEVAL_MCP_CONFIG_DIR. On POSIX systems the file must not be readable by
group or other users:
Successful eligible fetches are stored for 24 hours. Credential-bearing URLs,
userinfo, forced or positive freshness, and render="always" bypass replay. Cache
errors fail open to normal provider retrieval. Default auto enables caching only
when the Valkey client is installed on a non-Windows host; on still attempts the
configured Unix socket and fails open on errors, while off disables it. See the cache guide
for the privacy model, singleflight behavior, and a minimal sidecar configuration.
Streamable HTTP transport
stdio is the default and recommended transport for a local MCP client:
HTTP transport has no built-in authentication. Keep it on loopback unless you add an
authenticated perimeter and suitable network controls.
Optional Claude Code replacement policy
The package includes a PreToolUse hook that can deny Claude Code's built-in
WebSearch and WebFetch, directing agents to this MCP server instead. Preview the
exact settings change first:
The installer is idempotent and backs up an existing settings file before writing.
This optional hook is the only Claude Code-specific integration in the package. The
MCP server itself remains client-neutral and works as a complementary tool without it.
The release gate builds wheel and sdist artifacts, installs the wheel into a clean
virtual environment, exercises the installed MCP command against loopback provider
doubles, and runs the unit and transport suites. Browser or SSRF behavior changes also
require python test_ssrf_redirect_live.py. See the testing contract
for the feature matrix and acceptance contract.
Security model
Only http and https initial URLs are accepted.
Initial hosts must resolve exclusively to globally routable addresses.
The local browser observes document requests and validates redirect hops before
returning content to the caller.
Credentials are resolved in process and redacted from errors and displayed URLs.
Provider responses and page bodies are untrusted data; an agent must not treat
instructions embedded in retrieved content as authority.
Application checks cannot prove that no packet reaches a private address during a DNS
rebinding race. Deployments with that threat model need a validating forward proxy or
equivalent network egress policy. See the security policy, and report suspected
vulnerabilities through GitHub's private vulnerability reporting rather than a public
issue.
Frequently asked questions
What is web-retrieval-mcp?
It is an open-source MCP server that gives AI agents reliable web search, tiered web
page retrieval, research-paper discovery, and developer-source search. It works over
local stdio with any compatible MCP client and can also serve stateless Streamable
HTTP.
Is this an MCP web search server or an MCP web scraping server?
Both. web_search discovers and ranks pages; web_fetch extracts readable content
from one URL and can use a real local browser for JavaScript-heavy pages. The server
also exposes specialist paper and developer indexes that ordinary web scraping does
not provide.
Does it replace an agent's built-in web search?
It can, but it does not have to. The tool descriptions recommend an independent
complementary lane because two retrieval systems can provide useful source diversity.
Claude Code users can opt into the bundled replacement hook when they want one enforced
route.
Which AI agents and MCP clients can use it?
The server exposes standard MCP over stdio by default and optional Streamable HTTP.
Documented client examples include Codex CLI and the Codex IDE extension, Claude Code,
Claude Desktop, Cursor, and ChatGPT desktop; other MCP-capable agents are not excluded.
Compatibility depends on the client version and the MCP transport it supports, so use
the matching client documentation alongside the server commands above.
Why use Exa, Firecrawl, Tavily, and Camoufox together?
They cover different failure modes. Exa provides indexed search and contents; Tavily
is an alternate search provider and optional extractor; Camoufox renders pages locally;
Firecrawl provides the final web fallback plus specialist research and developer
indexes. The server chooses among them according to the request and discloses the tier
that succeeded.
Do I need every provider key?
No. Exa is the default for general search and first-tier contents. Firecrawl enables
fallbacks and all four research tools. Tavily is optional. The server lists all six
tools even when an optional route is not configured and returns an actionable error if
a call selects an unavailable capability.
Is web-retrieval-mcp free?
The project is MIT-licensed. Provider usage is billed under your own accounts; each
provider currently offers an entry allocation, but quotas and prices can change.
Camoufox runs locally without a metered retrieval API.
Which operating systems are supported?
The base Python server is platform-independent. Environment and key-file credentials
work across Linux, macOS, and Windows. Valkey UDS caching is non-Windows, and local
browser availability follows the supported Camoufox/Playwright platforms.
Does it support RAG and autonomous research agents?
Yes. The source-separated result contract, explicit URLs, controllable text budgets,
grounded page questions, specialist indexes, and deterministic provenance make the
tools suitable as a retrieval layer for RAG pipelines and research agents. The server
returns evidence; the calling application remains responsible for evaluation,
citation, and prompt-injection handling.
Contributions are welcome. Keep stdout reserved for JSON-RPC, send diagnostics to
stderr, add public-boundary acceptance coverage for changed behavior, and run
./run-tests.sh before opening a pull request.
Noch keine Tool-Liste. Remote-Server erhalten eine nach dem ersten Health-Check.
FAQ
Fragen
Wie installiere ich Web Retrieval?
Kopieren Sie auf dieser Seite die stdio-Konfiguration für Cursor, Claude Desktop, Claude Code oder VS Code. Sie nutzt die Installation des Herausgebers, keinen PlugMCP-Proxy.
Benötigt Web Retrieval einen API-Schlüssel?
Der Herausgeber nennt diese Umgebungsvariablen: EXA_API_KEY, FIRECRAWL_API_KEY.
Welchen Transport verwendet Web Retrieval?
Web Retrieval läuft lokal über stdio.
Wann wurde Web Retrieval zuletzt geprüft?
Zuletzt geprüft: 2026-11-09. MCP web search + tiered web fetch for AI agents (Exa, Firecrawl), SSRF-guarded, cross-platform.