Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.
Multi-engine web search, fetch, browser control. 11 tools, keyless lanes.
doctor — Run this first. Reports which lanes are usable right now and the exact command to enable the rest. Distinguishes "broken" from "needs optional setup". Never prints key values.
search — Open research questions. depth defaults to 3 when a model lane is configured (more independent sources; measured) and 1 otherwise. Depth also sets how many pages are fetched: max(5, depth*2), ceiling 8. synthesize returns a short, cited answer that never invents: it says which source backs each claim, surfaces disagreement instead of averaging it, and marks itself truncated when it ran out of budget.
fetch — URL to markdown/text. Direct fetch with auto-fallback to a Chromium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (offset). Optional distill (see below). For structured JSON use extract.
fetch_many — Fetch several URLs in one call. Dedups, caps concurrency, honours robots.txt, shares the cache, returns per-URL success/status/content plus counts. Use when you have 2+ URLs; use fetch for one.
crawl — Breadth-first site crawl with hard caps (per-page cap, depth cap, robots.txt, dedup, concurrency). Returns each page plus a stop_reason so you know whether the site was exhausted or a cap stopped it. Use for "read this whole docs site".
extract_links — Every link on a page with anchor text, title and rel. Resolves relative hrefs, drops javascript:/mailto:, dedups. Use before crawl or to find a "next" button.
extract_table — HTML tables to json, csv or markdown, by selector and index. Reports tables_found so you can pick one.
download — Save a URL to disk; returns the local path, byte count, sha256 and content type. Writes only inside dest_dir and refuses path traversal. Marks truncated if the byte cap stopped the transfer.
diff_page — Has this page changed since it was cached? Returns changed/old_hash/new_hash plus added and removed lines. First call for a URL returns had_previous: false.
screenshot — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
browser_action — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use fetch. Pass session_id to reuse a persistent tab.
browser_session — Create, list, inspect or close a persistent browser tab. Without one, every browser_action call gets a fresh tab and closes it, so logins and cookies cannot survive. See below.
form_fill_submit — Fill many fields, submit, and read the result in one call. Verifies each element exists, reads back each typed value, and refuses to submit if a value did not stick. Reports effect (navigated / content_changed / none) so a submit that silently did nothing is visible rather than assumed.
paginate — Click a "next" control, collecting a value per page, up to a cap. Stops honestly when the control is missing or has no effect, and reports stop_reason.
wikipedia — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
arxiv — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch papers).
extract — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
pdf_extract — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
map_site — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
ping — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.
What replaced the throwaway scripts
Every one of these exists because an agent otherwise writes a one-off script:
The task
The tool
Fetch 20 URLs
fetch_many
Read a whole docs site
crawl
Get every link on a page
extract_links
Pull a pricing table into CSV
extract_table
Save a PDF and get its path
download
"Did this page change?"
diff_page
Log in, then act across several calls
browser_session + browser_action(session_id=...)
Fill and submit a form
form_fill_submit
Walk "next" through pages
paginate
Persistent browser sessions
browser_action opens one fresh tab per call and closes it afterwards. That is
right for a single read, but it makes any interactive workflow impossible: no
login, no cookies, no scroll position, no multi-step flow across calls.
A session is a named, long-lived tab:
browser_session(action="create", label="login", url="https://site.example/login")
# -> {"session_id": "login-1a2b3c4d", ...}
browser_action(action="type", selector="#email", text="[email protected]", session_id="login-1a2b3c4d")
browser_action(action="click", selector="#submit", session_id="login-1a2b3c4d")
browser_action(action="text", session_id="login-1a2b3c4d") # same tab, still logged in
browser_session(action="close", session_id="login-1a2b3c4d")
Sessions are opt-in: without session_id, behaviour is exactly as before. They
are capped at 8 with least-recently-used eviction, tabs are always closed
(including on eviction), and metadata is mirrored to
~/.cache/websearch/sessions.json so list can explain what survived a
restart. Sessions need a browser, so they are local-only.
In-tool distillation (optional, off by default)
A fetched page can be 50k characters. The calling model pays for all of it even
when it needed one fact. fetch(distill=true) runs a small model inside the
tool first and returns a compressed result:
fetch(url="https://example.com/long-report", distill=True,
distill_question="What are the pricing tiers?")
The result never pretends to be the page:
Field
Meaning
distill.distilled
whether a summary was produced
distill.raw_chars
size of the original text
distill.grounded
whether every number and quoted span was found in the source
distill.grounding.numbers_unverified
figures in the summary that do not appear in the source
distill.grounding.quotes_unverified
quoted spans that are not verbatim
distill.how_to_get_raw
how to get the untouched text: distill=false
Below the size threshold (default 4000 chars) distillation is skipped and the
raw text is returned, because a summary would cost more than it saves. On any
failure the raw text is returned with distill_skipped explaining why - a
broken summary is never returned in place of the page.
Variable
Default
Meaning
DISTILL_BASE_URL
http://127.0.0.1:11435/v1
any OpenAI-compatible endpoint
DISTILL_MODEL
reflection/beam:free
model id
DISTILL_API_KEY
none
bearer token; without it the lane stays off
DISTILL_RETRIES
3
attempts per call (1-10)
DISTILL_MAX_TOKENS
3000
initial completion budget
DISTILL_MAX_TOKENS_CEILING
12000
ceiling when escalating after empty content
DISTILL_TIMEOUT
90
seconds per request
DISTILL_THRESHOLD_CHARS
4000
below this, skip distillation
The synthesis prompt (the answer you actually read)
search does not just rank results; it writes the answer. That answer is
generated from the retrieved text alone, under rules that exist to stop the
three failure modes of a search summary:
Failure
What the prompt does
Invention
Nothing from the model's own knowledge. A claim with no citable source is deleted, not softened. If the results do not answer the question, saying so is the correct answer.
Bias
Rank is not evidence. The prompt says so explicitly, caps how many sources can come from one domain, requires disagreement between sources to be reported rather than averaged into a number that appears in no source, corrects false premises in the question, and labels vendor claims as vendor claims.
Injection
Results are wrapped in <results> and declared untrusted data. A page that says "ignore previous instructions" is data, not instruction; the model must exclude it and say it did.
Concision is structural, not a wish: a sentence ceiling, an answer-first rule,
and a named ban on filler phrasing ("it is important to note", "overall", ...).
Everything is verifiable rather than claimed:
the result carries prompt_version, so a quality change can be tied to a
revision instead of guessed at
temperature is 0, so the same sources give the same text
finish_reason and truncated are returned; a half sentence is labelled
instead of looking like a complete thought
diversity reports the source mix and whether the per-domain cap had to relax
as_of reports the newest publication date among the sources, and undated
sources are labelled "Published: not stated", so freshness is never implied
duplicates_removed counts syndicated copies collapsed before synthesis, and
figures_unverified lists numbers the answer contains that appear in no
cited source (deterministic, no model call)
unusable_sources names the results that could not become evidence - a 404, an
error page, or a page with under 400 chars of text - with the reason, and
unreadable_sources counts the ones with no text at all
multi-hop: when the first answer admits a weakness (single source, a claim
it could not tie to a source, or a stated gap), one extra round runs on the
subquery that targets the gap. The response records follow_ups and marks
synthesis.round = 2. Bounded: once per call, only under a 75s budget, and
follow_up=false disables it
a second model pass (grounded) re-reads the answer against the exact text
the first pass saw and flags sentences no cited source supports. It is marked
advisory because it is another model's opinion, not a proof
Tool annotations
All 20 tools carry MCP annotations (readOnlyHint, destructiveHint,
idempotentHint, openWorldHint) so a client can tell a read from an action:
search and fetch are read-only, download writes a local file without being
destructive, browser_session changes local browser state only, and
browser_action / form_fill_submit are marked destructive because a click or a
form submit can have an effect that cannot be undone from here.
When a call takes too long (deadline, admission, partials)
MCP clients cut a tool call at their own timeout (jcode: 300 s) and report a
bare Request timeout after 300s with no payload. Measured over one day on a
single busy server: 314 calls, median 2.7 s, p90 123 s, max 225 s, and 33 hard
client timeouts — the tail is the problem, not the median. So the server bounds
its own work instead of hoping the client waits:
Env
Default
Meaning
WEBSEARCH_TOOL_DEADLINE
240
Seconds one tool call may run. Keep it under the client timeout. 0 disables the bound
WEBSEARCH_MAX_CONCURRENT
8
Simultaneous tool calls. Extra callers wait, then get busy. 0 = unlimited
WEBSEARCH_QUEUE_WAIT
20
Seconds a call may wait for a free slot before being answered busy
WEBSEARCH_SLOW_CALL_S
30
A successful call at or above this logs SLOW <tool> took Ns to stderr
WEBSEARCH_RERANK_TIMEOUT
60
Reranker read timeout. On timeout the call degrades to engine ranking (see below)
WEBSEARCH_RERANK_COOLDOWN
20
After a reranker timeout, skip it for this long instead of paying the timeout again
WEBSEARCH_RERANK_QUEUE_WAIT
15
Longest a call waits behind other calls for the single rerank worker before answering with base order. 0 = wait without bound
WEBSEARCH_OCR_IDLE_UNLOAD
90
Idle seconds before the cached OCR converter (and its ~1 GB of models) is dropped
WEBSEARCH_OCR_MIN_FREE_MB
900
Refuse OCR below this much available memory (ocr_skipped: "low_memory") instead of swap-thrashing the whole server
WEBSEARCH_DDG_TIMEOUT
0
Optional per-call timeout for the blocking ddgs network work; 0 = no extra bound
Three response shapes exist because of this, and each is deliberate:
{"success": false, "deadline_exceeded": true, "deadline_s": 240,
"elapsed_s": 241.3, "retryable": true,
"partial": {"success": true, "results": [...]},
"partial_note": "Partial result captured before the deadline: ..."}
partial — the stages that finished are still returned, because a cited
source list is worth more than an empty retry. search publishes its
snapshot as soon as ranking is done.
deadline_exceeded — the call was cut by us, not by the client, and it says
which knob to raise. Narrow the request (depth, count,
synthesize=false) before raising the deadline: the deadline protects the
server, it does not make the work cheaper.
busy + retryable — admission was full for WEBSEARCH_QUEUE_WAIT.
Retrying shortly is correct; queueing behind an unbounded swarm is what turns
one slow call into many timed-out ones.
Reranking degrades honestly rather than failing: search returns a reranker
field only when ranking was not the requested path (worker unavailable, timeout,
cooldown, bad reply), plus the ordering that was used instead. On this box the
reranker's own p90 was 56 s, so a 60 s timeout trades ~10 % of reranked calls
for never stalling a swarm call for the full timeout.
The worker is one socket serving one request at a time, so a burst queues: with
8 calls in flight the last one waits for seven completions. Measured on 12 lanes
that queue alone pushed a search to 253 s, which is why the wait for the worker
is now capped (WEBSEARCH_RERANK_QUEUE_WAIT, 15 s). A call that cannot get the
worker in time answers with base order and says so, instead of turning a 4 s
ranking step into a 4-minute search.
OCR (scanned PDFs) runs on a dedicated single-slot pool with a cached converter
and a memory guard, and reports ocr_ms. asyncio.wait_for cannot cancel a
thread, so a per-call to_thread build kept loading docling models after the
request was abandoned; one cached converter plus one slot removes both the
model reload and the pile-up.
Why the answer is prose, not JSON
The tool envelope is already structured: answer, sources[], citations_used,
grounded, diversity, as_of. Asking the model to also emit JSON moves
structure from a place where it is generated deterministically into a place where
it is asserted, and measured behaviour is worse: a required key_facts array
gets padded with restatements of a single fact, which reads as corroboration.
JSON parsed 3/3 in testing, so this is not a reliability argument - it is a
content-quality and duplication argument. See exec/websearch-format-test/.
Which model writes the answer
The in-tool model comes first by default, because it is free at the point of
use. Groq is the fallback if it fails. Both are plain OpenAI-compatible chat
endpoints, so one call path serves both.
Env
Default
Meaning
WEBSEARCH_SYNTH_PROVIDER
local
local (in-tool model first), groq (Groq first), auto (either, in-tool preferred). Applies to every model call, not just synthesis
WEBSEARCH_MODEL_FALLBACK
on
off = use only the preferred endpoint and report an error if it fails, instead of falling through to the other provider. Set it when the model choice is a requirement (cost, privacy, compliance), not a preference
WEBSEARCH_MIN_MODEL_TOKENS
512
Floor on the token budget of any model request. Reasoning models spend budget on reasoning_content first, so a 64-token label request comes back empty; the floor is what makes small helper calls work
WEBSEARCH_DEFAULT_DEPTH
auto
Pins the default depth. Auto = 3 when a model lane exists, else 1
WEBSEARCH_ENRICH_MAX_PAGES
8
Ceiling on page fetches per search. The count itself is max(5, depth*2)
WEBSEARCH_SYNTH_TOKENS
1200
Answer budget. On finish_reason=length the result says truncated
WEBSEARCH_SYNTH_MAX_SOURCES
8
Ceiling on sources in the context
WEBSEARCH_DOMAIN_CAP
2
Max sources from one registrable domain
WEBSEARCH_MIN_SOURCES
3
Floor before the domain cap relaxes
WEBSEARCH_SYNTH_SNIPPET_CHARS
1500
Chars of each source given to the model
doctor() reports all of these resolved (env.defaults) plus the active prompt
versions (env.prompts), and lists synthesis as its own lane, so "why is there
no answer?" has an answer instead of a blank field.
What a search costs in model calls
Search is not a model-free operation, and pretending otherwise made it
unpredictable. Every call that needs a model goes through llm.py, and every
response says where the tokens went:
the query depends on chat history (history argument)
no history, no call
classify
always (biases engine selection, never suppresses search)
google_ai_only
expand
short query, depth 1
query_expand=false
decompose
depth >= 2
query_expand=false
synthesis
results exist
synthesize=false
verify
after synthesis
synthesize=false
Two consequences worth knowing:
One endpoint preference, everywhere. Before this was unified, synthesis
used the free in-tool model while rewrite/classify/decompose called Groq
directly and returned nothing without a Groq key. The result was a
model-written answer produced from an unplanned, literal query. Now the
in-tool model plans the search too, and Groq is only the fallback
(WEBSEARCH_SYNTH_PROVIDER).
Small budgets are a trap. A reasoning model spends tokens on
reasoning_content before it emits content, so a 64-token "output one
label" request returns 200 with an empty message. Callers saw their fallback
value and it looked like a real classification. Every request now gets at
least WEBSEARCH_MIN_MODEL_TOKENS (512), and an unfilled request is reported
and logged instead of being mistaken for an answer.
What works out of the box, and what is optional
Everything keyless works immediately after pip install. No API keys, no browser, no GPU: search (DuckDuckGo, Bing RSS, Wikipedia, arXiv, Stack Exchange, GitHub, HN, grep.app, Google News RSS), fetch, map_site, pdf_extract, screenshot.
Four lanes need optional setup. Run doctor() and it tells you which are live and prints the exact command for the rest:
Lane
Needs
On a headless server
Google AI Mode (search(google_ai_only=True))
a Chromium-based browser with --remote-debugging-port+ a signed-in Google account in that profile
unavailable — no signed-in browser exists there
browser_action
same browser, no Google login needed
unavailable
CDP fallback for bot-walled pages
same browser
unavailable; those pages return a terminal verdict instead of hanging
Cross-encoder reranking
an optional local worker (separate download, ~500MB model)
works if installed; otherwise scoring falls back to blended relevance
Then open google.com in that window and sign in once (needed only for Google AI Mode). Point elsewhere with WEBSEARCH_CDP_URL=http://host:port.
# confirm
doctor() # per-lane report
search(query="...", google_ai_only=True) # explicit error + fix list if no browser
The browser lanes are local-only by design. On a server the HTTP lanes are unaffected, so the server is still fully useful — it just cannot query Google AI Mode. Without a browser, google_ai_only=True returns a structured error naming the cause and the fix, never an empty result set.
Setup
Requires Python 3.14+.
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp run.example run # fill in your API keys (run is gitignored — never commit it)
chmod +x run
fallback for every model call: rewrite, classify, expand, decompose, synthesis, grounding checks
Optional with an in-tool model (it becomes the fallback); required for those steps if no in-tool model is configured
TAVILY_KEYS
Tavily engine lane
Optional
NV_KEY
NVIDIA embedding endpoint
Optional
Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.
The websearch-mcp console script speaks MCP over stdio:
{ "websearch": { "command": "websearch-mcp" } }
Docker
docker build -t websearch-mcp .
docker run --rm -i websearch-mcp
Keys are passed at run time, never baked in: -e GROQ_API_KEYS=....
Heavy extras: docker build --build-arg EXTRAS="[all]" -t websearch-mcp .
Notes
Reranker (optional, local): ranking uses a local gte-reranker-modernbert-base cross-encoder over a Unix socket (RERANKER_MODEL overrides the model). Not bundled with the pip package — set RERANKER_WORKER=/path/to/worker.py to use one. Without it, ranking falls back to blended relevance scores; everything still works. doctor() reports whichever is in effect.
CDP browser (optional): bot-walled pages escalate to a Chrome DevTools endpoint (WEBSEARCH_CDP_URL, default 127.0.0.1:9222) when available; without it those pages return a terminal verdict instead of hanging.
Caching: fetch results cache for 1h (cache_ttl=0 forces fresh). No database, no state beyond the cache dir.
Safety: all user-URL fetches are SSRF-validated (private-IP blocking + redirect-hop revalidation + robots.txt gate).
One model call path (llm.py): synthesis, verification, query rewrite, intent classification, subquery decomposition and query expansion all go through llm.chat, so the endpoint preference, retry policy, budget escalation and counting are the same everywhere. On Groq, synthesis + grounding + extract-llm use openai/gpt-oss-120b and rewrite/classify/expand/decompose use openai/gpt-oss-20b (the previous llama-3.1-8b-instant was retired by Groq and now 404s); constants GROQ_SMART_MODEL/GROQ_CHEAP_MODEL in config.py.
Model calls are reported, not hidden: every search returns model_calls, e.g. {"rewrite":1,"classify":1,"decompose":1,"synthesis":1,"verify":1}. Search is not free of model use: it plans the query, writes the answer and verifies the answer. synthesize=false removes the last two, query_expand=false and depth 1 remove decomposition.
Keyless lanes (all advisory-only): OSV.dev vuln vertical + NVD CVE enrichment; grep.app MCP code search (Sourcegraph backup); Bing Web RSS + Bing News RSS (Brave's scrape is rate-limited, GDELT 429s from some networks); Wikipedia vertical; Jina Reader pre-CDP tier; Wayback CDX-history recovery in enrich(); Common Crawl domain-survey assist in map_site.
Local reranker: a gte-reranker-modernbert-base cross-encoder runs as one shared worker over a Unix socket; either server spawns it on demand and it unloads when idle. Optional — without it results are engine-ranked only.
Marginalia: best-effort lane only (shared public key pool, 503s under load); independent-index recall, no dedicated key wired ([email protected] for one).