Websearch

websearch MCP Server

mcp-name: io.github.SK-DEV-AI/websearch

Companion agent skills: SK-DEV-AI/skills · npx skills add SK-DEV-AI/skills

Multi-engine web search + fetch MCP server. Parallel fan-out across DuckDuckGo, Bing Web/News RSS, Tavily, Wikipedia, arXiv, Google News RSS, Brave (last resort — rate-limited from some networks), Google AI Mode, and intent-gated vertical lanes (MDN, Stack Exchange, GitHub, HN, Semantic Scholar, OSV vuln, grep.app/Sourcegraph code, GDELT news) — with consensus-merge dedup, a local cross-encoder reranker, and Groq synthesis with grounding checks.

Tools (20)

• doctor — Run this first. Reports which lanes are usable right now and the exact command to enable the rest. Distinguishes "broken" from "needs optional setup". Never prints key values.
• search — Open research questions. depth defaults to 3 when a model lane is configured (more independent sources; measured) and 1 otherwise. Depth also sets how many pages are fetched: max(5, depth*2), ceiling 8. synthesize returns a short, cited answer that never invents: it says which source backs each claim, surfaces disagreement instead of averaging it, and marks itself truncated when it ran out of budget.
• fetch — URL to markdown/text. Direct fetch with auto-fallback to a Chromium CDP browser for blocked/JS pages. PDF/EPUB/DOCX supported. Paginated (offset). Optional distill (see below). For structured JSON use extract.
• fetch_many — Fetch several URLs in one call. Dedups, caps concurrency, honours robots.txt, shares the cache, returns per-URL success/status/content plus counts. Use when you have 2+ URLs; use fetch for one.
• crawl — Breadth-first site crawl with hard caps (per-page cap, depth cap, robots.txt, dedup, concurrency). Returns each page plus a stop_reason so you know whether the site was exhausted or a cap stopped it. Use for "read this whole docs site".
• extract_links — Every link on a page with anchor text, title and rel. Resolves relative hrefs, drops javascript:/mailto:, dedups. Use before crawl or to find a "next" button.
• extract_table — HTML tables to json, csv or markdown, by selector and index. Reports tables_found so you can pick one.
• download — Save a URL to disk; returns the local path, byte count, sha256 and content type. Writes only inside dest_dir and refuses path traversal. Marks truncated if the byte cap stopped the transfer.
• diff_page — Has this page changed since it was cached? Returns changed/old_hash/new_hash plus added and removed lines. First call for a URL returns had_previous: false.
• screenshot — ARIA accessibility snapshot (LLM-readable page text and interactive structure).
• browser_action — Drive a real browser page (navigate, click, type, press a key, evaluate JS, read HTML/text, ARIA tree, screenshot). Use when a page needs interaction; for plain reading use fetch. Pass session_id to reuse a persistent tab.
• browser_session — Create, list, inspect or close a persistent browser tab. Without one, every browser_action call gets a fresh tab and closes it, so logins and cookies cannot survive. See below.
• form_fill_submit — Fill many fields, submit, and read the result in one call. Verifies each element exists, reads back each typed value, and refuses to submit if a value did not stick. Reports effect (navigated / content_changed / none) so a submit that silently did nothing is visible rather than assumed.
• paginate — Click a "next" control, collecting a value per page, up to a cap. Stops honestly when the control is missing or has no effect, and reports stop_reason.
• wikipedia — Full Wikipedia API client: articles, summaries, categories, links, pageviews, revisions, backlinks, geosearch, recent changes, random pages.
• arxiv — arXiv paper search with boolean/phrase/wildcard query syntax (single-source; multi-source paper search lives in codesearch papers).
• extract — Structured JSON from a page via CSS (fast, free), LLM, or regex strategies. Repeat calls cached.
• pdf_extract — PDF to markdown/json/html. PyMuPDF fast path for text-layer PDFs, Docling+OCR fallback for scanned PDFs.
• map_site — Survey a site's pages via sitemap XML (primary) + HTML link fallback, before fetching individually.
• ping — Connectivity check for the internet and key search endpoints. Use before expensive calls when uncertain.

What replaced the throwaway scripts

Every one of these exists because an agent otherwise writes a one-off script:

Persistent browser sessions

browser_action opens one fresh tab per call and closes it afterwards. That is right for a single read, but it makes any interactive workflow impossible: no login, no cookies, no scroll position, no multi-step flow across calls.

A session is a named, long-lived tab:

browser_session(action="create", label="login", url="https://site.example/login")
# -> {"session_id": "login-1a2b3c4d", ...}
browser_action(action="type", selector="#email", text="[email protected]", session_id="login-1a2b3c4d")
browser_action(action="click", selector="#submit", session_id="login-1a2b3c4d")
browser_action(action="text", session_id="login-1a2b3c4d")   # same tab, still logged in
browser_session(action="close", session_id="login-1a2b3c4d")

Sessions are opt-in: without session_id, behaviour is exactly as before. They are capped at 8 with least-recently-used eviction, tabs are always closed (including on eviction), and metadata is mirrored to ~/.cache/websearch/sessions.json so list can explain what survived a restart. Sessions need a browser, so they are local-only.

In-tool distillation (optional, off by default)

A fetched page can be 50k characters. The calling model pays for all of it even when it needed one fact. fetch(distill=true) runs a small model inside the tool first and returns a compressed result:

fetch(url="https://example.com/long-report", distill=True,
      distill_question="What are the pricing tiers?")

The result never pretends to be the page:

Below the size threshold (default 4000 chars) distillation is skipped and the raw text is returned, because a summary would cost more than it saves. On any failure the raw text is returned with distill_skipped explaining why - a broken summary is never returned in place of the page.

The synthesis prompt (the answer you actually read)

search does not just rank results; it writes the answer. That answer is generated from the retrieved text alone, under rules that exist to stop the three failure modes of a search summary:

Concision is structural, not a wish: a sentence ceiling, an answer-first rule, and a named ban on filler phrasing ("it is important to note", "overall", ...).

Everything is verifiable rather than claimed:

• the result carries prompt_version, so a quality change can be tied to a revision instead of guessed at
• temperature is 0, so the same sources give the same text
• finish_reason and truncated are returned; a half sentence is labelled instead of looking like a complete thought
• diversity reports the source mix and whether the per-domain cap had to relax
• as_of reports the newest publication date among the sources, and undated sources are labelled "Published: not stated", so freshness is never implied
• duplicates_removed counts syndicated copies collapsed before synthesis, and figures_unverified lists numbers the answer contains that appear in no cited source (deterministic, no model call)
• unusable_sources names the results that could not become evidence - a 404, an error page, or a page with under 400 chars of text - with the reason, and unreadable_sources counts the ones with no text at all
• multi-hop: when the first answer admits a weakness (single source, a claim it could not tie to a source, or a stated gap), one extra round runs on the subquery that targets the gap. The response records follow_ups and marks synthesis.round = 2. Bounded: once per call, only under a 75s budget, and follow_up=false disables it
• a second model pass (grounded) re-reads the answer against the exact text the first pass saw and flags sentences no cited source supports. It is marked advisory because it is another model's opinion, not a proof

Tool annotations

All 20 tools carry MCP annotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint) so a client can tell a read from an action: search and fetch are read-only, download writes a local file without being destructive, browser_session changes local browser state only, and browser_action / form_fill_submit are marked destructive because a click or a form submit can have an effect that cannot be undone from here.

When a call takes too long (deadline, admission, partials)

MCP clients cut a tool call at their own timeout (jcode: 300 s) and report a bare Request timeout after 300s with no payload. Measured over one day on a single busy server: 314 calls, median 2.7 s, p90 123 s, max 225 s, and 33 hard client timeouts — the tail is the problem, not the median. So the server bounds its own work instead of hoping the client waits:

Three response shapes exist because of this, and each is deliberate:

{"success": false, "deadline_exceeded": true, "deadline_s": 240,
 "elapsed_s": 241.3, "retryable": true,
 "partial": {"success": true, "results": [...]},
 "partial_note": "Partial result captured before the deadline: ..."}

• partial — the stages that finished are still returned, because a cited source list is worth more than an empty retry. search publishes its snapshot as soon as ranking is done.
• deadline_exceeded — the call was cut by us, not by the client, and it says which knob to raise. Narrow the request (depth, count, synthesize=false) before raising the deadline: the deadline protects the server, it does not make the work cheaper.
• busy + retryable — admission was full for WEBSEARCH_QUEUE_WAIT. Retrying shortly is correct; queueing behind an unbounded swarm is what turns one slow call into many timed-out ones.

Reranking degrades honestly rather than failing: search returns a reranker field only when ranking was not the requested path (worker unavailable, timeout, cooldown, bad reply), plus the ordering that was used instead. On this box the reranker's own p90 was 56 s, so a 60 s timeout trades ~10 % of reranked calls for never stalling a swarm call for the full timeout.

The worker is one socket serving one request at a time, so a burst queues: with 8 calls in flight the last one waits for seven completions. Measured on 12 lanes that queue alone pushed a search to 253 s, which is why the wait for the worker is now capped (WEBSEARCH_RERANK_QUEUE_WAIT, 15 s). A call that cannot get the worker in time answers with base order and says so, instead of turning a 4 s ranking step into a 4-minute search.

OCR (scanned PDFs) runs on a dedicated single-slot pool with a cached converter and a memory guard, and reports ocr_ms. asyncio.wait_for cannot cancel a thread, so a per-call to_thread build kept loading docling models after the request was abandoned; one cached converter plus one slot removes both the model reload and the pile-up.

Why the answer is prose, not JSON

The tool envelope is already structured: answer, sources[], citations_used, grounded, diversity, as_of. Asking the model to also emit JSON moves structure from a place where it is generated deterministically into a place where it is asserted, and measured behaviour is worse: a required key_facts array gets padded with restatements of a single fact, which reads as corroboration. JSON parsed 3/3 in testing, so this is not a reliability argument - it is a content-quality and duplication argument. See exec/websearch-format-test/.

Which model writes the answer

The in-tool model comes first by default, because it is free at the point of use. Groq is the fallback if it fails. Both are plain OpenAI-compatible chat endpoints, so one call path serves both.

doctor() reports all of these resolved (env.defaults) plus the active prompt versions (env.prompts), and lists synthesis as its own lane, so "why is there no answer?" has an answer instead of a blank field.

What a search costs in model calls

Search is not a model-free operation, and pretending otherwise made it unpredictable. Every call that needs a model goes through llm.py, and every response says where the tokens went:

"model_calls": {"rewrite": 1, "classify": 1, "decompose": 1, "synthesis": 1, "verify": 1}

Two consequences worth knowing:

• One endpoint preference, everywhere. Before this was unified, synthesis used the free in-tool model while rewrite/classify/decompose called Groq directly and returned nothing without a Groq key. The result was a model-written answer produced from an unplanned, literal query. Now the in-tool model plans the search too, and Groq is only the fallback (WEBSEARCH_SYNTH_PROVIDER).
• Small budgets are a trap. A reasoning model spends tokens on reasoning_content before it emits content, so a 64-token "output one label" request returns 200 with an empty message. Callers saw their fallback value and it looked like a real classification. Every request now gets at least WEBSEARCH_MIN_MODEL_TOKENS (512), and an unfilled request is reported and logged instead of being mistaken for an answer.

What works out of the box, and what is optional

Everything keyless works immediately after pip install. No API keys, no browser, no GPU: search (DuckDuckGo, Bing RSS, Wikipedia, arXiv, Stack Exchange, GitHub, HN, grep.app, Google News RSS), fetch, map_site, pdf_extract, screenshot.

Four lanes need optional setup. Run doctor() and it tells you which are live and prints the exact command for the rest:

Enabling the browser lanes:

# any Chromium-family browser: chromium, google-chrome, brave, Helium, edge, vivaldi
chromium --remote-debugging-port=9222 --user-data-dir="$HOME/.mcp-browser"

Then open google.com in that window and sign in once (needed only for Google AI Mode). Point elsewhere with WEBSEARCH_CDP_URL=http://host:port.

# confirm
doctor()                          # per-lane report
search(query="...", google_ai_only=True)   # explicit error + fix list if no browser

The browser lanes are local-only by design. On a server the HTTP lanes are unaffected, so the server is still fully useful — it just cannot query Google AI Mode. Without a browser, google_ai_only=True returns a structured error naming the cause and the fix, never an empty result set.

Setup

Requires Python 3.14+.

python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp run.example run   # fill in your API keys (run is gitignored — never commit it)
chmod +x run

Point your MCP client at run (stdio):

{ "websearch": { "command": "/path/to/websearch/run" } }

API keys (run)

Everything else (DDG, Bing RSS, Wikipedia, arXiv, RSS) is keyless. Brave's HTML scrape is currently rate-limited from some networks, so Bing RSS is the primary general-web door and Brave is a last resort.

Install from PyPI

pip install skdev-websearch-mcp          # core install, small
pip install skdev-websearch-mcp[all]     # + browser/OCR/extract extras (multi-GB)

The websearch-mcp console script speaks MCP over stdio:

{ "websearch": { "command": "websearch-mcp" } }

Docker

docker build -t websearch-mcp .
docker run --rm -i websearch-mcp

Keys are passed at run time, never baked in: -e GROQ_API_KEYS=.... Heavy extras: docker build --build-arg EXTRAS="[all]" -t websearch-mcp .

Notes

• Reranker (optional, local): ranking uses a local gte-reranker-modernbert-base cross-encoder over a Unix socket (RERANKER_MODEL overrides the model). Not bundled with the pip package — set RERANKER_WORKER=/path/to/worker.py to use one. Without it, ranking falls back to blended relevance scores; everything still works. doctor() reports whichever is in effect.
• CDP browser (optional): bot-walled pages escalate to a Chrome DevTools endpoint (WEBSEARCH_CDP_URL, default 127.0.0.1:9222) when available; without it those pages return a terminal verdict instead of hanging.
• Caching: fetch results cache for 1h (cache_ttl=0 forces fresh). No database, no state beyond the cache dir.
• Safety: all user-URL fetches are SSRF-validated (private-IP blocking + redirect-hop revalidation + robots.txt gate).
• One model call path (llm.py): synthesis, verification, query rewrite, intent classification, subquery decomposition and query expansion all go through llm.chat, so the endpoint preference, retry policy, budget escalation and counting are the same everywhere. On Groq, synthesis + grounding + extract-llm use openai/gpt-oss-120b and rewrite/classify/expand/decompose use openai/gpt-oss-20b (the previous llama-3.1-8b-instant was retired by Groq and now 404s); constants GROQ_SMART_MODEL/GROQ_CHEAP_MODEL in config.py.
• Model calls are reported, not hidden: every search returns model_calls, e.g. {"rewrite":1,"classify":1,"decompose":1,"synthesis":1,"verify":1}. Search is not free of model use: it plans the query, writes the answer and verifies the answer. synthesize=false removes the last two, query_expand=false and depth 1 remove decomposition.
• Keyless lanes (all advisory-only): OSV.dev vuln vertical + NVD CVE enrichment; grep.app MCP code search (Sourcegraph backup); Bing Web RSS + Bing News RSS (Brave's scrape is rate-limited, GDELT 429s from some networks); Wikipedia vertical; Jina Reader pre-CDP tier; Wayback CDX-history recovery in enrich(); Common Crawl domain-survey assist in map_site.
• Local reranker: a gte-reranker-modernbert-base cross-encoder runs as one shared worker over a Unix socket; either server spawns it on demand and it unloads when idle. Optional — without it results are engine-ranked only.
• Optional keys (graceful keyless fallback): SE_KEY/STACKEXCHANGE_KEY (10k/day SE), S2_KEY/S2_API_KEY (dedicated S2 1 RPS).
• Marginalia: best-effort lane only (shared public key pool, 503s under load); independent-index recall, no dedicated key wired ([email protected] for one).