The web crawler for AI agents that finds the answer, not just pages.
A small decision model picks every link and every passage, so your agent's LLM reads only what answers its question at a few cents per crawl.
Potencialize seu agente
Instalação do publicador. Não é um endpoint do PlugMCP.
Host a copy with our keys and logs.
Ainda não há uma versão empacotada no registro. Siga o README do repositório.
URL remota ou um comando na sua máquina? Como as duas instalações diferem
Rodar no PlugMCP
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
AbrirDo publicador
A small decision model picks every link and every passage, so your agent's LLM reads only what answers its question at a few cents per crawl.
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Agents that browse a site have two bad options. They can read everything a crawler fetches, which buries the answer in navigation and boilerplate and burns tokens. Or they can let the LLM pick each link, which is slow and costs dollars per question. jevcrawl does neither:
You need a Jev API key from console.typesafe.ai and uv.
Claude Code
claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl
Claude Desktop, Cursor and other MCP clients (mcpServers in the client's config file)
{
"mcpServers": {
"jevcrawl": {
"command": "uvx",
"args": ["jevcrawl"],
"env": { "TYPESAFE_API_KEY": "<your key>" }
}
}
}
Then ask your agent something like "Use jevcrawl to find which Python version added the walrus
operator, starting from docs.python.org". Check the setup with
uvx jevcrawl doctor. Agents that load skills
can also use skills/jevcrawl/SKILL.md.
Pages that need JavaScript (optional). jevcrawl reads plain HTML. To also render pages that
are empty without JavaScript, add the js extra and download Chromium once:
claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx --from "jevcrawl[js]" jevcrawl
uvx --from "jevcrawl[js]" playwright install chromium
(In mcpServers, use "args": ["--from", "jevcrawl[js]", "jevcrawl"].) Only pages detected as
JavaScript shells are opened in the browser, at most five per call, and they are listed in
stats.js_rendered_urls. Everything else is still fetched as plain HTML.
Every system starts on a site's home page with the same 30-page budget and the same question. What each one collects goes to the same answering LLM, and the same judge grades the answer, so the crawler is the only thing that changes.
Crawl4AI is one of the most popular open-source crawlers
for LLMs. Its BestFirstCrawlingStrategy with a keyword scorer is its way to crawl toward a question
without an LLM, so we compared the two on whether they fetch the page that holds the answer.
The benchmark code and how to reproduce every number: bench/.
{"blocks": [{"url": "https://docs.python.org/3/whatsnew/3.8.html",
"heading_path": "What’s New In Python 3.8 > New Features > Assignment expressions",
"text": "There is new syntax := that assigns values to variables as part of a larger expression. ...",
"char_start": 307, "char_end": 580, "relevance": 0.97}],
"uncertain": [],
"pages": [{"url": "https://docs.python.org/3/", "title": "3.15.0 Documentation", "lang": "en",
"text_sha256": "b624ce93..."}],
"stats": {"pages_fetched": 6, "tokens_raw": 80034, "tokens_returned": 1873, "jev_calls": 11,
"cache_hits": 0, "jev_input_tokens": 164139, "jev_cost_usd": 0.0069,
"model": "jev-1.13.0", "degraded": false, "degraded_reason": null,
"needs_js_urls": [], "stopped_because": "answered", "elapsed_seconds": 8.3}}
Trimmed to one block and one page. 80,034 tokens of pages became 1,873 tokens of answer, for $0.0069, in 8.3 seconds.
blocks are the relevant passages, best first; uncertain are borderline ones worth reading
before concluding something is missing.char_start/char_end index into the page text whose sha256 is in pages, so a quote can be
checked later.stopped_because is answered, frontier_empty, max_pages, max_seconds, budget or
auth_error.{"error": "<code>", "message": "<what to do>"}; the tools
never raise into the agent's session.The same operations work from the command line (uv tool install jevcrawl or
pip install jevcrawl puts jevcrawl on your path):
jevcrawl crawl https://docs.python.org/3/ --goal "What does the walrus operator do?" [--json]
jevcrawl scrape https://example.com/help/returns [--goal "..."] [--json]
jevcrawl map https://example.com --goal "..." [--limit 20]
jevcrawl doctor
uncertain, and sets stats.degraded.~/.jevcrawl/cache), so the same goal
on the same pages costs nothing.Everything else (thresholds, links per page, concurrency) has a measured default in
jevcrawl/config.py, and Python callers can pass their own Config.
Jev charges for input tokens only, $0.042 per million for jev-1.13.0; in the benchmark a crawl
cost $0.011 on average. max_decision_usd is a hard limit: a request that could pass it is never
sent, and the crawl stops with budget. Every result reports jev_cost_usd, and tokens_raw
(what a plain crawl would hand your LLM) against tokens_returned.
Page text is sent to TypeSafe (blocks, link labels and URLs) to be scored. The API key is read from the environment and never written to disk or logs.
jevcrawl follows each site's robots.txt by default (set JEVCRAWL_RESPECT_ROBOTS=0 to turn
that off) and does not rate-limit itself beyond fetching four pages at a time. You are
responsible for the sites you crawl, their terms of service and the law that applies to you. It
never logs in and does not try to get past paywalls or CAPTCHAs. Without the js extra it does
not run JavaScript (pages that need it are listed in stats.needs_js_urls). Every call is
bounded by max_pages, max_seconds, max_decision_usd and return_tokens.
uv sync
uv run ruff check . && uv run ruff format --check . && uv run mypy --strict jevcrawl
uv run pytest # offline: all HTTP is mocked
TYPESAFE_API_KEY=... uv run pytest -m live -s # one tiny real request, a fraction of a cent
Tests never touch the network or spend money; anything that does is marked live. The README
charts are drawn by assets/build.py from the measured numbers. Issues and
pull requests are welcome; please report security problems as described in
SECURITY.md.
MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.
Ferramentas
Ainda não há lista de ferramentas. Servidores remotos recebem uma após a primeira verificação de saúde.
Perguntas frequentes
Copie nesta página a configuração stdio para Cursor, Claude Desktop, Claude Code ou VS Code. Ela usa a instalação do publicador, não um proxy do PlugMCP.
Este registro não lista nenhuma variável de ambiente obrigatória.
Jevcrawl roda localmente via stdio.
Última verificação: 2026-10-27. MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl.