Agenten stärken
Der eigene Befehl des Herausgebers. Kein PlugMCP-Endpunkt.
Host a copy with our keys and logs.
npx -y [email protected]Claude Code hat keinen Installationslink für einen lokalen Befehl. Kopieren Sie den Befehl unten und führen Sie ihn in einem Terminal aus.
claude mcp add crawlforge-mcp-server -- npx -y crawlforge-mcp-server@6.25.1In einem Terminal ausführen.
{ "mcpServers": { "crawlforge-mcp-server": { "command": "npx", "args": [ "-y", "[email protected]" ] } }}claude_desktop_config.json
Remote-URL oder ein Befehl auf Ihrem Rechner? Worin sich die beiden Installationen unterscheiden
Auf PlugMCP ausführen
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
StartenVom Herausgeber
CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama
agent, a unified multi-format scrape, document processing, stealth browsing, stateful browser sessions, and more, callable directly from your AI assistant.extract_with_llm runs against a local Ollama model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.agent — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.Comparison reflects publicly documented capabilities at time of writing. CrawlForge is MIT-licensed and MCP-first — built to plug straight into AI coding assistants.
npm install -g crawlforge-mcp-server
Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start. The recommended path signs you in through the browser, so the key is never pasted into a terminal (a coding agent can run this for you and relay the URL):
crawlforge login
It prints an approval URL; open it, approve, and the key is stored in ~/.crawlforge/config.json. Then run crawlforge init to register the MCP server with your client. Or use the interactive wizard, which also configures your clients:
npx crawlforge-setup
This will:
Don't have an API key? Get one free at https://www.crawlforge.dev/signup
One-step setup (v4.6.0+):
crawlforge initdetects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Usecrawlforge init --all --yesto configure every detected client non-interactively.
Add to claude_desktop_config.json:
{
"mcpServers": {
"crawlforge": {
"command": "npx",
"args": ["-y", "crawlforge-mcp-server"]
}
}
}
Location:
~/Library/Application Support/Claude/claude_desktop_config.json%APPDATA%/Claude/claude_desktop_config.json~/.config/Claude/claude_desktop_config.jsonRestart Claude Desktop to activate.
The setup wizard automatically configures Claude Code by adding to ~/.claude.json:
{
"mcpServers": {
"crawlforge": {
"type": "stdio",
"command": "crawlforge-mcp"
}
}
}
After setup, restart Claude Code to activate.
The setup wizard automatically configures Cursor by adding to ~/.cursor/mcp.json:
{
"mcpServers": {
"crawlforge": {
"type": "stdio",
"command": "crawlforge-mcp"
}
}
}
Restart Cursor to activate.
n8n's built-in MCP Client Tool node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:
export CRAWLFORGE_API_KEY=your_api_key
npm run start:http # Streamable HTTP endpoint at http://localhost:10000/mcp
Then point the MCP Client Tool node at http://<host>:10000/mcp with transport HTTP Streamable and a Bearer credential set to the same API key. On self-hosted n8n you can instead use the community n8n-nodes-mcp node over STDIO (npx -y crawlforge-mcp-server).
Full guide: docs/n8n-integration.md
Which launch command?
npx -y crawlforge-mcp-serverneeds no global install and always runs the published version (recommended for Claude Desktop). For a global install (npm i -g crawlforge-mcp-server), use the dedicatedcrawlforge-mcpbin — it resolves on yourPATH, so it survives Node/nvm version switches. The barecrawlforgecommand still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — runcrawlforge mcpto start the server by hand.
CrawlForge requires a CrawlForge API key — every tool is metered and consumes credits. New accounts get 1,000 free trial credits to start. Get a key at crawlforge.dev/signup.
All Tools (API key required)
Ten tools (scrape, fetch_url, extract_content, crawl_deep, batch_scrape, stealth_mode, scrape_with_actions, process_document, deep_research, extract_embedded_state) accept max_inline_chars (default 40,000; env CRAWLFORGE_MAX_INLINE_CHARS): a result over it comes back as a preview plus a result_handle for read_result, with the full result kept for 1 hour under ~/.crawlforge/results/ on your own machine — nothing is uploaded.
For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see SKILL.md.
Every tool is metered and requires an API key. New accounts get 1,000 free trial credits — no credit card required to start.
All plans include:
# Optional: Set API key via environment
export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"
# Optional: Custom API endpoint (for enterprise)
export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
# As of v3.0.18, this variable is validated against an allow-list of CrawlForge backend hosts.
# Optional: Local LLM (Ollama) overrides — extract_with_llm, extract_structured
# and deep_research all use Ollama when no cloud key is set
export OLLAMA_BASE_URL="http://localhost:11434" # default; set https://ollama.com for Ollama Cloud
export OLLAMA_DEFAULT_MODEL="gemma3:4b" # optional; unset = pick the best installed model automatically
# deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
# conflict detection is on only with that model, or a cloud provider
export OLLAMA_EMBEDDING_MODEL="nomic-embed-text" # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
export OLLAMA_API_KEY="..." # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
export DISABLE_OLLAMA="true" # skip Ollama entirely and use CSS/keyword fallbacks
# Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
export OPENAI_API_KEY="sk-..."
export ANTHROPIC_API_KEY="sk-ant-..."
# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape" # unset = all tools; unknown names/groups are ignored with a warning
# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
export RESEARCH_STEALTH_ENGINE="auto" # auto (default) | camoufox | chromium
export RESEARCH_STEALTH_FALLBACK="true" # set to "false" to disable entirely
export RESEARCH_MAX_STEALTH_RETRIES="8" # cap on stealth retries per research run
# Optional: your own proxies for the stealth paths that have no caller to ask —
# the scrape escalation stage, the deep_research retry, the agent, browser_session.
# Comma-separated; a proxy passed on a call always wins. CrawlForge supplies none.
export CRAWLFORGE_STEALTH_PROXIES="http://user:[email protected]:8080"
# Unrelated to the PROXY_ROTATION_* variables, which belong to the localization tool.
# Optional: turn off the clearance jar (on by default) — see "Stealth engines and proxies"
export CRAWLFORGE_CLEARANCE_JAR="off"
# Optional: also refuse private networks (RFC1918, ULA, CGNAT, ...) — see "Security & Approvals"
export SSRF_STRICT="true"
# Optional: the only directories process_document may read local files from (":"-separated, ";" on Windows)
export CRAWLFORGE_ALLOWED_DIRS="$HOME/Documents:$HOME/Downloads"
CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:
scrape, map_site, serp_rank, reddit_search, search_web, extract_structured, and crawl_deep return machine-parseable structuredContent alongside the usual text, validated against a published outputSchema; legacy clients keep working off the text.isError: true result the calling model can read and retry from, instead of a raw JSON-RPC protocol error.tools/list ordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.crawl_deep, batch_scrape, deep_research, agent — for clients that support polling; synchronous results are still returned for clients that don't.See docs/mcp-spec-adoption.md for wire-level examples and client-compatibility notes.
extract_with_llm with Ollama)extract_with_llm defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).
# 1. Install Ollama: https://ollama.com
# 2. Pull any model from https://ollama.com/library
ollama pull llama3.2
# 3. Discover what's installed (from your MCP client)
# list_ollama_models()
# 4. Extract — defaults to Ollama with the model from step 2
# extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })
Every stealth entry point — scrape with escalate: true, stealth_mode, browser_session, scrape_with_actions and the deep_research retry — now defaults to auto: use Camoufox (Firefox anti-detect) when its binary is installed, fall back to Chromium when it is not, with the reason in the result's warnings[]. Every result names the engine that actually ran. Naming an engine explicitly behaves as it always has — camoufox fails rather than falling back, and chromium and playwright are the same engine (on scrape's escalate_engine, spell it playwright).
On browser_session and scrape_with_actions the engine applies only with stealth: true — their ordinary path is the standard Chromium pool.
auto is not free. Measured once each on an Apple Silicon Mac on 2026-09-21 (launch plus a context and a blank page): Chromium 148 ms and 253 MB, Camoufox 946 ms and 667 MB — roughly +0.8 s and +400 MB per stealth call. Pin engine: "playwright" for high-volume work on sites that do not block.
CrawlForge supplies no proxies, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with CRAWLFORGE_STEALTH_PROXIES (comma-separated URLs), which the escalation stage, stealth_mode, browser_session, scrape_with_actions, the deep_research retry and the agent tool's automatic stealth retry use when the caller passes none; a proxy passed on the call always wins. With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.
A challenge solved once is not solved again. When a stealth render gets past a Cloudflare or DataDome wall, the vendor's clearance cookies (cf_clearance, __cf_bm, datadome — nothing else) are kept in ~/.crawlforge/stealth-clearance.json (mode 0600) and replayed to the next stealth context with the same engine, User-Agent and proxy, for at most the cookie's own lifetime and never more than 24 hours. A render that still meets the wall drops them. Set CRAWLFORGE_CLEARANCE_JAR=off to disable.
Full detail, measurements and sources: docs/stealth-engines.md.
deep_research (Camoufox)deep_research automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a real fingerprinted browser, then re-extracts from the rendered HTML. It's bounded (RESEARCH_MAX_STEALTH_RETRIES, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.
Engine selection (RESEARCH_STEALTH_ENGINE):
auto (default) — prefer Camoufox (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.camoufox — force Camoufox.chromium — force the Chromium stealth engine.Camoufox gets through walls that stop headless Chromium: in a benchmark run on 2026-09-21 from a residential IP it was the only engine to clear Cloudflare Turnstile (indeed.com) and Akamai (harrods.com), both of which blocked Chromium stealth. Neither engine cleared DataDome or an interactive Turnstile, so this is a better fetch path, not a bypass. To enable it, install the optional dependency and run its one-time binary fetch:
# Camoufox is declared as an optional dependency, so a normal install already pulls it.
# If you installed with --no-optional, add it explicitly:
npm install camoufox
# One-time download of the Camoufox Firefox binary (~130 MB):
npx camoufox fetch
Without the Camoufox binary, deep_research silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with RESEARCH_STEALTH_FALLBACK=false.
Note: Hard IP-reputation blocks (e.g. Reddit's edge
403) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide — point the server at your own withCRAWLFORGE_STEALTH_PROXIES. See docs/stealth-engines.md for details.
Your configuration is stored at ~/.crawlforge/config.json:
{
"apiKey": "cf_live_...",
"userId": "user_...",
"email": "[email protected]"
}
Once configured, use these tools in your AI assistant:
"Search for the latest AI news"
"Extract all links from example.com"
"Crawl the documentation site and summarize it"
"Monitor this page for changes"
"Extract product prices from this e-commerce site"
~/.crawlforge/config.json127.0.0.0/8, ::1), the unspecified addresses (0.0.0.0/8, ::), link-local and cloud metadata (169.254.0.0/16 incl. 169.254.169.254, fe80::/10). It allows private networks (RFC1918 10/8, 172.16/12, 192.168/16, IPv6 ULA): run on your own machine, the server acts for you, and reading a router page, a NAS or an intranet docs site on your LAN is a legitimate use that the stricter level would break.SSRF_STRICT=true adds every private and reserved range: RFC1918, ULA (fc00::/7), CGNAT (100.64/10), multicast, reserved, IPv4-compatible (::/96) and NAT64 (64:ff9b::/96) IPv6. Set it if an agent that reads untrusted pages (and could be prompt-injected into fetching a LAN address) should not reach your local network, or whenever the server is shared.process_document with sourceType file or pdf_file; refused outright when the server is served to a network): a path is judged by its realpath, so a symlink counts as its target. Only .pdf, .docx, .txt, .md, .html and .htm files are read, and nothing inside a dot-directory (~/.ssh, ~/.aws, ~/.config, ~/.crawlforge, ...). CRAWLFORGE_ALLOWED_DIRS (directories separated by :, ; on Windows) narrows reads to those directories instead; inside them the dot-directory rule is lifted, the extension rule is not.{crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}, HTTPS required). Setting CRAWLFORGE_API_URL to an arbitrary host is blocked at parse time.scrape_with_actions accepts only 11 action types (snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select, hover, navigate). No download, file-write, or arbitrary cross-page navigation primitives exist — navigate goes through the same SSRF and robots.txt gate as the initial URL.executeJavaScript action throws by default. Set ALLOW_JAVASCRIPT_EXECUTION=true at deploy time to enable (not recommended in production).deep_research (>50 URLs), batch_scrape (sync mode, >25 URLs), crawl_deep (projected >500 pages), extract_structured (schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).withAuth() and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).See docs/sandboxing-and-approvals.md for the full reference.
v3.0.3 (2025-10-01): Removed authentication bypass vulnerability. All users must authenticate with valid API keys.
For the full security policy and how to report a vulnerability, see SECURITY.md.
MIT License - see LICENSE file for details.
Contributions are welcome! Please read our Contributing Guide first.
Built with ❤️ by the CrawlForge team
CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude
{ "mcpServers": { "crawlforge-mcp-server": { "command": "npx", "args": [ "-y", "[email protected]" ] } }}Übersicht
CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama
Tools
Noch keine Tool-Liste. Remote-Server erhalten eine nach dem ersten Health-Check.
FAQ
Kopieren Sie auf dieser Seite die stdio-Konfiguration für Cursor, Claude Desktop, Claude Code oder VS Code. Sie nutzt die Installation des Herausgebers, keinen PlugMCP-Proxy.
In diesem Eintrag ist keine erforderliche Umgebungsvariable aufgeführt.
Crawlforge läuft lokal über stdio.
Zuletzt geprüft: 2026-11-17. CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama