crawler
This repository holds , a polite web crawler for research and for AI agents, and its site.
Agenten stärken
Der eigene Befehl des Herausgebers. Kein PlugMCP-Endpunkt.
Host a copy with our keys and logs.
npx -y [email protected]Claude Code hat keinen Installationslink für einen lokalen Befehl. Kopieren Sie den Befehl unten und führen Sie ihn in einem Terminal aus.
POLITE_CRAWLER_DIR=<value> POLITE_CRAWLER_CONTACT=<value> POLITE_CRAWLER_ALLOW_PRIVATE=<value> POLITE_CRAWLER_ALLOW_USER_PROVIDED=<value> POLITE_CRAWLER_SIGN_KEY=<value> POLITE_CRAWLER_SIGNATURE_AGENT=<value> claude mcp add io.github.mark1russell7-polite-crawler -- npx -y polite-crawler-mcp@0.7.6In einem Terminal ausführen.
{ "mcpServers": { "io.github.mark1russell7-polite-crawler": { "command": "npx", "args": [ "-y", "[email protected]" ], "env": { "POLITE_CRAWLER_DIR": "<value>", "POLITE_CRAWLER_CONTACT": "<value>", "POLITE_CRAWLER_ALLOW_PRIVATE": "<value>", "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<value>", "POLITE_CRAWLER_SIGN_KEY": "<value>", "POLITE_CRAWLER_SIGNATURE_AGENT": "<value>" } } }}claude_desktop_config.json
Remote-URL oder ein Befehl auf Ihrem Rechner? Worin sich die beiden Installationen unterscheiden
Auf PlugMCP ausführen
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
Startenpolite-crawler fetches a source one time, keeps it, and reads it many times. It obeys robots.txt, sends one request at a time to each site, and answers each later question from a local full-text index.
packages/polite-crawler is on npm as polite-crawler.packages/site is the documentation: https://mark1russell7.github.io/crawler/. Its interactive figures and tools start the real code: hybrid search, the recipes, the daemon and the robots.txt tester.npx polite-crawler fetch https://en.wikipedia.org/wiki/Web_crawler
npx polite-crawler find "robots exclusion"
claude mcp add polite-crawler -- npx -y polite-crawler-mcp
What the crawler does for you and for the sites:
--allow-private permits loopback and private addresses.<untrusted-web-content> envelope, so an assistant can tell data from instructions. A page that refuses AI input stays held back until a person confirms.polite-crawler, and the MCP server is the package polite-crawler-mcp (its README).The package README tells the details.
The repository is a pnpm workspace. It needs Node.js 22.19 or later and pnpm 10 or later.
pnpm install
pnpm browser # the headless shell for the renderer test
pnpm check # typecheck, lint, test, lint:ste, build and the smoke test of the package
The prose of the repository (the README files, the doc comments and the site) follows ASD-STE100 Simplified Technical English. The linter ste-lint examines it in CI.
CONTRIBUTING.md tells how to set up, test and write a change.SECURITY.md tells how to report a vulnerability privately.CHANGELOG.md records the changes in each version.MIT
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
{ "mcpServers": { "io.github.mark1russell7-polite-crawler": { "command": "npx", "args": [ "-y", "[email protected]" ], "env": { "POLITE_CRAWLER_DIR": "<value>", "POLITE_CRAWLER_CONTACT": "<value>", "POLITE_CRAWLER_ALLOW_PRIVATE": "<value>", "POLITE_CRAWLER_ALLOW_USER_PROVIDED": "<value>", "POLITE_CRAWLER_SIGN_KEY": "<value>", "POLITE_CRAWLER_SIGNATURE_AGENT": "<value>" } } }}Tools
Noch keine Tool-Liste. Remote-Server erhalten eine nach dem ersten Health-Check.
FAQ
Kopieren Sie auf dieser Seite die stdio-Konfiguration für Cursor, Claude Desktop, Claude Code oder VS Code. Sie nutzt die Installation des Herausgebers, keinen PlugMCP-Proxy.
Der Herausgeber nennt diese Umgebungsvariablen: POLITE_CRAWLER_DIR, POLITE_CRAWLER_CONTACT, POLITE_CRAWLER_ALLOW_PRIVATE, POLITE_CRAWLER_ALLOW_USER_PROVIDED, POLITE_CRAWLER_SIGN_KEY, POLITE_CRAWLER_SIGNATURE_AGENT.
Polite Crawler läuft lokal über stdio.
Zuletzt geprüft: 2026-11-16. A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.