crawlerpeek
Check whether AI crawlers can access your site. For each bot (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and more) it tells you:
Power your agent
คำสั่งของผู้เผยแพร่เอง ไม่ใช่เอนด์พอยต์ของ PlugMCP
Host a copy with our keys and logs.
npx -y [email protected]Claude Code ไม่มีลิงก์ติดตั้งสำหรับคำสั่งแบบโลคัล ให้คัดลอกคำสั่งด้านล่างแล้วรันในเทอร์มินัล
claude mcp add crawlerpeek-crawlerpeek -- npx -y crawlerpeek@0.3.1รันในเทอร์มินัล
{ "mcpServers": { "crawlerpeek-crawlerpeek": { "command": "npx", "args": [ "-y", "[email protected]" ] } }}claude_desktop_config.json
URL แบบรีโมตหรือคำสั่งบนเครื่องของคุณ? การติดตั้งสองแบบต่างกันอย่างไร
รันบน PlugMCP
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
เปิดจากผู้เผยแพร่
Check whether AI crawlers can access your site. For each bot (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot and more) it tells you:
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
robots.txt allows the URL's path for that bot (RFC 9309: user-agent groups, wildcards, $, longest match wins), andIt also generates a robots.txt that welcomes AI crawlers. A library, a CLI for CI, and an MCP server. Node 20 or newer, no runtime dependencies, ESM and CommonJS, TypeScript types included.
The same checks as a web tool: crawlerpeek.com. Documentation with examples for CI and MCP clients: crawlerpeek.com/cli.
npm install crawlerpeek # library
npx crawlerpeek --help # CLI, no install needed
npx crawlerpeek check https://example.com
https://example.com
robots.txt: ok (HTTP 200) at https://example.com/robots.txt
Bot robots HTTP
GPTBot blocked 200 FAIL
ClaudeBot allowed 403 (block page) FAIL
Google-Extended allowed -
npx crawlerpeek check https://example.com/pricing --bots GPTBot,ClaudeBot --fail-on blocked
npx crawlerpeek check https://example.com --bots all --json
npx crawlerpeek check https://example.com --no-http # robots.txt only
npx crawlerpeek see https://example.com --bot ClaudeBot # one bot, full detail
npx crawlerpeek verify 66.249.66.1 --bot Googlebot # is this IP really Googlebot?
npx crawlerpeek robots --disallow /admin/,/cart/ --sitemap https://example.com/sitemap.xml
Exit codes: 0 ok, 1 the --fail-on condition was met, 2 usage error (including an unknown option) or request error. see exits 0 when the page was fetched, whatever its HTTP status, and 2 when the request failed or was refused.
verify <ip> [--bot GPTBot] [--json] checks whether an address from your server logs really belongs to a crawler. It uses only what the operator documents: the published IP-range JSON (Google, OpenAI, Anthropic, Perplexity, Bing, Apple, Common Crawl, DuckDuckGo, Ahrefs) or reverse DNS with forward confirmation (Google, Bing, Apple, Common Crawl, Ahrefs). Without --bot every bot with such a method is checked. Each result has method (ip-list, reverse-dns or none), verified (true, false, or null when it cannot be decided, for example the list is unreachable or the operator documents no method) and a detail. Lists are cached for one hour, and nothing is ever sent to the IP. Exit codes: 0 verified for at least one bot, 1 not verified or undecidable, 2 usage error.
Without --fail-on the exit code is 0 whatever the result. --bots takes tokens or all; the default is 15 bots: the main AI crawlers plus Googlebot and Bingbot. A robots.txt that cannot be fetched (timeout, 5xx, a redirect that is refused or has no target) is reported as unknown, with the reason in robotsTxt.error, and never counts as blocked. Note: this deliberately differs from RFC 9309 §2.3.1.4, where crawlers treat an unreachable robots.txt (5xx) as "disallow everything" (Google does too), so a site whose robots.txt returns 5xx is effectively closed to many bots even though --fail-on blocked passes. Treat unknown as a problem to fix.
import { checkAiAccess, seeAsBot, verifyBotIp, generateRobotsTxt, failingBots } from "crawlerpeek";
const report = await checkAiAccess("https://example.com/pricing", { bots: ["GPTBot", "ClaudeBot"] });
report.bots[0].robots; // { access: "allowed", pathAllowed: true, matchedGroup: "*" }
report.bots[0].http; // { status: 200, finalUrl, redirects, contentType, bytes, title, looksBlocked, outcome: "ok", ... }
failingBots(report, "blocked"); // ["ClaudeBot"] when it is closed or challenged
await seeAsBot("https://example.com", "PerplexityBot");
await verifyBotIp("66.249.66.1", { bot: "Googlebot" }); // [{ bot: "Googlebot", method: "ip-list", verified: true, detail, source }]
generateRobotsTxt({ disallow: ["/admin/"], blockBots: ["CCBot"], sitemap: "https://example.com/sitemap.xml" });
generateRobotsTxt names each AI bot explicitly with Allow: /. Disallowed paths are repeated in that group on purpose: a bot with its own group ignores the * group.
crawlerpeek mcp speaks the Model Context Protocol over stdio and exposes four read-only tools: check_ai_access, see_as_bot, verify_bot_ip, generate_robots_txt. Results are cached for 10 minutes per call. No LLM is called; the tools only fetch the URL you give them.
Generic configuration (Claude Desktop, Cursor, Windsurf, Cline and others use this mcpServers shape):
{ "mcpServers": { "crawlerpeek": { "command": "npx", "args": ["-y", "crawlerpeek", "mcp"] } } }
VS Code reads .vscode/mcp.json with servers as the root key and "type": "stdio" on the entry. Codex reads ~/.codex/config.toml; raise the startup timeout so the first npx download does not run into the default 10 seconds:
[mcp_servers.crawlerpeek]
command = "npx"
args = ["-y", "crawlerpeek", "mcp"]
startup_timeout_sec = 30
skills/crawlerpeek/ is an Agent Skill: a SKILL.md that tells an AI agent when to check AI crawler access, which commands to run, and how to read the result. It uses only the common frontmatter fields, so the same folder works in several agents.
git clone --depth 1 https://github.com/crawlerpeek/crawlerpeek.git
mkdir -p ~/.agents/skills && cp -r crawlerpeek/skills/crawlerpeek ~/.agents/skills/
Restart the agent afterwards so it picks up the new skill. The skill and the MCP server are independent: with the MCP server connected the skill uses its tools, otherwise it runs npx -y crawlerpeek.
A composite action lives in action/; see action/README.md.
- uses: crawlerpeek/crawlerpeek/action@v0
with: { url: "https://example.com", fail-on: blocked }
Googlebot here can therefore be normal; a 403 for an unusual bot with a 200 for a browser usually points at a WAF or CDN rule. robots.txt itself is fetched with an honest crawlerpeek/<version> user agent.One request per bot plus one for robots.txt, sequential with a 300 ms pause (--delay); 10 s timeout for the whole request chain; at most 5 redirects, each re-checked (a redirect to a refused target is reported for that bot or for robots.txt, it does not stop the run); response bodies capped at 2 MB; no cookies. Private, loopback, link-local and metadata addresses are refused after DNS resolution (and again at connect time), as are ports other than 80 and 443 and URLs with credentials. --allow-private (CLI and crawlerpeek mcp) or allowPrivate: true lifts these limits for local testing.
Issues and pull requests are welcome, see CONTRIBUTING.md. Report security problems privately, see SECURITY.md.
Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.
{ "mcpServers": { "crawlerpeek-crawlerpeek": { "command": "npx", "args": [ "-y", "[email protected]" ] } }}เครื่องมือ
ยังไม่มีรายการเครื่องมือ เซิร์ฟเวอร์แบบรีโมตจะได้รายการหลังการตรวจสอบสถานะครั้งแรก
FAQ
คัดลอกคอนฟิก stdio สำหรับ Cursor, Claude Desktop, Claude Code หรือ VS Code ได้ในหน้านี้ คอนฟิกนี้ใช้การติดตั้งของผู้เผยแพร่ ไม่ใช่พร็อกซีของ PlugMCP
ระเบียนนี้ไม่ได้ระบุตัวแปรสภาพแวดล้อมที่จำเป็น
Crawlerpeek ทำงานแบบโลคัลผ่าน stdio
ตรวจสอบล่าสุดเมื่อ 2026-10-27 Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.