Skip to content
plugmcp
DirectorySkillsCategoriesBlog
EN/δΈ­ζ–‡
Sign inSign up
Home/Servers/Crawlbit

Crawlbit

I
by io.github.amati032-dev

Four free tools to check if AI engines can find, read and recognise a site. No API key.

Power your agent

Add this server

The publisher's own command. Not a PlugMCP endpoint.

Local command
npx -y [email protected]
Credentials from the publisher
  • CRAWLBIT_BASE_URL β€” Optional. Point the server at another CrawlBit instance. Defaults to https://www.crawlbit.app. No API key or account is needed to use this server.
Provider
Claude Code
CRAWLBIT_BASE_URL=<value> claude mcp add io.github.amati032-dev-crawlbit -- npx -y [email protected]

Run in a terminal.

Claude Desktop
{
  "mcpServers": {
    "io.github.amati032-dev-crawlbit": {
      "command": "npx",
      "args": [
        "-y",
        "[email protected]"
      ],
      "env": {
        "CRAWLBIT_BASE_URL": "<value>"
      }
    }
  }
}

claude_desktop_config.json

Step-by-step: add this in Claude

Remote URL or a command on your machine? How the two installs differ

Run on PlugMCP

Host a copy with our keys and logs.

A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.

Launch

Related MCP

View more
  • FirecrawlπŸ”₯ Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
  • N
    NotionToken-auth Notion MCP: pages, databases, blocks, comments, files. Runs headless in CI and agents.
  • Context7Context7 Platform -- Up-to-date code documentation for LLMs and AI code editors

Related skills

View all
  • reaReverse engineer anything with agents, from app behavior down to native binaries.
  • rea-changelog-updatePrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.
  • rea-tool-designDesign or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.

Questions

How do I install Crawlbit?
Copy the stdio config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.
Does Crawlbit need an API key?
The publisher lists these environment variables: CRAWLBIT_BASE_URL.
What transport does Crawlbit use?
Crawlbit runs locally over stdio.
When was Crawlbit last checked?
Last checked 2026-10-16. Four free tools to check if AI engines can find, read and recognise a site. No API key.

Four free tools to check if AI engines can find, read and recognise a site. No API key.

CrawlBit MCP

Four free tools that answer one question: can AI engines find, read and recognise a website?

No account. No API key. No signup. Nothing here runs a language model, so nothing here costs you or us anything.

"Can ChatGPT read stripe.com?"
"Why does AI never mention my store?"
"Where is my brand missing outside my own site?"

Ask in plain language. Your AI client picks the right tool.

Install

Requires Node.js 18+.

Claude Code

claude mcp add crawlbit -- npx -y crawlbit-mcp

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "crawlbit": {
      "command": "npx",
      "args": ["-y", "crawlbit-mcp"]
    }
  }
}

Cursor

Add to .cursor/mcp.json:

{
  "mcpServers": {
    "crawlbit": {
      "command": "npx",
      "args": ["-y", "crawlbit-mcp"]
    }
  }
}

Restart your client. Four tools appear.

The four tools

ToolWhat it answers
crawlbit_crawler_watchCan GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended read this site?
crawlbit_entity_checkDoes this brand read as a clear entity AI can recognise?
crawlbit_offpage_gapsWhere is this brand missing outside its own website?
crawlbit_shoppingCan AI shopping agents understand and recommend these products?

Try these

Check whether AI crawlers can read example.com
Does example.com read as a clear brand entity to AI?
Where is example.com missing off-page, and which gap should we close first?
Compare example.com and competitor.com on AI crawler access

The tools compose. A useful sequence is: crawler access first (a blocked crawler makes everything else pointless), then the entity check, then the off-page gaps.

What this does NOT do

Written plainly, because an SEO tool that overstates itself is worse than no tool.

  • It does not test whether AI actually cites a brand. That means running real buyer questions against ChatGPT, Perplexity and Claude, reading the answers, and reporting who gets named instead of you. It costs real money per run and is not part of this free surface. These four tools measure the conditions that make a citation possible, which is a smaller claim.
  • crawler_watch reads robots.txt only. It cannot see blocking done at the CDN, firewall or rate-limit layer. A site can pass here and still refuse the crawler in practice.
  • entity_check reads what the site publishes about itself. It says nothing about the brand's reputation across the rest of the web.
  • shopping inspects published structured data. It cannot see a merchant feed submitted privately to a platform.

If a site scores well on all four and still is not cited, the answer is almost always the same and it is not technical: too few third-party pages mention it. These tools will tell you that honestly rather than sell you a fix that does not exist.

Why only four tools

CrawlBit also runs a full technical audit and generates llms.txt. Both use a language model, so both cost money per run, and neither is exposed here.

That is a deliberate line rather than a teaser. A free tool whose bill grows with its popularity gets rate-limited, degraded or withdrawn the moment it succeeds, and the people who installed it are the ones who pay for that. Everything in this server costs nothing to run, so nothing here has to be taken back later. The audit and the llms.txt generator live at crawlbit.app if you want them.

Rate limit

The API rate-limits per IP. Normal conversational use does not come close, and the server runs on your machine, so the budget is yours rather than shared. If you do hit it, you get a plain sentence rather than a stack trace.

Configuration

VariableDefaultPurpose
CRAWLBIT_BASE_URLhttps://www.crawlbit.appPoint the server at another instance

Privacy

The server runs locally and stores nothing. The only data leaving your machine is the URL you ask about, sent to the CrawlBit API to be analysed. There is no account, so there is nothing to tie a request to a person.

About

Built by CrawlBit, which measures AI visibility and does the off-page work that earns citations. The free tools here are the measurement half. The paid work is the other half, and you are not required to look at it to use this.

MIT licensed. Issues and pull requests welcome.

How do I install Crawlbit?
Copy the stdio config for Cursor, Claude Desktop, Claude Code, or VS Code on this page. It uses the publisher's install, not a PlugMCP proxy.
Does Crawlbit need an API key?
The publisher lists these environment variables: CRAWLBIT_BASE_URL.
What transport does Crawlbit use?
Crawlbit runs locally over stdio.
When was Crawlbit last checked?
Last checked 2026-10-16. Four free tools to check if AI engines can find, read and recognise a site. No API key.

No tool list yet. Remote servers get one after the first health check.

plugmcp

Plug any tool into any agent. MCP servers for people who do not write servers. Gateway for volume, n8n for apps.

  • github.com/TODO
  • x.com/TODO
  • linkedin.com/company/TODO

PlugMCP, 2026

Product
  • Browse
  • Install
  • Host your own
  • Pricing
  • Voice mode
  • Browser
Directory
  • All MCP servers
  • MCP Apps
  • Tools
  • Agent skills
  • Categories
  • New
  • Voice
  • Submit a server
Developers
  • Docs
  • Blog
  • Club
  • Work with us
  • What is MCP
  • What is WebMCP
  • Status
Company
  • About
  • Contact
  • Terms
  • Privacy
  • DPA