Web scraping & search for AI agents: extract, SearXNG search, rerank, robots compliance. No API key.
Publish HTML as live, shareable web pages with public, password-protected or email-restricted access.
Unofficial Decodo MCP — wraps the Decodo Web Scraping API (decodo.com, formerly Smartproxy).…
Which AI crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot) a website allows, and why.
Deterministic Zero-LLM Web Compiler, Scraper, Crawler, and Site Mapper for AI Agents.
MCP server (stdio): fetch web pages as clean readable markdown via the AgentForge API
Unified access to subreddit metadata, post threads, user activity, and search. Built for LLMs and automation — not one-off scraping.
Korean cloud for AI agents — servers, Docker apps with HTTPS, domains, scraping, 300+ AI models
{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key
Checks robots.txt for AI search vs training crawlers, llms.txt, JSON-LD, canonicals and sitemaps.
Detects GTM hiring activity from company career pages via Greenhouse, Lever, Ashby. Clay-ready.
Scrape Amazon products, search, and async batch ASIN lookups across 20 marketplaces
B2B contacts and account signals on Apify: new hires, email format, lookalikes, decision makers.
Structured recipes from Instagram cooking reels, any language, via Apify. No-recipe reels free.
Competitor ads from Google Ads Transparency Center by brand or domain: ad text, YouTube IDs.
Web pages for AI agents: may-I-use checks, prompt-injection flags, Markdown, JSON, screenshots
Google Trends for agents: interest over time, by region, rising queries, Trending Now. No API key.
Web pages and PDFs to clean, LLM-ready Markdown with token counts, small crawls and RAG chunks.
Unofficial Firecrawl MCP — wraps the Firecrawl API (firecrawl.dev) for web scraping / crawling /…
Is this page readable for AI search? Crawler access, structure and citability checks, runs locally.
MCP server for web scraping: URLs, links, metadata, CSS extraction.
Arachne MCP — scraping, browser, vision, OCR, RAG, Social Research, GitHub + VRT (56 tools).
Web scraping and AI sales workspace with scoped CRM, scraping, knowledge and sales workflow tools.
One key for agent data tools: emails, companies, search, scraping. Pay only for results that pass.
Web data for agents. Extract web pages, crawl websites, discover URLs, and search the web.
Capture web pages as signed, timestamped evidence; compare, watch and verify them.
Debug web pages in Chrome: performance traces, network, console, snapshots, screenshots.
Web search, URL content extraction to Markdown, site mapping, and recursive web crawler.
Monitor web pages and read item-level updates: the field that moved, with its old and new value.
Firecrawl web data: scrape, map, search, crawl, extract, research, and llms.txt.
Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.
See what GPTBot, ClaudeBot and other crawlers get from a site: robots.txt checks and AI scans.
Publish webpages and documents, send private review links, and collect comments and approvals.
BSE India filings as JSON: announcements, ex-dates, board meetings, result dates, deals. $3/1k
AI-crawler readiness checks and fixes: GEO score, robots.txt, schema.org, FAQ and llms.txt.
Audit a site for AI readiness: AI-crawler access, schema, FAQ, content structure. Generates fixes.
Gesamtes Verzeichnis
Das öffentliche Web crawlen, extrahieren und überwachen. 1,406 Server, die meistgenutzten zuerst. Jeder öffnet die Seite des Herausgebers.