Native-speed, browserless web crawler + MCP server for AI agents. A standalone Rust engine (turbo-dom + a V8 JS-render tier) — fetch + parse + extract + run page JS with no headless browser. This npm
Lighthouse-style CLI for auditing websites for AI agents, LLM crawlers, MCP clients, llms.txt, WebMCP, OpenAPI, and Schema.org readiness
Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
Audit how AI sees a website: per-bot robots.txt verdicts for AI crawlers, schema, sitemap, llms.txt, SPA detection and cross-model brand mentions.
Local-first web scraping, DuckDuckGo search, site mapping and crawling, using trafilatura with an optional Firecrawl fallback for JS-rendered pages.
Web scraping via ScraperAPI with JavaScript rendering, geotargeting, premium proxies and auto-parsing.
Identity and credential gateway for agents: Ed25519 identity, encrypted credential vault, OPA policies with audit logging, per-agent memory and web scraping.
URL fetcher and HTML-to-Markdown converter with layered prompt injection defense: hidden-element sanitization, risk scanning and content boundary wrapping.
Web pages as clean data, plus RugProbe Base token safety checks. Pay per call in USDC (x402).
Use Apify Actors, pre-built cloud tools, to extract data from websites, e-commerce, social media, search engines, maps and more.
Compare two web pages on desktop and phone: percent changed, changed regions, diff image, PASS/FAIL.
Fetch public web pages live and get clean Markdown, text or HTML.
Business emails from websites, UK & EU contract award winners, website tech stacks. Apify Actors.
AgentScraper CLI: live web pages as clean Markdown, text or HTML in your terminal, through the AgentScraper API. The commands follow the live API spec.
Free, open-source, local-first SEO crawler and regression audit tool.
Live Web Data CLI: live web pages as clean Markdown, text or HTML in your terminal, through the Live Web Data API. The commands follow the live API spec.
FetchMCP CLI: live web pages as clean Markdown, text or HTML in your terminal, through the FetchMCP API. The commands follow the live API spec.
Content licensing for the AI era. Licensed, rights-cleared content for AI agents, with a verifiable license key for every article. The MCP alternative to unlicensed scraping for RAG and AI search.
MCP server for Google Play and App Store scraping
iOS app reviews, ratings, details, as one MCP tool through Apify
Download Greenhouse, Lever and Ashby jobs from any employer's hiring site.
Abu Dhabi property sales since 2019 as JSON: AED price, price/sqm, district, project. $1/1k
Search Korean places and read reviews from Naver Map and Kakao Map. Pay per result via Apify.
Crawler diagnostics, AI visibility evidence and article drafts for review in Prerender Buddy.
A polite crawler for agents: obeys robots.txt, fetches each page once, finds and verifies quotes.
Autonomous AI agent readiness, crawler reachability, and 30-point audit specification engine.
Static and dynamic call tracing MCP server backed by a SQLite property graph of your codebase
Bulk PageSpeed Insights and Core Web Vitals (LCP, INP, CLS) for many URLs. Runs on Apify.
Merge Apify datasets and remove duplicates by website, email, phone or any field. Runs on Apify.
Tech stack, WHOIS, DNS, SPF/DMARC, SSL expiry and AI crawler rules per domain. Runs on Apify.
Scrape webpages, run crawls, and read Markdown and other results with Nstdata Crawl.
Web scraping MCP server: any URL to Markdown or JSON, with JavaScript rendering and batch jobs.
Google Web Search: Google Web Search API. Search the world’s information, including webpages.
Весь каталог
Обход, извлечение данных и мониторинг открытого веба. Серверов: 1,406, сначала самые используемые. Каждый ведёт на страницу издателя.