Crawl a website into clean Markdown for LLMs/RAG: main content only, linked docs converted.
Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only.
Firecrawl MCP for DeepSeek Harness: web crawling, scraping, search, and structured data extraction
Audit how AI sees a website: per-bot robots.txt verdicts for AI crawlers, schema, sitemap, llms.txt, SPA detection and cross-model brand mentions.
Local-first web scraping, DuckDuckGo search, site mapping and crawling, using trafilatura with an optional Firecrawl fallback for JS-rendered pages.
Web scraping via ScraperAPI with JavaScript rendering, geotargeting, premium proxies and auto-parsing.
Identity and credential gateway for agents: Ed25519 identity, encrypted credential vault, OPA policies with audit logging, per-agent memory and web scraping.
URL fetcher and HTML-to-Markdown converter with layered prompt injection defense: hidden-element sanitization, risk scanning and content boundary wrapping.
Web pages as clean data, plus RugProbe Base token safety checks. Pay per call in USDC (x402).
Use Apify Actors, pre-built cloud tools, to extract data from websites, e-commerce, social media, search engines, maps and more.
Compare two web pages on desktop and phone: percent changed, changed regions, diff image, PASS/FAIL.
Current pricing, cost estimates and recommendations for web scraping APIs, with official sources.
Fetch public web pages live and get clean Markdown, text or HTML.
Business emails from websites, UK & EU contract award winners, website tech stacks. Apify Actors.
Jobs, ads, SEO, keywords, YouTube and AliExpress data for AI agents. Runs on Apify.
Transcripts of YouTube videos and Shorts in bulk, with timestamps. Runs on Apify.
AI speech-to-text for public TikTok videos: SRT, VTT, word timings, speaker labels, 90+ languages.
Google Jobs listings for any role and city: company, salary, job type, apply link. Runs on Apify.
Geocode US addresses to census tracts and add ACS demographics. Runs on Apify.
Spreadsheet, census tract, PDF-to-Markdown, SEO/AI audit and video frame tools for agents, on Apify.
PDF, DOCX and scans to Markdown, CSV tables and RAG chunks with OCR. Runs on Apify.
Audit pages for SEO and AI search: scores, fixes, AI crawler access, llms.txt. Pay per use.
Clean, dedupe, join and reconcile CSV/Excel files. Runs on Apify, pay per use.
Evidence-bearing public webpage monitoring with hashes, diffs, signed webhooks, REST, and MCP.
Web pages your agent can quote: 1,000 full fetches a month free, no card, then $0.01 a page.
Instagram reel transcripts from the audio, in the language spoken, via Apify. Silent reels free.
MCP server for the Scrawly SEO crawler: query local audits and propose confirmation-gated fixes.
Google Trends for AI agents, no key: Trending Now, interest over time, rising searches, Pinterest.
Pinterest Trends for AI agents, no login: keyword research, forecast, audience, trending searches.
DNS lookup, WHOIS, SSL checker, and HTTP status tools. Uses public APIs only, no scraping.
Web scraping, extraction and search for AI agents. Hosted MCP with workspace API keys.
MCP server for loading authenticated web pages using Chrome DevTools Protocol
Search, scrape, and crawl the web for AI agents. Batch scraping and answers with citations.
Scan and fix your site's AI discoverability: crawler access, llms.txt, JSON-LD. Free.
Can ChatGPT, Claude and Perplexity crawlers read your page? Audit, fixes, compare, monitoring.
Free check whether a website is technically open to AI assistants (AI crawlers, llms.txt, JSON-LD).
完整目錄
爬取、擷取與監控公開網頁。共 1,406 個伺服器,最常用的排在前面。每個都會開啟發布者的頁面。