Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Crawl a website for links, file download links, HTML tables as rows, and lists, with no AI
Capture batches of public web pages in a real browser and download the screenshots as a zip.
Specialist workflows for marketing, web, content and automation. Paid account; OAuth required.
Scrape and analyze public LinkedIn posts as structured JSON via the Apify LinkedIn Posts API.
Live web search results and web pages as Markdown; page fetches use a residential network.
Finds the robots.txt lines that reserve nothing — retired crawler tokens, shadowed groups, rules abo
Scrape Hacker News stories, comments and user profiles as clean JSON with points, author…
Cookieless analytics for agents: visitors, pages, sources, AI crawlers, funnels and revenue.
70 pay-per-call APIs: DNS, email auth, security headers, web search, scraping, AI. x402 USDC/Base
Check what AI crawlers may read and what a page permits, generate llms.txt and schema.
Checks robots.txt line by line and says what each AI crawler token actually controls - training, AI
MCP server: webpage-to-Markdown extraction, paid per call in USDC on Base via x402. One tool, extract_markdown(url).
23 live research tools, from web search to SEC filings. Each call bills the scraper it runs.
Convert PDF, Word, Excel, PowerPoint, e-mails and web pages to clean Markdown for LLMs.
x402 paid API for AI agents. USDC on Base. CDP facilitator.
Reddit posts and comments by keyword, subreddit, or thread URL, via Arctic Shift or your search key
Web pages into decision-ready state: dates, numbers with units, budgeted chunks. No model, no key.
Does an AI crawler see your site? Free registry lookup, x402 pricing; checks paid in USDC on Base.
Convert HTML to Markdown or strip to plain text. For web-scraping agents.
Pay-per-call MCP tools: web search, scraping, on-chain wallet intel, extraction. x402 USDC on Base.
ChatGPT answers with web search as JSON: citations, sources, brands and products. Bulk prompts.
Scraping API for Google Search, Maps, Flights, Jobs, YouTube, LinkedIn, Trustpilot and more.
Apify MCP — run web-scraping Actors and fetch their dataset results.
Run up to 25 Apify Actors in order in one call, passing each step's dataset to the next.
Shows if GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can read each site in a list.
Google Web Search: Google Web Search API. Search the world’s information, including webpages.
Returns GROUP BY and pivot tables for any Apify dataset, file or Google Sheet by URL, or JSON...
Returns what was added, removed or changed between two Apify datasets, files, Google Sheets or...
Returns rows of any Apify dataset, CSV/Excel/JSON file URL or Google Sheet that pass filter...
Search public Siemens web content, product catalog, customer references and media.
Merge many Apify datasets, CSV files and Google Sheets into one list with duplicates removed.
Depop listings for searches or URLs, with alerts on new listings, price drops and sales.
Bulk phone validation for an Apify dataset, CSV/Excel/JSON file or Google Sheet
Affiliate product search for AI agents. Merchant feeds, not web scraping. Works with any MCP client.
SEO check of any website and AI crawler access test (access pass), and the full Audit.
ไดเรกทอรีทั้งหมด
ครอว์ล ดึงข้อมูล และมอนิเตอร์เว็บสาธารณะ 1,395 เซิร์ฟเวอร์ เรียงจากที่ใช้มากที่สุด แต่ละรายการจะเปิดหน้าของผู้เผยแพร่