🕷️ MCP Web Scrape
Clean, cached web content for agents—Markdown + citations, robots-aware, ETag/304 caching.
接入你的智能体
從發布者安裝。不是 PlugMCP 端點。
遠端 URL 還是本機指令? 兩種安裝方式的差異
來自發布者
Clean, cached web content for agents—Markdown + citations, robots-aware, ETag/304 caching.
🚀 mcp-web-scrape — Clean, cache-aware web content fetcher for AI agents. Fetch any URL → extract readable content → return Markdown/JSON with citations. ⚡ Fast caching, 🤝 robots.txt compliant, 📝 Markdown-ready output, �� works with ChatGPT/Claude Desktop.
Current Version: 1.0.7
# Extract content from any webpage
npx [email protected]
# Example: Extract from a news article
> extract_content https://news.ycombinator.com
✅ Extracted 1,247 words with 5 citations
📄 Clean Markdown ready for your AI agent
# Extract all forms from a webpage
> extract_forms https://example.com/contact
✅ Found 3 forms with 12 input fields
# Parse tables into structured data
> extract_tables https://example.com/data --format json
✅ Extracted 5 tables with 247 rows
# Find social media profiles
> extract_social_media https://company.com
✅ Found Twitter, LinkedIn, Facebook profiles
# Analyze sentiment of content
> sentiment_analysis https://blog.example.com/article
✅ Sentiment: Positive (0.85), Emotional tone: Optimistic
# Extract named entities
> extract_entities https://news.example.com/article
✅ Found 12 people, 8 organizations, 5 locations
# Check for security vulnerabilities
> scan_vulnerabilities https://mysite.com
✅ No XSS vulnerabilities found, 2 header improvements suggested
# Analyze competitor SEO
> analyze_competitors ["https://competitor1.com", "https://competitor2.com"]
✅ Competitor analysis complete: keyword gaps identified
# Monitor uptime and performance
> monitor_uptime https://mysite.com --interval 300
✅ Uptime: 99.9%, Average response: 245ms
# Generate comprehensive report
> generate_reports https://website.com --metrics ["seo", "performance", "security"]
✅ Generated 15-page analysis report
# Install globally
npm install -g [email protected]
# Try it instantly (latest version)
npx mcp-web-scrape@latest
# Try specific version
npx [email protected]
# Or start HTTP server
node dist/http.js
Add to your ~/Library/Application Support/ChatGPT/config.json:
{
"mcpServers": {
"web-scrape": {
"command": "npx",
"args": ["[email protected]"]
}
}
}
Add to your ~/Library/Application Support/Claude/claude_desktop_config.json:
{
"mcpServers": {
"web-scrape": {
"command": "npx",
"args": ["[email protected]"]
}
}
}
Deterministic Results → Same URL always returns identical content
Smart Citations → Every fact links back to its source
Robots Compliant → Respects robots.txt and rate limits
Lightning Fast → ETag/304 caching + persistent storage
Agent-Optimized → Clean Markdown instead of messy HTML
# Install specific version
npm install -g [email protected]
# Or use directly (latest)
npx mcp-web-scrape@latest
# Or use specific version
npx [email protected]
# Environment variables
export MCP_WEB_SCRAPE_CACHE_DIR="./cache"
export MCP_WEB_SCRAPE_USER_AGENT="MyBot/1.0"
export MCP_WEB_SCRAPE_RATE_LIMIT="1000"
STDIO (default)
mcp-web-scrape
HTTP/SSE
node dist/http.js --port 3000
Access cached content as MCP resources:
cache://news.ycombinator.com/path → Cached page content
cache://stats → Cache statistics
cache://robots/news.ycombinator.com → Robots.txt status
We love contributions! See CONTRIBUTING.md for guidelines.
Good First Issues:
MIT © Mahipal
Built with ❤️ for the Model Context Protocol ecosystem
🚀 mcp-web-scrape — Clean, cache-aware web content fetcher for AI agents. Fetch any URL → extract readable content → return Markdown/JSON with citations. ⚡ Fast caching, 🤝 robots.txt compliant, 📝 Ma
總覽
🚀 mcp-web-scrape — Clean, cache-aware web content fetcher for AI agents. Fetch any URL → extract readable content → return Markdown/JSON with citations. ⚡ Fast caching, 🤝 robots.txt compliant, 📝 Markdown-ready output, �� works with ChatGPT/Claude Desktop.
工具
尚無工具清單。遠端伺服器會在首次健康檢查後取得工具清單。
常見問題
在本頁複製適用於 Cursor、Claude Desktop、Claude Code 或 VS Code 的 stdio 設定。它使用發布者的安裝方式,而不是 PlugMCP 代理伺服器。
這筆記錄未列出必要的環境變數。
MCP Web Scrape 透過 stdio 在本機執行。
上次檢查時間:2026-10-27。 🚀 mcp-web-scrape — Clean, cache-aware web content fetcher for AI agents. Fetch any URL → extract readable content → return Markdown/JSON with citations. ⚡ Fast caching, 🤝 robots.txt compliant, 📝 Markdown-ready output, �� works with ChatGPT/Claude Desktop.