Freecrawl MCP सर्वर | PlugMCP
सामग्री पर जाएँ FreeCrawl MCP Server
A production-ready Model Context Protocol (MCP) server for web scraping and document processing, designed as a self-hosted replacement for Firecrawl.
🚀 Features
JavaScript-enabled web scraping with Playwright and anti-detection measures
A production-ready mcp server for web scraping and document processing. Drop-in replacement for FireCrawl.
किसी भी टूल को किसी भी एजेंट से जोड़ें। उन लोगों के लिए MCP निर्देशिका जो सर्वर नहीं लिखते।
उत्पाद
निर्देशिका
डेवलपर्स
कंपनी
Document processing with fallback support for various formats
Concurrent batch processing with configurable limits
Intelligent caching with SQLite backend
Rate limiting per domain
Comprehensive error handling with retry logic
Easy installation via uvx or local development setup
Health monitoring and metrics collection
MCP Config (using uvx) {
"mcpServers": {
"freecrawl": {
"command": "uvx",
"args": ["freecrawl-mcp"],
}
}
}
📦 Installation & Usage
Quick Start with uvx (Recommended) The easiest way to use FreeCrawl is with uvx, which automatically manages dependencies:
# Install browsers on first run
uvx freecrawl-mcp --install-browsers
# Test functionality
uvx freecrawl-mcp --test
Local Development Setup For local development or customization:
Clone from GitHub:
git clone https://github.com/dylan-gluck/freecrawl-mcp.git
cd freecrawl-mcp
Set up environment:
# Sync dependencies
uv sync
# Install browser dependencies
uv run freecrawl-mcp --install-browsers
# Run tests
uv run freecrawl-mcp --test
Run the server:
uv run freecrawl-mcp
🛠 Configuration Configure FreeCrawl using environment variables:
Basic Configuration # Transport (stdio for MCP, http for REST API)
export FREECRAWL_TRANSPORT=stdio
# Browser pool settings
export FREECRAWL_MAX_BROWSERS=3
export FREECRAWL_HEADLESS=true
# Concurrency limits
export FREECRAWL_MAX_CONCURRENT=10
export FREECRAWL_MAX_PER_DOMAIN=3
# Cache settings
export FREECRAWL_CACHE=true
export FREECRAWL_CACHE_DIR=/tmp/freecrawl_cache
export FREECRAWL_CACHE_TTL=3600
export FREECRAWL_CACHE_SIZE=536870912 # 512MB
# Rate limiting
export FREECRAWL_RATE_LIMIT=60 # requests per minute
# Logging
export FREECRAWL_LOG_LEVEL=INFO
Security Settings # API authentication (optional)
export FREECRAWL_REQUIRE_API_KEY=false
export FREECRAWL_API_KEYS=key1,key2,key3
# Domain blocking
export FREECRAWL_BLOCKED_DOMAINS=localhost,127.0.0.1
# Anti-detection
export FREECRAWL_ANTI_DETECT=true
export FREECRAWL_ROTATE_UA=true
🔧 MCP Tools FreeCrawl provides the following MCP tools:
freecrawl_scrapeScrape content from a single URL with advanced options.
url (string): URL to scrape
formats (array): Output formats - ["markdown", "html", "text", "screenshot", "structured"]
javascript (boolean): Enable JavaScript execution (default: true)
wait_for (string, optional): CSS selector or time (ms) to wait
anti_bot (boolean): Enable anti-detection measures (default: true)
headers (object, optional): Custom HTTP headers
cookies (object, optional): Custom cookies
cache (boolean): Use cached results if available (default: true)
timeout (number): Total timeout in milliseconds (default: 30000)
{
"name": "freecrawl_scrape",
"arguments": {
"url": "https://example.com",
"formats": ["markdown", "screenshot"],
"javascript": true,
"wait_for": "2000"
}
}
freecrawl_batch_scrapeScrape multiple URLs concurrently.
urls (array): List of URLs to scrape (max 100)
concurrency (number): Maximum concurrent requests (default: 5)
formats (array): Output formats (default: ["markdown"])
common_options (object, optional): Options applied to all URLs
continue_on_error (boolean): Continue if individual URLs fail (default: true)
{
"name": "freecrawl_batch_scrape",
"arguments": {
"urls": [
"https://example.com/page1",
"https://example.com/page2"
],
"concurrency": 3,
"formats": ["markdown", "text"]
}
}
freecrawl_extractExtract structured data using schema-driven approach.
url (string): URL to extract data from
schema (object): JSON Schema or Pydantic model definition
prompt (string, optional): Custom extraction instructions
validation (boolean): Validate against schema (default: true)
multiple (boolean): Extract multiple matching items (default: false)
{
"name": "freecrawl_extract",
"arguments": {
"url": "https://example.com/product",
"schema": {
"type": "object",
"properties": {
"title": {"type": "string"},
"price": {"type": "number"}
}
}
}
}
freecrawl_process_documentProcess documents (PDF, DOCX, etc.) with OCR support.
file_path (string, optional): Path to document file
url (string, optional): URL to download document from
strategy (string): Processing strategy - "fast", "hi_res", "ocr_only" (default: "hi_res")
formats (array): Output formats - ["markdown", "structured", "text"]
languages (array, optional): OCR languages (e.g., ["eng", "fra"])
extract_images (boolean): Extract embedded images (default: false)
extract_tables (boolean): Extract and structure tables (default: true)
{
"name": "freecrawl_process_document",
"arguments": {
"url": "https://example.com/document.pdf",
"strategy": "hi_res",
"formats": ["markdown", "structured"]
}
}
freecrawl_health_checkGet server health status and metrics.
{
"name": "freecrawl_health_check",
"arguments": {}
}
🔄 Integration with Claude Code
MCP Configuration Add FreeCrawl to your MCP configuration:
{
"mcpServers": {
"freecrawl": {
"command": "uvx",
"args": ["freecrawl-mcp"]
}
}
}
Using local development setup:
{
"mcpServers": {
"freecrawl": {
"command": "uv",
"args": ["run", "freecrawl-mcp"],
"cwd": "/path/to/freecrawl-mcp"
}
}
}
Usage in Prompts Please scrape the content from https://example.com and extract the main article text in markdown format.
Claude Code will automatically use the freecrawl_scrape tool to fetch and process the content.
🚀 Performance & Scalability
Resource Usage
Memory : ~100MB base + ~50MB per browser instance
CPU : Moderate usage during active scraping
Storage : Cache grows based on configured limits
Throughput
Single requests : 2-5 seconds typical response time
Batch processing : 10-50 concurrent requests depending on configuration
Cache hit ratio : 30%+ for repeated content
Optimization Tips
Enable caching for frequently accessed content
Adjust concurrency based on target site rate limits
Use appropriate formats - markdown is faster than screenshots
Configure rate limiting to avoid being blocked
🛡 Security Considerations
Anti-Detection
Rotating user agents
Realistic browser fingerprints
Request timing randomization
JavaScript execution in sandboxed environment
Input Validation
URL format validation
Private IP blocking
Domain blocklist support
Request size limits
Resource Protection
Memory usage monitoring
Browser pool size limits
Request timeout enforcement
Rate limiting per domain
🔧 Troubleshooting
Common Issues
Debug Mode export FREECRAWL_LOG_LEVEL=DEBUG
uvx freecrawl-mcp --test
export FREECRAWL_LOG_LEVEL=DEBUG
uv run freecrawl-mcp --test
📈 Monitoring & Observability
Health Metrics
Browser pool status
Memory and CPU usage
Cache hit rates
Request success rates
Response times
Logging FreeCrawl provides structured logging with configurable levels:
ERROR: Critical failures
WARNING: Recoverable issues
INFO: General operations
DEBUG: Detailed troubleshooting
🔧 Development
Running Tests # Basic functionality test
uvx freecrawl-mcp --test
# Basic functionality test
uv run freecrawl-mcp --test
Code Structure
Core server : FreeCrawlServer class
Browser management : BrowserPool for resource pooling
Content extraction : ContentExtractor with multiple strategies
Caching : CacheManager with SQLite backend
Rate limiting : RateLimiter with token bucket algorithm
📄 License This project is licensed under the MIT License - see the technical specification for details.
🤝 Contributing
📚 Technical Specification For detailed technical information, see ai_docs/FREECRAWL_TECHNICAL_SPEC.md.
FreeCrawl MCP Server - Self-hosted web scraping for the modern web 🚀
डायरेक्टरी स्कोर 6/100 सूचीबद्ध 10 में से 10
आधिकारिक MCP Registry में 18 में से 0
मेंटेनर द्वारा क्लेम किया गया 8 में से 0
टूल सूची के साथ जवाब देता है 12 में से 0
GitHub स्टार 16 में से 4
साप्ताहिक डाउनलोड 10 में से 0
SparkForge Access a versatile collection of tools for media generation, web scraping, and cryptocurrency research. Convert between data formats, summarize long documents, and extract text from images or PDFs wit Scrapling MCP Server Web scraping with stealth HTTP, real browsers, and Cloudflare bypass. CSS selectors supported. Scrapling 🕷️ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and trick Fetch Fetch a web page and return it as clean markdown for the agent, in chunks for long pages. The reference server from the Model Context Protocol project. Obscura The headless browser for AI agents and web scraping Douyin TikTok Download API 🚀 自托管的抖音、TikTok、B 站数据采集与无水印视频下载 API,支持 AI agent 通过 MCP 调用与离线语音转文字,Docker 一键部署。| Self-hosted TikTok, Douyin & Bilibili scraper and no-watermark video downloader — async REST API, MCP server for AI age Web Scout Search the web and extract clean, readable text from webpages. Process multiple URLs at once to speed up research with reliable throttling and error handling. Quickly compile sources and summaries for Apify MCP Server Extract data from any website with thousands of scrapers, crawlers, and automations on Apify Store ⚡ Apify The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automa Skill Seekers Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection SeleniumBase 📊 Python APIs for Web Automation, Testing, and Scraping. CDP Mode launches a stealth browser that bypasses bot-detection/CAPTCHAs. Adds stealth to Playwright and has an MCP server for AI agents. Mobile Model Context Protocol Server for Mobile Automation and Scraping (iOS, Android, Emulators, Simulators and Real Devices) Web scraping MCP सर्वर सबसे अच्छे Web scraping MCP सर्वर
अभी टूल सूची नहीं है। रिमोट सर्वर को पहले हेल्थ चेक के बाद सूची मिलती है।
Freecrawl कैसे इंस्टॉल करें? इस पेज पर Cursor, Claude Desktop, Claude Code या VS Code के लिए stdio कॉन्फ़िग कॉपी करें। यह पब्लिशर का इंस्टॉल इस्तेमाल करता है, PlugMCP प्रॉक्सी नहीं।
क्या Freecrawl के लिए API key चाहिए? इस रिकॉर्ड में कोई ज़रूरी एनवायरनमेंट वेरिएबल दर्ज नहीं है।
Freecrawl कौन सा ट्रांसपोर्ट इस्तेमाल करता है? Freecrawl stdio पर लोकल चलता है।
Freecrawl की पिछली जाँच कब हुई थी? पिछली जाँच 2026-10-28 को हुई। A production-ready mcp server for web scraping and document processing. Drop-in replacement for FireCrawl.
टूल के विवरण मौजूद हैं 6 में से 0
वॉइस के लिए तैयार 3 में से 0