A production-ready Model Context Protocol (MCP) server that enables AI agents to access structured web data using ScrapingBee as a tool.
This project demonstrates how to transform ScrapingBee into an AI-native data layer. Instead of embedding scraping logic directly inside prompts, AI agents register ScrapingBee as a tool and request structured web data on demand.
Scrape any web page, search Google/Amazon/Walmart/YouTube and extract data via the ScrapingBee API.
2026-11-08
ดาว
—
เครื่องมือ
—
อัปไทม์
—
If you are building:
• AI agents that require live web access
• Autonomous research systems
• Ecommerce intelligence bots
• Market monitoring agents
• LLM workflows with tool execution
• AI-powered web scraping infrastructure
This repository provides a complete MCP integration blueprint.
Quick start
The server is hosted by ScrapingBee at https://mcp.scrapingbee.com/mcp (Streamable HTTP) —
there is nothing to install. You need a ScrapingBee API key
(get one here; trial credits included),
sent as the header Authorization: Bearer <key>. Registry name: com.scrapingbee/mcp.
Fetch a URL and return its content as text or markdown (JavaScript rendered, blocks handled)
get_page_html
Fetch a URL and return the rendered HTML
get_screenshot
Fetch a URL and return a screenshot
get_file
Download a file (PDF, image, …) from a URL
extract_page_data
Fetch a URL and extract specific data with CSS or XPath selectors
fast_search
Fast web search returning clean organic results and top stories — the first choice for general searches
get_google_search_results
Google SERP as JSON for specialised searches: news, maps, Lens, shopping, images, AI mode
get_amazon_search_results
Amazon search results as JSON
get_amazon_product_details
Amazon product details by ASIN
get_amazon_pricing
Current Amazon price and seller offers by ASIN
get_walmart_search_results
Walmart search results as JSON
get_walmart_product_details
Walmart product details by ID
get_youtube_search_results
YouTube search results
get_youtube_video_metadata
Title, description, stats and more for a video
get_youtube_video_subtitles
Subtitles / transcript for a video
ask_chatgpt
Send a prompt to ChatGPT, optionally with web search
ask_gemini
Send a prompt to Gemini, with citations when available
get_scrapingbee_usage
Remaining credits and concurrency for your account
Each call consumes API credits exactly as the equivalent ScrapingBee API request would
(see pricing and the
API documentation). The server handles one page or
query per call; for hundreds of URLs, crawling a site, writing results to files or scheduling
recurring jobs, use the ScrapingBee CLI.
Relevant keywords:
scrapingbee mcp
model context protocol server
mcp scraping
ai agent web scraping
mcp web scraping tool
What Is Model Context Protocol (MCP)?
Model Context Protocol (MCP) defines how AI systems interact with external tools in a structured way.
Instead of letting a language model guess how to scrape or browse the web, MCP allows it to:
Discover available tools
Call a tool using structured parameters
Receive normalized JSON responses
Inject results back into the reasoning loop
The ScrapingBee MCP server exposes web scraping capabilities as a formally registered tool that AI systems can use safely and predictably.
The MCP server acts as a bridge between your AI agent and ScrapingBee’s managed scraping infrastructure.
This enables true AI agent web scraping without fragile prompt-based hacks.
Architecture Overview
AI Agent
→ MCP Client
→ ScrapingBee MCP Server
→ ScrapingBee API
→ Target Website
→ Structured JSON Response
→ Injected Back Into Agent Context
The AI agent never scrapes directly.
It calls tools.
The MCP server executes them.
Installation
There is nothing to install or run: the server is operated by ScrapingBee at
https://mcp.scrapingbee.com/mcp. The only setup is your API key, which every client below
sends as a Authorization: Bearer header:
export SCRAPINGBEE_API_KEY=your_api_key
Connecting to the ScrapingBee MCP Server
You can connect in two ways:
Via Claude Desktop (remote MCP)
Via a custom Python MCP client
Option 1: Connect to Claude Desktop
Add the ScrapingBee MCP server to your claude_desktop_config.json file:
async def main():
mcp_client = MCPClient(MCP_SERVER_URL, SCRAPINGBEE_API_KEY)
tools_response = await mcp_client.initialize()
genai.configure(api_key=GEMINI_API_KEY)
gemini_tools = format_tools_for_gemini(tools_response["result"]["tools"])
model = genai.GenerativeModel(
model_name="gemini-2.5-pro",
tools=gemini_tools
)
user_prompt = input("Ask the AI to perform a task:\n> ")
response = model.generate_content(user_prompt)
function_call = response.candidates[0].content.parts[0].function_call
tool_name = function_call.name
tool_args = dict(function_call.args)
result = await mcp_client.call_tool(tool_name, tool_args)
print(json.dumps(result, indent=2))
What This Enables
With this architecture you can:
• Build AI agents that reason about available tools
• Allow LLMs to dynamically choose scraping actions
• Execute structured web data extraction
• Inject real-world data into model context
• Create scalable AI-native web scraping systems
Conclusion
The ScrapingBee MCP Server transforms web scraping into a structured, tool-driven data layer for AI systems.
Instead of brittle prompt scraping, you get:
• Model Context Protocol compliance
• JSON-RPC structured execution
• AI agent web scraping
• Scalable tool orchestration
This is AI-native web data access.
To start building AI-powered agents today, Get your free API key and connect to the ScrapingBee MCP server in minutes.