Purify
Web Scraping API for AI Agents. One binary. Zero dependencies. Built-in MCP server.
The Go alternative to Firecrawl / Crawl4AI — no Python, no Node.js, no Redis. Just docker compose up and go.
Get Free API Key · API Docs · MCP Server · Self-Host
See it work
# Start Purify
docker compose up -d
# Scrape a page
curl -s -X POST http://localhost:8080/api/v1/scrape \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq .
{
"success": true,
"content": "# Hacker News\n\n1. Show HN: ...",
"tokens": {
"original_estimate": 11708,
"cleaned_estimate": 5572,
"savings_percent": 52.4
},
"timing": {
"total_ms": 400
}
}
Two commands. Clean Markdown back in under a second.
How it compares
Quick start
Option A: Docker
docker compose up -d
Option B: Build from source
git clone https://github.com/Easonliuliang/purify.git
cd purify && make build
PURIFY_AUTH_ENABLED=false ./bin/purify
Option C: Hosted API (no setup)
curl -s -X POST https://purify.verifly.pro/api/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://news.ycombinator.com"}' | jq .content
Get a free API key at purify.verifly.pro — 1,000 requests/month, no credit card.
Token savings — real numbers
Measured with tiktoken (GPT-4 tokenizer). Purify strips navigation, ads, scripts, and styling — your LLM only sees the content.
Low-savings sites (Hacker News, paulgraham.com) are already minimal — almost pure text with no cruft to remove. That's a feature, not a bug.
Use cases
- AI agents — Give your agent web access via MCP or REST API
- RAG pipelines — Scrape docs, get clean Markdown, embed into your vector DB
- Trading bots — Scrape prediction markets and news with sub-500ms latency
- Research assistants — Read and summarize any web page
MCP server
Purify includes a built-in MCP server with 5 tools:
Setup
Add to your Claude Desktop config (claude_desktop_config.json):
{
"mcpServers": {
"purify": {
"command": "purify-mcp",
"env": {
"PURIFY_API_URL": "https://purify.verifly.pro",
"PURIFY_API_KEY": "your-api-key"
}
}
}
}
For self-hosted instances, set PURIFY_API_URL to http://localhost:8080.
Then ask Claude:
- "Scrape https://paulgraham.com/greatwork.html and summarize it."
- "Crawl the Next.js docs site, max 20 pages."
- "Extract the product name and price from this page: ..."
API
POST /api/v1/scrape
Scrape a single page and return cleaned content. Supports JSON response or SSE streaming.
{
"url": "https://example.com/article",
"output_format": "markdown",
"extract_mode": "readability"
}
Response:
{
"success": true,
"status_code": 200,
"final_url": "https://example.com/article",
"content": "# Article Title\n\nClean markdown content...",
"metadata": {
"title": "Article Title",
"author": "Author Name",
"language": "en",
"source_url": "https://example.com/article",
"fetch_method": "http"
},
"links": {
"internal": [{"href": "/about", "text": "About"}],
"external": [{"href": "https://github.com/...", "text": "GitHub"}]
},
"images": [{"src": "https://example.com/hero.jpg", "alt": "Hero"}],
"tokens": {
"original_estimate": 32895,
"cleaned_estimate": 187,
"savings_percent": 99.43
},
"timing": {
"total_ms": 1172,
"navigation_ms": 1162,
"cleaning_ms": 9
},
"engine_used": "http"
}
SSE streaming
Add Accept: text/event-stream header to receive Server-Sent Events instead of JSON:
curl -X POST https://purify.verifly.pro/api/v1/scrape \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-H "Accept: text/event-stream" \
-d '{"url": "https://example.com"}'
Events: scrape.started → scrape.navigated → scrape.completed (or scrape.error).
Citation format
Use "output_format": "markdown_citations" to convert inline links to academic-style references:
See [Google][1] and [GitHub][2]
---
[1]: https://google.com
[2]: https://github.com
POST /api/v1/batch/scrape
Scrape multiple URLs in parallel. Returns a job ID for async polling.
{
"urls": ["https://a.com", "https://b.com", "https://c.com"],
"options": {"output_format": "markdown"},
"webhook_url": "https://your-server.com/callback",
"webhook_secret": "your-hmac-secret"
}
Poll status: GET /api/v1/batch/:id
POST /api/v1/crawl
Recursively crawl a website starting from a URL.
{
"url": "https://docs.example.com",
"max_depth": 3,
"max_pages": 100,
"scope": "subdomain",
"webhook_url": "https://your-server.com/callback",
"webhook_secret": "your-hmac-secret"
}
Poll status: GET /api/v1/crawl/:id
POST /api/v1/map
Discover all URLs on a site without scraping content.
{"url": "https://example.com"}
POST /api/v1/extract
Structured data extraction using your own LLM key (BYOK).
curl -X POST https://purify.verifly.pro/api/v1/extract \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/product",
"schema": {
"name": "string",
"price": "number",
"features": ["string"]
},
"llm_api_key": "your-openai-key"
}'
Webhook callbacks
Batch and Crawl endpoints support webhook notifications. When a job completes, Purify sends a POST request to your webhook_url with HMAC-SHA256 signature in the X-Purify-Signature header.
Events: batch.completed, crawl.page, crawl.completed, crawl.failed
Verify the signature:
HMAC-SHA256(webhook_secret, request_body) == X-Purify-Signature (sha256=<hex>)
GET /api/v1/health
Returns server status and uptime.
Configuration
All configuration via environment variables:
Self-hosting
Purify is a single Go binary. No Docker required, no Redis, no database.
# Local development (no auth)
PURIFY_AUTH_ENABLED=false ./bin/purify
# Production (with API key)
PURIFY_API_KEYS=your-secret-key ./bin/purify
Runs on any $5/month VPS. No usage limits when self-hosted.
System requirements
- Any Linux, macOS, or Windows machine
- ~15 MB disk space
- ~30 MB RAM idle
Pricing
Contributing
Contributions welcome. Please open an issue first to discuss what you'd like to change.
License
Apache 2.0 — use it however you want, commercially or otherwise. No AGPL restrictions.