PullMD
Anything in, clean Markdown out. Self-hosted conversion of web pages, documents, images, audio and YouTube videos into token-lean Markdown, for you and for your AI agents. Quick start · Use with AI agents · API · Configuration · Changelog
Why PullMD
Agents are bad at reading the web. Raw HTML burns context on navigation, cookie banners and scripts, and built-in fetch tools often stumble on JavaScript-heavy pages, PDFs and comment threads. PullMD sits in between: one URL or file in, clean Markdown out, with source and metadata in YAML frontmatter instead of the body.
curl "https://pullmd.example.com/api?url=https://example.com/article"
What that saves, measured on three public pages (tokens counted with the
cl100k_base tokenizer, 2026-10-16):
The curl column is what a naive fetch tool hands an agent. The last
column is query-scoped extraction: only the
sections relevant to a question.
Highlights
Reads almost anything
- Web pages through a fallback chain: Cloudflare's native Markdown, then Readability + Trafilatura (best result wins), then headless Chromium for JavaScript-heavy sites
- Reddit and Hacker News threads with their comment trees (depth and count configurable)
- Documents: PDF, Word, PowerPoint, Excel, EPUB, by URL or upload, plus an optional OCR tier for table-grade PDFs
- YouTube title, description and transcript with clickable timecodes, no API key needed
- Images and audio via any OpenAI-compatible or local model (captions, transcripts)
Built for agents
- MCP server for Claude Code and other MCP clients, with OAuth 2.1 for claude.ai and Claude Desktop
- Query-scoped extraction:
?query=returns only the sections relevant to a question, typically 70-95% fewer tokens on long pages - Lean output: clean body, metadata in frontmatter, unpadded tables, configurable field allowlist
- Live share links:
/s/:idre-fetches when the cached copy is older than an hour and serves the last good snapshot if the source dies
Comfortable to self-host
- One
docker compose up, zero config, multi-arch images (amd64/arm64) on Docker Hub and GHCR - Every sidecar is optional, and PullMD degrades gracefully without it
- Auth modes from open homelab to multi-user, SSRF protection, rate limits, a health endpoint for Uptime Kuma
- Per-site extraction recipes as declarative JSON
Four ways in: PWA · REST API · MCP server · Claude Code skill
Quick start
mkdir pullmd && cd pullmd
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.yml
docker compose up -d
# → http://localhost:3000
No .env needed: every variable has a sensible default. Add a .env next
to the compose file to override anything (see Configuration).
Then connect your agent: see AI-agent integration.
Upgrading? v3's only breaking change is the clean-body output format (one-line opt-out in
MIGRATION.md). Running behind a reverse proxy? Since v3.13 you needPULLMD_TRUST_PROXY. Full history in the changelog.
docker-compose.yml (zero-config, abridged)
services:
pullmd:
image: aeternalabshq/pullmd:latest
container_name: pullmd
restart: unless-stopped
ports:
- "${PORT:-3000}:3000"
environment:
- PUBLIC_URL=${PUBLIC_URL:-http://localhost:${PORT:-3000}}
- TRAFILATURA_URL=http://trafilatura:8001/extract
- PLAYWRIGHT_URL=http://playwright:8002/render
- MARKITDOWN_URL=http://markitdown:8003/convert
- CACHE_DB=/data/cache.db
volumes:
- ./data:/data
networks:
- pullmd-internal
depends_on:
- trafilatura
- playwright
- markitdown
trafilatura:
image: aeternalabshq/pullmd-trafilatura:latest
container_name: pullmd-trafilatura
restart: unless-stopped
networks:
- pullmd-internal
playwright:
image: aeternalabshq/pullmd-playwright:latest
container_name: pullmd-playwright
restart: unless-stopped
networks:
- pullmd-internal
markitdown:
image: aeternalabshq/pullmd-markitdown:latest
container_name: pullmd-markitdown
restart: unless-stopped
mem_limit: ${MARKITDOWN_MEM_LIMIT:-1g}
networks:
- pullmd-internal
networks:
pullmd-internal:
driver: bridge
Abridged for readability — the
docker-compose.ymlin the repo additionally passes every optional.envvariable through to the containers (Reddit credentials, auth, media/OCR keys, YouTube options, output shaping). Use thecurl -Ocommand above rather than copying this block, or.envoverrides beyond the basics won't reach the containers.
Note: the Playwright sidecar adds ~3.7 GB to your image cache (Chromium + Firefox + WebKit binaries from the official Playwright base image). It's optional — leave
PLAYWRIGHT_URLunset and theplaywrightservice block off, and PullMD silently degrades to static extraction with a fallback note in the metadata.
Note: the MarkItDown sidecar is optional. Leave
MARKITDOWN_URLunset and remove themarkitdownservice block to disable document conversion. Web-page URLs always work without it.
Mirror on GHCR:
ghcr.io/aeternalabshq/{pullmd,pullmd-trafilatura,pullmd-playwright,pullmd-markitdown}. Replace theimage:lines if you prefer GitHub's registry.
Behind Traefik
For deployments behind Traefik with TLS, use docker-compose.traefik.yml
instead. Same images, but with Traefik labels and the proxy external
network. Set HOST_DOMAIN in .env:
curl -O https://raw.githubusercontent.com/AeternaLabsHQ/pullmd/main/docker-compose.traefik.yml
echo "HOST_DOMAIN=pullmd.example.com" > .env
docker compose -f docker-compose.traefik.yml up -d
This file sets PULLMD_TRUST_PROXY=1: Traefik is the only peer (no port is
published), so PullMD takes the client address and the https scheme from
Traefik's forwarding headers.
Behind another reverse proxy
PullMD ignores X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host
unless PULLMD_TRUST_PROXY says the connecting peer is a proxy. Behind nginx,
Caddy, a tunnel or a load balancer, set:
PULLMD_TRUST_PROXY=1 # one proxy hop, port not reachable directly
# or: PULLMD_TRUST_PROXY=172.18.0.0/16 # the proxy's address/network
PUBLIC_URL=https://pullmd.example.com
A hop count trusts whatever connects, so use it only when the container port is
not published to anyone but the proxy; otherwise list the proxy's address.
Without the setting, all clients share one rate-limit bucket (login, signup,
share-link misses, OAuth, media budget) and PullMD logs a one-time warning.
PUBLIC_URL keeps the session cookie Secure and is accepted as this site's
host by the login form check. Do not send Referrer-Policy: no-referrer for
PullMD's pages: browsers then post forms with Origin: null, which is refused.
See MIGRATION.md.
Local development (no Docker)
git clone https://github.com/AeternaLabsHQ/pullmd.git
cd pullmd
npm install
npm start # http://localhost:3000
npm test # node --test
Configuration
All variables go in .env (copy from .env.example):
v3.0.0 output format change: the markdown body is clean by default - just
# Titlefollowed by content. The source URL, fetch date, and all extraction metadata remain in the YAML frontmatter unchanged - the body no longer duplicates them. SetPULLMD_SOURCE_HEADER=trueto restore the old inline header. UsePULLMD_FRONTMATTER_FIELDSto pick which frontmatter fields are emitted (handy for trimming tokens in agent pipelines).
PUBLIC_URL matters for self-hosting: the help page and downloadable
skill embed it as the canonical endpoint. Set it correctly and your
users get a copy-paste setup that points at your instance.
PullMD rotates its outbound User-Agent for the web fetch path from a
pool of current desktop browsers, refreshed every 48 hours from a
live feed of real-world UAs
maintained by @WinFuture23. A built-in
seed pool ensures rotation works even when the feed is unreachable. Set
PULLMD_USER_AGENT to pin a single UA, or PULLMD_UA_FEED_URL to point
at your own feed. The Reddit path keeps its dedicated REDDIT_USER_AGENT
because Reddit's API expects a stable, identifying UA.
DISABLE_PUBLIC_HISTORY=true is the privacy switch for shared
instances (multi-tenant VPS, office deployments). Conversions still
get cached and assigned share IDs; users just can't see what other
users have fetched (the MCP list_recent tool follows the same rule,
and /api/stats leaves out its per-domain lists). Anyone with a known
/s/:id link still gets their markdown back. Share ids are 128 random bits, so a link cannot
be guessed, and /s/:id throttles unknown-id lookups to 120 per
minute and IP (valid links are never throttled). With sign-in
enabled (PULLMD_AUTH_MODE=single-admin or multi-user) history is
scoped per user anyway; the switch matters for instances without
sign-in.
Authentication (v2.0+)
Version pinning:
:latesttracks the newest release (v3). v3's only breaking change is the clean-body output format — to stay on the v2.x output format instead, pin the explicit major tag:services: pullmd: image: aeternalabshq/pullmd:2
PullMD ships with three auth modes. Pick one with PULLMD_AUTH_MODE:
In single-admin and multi-user modes, PULLMD_ADMIN_EMAIL + PULLMD_ADMIN_PASSWORD bootstrap the first admin user on first startup. After that, changing these env vars does not change the password — use the admin CLI:
docker compose exec pullmd node scripts/admin.js reset-password [email protected]
Create an account without opening self-registration (useful when
PULLMD_ALLOW_SIGNUP is off):
docker compose exec pullmd node scripts/admin.js create-user [email protected]
Both commands read the password from stdin, so they need it attached:
docker compose exec, docker exec -it, or a pipe
(echo "…" | docker exec -i <container> node scripts/admin.js …). A plain
docker exec without -i aborts with an error and exit code 2 instead of
doing nothing.
Auth boundary
/api/stats returns its per-domain lists (lowQualityDomains,
fallbackByDomain) only to the admin, or to everyone when auth is off and
public history is on; the aggregate counts are public. Signed-in callers of
/api/history and MCP list_recent get their own conversions.
Cache deletes are scoped to the caller. An admin (and every caller in
disabled mode) removes the shared, URL-deduped cache row, which affects
every user's history. A regular user only unlinks the entry from their own
history - the shared row and its /s/:id share link keep working. The
response says which happened via "scope": "user" | "global".
Authentication paths
- Session cookies —
POST /loginsetspullmd_session(HttpOnly,SameSite=Lax, 90-day TTL with sliding expiry). The PWA uses this automatically. The cookie isSecurewhen the request arrived over HTTPS (directly, or through a proxy covered byPULLMD_TRUST_PROXY) or whenPUBLIC_URLstarts withhttps://.POST /loginallows 10 attempts per minute andPOST /signup5 per hour per client address (429withRetry-Afterbeyond that). Login, signup, logout and OAuth consent form posts are accepted only whenOrigin(orReferer) names this host or the host ofPUBLIC_URL; posts without either header are accepted. - API keys — generate at
/settings, send viaAuthorization: Bearer pmd_<32-char-base62>. Stored as SHA-256 hashes; only shown once at creation. - Legacy
PULLMD_AUTH_TOKEN— deprecated.single-adminmode only. Maps to admin user. Kept for migration compatibility; slated for removal in a future major release.
Migration from v1.x
See MIGRATION.md for the full upgrade checklist. The TL;DR: leave PULLMD_AUTH_MODE unset and v2.0 behaves exactly like v1.x.
OAuth 2.1 (claude.ai Web Connector)
PullMD ships with a full OAuth 2.1 Authorization Code flow so the claude.ai web app's Custom Connector feature can authenticate users against your PullMD instance. All endpoints needed by the spec are implemented: Dynamic Client Registration (RFC 7591), PKCE-S256 (RFC 7636), Authorization Server Metadata (RFC 8414), Protected Resource Metadata (RFC 9728), and Token Revocation (RFC 7009).
Setup:
- Set
PULLMD_AUTH_MODEtosingle-adminormulti-user(OAuth requires Phase-1 auth). - Set
OAUTH_JWT_SECRETto a 32+ character random string (openssl rand -hex 32). - Set
PUBLIC_URLto your instance's public origin (e.g.https://pullmd.example.com). - In claude.ai → Settings → Connectors → Add custom connector, point it at
https://pullmd.example.com/mcp— claude.ai discovers everything else automatically via the well-known endpoints. - The first time the user clicks the connector, they'll be redirected to PullMD's
/login, then to a consent screen, then back to claude.ai.
Tokens:
- Access tokens are JWTs (HS256), TTL 1 hour, audience-bound to your
/mcpURL, and accepted only on/mcp. Other routes,/api/meincluded, treat them as unknown bearers; use an API key there. - Refresh tokens are opaque (
pmd_rt_…), TTL 30 days, rotated on every refresh, with reuse-detection that invalidates the entire refresh chain on replay. - Revoke a token via
POST /oauth/revoke(RFC 7009).
Scope: Currently a single mcp:full scope (URL conversion + history read). Granular scopes are tracked for a future minor release.
Shipped in v2.3.0 (issues #6 and #10).
AI-agent integration
Three install paths. Once your instance is running, ${PULLMD_URL}/help
shows the same boxes with your URL pre-filled. Replace ${PULLMD_URL}
below with your hostname (e.g. https://pullmd.example.com).
1. Universal prompt
Drop into any chat agent (ChatGPT, Claude, Gemini, …):
When you need to read a web page, fetch it via PullMD instead of your
built-in fetch/browse tool - not just when that one fails:
GET ${PULLMD_URL}/api?url=<URL>
Returns clean Markdown (text/markdown). Optional query params:
comments=false skip Reddit comments
comment_depth=N comment nesting depth (default 3)
frontmatter=true prepend YAML metadata block
format=text strip Markdown, return plain text
nocache=true bypass the 1h cache and refetch
render=force|skip override the auto Playwright fallback
pdf=ocr high-quality PDF conversion (tables)
query=<text> got a specific question? pass it and get back only
the matching sections (70-95% fewer tokens)
max_tokens=N budget for query= (default 600, 64-20000)
lang=de|en language for the comments section header
Response headers worth checking:
X-Source reddit | hackernews | cloudflare | readability | trafilatura |
playwright | markitdown | youtube | pdf-ocr | ...
X-Quality 0.0-1.0 extraction confidence
X-Share-Id 32-hex permalink, openable as /s/<id>
X-Transcript-Status youtube only: ok | none | blocked | error
(blocked/error = transient, not cached — retry later)
Reddit URLs are auto-detected (incl. redd.it short links and /s/ shares).
Hacker News URLs are auto-detected too — items, comment permalinks, and the
front/newest/ask/show/jobs listings.
Use this whenever you would otherwise fetch raw HTML — the markdown is
much cleaner and saves significant context window space. When you only
need specific information rather than the whole document, pass your
question as query= and get back just the relevant sections.
2. Claude Code skill
pullmd.zip is auto-built with your URL embedded:
curl -O ${PULLMD_URL}/pullmd.zip
mkdir -p ~/.claude/skills
unzip pullmd.zip -d ~/.claude/skills/
# Restart Claude Code; the skill activates on web-reading requests.
Upgrading from pre-v3? The skill was renamed from
web-readertopullmdin v3.0.0. Installing the new zip does not replace an existing install — remove the old one first, or both skills will be active side by side:rm -rf ~/.claude/skills/web-reader(The old download URL
/web-reader.zipkeeps working as a redirect.)
3. MCP server
Remote MCP server at ${PULLMD_URL}/mcp (Streamable-HTTP transport, stateless).
Three tools: read_url, get_share, list_recent. Server-side updates reach
every client automatically — no local install needed. With auth enabled,
list_recent lists the caller's own conversions.
Claude Code (CLI):
claude mcp add --transport http pullmd ${PULLMD_URL}/mcp
Claude Desktop / Cursor / other MCP hosts — JSON config:
{
"mcpServers": {
"pullmd": {
"type": "http",
"url": "${PULLMD_URL}/mcp"
}
}
}
Once registered, the three tools surface natively in the agent — no prompt instructions needed, the LLM picks them up via their schema descriptions.
MCP client compatibility
OAuth shipped in v2.3.0 — enable it via OAUTH_JWT_SECRET (see above) and
Claude Desktop / claude.ai connect natively. The reverse-proxy workaround
below is only needed on instances that keep OAuth disabled.
Claude Desktop limitation (without OAuth)
The Claude Desktop "Add custom connector" UI accepts URL + OAuth
Client ID/Secret but no custom-header field. Additionally,
claude_desktop_config.json entries with "type": "http" are silently
rewritten to {} after Desktop launches (current Desktop only honors
stdio servers in that file).
If OAuth is not enabled on your instance, the practical workaround for Claude Desktop users is a reverse proxy that accepts the auth token as either a bearer header (for CLI) or as a URL path prefix (for Desktop, which has no header field).
Caddy workaround for Claude Desktop
Contributed by @WinFuture23:
@bearer header Authorization "Bearer {$AUTH_TOKEN}"
handle @bearer { reverse_proxy pullmd:3000 }
@token_path path /{$AUTH_TOKEN}/* /{$AUTH_TOKEN}
handle @token_path {
uri strip_prefix /{$AUTH_TOKEN}
reverse_proxy pullmd:3000
}
Then in Claude Desktop's connector dialog, use the URL with the token
path prefix: https://your-instance.com/<TOKEN>/mcp. CLI clients keep
using the Authorization header as normal.
This is a stopgap pattern; enabling the built-in OAuth removes the need for it.
API
/api parameters
Both query and max_tokens are also available on the MCP read_url tool.
Response headers
X-Source—reddit·cloudflare·readability·readability-fallback·trafilatura·playwright·recipe-content·coverage-guard·markitdown·youtube·image-caption·audio-transcript·pdf-ocrX-Quality—0.0–1.0extraction confidenceX-Share-Id— the 32-hex permalink idX-Suggested-Filename— a download filename for this conversion, e.g.YT-some-talk-dQw4w9WgXcQ.md. YouTube gets title plus video id, images/audio/documents keep the original file name from the URL, everything else uses the title;PULLMD_FILENAME_DATE_PREFIXprepends a date. Also on/s/:id,POST /api/htmlandPOST /api/file; on/api/streamthe same value arrives assuggestedFilenamein theresultevent, since SSE has no headers to carry it.X-Transcript-Status— YouTube only:ok·none·blocked·error.blocked(YouTube rate-limited the transcript fetch, HTTP 429) anderrorare transient and not cached — retry later;nonemeans the video genuinely has no transcriptX-Extracted—true/false. Present only whenqueryis active.X-Extract-Confidence—high·medium·low. Present wheneverX-Extractedis present, except when the page was already small enough that extraction was skipped.X-Extract-Sections— number of sections/regions included in the returned markdown.X-Extract-Original-Tokens/X-Extract-Returned-Tokens— estimated token counts (chars / 4) of the full page and of the returned markdown.
Bot-protection challenges
PullMD does not solve bot-protection interstitials, and the Playwright sidecar
is rejected by them as well. When the upstream answers with a Cloudflare
managed challenge (cf-mitigated: challenge) or a Bunny Shield proof-of-work
page (cdn-challenge: true, or the same page served with HTTP 200), /api
answers 502 with a machine-readable body instead of a bare 403 or the
interstitial rendered as Markdown:
{"error":"Site is behind a bot-protection challenge (Cloudflare); PullMD does not solve these. Workaround: save the page from your browser and upload the HTML via POST /api/html.","code":"bot_challenge","vendor":"Cloudflare"}
Nothing is cached for such a response. The workaround is the one in the body:
save the page from a real browser and convert it via POST /api/html or
drag-and-drop in the PWA.
Query-scoped extraction
When you have a specific question about a long page, you rarely need the whole
document. Pass the question as ?query=<text> on GET /api (or the query
param on the MCP read_url tool) to get back only the sections relevant to it
instead of the full Markdown body - typically 70-95% fewer tokens on long
pages:
GET /api?url=https://example.com/long-article&query=how+does+caching+work
It's a BM25 ranking over the page's heading-based sections (falling back to paragraph-level blocks for pages with fewer than two headings) — no network calls, no LLM involved.
- Budget —
max_tokenscaps the returned size (default600, range64–20000, estimated aschars / 4). - Whole-page fallback — the full page is returned unchanged when either
the page is already small enough that extraction wouldn't help (estimated
tokens ≤
max(800, 1.25 × max_tokens)), or nothing in the page scores against the query's terms. In both casesextractedisfalse; the no-match case reportsconfidence: low, the small-page case omitsconfidence. - Atomic blocks — code fences, tables, lists, and blockquotes are never split; if any part of one is relevant, the whole block is kept. Non-contiguous output regions are joined by an elision marker (``).
?frontmatter=trueaddsextracted,extract_confidence,sections_selected,original_tokens,returned_tokensto the YAML block.format=jsonadds a top-levelextractobject with the same data (extracted,confidence,sectionsSelected,originalTokens,returnedTokens).- Extraction runs on the already-cached full page — the cache always stores
the complete Markdown, so several different
queryvalues against the same URL cost one fetch (within the normal 1-hour cache TTL), not one per query. - Omitting
query(or passing an empty/whitespace-only string) leaves/apibehavior completely unchanged — no extra headers, no extra JSON or frontmatter fields. - MCP
read_urlhas no response headers; when nothing matches, it prepends an in-band comment (``) to the returned markdown instead.
SSRF protection
A URL-fetching service will happily be pointed at its own network if you let
it. Since v3.3.0 PullMD does not: before any fetch, the target host is
resolved and the resulting addresses are checked. Anything in a private,
loopback, link-local, CGNAT or cloud-metadata range is refused with 403 —
including 169.254.169.254 (AWS/GCP/Azure) and 100.100.100.200 (Alibaba).
GET /apiand the MCPread_urltool check the URL up front and answer403before doing any work.GET /api/streamis covered by the same guard one layer down, in the Reddit / web / Playwright fetch paths, and surfaces the refusal as an SSEerrorevent.- Every redirect hop is re-checked before it is followed, so a public URL that redirects inward is blocked at the hop, not after it.
- IPv6 is covered too, including blocked v4 ranges reached through transition addressing (IPv4-mapped, NAT64, 6to4, IPv4-compatible).
- The rejection happens before the cache is touched — a blocked URL never produces a cache row or a share id.
- The Playwright sidecar enforces the same policy inside the browser
(since 3.15.2). Chromium's own navigations never pass through Node, so the
sidecar checks every request it makes: redirects,
<meta refresh>,locationchanges, iframes, subresources, WebSockets. A page that sends the browser to an internal host fails the render with403; the allowlist below travels with each render request.
To reach an internal host on purpose (an intranet wiki, a document server on the same network), allowlist it:
# CIDRs, exact hostnames, or both
PULLMD_ALLOWED_HOSTS=10.0.5.0/24,wiki.internal
Empty (the default) means every internal target stays blocked.
Known residual: the guard checks resolved IPs at request time but does not pin the outbound socket to the address it checked, so a DNS-rebinding attack (where the name resolves differently between check and connect) is not fully closed. When PullMD runs behind an outbound HTTP proxy, the proxy resolves names itself — its egress filtering, not this guard, is the authoritative layer for traffic it forwards.
Reported as #41, shipped in v3.3.0.
Cache & TTLs
/api?url=…re-fetches from source if the cache row is older than 1 hour./s/:iddoes the same on-demand refresh, so share links double as live endpoints.- Cache rows are pruned
PULLMD_CACHE_RETENTION_DAYSdays (default 90;0keeps them forever) after the last write./s/:idhits keep the row alive (since they trigger refresh + write); read-only access does not extend the TTL. - If the source is unreachable on refresh, the last good snapshot is served — share links keep working even when the original URL dies.
Monitoring
GET /api/status reports whether the extraction sidecars are reachable:
{
"ok": false,
"version": "3.9.0",
"services": {
"trafilatura": { "status": "ok", "latencyMs": 3 },
"playwright": { "status": "down", "error": "unreachable" },
"markitdown": { "status": "not-configured" }
}
}
statusisok,down, ornot-configured. A sidecar you never configured is not a failure — self-hosting without Playwright or markitdown is a supported setup.erroris one ofunreachable,timeout,unhealthy(answered200but reports itself broken),http <code>, ormisconfigured. Deliberately a closed vocabulary: the endpoint is public and must not echo internal hostnames.- HTTP
200when every configured sidecar answers,503when at least one is down, so a plain HTTP check catches it without keyword matching. - Results are cached for 5 seconds and concurrent callers share one round of probes, so polling it cannot amplify into internal traffic.
Why it exists: a broken sidecar does not take PullMD down, it silently degrades extraction quality — pages that need a browser render come back near-empty while everything still answers 200. Without an explicit check that can go unnoticed for a long time.
Uptime Kuma: monitor type HTTP(s), URL https://<your-host>/api/status, accepted status codes 200-299. On an instance with authentication enabled the endpoint stays reachable without credentials.
Document conversion
When the markitdown sidecar is running (set MARKITDOWN_URL=http://markitdown:8003/convert), PullMD can convert document files to Markdown in addition to web pages.
Supported formats: PDF, DOCX, PPTX, XLSX/XLS, EPUB, ZIP (contents listed), CSV, JSON, XML.
Two ways to convert documents:
-
By URL — pass a direct document link to the regular API:
GET /api?url=https://example.com/report.pdfPullMD detects the document content type and routes it through the sidecar automatically.
-
By upload — POST the raw file bytes as the request body (max 25 MB):
POST /api/file Content-Type: application/pdf # the file's MIME type X-Filename: report.pdf # URI-encoded; or use ?filename= # body: raw file bytesThe PWA supports both drag-and-drop and a file picker (desktop and mobile) for the same path.
If MARKITDOWN_URL is unset, document conversion is unavailable: a document URL (GET /api?url=…) and POST /api/file both return 502. Regular HTML pages are unaffected, and the PWA hides the document-upload affordance (it reads the markitdown flag from /api/config).
The default docker-compose.yml includes the markitdown sidecar with MARKITDOWN_URL pre-wired. To opt out, remove the markitdown service block and unset the env var.
Media tier (image captions + audio transcription)
Image captioning and audio transcription run inside pullmd itself - the markitdown sidecar is not required for media features (it is only needed for document conversion). By default pullmd processes images with EXIF metadata only and audio files with track metadata only - no model calls, no external traffic. Set the relevant PULLMD_VISION_* or PULLMD_STT_* credentials to unlock richer extraction:
- Images - a vision-capable model generates a text caption for each image, embedded in the Markdown output. Enabled when
PULLMD_VISION_API_KEY(or the shared fallbackPULLMD_LLM_API_KEY) is set. - Audio - a speech-to-text model transcribes the audio; the transcript replaces the bare metadata block. Enabled when
PULLMD_STT_API_KEY(or the shared fallbackPULLMD_LLM_API_KEY) is set.
Both modalities are independently configurable via PULLMD_* env vars on the pullmd service:
Per-modality vars override the shared LLM_* fallback when set. The fallback deliberately stops at the key and the base URL: a single model name cannot serve captioning, transcription and OCR, so PULLMD_LLM_MODEL does not exist. Set PULLMD_VISION_MODEL / PULLMD_STT_MODEL / PULLMD_PDF_OCR_MODEL instead - if a PULLMD_LLM_MODEL is set anyway, the server says so at startup and the modalities stay on their defaults. Any OpenAI-compatible endpoint works - point *_BASE_URL at a local server (e.g. Ollama, LM Studio) to keep everything on-host and avoid per-call cloud costs.
Note: cloud endpoints send image and audio content to a third-party API. Use a local model server if data-residency matters.
Provider calls are budgeted: image captioning, audio transcription and PDF OCR share PULLMD_LLM_RATE_LIMIT requests per hour (default 30, 0 = off), counted per signed-in user or, without sign-in, per client address. Over budget the request gets what an instance without the provider returns (metadata only, or the markitdown PDF conversion), and that result is not cached. A tier without a configured provider never consumes budget.
YouTube transcripts
When MARKITDOWN_YOUTUBE=true is set on the pullmd service, YouTube video URLs are routed through the markitdown sidecar instead of the regular web-extraction pipeline. The sidecar fetches the video's title, description, and auto-generated or community transcript, and returns them as clean Markdown. No API key is required.
GET /api?url=https://www.youtube.com/watch?v=dQw4w9WgXcQ
The transcript format is configurable via environment defaults on the sidecar, and can also be overridden per request:
Both params are also available on the MCP read_url tool.
Language preference: set MARKITDOWN_YT_LANGS=de,en on the sidecar to prefer German transcripts, falling back to English if no German track exists. If none of the preferred languages are available, the sidecar falls back to the first available track.
Rate-limit caveat: YouTube's transcript API aggressively rate-limits requests from datacenter IP ranges. If you run PullMD on a cloud VPS and see frequent failures, set MARKITDOWN_YT_PROXY on the sidecar to route requests through a residential or ISP proxy.
Note: YouTube extraction requires the markitdown sidecar (
MARKITDOWN_URL) to be configured. IfMARKITDOWN_URLis unset, YouTube URLs are processed by the regular web pipeline (HTML extraction via Readability/Trafilatura), which does not include a transcript.
High-quality PDF (OCR)
By default, PullMD converts PDFs through the free markitdown path, which works well for text-heavy files but loses complex tables. For table-grade output, enable the opt-in OCR tier:
- Set
PULLMD_PDF_OCR_API_KEYto your OCR provider key (reference provider: Mistral OCR, ~$0.002/page). - Request
?pdf=ocron any PDF URL (also on/api/streamandPOST /api/fileuploads), or setfetch.pdf: ocras a recipe default.
PULLMD_PDF_OCR_API_KEY (and optionally PULLMD_PDF_OCR_BASE_URL) must be set explicitly - unlike vision and STT, PDF-OCR does NOT fall back to the shared PULLMD_LLM_* key, because the OCR endpoint is typically a different provider (Mistral OCR vs. a chat LLM).
On success, source becomes pdf-ocr and the frontmatter gains a pdf_pages field with the page count. If the OCR call fails or no key is configured, pullmd falls back to the standard markitdown conversion automatically - no error is returned to the caller.
Note: the OCR provider charges per page. Check your provider's pricing before enabling this on high-volume instances. OCR requests count against the hourly
PULLMD_LLM_RATE_LIMITbudget described under Media tier.
Self-hosted / heavy local engines (advanced)
PULLMD_PDF_OCR_BASE_URL can point at any compatible endpoint. Operators who want fully local OCR can run an engine such as Docling themselves and set _BASE_URL to its address. Be aware that such engines are heavy - a multi-GB container image, and model cold-start times of tens of seconds on first request are typical. This is deliberate: pullmd does not bundle any heavy OCR engine. The reference Mistral OCR path uses a lightweight cloud API call; self-hosting a heavy local engine is an advanced operator choice.
Source-specific frontmatter fields
When you request ?frontmatter=true, Reddit, media, and YouTube results include extra fields in the YAML block - post metadata for Reddit, usage fields that let agents compute cost or log telemetry for media. These fields appear only in the frontmatter and are never inserted into the body — without ?frontmatter=true the body is pure content.
A few things worth noting:
- PullMD reports no computed cost. Token counts and the model name are what pullmd receives from the upstream API; multiply by the model's per-token rate on your end.
- Image captioning and audio transcription are direct API calls made by pullmd.
- Frontmatter fields for ordinary web pages (
title,url,source,quality, …) are always present when?frontmatter=true; the LLM fields above are additive.
Site recipes
Site recipes are small declarative JSON entries that adjust extraction for specific hosts — force a browser render, strip site-specific noise, unwrap wrappers that confuse the reader, rebuild CSS-grid comparison tables as real tables, or promote structured data (JSON-LD / CSS selectors) into the frontmatter. The built-ins ship in site-recipes.default.json; self-hosters can add their own via data/site-recipes.json or the PULLMD_SITE_RECIPES env var (see .env.example). GET /api/recipes/status reports what loaded.
To write or contribute a recipe, see the site-recipe contributor guide.
Architecture
server.js— Express app factory (createApp) with dependency injection for tests. Exposes/apiand/api/stream(SSE).lib/reddit.js— Reddit URL normalization, redirect resolution, post + comment extraction.lib/web.js— Orchestrator: Cloudflare-Markdown short-circuit, then static Readability + Trafilatura withpickBest, then optional Playwright re-render + re-extract on body-soup / low-quality output.lib/render-decision.js— Predicate that decides when to fall back to Playwright (readability-fellback + thin, body-soup signature, or quality < 0.5; plusforce/skipoverrides).lib/playwright-client.js— HTTP client for the Playwright sidecar withAbortSignalpropagation for SSE-disconnect cancellation.lib/scoring.js— Quality scoring used to pick between extractors and as a render-trigger heuristic.lib/cache.js— SQLite cache (better-sqlite3) with configurable retention (default 90 days) and 32-hex (128-bit) share ids.lib/mcp.js— Stateless MCP server registering the three tools.lib/distrib.js— Public-URL substitution in/helpand/pullmd.zip.trafilatura-sidecar/— Python sidecar (FastAPI) wrapping Trafilatura.playwright-sidecar/— Python sidecar (FastAPI + Playwright + Chromium) for JS-rendered pages.markitdown-sidecar/— Python sidecar (FastAPI) wrapping MarkItDown for document conversion (PDF, Office, EPUB, …) plus the keyless/youtubetranscript endpoint.lib/markitdown-client.js— HTTP client for the markitdown sidecar (documents + YouTube).lib/youtube.js— YouTube URL detection and normalization.lib/llm/— provider resolver + adapters for image captioning (vision), audio transcription (STT), and PDF OCR against OpenAI-compatible endpoints.lib/frontmatter.js— YAML frontmatter builder with thePULLMD_FRONTMATTER_FIELDSallowlist.lib/coverage-guard.js— detects extractions that kept only a sliver of the page and recovers the container holding the body.lib/query-extract.js— Opt-in query-scoped extraction (?query=): BM25 over heading-based sections of the converted markdown, returning only the relevant sections/blocks. Composeslib/markdown-sections.js(block/section parser) andlib/bm25.js(tokenizer + scorer).public/— PWA frontend (vanilla JS, dark/paper themes, service worker, EventSource client for/api/stream).skill/pullmd/— Claude Code skill source (templated with__PULLMD_URL__).
License
GNU AGPL v3 — Copyright © 2026 Aeterna Labs.
PullMD is free software: you can redistribute it and modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, version 3 or later. If you run a modified version as a network service, you must make your modifications available to its users.