Distill

distill

Turn any webpage into clean Markdown an AI agent can actually use.

distill is a local CLI (and MCP server) that takes a URL or HTML file and returns Markdown with:

• nav / ads / footers stripped
• tables, code blocks, and lists kept intact
• relative links resolved to absolute URLs
• title / author / date in a YAML frontmatter block

No cloud API. No per-page cost. Same input → same Markdown every time.

distill https://en.wikipedia.org/wiki/Markdown
# → clean Markdown on stdout

What it is (in one minute)

Most “reader mode” tools were built for blog posts. Agents usually need API docs, pricing tables, forum threads, and reference pages. Distill is built for that: structural fidelity first, then speed and privacy.

Ways to use it

• CLI — distill <url> in a shell or script
• MCP server — tools distill_url, distill_urls, distill_html for local agents (Claude Code, etc.)

Published as distill-md on crates.io / npm (the name distill is taken). The installed commands are still distill and distill-mcp.

Install

# Agents (MCP) — no install, always latest
npx -y -p distill-md distill-mcp

# Shell installer (macOS / Linux) — prebuilt binary
curl -LsSf https://github.com/gokulnair2001/distill/releases/latest/download/distill-md-installer.sh | sh

# Rust toolchain
cargo binstall distill-md      # prebuilt
cargo install distill-md       # from source

# From this repo
cargo build --release --features mcp
# → ./target/release/distill  and  ./target/release/distill-mcp

Prebuilt binaries for macOS (arm64/x64), Linux (arm64/x64), and Windows (x64) ship with every GitHub Release. See RELEASING.md for how releases are cut.

Usage

# URL
distill https://en.wikipedia.org/wiki/Markdown

# Local HTML file
distill page.html

# Stdin
curl -s https://example.com | distill -

# Several inputs → one .md file each (-o is a directory)
distill https://a.com https://b.com page.html -o out/
# → out/a.com.md, out/b.com.md, out/page.md
# A failing input is reported and skipped; the rest still write.

# Common flags
distill <url> \
  --render auto \      # JS rendering: never | auto | always (default: auto)
  --no-frontmatter \   # omit YAML metadata
  --no-links \         # keep link text only
  --no-images \        # drop images
  --raw \              # skip main-content extraction (convert whole page)
  --base <url> \       # base for resolving relative links
  -o out.md            # write to a file (or a directory, for multiple inputs)

# Agent-ready JSON (opt-in): sectioned Markdown + RAG chunks + schema
distill <url> --agent-ready        # long form
distill <url> --ars                # alias
distill <url> -A -o page.json      # short form

Agent-ready output (--agent-ready / --ars / -A)

Default output stays plain Markdown. With --agent-ready, Distill emits JSON with three layers agents can use without re-parsing a flat blob:

Deterministic and local — no LLM. Same extract/convert pipeline; only the final shape changes.

JavaScript rendering

Some sites (SPAs) only fill in content after JavaScript runs. Over plain HTTP you get an empty shell.

Distill is static-first:

Needs an installed Chrome / Chromium / Brave / Edge. Set DISTILL_CHROME to point at a specific binary. If none is found, distill falls back to static HTML.

Most docs sites (VitePress, Next.js SSG, etc.) already ship full HTML — rendering them changes nothing. The browser path only helps genuinely client-rendered pages.

How it works

1. fetch — Download the page with browser-like headers; follow redirects; decode gzip/brotli and character encodings.
2. render? — Optional headless Chrome, only when the static HTML looks empty or extraction finds almost no content.
3. metadata — Pull title / author / date / lang / canonical from <head>.
4. clean — Remove scripts, nav, footer, aside, ads, and other boilerplate (matched by tag and by class/id, with care not to delete real content like forum threads).
5. extract — Score text blocks (Readability-style) and pick the main content region; merge nearby siblings when the page splits content across wrappers.
6. convert — Walk the DOM and emit Markdown: real tables, fenced code with language hints, nested lists, blockquotes, absolute links and images.

Why it's fast

Most of the speed comes from avoiding work, not micro-optimizations.

Process-only median on cached HTML: ~16 ms/page (see Benchmarks). Network time dominates end-to-end; that number is the stable, network-free figure.

Key terms

Four reasons

1. Static-first — Docs and Wikipedia-style pages already have content in the HTML. Distill uses that and skips the browser. Chrome runs only when the body looks empty, SPA mounts (#root, #app, …) are empty, or extraction yields almost nothing (under 200 characters of main content).

2. Rust + lean deps — Native binary, small HTML parser (kuchikiki), blocking HTTP client. No LLM in the convert path. Release builds use LTO and strip.

3. Cheap JS when needed — Shells out to an installed Chrome with --dump-dom and a virtual-time budget. No heavy browser-automation library inside the binary. Rendered HTML then reuses the same clean → extract → convert path.

4. One deterministic pass — No model, no sampling, no “rewrite for quality.” One pipeline, same output every time.

In one line: skip the browser whenever possible; only pay for Chrome on real JS-heavy pages.

MCP server

Expose distill as tools any local agent can call (Model Context Protocol, over stdio):

Same knobs as the CLI (include_links, include_images, frontmatter, raw, base, agent_ready; URL tools also take render).

# Register with Claude Code (published package)
claude mcp add distill -- npx -y -p distill-md distill-mcp

# Or a local build
cargo build --release --features mcp
claude mcp add distill -- /absolute/path/to/target/release/distill-mcp

Or in mcp.json:

{
  "mcpServers": {
    "distill": {
      "command": "npx",
      "args": ["-y", "-p", "distill-md", "distill-mcp"]
    }
  }
}

SSRF guard

Agents can pass any URL into distill_url. By default, requests to non-public addresses are refused — loopback, private ranges, link-local (including the 169.254.169.254 cloud metadata endpoint), and IPv6 equivalents — on the initial URL and every redirect / meta-refresh hop.

To allow a local or internal host:

DISTILL_ALLOW_PRIVATE_HOSTS=1 distill http://localhost:3000

Same env var for CLI and MCP.

Benchmarks

50-URL corpus (bench/corpus.jsonl): articles, docs, product pages, forums, table-heavy pages, and SPAs. Measured 2026-08-09 against Jina Reader, Firecrawl, trafilatura, readability-lxml, and markitdown. Reproduce with bench/.

Structural preservation (micro-average kept ÷ source; tables and code blocks are the high-signal features — link/heading ratios often just mean “kept more nav junk”):

* Jina’s coverage is real, but its link/heading/image keep rates are inflated by boilerplate. On tables and code blocks it is the weakest of the group.

Known limitation: a few heavy client-rendered docs sites do not finish populating within the current headless-Chrome time budget. Bug history and methodology notes live in bench/README.md.

Roadmap

• JS rendering — static-first, Chrome only when needed
• Benchmark harness vs cloud and local alternatives (bench/)
• MCP server (distill_url / distill_urls / distill_html, SSRF-guarded)
• Agent-ready structure — opt-in --agent-ready / --ars (sectioned MD + RAG chunks + schema)
• Page-type awareness — distinct strategies for docs / listings / tables
• More structural fidelity — colspan/rowspan, <picture> / srcset, …
• Schema-guided extraction — caller-supplied JSON schema (LLM or rules)

Development

cargo test                     # unit + integration tests
cargo test --features mcp      # include the MCP server
cargo build --release

For LLM-facing checks of Distill’s Markdown / --agent-ready JSON on a fixed 12-URL corpus, see bench/llm_suite/ (python3 bench/llm_suite/generate.py → gitignored out/).

License

GNU General Public License v3.0 or later — see LICENSE.