Codearia Sieve

What it does

An agent that needs a web page fetches the whole thing: navigation, cookie banner, footer, ad slots, a megabyte of framework markup. Then a model paid per token digs through the pile for one paragraph.

codearia-sieve does the digging before the model sees anything — and returns the page as state, not prose:

Dates become dates

"Published September 15, 2026" → "2026-09-15". Read from JSON-LD, meta tags and <time> first; from a byline only when the markup is silent, and never guessed.

Numbers become facts

"$42 per billion tokens" → { value: 42, unit: "USD_per_billion" }. Works in nine languages; the decimal comma follows the page's language. A number without a unit is not a fact. A year is never a fact.

Text becomes chunks that fit

Each chunk knows its size in tokens and characters, the blocks it was built from, and the #anchor on the page where a decision can be checked.

Everything else — menus, footers, banners, tag rows, "read more" — is removed, and with trace: true you get the list of what was removed and why.

People who build agents and have seen the bill. Every fetched page costs tens of thousands of tokens before the agent has read a word of it. Median page in the sample: 53 718 tokens in, 1 106 out.

People who run cheap decision models. Classifiers, rankers, System One models like Jev that judge instead of write. They are nearly free and very fast, and they have hard edges: they cannot count, they read dates as text, and their accuracy drops as irrelevant material fills the context. Every "page to markdown" tool prepares input for a reader. This one prepares input for a judge.

People who need answers they can check. A verdict from scraped text is unprovable unless each piece points back to its source. Here every fact names its block and every chunk carries an anchor.

Eight steps of ordinary code. No model runs unless you plug one in. The same HTML gives the same JSON, byte for byte.

1. Fetch — robots.txt first; a refusal is reported, not bypassed. Plain HTTP, honest user agent.
2. Parse — HTML into a DOM with linkedom. No browser.
3. Dates and ids from the untouched tree — cleaning strips <head>, bylines and attributes, so both are read before it runs.
4. Clean — Defuddle removes chrome; Readability takes over if it comes back empty. Inline elements get a space first, so <span>20 Sept</span><span>10 min</span> never becomes 202610 min.
5. Blocks — headings, paragraphs, lists, tables, code, quotes, in order, with positional ids. Old pages set with <br><br> become paragraphs too; table rows keep their column headers.
6. Facts and anchors — ids restored; numbers become facts only beside a unit; ranges keep both ends.
7. Chunk — greedy, in order, under two budgets at once: 20 000 tokens and 50 000 characters by default.
8. Assemble — state, markdown, usage, warnings, and the trace on request.

const r = await sieve({ kind: 'url', url: 'https://docs.typesafe.ai/models' });

r.state.title         // "Models"
r.state.facts[0]      // { value: 42, unit: "USD_per_billion", label: "price_btok_mtok",
                      //   context: "Price (per Btok / per Mtok) | jev-1.13.0: $42 / $0.042", from: "b3" }
r.state.facts[1]      // { value: 0.042, unit: "USD_per_million", … }   — paired by position
r.state.chunks[0]     // { id: "c1", tokens: 1210, chars: 5357, anchor: "Current models",
                      //   headings: ["Current models", "Pricing", …], blocks: ["b1", …, "b36"], text: "…" }
r.usage               // { rawTokens: 127413, stateTokens: 1211,
                      //   visibleChars: 4939, stateChars: 5357, chunks: 1, ms: 1503 }
r.warnings            // []
r.markdown            // the same article, for a human or a generative model

Expected outcomes never throw. They come back as warnings, each named:

You say what you want in plain words. The agent finds the pages, calls sieve_page for each, hands the state to a decision model with a typed question, and writes up the result. Sieve prepares. The judge judges. The agent writes.

{ "mcpServers": { "sieve": { "command": "npx", "args": ["-y", "codearia-sieve"] } } }

Listed in the official MCP Registry as io.github.AntonG87/codearia-sieve; clients that read the registry can install it by name.

sieve_page — url or html

Returns typed structuredContent with an output schema: source, state, usage, warnings. In the default summary mode chunks carry sizes, anchors and their headings but no text — the agent sees the outline of what exists without paying for it. mode: "full" and mode: "markdown" when you want everything.

sieve_chunk — url, id

The text of one chunk from the last result for that URL, no refetch. Overview first, then only what is needed — the tool applies its own idea to itself.

Pairs with jev-mcp: chunks are sized to fit its fields, so state goes straight into a typed question.

Checked by a judge

The claim is that a decision model gets better input from Sieve than from raw text. So the output was handed to one. examples/jev.ts drives both MCP servers with the official client — codearia-sieve prepares six pages (API docs, a release note, two Wikipedia articles in two languages, two pricing pages), Jev judges them through jev-mcp. Same run, 21 September 2026:

The first pass of this test did its job the other way round: Jev sent three facts to review and contradicted one. All four traced to Sieve — a table row labelled by its column header instead of its row header, two rates in one header left unpaired, and a Russian bibliographic "256 с." read as seconds. Fixed, tested, rerun: 11 of 11. A judge that can tell you when your parser is wrong is the point of the whole pairing.

TYPESAFE_API_KEY=… node --experimental-strip-types examples/jev.ts

Use it as a library

import { sieve } from 'codearia-sieve';

await sieve({ kind: 'url', url });                    // fetch it
await sieve({ kind: 'html', html, url });             // already have it; url only for anchors

await sieve(input, {
  budget:    { maxTokens: 8000, maxChars: 30000 },    // chunk limits
  trace:     true,                                    // everything discarded, and why
  tokenizer: myTokenizer,                             // o200k by default; swap for your model's
  fetcher:   myFetcher,                               // your transport, or a file reader in tests
  now:       () => fixedDate,                         // injected clock: identical output on identical input
  selector:  mySelector, task: 'is this about pricing?', // relevance judge; nothing runs without one
});

parseDate, findDates and the vendor limits (JEV, JEV_MCP, DEFAULT_BUDGET) are exported too.

Where it stops

• Front pages, listings and product pages have no article to find. You get the headlines and a thin-content warning, not a fake win.
• Articles rendered by JavaScript come back as empty-without-js when the container is empty. A site that ships a teaser and streams the rest cannot be told apart without a browser; you get the teaser.
• Text-heavy pages save less. A whole novel saves 22 %, an RFC 84 %: there is no wrapping to remove and the text is kept in full. That is the tool working.
• A pricing grid is not a table. A fact knows the block it came from, not the plan column it sits under; Sieve does not guess the pairing. Send the chunk — a pricing page is about a thousand tokens after cleaning — and let the judge read it: examples/pricing-watch.ts.
• Tokens are counted with o200k as an approximation. Pages over a megabyte get a sampled count and usage.rawTokensEstimated: true.

Develop

npm install
npm test                    # 71 tests, offline, a few seconds
npm run demo -- <url>       # the token bill for one page
npm run bench               # the 20-page benchmark set
npm run analytics           # the 56-page random sample: rows, CSV, summary

Design notes — vision and architecture — are in docs/.