NicheDB

NicheDB

Sources in, feeds out. An open platform for databases that only ever grow: game releases, package registries, government filings, and any other public data with a big seed and daily growth. One Postgres, one Redis, and every row reachable by web, RSS, JSON, API, CLI and MCP.

The first deployment is nichedb.dev. Run your own on anything that has Postgres and Redis.

The model

Adapters are one file each in packages/adapters/src. One hundred and sixty-one ship today across forty-six collections:

The way out (OpenSaaS)

nichedb serves its own OpenSaaS descriptor at /.well-known/opensaas.json: the plans as the checkout sells them (prepaid terms, none renews itself) and the actions that are real here, each as the page a person opens and the endpoint an agent calls with an OpenAccess bearer: subscribe (POST /api/v1/billing/subscribe, answers pending with the checkout page), cancel (POST /api/v1/billing/cancel, one click, today, nothing refunded), unsubscribe (a page: /settings, where the follows that send mail are turned off), export (POST /api/v1/account/export, one JSON document at once), delete (POST /api/v1/account/delete, confirmed by a link to the account email, immediate, nothing kept). The same doors as pages: /account/billing, /account/export, /account/delete. The opensaas adapter reads other services' descriptors into the saas collection, one row per plan with the service's own steps and confirm for the way in and the way out shown beside each other.

People (/c/profiles)

One entry per person, assembled from every app that serves their OpenProfile.md. The openprofiles adapter pages each app's listing (/api/openprofiles?since=…), fetches every document from the host that listed it, and merges documents that share an identity key (the Web URL, any Accounts URL, Email, DID) into one profile; a shared name alone never merges. The page is /c/profiles/<slug>-<id> (the id resolves, the name is cosmetic, a wrong name 301s) or /c/profiles/<handle> once the owner took one, and …/openprofile.md next to it is the file, text/markdown, CORS open, with <link rel="openprofile"> on the page.

A person claims their entry with This is me (proven by the email the profile lists, or a link back to the page from their site or show; admins may claim for an email) and edits it from the page, PUT /api/v1/profiles/<ref> (JSON overlay or a whole text/markdown file), nichedb profile edit, or the update_profile MCP tool, with a session, an API key, or an OpenAccess token carrying openprofile:edit. What the owner writes is an overlay that wins over every source and survives every re-read; a section written as none is removed. The Broadcast and Guest sections (OpenBroadcast, OpenGuest) are what the podcasters and guests feeds cut on.

The page shows the three identity fields OpenProfile 0.4 makes defaults, read from the Markdown: Emoji (one grapheme or an OpenEmoji :shortcode:) beside the name, Pronouns as written beside the handle (absent is unstated, never inferred), and Web (with Website, Homepage and Site read as the same key). The API and MCP answers carry them as emoji, pronouns and web; owners set them in the editor, as emoji/pronouns/web in a PUT or update_profile, or with nichedb profile edit <ref> --emoji … --pronouns … --web ….

Professional facets come only from what the person (or a public page an adapter already reads) published: a Title: (or Job Title:, Role:, Position:) identity line and a Company: (or Employer:) line, read case-insensitively; Company: Acme (https://acme.com), [Acme](https://acme.com), acme.com and a bare name all read, and Title: CEO at Acme names both. A seniority (c-suite, vp, director, manager, senior, entry, or none) is derived from keywords in the title. The API and MCP answers carry title, company ({name, domain}), seniority and facets; the collection items are tagged title:<slug>, company:<domain> (a slug of the name when there is no domain) and seniority:<level>, so ?tags= filters them like any other facet, and GET /api/v1/profiles (and search_profiles, nichedb profiles) take title, company, seniority and tags. GET /api/v1/profiles?linkedin=<url> (find_profile_by_linkedin, nichedb profiles --linkedin) answers the public profile a LinkedIn page belongs to. There is no lookup by email, on purpose: it would tell anyone holding an address whether its owner is listed. | directory | outreachgraph | no | | sites | opensite | no | | algorithms | nist-dads (NIST's Dictionary of Algorithms and Data Structures, every entry with its definition and relations; public domain), wikidata-algorithms (every algorithm and data structure on Wikidata, with who devised it, when, and its complexity; CC0), pwc-archive (the Papers With Code methods and datasets catalogues as archived on Hugging Face, call-centre spam dropped; CC BY-SA 4.0), rosetta-code (every programming task with the languages it is solved in; GFDL), thealgorithms (The Algorithms' implementations by language from each repo's DIRECTORY.md; MIT). docs/algorithms.md | no | | models | modelsdev (every AI model and what it costs, from models.dev; MIT). One row per provider offering rather than per model, because the same model reaches you from many of the 223 providers at as many prices: price per million tokens in, out and cached, context and output limits, modalities and capability flags, 8,179 of them. Plus the lab catalogue, one entry per model as its maker published it with a page of its own, and the providers themselves. openmodel (a provider's own file at /.well-known/openmodel.json, believed only when served from the origin it describes; the same vocabulary as the models.dev rows beside it, so one feed catches both). docs/models.md | no | | mcp | mcp-registry (the official registry: one row per server at its newest version, with packages, remotes and status), docker-mcp (Docker's catalogue: pull counts, categories, licences and the tools each server exposes), smithery (how often each server is actually called through Smithery's proxy), awesome-mcp (the punkpeye, wong2 and appcypher lists plus the protocol repo's reference servers), aitmpl-mcps (ready-made configs), github-mcp-topics (repositories tagged for MCP, newest push first), installmap-yc-mcp (Y Combinator companies that run a server, from Installmap's CC BY crawl, as OpenMCP records with the endpoint probed, plus each company's accounts and founders; the companies and founders also go to profiles as OpenProfiles through installmap-yc-people), glama-mcp (Glama's directory, last, so it adds only the servers none of the others carry, and only those Glama has quality-scored at 3.5 or more whose repository belongs to the lister, which leaves out the clone spam; licensed data, so every Glama row shows "MCP data from Glama" and a link to its Glama listing). Deduplicated on URL: the first source to carry a server keeps it. docs/mcp-and-workflows.md | keyless (GITHUB_TOKEN raises the search and enrichment rate limits) except glama-mcp, which needs GLAMA_API_KEY and stays paused without it | | workflows | reddit-workflows (the r/ClaudeWorkflows library, one post per workflow with its rating, freshness, level and categories parsed out of the post), awesome-claude-code (read from the CSV the list is generated from, so entries its maintainer has marked dead are skipped), aitmpl-components (settings bundles, agent loops and behaviour mods), github-agent-topics (repositories tagged for Claude Code, skills and subagents). Served at /workflows as well as /c/workflows. docs/mcp-and-workflows.md | keyless (GITHUB_TOKEN optional, as above) | | skills | aitmpl-skills (890 skills with install counts and the command that installs each), awesome-claude-code-skills (the hand-picked ones). /skills | no | | agents | aitmpl-agents (420 subagents by category), awesome-agents (the community lists that name each subagent rather than the bundle it ships in; VoltAgent seeded). /agents | no | | commands | aitmpl-commands (290 slash commands with install counts). /commands | no | | plugins | plugin-marketplaces (every plugin declared in the .claude-plugin/marketplace.json of the repos you name, the official one included). /plugins | no | | hooks | aitmpl-hooks (what runs before and after a tool call, a prompt or a session). /hooks | no |

Moderators

A niche can have moderators: people who review what is suggested for its collection without operating it. Anyone signed in applies at /<niche>/moderate (three questions); the application is a claim marked with the role, listed on /admin/knowledge with every other claim, and only an admin decides it. An approved moderator is a niche member with a share cap of zero and no score row: moderating earns nothing and is not a rung on the ladder. More than one person may moderate a niche. Moderators see the suggestion queue at /admin/submissions filtered to the collections their niches feed, approve into that collection only, and get the Queue link in the nav.

Sites (/c/sites)

One record per page, read the way OpenSite says: the title, the line, the picture and the kind a card is drawn from, the canonical address it is keyed by, the author, the feeds, and the og:, twitter: and JSON-LD tags verbatim. The opensite adapter walks each house site from its front page and sitemap, a hundred pages a run, honouring robots.txt and a site's own /.well-known/opensite.json. Any other address is read the moment somebody pastes it at /c/sites/add, which then shows every tag and the card as X, Slack, iMessage, Discord, LinkedIn and WhatsApp would draw it; the same read is POST /api/v1/sites { "url" }, and GET /api/v1/sites?url= answers the kept record or reads the page when it is older than an hour. A whole list, up to ten thousand addresses pasted or uploaded on the same page (or POST /api/v1/sites/bulk with JSON urls or a text body), becomes a source of the submitter's: the worker walks it 250 pages a run, a minute apart, and /s/<slug> shows the progress; a signed-in account is needed, since a list makes the deployment fetch on a schedule. The page is /c/sites/<host>/<path>, query string and all, because a page's identity can be in its query.

Lyrics (/lyrics)

Lyrics for Profullstack's own bands (Þrøngva, Velocity Vibe, Meine Mutter), read and written by the nixamp radio servers: /lyrics, /lyrics/<artist>, /lyrics/<artist>/<title>, and GET/POST/DELETE /api/v1/lyrics. Our own words only (source is embedded, author or transcribed); nothing is imported from lyrics sites. Writing takes a key with the lyrics permission (scripts/lyrics-key.js). See docs/lyrics.md.

Top-level domains (/tlds)

Every label in IANA's root, read daily and diffed into a change log nobody else publishes, with each registrar's one-year price to register, renew, transfer and restore beside it: Porkbun, Dynadot, Cloudflare and OVHcloud today, and any registrar that serves an OpenTLD file. Faceted by type, renewal band, registrar, script, status and registry, sortable on every column, renewal first, with the renewal trap (a second year at twice the first or more) as a filter of its own. /tlds/check asks each registry over RDAP whether a name is taken, and never calls a 404 "available". The same rows are GET /api/v1/tlds (JSON or CSV), MCP (search_tlds, get_tld, tld_changes, check_domain) and the CLI (nichedb tlds, nichedb check). See docs/tlds.md.

House aggregators

Ten of the sites we run publish a public feed or API of their own, and each is read here through it, keyless, the way any other reader would. The endpoint is the one checked live on 2026-09-12 (c0upons on 2026-09-13); the slug is the seeded source.

saasrow's /api/v1/listings is per-account and needs a key, so only the public products directory is read. A submission aiornot carries on more than one feed is stored once and tagged with each feed. tsbb's cross-board /api/v1/latest does not say which forum a topic is in, which is why the walk is per forum.

Ruuster housing searches

ruuster-san-jose-homes reads the Real Estate Experts / Talar Davoudi saved search hourly and feeds /f/san-jose-homes. The search asks for San Jose houses that are Active or Coming Soon, with 2+ bedrooms, 1+ bathrooms, 750+ square feet, a 4,500+ square foot lot and a build year of 2000 or later. Ruuster calls the lot filter lotSizeAcresMin but accepts square feet there; property records return acres, and NicheDB keeps both units. Results and dimensions are upstream observations; conflicting MLS values are not silently corrected.

Create another ruuster source with its savedSearchUrl, pages (20 by default, ten records per page) and currency (USD or CAD). Tracking parameters are removed and repeated status filters are combined. Public detail requests add photos, property facts and MLS attribution. Syndicated copies are joined by MLS listing number and address. Hidden-address and deleted records are omitted. The worker's detail budget and deadline are respected, with a cursor for unfinished pages; completed walks start from page one on the next refresh.

Items retain the last status observed while matching the search. A property disappearing from a search is not evidence that it sold, and this source does not revisit every historical listing after it leaves the search.

Run bun run ingest ruuster-san-jose-homes against the configured database, or export without a database with bun scripts/scrape-ruuster.js > properties.json. The export also accepts a saved-search URL and an optional new output filename as positional arguments. Ruuster's terms restrict automated access without written consent, and its IDX notice limits reuse; deploying this adapter does not grant redistribution rights: https://realestateexperts.ruuster.com/terms-of-service.

MusicBrainz web-service requests share a process-wide queue with at least 1.1 seconds between requests. A 429 or 503 retries up to three times with 5, 15 and 45 second backoff, honoring longer Retry-After values, including HTTP dates. A zero retry hint cannot cause an immediate retry. This queue covers one worker process; deployments sharing an egress IP across multiple processes need coordination across those processes as well.

Enrichment

After ingest, every item is enriched by the enrichers that apply to it (packages/enrichers/src), and the results live on the item under enrichment.<name>:

A feed's enrichers list picks which of these it shows; absent means the collection's defaults. The feed builder exposes them as checkboxes. Items are enriched once, newest first with a fair share per collection (ENRICH_PER_RUN a tick, ENRICH_PER_COLLECTION of them from any one collection), and a missing image, summary or tags are filled from whatever the enrichers found while the source's own words always win.

Run it

git clone https://github.com/profullstack/niche-db && cd niche-db
bun install
docker compose up -d db redis          # or point at your own Postgres 16+ and Redis
cp .env.example .env                    # DATABASE_URL, SITE_URL, CONTACT_EMAIL at minimum
bun run build:client
bun run dev                             # web + worker in one process on :3000

Migrations apply themselves on boot. Every collection, its default sources and its feeds are seeded on first boot; sources whose adapter needs a credential the deployment lacks are created paused. The first account to sign in is an admin.

bun run ingest [slug ...] runs sources from a terminal without Redis. bun test runs the suite against an in-process Postgres (PGlite), so it needs no server.

Surfaces

• Web: a PWA. /sources manages sources (status, last run, next run, run now, pause, edit config). /feeds/new builds a feed with a live preview. /settings holds passkeys, API keys and notifications.
• RSS / JSON Feed: /f/<slug>.rss, /f/<slug>.json.
• API: /api/v1, documented at /docs/api. Reads need no key.
• CLI: curl -fsSL https://nichedb.dev/install.sh | sh (bun if present, else npm with Node 22+, no sudo), or npm i -g @profullstack/nichedb → nichedb. Documented at /docs/cli.
• MCP: /mcp (Streamable HTTP, stateless) or nichedb mcp over stdio. Documented at /docs/mcp.
• llms.txt: /llms.txt describes the whole deployment in one document.

Accounts and money

Magic link and passkey only; there is no password column. API keys are minted from settings and shown once.

Free pages and feeds carry one CrawlProof ad and a tracker (CRAWLPROOF_AD_SLOT, CRAWLPROOF_SITE_ID). Two tiers buy them away, both through CoinPay:

• Premium — $1 a day, $30 a month or $300 a year (/premium). No ads, no tracking, the members' Lounge, 1,000 credits a month to give awards with, a badge, six themes and five app icons, early access to new collections, 30,000 API requests an hour, unlimited feeds, own sources, and the metered vehicle lookups included. /premium carries a line-by-line comparison with Reddit Premium; GET /api/v1/premium is the same thing as JSON. See docs/premium.md.
• Pro — $120 a month (/pro). Everything Premium has, plus 120,000 API requests an hour and a crawl pass for the whole term (GET /api/v1/crawl-pass) so a member's own agents walk through the paywall.

Referral links give the new customer 20% off and the referrer 60% of the first payment.

Training crawlers (GPTBot, ClaudeBot, CCBot, meta-externalagent…) get a 402 with an x402 offer and buy a pass at /crawl: $1 a day for everything, and the more a buyer has paid here the less a day costs (CRAWL_LOYALTY, default 20% off after $10, 40% after $50, 60% after $100; every sale is a row in crawl_sales). A pass may switch ads or tracking off for its own requests with ?disable=ads,tracking. People, search engines and retrieval crawlers pass through untouched.

Knowledge Influencers

Every collection here is also a niche someone can operate. A Knowledge Influencer is a person who knows how an industry actually works and supervises the agents building software, data and promotion for it — no code required. They start at 20% of what the niche makes and climb to 80% as verified contribution accumulates.

/opportunities lists the niches looking for one. /<niche> is the niche's public page (plus skill.md and manifest.json for agents), /@<handle> is an operator's profile, /dashboard/niches is their own view, and /admin/knowledge is where claims and contributions are verified.

When an agent gets stuck on something only a person who has done the job can settle, it asks: the question lands on the operator's dashboard, the answer becomes niche knowledge and a scored contribution, and Chovy is told. Saying "not enough context" is a first-class answer that costs nothing, because a guess that gets verified is worse for the niche than an open question. See docs/agent-questions.md.

Score comes only from contributions somebody verified, and volume does not buy it: repeated submissions of a type pay less each time, anything claiming a customer or a payment needs an outside reference, and a duplicate books once. The ladder lives in contribution_tiers, so a deployment can tune it. What a niche earns is divided at the moment it settles, using the shares in force right then, so a tier that moves tomorrow never re-prices yesterday's sale (docs/revenue-ledger.md). See docs/knowledge-influencers.md and docs/revenue-share.md.

Stack

Bun, Hono (server-rendered JSX), Postgres via Bun's native driver, BullMQ on Redis, Biome. bun apps/worker/src/enrich-cli.js [n] runs enrichment from a terminal. Shared Profullstack modules: @profullstack/x402-gateway, @profullstack/emailer, @profullstack/coinpay, @profullstack/referrals, @profullstack/api-key-manager, @profullstack/autoblog (signed webhooks), @profullstack/favicon-generator.

Brand assets

brand/logo.svg is the editable SVG master. Run bun run build:brand to crush it into favicon.svg with SVGCrusher, then run the installed fav CLI to regenerate apps/web/public/icons. The build also updates /logo.svg, /favicon.svg, member palettes, and maskable app icons. SVG is preferred in the page and app manifest; PNG and ICO remain fallbacks. Preview everything in brand/preview.html.

SVGCrusher's initial CLI release is bundled in vendor/ so this command works from a clean checkout before an npm release. Its standalone source lives in profullstack/svgcrusher.

Configuration

Everything is read once from the environment in packages/config/src/index.js; .env.example lists every variable. Production secrets live in the nichedb--prod logicsrc vault:

bun run secrets:pull     # logicsrc teams pull profullstack nichedb prod --env .env
bun run secrets:push     # after adding a key locally

ROLES=web,worker runs everything in one container; ROLES=worker on a second service splits ingestion out when one instance stops being enough.

Deploy

One Dockerfile, one Railway service, plus Railway's Postgres and Redis. railway.json sets the healthcheck at /healthz. Set SITE_URL to the public origin before anyone registers a passkey: the passkey relying-party id is derived from it.

Adding an adapter

import { defineAdapter } from '@nichedb/core/adapter';

export const example = defineAdapter({
  name: 'example',
  title: 'Example API',
  collection: 'packages',
  description: 'What it watches, in one paragraph.',
  kinds: ['thing'],
  cadenceMinutes: 30,
  configFields: [{ key: 'topic', label: 'Topic', required: true }],
  defaultSources: [{ slug: 'example-default', name: 'Example', config: { topic: 'all' } }],
  async pull({ config, cursor, http, log, budget, deadline }) {
    const page = await http.json(`https://example.com/api?topic=${config.topic}&since=${cursor.since ?? ''}`);
    return {
      items: page.results.map((r) => ({ externalId: r.id, kind: 'thing', title: r.name, url: r.url, publishedAt: r.at, tags: r.tags, data: r })),
      cursor: { since: page.next },
      note: `${page.results.length} things`,
    };
  },
});

Register it in packages/adapters/src/index.js. The core normalises items, hashes them so unchanged rows cost no write, writes in batches, records the run and reschedules.

Streaming a dump

A source that reads a multi-gigabyte file (MusicBrainz, Discogs, Open Library, Podcast Index) cannot return one array: the file does not fit in memory and the walk does not fit in one run. Instead pull yields batches and the core drains them one at a time, writing each and saving its cursor before reading the next. pull is an async * generator (or returns { items: <async iterable> }, which is the same thing), each yielded value is { items, cursor }, and the generator's return value is the run's { cursor, note, nextInMinutes }.

import { dumpDir, gzipLines } from '@nichedb/core/dump';

export const example = defineAdapter({
  name: 'example-dump',
  // ...
  cadenceMinutes: 60,
  budgetMs: 60 * 60_000,
  async *pull({ cursor, http, log, deadline }) {
    const version = await http.text('https://example.com/dumps/LATEST');
    const skip = cursor.version === version ? Number(cursor.skip) || 0 : 0;
    if (cursor.version === version && cursor.done) return { cursor, note: 'unchanged' };

    const dir = await dumpDir('example');
    const file = `${dir}/${version}.ndjson.gz`;
    const dl = await http.download('https://example.com/dumps/latest.ndjson.gz', file);
    if (!dl.complete) return { cursor: { version, skip }, note: 'download in progress', nextInMinutes: 1 };

    let n = skip;
    let batch = [];
    for await (const line of gzipLines(file, { skip })) {
      n += 1;
      batch.push(toItem(JSON.parse(line)));
      if (batch.length === 500) {
        yield { items: batch, cursor: { version, skip: n } };
        batch = [];
        if (Date.now() > deadline) return { cursor: { version, skip: n }, nextInMinutes: 1 };
      }
    }
    if (batch.length) yield { items: batch, cursor: { version, skip: n } };
    return { cursor: { version, skip: n, done: true }, note: 'complete' };
  },
});

The rules:

• budgetMs is the run's wall-clock budget and replaces INGEST_RUN_DEADLINE_MS (4 minutes) for this adapter. The reaper window and the BullMQ lock widen to the largest budget declared, so an hour-long budget is safe to declare. The core hands it back as deadline; check it between batches and return. A batch written past the deadline ends the run from the core's side too, rescheduling in a minute, but the adapter checking first is what keeps a run inside its lock.
• Each batch's cursor is the position after that batch's items (a line count, a byte offset, a last id) plus whatever identifies the file (a dump date, a version, a LATEST value), so a new upstream file resets the walk. It is saved as soon as the batch is in the table. At-least-once delivery is fine: a batch re-written after a crash is an idempotent upsert on (source, externalId) and unchanged rows cost nothing.
• return the final { cursor, note, nextInMinutes }. nextInMinutes: 1 keeps an unfinished walk moving between cadences; done: true in the cursor lets the next run short-circuit when the upstream file has not changed.
• http.download(url, path, { timeoutMs, headers, onProgress }) streams to disk and resumes with Range from whatever is already there (append on 206, restart on 200, 416 means whole). It returns { path, bytes, complete }; on complete: false, return and call it again next run. The user agent is sent; Podcast Index refuses requests without one. Discogs ignores Range and rate-limits hard: expect a restart and keep requests to a handful an hour.
• dumpDir(name) is where the file goes: under INGEST_DATA_DIR when set (mount a volume there), else the OS temp dir, which a redeploy wipes. The cursor is the walk; the directory is a cache.
• Readers: gzipLines(path, { skip }), xzLines(path, { member, skip }) (MusicBrainz: member: 'mbdump/artist', needs xz on the host), tsvJsonLines(path, { skip }) for Open Library's five-column rows (each record carries its lineNo), lineOffsetReader(path, { offset }) for a plain file you want to seek in, untar(path, dir) and sqliteRows(dbPath, sql) for Podcast Index, and bzip2CsvRows(path, { skip, escape }) for CourtListener's bzip2 CSV tables (pure JS decoder, no bzip2 binary; records span lines, an unquoted empty field is null and "" is the empty string; csvRecords(chunks, opts) underneath reads RFC 4180 or COPY's ESCAPE '\'). Every reader yields every line, so skip + lines read is the file position. Skipping into a compressed file re-inflates from the top (seconds per gigabyte for gzip, slower for xz, slower still for bzip2 at about 10 MB/s); a plain-file offset is a seek. A bzip2 file cut short ends the iteration cleanly after its last whole record and reports through onTruncated; a corrupt one throws.

License

MIT

Geographic filtering and scanner context

Use ?lat=41.88&long=-87.62&radius=10 across located collections, searches and feeds. Geographic queries documents the API, CLI/MCP flags, permissioned scanner catalogs, coverage semantics and migration.