OpenChainBench

OpenChainBench

Open, reproducible benchmarks for crypto infrastructure. Aggregators, bridges, RPCs, oracles, price feeds, prediction markets, perp DEXes, cross-chain messaging. Same metric, same conditions, every provider. Live at openchainbench.com.

OpenChainBench publishes one benchmark at a time, each shipping with the harness that produces its data. The goal is to make performance an observable property of crypto infra, measured in the open, by anyone who wants to add a provider or a metric.

Community-run, MIT-licensed harnesses + CC-BY-4.0 data, PRs from any party including the providers we benchmark.

At a glance

• 94 live benchmark specs across 8 categories (RPCs, Trading, Aggregators, Bridges, Blockchains, RWA, Explorers, NFT APIs)
• 50 self-contained Go harnesses, each shipping a /metrics Prometheus endpoint
• 32 curated question/answer pages · 23 "alternatives to X" landing pages
• 26 machine-readable API routes (citation, badges, MCP, feeds, per-format quotes)
• One shared Prometheus, one materialization worker, CDN-fastpath snapshot store: cold reads in < 100 ms edge, no origin fan-out on hot paths

Quickstart

pnpm install
pnpm dev              # site at http://localhost:3000
pnpm validate         # bench spec Zod schema check (run before opening a PR)
pnpm typecheck        # tsc --noEmit

Adding a benchmark, writing a harness, or fixing a number: see CONTRIBUTING.md + the visual walkthrough at /contribute.

Machine-readable surfaces

Every benchmark is licensed CC-BY-4.0 and exposed through endpoints designed for citation by journalists, devs, and AI agents:

Structured data on every bench page: Dataset + StatisticalReport + Observation + TechArticle + FAQPage + BreadcrumbList JSON-LD, plus a schema.org Dataset microdata graph layered on the visible "At a glance" infobox.

Journalist citation format (what /api/stat/<slug>.headlineSentence emits):

"Mobula leads head lag at 0.8s (p50, 24h) on Fastest onchain data provider.
Source: OpenChainBench (https://openchainbench.com/benchmarks/aggregator-head-lag)"

Site surfaces

Architecture

┌─────────────────────────┐    ┌───────────────────┐    ┌──────────────────────┐
│  50 Go harnesses on     │    │  Shared Prometheus │    │ Materialization worker│
│  ocb-par-main (Paris)   │─── ▶│  (same VPS)        │───▶│ (same VPS, docker)   │
│  each exposes /metrics  │    │  federates a few   │    │ sweeps every 60s     │
└─────────────────────────┘    │  Railway regional  │    └──────┬───────────────┘
                                │  harnesses         │           │
                                └───────────────────┘           ▼
                                                    ┌─────────────────────────┐
                                                    │ Redis + CDN blob store   │
                                                    │ (SRH at kv.opencha…)     │
                                                    └──────────┬──────────────┘
                                                               │
                                                    ┌─────────────────────────┐
                                                    │ Vercel Fluid Compute:    │
                                                    │ pages read CDN blobs     │
                                                    │ first, SRH as fallback,  │
                                                    │ last-known-good on cold  │
                                                    └─────────────────────────┘

• Harnesses + Prometheus + worker + Redis + SRH run on one VPS (ocb-par-main in Paris) via docker compose. A handful of multi-region harnesses (rpc-capabilities-us/eu/sgp, evm-quote-latency, aggregator-head-lag regional) stay on Railway and federate into the VPS Prom.
• Worker publishes atomic snapshots every 60 s to Redis + CDN blobs at kv.openchainbench.com/aggregate/{latest,benches/<slug>,variants/<slug>/<sig>}.json. Site reads CDN first (~20 ms edge), falls back to SRH, then to the last-known-good snapshot key on cold cache.
• Site is Next.js 16 App Router on Vercel Fluid Compute. Auto-deploy from main on every push (with sitemap smoke test + auto-rollback + bench-page warm-up); dev branch runs on staging preview.
• Middleware (src/middleware.ts) handles lowercase 308 normalization, /compare/<b>-vs-<a> → canonical alphabetical, 410 Gone for removed benches, 301 for renamed ones, and cache-key normalization on read-only public APIs.
• Live stream relay (WebSocket at wss://stream.openchainbench.com/ws) is a separate service run by the OpenChainBench maintainers because it holds upstream API keys; the browser talks to it directly, Vercel only serves the static shell.

Full walkthrough with diagrams + a "clone-and-run-it-yourself" section is in docs/architecture.md.

Repo layout

benchmarks/        94 YAML specs, one per public benchmark
harnesses/         50 Go harnesses, each self-contained + Dockerized
answers/           32 Q&A YAMLs backing /answers/<slug>
alternatives/      23 "alternatives to X" YAMLs backing /alternatives/<slug>
worker/            Materialization worker (sweeps Prom → publishes CDN + Redis blobs)
infrastructure/    Shared Prometheus config
src/               Next.js 16 site (App Router, ISR, Tailwind 4, TypeScript, Zod)
docs/              architecture.md, methodology, walkthrough.md
scripts/           Spec validators + prebuild manifest generators
.github/           CI: prod deploy, staging preview, sync dev from main

Contributing

Read CONTRIBUTING.md for the long-form guide. TL;DR:

1. Float an idea in Discussions
2. Open a 📊 Propose a benchmark issue with the metric + methodology
3. Write the harness (Go recommended, any language shipping /metrics works)
4. Host it (anywhere with HTTPS + public /metrics)
5. Open a PR against dev: spec YAML + harness ref + Prom scrape config
6. Review on the staging preview; merge to main when data lands

Editorial conventions

• Same metric, same conditions. If two providers can't be measured identically, they don't share a bench.
• Provider marketing does not shape published numbers. Public methodology + reproducible harness are the only source of truth.
• Retirement over removal. Retired benches return 410 Gone with a link to the current catalog so previously-indexed URLs decay cleanly.
• Freshness signals everywhere. Every citable surface carries dateModified + asOf so LLMs can tell fresh data from stale.
• Draft benches never quote a number. If quorum drops, the page renders "awaiting samples" rather than a fabricated leader.

Community + license

• Discussions: github.com/ChainBench/OpenChainBench/discussions
• Twitter/X: @OpenChainBench
• Wikidata: Q140172649
• Zenodo DOI: 10.5281/zenodo.20800311
• Hugging Face dataset mirror: OpenChainBench/benchmarks (daily parquet snapshots)

License: harness code MIT · dataset CC-BY-4.0 · attribution required. See LICENSE.