SQL Benchmarking Laboratory
A deterministic, orchestrated harness for verifying database performance at scale. Developed by Ramona C. Truta
Agenten stärken
Der eigene Befehl des Herausgebers. Kein PlugMCP-Endpunkt.
Host a copy with our keys and logs.
uvx sqlbenchdag==0.1.2Claude Code hat keinen Installationslink für einen lokalen Befehl. Kopieren Sie den Befehl unten und führen Sie ihn in einem Terminal aus.
SB_RESULTS_DIR=<value> SB_API_BASE=<value> claude mcp add io.github.rctruta-sqlbenchdag -- uvx sqlbenchdag==0.1.2In einem Terminal ausführen.
{ "mcpServers": { "io.github.rctruta-sqlbenchdag": { "command": "uvx", "args": [ "sqlbenchdag==0.1.2" ], "env": { "SB_RESULTS_DIR": "<value>", "SB_API_BASE": "<value>" } } }}claude_desktop_config.json
Remote-URL oder ein Befehl auf Ihrem Rechner? Worin sich die beiden Installationen unterscheiden
Auf PlugMCP ausführen
A new server you create. It is not this listing's official endpoint. Use the publisher install when you want their URL and their credentials.
StartenThis platform is a specialized laboratory for testing SQL performance hypotheses. It transforms query tuning from intuition into a reproducible science.
The core of the system is a Deterministic Orchestration Harness that guarantees that if the logic or the environment changes, the benchmark result changes. If they do not, the result is addressable and cached.
Scope: the focus to date is synthetic and canonical (TPC-H) data, which is the right instrument for mechanism experiments — where controlled, reproducible data isolates the variable under test. Real-data support exists but is experimental; see the FAQ for the synthetic-vs-real rationale, the container model, and the roadmap (AI-security testbed, real-data trust chain).
The Heart of the system is the Experiment ID, an 8-character hash that governs the entire lifecycle. This hash is a SHA-256 fingerprint generated from:
assets/), engine clients (resources/), and data generators (plugins/).Semantic Normalization: The hashing engine distinguishes between a logic change and a formatting change.
To ensure IO-bound performance is not masked by memory buffers, we implement a dual-layer cold start mechanism:
thrash_os_cache primitive that maps and dirties an anonymous memory region sized at half of available RAM, capped at 4 GB. This evicts OS Page Cache entries before each measurement, for both containerized and in-process (DuckDB) engines.The platform is built for the future of Autonomous Engineering. The Experiment ID allows AI agents to treat the laboratory as a Deterministic Performance API.
Benchmarks are defined as N-dimensional matrices in YAML. The platform expands these into a Cartesian product of Independent Dagster Partitions. This allows for parallel dispatch and granular retries.
setup.sh uses it, and it can install Python 3.11 for you.From PyPI — the lab is installable, which gives you the tooling and the sqlbench CLI:
pip install sqlbenchdag
From source — clone the repo to also get the published capsules (sql_benchmarks/experiments/results/) and the full harness. The laboratory includes a comprehensive setup script that manages virtual environments, dependencies, and directory initialization.
# 1. Automate Setup
chmod +x setup.sh && ./setup.sh
# 2. Activate Laboratory
source venv/bin/activate
While the system is powered by Dagster, the primary interface is the CLI for automated workflows.
# QUICKSTART: DuckDB only, no Docker required — runs in seconds
./run.sh sql_benchmarks/experiments/queue/quickstart.yaml --auto
# FULL BENCHMARK: Run all queued experiments (requires Docker for Postgres)
./run.sh queue --auto
# SINGLE EXPERIMENT: Pass a specific config path
./run.sh sql_benchmarks/experiments/queue/quack_execution_modes.yaml --auto
To write your own experiment, start from the annotated template and see the catalog of published runs in docs/experiments.md.
Managed with a strict separation between Harness and Scenario:
/sql_benchmarks
├── /assets # THE HARNESS: Dagster factories & pipeline logic
├── /resources # THE INFRASTRUCTURE: DB drivers & Docker management
├── /scripts/sql # THE SCENARIOS: Raw SQL partitioned by scenario
├── /plugins # THE DATA: Declarative generators & scenario providers
├── /utils # THE BRAIN: AST-hashing, common logic, & system primitives
└── /experiments # THE LABORATORY
├── active.yaml # LOCAL runtime staging — the coordinator overwrites
│ # this on every run (gitignored; not shared state)
├── /queue # Staging area for new experiment configs
├── /archive # Library of previously defined experiment templates
├── /configs # Registry of immutable, hash-addressed experiment capsules
└── /results # Data capsules: fragments, CSVs, and Dashboards
The first published study with this lab: an independent measurement of DuckDB's Quack client-server protocol (beta, v1.5.3). Each row links the runnable config to its committed, verifiable capsule. Full numbers, scaling exponents, and verification steps in docs/published_capsules.md.
Act 0 (b82b4eae) is the exploratory scout that started it — see docs/published_capsules.md.
The lab also ships scenario suites for other studies — selectivity/,
null_logic/, null_sentinel/, recursion/, sort_spill/, tpch/, and more
under sql_benchmarks/scripts/sql/ — each runnable across the engine matrix.
The lab works at two layers, named two different ways. Keeping them distinct is the difference between a number you can trace and a number you have to trust.
A release contains capsules; a capsule is one experiment. The machine names
capsules (hashes you can't forge); the author names releases (words that mean
something). sqlbenchdag is the lab's maker's mark — sql + bench +
dag (Dagster, the orchestration that distinguishes this lab from a bare query
timer). It is carried on every release this lab produces.
Two hashes, two jobs (a capsule contains both — they are not the same thing):
integrity.seal) is a checksum — it
verifies the capsule's contents are untouched. You file by the address; the
seal is the tamper-evident lid.Reproducibility is pinned to the release. Because the Experiment ID hashes the measurement code, it is stable only at a fixed code revision — which is exactly what the signed release tag freezes. To reproduce a published capsule, check out its release tag and re-run its config; the same question yields the same ID, or the comparison is refused by construction.
Provenance roadmap: a per-capsule
generatorstamp (sqlbenchdag@<commit>) will record the exact build that produced each capsule. It writes through the finalization code, which lives inside the Experiment-ID hash, so it ships with a future release rather than re-numbering capsules already published.
See docs/published_capsules.md for the full trust chain (reproducibility, integrity, timestamp, authorship) and how to verify each. Reproducibility and integrity are automatic on every run; timestamp and signature are optional steps you add only when publishing — the producer workflow is docs/PUBLISHING.md.
Copyright 2025-2026 Ramona C. Truta. Licensed under the Apache License 2.0.
Built with Dagster, Polars, Docker, DuckDB, and Postgres.
Query pre-computed SQL engine benchmarks. Runs standalone, no server setup required.
{ "mcpServers": { "io.github.rctruta-sqlbenchdag": { "command": "uvx", "args": [ "sqlbenchdag==0.1.2" ], "env": { "SB_RESULTS_DIR": "<value>", "SB_API_BASE": "<value>" } } }}Tools
Noch keine Tool-Liste. Remote-Server erhalten eine nach dem ersten Health-Check.
FAQ
Kopieren Sie auf dieser Seite die stdio-Konfiguration für Cursor, Claude Desktop, Claude Code oder VS Code. Sie nutzt die Installation des Herausgebers, keinen PlugMCP-Proxy.
Der Herausgeber nennt diese Umgebungsvariablen: SB_RESULTS_DIR, SB_API_BASE.
SQL Benchmarks Lab läuft lokal über stdio.
Zuletzt geprüft: 2026-11-10. Query pre-computed SQL engine benchmarks. Runs standalone, no server setup required.