A database for benchmark results. Store every run, analyse regressions across commits, and share the findings with a link — before the screenshot ends up buried in a PR thread.
$ critdb query --bench parse_json --since 2026-01 ┌──────────┬───────────┬───────────┬─────────┐ │ commit │ mean_ns │ p99_ns │ change │ ├──────────┼───────────┼───────────┼─────────┤ │ a3f1b2c │ 142,330 │ 198,221 │ +0.4% │ │ 7e9d0a4 │ 141,802 │ 195,014 │ 0.0% │ │ 2c8b1f5 │ 138,990 │ 189,443 │ -2.0% │ │ 9f4e8c1 │ 141,005 │ 194,872 │ +1.8% │ │ b8a3d27 │ 137,442 │ 188,001 │ -2.5% │ └──────────┴───────────┴───────────┴─────────┘ 5 runs · 14d · trend ▂▃▂▄▃▅▄▆▅▇▆█ ±2.1% significant regressions: 0
Today, performance regressions are caught by luck. A developer runs Criterion on their laptop, screenshots the terminal output, pastes it into a PR, and the data dies there. Six months later, nobody can find the baseline.
CriterionDB is a long-term storage layer for the numbers that benchmark harnesses already produce — Criterion, JUnit, custom scripts. Queryable, diffable, shareable. It does not run your benchmarks. It remembers them.
The schema is format-agnostic. Whether a run came from Rust's Criterion, a JUnit XML report, or a custom script timing your TSP solver — it lands in the same shape, queryable with the same CLI.
Push results from Rust's Criterion, JUnit XML from your test suite, or raw timings from your own TSP solver. The schema is format-agnostic — each run is tagged with commit, branch, hostname and environment. Your harness decides what to record; CriterionDB remembers the rest.
$ critdb push results.json \ --format criterion \ --tag commit=$CI_COMMIT_SHA \ --tag branch=$CI_COMMIT_BRANCH # also supported: # --format junit # --format custom
Compare runs across commits, branches, or arbitrary time windows. Built-in statistical comparison — Mann-Whitney U on the underlying samples when available, Welch's t-test otherwise — so you don't ship a 12% regression thinking it's noise.
$ critdb diff a3f1b2c..7e9d0a4 --bench parse_json → mean: -2.0% (p=0.003, significant) → p99: -2.8% (p=0.001, significant) → samples: 100 vs 100
Generate a shareable URL for any query or comparison. Paste it in a PR description, an issue, a Slack thread — the recipient sees the same chart and the same numbers, no login required. Links expire when you say they expire.
$ critdb share --query "parse_json since=2026-01" → https://criteriondb.dev/s/a3f9c2 → expires: 30d · views: 0
Built by people who have shipped performance regressions and lied to themselves about whether the chart was noise. The design follows from that.
We store raw observations — every sample, every percentile, every environment variable that could explain a slowdown. No composite "performance scores" that hide what actually happened. The chart is the data.
A 5% change might be noise. A 0.5% change might be real. We run the right test — Mann-Whitney U for non-parametric, Welch's t for normal-ish — and show you the p-value. You decide the threshold.
Every result is exportable as JSON or Parquet. No proprietary format, no lock-in. If CriterionDB disappears tomorrow, your data walks out the door with you, in the same shape it came in.
The server is MIT-licensed and ships as a single binary backed by SQLite or Postgres. Run it on your laptop, on your team's VPC, or use the hosted instance when it's ready. Your choice, always.
The project is pre-alpha — day one. Signups are closed while the storage schema and CLI stabilise. Below is the order of operations.
Defined the run/result/tag schema. Append-only storage working locally with SQLite. CLI scaffold in place — push and query work, but only for Criterion output so far.
First-class ingest for Criterion JSON. JUnit XML and a generic custom format follow — so your TSP solver timings land in the same database. The tag system stabilises. This is when critdb becomes useful for a solo developer.
Statistical comparison across runs. Shareable links for any query or diff. A read-only web view that doesn't require an account. This is when it becomes useful to a team.
First hosted instance for a small set of early teams. Still invite-only. Free for open-source projects, paid for private workspaces.
Self-serve signup. Public sharing with deduplicated URLs. Integrations with GitHub Actions, GitLab CI and Buildkite. The thing you can put in your README.