Leaderboard
Season 1 board
The board compares memory systems on sealed lives no one has seen, with gates measured on control arms in the same blocks. It is published only after the season-1 sealed board (v1.0) has run.
Status
Not published yet
No system has run on a sealed board, so there is no ranking here. Until v1.0, this site carries the method, the developer packs and the roadmap, and every developer run is published on the runs page with its labels.
Next release: v0.3 Developer tools: close the loop.
- v0.1 Kitreleased
- v0.2 Developer loopreleased
- v0.3 Developer tools: close the loopnext
- v0.4 Developer tools: search and sensitivityplanned
- Pilotplanned
- v1.0 Boardplanned
What each row will show
One table per season, built from sealed result cards. The preview below has the board's columns and no rows.
| Stack | Act, penalised | Action accuracy | Harmful actions | Twin consistency | Certification | Operating |
|---|
Season 1 has not run. Rows appear here from sealed result cards only.
- Stack
- memory × harness × model, plus the vision provider and spend
- Act, penalised
- −100 to +100, with its confidence interval; the 0-100 scale is for display only
- Action accuracy
- per-item action accuracy, beside the penalised score
- Harmful actions
- actions driven by a false, stale or planted memory, and violations
- Twin consistency
- reported beside the scores
- Certification
- on Act per declared profile, with its stability
- Operating
- latency p50 and p95, added turn latency, store growth, time-to-correct, cost and wall time
- The capability map: track rows by profile, with cell verdicts, cause labels such as "unsupported", and the non-results N/A, N/A_READER and BLOCKED.
- The baselines of each profile: the no-memory floor, the oracle-evidence ceiling, the full-history oracle ceiling where it fits, and naive RAG.
- Sensitivity rows beside the headline: the −1 outcomes rescored at 0 (plain accuracy) and at −2.
- G1 task completion, beside the scores and outside the headline.
- A raw-archive marker column, and a small crossover panel of other models and harnesses.
Board rules
- 1Every system's latest version is shown, never its best.
- 2Every scheduled run is published.
- 3Delta boards run only on a new GA, with the previous version re-run alongside.
- 4Certification is issued only on a sealed board at the pre-registered size. A season cut below the powered size issues no certificate, and its results are still published.
- 5No claim of rank without separation: systems whose intervals overlap are shown as tied bands, not as ranks.
- 6The memory headline holds the reader and harness fixed.
Joining the board
A system joins with an open-source, self-hostable memory that has a Hermes memory plug-in or a documented ingest and query API, under the same configuration budget as every other system.
- Model-native memories join a separately labelled stack view, each row vendor-submitted with a disclosed configuration.
- Model labs use a sealed slice (v1.x): the benchmark runs the lab's model inside the sealed runner.
- Before then, registered systems on a Hermes harness can run the public developer tiers; bringing your own system from outside the benchmark checkout arrives in v0.3: start here.