Summary / The 27 components
27 Benchmarking and objective scoring
Measuring real engineering ability, not the model's opinion of itself.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
The whole rig (see the Benches section): stratified SWE-bench Verified sample split 50 / 30 / 20, the official grader in pristine containers, an A/A run for the noise floor, a paired bootstrap with a written adoption rule, pre-registration enforced at launch, and holdout burn counting with an audit.
What is missing
A smoke test on every harness commit, gold-based diagnostics (recall@5), growing piperbench to 40 tasks, and a harder set, now that dev is saturated.
Reviewer Claude, 2026-10-04
Reviewer's read: Built first, and it held
Building the measurement before any component was the programme's best decision. It caught a budget confound, a model-alias change under the arms' feet (09-10), timestamps on a registration that came after its run, and a model id that would have fallen back silently. Its limit now is ceiling: h1 resolves 97% of dev and the held-out sets are near-perfect, so the rig can no longer show a gain. The next component needs a harder, fresh set before it is worth running.
How it is measured built
See How the tests work.
Adoption rules shared by every arm: How the tests work.
Research & attempts
- Build Spec 27 — Benchmarking / Objective Scoring Spec 2026-09-06
- Component 27 — Benchmarking / Objective Scoring Paper 2026-09-06
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.