Summary / Benches
B5 Replay bench
Her own past builds, re-run from their original starting state with the tests hidden, graded by their original acceptance tests.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
A pool of her past delegate builds whose later undo was complete. Each case rebuilds the pre-state, hides every answer source, lets a worker attempt the task, restores the tests byte for byte, and grades with the original bar (it must be red before and green after). Results 09-27: tests visible 20/20; tests hidden 11/22 (50%); hidden plus an API hint 12/22. A fairness pass (09-28) adds the exact literals the tests check whenever they are not in the task.
What is missing
Repetitions (k≥2), a larger pool (about 60 needed; 55 valid on 09-28), and a trigger that runs it on changes to her worker path.
Reviewer Claude, 2026-10-04
Reviewer's read: The right yardstick for her own work
The visible-against-hidden gap is the single most important number in the programme: a worker that can see its tests looks twice as capable as one that cannot. This is the bench that measures what the harness is for, which is her real tasks rather than public ones, and it is where the vote (16) should be tested before it is wired into production. Run-to-run noise is about the size of the hint effect (5 of 22 flipped), so single runs decide nothing.
How it scores built; pool growing
Hidden tests plus spec hint, k=3, paired; pass = red before and green after with the original bar.
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-09-27 | v1 | sealed | 50 | — | — | $0.38 | Tests hidden: 11/22. With tests visible the same worker passes 20/20. With an API hint: 12/22 ($0.40). |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.