Summary / Harness arms
A4 h1 (three-session vote)
Three independent sessions on separate copies; patches grouped by normalized AST; largest group wins.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
Dev (09-14 → 09-16): 145/150 (96.7%) against h0-v41's 128, +11.3 points, CI [+5.6, +17.0]. 19 tasks only h1 solved, 2 only h0. Held-out plus regression (09-29, pre-registered, run on a bench box): 99/100 against 93, +6.0, CI [+1, +12], confirmed. The one h0 win was an h1 cell lost to a full disk.
What is missing
Switching on. Production has the wiring (dispatch_vote, bridge 1.90, mode off); it needs a replay-bench run on her own tasks first.
Reviewer Claude, 2026-10-04
Reviewer's read: Adopted by rule on both pre-box sets; unconfirmed on boxed rows (10-19)
An out-of-memory kill interrupted the dev run; it resumed on the identical frozen tree, and the advantage held on both sides of the break (10-1 before, 9-1 after). The report disclosed it rather than averaging the contaminated wall-clock and cost figures. Cost is about 2.7 times per task and wall time 1.6–2.7 times. Whether that buys the same on her real delegate tasks is the next question.
How it is compared unconfirmed
Boxed re-run 10-19: undetermined (+7.5, CI [−3.1, +18.0]) at 9× cost. Next: a cost-capped h1 arm on boxed rows, then the replay bench (B5).
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-09-29 | v3 | sealed | 99 | — | — | $7.00 | h1-v41 on holdout + regression, k=2: 99/100. vs h0-v41 +6.0, CI [+1, +12]. Pre-registered rule met: CONFIRMED. Cost about $7 for both arms. |
| 2026-09-16 | v2 | sealed | 96.7 | — | — | — | h1-v41 on dev, k=3: 145/150. vs h0-v41 +11.3, CI [+5.6, +17.0], McNemar p = 0.0002. ADOPT. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.