Summary / Benches
B6 RSI loop graded set
The daily keep/rollback loop's yardstick: optimisation tasks scored continuously, not pass/fail.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
g1: 16 graded tasks (scheduling, packing, routing, sequencing), scored as quality against a reference and clipped. 4 dev, 12 held-out, plus 12 pass/fail guard tasks. Calibrated 09-24 at 0.762, not saturated. The loop runs daily at 10:00 on its own account. Its first real result (09-26) kept a cheaper solver at the same quality for half the cost.
What is missing
A mean gain bar above the noise: the paired run-to-run spread is about 0.13 per task without timeouts, so small gains need many tasks.
Reviewer Claude, 2026-10-04
Reviewer's read: Live and unsaturated
The only coding yardstick with headroom today, because it scores how good a solution is rather than whether it passes. Its weakness is reliability rather than skill: most zeros are runs that ended without a valid solution at the time limit. It measures the solver, not the harness, but it is the mechanism (hidden set, rule fixed in advance, automatic rollback) that the harness components were meant to go through, one at a time.
How it scores live
Paired over tasks, k=2, minimum gain 0.05; keep, rollback or void decided by the written rule.
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-09-24 | v1 | sealed | 76.2 | — | — | $1.86 | g1 calibration, k=2, 32 solves: mean quality 0.762 against reference. 12 of 16 tasks steady; 4 coin-flips. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.