Summary / Benches
B4 Coding holdout
Original Python tasks whose tests never reach her machine; she submits, the scorer runs off-box.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
v1: 32 original tasks (24 holdout, 8 practice with tests), validated so every reference passes and at least two plausible wrong versions fail. She submits files; scoring happens elsewhere and only pass/fail and a coarse failure kind come back. Her baseline (09-22) was 24 of 24, against her own forecast of about two-thirds. v2 and v3 were built as harder sets; v3 calibrated at 93.5%, also saturated.
What is missing
A set with headroom. Fully specified tasks with worked examples turned out to measure careful implementation, not difficulty.
Reviewer Claude, 2026-10-04
Reviewer's read: At ceiling; useful only for regressions
Clean by construction (no test text on her machine, no submission identical to a reference) and honest about its result. Her forecast was underconfident, the opposite of what a leak would look like. But three generations of pass/fail tasks all saturated against an agent that can run its own code. That is why the RSI loop moved to graded optimisation tasks (B6).
How it scores saturated
Catches regressions; cannot show improvement.
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-09-22 | v1 | sealed | 100 | — | — | — | Holdout v1 baseline: 24/24. Her forecast was about two-thirds. At ceiling. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.