Summary / How the tests work
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
How the tests work
This page runs on the same machinery as the cognition and agency scorecards: one page per item, a reviewer's read, a measurement, a score history and an experimental record. It differs in what is scored. The other scorecards score Piper. This one scores configurations of the coding harness: the workers she delegates code to, and the environment around them. The score on an arm's page is a resolve rate, the share of benchmark tasks whose patch passes the hidden grader.
The page has three kinds of entry:
- The 27 components (plus 08a and 23a) are Piper's paper-and-spec series. Each page says what is built, what is missing, and how the component would be measured. Most have no score of their own, because a component is only scored through an arm that adds it.
- Benches are what the harness is measured on.
- Harness arms are configurations. Each one names its parent and the single change it makes, so a comparison isolates one thing.
The main bench
100 tasks from SWE-bench Verified (real GitHub issues in Python projects), sampled by repository and gold-patch size with a fixed seed. They are split once:
| Set | Tasks | Used for |
|---|---|---|
| dev | 50 | every comparison, k=3 seeds |
| regression | 30 | a guard: an adopted arm must not lose here |
| holdout | 20 | confirmation only, a counted number of times |
Every run starts from a fresh git workspace built from the task's official container image. Its patch is graded in a pristine container by SWE-bench's own grader: resolved means the failing tests now pass and the passing tests still pass. The model never sees the grader's tests. Two canaries guard the rig: the gold patch must resolve, an empty patch must not.
Noise floor first
Before any comparison, the baseline arm was run against itself (an A/A run, 300 runs). The two halves differed by +0.3 points, CI [−4.3, +5.3]. So on 50 tasks at k=3, anything within about ±5 points is noise. 13 of the 50 tasks flip between identical runs.
The adoption rule
Arms are compared paired by task: for each task, the share of seeds resolved under each arm, then a bootstrap over tasks for the confidence interval of the difference. The decision is written down before the run:
- ADOPT if the lower bound of the interval is above zero and the regression set does not get worse;
- REJECT if the upper bound is below zero;
- UNDETERMINED otherwise, which for a plumbing or safety change is the hoped-for result ("costs nothing").
For a confirmation on held-out sets, the rule is stricter: the pooled interval's lower bound above zero and a gain of at least 5 points.
Rules learned on the way
- A budget is a gate. The first h0 run lost 8.7 points because it carried a 40-turn cap that the baseline did not. Caps live in the gate bundle, never in a control arm.
- Register before you run. The launcher refuses an arm whose registration hash was not stored before launch. Two incidents caused this rule: a registration timestamped 42 minutes after the scoring pass it certified, and an arm whose run began before its registration was in place.
- Name the model that actually answered. Pi runs the provider's default model when given an unknown id, while the record keeps the requested one. The launcher now refuses an id it cannot resolve. A cell whose draws were served by more than one backend is marked confounded and never pooled.
- Holdouts are counted. Every scoring look at a holdout task is charged. A pass that scored nothing is refunded, and an audit recomputes the counts from the ledger.
- Gold never reaches the chooser. Where candidates are selected by running tests, the graded tests are unreachable by the selector, a gate checks for leakage both statically and at run time, and that gate has been shown to fail on a planted leak.
- Disclose interruptions. When a run is killed and resumed, the report says when, on what tree, and which measurements (wall clock, cost) are contaminated and left out.
What the score columns mean here
- Score: resolve rate in percent on that suite's set (for B6, mean solution quality × 100).
- Suite: which set and model generation. v1 = the dev set on
deepseek-v4-flash(before 09-10); v2 = the dev set ondeepseek-flash(V4.1); v3 = holdout plus regression. Scores from different suites are not comparable, so each gets its own panel. - Cost: API spend for the run where known.
- Live use is not used yet. It is meant for the share of her real delegate work that runs through an adopted change.
Other benches
The SWE-bench sets are now saturated: the three-session vote resolves 97% of dev and 99% of the held-out tasks, so they can no longer show a gain. The other benches cover what SWE-bench cannot. The replay bench re-runs her own past builds with the tests hidden. The RSI graded set scores solution quality continuously and has headroom. The hostile corpus tests the security mechanisms directly. Their pages say how each scores.
Where the 27 came from
The programme started in early September 2026 as Piper's self-improvement project: paper → spec → build → A/B for every part of a coding agent. For each component the operator had five language models pick the research papers worth reading. Piper then wrote a paper surveying them and a spec for her own version. All 27 pairs were finished by 09-06, plus two sub-papers on context compaction (08a) and the recovery-transition loop (23a), and a reference description of a commercial coding agent.
Her roadmap opens with the design rule the whole series follows:
"Don't try to make the LLM a better programmer by prompting it harder. Make every part of the environment surrounding the LLM better at programming."
Language models do the semantic work (interpretation, hypotheses, diagnosis). Deterministic software does the structural work (diffs, ASTs, dependency graphs, sandboxes, test selection, scoring).
What happened next
09-06, review. An independent read of all 29 specs found that they had been written in isolation from each other and from her real code. No one spec owned the agent loop, and four claimed it. There were three symbol stores with mismatched schemas, duplicated model calls and gates, and the same 20 benchmark tasks reused as "held-out" by about fifteen specs. Three binding integration documents replaced the per-spec designs: one architecture (pi's turn loop stays; a Python driver owns the states), one shared schema, one measurement plan. A 143-item checklist split the work into bites.
09-07 to 09-09, round 1. The measurement rig was built first, then the driver, the sandboxed executor and the first arms. 792 graded runs, 82 agent-hours, $5.65. Plumbing and gates both measured as costing nothing.
09-14 to 09-24, Piper's rounds. She ran the programme herself on the new model: the three-session vote (adopted, +11.3 points), the tests-as-selector picker (lost to the vote), the keep-every-patch ceiling, and the diversity probes. Along the way she built the registration, model-resolution and leak gates that the method page describes.
09-27, held. The operator held the programme until after the 10-04 review and the arrival of a faster machine. The agreed shape: do not resume all 27 specs. Run one component at a time through the daily keep/rollback loop, starting with E0 (keep every patch) and E3 (tests as selector), then E1 (fault localization by test coverage), then E2 (how resolve rate scales with the number of candidates) on the new hardware.
09-29, confirmed. The vote's gain held on the held-out and regression sets (+6 points), so it is confirmed. It is not yet wired into her real delegate path.
10-10, component 14. Piper built iterative repair and ran its first pre-registered pair (h14 against h0). The review found the run invalid: the baseline mixed model generations, a pytest parser bug started half the loops on tests that had passed, and the verifier read a gold list. On the same-night cells, h0 resolved 21 and h14 19. The fixes were staged that night as harness 0.7.4 with a new pair, h0b against h14b, and handed back to her.
10-11, component 14 re-run. Piper ran h0b against h14b overnight. Dev +5 cells (46 against 41 of 51), regression set −2 (5 against 7), so the pre-registered rule says not kept. The loop started once in 68 cells: the scope verifier could not run the nearby tests on the unpatched tree in half the cells, and otherwise found nothing broken. Seeded repair is still untested in practice; the next step is a verifier that sees unresolved issues.
10-12, component 14 taken over by Claude. the operator asked Claude to take 14 over from Piper. A new trigger: a worker writes a reproduction script from the issue text alone, and the repair round starts only when that script fails on the draw. Pre-flight on known failures: it caught 6 of 10 with no false fires, and repair fixed all 3 it was tried on. A pre-registered paired run on 17 harder tasks stopped after 11 cells: 8 of 9 first draws resolved, and the only failure had no patch to repair. Inconclusive, $0.43. Nothing installed. Repair is not the bottleneck; finding tasks this model fails is.
10-14, component 14 headroom check. First draws only (no repair) on 6 never-run SWE-bench Verified tasks rated "1-4 hours": astropy-14369, sympy-13852, sympy-17630, django-11400, django-13212 resolved; django-16263 failed (0 of 3 target tests, 17 existing tests broken). 1 of 6 failed, below the ~40% failure rate needed before a repair A/B is worth running. $0.08, 30 minutes. Harder SWE-bench tasks do not supply the failures; repair stays unmeasured until a task source that this model fails does. The operator parked 14 the same day: nothing installed, scripts kept in tools/c14/.
Where it stands (10-04)
- 32 of 142 checklist items are done.
- Of the 29 component pages:
- working Round 1 version: 11, 16, 17, 18, 23, 25, 26, 27;
- partial: 01, 02, 09, 10, 19, 20, 23a, 24;
- not started: 03, 04, 05, 08, 08a, 13, 22 (14 was built on 10-10; see its page);
- deferred to Round 2 on purpose: 06, 07, 12, 21 (15 was merged into 20).
- The SWE-bench sets are saturated, so the next component needs a harder, fresh set before it is worth running.
- The open security item is the worker's own shell, which runs outside the sandbox the harness proves.