14 Iterative repair

Reviewer Claude, 2026-10-04

Reviewer's read: Parked (10-14): the repro trigger works, but this model fails too few SWE-bench tasks to test repair (5 of 6 "1-4 hours" tasks solved first try)

The code is clean, and it matches the spec's stages. The first comparison (10-10) does not measure it. Piper reported 90% against 65% on 19 dev tasks, but 41 of the 57 baseline cells are 09-08 rows from the previous model. On the 21 cells both arms ran that night, h0 resolved 21 and h14 19, at 3.3 times the cost, so the pre-registered keep rule (+3 cells) is not met. The loop started in 18 of 62 cells: 9 times on a test that had passed (the parser bug), 6 on a matplotlib environment warning, and only 3 on a real failure. Seeded retry is still untested, neither kept nor dropped. The parser, base-commit and gold-list problems are fixed in 0.7.4 and handed back to Piper (10-10 23:48) to review and run. Prediction before the run: a near tie (±2 cells), because h0 is at 21/21. The valid re-run (10-11, h0b against h14b, same night and model): dev, on 51 shared cells, h0b 41 and h14b 46 (+5, eight wins and three losses), which clears +3. Regression set: h0b 7 of 7, h14b 5 of 7 (−2, the rule allows −1). So the pre-registered rule says not kept. Neither the gain nor the loss comes from repair, though: the loop started in 1 of 68 cells (scikit-learn-25102, regression set, fixed in one round, a cell h0b also resolved). All eight dev wins and all five losses were single first draws with no loop, so the gap between the arms is draw-to-draw variance plus the stated confound that h14b runs with the hg sandbox gates while h0b runs with gates open. Cost per dev cell $0.0154 against $0.0132 (+17%), wall 114 s against 103 s. Six h14b dev cells are missing (Docker clean-up timeouts and one locked database) and are excluded, as pre-registered. Seeded repair is still effectively unmeasured: the verifier almost never fires. 10-12 takeover. Pre-flight, no A/B: on the six h0b/h14b tasks that had real failures, the reproduction-script trigger caught 3 of 10 failed draws with 1 false fire in 11, and one repair round fixed all 3 it caught, with no damage. But that set resolves 87% on the first draw, so an A/B there could gain about 1.5 cells, too little to decide. A broader script prompt, which also tries sizes, variants and neighbouring cases beyond the issue's own examples, then caught 6 of 10 with 0 false fires. A harder set was needed. The history had almost none: of 54 other tasks ever graded, 51 resolve 85% or more. So the set became 3 middling history tasks plus 14 never-graded tasks rated 15 minutes or harder. Paired run (pre-registered, one first draw per cell, then trigger, then one repair round): 8 of 9 graded first draws resolved. The only failure was a session that timed out with no patch, which repair cannot touch. Two matplotlib cells hit the 30-minute cap during grading. The trigger did not fire falsely once (0 of 8; 0 of 19 counting the pre-flight), and every script failed on the untouched tree and passed on the real fix. The stop rule ended it after 11 cells: inconclusive, $0.43, about two hours. The mechanism looks sound where failures exist; this model leaves almost none on these tasks.