Summary / 14. Iterative repair / Document
Attempt 1 · review 2026-10-10
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Iterative repair, attempt 1: h14 seeded retry (review, 2026-10-10) Reviewer's write-up of Piper's 10-10 build and run of component 14. Evidence: `build/harness` on Firefly (`repair.py`, `driver.py`, `bench/runner.py`, `testrun.py`, the registration and pre-registration, `harness.db` scores, ledger and run rows), read after the run finished at 21:28. ## Update, 10-10 late: the fixes are staged as h14b (not yet run) At the operator's request, the fixes below went into Piper's harness as **0.7.4**, and a new same-night pair was pre-registered. Nothing has been registered or launched yet. Piper was sent the findings and the staged fix at 23:48, and the run is hers. | Problem found | Status in 0.7.4 | |---|---| | `parse_pytest` reads colourised output as zero results | **Fixed.** ANSI codes are stripped before parsing (`tests/test_testrun_ansi.py`). This applies to every arm. | | A test that fails before any patch (the matplotlib warning) starts the loop | **Fixed in h14b.** The verifier runs on a fresh fork of the unpatched tree first. The loop starts only when a test that passed before the patch fails, errors or vanishes after it. | | The verifier target comes from `PASS_TO_PASS`, and its node id lands in the prompt | **Fixed in h14b.** The verifier is `picker.scope_targets`, the repository's own tests nearest the edit. It never reads `FAIL_TO_PASS`, `PASS_TO_PASS` or `repro_target`. | | The baseline mixes model generations, and only seed 0 of h0 was re-run | **Addressed by the new pre-registration.** `h0b` (h0 byte-for-byte) and `h14b` run together on the same night and model over all 64 cells, with `--redo`, so the 10-10 rows are kept. | | The diagnosis is a stub; escalation targets a missing component | Unchanged. It is stated in the pre-registration and needs component 13. | - **Tests:** `test_driver_repair_scope.py` is new. With the ANSI test and `test_driver_repair_arm.py`, 12 tests pass (re-run 10-10, about 23:50). - **Pre-registration:** `logs/h14gen/PREREG-h14b-2026-10-10.md`. The keep rule is unchanged: at least +3 shared cells, and the regression set no worse by more than 1. A mechanism report is required whatever the verdict: loop starts by reason, genuine failures by spot-check, resolve by attempt, and cost and wall time per cell. - **Stated confounds:** - h14b's parent is hg (`gates="repair"`) and h0b runs with gates open. - The h14b verifier sees only regressions the patch causes, so it never retries an incomplete fix that breaks nothing. - **Prediction (reviewer's, made before the run):** a near tie, within ±2 cells. h0 already resolved 21 of 21 on the same night, so there is little headroom. Loops should start in well under a quarter of cells. - **Comparability warning:** the parser fix changes how VERIFY reads pytest output in images that force colour, for every arm. Earlier rows from those images may have under-counted passes. Check which ones before comparing new rows with old ones. This is not yet checked. ## Verdict **The code is good. The +25 points she reported are not a result.** Three things are wrong with the comparison: 1. The baseline mixes two model generations. 2. On the cells where both arms ran tonight, h14 is two cells behind. 3. The retry loop under test fired on a genuine failure only 3 times in 62 cells. By the pre-registered rule, h14 is not kept. The seeded-retry idea is still untested: neither "keep" nor "drop" applies to it yet. ## What she built - **`piperharness/repair.py` 1.0.0.** A deterministic, stdlib-only retry loop: - A failure signature (class, normalised message, site) hashed into an append-only ledger. - A per-signature cap of 3 and an absolute ceiling of 35. - A `RoundRecord` seed holding the prior diff, the failure evidence, a diagnosis and a "must not break" boundary. - `repair_loop`: seed → ledger lookup → one attempt → verify → write back → gate. - Components 12 (evidence) and 13 (diagnosis) don't exist yet. They sit behind injectable interfaces with deterministic stubs, so the loop can be tested today and the real components can drop in later. - **Arm `h14`** (parent `hg`, harness 0.7.3): - One ordinary first draw. - If that draw's verifier test fails, the failed workspace is committed and forked, and the loop runs on the fork. - Each round is one session using the new `seeded_repair_v1` prompt, which carries the previous diff, the failing output, the diagnosis and the boundary. - Other arms' prompts are byte-identical, because the new placeholders are no-ops in their templates. - **Pre-registration and tests.** The pre-registration (`PREREG-h14pair-2026-10-10`) and arm registrations were stored at 13:49, before the 13:50 launch. It names one confound honestly: the gate bundle differs between hg and h0. The 33 unit tests (`test_repair.py`, `test_driver_repair_arm.py`) pass; I re-ran them. This is clean, well-separated work, and it matches the spec's shape: stage-by-stage, counter-based gating, providers that can be swapped. ## What the run measured | | h14 (tonight) | h0 as reported | h0 cells re-run tonight | |---|---|---|---| | dev cells scored | 55 of 57 | 57 | 16 (seed 0 only) | | dev resolved | 50 (90.9%) | 37 (64.9%) | 16 (100%) | | regression resolved | 6 of 7 | n/a | 7 of 7 | The 90% against 65% compares tonight's h14 with an h0 column where **41 of 57 cells are 09-08 rows**. The served model changed on 09-10, and the scorecard keeps those rows in suite v1 because the new model's numbers aren't comparable with them. On these tasks, the old h0 rows resolve 21 of 41. Every h0 cell re-run tonight resolved: 23 of 23. Most of the 25 points is the model change, not seeding. **Same-night pairs.** 21 cells were scored for both arms tonight: - **h0:** 21 resolved. - **h14:** 19 resolved. It lost pylint-4604 (seed 0) and sphinx-7590. - **Cost per cell:** $0.013 for h0 against $0.043 for h14 (3.3×). - **Wall time per cell:** 460 s for h0 against 703 s for h14. The keep rule needed h14 to resolve at least 3 more cells than h0. It resolved 2 fewer. The pre-registration also called for h0 on all 57 dev cells, but only seed 0 was re-run. Piper's own caveat at 21:28, that the baseline rows are older runs, was correct. But the same-night rows already in the database answer the question, and they contradict the headline. ## Did the seeded loop actually run? The loop starts only when the first draw fails its verifier test. That happened in 18 of 62 cells: | Why the verifier said "failed" | Loops | Real failure? | |---|---|---| | The test **passed** (exit 0, "1 passed"), but the colourised pytest output parses to zero results, so `passed` is false | 9 (astropy ×8, scikit-learn ×1) | no: parser bug | | A `setuptools_scm` UserWarning in the matplotlib images fails the test whatever the patch | 6 | no: environment | | The patch removed `unique_path`, which `test_conftest.py` imports | 3 (pytest-5840) | **yes** | So the mechanism under test met a genuine failure 3 times. In nine runs, the model was told its passing patch had failed and asked to repair it. In the other 44 cells, h14 is one ordinary draw with a different prompt. **The parser bug.** `testrun.parse_pytest` matches `PASSED`/`FAILED` lines without stripping ANSI colour codes. I confirmed it on the 15:38:48 astropy run: the output ends in `\x1b[32mPASSED\x1b[0m … 1 passed`, and `parse_pytest` returns `{'results': [], 'summary': None}`. Any arm whose VERIFY step parses pytest output in these containers is probably affected too. That may explain why so many VERIFY verdicts read "inconclusive". It's worth checking; I haven't verified it. ## Protocol problems - **The verifier target comes from the gold list.** `repro_target` is always `None`, so the target falls back to the first node id in the task's `PASS_TO_PASS` list. The harness's own `gold_leak_check.py` lists `pass_to_pass` as a gold accessor and forbids node ids in prompts. h14 writes that node id into the prompt's Boundary line. The leak is small, because these are tests that already pass, not the hidden failing ones, but it breaks the rule the picker experiment was built to honour. - **One unrelated guard test is no check on the fix.** It passes for almost any patch. When it fails, the cause is more often the environment than the patch. - **The diagnosis is a stub.** `failure_class` was "unknown" in all 18 loops, so "diagnosis-guided" repair hasn't been tried yet. - **Escalation stops the loop.** Escalation targets "component12", which doesn't exist yet. Fine for now, but say so in reports. ## What would make the next run count 1. **Fix the parser.** Strip ANSI codes in `parse_pytest`, or run pytest with `--color=no`, and add the astropy output as a regression test. 2. **Check the target on the base commit first.** A test that fails before any patch (the matplotlib warning) is environment, not evidence. Skip the loop and record why. 3. **Verify against something the worker could know.** Use the worker's own reproduction, or the repository's tests nearest the edit, which is the picker's leak-checked selector. Use nothing from `PASS_TO_PASS` or `FAIL_TO_PASS`. 4. **Re-run both arms the same night, on the current model, on all 64 cells.** Report resolve rate by attempt, and how many loops started on a genuine failure. 5. **Then compare against the vote (h1) at matched cost.** Sequential retry competes with parallel sampling for the same slot, and the vote is the incumbent.