Summary / 14. Iterative repair / Document
Spec 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 14 — Iterative Repair (the ≈80% fraction) > Derived from `papers/14-iterative-repair.md` — "The Failed Attempt Is the Asset." This document > drives the build of the **iterative-repair stage**: the deterministic-first, evidence-seeded, > minimal-delta **repair loop** that sits immediately AFTER component 13 (error interpretation), which > converts the failed run's complaint into a grounded `Diagnosis`, and BEFORE the verification and > repair-consolidation components, which judge whether a repair is actually good and then bank it. Where > component 13 reads the failed attempt and names its cause, this component takes the diagnosis and > *acts* — converting the diagnosis into a **minimal-delta repair** against the prior candidate, which > is a **partially-finished solution** correct at every site it did not touch, never a from-scratch > re-roll that throws surviving correct code away with the bug. Each round is seeded with the diff, the > failure evidence, and the diagnosis; emits the smallest delta that moves the candidate toward passing; > keeps a failure-signature ledger so recurrence is recognized instead of re-debugged; and > cap-and-escalates so a stuck loop becomes a prompt for a better instrument, never a losing guess. The > guiding rule is the paper's strong rule for Piper, made mechanical: **Never say "that didn't work — > try again"; say exactly what the last patch fixed, what remains wrong, what evidence establishes both, > and what she is not allowed to break.** This spec translates the paper's design into exact primitives, > the exact `RoundRecord`, `FailureSignature`, `MinimalDelta`, and ledger schemas, exact per-stage > input/output/schema/caps, exact L-column caps and firing conditions, the exact attempt cap and the > absolute ceiling, exact A/B instruments and thresholds, exact ablation mechanics, and an exact build > sequence — no further research required. Where the paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible iterative-repair stage** — a deterministic-first, evidence-seeded, minimal-delta repair loop (round seeding + boundary assembly → failure-signature ledger lookup → delta validation + boundary enforcement → ledger write-back → the cap-and-escalate gate, plus the **single weak-LLM minimal-delta-repair-generation call**) that turns a failing candidate, its diff, its component 12 failure evidence, and its component 13 `Diagnosis` into a validated **minimal-delta repair** that the verification component judges. It is distinct from its predecessor in *kind*, not merely in order: component 13 asks the failure *what it is* and emits the cause; repair asks the loop *what minimal change converts the partially-finished candidate into a passing one* and emits that delta. A diagnosis is not yet a fix. When a candidate fails, the pipeline now knows the failing test, the observed evidence behind the verdict (12), and the symptom→root-cause hypothesis that names the cause (13) — but between that knowledge and a repaired candidate there lies a decision: what to change, and how little. This stage performs that decision as **evidence-seeded minimal-delta repair rather than as an LLM asked to regenerate the whole solution**. The verification component judges the delta that passes this loop's deterministic gates. The stage is governed by the paper's seven principles, restated because they decide every design choice below. **P1 Seed every round with the diff + failure evidence + diagnosis** — no round is a fresh generation from the problem statement; the seed is the round record, and a round without it is a re-roll, not a repair. **P2 Emit a minimal delta: repair, not re-roll** — the unit of repair is the smallest set of token/line deltas that moves the candidate toward passing, never a from-scratch regeneration that discards surviving correctness. **P3 State what the last patch fixed and what remains wrong** — each round's output credits the prior candidate's surviving correctness and names the residual gap the next delta must close. **P4 Ground every claim in evidence** — what the last patch fixed and what remains wrong each point at observed evidence from components 11–12, never at bare assertion. **P5 Keep a failure-signature ledger** — every failure is hashed into a signature and the ledger records what each round tried and whether it worked, so recurrence is met with the record of prior attempts rather than re-debugged from zero. **P6 Cap-and-escalate to stop thrash** — the loop has a hard attempt cap; at the cap it escalates to a stronger instrument (more evidence, deeper diagnosis, wider delta scope) rather than a fourth losing guess, and the gate is a counter, never a model's judgment. **P7 Declare and enforce the not-allowed-to-break boundary** — each round declares what it may not break — the passing tests it must keep green, the API contract the task preserves — and the delta is validated against that boundary deterministically before it is believed. **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — round seeding and boundary assembly, failure-signature ledger lookup, delta minimality validation, boundary enforcement (re-running the boundary's passing tests under component 11's execution), ledger write-back, and the cap-and-escalate gate — plus the *bound* of the *one* minimal-delta-repair-generation call. The heavy pieces (a trained minimal-delta repair generator, RePair-style reward-critic iterate-until-no-gain calibration, a persistent cross-repository repair-exemplar retrieval surface beyond the in-process ledger, replay-grade delta verification) are **not** built in Round 1 and are **not excluded** — later-round content climbing toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by the **boundary-validated minimal-delta resolve rate at cap** — never the end-to-end resolve rate alone (§7). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element is classified below by *which round* builds it. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–8 plus the bounded stub of row 9). - **DEFERRED = later rounds** — the trained-generator / reward-critic / persistent-repository-learning / replay-grade-verification pieces (rows 10–13). They are **not excluded** — they are the content of later rounds, gated by the same §7 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | (a) Round seeding + boundary assembly — assemble the failed candidate, the diff of the last patch, the failure evidence (12), and the diagnosis (13) into the `RoundRecord` seed; extract the not-allowed-to-break boundary deterministically from the task spec + passing tests | **BUILD-NOW** | | 2 | (b) Failure-signature ledger lookup — hash the current failure into a `FailureSignature` (failure class, message template, site) and look it up in the ledger; a recurring signature seeds the round with what prior attempts tried and whether they worked | **BUILD-NOW** | | 3 | (c) Delta minimality validation — check the proposed delta touches only the residue the diagnosis and remaining-wrong statement implicate; reject a disguised re-roll that rewrites correct regions | **BUILD-NOW** | | 4 | (c) Boundary enforcement — re-run the boundary's passing tests under component 11's execution; reject any delta that breaks a test it was told to keep green (the regression guard) | **BUILD-NOW** | | 5 | (d) Ledger write-back — record the round's outcome (signature, delta, and once verification returns, whether it resolved) deterministically in the ledger | **BUILD-NOW** | | 6 | (e) Cap-and-escalate gate — the deterministic attempt counter advanced against the attempt cap (default 3) and the absolute ceiling (35); at the cap, escalate to a stronger instrument rather than permitting another attempt at the same evidence level | **BUILD-NOW** | | 7 | Ledger accumulation across rounds and repositories — the same failure-signature ledger is shared, so a signature that recurs in a later repository is met with the record of prior attempts | **BUILD-NOW** | | 8 | A/B harness (§7: boundary-validated minimal-delta resolve rate at cap + per-attempt curve + minimality score + thrash rate + ledger hit-rate + escalation firing rate + ablation runner + failure-mode audits) | **BUILD-NOW** | | 9 | (L) Minimal-delta repair generation — **the single learned entry**, included as a *bounded stub* (cost rule below) | **BUILD-NOW** (bounded stub only) | | 10 | **Trained minimal-delta repair generator at scale** | **DEFERRED** — Round 1 ships only the bounded stub (§4); full training/calibration deferred | | 11 | **RePair-style reward-critic iterate-until-no-gain calibration** | **DEFERRED** — Round 1's cap-and-escalate is a deterministic counter (§3.5); the learned "when to stop" critic is deferred and never replaces the counter | | 12 | **Persistent cross-repository repair-exemplar retrieval** — a retrieval surface beyond the in-process ledger that pulls successful deltas from many prior repositories as few-shot exemplars | **DEFERRED** — Round 1's ledger accumulates in-process across rounds/repos (§3.7) but ships no separate retrieval index | | 13 | **Record/replay (rr-style) delta verification** | **DEFERRED** — Round 1 verifies deltas by re-running under component 11's execution (§3.3); replay-grade determinism is inherited, not built fresh | **The cost rule (explicit).** The paper's single learned entry — minimal-delta repair generation — is the one L-column entry this architecture permits. In Round 1 it is included as a **bounded stub**: the stage exists, wired into the pipeline at its firing point (§4), gated, seeded by the deterministic round record, never auto-promoted — but its full training and calibration (the paper's recursive self-improvement loop, §5) is **deferred**. The stub's *cost* is bounded before invocation: one call, temperature zero, schema-constrained output, a bounded payload, parse-failure fallback that returns control to the deterministic stage with the constraint violation fed back (§4 caps). It is kept **only if** the §7 no-L-vs-L ablation shows it improves the KEY measure on held-out repositories; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the repair path by default — the full deterministic pipeline (rows 1–7) plus the *bounded stub* of the single repair call (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a learned stopping decision, a separate retrieval index, or break the one-entry L column — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out repositories, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Five stages, all **D** through the cap-and-escalate gate. The LLM column holds exactly one thing: minimal-delta repair generation (§4), reached once per round between ledger lookup and delta validation. The pipeline: ``` FAILING CANDIDATE + DIFF OF LAST PATCH + FAILURE EVIDENCE (component 12) + DIAGNOSIS (component 13) + EXISTING TESTS + TASK SPEC → (a) round seeding + boundary assembly [D] → the RoundRecord seed (candidate/diff/evidence/diagnosis/boundary) → (b) failure-signature ledger lookup [D] → signature + ledger recurrence knowledge (or escalation if at cap) → (§4) minimal-delta repair generation [L — the ONE entry; emits the MinimalDelta] → (c) delta validation + boundary enforcement [D] → accept, or reject + feed the violation back → (d) ledger write-back [D] → signature + delta + outcome recorded deterministically → (e) cap-and-escalate gate [D] → advance the counter; cap → escalate, ceiling → out of the loop → to verification (the delta that passed the deterministic gates) ``` The pipeline reasons over **the failed attempt made spendable** — the RoundRecord seed is the only thing the model repairs against, and the delta it emits is validated, boundary-enforced, ledgered, and capped deterministically. The candidate is **repaired, never re-rolled**. All artifacts land in `build/14-iterative-repair/`. Nothing writes into the live harness. ### 3.1 (a) Round seeding + boundary assembly (D) **Input:** the failing candidate, the diff of the last patch, the failure evidence from component 12, the `Diagnosis` from component 13, the existing passing tests, and the task spec. **Output:** the structured **`RoundRecord`** — the seed of the round, the failed attempt made spendable. **No language model runs here.** The round never regenerates from the problem statement; it refines from this record (P1). The seed carries exactly the prior candidate's current state, what was already changed, what failed and the observations behind it, what the diagnosis believes caused it and on what evidence, and what the repair is not allowed to break. **The `RoundRecord` schema (exact — EXACTLY these fields):** ```json { "candidate": "…", // the candidate at its current state (the partially-finished solution) "diff": "…", // the diff of the last patch — what was already changed "failure_evidence": [ "…" ], // typed observations from component 12 of what actually failed "diagnosis": "…", // the component-13 artifact: symptom → root-cause hypothesis → cited evidence → fix direction "boundary": [ "…" ] } // the not-allowed-to-break set (P7) ``` **Field meanings (exact):** `candidate` is the current state of the partially-finished solution; `diff` is the last patch's diff, so the round credits what survived and sees exactly what it already changed (P3); `failure_evidence` holds the **typed observations from component 12** — never raw narration of the run, only the observed evidence; `diagnosis` is the **component-13 artifact** — the symptom → root-cause hypothesis → cited evidence → fix direction chain that names *why* the candidate fails; and `boundary` is the **not-allowed-to-break set extracted deterministically from the task spec + passing tests** — what must stay green (the API contract, the behavior the task preserves), so the model never gets to choose what it is not allowed to break (P7). **Boundary-extraction rule (exact, deterministic):** the boundary is assembled from (1) the passing tests in the task spec's harness that currently pass, and (2) the explicit API/behavioral contracts the request names or preserves. These are collected into the not-allowed-to-break set **before** any repair is generated; the extraction consults no model. The boundary's passing tests are the ones stage (c) re-runs under component 11 to enforce the boundary (§3.3). **Output schema (exact):** ```json { "round_record": { "<the RoundRecord schema above>" }, "round_number": int, // 1..ceiling "seed_complete": true } // candidate, diff, evidence, diagnosis, boundary all present ``` **Caps:** exactly one `RoundRecord` per round, assembled from the deterministic inputs — never a fresh generation from the problem statement. A round whose seed lacks the diff, the evidence, or the boundary is a rejected deliverable (it would be a re-roll, P1/P7). ### 3.2 (b) Failure-signature ledger lookup (D) **Input:** the current failure (its failure class, message, and site) plus the persistent failure-signature ledger. **Output:** the failure's **signature**, looked up in the ledger; on a hit, the **ledger recurrence knowledge** — what prior rounds tried for this signature and whether it worked; on a count at or past the cap, the **escalation trigger**. **No language model runs here.** This is P5 made concrete: the ledger is what turns a repeated failure from a mystery into a lookup, so the harness does not re-debug a bug it has already debugged. The lookup is a **deterministic hash-and-index**; the model never reasons about whether it has tried this before, the ledger answers that. **The `FailureSignature` schema (exact — EXACTLY these fields):** ```json { "failure_class": "syntax | type | null | off-by-one | async-order | env-config | concurrency", "message_template": "…", // the normalized message shape, not the raw text "site": "file:line | symbol" } ``` **Field meanings (exact):** `failure_class` is the component-13 taxonomy label; `message_template` is the normalized message shape (variable parts abstracted) so recurrence is recognized across near-identical messages; `site` is the file/line or symbol the failure is pinned to. The triple is **hashed deterministically into a ledger key**, so identical triples collide to the same key and the ledger indexes by that key. **Ledger lookup procedure (exact, deterministic):** 1. **Hash** the failure's class/message-template/site triple into the ledger key. 2. **Look up** the key; if absent, this is a first-seen signature — no recurrence knowledge, seed from the round record alone. 3. **If present**, return the prior attempts for the signature — each what-was-tried and whether it worked — so the round is seeded with the knowledge that an approach already failed rather than re-trying it blind. 4. **If the round count for this signature is at or past the attempt cap**, the ledger triggers the escalation path rather than permitting another attempt at the same evidence level (§3.5). **Output schema (exact):** ```json { "signature": "<FailureSignature>", "ledger_key": "<SHA-256 hash of the signature triple>", "recurrence": [ { "attempt_number": int, "delta_summary": "…", "worked": bool } ], "first_seen": bool, "escalation_triggered": false } ``` **Caps:** the lookup is pure deterministic hashing and indexing on CPU — no model consults the ledger. A recurring signature returns its prior attempts, bounded to the signature's recorded history; the ledger grows only through stage (d)'s deterministic write-back (§3.4). ### 3.3 (c) Delta validation + boundary enforcement (D) **Input:** the proposed `MinimalDelta` from §4 plus the `RoundRecord` seed's diagnosis and boundary. **Output:** an **accepted delta** (passed validation and boundary enforcement, forwarded to verification) or a **rejection with the constraint violation fed back** to the deterministic stage — never silently accepted as a repair. **No language model runs here.** This is the deterministic judge of the model's one proposal: the model proposes the delta; the harness decides whether the delta is legal, minimal, bounded, and boundary-respecting before it is believed (P2, P4, P7). The model is **not asked to judge its own repair**, only to propose it. **The `MinimalDelta` output schema (exact — EXACTLY these fields):** ```json { "fixed_claim": "…", // exactly what the last patch / this delta fixes "remains_wrong": "…", // exactly what remains wrong, the residual gap the next delta closes "evidence_citations": [ "…" ], // component 11-12 observations grounding fixed_claim AND remains_wrong "boundary_confirmation": "…", // the delta respects the not-allowed-to-break boundary "edits": [ "…" ] } // the smallest set of token/line deltas toward passing ``` **Field meanings (exact):** `fixed_claim` and `remains_wrong` state, respectively, what the delta credits as fixed and what remains wrong (P3) — and **each must carry `evidence_citations` pointing at component 11–12 observations, never bare assertion** (P4): "Fixed X" points at the passing test that now passes; "Y remains wrong" points at the failing test that still fails, the traceback that moved, the value that changed. `boundary_confirmation` states that the delta keeps green what it was told to keep green (P7). `edits` is the **smallest set of token/line deltas that moves the candidate toward passing** — bounded, explicit, and minimal (P2), so downstream stages audit exactly what changed rather than diffing two opaque generations. **Validation rules (exact, deterministic):** 1. **Minimality check** — the delta must touch **only the residue the diagnosis and remaining-wrong statement implicate**. The check rejects any delta touching regions beyond that residue — a disguised re-roll that rewrites correct regions the candidate already got right (P2). A delta that rewrites whole functions or files the diagnosis never implicated is rejected as re-roll-in-disguise. 2. **Boundary enforcement** — the delta is validated against the not-allowed-to-break boundary by **re-running the boundary's passing tests under component 11's execution**: did the delta keep green what it was told to keep green? Any delta that breaks a passing test or an API contract it was told to preserve is **rejected** as a regression-wearing-a-fix — never believed as a repair (P7). 3. **Evidence-plausibility check** — the delta is checked against the failure evidence: does it plausibly address the diagnosed cause rather than a surface symptom? A delta that addresses a symptom the diagnosis did not name as the cause is rejected or routed back for a deeper diagnosis. **Rejection semantics (exact):** a delta that fails minimality, breaks the boundary, or misses the diagnosed cause is **rejected and fed back as a constraint violation** to the deterministic stage — never believed as a repair — and the violation is recorded so the next round is seeded with it (P5). **Output schema (exact):** ```json { "accepted": true, "validated_delta": { "<the MinimalDelta schema above>" }, "minimality": "pass | reject", // touches only the diagnosed residue "boundary": "green | violated", // re-ran the boundary's passing tests under component 11 "plausibility": "pass | reject" } // addresses the diagnosed cause, not a surface symptom ``` **Caps:** exactly one validation verdict per proposed delta; minimality and boundary checks are pure deterministic software on CPU running under component 11's execution for the boundary tests. A rejected delta never reaches verification. ### 3.4 (d) Ledger write-back (D) **Input:** the round's signature, the emitted delta, and — once verification returns — whether it resolved the failure. **Output:** a **ledger record** written deterministically to the failure-signature ledger. **No language model runs here.** This is the mechanism by which the harness learns from every failed attempt, so a failure that recurs in a later round or a later repository is recognized and met with the record of what has already been tried, rather than re-debugged from zero (P5). Ledger accumulation across rounds and repositories is BUILD-NOW (§2 row 7): the same ledger is shared, so cross-repository recurrence is recognized too. **The ledger record schema (exact — EXACTLY these fields):** ```json { "signature": "<FailureSignature>", "attempt_number": int, "delta": "…", // the delta that was emitted this round "outcome": "resolved | not_resolved", "timestamp": "…" } ``` **Field meanings (exact):** `signature` is the round's `FailureSignature` (the ledger key under which the record is indexed); `attempt_number` is the round's attempt counter value; `delta` records what was tried; `outcome` records whether the round resolved the failure (filled once verification returns — the write-back may be finalized after the verdict); `timestamp` ties the record to the exact round. **Procedure (exact, deterministic):** after the round's outcome is known, the record is written under the signature's ledger key. A later round whose signature hashes to the same key will therefore return this record at lookup (§3.2), seeding the recurrence knowledge that this delta was already tried and whether it worked. **Caps:** the write-back is pure deterministic bookkeeping on CPU — no model records its own outcome. The ledger is append-only; a record is never overwritten, only added, so the full history of attempts per signature is preserved. ### 3.5 (e) Cap-and-escalate gate (D) **Input:** the round's outcome and the persistent attempt counter. **Output:** a decision to **continue** (seed the next round), **escalate** (reach the cap → route to a stronger instrument), or **exit the loop** (reach the absolute ceiling → escalate out of the repair loop entirely). **No language model runs here.** This is P6 made mechanical: the gate is a **counter, not a model's judgment about whether to keep going**, so thrash is impossible by construction. **The caps (exact):** - **Attempt cap default 3** — escalate on reaching it. The art-of-repair result that gains concentrate in the first ~3 iterations and then saturate [11] is the warrant: past the cap, another attempt at the same evidence level is thrash, not repair. Reaching the cap **escalates deterministically** — to component 12 for more evidence, to component 13 for a deeper diagnosis, or to a wider delta scope once minimal deltas at the obvious site have failed — and the escalated round is **seeded afresh** rather than repeating the exhausted one. - **Absolute ceiling 35** — the DynaFix empirical bound [3]: a bug unresolved at 35 attempts **escalates out of the loop entirely**, never thrashes further. DynaFix reached 38 previously-unfixed bugs within ≤35 attempts, so a bounded loop is a disciplined one, not a crippled one. Past the ceiling the harness does not keep trying at any scope; it records the failure in the ledger and hands the bug to a stronger instrument. **Procedure (exact, deterministic):** 1. **Advance** the attempt counter for the current signature by one. 2. **Compare** the counter against the attempt cap (default 3). If reached without resolution, **escalate**: route to component 12 for more evidence, or component 13 for a deeper diagnosis, or widen the delta scope; seed the escalated round afresh. 3. **Compare** the counter against the absolute ceiling (35). If reached without resolution, **exit the loop** — escalate out of the repair loop entirely rather than permitting further attempts at any level. **Output schema (exact):** ```json { "attempt_number": int, "decision": "continue | escalate | exit", "escalation_target": "component12 | component13 | wider_delta | out_of_loop", "at_ceiling": false } ``` **Caps:** the gate is pure deterministic arithmetic on CPU — the attempt counter, the attempt cap (3), and the absolute ceiling (35) are constants, never model judgments — a stuck loop always routes to a different instrument rather than repeating the same losing guess. **The L column holds exactly one entry.** Stages (a)–(e) are all deterministic software on CPU. The single learned node is stage (§4), minimal-delta repair generation, reached once per round between ledger lookup and delta validation. Any proposed addition that would put a language model elsewhere in the pipeline — round seeding, ledger lookup, delta validation, boundary enforcement, ledger write-back, the cap-and-escalate gate, or letting the model judge its own repair or choose when to stop — is a regression against P1–P7 and the L-column invariant, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of §7, on held-out repositories, at a defensible cost, or be rejected (paper §4). --- ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **`repair_minimally`** — minimal-delta repair generation. The weak model, seeded with the `RoundRecord` and the ledger's recurrence knowledge, emits the `MinimalDelta` — the smallest set of edits that moves the candidate toward passing, and nothing more (P2). Its output opens by crediting the prior candidate — exactly what the last patch fixed — then states exactly what remains wrong and what evidence establishes both, and closes by confirming the delta respects the not-allowed-to-break boundary (P3, P4, P7). This is **ONE call**, in which the model decides what to change next; every other judgment about whether the change is legal, minimal, bounded, or recorded is deterministic (paper §4 stage L). The model is **not asked to judge its own repair**, only to propose it — validation is deterministic (§3.3). **Prompt inputs (assembled deterministically, in this exact order):** 1. **`round_record`** — the `RoundRecord` seed from stage (a): the candidate, the diff of the last patch, the typed failure evidence (12), the diagnosis (13) with its cited evidence and fix direction, and the not-allowed-to-break boundary. 2. **`recurrence`** — the ledger recurrence knowledge from stage (b): what prior attempts for this signature tried and whether they worked, so the round is seeded with the knowledge that an approach already failed rather than re-trying it blind. 3. **`constraint_violations`** — any rejection fed back from stage (c) in a prior round, so a delta that already broke the boundary or re-rolled correct code is not proposed again. **Input → output schema (exact):** ```json { "fixed_claim": "…", // exactly what the last patch fixed (credits surviving correctness) "remains_wrong": "…", // exactly what remains wrong — the residual gap this delta closes "evidence_citations": [ "…" ], // component 11-12 observations grounding fixed_claim AND remains_wrong "boundary_confirmation": "…", // confirms the delta respects the not-allowed-to-break boundary "edits": [ "…" ] } // the smallest set of token/line deltas toward passing ``` **Caps:** **ONE call**, `temperature = 0`, `max_completion_tokens = 2048`; a bounded payload — the `edits` list is bounded to the residue the diagnosis implicates, and the whole output is schema-constrained. The output is the `MinimalDelta`, whose fields stage (c) validates deterministically (§3.3). The harness, not the model, decides what the delta means. **Gating (exact):** the model **proposes only** — it emits the delta it believes moves the candidate toward passing; validation is deterministic. The model does **not** judge whether its own delta is good, minimal, or boundary-respecting — those are stage (c)'s checks (§3.3), and a proposal that fails them is **rejected and fed back as a constraint violation**, never believed on the model's say-so (§8 invariant 3). **Parse-failure fallback (exact):** any JSON parse failure, schema violation, out-of-bound output (edits beyond the diagnosed residue, an oversized payload), or empty/malformed output is **not silently retried as a fresh re-roll**: the stage **returns to the deterministic stage with the constraint violation fed back** — the malformed proposal is recorded as a violation and the deterministic path seeds the next consideration from it — **never a re-prompt, never a second call at the same seed**. The L column is reached **once** per round. This is the mechanical enforcement of the one-entry L column and the anti-over-prompting rule. **Re-invocation on a later round (exact):** repair generation is re-considered only when a *new* round is seeded (a fresh attempt after escalation, §3.5) and is scoped strictly to that round's `RoundRecord` + recurrence; it re-uses none of a prior firing's `edits` except as ledger recurrence knowledge. It remains the **single conceptual learned entry** — a bounded, once-per-round call — **never a model that runs over the whole pipeline**. **Invariant 1 (§8): the L column contains exactly one entry — and that entry is only ever a bounded, seeded minimal-delta repair generator whose output the deterministic stages validate.** --- ## 5. Implementation Order Seven phases, each independently testable and reversible, landing in `build/14-iterative-repair/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Round seeding (a)** | `seed.py` (§3.1 assembly of the `RoundRecord` from candidate + diff + failure evidence (12) + diagnosis (13) + deterministically-extracted boundary from the task spec + passing tests) | Feed a candidate + diff + evidence + diagnosis + task spec + passing tests: the `RoundRecord` has exactly the five fields; `boundary` is extracted deterministically (never by a model); a seed missing diff/evidence/boundary is rejected; assert **no LLM consulted**. | Delete `seed.py`. | | **2. Ledger lookup (b)** | `ledger.py` (§3.2 hashing the failure's class/message-template/site triple into the ledger key + lookup returning recurrence knowledge or escalation) | Feed signatures: identical triples hash to the same ledger key; a first-seen signature returns `first_seen: true`; a recurring signature returns prior attempts with whether they worked; a count at/past the cap triggers escalation; assert **no LLM consulted**. | Delete `ledger.py`. | | **3. Repair-stub (L)** | `repair.py` (§4 the bounded L stub: firing + ONE call + temperature 0 + seeded with `round_record` + recurrence + violations + schema-constrained `MinimalDelta` output + parse-failure fallback that returns to the deterministic stage with the violation fed back) | With a mocked LLM: the call is **one**, temperature 0; output has exactly the `MinimalDelta` fields with `fixed_claim` and `remains_wrong` each evidence-cited; a malformed/over-cap response → returns to the deterministic stage with the violation recorded, **no second call**; the output feeds stage (c) deterministically. | Delete `repair.py`. | | **4. Delta validation + boundary enforcement (c)** | `validate.py` (§3.3 the minimality check — touches only the diagnosed residue, rejects a disguised re-roll — + boundary enforcement re-running the boundary's passing tests under component 11 + evidence-plausibility; rejection feeds the violation back) | `validate.py` on fixtures: a minimal delta touching only the diagnosed residue passes; a delta rewriting correct regions is rejected as re-roll; a delta breaking a boundary passing test (under component 11 execution) is rejected as regression; the rejection is recorded as a constraint violation; assert **no LLM consulted**. | Delete `validate.py`. | | **5. Ledger write-back (d)** | `record.py` (§3.4 writing the `{ signature, attempt_number, delta, outcome, timestamp }` ledger record deterministically; append-only) | Feed a round's outcome: the record is written under the signature's ledger key with exactly the five fields; the ledger is append-only (a prior record is never overwritten); the write-back consults no model; assert **no LLM consulted**. | Delete `record.py`. | | **6. Cap-and-escalate gate (e)** | `gate.py` (§3.5 the attempt counter advanced against the attempt cap default 3 and the absolute ceiling 35; at the cap escalate to component 12/13/wider delta with a fresh seed; at the ceiling exit the loop entirely) | Feed a sequence of unresolved rounds: the counter advances; reaching attempt 3 escalates (target named); reaching attempt 35 exits the loop (never thrashes further); the caps are constants, never model judgments; assert **no LLM consulted**. | Delete `gate.py`. | | **7. Loop driver + A/B harness** | `loop.py` (§3 the round loop wiring (a)→(b)→L→(c)→(d)→(e) with the ledged, capped sequence), `ab.py` (§7 KEY-measure + secondary-instrument capture + ablation runner + no-L-vs-L runner + decision) | `loop.py` on a failing fixture with a mocked repair stub: the round sequence runs seed→lookup→repair→validate→record→gate correctly; a cap-triggered round escalates and is seeded afresh. `ab.py` runs the §7 pipeline-vs-baseline and the §7 ablations, emitting the §7 decision on held-out fixtures. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 7) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. The loop driver reuses component 11's execution and component 13's diagnosis wholesale; it invents no new execution substrate. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of component 11's execution and component 12's instrumentation/evidence and component 13's diagnosis)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `hashlib` | §3.2 deterministic hashing of the `FailureSignature` triple into the ledger key; §3.4 artifact/provenance hashing | CPython stdlib. | **No** | | stdlib `re` | §3.2 normalizing message text into the `message_template` shape | CPython stdlib. | **No** | | stdlib `json`, `os`, `sys`, `time` | §3 schemas, ledger persistence, provenance timestamps | CPython stdlib. | **No** | | stdlib `sqlite3` | §3.2/§3.4 the append-only failure-signature ledger store | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3.3 computing token/line deltas; the candidate runtime | In-repo, pinned interpreter. | **No** | | Component 11 execution substrate | §3.3 boundary enforcement re-runs the boundary's passing tests under component 11; §3.5 verification of each round's outcome | Built by spec 11; reused, not duplicated. | **No** | | Component 12 instrumentation + evidence pack | §3.1 the `failure_evidence` in the seed; §3.3 the evidence-plausibility check; §3.5 the escalation target for more evidence | Built by spec 12; reused, not duplicated. | **No** | | Component 13 diagnosis artifact | §3.1 the `diagnosis` in the seed; §3.3 the fix-direction the delta must address; §3.5 the escalation target for a deeper diagnosis | Built by spec 13; reused, not duplicated. | **No** | | Component 01 tool layer / existing test runner | §3.1 the passing tests the boundary is extracted from; §3.3 the boundary's tests re-run under component 11 | In-repo, already vetted. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single minimal-delta-repair call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: seeding, ledgering, delta validation, boundary enforcement, and the cap-and-escalate gate use stdlib `hashlib`/`re`/`json`/`sqlite3`; execution, instrumentation, and diagnosis are **reused wholesale from components 11, 12, and 13**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The four DEFERRED rows (§2 rows 10–13) are a trained generator on known architectures, a learned stopping critic, a separate retrieval index over an existing ledger, or replay-grade determinism on the existing substrate — not new third-party Python installs. Any future trained model download, reward-critic training run, cross-repository retrieval index dependency, or replay substrate must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — and is out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, and 13** (Python-only, single-repo, no-Docker reproduction) as the source of a held-out **(failing candidate, diff, component-12 evidence pack, component-13 diagnosis, known fix delta, boundary) corpus**: for each instance, a failing candidate with its diff, evidence, and diagnosis, the mechanically-known **minimal fix delta** the repair had to produce, and the declared not-allowed-to-break boundary. Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. **Baseline (concrete).** The **re-roll-from-scratch** baseline — the naïve path the paper and this component reject: on each failure the model is handed the original problem statement and asked to regenerate a fresh candidate, with **no diff seeding, no evidence, no diagnosis, no boundary, no ledger, no cap** (an unbounded regenerate-until-pass loop). The structured loop must beat *this* on the honest measures below, and the ablation (7.3) removes loop mechanisms against those same honest measures. ### 7.1 (a) Boundary-validated minimal-delta resolve rate at cap — the KEY measure The primary instrument, the direct score of the whole repair loop: **of failures attempted, the fraction resolved within cap by a delta that (a) resolves the failing test(s) under component 11 execution, (b) keeps the boundary green, and (c) is minimal (not a disguised re-roll)**. This is the honest measure of the loop, because it refuses to reward a repair that trades the target failure for a newly broken test, or that regenerates broadly and throws away the surviving correct code. **State explicitly: "Never the end-to-end resolve rate alone"** — a lucky patch must not mask a badly-shaped loop. A candidate that resolves the failing test but breaks a passing one it was told to keep green is **not** a resolved repair; a "repair" that rewrites whole correct regions is **not** a resolved repair. **This is the KEY measure — the structured loop is adopted only if it STRICTLY beats the re-roll baseline on this measure AND does not regress resolve (7.2); a resolve win that does not beat the baseline on boundary-validated minimal-delta resolve at cap is not credited.** ### 7.2 The end-to-end regression guard + the secondary instruments The coarse end-to-end instrument is kept for parity — **deliberately not** the gate for repair-loop changes, because resolve rate alone cannot separate loop quality from the shape of the loop: 1. **Resolve rate** = `resolved / total` (spec 01's score) — the no-regression guard, never the primary gate. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). 3. **Tokens** = `tokens / instance`. **Secondary instruments (report but do not gate on)** — the loop-shape measures from the paper's §6 that reveal *why* the loop resolves or thrashes, reported alongside the KEY measure but never replacing it: 1. **Per-attempt resolve curve** — the fraction of failures resolved by attempt `n` or earlier; reveals where the cap belongs (resolution that keeps climbing past the cap = under-capped; flatline = budget spent on thrash, echoing the art-of-repair saturation in the first ~3 iterations [11]). 2. **Delta-minimality score** — the fraction of the candidate the delta rewrote and the number of regions touched beyond the diagnosed residue; catches the silent abandonment of P2 (a loop rewriting whole functions is re-rolling, not repairing) [3][6]. 3. **Thrash rate** — attempts that repeat a prior losing guess at the same evidence level, detected via the ledger; the feedback-friction result that models under-absorb even near-perfect feedback [7] is the warrant a loop can repeat without learning. 4. **Ledger hit-rate** — of failures that recurred across rounds or repositories, how many were met with the record of a prior attempt rather than re-debugged from zero — the cumulative measure of the thesis. 5. **Escalation firing rate** — how often the cap is reached and a stronger instrument is invoked; a healthy rate means the cap is catching stuck loops and routing them to better instruments. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add a repair-loop mechanism because it sounds sensible" mechanically (paper §5, §6). Two passes, everything else held fixed, the delta measured on the honest measures (boundary-validated minimal-delta resolve rate at cap, plus resolve/cost/tokens and the secondary instruments): 1. **Remove the seeding stages** — feed only the failing test's raw message (no diff, no evidence, no diagnosis, no boundary) instead of the `RoundRecord` seed, and measure the KEY-measure drop. This is the direct test of P1/P4: does the seed measurably improve the resolve-at-cap, or is it decoration? 2. **Remove the boundary enforcement** — accept deltas without re-running the boundary's passing tests, and measure the regression-wearing-a-fix rate. This is the direct test of P7: does the guard stop the loop from trading the target failure for a newly broken test? 3. **Remove the ledger** — never look up or write back signatures, so every recurrence is re-debugged from zero, and measure the thrash-rate and resolve drop. This is the direct test of P5. 4. **Remove the cap-and-escalate gate** — let the loop run unbounded, and measure thrash rate and spend. This is the direct test of P6. 5. **Remove the minimality check** — accept any delta, including whole-function rewrites, and measure the delta-minimality score. This is the direct test of P2. 6. **Ablate the single L repair step** — run **no-L vs L** (never generate a repair vs. generate one) and measure the **KEY measure** on held-out repos. This is the test of whether the bounded repair stub (§4) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L repair stub, which must survive the no-L-vs-L ablation on the KEY measure to be kept at all), then to any other mechanism. **Never add a repair-loop mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run loop-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured loop **wins iff**: ``` Boundary-validated minimal-delta resolve rate at cap STRICTLY beats the re-roll baseline (the KEY measure) AND resolve_pipeline ≥ resolve_baseline (no end-to-end regression) AND cost_pipeline ≤ 1.5 × cost_baseline AND tokens_pipeline ≤ 1.5 × tokens_baseline ``` The KEY measure must **strictly beat the re-roll baseline**; resolve must not regress; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — a candidate that improves resolve but leaves boundary-validated minimal-delta resolve at cap flat against the baseline is **not adopted** (a lucky patch would be masking a badly-shaped loop). If the structured loop wins, it replaces the re-roll path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways the repair loop can lie) Beyond the positive instruments, the harness audits the specific ways a repair loop can lie, each with a mechanical guard (paper §6): 1. **Re-roll-in-disguise** — a "repair" regenerates broadly and rewrites correct regions; guarded by the minimality check asserting the delta touches only the residue the diagnosis implicates (§3.3). 2. **Fix-the-surface** — the delta addresses the symptom but not the diagnosed cause; guarded by the evidence-plausibility check and the per-attempt curve (a shallow fix recurs next round). 3. **Regression-wearing-a-fix** — a delta resolves the target by breaking the boundary; guarded by the boundary-enforcement audit asserting the boundary's passing tests stayed green under component 11. 4. **Silent-repeat** — the loop emits the same losing guess it emitted last round; guarded by the ledger and the thrash-rate instrument. 5. **Infinite-refine** — the loop never escalates and spins against the same wall; guarded by the cap-and-escalate gate (§3.5). 6. **No-credit** — the round discards the prior candidate's surviving correctness and re-rolls it; guarded by the minimality instrument and the seed's fixed-claim audit. 7. **No-boundary** — the round repairs without declaring what it is not allowed to break, so a regression is indistinguishable from a fix; guarded by the seed-completeness audit asserting `boundary` is present in every `RoundRecord` (§3.1). ### 7.6 Two honest caveats Two caveats close the section, inherited from the paper's §6. **First, the strong-model-and-scale caveat:** the headline numbers of the paper's review — REFINE's +14.67pp to 51.67% on SWE-bench Lite from iterative refinement [1], DynaFix's ~10% over SOTA with 38 previously-unfixed bugs inside 35 attempts [3], TokenRepair's 88 of 139 [6], Golubev et al.'s 11→39% multi-turn training [12] — were obtained with models far stronger than this harness's target; a weak-model harness should expect materially lower absolute resolve rates. Its repair loop exists precisely so that weak repair *judgment* is compensated by strong deterministic repair *discipline* — seeding, validation, ledgering, and cap-and-escalate — and the honest instruments here, above all the boundary-validated minimal-delta resolve rate at cap, are what will show whether that compensation is real. **Second, the held-out and memorization caution** inherited from the earlier components: all repair metrics must be reported on held-out and out-of-benchmark repositories, so a loop that has memorized a repository's bugs cannot score well without its deltas having been validated and its failures ledgered against evidence the harness actually ran. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/14-iterative-repair/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `REPAIR=loop14|roll`, default `roll`) routes the A/B; the flag alone switches between the new structured minimal-delta loop and the re-roll-from-scratch baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. - The failure-signature ledger and every round record are append-only and versioned; rolling the build back never loses the ledger, which the reverted path can still consult. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains exactly one entry** — minimal-delta repair generation (§4), the one bounded, seeded call that emits the `MinimalDelta`, reached once per round between ledger lookup and delta validation, never re-prompted on parse failure (the fallback returns to the deterministic stage with the violation fed back). Any phase that introduces a language model elsewhere in the pipeline (seeding, ledger lookup, delta validation, boundary enforcement, ledger write-back, the cap-and-escalate gate, judging its own repair, choosing when to stop) is rejected outright. An **empty** L column (the repair stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **No re-roll — repair is always a delta against the prior candidate.** The unit of repair is the smallest set of token/line deltas that moves the partially-finished candidate toward passing, never a from-scratch regeneration that throws the surviving correct code away with the bug. A round that regenerates from the problem statement instead of refining from the `RoundRecord` seed is a rejected deliverable (P2). 3. **Every fixed/remains claim is evidence-cited.** `fixed_claim` and `remains_wrong` each carry `evidence_citations` pointing at component 11–12 observations — never bare assertion. A round that asserts a fix or a residual gap it cannot support with observed evidence is rejected by the ledger and the gate (P4, mirroring component 13's evidence-citation gate). 4. **Boundary enforced deterministically.** Every repair is validated against the not-allowed-to-break boundary by re-running the boundary's passing tests under component 11's execution; a delta that breaks a test it was told to keep green is rejected, never believed (P7). A repair without a declared boundary is a rejected deliverable. 5. **Every failure signature is ledgered.** Every failure the loop encounters is hashed into a signature and recorded in the ledger with what was tried and whether it worked; a recurrence is met with that record, not re-debugged from zero (P5). A failure the loop resolves without a ledger write-back is a rejected deliverable. 6. **Cap-and-escalate is a counter, not a model judgment.** The attempt cap (default 3) escalates to a stronger instrument; the absolute ceiling (35) escalates out of the loop entirely. Both are deterministic constants; the model never decides whether to keep going. Thrash is impossible by construction; a loop that repeats a losing guess past the cap is a rejected deliverable (P6). 7. **All metrics on held-out repositories.** The KEY measure and every secondary instrument are reported on held-out and out-of-benchmark repositories, never on tuning or in-benchmark data, so a loop that has memorized a repository's bugs cannot score well without its deltas having been validated and its failures ledgered against evidence the harness actually ran. 8. **A delta that breaks the boundary is rejected before it is believed.** No accepted-repair bookkeeping, no verification hand-off, and no "success" credit precedes the deterministic boundary-enforcement re-run — a regression-wearing-a-fix never reaches verification, the ledger's resolved-outcome, or the resolve rate. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*