Summary / 19. Regression protection / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 19 — Regression Protection (the reversible, differential, impact-driven preservation stage) > Derived from `papers/19-regression-protection.md` — "The Unchanged Must Be Proved, Not Assumed: Differential, Impact-Driven, Preservation-Oriented Regression Protection for a Weak-Model Coding Harness." > This document drives the build of the **regression-protection stage**: the reversible, differential, impact-driven > preservation pipeline that sits *after* the patch is produced and *before* the acceptance verdict (component 18) is > trusted — the stage that answers the one question every bug fix poses silently: **did everything the fix did not > intend to change provably stay the same?** It sits at the point where the repair loop (14) has produced a candidate > patch, reflection (15) has extracted its meaning, and candidate generation (16) has formed the competing hypotheses; > this component proves — or fails to prove — that the candidate did not trade the fix for the surrounding behavior. > The sharp distinction is between an **ASSUMPTION** and a **PROOF**: an assumption is "the full suite is green, so > nothing broke" — a boolean that flatters the loop while its net never covered the blast radius; a proof is a > `PreservationLedger` that names every preservation obligation derived from impact analysis, attaches evidence or > records it as untested, logs every skipped test as *unrun* with its selection reason, and reports a > `regression_status` of PROTECTED, DIVERGENT, or UNPROVEN with the gaps in plain sight. Naive harnesses run the full > suite and read green; this stage pins the unchanged behavior *before* the edit (characterization tests plus mined > invariants — the pre-patch program as a recorded executable spec), derives preservation obligations from the impact > graph (component 06), selects the smallest *safe* regression set by checksum-based reachability with every skip > logged as unrun, executes the changed region differentially against the known-good pre-patch baseline, gates the > surface that no test can see (types, semver, API diff), and renders every observed behavioral change as an > intentional-versus-accidental manifest that routes to review, never to auto-acceptance. The output of component 19 > is **not a green light and not a patch** — it is a **PRESERVATION LEDGER**: a structured verdict naming exactly > which unchanged behaviors it proved, which it could not, and where the patch diverged from the baseline. Every node > — the behavior net, the obligation derivation, the selection, the differential execution, the surface gates, the > manifest, and the ledger itself — is deterministic software on CPU; the weak model's AT MOST ONE contribution is a > single learned step of INTENT INTERPRETATION (read each observed divergence against the change's stated intent and > propose whether it is the fix or collateral regression), and it never authors the preservation verdict. The guiding > rule, made mechanical: **Do not ask "Did the tests still pass?" Ask "Which unchanged behaviors did we prove, which > did we skip, and where did the patch diverge from the baseline it did not mean to touch?"** This spec pins the > paper's design to exact primitives, the exact `PreservationLedger` schema, the exact pre-edit net, the exact > obligation derivation, the exact selection-with-skip-log, the exact differential classification, the exact surface > gates, the exact L-column firing, exact A/B instruments and thresholds, exact ablation mechanics, and an exact build > sequence. Where the paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible, differential, impact-driven regression-protection stage** — a pipeline that, given a candidate patch (or candidate repository state), the pre-patch baseline, the repository graph and impact analysis from component 06, and the dynamic coverage from component 12, produces a **PreservationLedger** certifying which unchanged behaviors the patch provably preserved. The stage has six deterministic tiers and one learned entry, in the paper's order: **pre-edit behavior nets** (auto-generated characterization tests over the blast radius plus mined invariants where instrumentation is cheap), **preservation obligations** derived from impact analysis, **change-scoped selection** (safe first, small second, every skip logged as *unrun*), **differential execution** against the pre-patch baseline, **surface gates** (types/API compatibility), and the **characterization-diff review manifest** (intentional versus accidental). It differs from its predecessors in *kind*, not just position: test execution (17) asks *"which tests passed, what property did each prove?"* and certifies the **EVIDENCE**; regression protection asks *"did everything the fix did not intend to change stay the same?"* and certifies the **PRESERVATION** — without which a green suite is the most expensive lie the harness tells, in its cheapest-looking form. A patch that fixes the bug while silently breaking a neighbor, a serializer, or a public API consumer is exactly the failure this stage exists to surface. The paper's eight principles decide every design choice. **P1 The unchanged behavior is the specification, and it is pinned before the edit** — before editing, the harness generates characterization tests over the blast radius and mines invariants where instrumentation is cheap, producing an executable, named spec of everything that must not change (Feathers's cover-and-modify workflow fused with Daikon's dynamic invariant detection); the pre-edit net pre-dates the patch, which is what makes it trustworthy. **P2 Preservation is proven by obligations, not by whatever tests exist** — the harness derives preservation obligations from impact analysis and proves every identified one, rather than running whatever tests happen to exist; the regression surface is proportional to the diff's blast radius, not to the bug's. **P3 Select the smallest safe regression set, and log skips as unrun** — checksum-based reachability (Ekstazi) is the floor, risk-ranked top-slicing (Predictive Test Selection) is the budget policy when the safe set exceeds what the loop can afford; safety is the first property and smallness the second; every unselected test is recorded as *unrun* with its reason, and there is no silent assume-pass, ever. **P4 The pre-patch program is the oracle for unchanged behavior** — preservation is a comparison against a known-good baseline, not an absolute pass/fail; the differential tier executes the changed region under the pre- and post-patch programs side by side and flags divergence in return values, raised exceptions, execution paths, or mined invariants. **P5 Distinguish the two overfitting directions** — a divergence that makes the bug test pass while a neighbor fails is *regression introduction*; a patch that passes everything but still fails the bug on inputs B and C is *incomplete fix*; regression protection targets the first, and the differential tier classifies the two deterministically. **P6 Gate what tests cannot see** — type checks, builds, and API/semver compatibility run alongside the behavioral tiers, because duck-typed call sites, string-keyed contracts, and public API consumers outside the repository are invisible to any suite. **P7 The characterization diff is a manifest for review, not auto-acceptance** — post-edit, every behavioral change is presented as an intentional-versus-accidental manifest; intentional entries update the pinned tests *through review*, never by the repair loop's own hand, because pinned bugs are protected too. **P8 The L column holds at most one entry: intent interpretation** — selection, differential execution, invariant mining, surface gating, and the ledger are deterministic software on CPU; the single learned entry is intent interpretation, judging which observed divergences are the intended fix versus collateral regression, and it is downstream of the deterministic divergence (a fact), never the verdict on preservation. The stage lives in `build/19-regression-protection/`, writes nothing into the live harness, and is kept **only if** it strictly beats the naive "run full suite, green = safe" baseline on the honest measures of §7 — above all the **regression-detection rate** and the **false-alarm rate** — never the end-to-end resolve rate alone and never model self-assessment. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–9); **DEFERRED = later rounds** (rows 10–13), **not excluded** but gated by the same §7 A/B protocol. Every later round of this component still builds within the same `build/19-regression-protection/` tree. | # | Component (paper §4) | Classification | |---|---|---| | 1 | Pre-edit behavior net — `pin_behavior_net` generates characterization tests over the blast radius and mines invariants where instrumentation is cheap (Feathers [15], Daikon [14]) | **BUILD-NOW** | | 2 | Preservation-obligation derivation — `derive_preservation_obligations` turns the impact graph into named obligations with evidence sources (P2) | **BUILD-NOW** | | 3 | Ekstazi-style selection — `select_regression_set` computes the smallest safe set by checksum-based reachability, logs every skip as *unrun* with its reason (P3) | **BUILD-NOW** | | 4 | Differential execution — `run_differential` executes the changed region under baseline and candidate side by side and classifies each divergence (P4/P5) | **BUILD-NOW** | | 5 | Surface gates — `surface_gates` runs type checks, build, and API/semver compatibility on the changed surface (P6) | **BUILD-NOW** | | 6 | Characterization-diff manifest — `compute_characterization_diff` renders every post-edit behavioral change as intentional-versus-accidental (P7) | **BUILD-NOW** | | 7 | (L) Intent interpretation — `interpret_intent` reads each divergence against the stated intent and proposes fix-vs-regression (Testora [6], PatchGuru [7]) | **BUILD-NOW** (bounded stub only) | | 8 | PreservationLedger — `maintain_preservation_ledger` assembles obligations, evidence, differences, untested obligations, and regression_status (P2/P8) | **BUILD-NOW** | | 9 | Pipeline driver + A/B harness (§7 KEY + companion + secondaries + ablation runner + decision rule) | **BUILD-NOW** | | 10 | **Risk-ranked predictive selection** — learned per-test failure-probability top-slicing (Predictive Test Selection [13]) | **DEFERRED** — Round 1 ships Ekstazi-style reachability as the floor; a trained risk model that ranks the safe set by predicted failure is deferred | | 11 | **Full property/fuzz augmentation** — QuickCheck-style property probes and mutation-guided test generation to widen the net (QuickCheck [16], CodaMosa [19], Probe to Generate [18]) | **DEFERRED** — Round 1 ships sampled characterization + mined invariants; generation over the whole input domain is deferred | | 12 | **Formal equivalence** — P³ product-program reasoning and SymDiff-style semantic diff over the untouched input space (P³ [8], SymDiff [26], Godlin & Strichman [27]) | **DEFERRED** — Round 1 ships sampled differential execution; *proving* equivalence is deferred to a later round as the aspirational ceiling | | 13 | **Cross-task regression memory** — a durable store of which behaviors regressed across tasks, feeding selection and risk ranking | **DEFERRED** — Round 1 ships per-patch ledgers; a cross-task regression corpus is deferred | **The cost rule (explicit).** The single learned entry — intent interpretation — is the AT MOST ONE L-column entry this architecture permits. In Round 1 it is a **bounded stub**: wired in at its firing point (§4), seeded, schema-constrained, never auto-promoted; its full training and calibration is **deferred**. Its cost is bounded before invocation: one call per protection run, temperature zero, schema-constrained, bounded payload, parse-failure fallback that returns to the deterministic stage with the violation fed back (§4 caps). It is kept **only if** the §7 no-L-vs-L ablation shows it improves the KEY measures on held-out tasks; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the net, the obligation derivation, the selection, the differential execution, the surface gates, the manifest, or the ledger path by default — the full deterministic pipeline (rows 1–6, 8–9) plus the *bounded stub* of the single intent call (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a learned selection/equivalence judgment, a full training run, or break the at-most-one L-column invariant — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out tasks, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Six deterministic services plus the single learned entry (L). The L column holds AT MOST ONE thing: intent interpretation (§4), reached only after the deterministic divergence has been computed. A pure protection run of a candidate patch has exactly ONE L entry — the intent-interpretation call — and everything else is deterministic software on CPU. The invariant that governs the whole: **the L column contains at most one entry, and the deterministic preservation machinery is the default.** ``` CANDIDATE PATCH (C) + PRE-PATCH BASELINE (B) + REPO GRAPH + IMPACT (06) + COVERAGE (12) → pin_behavior_net [D] → pre-edit characterization tests + mined invariants (P1) → derive_preservation_obligations [D] → named obligations from the impact graph (P2) → select_regression_set [D] → smallest safe set; every skip logged as unrun (P3) → run_differential [D] → baseline vs candidate, side by side; classify divergences (P4/P5) → surface_gates [D] → type check, build, semver/API diff (P6) → compute_characterization_diff [D] → intentional-vs-accidental manifest (P7) → (L) interpret_intent [L] → the at-most-one learned entry (P8) → maintain_preservation_ledger [D] → PROTECTED | DIVERGENT | UNPROVEN + untested obligations ``` The pipeline reasons over **the candidate patch made into a preservation verdict** — the pre-patch baseline, the impact graph, the coverage, and the pinned net are the only things it runs against, and every result it produces is pinned, derived, selected, differentially compared, surface-gated, and manifested deterministically. The model never decides whether a divergence is a regression; it judges, against the change's stated intent, which divergences are the fix and which are collateral, and that judgment is consumed as a hypothesis attached to the ledger, never as a verdict. All artifacts land in `build/19-regression-protection/`. ### 3.1 The preservation contract: `PreservationLedger` (D) **Input:** the obligations, evidence, differential divergences, and untested obligations assembled by the pipeline. **Output:** a `PreservationLedger` with `regression_status` one of PROTECTED, DIVERGENT, or UNPROVEN. **No language model runs here — the ledger is written by deterministic software only; the sole learned input is the intent annotation on `unexpected_differences`, and it is an annotation, never the verdict.** The ledger is the row's contract, and its shape makes the discipline visible: ``` PreservationLedger { patch_id obligations: [ // impact-derived, P2 { behavior, evidence: [test, baseline, candidate], impact_source: "SessionConfig -> SessionManager" } ] unexpected_differences: [] // differential tier, P4/P5 untested_obligations: [] // obligations with no evidence yet, P2 regression_status: PROTECTED | DIVERGENT | UNPROVEN } ``` `patch_id` identifies the candidate under protection. `obligations` carries every impact-derived preservation obligation, each with the `behavior` it names (the unchanged contract), its `evidence` (the test plus baseline and candidate results that discharged it), and its `impact_source` (the dependency edge from which it was derived). A non-empty `untested_obligations` means the patch is *partially evidenced*, not *fully protected* — the acceptance stage (component 18) must see that distinction. `unexpected_differences` collects every differential divergence; each entry is annotated by the single learned step (§4) but its existence is a deterministic fact. `regression_status` is computed deterministically from the evidence and the untested obligations: ``` PROTECTED — every derived obligation has evidence, and no unexpected difference was marked accidental regression DIVERGENT — at least one unexpected difference survives as a collateral-regression annotation UNPROVEN — untested_obligations is non-empty, or a derived obligation received no evidence ``` **Caps:** ledger assembly is pure deterministic software on CPU — the model never writes a field, never authors the `regression_status`, and never upgrades UNPROVEN to PROTECTED by fiat; the three-valued status is computed by table lookup over the evidence and the untested obligations. ### 3.2 Pre-edit behavior net — characterization + invariant mining (D) **Input:** the blast radius from the impact graph (component 06) and the pre-patch baseline state. **Output:** the pre-change executable spec — characterization tests over every function in the blast radius plus mined invariants where instrumentation is cheap. **No language model runs here.** Before the edit, the harness auto-generates characterization tests for every function the diff reaches (impact row's consumers), recording current outputs, side effects, and ordering as the baseline (Feathers [15]), and mines invariants where instrumentation is cheap (Daikon [14]). The output is the standing claim of what must not change — pinned *before* the edit, which is what makes it trustworthy: it was built from behavior that already worked. **Procedure (exact, deterministic):** 1. From the impact graph, compute the blast radius — every function, class, and consumer the diff reaches. 2. For each function in the blast radius, generate a characterization test that pins its current outputs, side effects, and ordering against the pre-patch baseline. 3. Where instrumentation is cheap, mine likely invariants (pre/post-conditions, ranges, ordering constraints) over the observed execution. 4. Record the net — characterization tests plus invariants — as the baseline executable spec, keyed by behavior. **Caps:** the net is pure deterministic software on CPU — the model neither chooses what to pin nor authors the characterization tests; the pre-patch program is the only source of the net's claims, and a pinned bug is pinned honestly, which is why the post-edit manifest routes to review rather than auto-acceptance. ### 3.3 Derive preservation obligations from impact analysis (D) **Input:** the impact graph (component 06) and the candidate diff. **Output:** the named preservation obligations — which behaviors must remain unchanged, which call sites, serializers, config paths, and consumers each obligation reaches, and what evidence would discharge it. **No language model runs here.** This is the "prove every identified preservation obligation" discipline of P2, replacing "run whatever tests happen to exist": a changed function reaches direct callers, indirect callers, subclasses, serializers, config paths, and tests elsewhere, and every one of those is an obligation with evidence attached or a gap recorded as untested. **Procedure (exact, deterministic):** 1. From the impact graph, enumerate every consumer of every changed symbol — direct and indirect callers, subclasses, serializers, config paths, and downstream tests. 2. For each consumer, derive an obligation: the `behavior` it expects to remain unchanged, and the `impact_source` edge (e.g. `"SessionConfig -> SessionManager"`) from which it is derived. 3. Attach the evidence that would discharge each obligation (the test plus its baseline and candidate results), or leave the obligation in `untested_obligations` if no evidence yet exists. **Caps:** obligation derivation is pure deterministic software on CPU — the model neither narrows the set of obligations nor declares one "obviously unaffected"; a patch whose untested obligations are non-empty is not "fully protected", it is "partially evidenced". ### 3.4 Select the regression set — Ekstazi-style, skips logged as unrun (D) **Input:** the obligations (§3.3), the repository graph, and the coverage (component 12). **Output:** the smallest *safe* regression set — a superset of what the change could affect — with every unselected test logged as *unrun* with its selection reason. **No language model runs here.** Checksum-based reachability (Ekstazi [9]) is the floor: a test is selected iff the change could affect it by transitive file/symbol dependency. Safety precedes smallness (Rothermel and Harrold [11]); the set must be a superset of what the change could affect, and smallness is the second property. When the safe set exceeds the loop's budget, Round 1 logs the over-budget tail as *unrun-by-budget* — never silently green — and leaves risk-ranked top-slicing to a deferred round (§2 row 10). **Procedure (exact, deterministic):** 1. Compute the transitive dependency closure of every changed symbol/file (checksum-based reachability). 2. Select every test in that closure — the smallest safe superset. 3. For every unselected test, record `{ test, reason: unrun-by-reachability | unrun-by-budget }` — a skip is *unrun*, never silently green. **Caps:** selection is pure deterministic software on CPU — the model never narrows the safe set to make itself look good, never greases a skip, and never converts an unrun test into an assumed pass; a harness rollup that cannot distinguish green-from-running from green-from-assumption is a defect, and the skip log is the instrument that makes the shrinking visible. ### 3.5 Differential execution — baseline vs candidate (D) **Input:** the changed region, the pre-patch baseline program, and the candidate program. **Output:** every divergence in return values, raised exceptions, execution paths, or mined invariants, each classified deterministically. **No language model runs here.** The differential tier executes the changed region under the pre- and post-patch programs side by side (ChangeGuard [3], PATCH-SIM [4]) and flags divergence — divergence on previously correct behavior is regression evidence even when the suite is green. The asymmetry is PATCH-SIM's: known-good behavior prefers stability, known-bad behavior requires change. The pre-patch program is the oracle for everything the fix did not intend to touch. **Procedure (exact, deterministic):** 1. Run the changed region under the pre-patch baseline; record its outputs, exceptions, and paths. 2. Run the same region under the candidate; record the same. 3. Compare; for each divergence, classify deterministically: divergence on previously-passing behavior is *regression evidence* (recorded into `unexpected_differences`); divergence on the bug-fix behavior is the *intended change* (the fix doing its job). **Caps:** differential execution is pure deterministic software on CPU — the model never decides whether a divergence exists, never reclassifies a divergence to spare the patch, and never skips the baseline run; the pre-patch program is the oracle, and a divergence on previously correct behavior is a fact, not an opinion. ### 3.6 Surface gates — types, build, semver/API diff (D) **Input:** the changed surface — changed signatures, public API, and build targets. **Output:** a surface gate verdict over types, build, and API/semver compatibility. **No language model runs here.** Type checks, builds, and API/semver compatibility run on the changed surface — the signature-level guard for consumers no test can see (Brito [24], Jayasuriya [25]). The dangerous regressions are the ones with unchanged signatures — behavioral changes invisible to a compiler and to any test that only exercises the changed module — so the harness gates them by signature rather than by test. **Procedure (exact, deterministic):** 1. Type-check and build the changed targets; record the compiler's structured diagnostics. 2. Compute the API/semver diff over the changed public surface (removed symbols, narrowed types, changed contracts). 3. A signature-unchanged behavioral break is the class this tier exists to catch; a breaking surface change is recorded as an unexpected difference, not excused. **Caps:** the surface gates are pure deterministic software on CPU — the model never reinterprets a compiler error into a pass and never waives a breaking API change; the gates are the complement to the suite, not a duplicate of it, and a zero catch rate is a signal the gates are not installed (§7.2). ### 3.7 Characterization-diff manifest — intentional vs accidental (D) **Input:** the pinned net (§3.2) and the post-edit behavior. **Output:** the intentional-versus-accidental manifest — every behavioral change after the patch, marked against the pinned net. **No language model runs here.** The manifest is a diff, not a verdict: each entry is marked *intentional* (part of the fix) or *accidental* (regression), and it routes to review. The net is honest about protecting current behavior including its bugs, so the manifest routes to review rather than to auto-acceptance. **Procedure (exact, deterministic):** 1. Re-run the pinned characterization tests and invariant checks against the candidate. 2. For every behavioral change, produce a manifest entry: the behavior, its pinned baseline, its candidate result, and a provisional intentional/accidental marker. 3. Route the manifest to review — never auto-accept; an intentional entry updates the pinned tests *through review*, never by the repair loop's own hand. **Caps:** the manifest is pure deterministic software on CPU — the model never updates a pinned test to match new behavior, never auto-accepts an accidental entry, and never suppresses an entry; the manifest is the review gate's input, and net-tampering (the loop "fixing" regressions by updating characterization tests) is the failure mode this tier exists to catch. ### 3.8 Assemble the PreservationLedger (D) **Input:** the obligations (§3.3), the differential differences (§3.5), the surface-gate findings (§3.6), and the manifest (§3.7). **Output:** the `PreservationLedger` (§3.1) handed to the acceptance stage (component 18). **No language model runs here.** The ledger assembles obligations, evidence, unexpected differences, untested obligations, and `regression_status`; if `untested_obligations` is non-empty, acceptance sees *acceptance evidence incomplete*, not full protection — even when every known test is green. **Procedure (exact, deterministic):** 1. Collect the obligations and their evidence. 2. Collect the unexpected differences (annotated by §4, but their existence is deterministic). 3. Collect the untested obligations. 4. Compute `regression_status` by the §3.1 table over evidence and untested obligations. **Caps:** ledger assembly is pure deterministic software on CPU — the model never writes a field and never authors the status; the ledger is handed to acceptance with its untested obligations in plain sight. ### 3.9 The single L entry — intent interpretation (L) **(L) Intent interpretation [L — the at-most-one learned entry].** After the deterministic divergence is computed, the model reads each observed divergence in the manifest against the change's stated intent and proposes whether it is the intended fix or collateral regression (Testora [6], PatchGuru [7]). Its output is an annotation on the manifest, a hypothesis, never the verdict: the model reads what a divergence *means*, never whether a divergence *exists*, and never authors the preservation verdict. This entry is detailed in §4. **The L column holds at most one entry, and the deterministic preservation machinery is the default.** Services §3.1– §3.8 are deterministic software on CPU; the single learned node is §4, reached only after the deterministic divergence. Any addition putting a model elsewhere — deciding whether a divergence is a regression, silently greening a skipped test, updating a pinned characterization test to match new behavior, or authoring the final `regression_status` — is a regression against P8 and the Piper rule, and must clear the §7 bar on held-out tasks at defensible cost, or be rejected. **Enforced mechanically: "the L column contains at most one entry."** --- ## 4. The LLM Step (L, AT MOST ONE entry) The **single** learned node in the entire stage: **`interpret_intent`** — intent interpretation. The weak model reads each observed divergence in the characterization-diff manifest against the change's stated intent and proposes whether it is the **intended fix** or **collateral regression** (Testora [6], PatchGuru [7]). This is the INTENT-INTERPRETATION step and nothing more: **judge which observed divergences are the fix and which are collateral.** The model emits only an ANNOTATION on the manifest — a hypothesis about each divergence's meaning. It is **downstream of the deterministic divergence (which is a fact), never the verdict on preservation**: the model reads what a divergence means, never whether a divergence exists, and never authors the `regression_status`. The model is **forbidden from deciding whether a divergence is a regression, greening a skipped test, updating a pinned characterization test to match new behavior, or authoring the preservation verdict** — those are deterministic and owned by the harness (§3). **Prompt inputs (assembled deterministically, in this exact order):** 1. **`manifest_entries`** — the deterministic characterization-diff entries (§3.7): each observed divergence, its pinned baseline, and its candidate result. Existence is already established; only meaning is in question. 2. **`issue_intent`** — the extracted specification/issue intent the patch is meant to satisfy (component 05/07). 3. **`impact_context`** — the `impact_source` edges from which each obligation was derived, so the model's reading cannot ignore the blast radius. **Input → output schema (exact):** the model emits a JSON object conforming exactly to `{ annotations: [ { divergence_id, classification: intentional | accidental } ] }`. `classification` is the model's proposal about whether the divergence is the intended fix or collateral regression. The model may NOT emit an entry for a divergence the deterministic tier did not produce, may NOT assert "no divergence" where the manifest records one, and may NOT emit a `regression_status` — that field belongs to the ledger (§3.1/§3.8), never to the model. **Caps:** **ONE call per protection run**, `temperature = 0`, `max_completion_tokens` bounded, schema-constrained; a bounded payload. The model **ANNOTATES only, never certifies, judges the verdict, greens a skip, or updates a pin**. **Gating (exact):** the model proposes intentional/accidental for each manifest divergence. It does **not** decide whether a divergence exists, does **not** green a skipped test, does **not** update a pinned characterization test, and does **not** author `regression_status` — those are the pipeline's deterministic properties (§3); an output asserting any of them is rejected deterministically and never admitted on the model's say-so (§8 invariants 1 and 5). **Parse-failure fallback (exact):** any JSON parse failure, schema violation (a missing `annotations`, a `classification` asserting "no divergence" over a recorded one, or an emitted `regression_status`), or empty/malformed output is **not silently retried as a fresh interpretation call**: the stage **returns to the deterministic stage with the violation fed back** — the malformed annotation is recorded as a violation and the ledger's deterministic verdict stands unmodified, with the divergence's meaning left unresolved for review — **never a re-prompt, never a second call at the same seed**. The L column is reached **once** per protection run. **Re-invocation on a later round (exact):** interpretation is re-considered only when a *later round* of the same repair loop is processed, scoped strictly to that round's candidate patch and fed the accumulated deterministic manifest (§3.7) as its `manifest_entries`. It remains the **single conceptual learned entry** — a bounded, once-per-opportunity call — **never a model that runs over the whole pipeline**. **Invariant 1 (§8): the L column contains at most one entry — a bounded, seeded intent-interpretation call whose output the deterministic pipeline disposes of.** --- ## 5. Implementation Order Ten phases, each independently testable and reversible, landing in `build/19-regression-protection/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Ledger contract (P2/P8)** | `ledger.py` (§3.1 the `PreservationLedger` schema + `maintain_preservation_ledger` + the three-valued status table) | Assemble a fixture ledger: non-empty `untested_obligations` → `UNPROVEN`; a surviving accidental difference → `DIVERGENT`; all obligations evidenced and no accidental difference → `PROTECTED`; assert **no LLM consulted**. | Delete `ledger.py`. | | **2. Pre-edit behavior net (P1)** | `net.py` (§3.2 `pin_behavior_net`: characterization tests over the blast radius + mined invariants) | Given a blast radius, characterization tests pin outputs/side effects/ordering for every reached function; a pinned bug is pinned honestly; assert **no LLM consulted**. | Delete `net.py`. | | **3. Obligation derivation (P2)** | `obligations.py` (§3.3 `derive_preservation_obligations` from the 06 impact graph) | A diff touching `SessionManager` derives obligations reaching `SessionConfig -> SessionManager` and every consumer; an un-evidenced obligation lands in `untested_obligations`; assert **no LLM consulted**. | Delete `obligations.py`. | | **4. Ekstazi-style selection (P3)** | `select.py` (§3.4 `select_regression_set`: checksum-based reachability + skip log) | A change reaching symbol S selects exactly the tests whose transitive dependency reaches S; every unselected test is logged `unrun` with a reason; assert **no LLM consulted**. | Delete `select.py`. | | **5. Differential execution (P4/P5)** | `differential.py` (§3.5 `run_differential`: baseline vs candidate, classify each divergence) | A patch that flips a previously-passing neighbor diverges on that behavior → recorded in `unexpected_differences`; the bug-fix behavior's change is the intended change; assert **no LLM consulted**. | Delete `differential.py`. | | **6. Surface gates (P6)** | `surface.py` (§3.6 `surface_gates`: type check, build, semver/API diff) | A signature-unchanged behavioral break and a removed public symbol are both caught; a compiler error returns the structured diagnostic; assert **no LLM consulted**. | Delete `surface.py`. | | **7. Characterization-diff manifest (P7)** | `manifest.py` (§3.7 `compute_characterization_diff`: intentional-vs-accidental) | A post-edit behavioral change is surfaced as a manifest entry and routes to review; no entry is auto-accepted; assert **no LLM consulted**. | Delete `manifest.py`. | | **8. L stub (intent interpretation)** | `intent.py` (§4 ONE call + temperature 0 + seeded + schema-constrained + parse-failure fallback) | With a mocked LLM: the call is **one**, temperature 0; output has exactly the annotations fields; an emitted `regression_status` → returns to the deterministic stage, **no second call**; assert the ledger verdict is unchanged. | Delete `intent.py`. | | **9. Pipeline driver** | `pipeline.py` (§3 wiring pin→obligations→select→differential→surface→manifest→(L)→ledger) | `pipeline.py` on a fixture with a mocked stub: net→…→manifest→intent→ledger; the ledger reports PROTECTED/DIVERGENT/UNPROVEN with untested obligations visible. | Feature flag `off` → live harness untouched. | | **10. A/B harness** | `ab.py` (§7 KEY + companion + secondaries + ablation runner + decision rule) | `ab.py` runs the §7 pipeline-vs-baseline and ablations on held-out fixtures. | `ab.py` read-only. | The A/B runner (Phase 10) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The pipeline reuses components 06 (impact graph), 12 (coverage), 11/17 (execution substrate / evidence battery), and 05/07 (issue intent) for net-pinning, obligation derivation, selection, and the baseline; it invents no new execution substrate and no new diagnosis path. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of the earlier components' impact graph, coverage, execution substrate, and evidence battery)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `ast` | §3.2/§3.4 blast-radius and symbol-touched computation; §3.6 signature extraction | CPython stdlib. | **No** | | stdlib `json` | §3.1/§4 the `PreservationLedger` schema and the intent-annotation schema | CPython stdlib. | **No** | | stdlib `sqlite3` | §3.1 the durable ledger store and skip log | CPython stdlib. | **No** | | stdlib `hashlib` | §3.4 checksum-based reachability (Ekstazi-style file/symbol checksums) | CPython stdlib. | **No** | | stdlib `subprocess` (sandboxed) | §3.2–§3.7 running the net, the differential execution, and the surface gates under the pinned, no-network environment | CPython stdlib; sandbox enforced by component 11. | **No** | | stdlib `re` | §3.6 API-diff and semver-signature parsing templates | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic services; the stores | In-repo, pinned interpreter. | **No** | | Component 05/06 impact graph | §3.2/§3.3 blast radius and preservation-obligation derivation | Built by spec 05/06; reused. | **No** | | Component 11 execution substrate / sandbox | §3.2–§3.7 the harness's own pinned execution | Built by spec 11; reused. | **No** | | Component 12 coverage | §3.4 regression-set selection | Built by spec 12; reused. | **No** | | Component 17 evidence battery | §3.4/§3.5 baseline execution and the typed evidence for discharging obligations | Built by spec 17; reused. | **No** | | Component 05/07 issue intent | §4 the intent the manifest is read against | Built by spec 05/07; reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single intent-interpretation call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all services use stdlib `ast`/`json`/`sqlite3`/`hashlib`/`subprocess`/`re` (with the component-11 sandbox), and the impact graph, coverage, execution substrate, evidence battery, and issue intent are **reused wholesale from components 05/06, 12, 11, 17, and 05/07**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The DEFERRED rows (§2 rows 10–13) are a learned selection model, a property/fuzz augmentation scale-up, a formal-equivalence proof tier, and a cross-task regression memory — not new third-party Python installs. Any future trained-model download, fine-tuning run, or learned-judgment dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol Regression protection is measured *separately* and *honestly*, because a stage whose whole purpose is to prove the unchanged stayed the same can be defeated in either of two opposite ways: by certifying preservation on a net that never covered the blast radius, or by declaring safety while silently skipping the tests that would have caught the break. The instruments keep the net accountable to what each run actually proved and what it actually skipped, not to the count of green lights. **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, 13, 14, 15, 16, and 17** (Python-only, single-repo, no-Docker), each annotated additionally with injected and baseline regressions (a known regression corpus so the stage's detection can be scored against ground truth). Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. Held-out tasks feed the net, the obligations, the selection, the differential execution, and the outcome statistics; in-benchmark tasks feed nothing that scores the stage. **Baseline (concrete).** The **naive "run full suite, green = safe" protection** — the failure mode this component rejects: on each candidate patch the harness runs the full suite once, reads a bare exit code, and reports "green = nothing broke" with **no pre-edit net, no preservation obligations, no change-scoped selection, no differential execution, no surface gates, no characterization-diff manifest, and no skip log** — a green light is whatever the exit code says, and the untouched behaviors are *assumed* unchanged rather than proved. The preservation stage must beat *this* on the honest measures below, and the ablation (7.3) removes mechanisms against those same measures. ### 7.1 Regression-detection rate + false-alarm rate — the KEY measures The primary instruments, the direct score of the whole stage. **Regression-detection rate** — of every regression the corpus ground-truth labels as present (injected or baseline), what fraction the stage catches (as DIVERGENT, an unexpected difference, a surface-gate hit, or a characterization-diff accidental entry). A stage that reports "green = safe" while half the ground-truth regressions slip past scores near zero, because it has not proved the unchanged stayed the same — it has only not-run the checks that would have said otherwise. **False-alarm rate** — of every candidate that is in fact regression-free, what fraction the stage wrongly flags as DIVERGENT. A stage that cries regression at every real fix is not a protection net, it is a cage; the false-alarm rate keeps the net honest in the opposite direction. Both are computed deterministically against the ground-truth regression corpus and outside the model, because the model cannot be trusted to score its own protection. **State explicitly: "Never the end-to-end resolve rate alone"** — protection must not be judged by whether the eventual patch resolved (component 14's measure) but by whether each regression the corpus planted was caught and each clean patch passed un-flagged. **State explicitly: "Never introspective fluency or model self-assessment"** — a stage scoring highly on the model's own sense that "nothing broke" while the ground-truth regressions slip through uncaught is the thesis's exact failure and scores near zero here. **These are the KEY measures — the preservation stage is adopted only if it STRICTLY beats the naive "run full suite, green = safe" baseline on BOTH AND does not regress the false-alarm rate beyond noise.** ### 7.2 The companion + secondary instruments Report alongside the KEY measures; the secondaries report, do not gate on: 1. **Selection cost** — the companion: how many tests were actually run versus the budget, i.e. `tests_run / tests_in_full_suite`. A stage that "protects" by running the whole suite has bought nothing over the baseline; a stage that protects at a fraction of the suite has earned the selection tier's keep. The cost is bounded at 1.5× the baseline's full-suite cost (§7.4). **Secondary instruments (report, do not gate on):** 1. **Skip-log completeness** — what fraction of unselected tests carry an explicit `unrun` reason (unrun-by-reachability vs unrun-by-budget). A skip log with gaps is assume-pass amnesia in disguise; the instrument drives that fraction toward one. 2. **Characterization-diff precision** — of the manifest entries that review ultimately confirmed, what fraction were true behavioral changes versus noise. A manifest that floods review with false entries is a net that mis-labels its own pins. 3. **Surface-gate coverage** — how often the surface gates caught a break no behavioral test could see (the Brito/Jayasuriya class). A zero rate is not proof of honesty; it is a signal the gates are not installed. 4. **Divergence-explanation rate** — what fraction of `unexpected_differences` received a well-formed intent annotation (§4) versus were left unresolved. A low rate means the single L entry is not actually discharging its one job, or the parse-failure fallback is firing too often. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add a protection mechanism because it sounds sensible" mechanically. Two passes, everything else held fixed, the delta measured on the honest measures (regression-detection rate + false-alarm rate + selection cost + secondaries): 1. **Ablate differential execution** — never run baseline vs candidate side by side, rely on absolute pass/fail: the direct test of P4/P5. 2. **Ablate invariant mining** — characterization tests only, no mined invariants: the direct test of P1's Daikon half. 3. **Ablate intent interpretation** — no-L-vs-L (deterministic-only vs ask) on the KEY measures: the test of whether the bounded stub (§4) earns its single learned entry at all. 4. **Ablate surface gates** — no type check, no build, no semver/API diff: the direct test of P6. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L stub, which must survive the no-L-vs-L ablation to be kept), then to any other mechanism. **Never add a protection mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run protection-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The preservation stage **wins iff**: ``` Regression-detection rate STRICTLY beats the naive "run full suite, green = safe" baseline (KEY) AND false-alarm rate not regressed (≤ baseline + noise; a rising false-alarm rate is a defect to fix) AND selection cost ≤ 1.5 × cost_baseline (tests run / budget) AND divergence-explanation rate not regressed (a falling rate signals the L entry is not discharging its job) ``` The KEY measure must **strictly beat the naive "run full suite, green = safe" baseline**; the false-alarm rate must not regress; selection cost is bounded at 1.5×; the divergence-explanation rate must not regress. **Never the end-to-end resolve rate alone** — resolve is component 14's measure, and a stage that improves resolve while leaving the regression-detection rate flat against the baseline is **not adopted**. **Never introspective fluency or model self-assessment** — a stage that reports the model's confidence that "nothing broke" while the ground-truth regressions slip through scores *worse* and is rejected. If the stage wins it replaces the naive full-suite-green path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways a preservation net can lie) 1. **Assume-pass amnesia** — skipped tests counted green, the suite quietly shrinking while the report stays green. Caught by the skip log and the skip-log-completeness instrument. 2. **Characterization lock-in** — pinned behavior (bugs included) treated as correct forever, the net becoming a cage that vetoes the actual fix. Caught by the false-alarm rate and the review-routed manifest. 3. **Net-tampering** — the loop "fixing" regressions by updating characterization tests to match new (wrong) behavior. Caught by the manifest-routes-to-review rule and the characterization-diff precision instrument. 4. **Over-minimisation** — aggressive pruning for compute whose coverage holes are discovered by the incident. Caught by the selection-cost-vs-safety trade and the red-flag on minimization. 5. **Stale invariants** — mined specs encoding obsolete behavior firing rejection storms, and getting ignored, taking real violations' credibility with them. Caught by the false-alarm rate and invariant-revalidation checks. 6. **Surface blindness** — all internal tests green while the public API is broken, downstream consumers as the regression detector of last resort. Caught by the surface-gate coverage instrument. 7. **Intermediate-state rot** — a final diff green while rounds 2 through 4 broke and un-broke neighbors — invisible without per-edit runs. Caught by the per-edit differential execution, run at every round, never only at the end. 8. **Incomplete-versus-introduced conflation** — the two overfitting directions collapsed: a patch that never fully fixed the bug recorded as "protected" because the net was aimed only at the regression-introduction direction. Caught by the P5 classification and the differential tier's separation of intended change from collateral. ### 7.6 Two honest caveats **First, the strong-model caveat:** the paper's headline numbers — ChangeGuard's 77.1%/69.5%/7.6%, PATCH-SIM's 56.3%, Probe to Generate's 77% and 4.2%–9.0%, HyRTS's 18.35%, TestPrune's 9.4%–12.9% — will **NOT** reproduce on a weak model, and were measured on repositories and models far larger than a weak-model harness controls; a weak-model harness should expect materially smaller absolute effects while leaning all the harder on the cheap tiers, the pinned net, and the smallest-safe-set discipline to make its small budget count. **Second, the held-out caution:** all preservation metrics must be reported on held-out and out-of-benchmark tasks, so a net that has memorized a task set's answers cannot score well on regression detection without its preservation verdicts having been measured on problems it never saw. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/19-regression-protection/`; it never writes into the live harness's source. - A **feature flag** (env var `REGRESSION=19net|fullsuitegreen`, default `fullsuitegreen`) routes the A/B; the flag alone switches between the preservation stage and the naive "run full suite, green = safe" baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running old-vs-new is always possible in the same session, repo, model. - The durable `PreservationLedger` store and the skip log are append-only and versioned; rolling the build back never loses them. A ledger superseded by later evidence is kept with its obligations, its differences, and its status — never silently dropped and never silently re-authored by a model. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains at most one entry** — intent interpretation (§4), the one bounded, seeded call judging which observed divergences are the fix and which are collateral, reached only after the deterministic divergence, never re-prompted on parse failure. Any phase introducing a model elsewhere (deciding whether a divergence exists, greening a skipped test, updating a pinned characterization test, authoring the regression_status, or selecting the regression set) is rejected outright. An **empty** L column (the stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **A skipped test is logged as unrun, never silently green** — every unselected test carries an explicit `unrun` reason, and no assume-pass exists (P3). 3. **Selection is safe first, small second** — the selected regression set is a superset of what the change could affect, and a regression that escapes selection to the full suite is a selection-safety defect, never excused (P3). 4. **The pre-patch program is the oracle for unchanged behavior** — preservation is a comparison against the known-good baseline, not an absolute pass/fail; a divergence on previously correct behavior is regression evidence even when the suite is green (P4). 5. **The characterization diff routes to review, never auto-accept** — intentional entries update the pinned tests *through review*, never by the repair loop's own hand; the manifest is a diff, not a verdict (P7). 6. **Minimization is a red-flagged operation** — deleting tests to save compute spends safety and is never done casually; the harness leans on selection and prioritization, not minimization (P3, Yoo & Harman [12]). 7. **The ledger is written by deterministic software only** — every field except the intent annotation on `unexpected_differences` is written by software on CPU, and the `regression_status` is computed by table lookup, never by a model (P8). 8. **All metrics on held-out/out-of-benchmark tasks** — the KEY measures and every secondary are reported on held-out tasks, never tuning or in-benchmark data (§7). --- *End of spec. This document is the build contract; the next document records the A/B outcome.*