Summary / 18. Validation and patch acceptance / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 18 — Validation / Patch Acceptance (the adversarial, evidence-thresholded acceptance stage) > Derived from `papers/18-validation-patch-acceptance.md` — "The Validator's Job Is to Prove the Patch Wrong: Adversarial, Evidence-Thresholded Patch Acceptance for a Weak-Model Coding Harness." > This document drives the build of the **patch-acceptance stage**: the reversible, adversarial, evidence-thresholded > decision gate in which the harness rules whether a candidate patch may become repository state, and on what recorded > grounds. It sits at the point where every upstream claim — the decomposed request, the localized fault, the generated > candidate, the competing hypotheses, and component 17's typed evidence battery — converges into exactly one deliverable, > and this is the referee that rules whether the deliverable ships. The sharp distinction is between *"it ran"* and > *"it ships"*: component 17 established that the tests were executed honestly; this component establishes whether the > accumulated evidence is *sufficient* to declare the patch correct. The spine of the design is the **two-part verdict** — > *fixed?* (the FAIL_TO_PASS tests flip under harness execution) and *unbroken?* (the guard set holds) are two different > questions answered by two different kinds of evidence, and a patch that satisfies only one is not accepted — plus the > **evidence ladder A→E** whose acceptance is monotonic, the **risk-sensitive threshold table** that maps change class to > minimum required evidence, **differential execution against the pre-edit repository** as the free executable > specification of unchanged behavior, a **semantic-assessment veto layer** that can demote a verdict but never promote > one, and an **explicit `PatchAcceptance` record** linking every verdict to its supporting and unresolved evidence. The > guiding rule, made mechanical: **The patch generator's job is to make a patch look right. The validator's job is to > prove it wrong. Only when the validator fails after sufficiently strong attacks does the patch earn acceptance.** When > the evidence falls short of that bar, the correct result is **INCONCLUSIVE, never ACCEPT**. This spec pins the paper's > design to exact primitives, the exact `PatchAcceptance` schema, the exact evidence-ladder levels A→E, the exact > risk-sensitive threshold table, the exact two-part verdict, the exact L-column firing, exact A/B instruments and > thresholds, exact ablation mechanics, and an exact build sequence. Where the paper says "≈80%", this spec pins *which* > 80%. --- ## 1. Purpose & Scope Build a **reversible, adversarial, evidence-thresholded acceptance stage** — a decision function over a structured evidence battery whose inputs are the candidate patch, the buggy baseline, the pre-edit repository, and the typed evidence already produced by component 17 plus the change-impact (06) and coverage (12) information, and whose output is a **verdict — ACCEPT, REJECT, or INCONCLUSIVE** — together with a `PatchAcceptance` record. It differs from its predecessors in *kind*, not just position: generation (14, 16) asks *"how could this patch be right?"*; test execution (17) asks *"which tests passed, what property did each one prove?"*; reflection (15) asks *"what did this failure teach?"*; acceptance asks *"did the validator, having attacked the patch as hard as the change's risk demanded, fail to break it?"* — and rules the **VERDICT** those stages act on. The naive harness asks "did the tests pass?" and commits; this stage treats a green suite as **one kind of evidence**, not a verdict, and the strongest available evidence is a failed attempt to prove the patch wrong. The paper's eight principles decide every design choice. **P1 Acceptance is a two-part verdict, never conflated** — *fixed?* and *unbroken?* are answered by different evidence (the targeted FAIL_TO_PASS tests must flip under harness execution AND the guard set — the impact row's affected tests plus the PASS_TO_PASS neighborhood — must hold), and a patch that fixes the bug while destroying its neighbors, and a patch that preserves everything while never fixing the defect, each fail a different clause, typed to the clause it violated. **P2 The verdict is adjudicated harness-side; the model's "tests pass" is a claim, not a fact** — the certification runs in the pinned, tamper-protected environment, and a model's report that everything passed triggers a re-run, never substitutes for one. **P3 Evidence is monotonic and thresholded; INCONCLUSIVE is a first-class verdict** — a patch that has passed a stronger challenge has automatically passed every weaker one, a patch is accepted only at or above the level its change class demands, and insufficient evidence yields INCONCLUSIVE, never ACCEPT. **P4 The division of labor is adversarial** — generation asks how the patch could be right, validation asks how to prove it wrong, and where the validator can be deterministic software, it is. **P5 The pre-edit repository is a free executable specification** — differential execution against the pre-edit state answers "did we break anything that worked?" with the cheapest real evidence there is, and any divergence is a regression. **P6 The semantic assessment is a veto, never an override** — the root-cause-vs-overfit judgment can only demote a verdict, never promote one, and a hard execution failure always wins over a confident semantic assessment. **P7 Risk scales the required evidence** — change class maps to a minimum required evidence level, so the harness spends its validation budget in proportion to the blast radius of the change. **P8 Every verdict carries an explicit acceptance record** — the `PatchAcceptance` record links the verdict to its supporting and unresolved evidence, so six months later the harness can answer "why did I accept this?" with provenance, not "tests passed". **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — the two-part verdict, the evidence ladder A→E with monotonic acceptance, the risk-sensitive threshold table, differential execution against the pre-edit repository, the overfit-tell scan, the tamper audit, and the `PatchAcceptance` record — plus the *bound* of the *single* learned entry, the semantic assessment (§4), delivered as a **bounded stub**. The heavy pieces (full property and fuzz automation at scale, a learned execution-free verifier ranking, formal equivalence via SymDiff/Godlin–Strichman, cross-task acceptance-memory) are **not** built in Round 1 and are **not excluded** — later rounds climb toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by **correctness-at-acceptance** and its co-signatory **false-reject rate**, with companion **evidence cost per acceptance**, never the resolve rate alone and never model self-assessment (§7). The stage lives in `build/18-validation-patch-acceptance/` and writes nothing into the live harness. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–9); **DEFERRED = later rounds** (rows 10–13), **not excluded** but gated by the same §7 A/B protocol. Every later round of this component still builds within the same `build/18-validation-patch-acceptance/` tree. | # | Component (paper §4) | Classification | |---|---|---| | 1 | Two-part verdict — `compute_two_part_verdict` computes *fixed?* and *unbroken?* as separate clauses, both required, never conflated (P1) | **BUILD-NOW** | | 2 | Evidence ladder A→E — `assemble_evidence` folds component 17's typed results into the five monotonic levels and records the failing tier (P3) | **BUILD-NOW** | | 3 | Risk-sensitive threshold table — `compute_change_class` + `require_evidence_level` map change class to minimum required evidence (P7) | **BUILD-NOW** | | 4 | Differential execution — `differential_execute` runs pre-edit and patched on unchanged-contract inputs and flags divergence (P5) | **BUILD-NOW** | | 5 | Overfit-tell scan — `scan_overfit_tells` statically scans the diff for functionality-deletion, special-case, and test-specific-hack tells (P6, ODS-shaped) | **BUILD-NOW** | | 6 | Tamper audit — `audit_tamper` re-applies protected test files / diff-scope-checks that no test, fixture, or skip marker changed (P2) | **BUILD-NOW** | | 7 | The explicit acceptance record — `write_acceptance_record` persists the `PatchAcceptance` record with evidence provenance (P8) | **BUILD-NOW** | | 8 | (L) Semantic assessment / review gate — the root-cause-vs-overfit + minimality/idiom judgment as ONE learned veto (P6, demote-only) | **BUILD-NOW** (bounded stub only) | | 9 | Loop driver + A/B harness (§7 metrics + ablation runner + failure-mode audits) | **BUILD-NOW** | | 10 | **Full property/fuzz automation at scale** | **DEFERRED** — Round 1 ships a bounded Level-E attack gated by change class; systematic property synthesis and large-budget fuzzing are deferred | | 11 | **Learned execution-free verifier ranking** | **DEFERRED** — Round 1 ships no second learned judge; a trained ranker (SWE-RM/RePaCA-class) is deferred and must beat the deterministic battery on honest metrics | | 12 | **Formal equivalence (SymDiff / Godlin–Strichman regression verification)** | **DEFERRED** — Round 1 ships differential *sampling*; damage-absence *proof* of unchanged sub-parts is deferred | | 13 | **Cross-task acceptance-memory** | **DEFERRED** — Round 1 records per-verdict provenance; a shared acceptance-memory that transfers evidence patterns across tasks is deferred | **The cost rule (explicit).** The single learned entry — the semantic assessment / review gate — is the AT MOST ONE L-column entry this architecture permits. In Round 1 it is a **bounded stub**: wired in at its firing point (§4), seeded, schema-constrained, never auto-promoted; its full training and calibration is **deferred**. Its cost is bounded before invocation: one call per patch, temperature zero, schema-constrained, bounded payload, parse-failure fallback that returns to the deterministic stage with the violation fed back (§4 caps). It is kept **only if** the §7 no-L-vs-L ablation shows it improves the KEY measures on held-out tasks; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the verdict, ladder, threshold, differential, overfit-scan, tamper, or record path by default — the full deterministic battery (rows 1–7, 9) plus the *bounded stub* of the single learned entry (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a learned ranking/scope/judgment, a formal-verification backend, or a cross-task memory — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out tasks, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) The acceptance stage is a decision function over a structured evidence battery; every node is labeled **D** (deterministic software on CPU) or **L** (a learned model). The invariant that governs the whole is: **the L column contains at most one entry, and the deterministic evidence battery is the default.** The ladder assembly, the threshold computation, the two-part verdict, the differential battery, the overfit scan, the tamper audit, and the record are all deterministic software on CPU; the single L entry is the *semantic assessment / review gate* of §4, which contributes its only legitimate value — a veto against overfit — and never the verdict: ``` CANDIDATE PATCH (C) + BUGGY BASELINE (B) + PRE-EDIT REPO + EVIDENCE (17) + CHANGE-IMPACT (06) + COVERAGE (12) → compute_change_class [D] → risk classification from impact analysis (P7) → require_evidence_level [D] → the class → minimum-level threshold (P7) → assemble_evidence [D] → fold 17's typed results into the ladder A→E (P3) → scan_overfit_tells [D] → static scan for functionality-deletion/special-case tells (P6) → audit_tamper [D] → reapply protected test files / diff-scope-check (P2) → differential_execute [D] → run pre-edit vs patched on unchanged-contract inputs (P5) → run_property_and_fuzz [D] → the Level-E attack, gated by change class (P5) → compute_two_part_verdict [D] → fixed? AND unbroken?, both required (P1) → (L) apply_semantic_veto [L] → the at-most-one learned entry, demote-only (P6) → write_acceptance_record [D] → the PatchAcceptance provenance record (P8) ``` The pipeline reasons over **the candidate patch made into a verdict** — the buggy baseline, the pre-edit repository, and the typed evidence from component 17 are the only things the decision is computed over, and every verdict is computed deterministically with the single learned node contributing only a demote-only veto. All artifacts land in `build/18-validation-patch-acceptance/`. ### 3.1 The evidence ladder A→E (D) **Input:** the typed evidence from component 17 plus the results of this component's own scans and attacks. **Output:** the highest level the patch has *demonstrably* reached, plus the failing tier if one failed. **No language model runs here.** The evidence is assembled in five monotonically stronger levels, each subsuming the last. Acceptance is monotonic in evidence strength: a patch accepted at a higher level has passed every lower one, and a patch that fails any level fails the ladder at that level, with the failing tier itself recorded as debugging evidence. - **Level A — syntactic plausibility [D].** The patch parses, lints, type-checks, and builds in the pinned environment. Not remotely sufficient, but a failure here halts immediately (a REJECT, tier A). - **Level B — plausible repair [D].** The bug-specific reproducer — the repository test that exposes the defect, or a generated reproduction test validated against the buggy baseline — demonstrates fail-on-buggy and pass-on-candidate, so the original failure is proven to have changed from fail to pass. - **Level C — regression-safe [D].** The guard set holds — the impacted unit tests and the affected regression subset, selected from dependency and coverage, all PASS_TO_PASS obligations in the affected neighborhood — so previously correct behavior is preserved through impact-aware regression testing. - **Level D — semantically supported [D].** The semantic assessment (§4, the single L entry) finds the patch addresses the diagnosed root cause rather than overfitting the visible tests, with a recorded confidence. The *ladder* folds this L output in deterministically; the *assessment* itself is the one learned node. - **Level E — behaviorally challenged [D].** Differential execution against the pre-edit repository, property-based checks, and — where the change class warrants it — fuzz testing actively search for a behavioral counterexample and find none. **Procedure (exact, deterministic):** 1. For each level A→E in order, fold the available evidence into that level's verdict. 2. If a level fails, record the failing tier and halt with REJECT at that tier. 3. Record the highest level reached; the two-part verdict (§3.4) then compares it against the required level (§3.3). **Caps:** the ladder assembly is pure deterministic software on CPU — the model never decides which level a patch reached; the monotonic progression is arithmetic over the typed evidence, and a patch's level is computed by the harness, not asserted by the model. (Level D's *evidence* is the single L entry's recorded, demote-only output, folded in deterministically; it never authors the level.) ### 3.2 The `PatchAcceptance` record schema (D) **Input:** a verdict and its assembled evidence. **Output:** a persisted `PatchAcceptance` record. **No language model runs here.** Every verdict is written with evidence provenance, so six months later "why did I accept this?" has an answer better than "tests passed". The record is keyed to a patch, not to a repository state. **The `PatchAcceptance` schema (exact — EXACTLY these fields):** ```json { "patch_id": "…", // the candidate patch's identifier (keyed to the diff, not the repo state) "syntax": { // Level A — syntactic plausibility "parsed": true, "linted": true, "typechecked": true, "built": true, "failure": null }, // structured diagnostic, or null on pass "bug_reproduction": { // Level B — plausible repair "reproducer": "…", // the repo test or generated reproducer identifier "baseline_result": "…", // result on the buggy state B "candidate_result": "…", // result on the candidate state C "bug_discriminating": false, // B FAIL / C PASS, set deterministically "fixed": false }, // FAIL_TO_PASS flipped under harness execution "regressions": { // Level C — regression-safe "guard_set": 0, // the impacted + affected PASS_TO_PASS set size "passed": 0, "failed": 0, // pass/fail counts over the guard set "unbroken": false }, // the guard set held (no flips) "impact_tests": { // the impact row's affected tests (06/12 selection) "selected": 0, "passed": 0, "failed": 0, "coverage_reached": 0.0 }, // fraction of the change's impacted symbols covered "semantic_assessment": { // Level D — the single L entry (demote-only) "judgment": "…", // root-cause-addressed | overfit | non-minimal | unidiomatic "confidence": 0.0, // 0..1, recorded not trusted "demoted": false, // whether it demoted the verdict, and from/to what "from_verdict": null, "to_verdict": null }, "differential_testing": { // Level E — differential execution vs pre-edit "probes": 0, "divergences": 0, // probe count and divergence count "outcome": "…" }, // no-divergence | diverged | not_required (with reason) "fuzzing": { // Level E — the property/fuzz attack, gated by change class "ran": false, "iterations": 0, "counterexamples": 0, "outcome": "…" }, // none-found | counterexample-found | not_required (with reason) "unresolved_risks": [ "…" ], // risks that remain unresolved (P3's INCONCLUSIVE triggers) "verdict": "…", // ACCEPT | REJECT | INCONCLUSIVE "evidence_strength": "…" } // the highest ladder level reached: A | B | C | D | E ``` `patch_id` identifies the candidate; `syntax` the Level-A gate; `bug_reproduction` the Level-B B/C contrast and the *fixed?* clause; `regressions` the Level-C guard-set counts and the *unbroken?* clause; `impact_tests` the impact-row coverage; `semantic_assessment` the Level-D learned judgment with its recorded confidence and demote path; `differential_testing` and `fuzzing` the Level-E attacks with their outcomes and `not_required` reasons; `unresolved_risks` the named risks that kept the verdict INCONCLUSIVE; `verdict` the ruling; `evidence_strength` the highest level reached. REJECTED patches keep their prune reasons and the tier that caught them; INCONCLUSIVE patches keep the specific missing evidence that, if obtained, would reopen the decision. **Caps:** record-writing is pure deterministic software on CPU — no model authors or edits a record's evidence; the record is a projection of the deterministic battery's outputs, and a verdict without a complete record is a scored defect (§7.2 record completeness), never silently accepted. ### 3.3 Risk-sensitive thresholds (D) **Input:** the change-impact analysis from component 06. **Output:** the change class and the minimum required evidence level. **No language model runs here.** The minimum required evidence level is a deterministic function of the change class, computed from the impact analysis — change radius, API exposure, persistence sensitivity, security sensitivity, and coverage — so the harness spends its validation budget in proportion to the blast radius of the change (P7). **The threshold table (exact):** | Change class | Minimum required evidence | The above plus | |---|---|---| | comment / documentation | **Level A** | syntax/build where applicable | | local refactor | **Level C** | targeted plus impacted regression | | normal bug fix | **Level C** | reproducer plus impacted/full regression | | public API / schema | **Level D** | compatibility and impact validation | | persistence / data migration | **Level D** | migration and round-trip tests | | auth / security | **Level E** | adversarial, fuzz, and property validation | | concurrency | **Level E** | repeated, stress, and race validation | A patch that reaches only a lower level than its class demands is **INCONCLUSIVE, never accepted** — the risk, not the patch's confidence, sets the bar. `compute_change_class` produces the class; `require_evidence_level` maps it to the level; the two-part verdict then enforces it. **Caps:** classification and thresholding are pure deterministic software on CPU — the model never argues a change down in risk to lower the bar; an auth change accepted at Level C is a proportionality failure scored by §7.2 risk-threshold adherence, never excused. ### 3.4 The two-part verdict (D) **Input:** the assembled evidence and the required level. **Output:** the verdict — ACCEPT, REJECT, or INCONCLUSIVE — typed to the clause it turns on. **No language model runs here.** The harness computes two clauses separately and never conflates them: *fixed?* — the FAIL_TO_PASS tests flip under harness execution (Level B); *unbroken?* — the guard set holds (Level C). The verdict is ACCEPT only when both clauses are established **at or above the required evidence level**. One clause is unaccepted, with the verdict typed to the missing clause. REJECT is returned when any deterministic tier fails (a Level-A parse failure, a Level-B reproduction that does not flip, a Level-C regression); INCONCLUSIVE is returned when the patch survives every tier it was given but the evidence does not reach its class's required level — the first-class verdict of P3. **Procedure (exact, deterministic):** 1. `fixed?` = `bug_reproduction.bug_discriminating` AND the FAIL_TO_PASS flip is recorded under harness execution. 2. `unbroken?` = `regressions.unbroken` AND the guard set held. 3. If any deterministic tier failed → REJECT (typed to the tier). 4. Else if `evidence_strength` < required level → INCONCLUSIVE (typed to the missing evidence). 5. Else → ACCEPT. **Caps:** the verdict is pure deterministic software on CPU — the model never authors the verdict, never waives a failing tier, and never converts INCONCLUSIVE into ACCEPT; the two clauses are a conjunction computed by the harness. ### 3.5 Differential execution (D) **Input:** the pre-edit repository and the patched repository. **Output:** a divergence report. **No language model runs here.** Differential execution answers "did we break anything that worked?" with the cheapest real evidence there is: run the original and the patched program on inputs where the original was correct, and treat any divergence as a regression — DiffTGen's unchanged-contract check, generalized to the full differential battery (P5). The old code is the specification of everything that was not supposed to change. **Procedure (exact, deterministic):** 1. Collect unchanged-contract inputs — inputs on which the pre-edit program was observed correct. 2. Run the pre-edit and patched programs on each; record any behavioral divergence. 3. A divergence is a regression and demotes the Level-E outcome; zero divergence over N probes records "no divergence in N probes". **Caps:** differential execution is pure deterministic software on CPU — the model never decides a divergence does not count; divergence is arithmetic over two observed behaviors, and the probe budget is set by the change class, not the model. ### 3.6 Overfit-tell scan (D) **Input:** the candidate diff and the buggy code. **Output:** a static overfit-tell report. **No language model runs here.** Before spending a differential battery, the harness statically scans the diff for the tells that Qi et al.'s functionality-deletions operationalize (P6, ODS-shaped): a hard-coded constant where logic used to be, a special case for the tested input, a deleted branch, a broadened or skipped condition — the *shape* of a workaround rather than a repair. This is the cheapest rung of the overfit judgment, and it is deterministic. **Procedure (exact, deterministic):** 1. Diff-normalize the change (AST-level, not raw text). 2. Match each edit against the overfit-tell patterns: constant-substitution, input special-casing, branch deletion, condition weakening, scope-broadening, test-hack. 3. Record matched tells; they feed the semantic assessment's context and the record's `unresolved_risks`, but never alone set the verdict. **Caps:** the scan is pure deterministic software on CPU — the model never re-labels a tell as benign; a matched tell is recorded evidence, and the scan alone never certifies or rejects, it only flags. ### 3.7 Tamper audit (D) **Input:** the protected test files and the candidate patch's diff scope. **Output:** a tamper report. **No language model runs here.** The certification runs in the tamper-protected environment: protected test files are reapplied and compared, or the diff is scope-checked so no test, fixture, or skip/xfail/mock-of-SUT marker changed — because deleting a flaky test or weakening an assertion is not a fix (P2). A tamper detection is a REJECT, typed to the tamper. **Procedure (exact, deterministic):** 1. Recompute protected-file hashes, or diff-scope-check that no test/fixture/skip marker changed. 2. On violation, reapply the protected files and record a tamper detection → REJECT. 3. On clean scope, record and continue. **Caps:** the tamper audit is pure deterministic software on CPU — the model never argues a changed test back into scope; a tamper detection is a deterministic REJECT, and a zero tamper-detection rate is a signal (§7.2) that the protection is not installed, never proof of honesty. ### 3.8 Cross-cutting deterministic services (D) Eight services make the gate a decision function rather than a script, each pure deterministic software on CPU except `apply_semantic_veto` (the L entry's wrapper): `compute_change_class` (the impact-analysis-derived risk classification of P7); `require_evidence_level` (the class→level threshold mapping); `assemble_evidence` (fold component 17's typed results into the ladder levels A→E, P3); `compute_two_part_verdict` (the fixed?/unbroken? conjunction of P1); `scan_overfit_tells` (§3.6); `audit_tamper` (§3.7); `differential_execute` (§3.5); `run_property_and_fuzz` (the Level-E attack, gated by change class); `apply_semantic_veto` (the L entry, demote-only); and `write_acceptance_record` (the PatchAcceptance provenance of P8). Every one except `apply_semantic_veto` is deterministic software on CPU. **Caps:** the deterministic services are pure software on CPU — the model never classifies the change, maps the threshold, assembles the ladder, computes the verdict, scans tells, audits tamper, runs differential/fuzz, or writes the record. The only L entry is §4, and it is a veto, never the verdict. ### 3.9 The single L entry — the semantic assessment / review gate (L) **(L) Semantic assessment / review gate [L — the at-most-one learned entry].** A learned assessor — RePaCA-class, or a budget-appropriate reasoning model — reads the patch, the buggy code, and the issue, and judges whether the transformation addresses the diagnosed root cause or overfits the visible tests, attending specifically to functionality-deletions, incomplete fixes, out-of-scope regressions, and test-specific hacks. The same single entry also carries the review facet — the minimality, scope-containment, and idiom judgment of the diff-level review gate. It reports a judgment and a confidence. Its authority is a **veto only**: it can demote an execution-green patch (ACCEPT→INCONCLUSIVE or REJECT); it can never promote one and never override a deterministic failure. This entry is detailed in §4. **The L column holds at most one entry, and the deterministic evidence battery is the default.** The ladder, thresholds, two-part verdict, differential battery, overfit scan, tamper audit, and record are deterministic software on CPU; the single learned node is §4, and it is a demote-only veto. Any addition putting a model elsewhere — authoring the verdict, deciding whether green counts, waiving a failing tier, promoting an execution-green patch on its own confidence, or judging a divergence benign — is a regression against P1–P8 and must clear the §7 bar on held-out tasks at defensible cost, or be rejected. **Enforced mechanically: "the L column contains at most one entry."** --- ## 4. The LLM Step (L, AT MOST ONE entry) The **single** learned node in the entire stage: **`apply_semantic_veto`** — the semantic assessment / review gate. The weak model reads the candidate patch, the buggy baseline, and the issue intent, and judges whether the transformation addresses the diagnosed root cause or overfits the visible tests, combined with the minimality-and-idiom judgment of the review gate. This is **one learned veto with two facets** — root-cause-vs-overfit and minimality/idiom — and nothing more: it may **demote** an execution-green verdict (ACCEPT→INCONCLUSIVE or REJECT), never **promote** one, and never **override** a deterministic execution failure. A hard execution failure always wins over a confident semantic assessment, because no static judgment of "does this look like a repair" is stronger than a concrete counterexample. The model is **forbidden from authoring the verdict, deciding whether a green result counts, waiving a failing tier, promoting an execution-green patch on its own confidence, or overriding a deterministic failure** — those are deterministic and owned by the harness (§3). **Prompt inputs (assembled deterministically, in this exact order):** 1. **`candidate_patch`** — the candidate diff under judgment. 2. **`buggy_code`** — the buggy baseline the patch transforms. 3. **`issue_intent`** — the extracted specification/issue intent the patch is meant to satisfy. 4. **`deterministic_evidence`** — the deterministic battery's outputs so far: the ladder level reached, the B/C pair, the guard-set counts, the differential/fuzz outcomes, and the overfit-tell scan — so the model cannot re-argue a tier that already failed. **Input → output schema (exact):** the model emits a JSON object conforming exactly to `{ judgment, confidence }`. `judgment` names the finding — `root-cause-addressed`, `overfit`, `non-minimal`, `unidiomatic` — or `null` if it has nothing to add; `confidence` is the recorded 0..1 self-confidence. The output is interpreted by the harness as a **veto only**: an `overfit`/`non-minimal`/`unidiomatic` judgment demotes the verdict; a `root-cause-addressed` judgment supports continuation but never overrides a failing tier and never certifies by itself. Neither field may contradict the deterministic result it is handed: a `judgment` of "root-cause-addressed" over a deterministic REJECT is a schema violation. **Caps:** **ONE call per patch**, `temperature = 0`, `max_completion_tokens` bounded, schema-constrained; a bounded payload. The model **VETOES only — it never certifies, judges green, promotes, or overrides**. **Gating (exact):** the model may demote an execution-green patch and record its confidence. It does **not** decide whether a green result counts, does **not** waive a failing tier, does **not** promote a patch, and does **not** author the verdict — those are the ladder's deterministic properties (§3); an output asserting any of them is rejected deterministically and never admitted on the model's say-so (§8 invariants 3 and 4). **Parse-failure fallback (exact):** any JSON parse failure, schema violation (a missing `judgment`, a `judgment` that asserts root-cause-addressed over a deterministic failure, or a judgment that promotes a verdict), or empty/malformed output is **not silently retried as a fresh assessment call**: the stage **returns to the deterministic stage with the violation fed back** — the malformed assessment is recorded as a violation and the deterministic verdict stands unmodified — **never a re-prompt, never a second call at the same seed**. The L column is reached **once** per patch. **Re-invocation on a later round (exact):** the semantic assessment is re-considered only when a *later round* of the same repair loop produces a *new* candidate patch, scoped strictly to that patch and fed the accumulated deterministic results (§3) as its `deterministic_evidence`. It remains the **single conceptual learned entry** — a bounded, once-per-patch call — **never a model that runs over the whole gate**. **Invariant 1 (§8): the L column contains at most one entry — a bounded, seeded semantic-assessment call whose output the deterministic battery disposes of, demote-only.** --- ## 5. Implementation Order Ten phases, each independently testable and reversible, landing in `build/18-validation-patch-acceptance/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Two-part verdict (P1)** | `verdict.py` (§3.4 `compute_two_part_verdict`: fixed? AND unbroken?, both required) | A patch fixing the bug but breaking a neighbor → REJECT typed to `unbroken?`; a patch preserving all but never fixing → REJECT typed to `fixed?`; assert **no LLM consulted**. | Delete `verdict.py`. | | **2. Evidence ladder (P3)** | `ladder.py` (§3.1 `assemble_evidence`: fold 17's typed results into A→E, record the failing tier) | Evidence reaching Level C assembles to `evidence_strength = "C"`; a Level-A parse failure halts with tier A; assert **no LLM consulted**. | Delete `ladder.py`. | | **3. Risk thresholds (P7)** | `risk.py` (§3.3 `compute_change_class` + `require_evidence_level` + the exact table) | An auth change → Level E; a comment → Level A; a patch reaching C on an auth change → INCONCLUSIVE; assert **no LLM consulted**. | Delete `risk.py`. | | **4. Differential execution (P5)** | `differential.py` (§3.5 run pre-edit vs patched on unchanged-contract inputs) | A divergence on an unchanged-contract input is recorded as a regression; zero divergence over N probes records "no divergence in N probes"; assert **no LLM consulted**. | Delete `differential.py`. | | **5. Overfit-tell scan (P6)** | `tells.py` (§3.6 static scan for constant-substitution, special-casing, branch deletion, condition weakening) | A hard-coded constant replacing logic matches; a genuine small repair does not; matches feed `unresolved_risks`, never alone the verdict; assert **no LLM consulted**. | Delete `tells.py`. | | **6. Tamper audit (P2)** | `tamper.py` (§3.7 reapply protected files / diff-scope-check) | A diff touching a test/fixture/skip marker → REJECT typed to tamper; clean scope passes; assert **no LLM consulted**. | Delete `tamper.py`. | | **7. Acceptance record (P8)** | `record.py` (§3.2 the exact `PatchAcceptance` schema + `write_acceptance_record`) | A verdict writes a complete record with all eleven fields; a bare verdict is refused and scored as record-incomplete; assert **no LLM consulted**. | Delete `record.py`. | | **8. L stub (semantic veto)** | `veto.py` (§4 ONE call + temperature 0 + seeded + schema-constrained + parse-failure fallback) | With a mocked LLM: the call is **one**, temperature 0; an `overfit` judgment demotes ACCEPT→INCONCLUSIVE; a `root-cause-addressed` over a deterministic REJECT → returns to the deterministic stage, **no second call**; assert the verdict is unchanged. | Delete `veto.py`. | | **9. Loop driver** | `pipeline.py` (§3 wiring risk→ladder→verdict→(L)→record) | `pipeline.py` on a fixture with a mocked stub: risk→ladder→verdict→veto→record. | Feature flag `off` → live harness untouched. | | **10. A/B harness** | `ab.py` (§7 KEY + companion + secondaries + ablation runner + no-L-vs-L + decision) | `ab.py` runs the §7 pipeline-vs-baseline and ablations on held-out fixtures. | `ab.py` read-only. | The A/B runner (Phase 10) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The pipeline reuses component 17 (the evidence battery it reads), component 06 (the impact graph for change class and the guard set), and component 11 (the execution substrate / sandbox); it invents no new execution substrate and no new diagnosis path. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of the earlier components' evidence battery, impact graph, and execution substrate)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `json` | §3.2/§4 the `PatchAcceptance` schema and the veto schema | CPython stdlib. | **No** | | stdlib `sqlite3` | §3.2 the durable `PatchAcceptance` store / acceptance ledger | CPython stdlib. | **No** | | stdlib `ast` | §3.6 AST-level diff normalization and overfit-tell matching | CPython stdlib. | **No** | | stdlib `hashlib` | §3.7 tamper-protection file hashes (protected test files) | CPython stdlib. | **No** | | stdlib `subprocess` (sandboxed) | §3.5 differential execution under the pinned, no-network environment | CPython stdlib; sandbox enforced by component 11. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic tiers; the stores | In-repo, pinned interpreter. | **No** | | Component 17 evidence battery | §3.1 the typed evidence the ladder folds in (`assemble_evidence`) | Built by spec 17; reused. | **No** | | Component 06 impact graph | §3.3 change class + §3.4 the guard set | Built by spec 06; reused. | **No** | | Component 11 execution substrate / sandbox | §3.5/§3.7 the harness's own pinned execution | Built by spec 11; reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single semantic-assessment veto (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all tiers use stdlib `json`/`sqlite3`/`ast`/`hashlib`/`subprocess` (with the component-11 sandbox), and the evidence battery, impact graph, and execution substrate are **reused wholesale from components 17, 06, and 11**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The DEFERRED rows (§2 rows 10–13) are a property/fuzz scale-up, a learned verifier ranking, a formal-equivalence backend, or a cross-task acceptance-memory — not new third-party Python installs. Any future trained-model download, fine-tuning run, or learned-judgment dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol Acceptance is measured *separately* and *honestly*, because a stage whose whole purpose is to rule can be defeated in either of two opposite ways: by shipping plausible patches that pass the suite and fix nothing, or by holding correct patches hostage to evidence the harness never bothered to collect. The instruments keep the gate accountable to what each verdict actually demonstrated and what it cost, not to the count of commits. **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, 13, 14, 15, 16, and 17** (Python-only, single-repo, no-Docker), each annotated with the repository's own battery and the gold developer patch, so *correctness* can be judged independently of the harness's own run (by differential/fuzz against the gold plus the gold's own FAIL_TO_PASS/PASS_TO_PASS). Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. Held-out tasks feed the verdict, the evidence ladder, the differential/fuzz attacks, and the outcome statistics; in-benchmark tasks feed nothing that scores the stage. **Baseline (concrete).** The **naive "green = commit" gate** — the failure mode this component rejects: on each candidate patch the harness reads the exit code of `pytest` once, and if green, commits, with **no two-part verdict, no evidence ladder, no risk thresholds, no differential execution, no overfit scan, no tamper audit, no acceptance record**, and "green" is whatever the exit code says. The structured gate must beat *this* on the honest measures below, and the ablation (7.3) removes mechanisms against those same measures. ### 7.1 Correctness-at-acceptance + false-reject rate — the KEY measures The primary instruments, the direct score of the whole stage. **Correctness-at-acceptance** — of all patches the gate ACCEPTed, what fraction are *genuinely correct*, judged by differential/fuzz execution against the gold developer patch and the gold's own FAIL_TO_PASS/PASS_TO_PASS obligations (never by the harness's own suite, never by the model). A gate whose ACCEPTed patches are mostly plausible-but-wrong — Qi et al.'s functionality-deletions shipped — scores near zero no matter how green it was, because it has ruled on a passing suite, not on the patch. **False-reject rate** — of all genuinely correct patches in the sample, what fraction the gate REJECTed or held INCONCLUSIVE when it should have accepted. A gate that accepts nothing is trivially "correct" but blocks every real fix, and this instrument exists to keep the other KEY measure honest. Both are computed deterministically and outside the model, because the model cannot be trusted to judge its own patch. **State explicitly: "Never the resolve rate alone"** — acceptance must not be judged by whether the eventual patch resolved (component 14's measure) but by whether each ACCEPTed patch was actually correct and each correct patch was actually accepted. **State explicitly: "Never model self-assessment"** — a gate scoring highly on the model's own confidence that a patch is correct while its ACCEPTed patches are plausible-not-correct functionality-deletions is the thesis's exact failure and scores near zero here. **These are the KEY measures — the structured gate is adopted only if it STRICTLY beats the naive "green = commit" baseline on BOTH AND does not regress false-reject rate beyond noise (7.4).** ### 7.2 The companion + secondary instruments Report alongside the KEY measures; the secondaries report, do not gate on: 1. **Evidence cost per acceptance** — the companion: how much validation budget was spent per patch actually ACCEPTed — how many patches halted at a cheap tier versus how many burned an expensive differential/fuzz attack that a cheaper tier would have caught. A gate that spends an hour of Level-E attack to learn what a Level-A parse would have shown in milliseconds is paying for information it already had. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). **Tokens** = `tokens / instance`. **Secondary instruments (report, do not gate on):** 1. **INCONCLUSIVE rate** — the fraction of decisions that were INCONCLUSIVE. This rate must be able to be nonzero: a gate that never withholds is not applying the threshold, and a harness whose INCONCLUSIVE rate is always zero has silently converted every "not enough evidence" into an accept. 2. **Overfit-detection recall** — how often a genuinely overfit patch was caught (by the semantic veto, the overfit-tell scan, or the differential attack) before it shipped. Against RePaCA's 85.7% recall and LLM4PatchCorrect's baselines; a veto that fires on fluency rather than overfit shows up as precision collapse. 3. **Tamper-detection rate** — how often the diff-scope check or protected-file reapplication caught a change to a test, fixture, or skip marker. A zero rate is not proof of honesty; it is a signal to check that the tamper audit is actually installed. 4. **Evidence-strength distribution** — the distribution of accepted verdicts over ladder levels A→E, per change class. A gate accepting auth changes at Level C is a proportionality failure with security consequences, and the instrument exists to surface it. 5. **Record completeness** — the fraction of verdicts carrying a complete `PatchAcceptance` record with evidence provenance and unresolved risks. A verdict without a record is a decision the harness cannot later explain. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add an acceptance mechanism because it sounds sensible" mechanically. Two passes, everything else held fixed, the delta measured on the honest measures (correctness-at-acceptance + false-reject rate + evidence cost per acceptance + cost/tokens + secondaries): 1. **Ablate differential execution** — no pre-edit-vs-patched runs: the direct test of P5. 2. **Ablate the semantic veto** — no L entry at all: the direct test of P6 and the at-most-one learned entry. 3. **Ablate risk-sensitivity** — accept everything at a flat Level C: the direct test of P7's proportionality. 4. **Ablate INCONCLUSIVE** — convert every INCONCLUSIVE into ACCEPT: the direct test of P3's first-class verdict. 5. **Ablate the overfit-tell scan** — no static tells: the direct test of the cheapest rung. 6. **Ablate the tamper audit** — no reapplication, no diff-scope check: the direct test of P2's tamper protection. 7. **Ablate the acceptance record** — no provenance: the direct test of P8. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L stub, which must survive the no-L-vs-L ablation to be kept), then to any other mechanism. **Never add an acceptance mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run gate-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured gate **wins iff**: ``` Correctness-at-acceptance STRICTLY beats the naive "green = commit" baseline (KEY) AND false-reject rate not regressed beyond noise (≤ baseline + 5pp) (KEY) AND evidence cost per acceptance STRICTLY beats the naive "green = commit" baseline (companion) AND INCONCLUSIVE rate able to be nonzero (not pinned at zero) (report) AND tamper-detection rate not regressed (a zero rate signals the audit is not installed) AND cost_gate ≤ 1.5 × cost_baseline AND tokens_gate ≤ 1.5 × tokens_baseline ``` The KEY measure **correctness-at-acceptance** must **strictly beat the naive "green = commit" baseline**; **false-reject rate** must not regress beyond noise; the companion must **strictly beat** it; cost and tokens are bounded at 1.5×. **Never the resolve rate alone** — resolve is component 14's measure, and a gate that improves resolve while leaving correctness-at-acceptance flat against the green-to-commit baseline is **not adopted**. **Never model self-assessment** — a gate that reports the model's confidence that a patch is correct while its ACCEPTed patches are plausible-not-correct scores *worse* and is rejected. If the gate wins it replaces the naive green-to-commit path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways an acceptance gate can lie) 1. **Plausible-not-correct acceptance** — the suite too weak to discriminate, the patch passing everything and solving nothing: Qi et al.'s functionality-deletion shipped. Caught by correctness-at-acceptance and the overfit-tell scan. 2. **Damage-blindness** — FAIL_TO_PASS flipping while the neighbors regress, one clause satisfied and the other never checked. Caught by the two-part verdict and the `unbroken?` clause. 3. **Green-by-weakening** — tests edited, skipped, or mocked, or rerun-until-green inherited from a flaky suite. Caught by the tamper audit and the protected-file reapplication. 4. **Circular certification** — the patch validated by tests the same model just wrote to match the patch: agreement with one's own misunderstanding. Caught by the restricted authority of generated tests (they may reject, never alone certify) and the KEY measures' gold judgment. 5. **Special-casing** — the patch returns the expected value for the tested input and the buggy behavior everywhere else: passing tests as a false flag. Caught by the overfit-tell scan and the differential attack. 6. **Review theatre** — an LGTM with no checklist grounding, or the harness routing correctness through review and repeating the expectation that review alone detects defects. Caught by the demote-only veto (which cannot certify) and the record-completeness instrument. 7. **Stale acceptance** — the patch validated against a snapshot that drifted, so by merge time the guard set no longer describes the mainline. Caught by the pinned environment and the recorded pre-edit snapshot the differential run used. ### 7.6 Two honest caveats **First, the strong-model caveat:** the paper's headline numbers — AutoPatchBench's ~60% plausible collapsing to 5–11% correct, RePaCA's 83.1% accuracy / 85.7% recall, Wang–Pradel–Liu's 6.2pp inflation estimate — will **NOT** reproduce on a weak model, and were measured on repositories, models, and vulnerability corpora far larger than a weak-model harness controls; a weak-model harness should expect materially smaller absolute effects while leaning all the harder on the two-part verdict, the risk thresholds, and the deterministic battery to make its small budget count. **Second, the held-out caution:** every acceptance metric must be reported on held-out and out-of-benchmark tasks, so a gate that has memorized a task set's correct patches cannot score well on discrimination without its verdicts having been measured on problems it never saw. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/18-validation-patch-acceptance/`; it never writes into the live harness's source. - A **feature flag** (env var `ACCEPT=18gate|greencommit`, default `greencommit`) routes the A/B; the flag alone switches between the structured gate and the naive "green = commit" baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running old-vs-new is always possible in the same session, repo, model. - The durable `PatchAcceptance` store is append-only and versioned; rolling the build back never loses a verdict. A verdict later falsified by new evidence is kept with its record — never silently dropped, never silently re-typed by a model. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains at most one entry** — the semantic assessment / review gate (§4), the one bounded, seeded call judging root-cause-vs-overfit and minimality/idiom, demote-only, never re-prompted on parse failure. Any phase introducing a model elsewhere (authoring the verdict, deciding whether green counts, waiving a failing tier, promoting a patch on its own confidence, judging a divergence benign, or writing the record) is rejected outright. An **empty** L column (the stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **The generator never judges its own patch** — the semantic assessor is a different agent than the generator, and the verdict is adjudicated harness-side; a model that both proposes and accepts a patch is a confirmation-bias engine (P2, P4). 3. **`fixed?` and `unbroken?` are separate clauses, both required** — a patch is ACCEPTed only when both are established at or above the required evidence level; one clause is no verdict (P1). 4. **The semantic veto demotes but never promotes or overrides** — a "likely overfit" or "non-minimal / unidiomatic" judgment demotes an execution-green patch; a "root-cause-addressed" judgment supports continuation but never overrides a failing tier and never certifies by itself; a hard execution failure always wins (P6). 5. **Insufficient evidence → INCONCLUSIVE, never ACCEPT** — a patch that survives every tier it was given but does not reach its class's required level is INCONCLUSIVE, and a gate whose default under uncertainty is accept has no gate (P3). 6. **Every verdict carries an evidence record** — a complete `PatchAcceptance` record with the syntax result, the B/C reproduction, the regression counts, the impact coverage, the semantic assessment and confidence, the differential and fuzz outcomes, the unresolved risks, the verdict, and the evidence strength; a verdict without a record is a scored defect, never silently accepted (P8). 7. **Risk scales the evidence** — the minimum required level is a deterministic function of the change class, and a patch accepted below its class's level is a proportionality failure, never excused (P7). 8. **All metrics on held-out/out-of-benchmark tasks** — the KEY measures and every secondary are reported on held-out tasks, never tuning or in-benchmark data (§7). --- *End of spec. This document is the build contract; the next document records the A/B outcome.*