Summary / 18. Validation and patch acceptance / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# The Validator's Job Is to Prove the Patch Wrong: Adversarial, Evidence-Thresholded Patch Acceptance for a Weak-Model Coding Harness *Component 18 of 27 — Coding-Harness Program. This document specifies the validation / patch-acceptance stage, and it is precise about what it is not: it is not test execution (component 17 ran the battery and produced typed evidence), and it is not candidate generation (component 16 produced the competing hypotheses). It is the decision layer — the gate that reads all prior evidence and rules whether a candidate patch may become accepted repository state, and on what recorded grounds. The spine distinction is between "it ran" and "it ships": component 17 established that the tests were executed honestly; this component establishes whether the accumulated evidence is sufficient to declare the patch correct, and it is the point where the whole loop's output converges — the traces and competing hypotheses have produced exactly one deliverable, and this is the referee that rules whether the deliverable ships, annotating the accepted/rejected hypothesis in the reflection ledger with its evidence outcome. A green suite in this component is not a verdict, it is one kind of evidence, and the strongest available evidence is a failed attempt to prove the patch wrong. The naive harness asks "did the tests pass?" and commits; this component argues that the only decision worth making is whether the validator, having attacked the patch as hard as the change's risk demanded, failed to break it — and that when the evidence falls short of that bar the correct result is INCONCLUSIVE, never ACCEPT. This paper translates the program's thesis into the acceptance build contract: a two-part verdict (fixed? and unbroken?, never conflated) adjudicated harness-side in the pinned environment; an evidence ladder from syntactic plausibility to behavioral challenge whose acceptance is monotonic; a risk-sensitive threshold mapping change class to minimum required evidence; differential execution against the pre-edit repository as the free executable specification of unchanged behavior; a semantic patch assessment as a veto layer that can demote but never promote; and an explicit acceptance record linking every verdict to its supporting and unresolved evidence — with the invariant that the deterministic evidence battery is the default and the L column holds at most one entry.* --- ## Abstract Patch acceptance is where a coding harness makes its least-forgiving decision on its most-forgivable evidence: a green test suite. This paper argues that "green" is a claim made by the suite, not a verdict about the patch, and that a weak-model harness must therefore make acceptance a separate, adversarial, evidence-thresholded decision — one that requires proof the original failure flipped from fail to pass, requires the preservation of previously correct behavior through impact-aware regression, independently assesses whether the patch addressed the diagnosed root cause rather than overfit the visible tests, actively searches for behavioral counterexamples through differential, property-based, or fuzz testing where the risk warrants it, scales the required evidence to the blast radius of the change, and preserves an explicit record linking every verdict to its supporting and unresolved evidence. The review establishes the case across thirty-one sources. Qi, Long, Achour, and Rinard quantified the founding danger — the overwhelming majority of "plausible" patches in generate-and-validate repair are functionality-deletions that pass the suite and fix nothing [1], with Long and Rinard's search-space analysis the companion reading [2] — and the overfitting-detection lineage that followed (Xin and Reiss's DiffTGen [3], Ye et al.'s static-feature classifier [4], Tian et al.'s representation-learning assessment [5]) built the machinery to catch it before the LLM era. The LLM-era assessment papers — LLM4PatchCorrect [6] and RePaCA [7] — made patch-correctness assessment a trained artifact, and the execution-free verifiers (R2E-Gym [8], SWE-Gym [9], SWE-RM [10], DeepSWE [11], OpenHands Critic [12]) made the acceptance judge itself a trainable component, with a confident-language failure mode that is the single most important caveat against trusting any learned gate. The benchmark audits — Wang, Pradel, and Liu's PatchDiff study [13], SWE-Bench+ [14], ImpossibleBench [15] — showed that a benchmark can still label incorrect patches solved, that thin suites leak resolution, and that "passing" is often tampering. Agentless [16] and SWE-bench [17] supplied the architectural half — validation as a named, harness-owned final phase, resolution defined by executed FAIL_TO_PASS/PASS_TO_PASS verdicts never by self-report — and SpecRover [18] the confidence-score handoff. AutoPatchBench [19] is the security outlier whose generation-to-verification collapse (around 60% plausible, 5–11% correct) is the strongest warning in the series, with QuickCheck [20] the property-based attack it inherits. CodeReviewer [21] and Bacchelli and Bird's review study [22] supply the review gate and its honest human ceiling, while Godlin and Strichman's regression verification [23], SymDiff [24], and Clover [25] mark the formal ceiling where "did we break anything?" becomes a proof rather than an observation. ExpressAPR [26], Lou et al.'s two-million-patch study [27], PatchPilot [28], TestGen-LLM [29], SWE-bench Verified [30], and TestGenEval [31] close the row with the economics of validation and the restricted authority of generated tests. From this review the paper derives eight design principles, then specifies an evidence ladder (A through E) with monotonic acceptance, a risk-sensitive threshold table, and an explicit PatchAcceptance record — all adjudicated by a deterministic battery in which the single learned entry is the semantic assessment, a veto that can demote a verdict but never promote one, with the invariant that the L column holds at most one entry — and closes with a recursive self-improvement protocol and a measurement plan whose instruments exist to keep the harness from mistaking a green suite, a passing benchmark, or a confident model for a correct patch. ## 1. Introduction The coding-harness program has, through components 01 through 17, assembled machinery that reads a repository, localizes a fault, maps its dependencies, generates candidate patches, edits them into place, runs them, distills failures into memory, and — in component 17 — executes the repository's battery honestly, typing every result by the obligation it discharged. Component 18 is the moment where all of that work either becomes repository state or evaporates: it is where the harness decides, on evidence, whether a candidate patch is allowed to ship. Everything upstream produces a claim — "this patch fixes the bug" — and everything downstream trusts that claim. Acceptance therefore fails in exactly one way that matters: it lets a plausible patch through on green that proves nothing, or blocks a correct patch for want of evidence the harness never thought to demand, and in either case the failure is invisible because the harness asked only whether the tests passed. The thesis that governs this component is stated plainly. **Patch acceptance should be a separate, adversarial, evidence-thresholded decision rather than the consequence of a green test suite: require proof that the original failure has changed from fail to pass, preserve previously correct behavior through impact-aware regression testing, independently assess whether the patch addresses the diagnosed root cause rather than overfitting visible tests, actively search for behavioral counterexamples through differential, property-based, or fuzz testing where warranted, scale the required evidence to the risk and blast radius of the change, and preserve an explicit acceptance record linking every verdict to its supporting and unresolved evidence. When evidence is insufficient, the correct result is INCONCLUSIVE—not ACCEPT.** And it carries a strong rule for Piper that compresses the whole thesis into one sentence of discipline, stated verbatim: **The patch generator's job is to make a patch look right. The validator's job is to prove it wrong. Only when the validator fails after sufficiently strong attacks does the patch earn acceptance.** The first clause of the thesis is the discipline of separation — acceptance is its own stage with its own authority, not the tail of generation, because the model that proposed a patch is the worst possible judge of whether it is correct. The second and third clauses are the discipline of the two-part verdict — "fixed?" and "unbroken?" are different questions answered by different evidence, and a patch that satisfies only one is not accepted. The fourth clause is the discipline of the root-cause audit — a patch that special-cases the tested input is passing the test, not solving the problem. The fifth clause is the discipline of the attack — the harness actively seeks the counterexample that breaks the patch, rather than collecting evidence that flatters it. The sixth clause is the discipline of proportionality — a typo fix and an authentication change do not owe the same evidence. The seventh clause is the discipline of the record — every verdict carries the evidence that justified it, so that six months later the harness can answer "why did I accept this?" with provenance instead of "tests passed." The eighth is the discipline of withholding — the gate's default under uncertainty is INCONCLUSIVE, because a harness that auto-accepts on insufficient evidence has no gate at all. The weak-model problem makes acceptance the make-or-break stage for a specific reason that has nothing to do with raw capability: a weak model is simultaneously the least trustworthy author of a patch and the least trustworthy judge of one, and if the same weak model is asked to both generate and validate — asked first "how could this be right?" and then "is this right?" — confirmation bias is guaranteed. The escape is the adversarial division of labor that is this component's spine: generation and validation are played by different agents with different objectives, and where the validator can be made deterministic software, it is made deterministic software, because a verdict a weak model could flatter is not a verdict. The economics make this urgent: AutoPatchBench measured the collapse from plausible to correct — around 60% of generated patches looked superficially viable, but only 5–11% survived real verification, and Gemini 1.5 Pro's 61.1% generation success collapsed to 5.3% correct [19]. That is the loop's whole budget spent on patches that were never going to ship, and it is precisely the fraction an acceptance gate exists to catch before it becomes repository state. The design implication is decisive and it is the thesis: a weak-model harness must make acceptance a separate, adversarial, evidence-thresholded decision, because the alternative — committing on green — is the program's most expensive lie told in its cheapest-looking form. The paper proceeds as follows. Section 2 reviews the acceptance literature in ten parts — the danger quantified (plausibility is not correctness), the classical overfitting-detection lineage, the LLM-era patch-correctness assessors, the execution-free verifiers, the benchmark audits, the validation stage as architecture, the security outlier that actively seeks counterexamples, the review gate and its human ceiling, the formal ceiling of proving damage-absence, and the economics and restricted authority of validation tests. Section 3 derives eight design principles. Section 4 specifies the architecture — the evidence ladder A through E, the risk-sensitive threshold table, the two-part verdict, and the explicit PatchAcceptance record — labeling every node deterministic (D) or learned (L) and stating the invariant that the L column holds at most one entry. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into ten parts that together answer one question: what must an acceptance decision be, so that a weak model's patch is not shipped merely because nothing cheap happened to catch it? Each part supplies one lever the acceptance gate must pull. ### 2.1 The danger quantified: plausibility is not correctness **An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems** (Zichao Qi, Fan Long, Sara Achour, and Martin Rinard, International Symposium on Software Testing and Analysis, ISSTA 2015, DOI 10.1145/2771783.2771791) is the founding quantification of the row's central danger, and it must be cited for its real, stronger finding rather than the weakened paraphrase that has circulated [1]. The paper separates two properties that generate-and-validate repair confuses: *plausibility* — the patch passes the given test suite — and *correctness* — the patch does what the issue actually demands on all inputs, not just the tested ones. Analyzing the plausible patches produced by GenProg and its siblings, the authors found that the overwhelming majority of them were semantically equivalent to a single functionality-deletion modification: 104 of 110 GenProg patches, 37 of 44 RSRepair patches, and 22 of 27 AE patches were, on inspection, functionality-deletions that passed the tests by deleting the behavior the tests did not cover [1]. This is not "about a third of plausible patches are wrong"; it is that the overwhelming majority are wrong in the same, specific way — they delete the failing functionality. The distinction is the row's foundation: a suite defines a lower bound on correctness, and a generate-first-then-pass patch satisfies the bound while violating the intent, and the cheapest way to satisfy the bound is to delete the behavior the bound does not measure. Acceptance therefore cannot be defined as "the failing test passes"; it must add evidence that the patch solved the problem instead of the test. **An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems** (Fan Long and Martin Rinard, International Conference on Software Engineering, ICSE 2016, DOI 10.1145/2884781.2884872) is the companion reading, and it explains *why* the plausible patches are the way they are [2]. By analyzing the search spaces that generate-and-validate systems actually traverse, Long and Rinard show that only a minority of the plausible patches such systems enumerate are correct, and that the search is structurally biased toward the cheap, test-satisfying, behavior-deleting patches Qi et al. observed [1][2]. The transfer is the argument for adversarial architecture rather than better prompts: if the search space itself is biased toward plausible-but-wrong patches, then no amount of generation-side polish fixes the problem — only a validation stage that is structurally independent of the generator, and that is rewarded for finding the wrongness, closes the gap. ### 2.2 The classical overfitting-detection lineage **Identifying Test-Suite-Overfitted Patches through Test Case Generation** (Qi Xin and Steven P. Reiss, International Symposium on Software Testing and Analysis, ISSTA 2017, DOI 10.1145/3092703.3092718) is the DiffTGen paper, and it must be cited with its corrected attribution: the authors are Xin and Reiss, and the venue is ISSTA 2017 — not "Xiong et al., FSE 2018," and certainly not the arXiv identifier 1806.10786, which resolves to an unrelated mathematics paper and must never be used for this work [3]. DiffTGen is the remedy to Qi et al.'s disease: it detects test-suite overfitting by generating new test inputs and executing both the original and the patched program on them, using the original program's behavior on previously passing inputs as a behavioral specification of the unchanged contract and its behavior on failing inputs as the required change. A divergence on an "unchanged contract" input — the original and the patch disagree where they were supposed to agree — is the signature of an overfitted patch, and DiffTGen rejects it with strong precision on patches the original suite had already certified [3]. The transfer is the single cheapest piece of real evidence this component owns: the pre-edit repository is a free, executable specification of everything that was not supposed to change, and differential execution against it is the cheapest real answer to "did we break anything?" — no formal spec required, only the pre-patch program's observed behavior. **Automated Classification of Overfitting Patches with Statically Extracted Code Features** (He Ye, Jian Gu, Matias Martinez, Thomas Durieux, and Martin Monperrus, IEEE Transactions on Software Engineering, 2021, arXiv:1910.12057) is the ODS paper — the static-feature overfitting classifier that runs on the patch text without generating a single new test [4]. It extracts syntactic and semantic features from the patch and the surrounding code and classifies a patch as overfitting or not, demonstrating that a meaningful overfit signal lives in the patch's *shape* — its size, its locality, the kinds of edits it makes — before any behavior is observed. The transfer is the cheapest rung of the semantic assessment: before spending a differential battery, the harness can look at the patch and ask whether it *looks like* a repair (a small, local, root-cause edit) or a workaround (a hard-coded constant, a special case, a deleted branch) — the "overfit tells" of Section 3 are Qi et al.'s functionality-deletions operationalized as a static scan. **Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair** (Haoye Tian, Kui Liu, Abdoul Kader Kaboré, Anil Koyuncu, Li Li, Jacques Klein, and Tegawendé F. Bissyandé, International Conference on Automated Software Engineering, ASE 2020, arXiv:2008.02944) closes the classical lineage by learning a representation of the code *change* itself — the diff, not the file — and predicting correctness from it [5]. Its citation is corrected here: the informal descriptor "Is this the correct patch?" is not the paper's title, and the real title and identifier above are the ones used. The transfer is the unit of analysis: acceptance operates on the *change* — the diff — not on the resulting file, and a correct judgment of a patch is a judgment about whether this specific transformation, in this specific context, is a genuine repair. That is the same unit the review gate of Section 2.8 operates on, and it is why the acceptance record of Section 4 is keyed to a patch, not to a repository state. ### 2.3 The LLM-era patch-correctness assessors **Leveraging Large Language Model for Automatic Patch Correctness Assessment** (Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo, IEEE Transactions on Software Engineering, Vol. 50, No. 11, pp. 2865–2883, November 2024, DOI 10.1109/TSE.2024.3452252) is the LLM4PatchCorrect paper, and it is cited here with its venue and DOI because no arXiv identifier exists for it [6]. (The identifier 2303.00202 that has been associated with it is wrong — that is "PatchZero: Zero-Shot Automatic Patch Correctness Assessment," a distinct Zhou-et-al. paper — and must not be used.) The paper carries automated patch correctness assessment (APCA) from the classical era into the LLM era: an LLM given in-context examples of labeled patches predicts correctness without running anything beyond the tests already available, beating the prior learned APCA approaches on the same Defects4J-derived benchmarks [6]. Its value to this component is the ready-made taxonomy it inherits from a decade of APCA work — incomplete fixes, regressions outside the test scope, and test-specific hacks are the named species of overfitting an acceptance gate should look for — and its caveat is the one all learned assessors share: it predicts, it does not decide, and the harness must treat its output as one veto among many, never as the verdict. **RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment** (Marcos Fuster-Pena, David de-Fitero-Dominguez, Antonio Garcia-Cabot, and Eva Garcia-Lopez, Neurocomputing, Elsevier, 2026, arXiv:2507.22580, DOI 10.1016/j.neucom.2026.134583) is the most directly relevant recent paper to the acceptance problem, and its citation is corrected here: the identifier is 2507.22580 and the authors are Fuster-Pena, de-Fitero-Dominguez, Garcia-Cabot, and Garcia-Lopez [7]. RePaCA trains a reasoning-oriented LLM to distinguish correct from overfitting patches given the buggy and patched source, reporting 83.1% accuracy, 84.8% F1, and 85.7% recall on overfitting patches, with a Qwen2.5-Coder-3B backbone selected partly for computational feasibility — a backbone unusually close to the constrained-model spirit of this harness [7]. The transfer is the semantic assessment of Section 4: an explicit, trained judgment of whether the code transformation looks like a genuine repair or a test-specific workaround. Its boundary is the same one drawn for LLM4PatchCorrect: a semantic assessor may *reject* a suspicious patch and may *support* the acceptance pipeline, but it may never override a hard execution failure, because no static judgment of "does this look like a repair" is stronger than a concrete counterexample. ### 2.4 The execution-free verifiers: the judge as a trainable artifact **R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents** (Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica, Conference on Language Modeling, COLM 2025, arXiv:2504.07164) is the paper that made "which kind of verifier" an empirical question [8]. It compares execution-based verifiers — that evaluate patches through test cases — with execution-free verifiers — that assess trajectories through learned models — and finds complementary strengths: execution-based verifiers give direct correctness signals but struggle to discriminate between solutions with the same pass count, while execution-free verifiers discriminate better but can be misled, the illustrative failure being a verifier attending to confident language in the agent's own thoughts ("Successfully," "Great," "works") when predicting correctness [8]. The hybrid — tests to filter, a learned model to rank — became the default test-time-scaling recipe, and that confident-language failure mode is the single most important caveat for anyone using an LLM judge as a gate: a learned verifier can be right for the wrong reason, and the wrong reason is often the model's own fluency. **Training Software Engineering Agents and Verifiers with SWE-Gym** (Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang, arXiv 2024, arXiv:2412.21139) supplied the first trained trajectory verifier, and **SWE-RM: Execution-free Feedback For Software Engineering Agents** (KaShun Shum, Binyuan Hui, Jiawei Chen, Lei Zhang, X. W., Jiaxi Yang, Yuzhen Huang, Junyang Lin, and Junxian He, arXiv 2025, arXiv:2512.21919) scaled it up, training an execution-free verifier on roughly 100k trajectories across about 170 repositories with 256k context [9][10]. SWE-RM's related-work framing is the cleanest statement of why execution alone is not enough for acceptance: execution-based verifiers cannot distinguish patches with the same number of passing tests and inherit the unreliability of poorly written or model-generated tests, whereas execution-free verifiers give a continuous score for finer discrimination [10]. Read together, they establish that the acceptance judge is a trainable artifact in its own right, whose quality scales with data the same way the generator's does — and, by the same token, that the judge must be trained and audited as carefully as the generator, not trusted because it is downstream. **DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks** (Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge, arXiv 2026, arXiv:2607.07946) and **A Rubric-Supervised Critic from Sparse Real-World Outcomes** (Xingyao Wang, Valerie Chen, Heng Ji, and Graham Neubig, arXiv 2026, arXiv:2603.03800) round out the execution-free verifier family that SWE-RM benchmarks against [11][12]. DeepSWE pairs a 113-task long-horizon benchmark with hand-written verification scripts built to defeat pretraining recall and fix-specific tests, and its execution-free verifier is benchmarked against its execution-based variant inside SWE-RM [11]. OpenHands Critic is an LLM verifier that scores agent trajectories without executing code, trained from sparse real-world outcomes [12]. Their shared lesson for this component is boundary, not core: an execution-free judge is a steering and ranking instrument, valuable where the harness cannot run the suite, but it is not a substitute for the deterministic battery — it is exactly the learned entry whose confident language R2E-Gym warns against [8], and it belongs in the L column, not the D column. ### 2.5 The benchmark audits: "solved" is not "correct" **Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study** (You Wang, Michael Pradel, and Zhongxin Liu, arXiv 2025, arXiv:2503.15223) attacks the exact assumption this component must not make: that passing benchmark validation means the issue is actually solved [13]. Re-examining patches the benchmark classified as solved, the authors found that 7.8% of analyzed patches were counted correct despite failing developer-written tests, that 29.6% of plausible patches behaved differently from the developer's ground-truth patch, that manual inspection found 28.6% of behaviorally divergent patches certainly incorrect, and that these weaknesses inflated reported resolution rates by 6.2 absolute percentage points [13]. The paper's remedy is PatchDiff, a differential patch-testing technique that searches for behavioral differences between the candidate patch and the developer patch rather than asking whether both pass the available suite [13]. The transfer is the behavioral-divergence evidence class: even when every known test is green, the harness should ask whether the candidate and the reference disagree on some input, and the disagreement — not the shared green — is where the overfit becomes visible. **SWE-Bench+: Enhanced Coding Benchmark for LLMs** (Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang, arXiv 2024, arXiv:2410.06992) is the audit of what "resolved" actually meant in the benchmark this program inherits [14]. Manually inspecting SWE-bench "successes," the authors found 32.67% had the solution directly leaked in the issue text or comments and 31.08% were suspicious patches that passed owing to inadequate tests, with resolution rates dropping sharply once those were removed [14]. **ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases** (Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini, arXiv 2025, arXiv:2510.20270) measures the attack surface the tamper protection exists to close: how often agents "pass" by exploiting the checks rather than satisfying them [15]. Together the three make the empirical case that "tests pass" is an acceptance *heuristic*, that the suite itself is the agent's optimization target, and that whatever the battery measures the loop will deliver — so a gate that accepts on green, without auditing what the green actually demonstrated, is not a gate but a rubber stamp. ### 2.6 The validation stage as architecture **Agentless: Demystifying LLM-Based Software Engineering Agents** (Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang, arXiv 2024, arXiv:2407.01489) is the architectural half of this component's strongest directly applicable pair, because it turns patch validation into its own named, harness-owned pipeline phase — LOCALIZE, REPAIR, VALIDATE — executed by scripts, never left to the model's self-report [16]. Its validation combines existing regression tests (did the patch break behavior that worked before?), generated reproduction tests (did the patch actually fix the issue?), and AST-normalized candidate comparison with majority voting, reaching 32.00% on SWE-bench Lite at about $0.70 per issue [16]. The transfer is the separation principle made concrete: the model proposes, the harness disposes, and the model's own opinion of its patch is nowhere in the decision path. **SWE-bench: Can Language Models Resolve Real-World GitHub Issues?** (Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan, ICLR 2024, arXiv:2310.06770) is the contract the acceptance gate inherits: resolution is defined by executed verdicts — FAIL_TO_PASS tests must flip, PASS_TO_PASS tests must hold — never by self-report, and the evaluation runs harness-side in per-instance Dockerized environments with the gold test patch applied mechanically over the model's patch so test-file tampering is neutralized by construction [17]. That two-sided vocabulary — the bug-fixing tests and the regression-preservation tests are different obligations, and a patch is resolved only when both are discharged — is the exact distinction this component's two-part verdict institutionalizes. **SpecRover: Code Intent Extraction via LLMs** (Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury, ICSE 2025, arXiv:2408.02232) attaches a confidence score to each accepted patch: beyond running regression tests per candidate, a reviewer agent interprets failures against the issue's extracted intent and exposes a confidence score so a downstream human or gate can decide whether to trust the patch [18]. The transfer is the boundary of the learned entry: interpreting a failure's meaning against the issue's intent is legitimate steering, but it is downstream of — and cannot substitute for — the deterministic verdict that a specific test failed. The confidence score is a handoff, not a ruling. ### 2.7 The security outlier: actively seek the counterexample **AutoPatchBench: A Benchmark for AI-Powered Security Fixes** (TJ Byun, Cornelius Aschermann, Kai Yuan Thng, Wu Zhou, Yupeng Yang, Lauren Deason, and Joshua Saxe, Engineering at Meta, 2025-04-29) is the outlier this component leans on hardest, and it is cited here with its blog venue and no arXiv identifier, because none exists and none should be fabricated [19]. Built from 136 real C/C++ fuzzing-discovered vulnerabilities with a 113-instance Lite subset, its acceptance pipeline goes well beyond "does it compile and did the original crash disappear?": build verification, original-crash reproduction, fuzz testing, and white-box differential testing against the human repair [19]. The result is the strongest warning in the whole series: models produced superficially viable patches at roughly 60%, but after deeper verification only about 5–11% of tasks remained successful depending on model — for Gemini 1.5 Pro, 61.1% generation success collapsed to 5.3% correct [19]. That is the collapse from *plausible* to *correct* the acceptance gate exists to catch. Fuzzing is the philosophical turning point: ordinary tests exercise examples developers thought of; fuzzing actively searches the input space for a counterexample to the patch — a different acceptance philosophy, "don't merely collect supporting evidence, actively try to falsify the patch." The caution is equally important: even differential testing is not magic; in the sampled validation it achieved 84.1% agreement but only 41.7% precision, admitting some patches human experts rejected [19]. Differential testing is strong additional evidence, not proof of correctness. **QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs** (Koen Claessen and John Hughes, International Conference on Functional Programming, ICFP 2000, DOI 10.1145/351240.351266) is the property-based ancestor of the fuzz tier [20]. Its contract is the one the behavioral-challenge rung inherits: instead of testing examples, the harness checks a *property* over randomly generated inputs, and the first counterexample is the patch's death certificate. Where a property exists — and typed code increasingly supplies them — the cheapest strong attack is not more tests but a property that the patch must not violate, run until it finds the violation or the budget runs out. ### 2.8 The review gate and its honest human ceiling **Automating Code Review Activities by Large-Scale Pre-training** (Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan, ESEC/FSE 2022, arXiv:2203.09095, DOI 10.1145/3540250.3549081) is the CodeReviewer paper, and it is the pre-LLM proof that diff-level defect detection — reviewing the *change*, not the file — is learnable and automatable [21]. It performs the three things reviewers do to a diff: quality estimation, comment generation, and code refinement, establishing that the unit acceptance operates on (the diff) is the unit a trained model can judge. The transfer is the non-execution component of acceptance: "is this the right minimal change, done the way this repository does things?" is model-checkable against the repository's own conventions, and it is a facet of the single learned entry this component admits — the review gate, one face of the same learned veto as the semantic assessment of Section 4. **Expectations, Outcomes, and Challenges of Modern Code Review** (Alberto Bacchelli and Christian Bird, International Conference on Software Engineering, ICSE 2013, DOI 10.1109/icse.2013.6606617) is the human-gate reality check, cited here with its corrected venue — it is ICSE 2013, not ESEM 2013 [22]. The study's large analysis of industrial and open-source review practice finds that developers *expect* review to detect defects, but the outcomes are dominated by code improvement, knowledge transfer, and team awareness — explicit defect detection is comparatively rare [22]. For harness design this is both a warning and an assignment: do not route the loop's entire correctness burden through a review-shaped stage, because humans never carried it that way; but do use review for what it empirically delivers — catching the diffusely obvious quality problems and building shared understanding of the change. An agent-generated diff summary for human handoff is not ceremony; it is the empirically validated function of the gate. The transfer is the split of Section 4: *mechanical* correctness (executed evidence, the deterministic battery) is cleanly separated from *reviewable* change (summary, scope, idiom), and neither substitutes for the other. ### 2.9 The formal ceiling: proving damage-absence instead of sampling it **Regression verification** (Benny Godlin and Ofer Strichman, Proceedings of the 46th Annual Design Automation Conference, DAC 2009, DOI 10.1145/1629911.1630034) is the outlier that marks the mathematical ceiling of this component's second question [23]. Instead of *testing* for damage, it *proves* its absence: given two related program versions, it generates mutual summary functions that let an SMT solver establish behavioral equivalence of the unchanged parts under unbounded inputs. Equivalence testing can only sample; regression verification decides, for restricted but real program classes [23]. The transfer is a confidence-budget discipline: most patches will never admit a proof, but *sub-parts* of many patches do — a rename, a mechanical refactor, a migration — and where proof is available, the second clause of the verdict ("unbroken?") becomes a certificate rather than evidence. **SymDiff: A Language-Agnostic Semantic Diff Tool for Imperative Programs** (Shuvendu Lahiri, Chris Hawblitzel, Ming Kawaguchi, and Henrique Rebêlo, Computer Aided Verification, CAV 2012) is the industrial descendant of that idea, cited here with its corrected attribution — the authors are Lahiri, Hawblitzel, Kawaguchi, and Rebêlo, and the venue is CAV 2012, not "Lahiri, McMillan & Sharma, CAV 2013," which is the related-but-distinct Differential Assertion Checking paper [24]. SymDiff computes a semantic diff between two versions of a program, and it is the tool-shaped statement of the row's differential principle: the unchanged contract is an object a tool can reason about, not just a hope. **Clover: Closed-Loop Verifiable Code Generation** (Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett, arXiv 2023, arXiv:2310.17807) is the existence proof that acceptance can be an *argument* rather than an observation [25]. Clover checks three artifacts against each other — the code, its docstring, and its formal annotations in Dafny — and accepts only when all pairwise consistency checks pass: the code satisfies the annotations (proved by the verifier), the annotations match the docstring, and the docstring matches the code, with the LLM doing the informal-to-formal translations and a theorem prover doing the rest [25]. It never asks "did the tests pass." It is impractical for a Django PR today, but it is the clearest demonstration that "did we fix it without breaking anything" can be discharged by proof, and as the harness writes more typed code the cost of that argument keeps falling. ### 2.10 The economics and the restricted authority of validation tests **Accelerating Patch Validation for Program Repair with Interception-Based Execution Scheduling** (Yuan-An Xiao, Chenyang Yang, Bo Wang, and Yingfei Xiong, IEEE Transactions on Software Engineering, Vol. 50, No. 3, pp. 618–635, 2024, arXiv:2305.03955) is the ExpressAPR paper, cited here with its corrected venue — it is IEEE TSE 2024, not "ACM DL 2023" [26]. It accelerates patch validation by two orders of magnitude over plain validation by systematically adapting five families of mutation-testing acceleration techniques, letting the harness validate more candidates per budget [26]. The transfer is the cost side of the acceptance gate: every tier of the evidence ladder buys confidence at a price, and the harness must be able to afford the tier its own threshold demands. **When Automated Program Repair Meets Regression Testing — An Extensive Study on 2 Million Patches** (Yiling Lou, Jun Yang, Samuel Benton, Dan Hao, Lin Tan, Zhenpeng Chen, Lu Zhang, and Lingming Zhang, ACM Transactions on Software Engineering and Methodology, 2024, DOI 10.1145/3672450, arXiv:2105.07311) is the RTS-for-APR study, cited here with its real title (the paraphrase "Regression Test Selection for APR Validation" is a descriptor, not the title) [27]. Its two-million-patch study of regression-test selection and prioritization for APR validation establishes that finer-grained selection buys real efficiency, and that the right metric of repair efficiency is the number of test executions, not the number of patches [27]. The transfer is the proportional-threshold discipline: the acceptance gate inherits the test-execution row's impact-aware selection, and the threshold table of Section 4 scales the *amount* of regression evidence to the change class so the harness does not run two million tests to accept a comment. **PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification** (Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo, arXiv 2025, arXiv:2502.02747) is cited here with its real title — the descriptor "multi-stage validation with in-generation checks" is a paraphrase of its Validation stage, not a separate paper [28]. Its Reproduction→Localization→Generation→Validation→Refinement workflow, with in-generation lint and syntax checks and formal-verification attempts, is the in-loop form of the acceptance discipline: validation is not a single terminal event but a recurring check, with the cheapest gates run as early as possible [28]. **Automated Unit Test Improvement using Large Language Models at Meta** (Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang, arXiv 2024, arXiv:2402.09171) and **TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark** (Kush Jain, Gabriel Synnaeve, and Baptiste Rozière, arXiv 2024, arXiv:2410.00752) close the row with the restricted authority of generated tests [29][31]. TestGen-LLM applies this row's rules to a different artifact — generated tests are filtered through run, pass, coverage, and flake checks before human review [29]. TestGenEval, cited here with its corrected authorship (Jain, Synnaeve, and Rozière — not "Löbel et al.", a name that appears on no TestGenEval paper), establishes test generation under realistic repository conditions as a distinct, measurable skill [31]. The shared transfer is the deliberate asymmetry this component adopts: generated tests may *reject* a patch — overfit detection, DiffTGen-style — but may never alone *certify* one, because a generated test that encodes the same misunderstanding as the patch validates nothing, and certification flows only from the pre-existing, human-owned suite and the deterministic battery. **Introducing SWE-bench Verified** (OpenAI, 2024-08-13) is the audit of the acceptance criterion itself, cited with no arXiv identifier because none exists [30]. Its 500 human-verified instances of SWE-bench — kept from 2,294 after discarding a large share for faulty or underspecified tests — are the reminder that the referee needed refereeing, and that the acceptance gate's own criterion is an object worth auditing, not an axiom [30]. ## 3. Design Principles Eight principles follow from the review, each tied to the evidence above and each a clause of the strong Piper rule made mechanical. **P1 — Acceptance is a two-part verdict, never conflated.** "Fixed?" and "unbroken?" are different questions answered by different evidence: the targeted FAIL_TO_PASS tests must flip under harness execution, AND the guard set — the impact row's affected tests plus the PASS_TO_PASS neighborhood — must hold. A patch that fixes the bug while destroying its neighbors, and a patch that preserves everything while never fixing the defect, each fail a different obligation, and each is unaccepted with its failure typed to the clause it violated [17]. One clause is not half a verdict; it is no verdict. **P2 — The verdict is adjudicated harness-side; the model's "tests pass" is a claim, not a fact.** The certification run executes in the pinned environment — containerized parity with CI, locked dependencies, seeded randomness, no network — through the harness's own invocation, with test files tamper-protected: protected test files reapplied and compared, or diff-scope-checked so no test, fixture, or skip marker changed, because deleting a flaky test or weakening an assertion is not a fix [17][15]. The model's report that everything passed triggers a re-run; it never substitutes for one. **P3 — Evidence is monotonic and thresholded; INCONCLUSIVE is a first-class verdict.** Acceptance is monotonic in evidence strength — a patch that has passed a stronger challenge has automatically passed every weaker one — and a patch is accepted only when it reaches the evidence level its change class demands. When the evidence falls short, the correct result is INCONCLUSIVE, never ACCEPT: a patch for which the harness cannot prove enough does not become accepted merely because nothing cheap happened to catch it. A gate whose default under uncertainty is accept has no gate. **P4 — The division of labor is adversarial.** Generation asks "how could this patch be right?"; validation asks "how can I prove it wrong?" The two roles are played by different agents with different objectives, and where the validator can be deterministic software, it is, because a validator that shares the generator's context and objective is a confirmation-bias engine [19]. The validator's success metric is the counterexample it finds, not the green it confirms. **P5 — The pre-edit repository is a free executable specification.** Differential execution against the pre-edit state answers "did we break anything that worked?" with the cheapest real evidence there is: run the original and the patched program on inputs where the original was correct, and treat any divergence as a regression — DiffTGen's unchanged-contract check, generalized from the original paper's test-generation setting to the harness's full differential battery [3][13][19]. The old code is the specification of everything that was not supposed to change. **P6 — The semantic assessment is a veto, never an override.** The root-cause-vs-overfit judgment — does the patch address the diagnosed cause, or does it hard-code the tested value, special-case the issue, or delete the failing behavior? — can only *demote* a verdict (ACCEPT to INCONCLUSIVE or REJECT); it can never *promote* one [1][7]. A hard execution failure always wins over a confident semantic assessment, because no static judgment of "does this look like a repair" is stronger than a concrete counterexample. The asymmetry is deliberate: the assessor is a safety net, not a fast lane. **P7 — Risk scales the required evidence.** A typo fix and an authentication change do not owe the same evidence. The change class — computed from the change radius, API exposure, persistence sensitivity, security sensitivity, and coverage — maps to a minimum required evidence level, so the harness spends its validation budget in proportion to the blast radius of the change [19][27]. This is the proportionality clause of the thesis, and it is what keeps the gate from running two million tests to accept a comment. **P8 — Every verdict carries an explicit acceptance record.** The PatchAcceptance record links the verdict to its supporting and unresolved evidence — the syntax result, the fail-to-pass reproduction, the regression counts, the impact coverage, the semantic assessment and its confidence, the differential and fuzz outcomes, and the list of risks that remain unresolved — so that six months later the harness can answer "why did I accept this?" with provenance, not "tests passed" [18]. Rejected patches keep their prune reasons and the tier that caught them; the record is the loop's institutional memory of its own refereeing. ## 4. The Architecture The acceptance stage is a decision function over a structured evidence battery; every node is labeled D (deterministic software) or L (a learned model). The invariant that governs the whole is: **the L column contains at most one entry, and the deterministic battery is the default.** The evidence battery — syntax, reproduction, regression, differential, and fuzz — is deterministic software on CPU, and the verdict is computed by the harness. The single L entry is the *semantic assessment / review gate*: the root-cause-vs-overfit judgment of P6 combined with the minimality-and-idiom judgment of the review gate [21], two facets of one learned veto — the one place a learned model legitimately contributes, and even there it contributes as a veto, able to demote a verdict, never to promote one, and never to override a deterministic failure. The concrete input is a candidate patch, the buggy baseline, the pre-edit repository, and the evidence already produced by component 17 plus the change-impact and coverage information from components 06 and 12; the output is a verdict — ACCEPT, REJECT, or INCONCLUSIVE — together with a PatchAcceptance record. The stages, in order: **1. Evidence ladder (A → E) [D].** The evidence is assembled in five monotonically stronger levels, each subsuming the last. *Level A — syntactic plausibility*: the patch parses, lints, type-checks, and builds in the pinned environment; not remotely sufficient, but a failure here halts immediately. *Level B — plausible repair*: the bug-specific reproducer — the repository test that exposes the defect, or a generated reproduction test validated against the buggy baseline — demonstrates fail-on-buggy and pass-on-candidate, so the original failure is proven to have changed from fail to pass. *Level C — regression-safe*: the guard set holds — the impacted unit tests and the affected regression subset, selected from dependency and coverage, all PASS_TO_PASS obligations in the affected neighborhood — so previously correct behavior is preserved through impact-aware regression testing. *Level D — semantically supported*: the semantic assessment finds the patch addresses the diagnosed root cause rather than overfitting the visible tests, with a recorded confidence. *Level E — behaviorally challenged*: differential execution against the pre-edit repository, property-based checks, and — where the change class warrants it — fuzz testing actively search for a behavioral counterexample and find none. Acceptance is monotonic in evidence strength: a patch accepted at a higher level has passed every lower one, and a patch that fails any level fails the ladder at that level, with the failing tier itself recorded as debugging evidence. **2. Risk-sensitive thresholds [D].** The minimum required evidence level is a deterministic function of the change class, which the harness computes from the impact analysis of component 06 — change radius, API exposure, persistence sensitivity, security sensitivity, and coverage. The mapping: *comment/documentation* requires Level A (syntax/build where applicable); *local refactor* requires Level C (targeted plus impacted regression); *normal bug fix* requires Level C (reproducer plus impacted/full regression); *public API/schema* requires Level D (the above plus compatibility and impact validation); *persistence/data migration* requires Level D (the above plus migration and round-trip tests); *auth/security* requires Level E (the above plus adversarial, fuzz, and property validation); *concurrency* requires Level E (the above plus repeated, stress, and race validation). A patch that reaches only a lower level than its class demands is INCONCLUSIVE, never accepted — the risk, not the patch's confidence, sets the bar. **3. The two-part verdict [D].** The harness computes two clauses separately and never conflates them: *fixed?* — the FAIL_TO_PASS tests flip under harness execution; *unbroken?* — the guard set holds. The verdict is ACCEPT only when both clauses are established at or above the required evidence level. One clause is unaccepted, with the verdict typed to the missing clause. REJECT is returned when any deterministic tier fails; INCONCLUSIVE is returned when the patch survives every tier it was given but the evidence does not reach its class's required level — the first-class verdict of P3. **4. (L) The semantic assessment — the at-most-one learned entry.** A learned assessor — RePaCA-class [7], or a budget-appropriate reasoning model — reads the patch, the buggy code, and the issue, and judges whether the transformation addresses the diagnosed root cause or overfits the visible tests, attending specifically to Qi et al.'s functionality-deletions and the APCA taxonomy's incomplete fixes, out-of-scope regressions, and test-specific hacks [1][6]. The same single entry also carries the review facet — the minimality, scope-containment, and idiom judgment of CodeReviewer's diff-level check [21] and Bacchelli and Bird's empirically grounded division of labor [22]. It reports a judgment and a confidence. Its authority is a veto only: a "likely overfit" or "non-minimal / unidiomatic change" demotes an execution-green patch to INCONCLUSIVE or REJECT; a "root-cause addressed" supports continuation but never overrides a failing tier and never certifies by itself. It is the single L entry, and the invariant is enforced mechanically: **the L column contains at most one entry.** **5. The explicit acceptance record [D].** Every verdict is written as a PatchAcceptance record with evidence provenance — the patch identifier; the syntax result; the fail-to-pass reproduction; the regression pass/fail counts; the impact-test coverage; the semantic assessment and its confidence; the differential outcome (e.g., "no divergence in 2,400 probes"); the fuzz outcome (including "not_required" with the reason from the threshold table); the unresolved risks; the verdict; and the evidence strength. Rejected patches keep their prune reasons and the tier that caught them; INCONCLUSIVE patches keep the specific missing evidence that, if obtained, would reopen the decision. The record is appended to the reflection ledger, closing the loop between the competing hypotheses of component 16 and the referee's ruling. The cross-cutting deterministic services that make the gate a decision function rather than a script: `compute_change_class` (the impact-analysis-derived risk classification of P7), `require_evidence_level` (the class-to-level threshold mapping), `assemble_evidence` (fold component 17's typed results into the ladder levels), `compute_two_part_verdict` (the fixed?/unbroken? conjunction of P1), `differential_execute` (run original and patch on unchanged-contract inputs and flag divergence, P5), `run_property_and_fuzz` (the Level E attack, gated by change class), `apply_semantic_veto` (the L entry, demote-only), and `write_acceptance_record` (the PatchAcceptance provenance of P8). Every one except `apply_semantic_veto` is deterministic software on CPU. The target is explicit and checkable: the ladder, the thresholds, the two-part verdict, the differential battery, and the record are software, and the L column holds at most one entry — the semantic assessment, contributing its only legitimate value (a veto against overfit) and never the verdict. Any proposed addition that would put a language model elsewhere in the gate — having the model decide whether green counts, waive a failing tier, promote an execution-green patch on its own confidence, or author the final verdict — is a regression against P2 and the Piper rule, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of Section 6, on held-out tasks, at a defensible cost, or be rejected. The invariant is stated again so it can be enforced mechanically: **"the L column contains at most one entry."** ## 5. Recursive Self-Improvement Protocol Component 18 of 27 follows the program's uniform protocol: the executable specification is written to 100%, round 1 lands at approximately 80% of the specified behavior, and later rounds close the gap. Each round is A/B-gated against the honest metrics of Section 6 — above all the plausible-to-correct collapse ratio and the INCONCLUSIVE rate — never the acceptance count alone. The three-way split disciplines every change: the learned veto goes to the LLM's single entry (the semantic assessment), the mechanics go to deterministic software (the ladder, the thresholds, the two-part verdict, the differential battery, the record), and every change to either is A/B-gated on honest metrics, never on vibes. Because acceptance decides what "fixed" even means, its improvement protocol measures the *discrimination* and *proportionality* of the gate rather than trusting the model's sense that a patch is correct, improving itself in four loops. First, *threshold calibration*: the harness measures, per change class, whether the required evidence level actually separates correct from plausible patches — whether patches accepted at Level C that later regressed would have been caught at Level D, and whether patches held INCONCLUSIVE at Level E were in fact correct — and adjusts the class-to-level mapping, guided by AutoPatchBench's collapse ratio [19] and Lou et al.'s two-million-patch efficiency [27]. Second, *veto calibration*: the harness measures the semantic assessment's demote precision — how often an "overfit" veto was later confirmed by a counterexample versus a false alarm — and recalibrates the confidence at which a veto fires, guided by RePaCA's 83.1% accuracy and 85.7% recall [7] and LLM4PatchCorrect's learned baselines [6]; a veto that fires on every patch is a gate that never opens, and one that never fires is a gate that does not exist. Third, *differential-probe tightening*: the harness measures the counterexample yield of the Level E attack — how many execution-green patches the differential, property, and fuzz tiers broke that the suite certified — and widens the probe budget where the yield is high, guided by AutoPatchBench's finding that even 84.1%-agreement differential testing admits wrong patches [19]; an attack that never finds a counterexample is not evidence of correctness, it is evidence the attack is weak. Fourth, *record-completeness auditing*: the harness measures what fraction of verdicts carry a complete PatchAcceptance record and, six months later, whether the records would have supported the decisions — closing the loop between the recorded evidence and the observed outcome, guided by the provenance discipline that P8 exists to enforce. Because every verdict carries its evidence, a regression in acceptance is visible in the discrimination and proportionality metrics before it surfaces as a wrong ship, and each A/B change must justify itself on held-out tasks at a defensible cost before it ships. ## 6. Measurement & Evaluation Acceptance is measured *separately* and *honestly*, because a stage whose whole purpose is to rule can be defeated in either of two opposite ways: by shipping plausible patches that pass the suite and fix nothing, or by holding correct patches hostage to evidence the harness never bothered to collect. The instruments keep the gate accountable to what each verdict actually demonstrated and what it cost, not to the count of commits. **Plausible-to-correct collapse ratio.** The primary instrument reports, of all patches that passed the visible suite, what fraction survived the acceptance gate, and — where the ground truth is known — what fraction of *accepted* patches were actually correct. AutoPatchBench's numbers are the calibration: around 60% plausible collapsing to 5–11% correct is the shape of an honest gate's job, and a harness whose acceptance gate passes nearly everything that is green is not validating, it is rubber-stamping [19]. **Two-clause rate.** The instrument reports, of all accepted patches, what fraction established *both* clauses of the verdict — fail-to-pass flips AND a held guard set. A patch accepted with only the bug fixed and its neighbors unexamined is the "fixed it by breaking it elsewhere" failure shipped, and the instrument exists to make that fraction zero [17]. **INCONCLUSIVE rate.** The instrument reports the fraction of decisions that were INCONCLUSIVE. This rate must be able to be nonzero: a gate that never withholds is not applying the threshold, and a harness whose INCONCLUSIVE rate is always zero has silently converted every "not enough evidence" into an accept — the exact auto-accept the thesis forbids. **Veto precision and recall.** The instrument reports, for the semantic assessment, how often an "overfit" veto was later confirmed by a counterexample (precision) and how often a genuinely overfit patch slipped through unvetoed (recall), against RePaCA's 83.1% accuracy and 85.7% recall and LLM4PatchCorrect's baselines [7][6]. A learned veto that fires on fluency rather than overfit — R2E-Gym's confident-language failure [8] — shows up here as precision collapse. **Differential counterexample yield.** The instrument reports how many execution-green patches the differential, property, and fuzz tiers broke. A Level E attack that never finds a counterexample is either facing perfect patches or a weak attack, and the audit loop of Section 5 treats the two as indistinguishable until proven otherwise [19][20]. **Risk-threshold adherence.** The instrument reports, per change class, what fraction of patches were required to reach the level their class demands, and what fraction were accepted below it. A harness that accepts an auth change at Level C is a proportionality failure with security consequences, and the instrument exists to surface it [19]. **Record completeness.** The instrument reports what fraction of verdicts carry a complete PatchAcceptance record with evidence provenance and unresolved risks. A verdict without a record is a decision the harness cannot later explain, and six months later "why did I accept this?" must have an answer better than "tests passed" [18]. **Cost per verdict.** The instrument reports the validation budget spent per verdict, and per patch that was rejected at each tier. This is the proportionality clause's other face: the gate must afford the tier its own threshold demands, and ExpressAPR's two-orders-of-magnitude acceleration [26] and Lou et al.'s test-execution metric [27] are the calibration for what "affordable" means. A failure-mode audit closes the positive instruments by naming how an acceptance gate lies. *Plausible-not-correct acceptance* is the suite too weak to discriminate, the patch passing everything and solving nothing — Qi et al.'s functionality-deletion shipped [1]. *Damage-blindness* is FAIL_TO_PASS flipping while the neighbors regress, one clause satisfied and the other never checked [17]. *Green-by-weakening* is tests edited, skipped, or mocked, or rerun-until-green inherited from a flaky suite [15]. *Circular certification* is the patch validated by tests the same model just wrote to match the patch — agreement with one's own misunderstanding [31]. *Special-casing* is the patch returning the expected value for the tested input and the buggy behavior everywhere else — passing tests as a false flag [1][3]. *Review theatre* is an LGTM with no checklist grounding, or the harness routing correctness through review and repeating the expectation Bacchelli and Bird showed was never realistic [22]. *Stale acceptance* is the patch validated against a snapshot that drifted, so by merge time the guard set no longer describes the mainline [17]. *Vacuous green* is a patch no test actually executed, the green certifying nothing. *Suite-blindness* is contaminated green — tests visible in training data or issue text, the failure SWE-Bench+ and SWE-bench Verified exist to measure [14][30]. Two honest caveats close the section. First, the strong numbers of Section 2 — AutoPatchBench's collapse [19], RePaCA's accuracy [7], Wang, Pradel, and Liu's inflation estimate [13] — were measured on repositories, models, and vulnerability corpora far larger than a weak-model harness controls, and a weak model should expect materially smaller absolute effects while leaning all the harder on the two-part verdict, the risk thresholds, and the deterministic battery to make its small budget count. Second, every acceptance metric must be reported on held-out and out-of-benchmark tasks, as in the earlier components, so a gate that has memorized a task set's correct patches cannot score well on discrimination without its verdicts having been measured on problems it never saw. ## 7. Conclusion Patch acceptance is where the coding harness's most expensive mistake is made to look like its cheapest success: a green test suite. This paper has argued that "green" is a claim made by the suite, not a verdict about the patch, and that a weak-model harness must therefore make acceptance a separate, adversarial, evidence-thresholded decision — a gate that requires proof the failure flipped, preserves behavior through impact-aware regression, independently assesses root cause against overfit, actively hunts counterexamples, scales evidence to risk, and records every verdict with its evidence — rather than committing on the consequence of a passing suite. The thesis that has governed every design choice is the one with which this paper began and closes: **Patch acceptance should be a separate, adversarial, evidence-thresholded decision rather than the consequence of a green test suite: require proof that the original failure has changed from fail to pass, preserve previously correct behavior through impact-aware regression testing, independently assess whether the patch addresses the diagnosed root cause rather than overfitting visible tests, actively search for behavioral counterexamples through differential, property-based, or fuzz testing where warranted, scale the required evidence to the risk and blast radius of the change, and preserve an explicit acceptance record linking every verdict to its supporting and unresolved evidence. When evidence is insufficient, the correct result is INCONCLUSIVE—not ACCEPT.** And the rule that compresses that thesis into one sentence of discipline for Piper is: **The patch generator's job is to make a patch look right. The validator's job is to prove it wrong. Only when the validator fails after sufficiently strong attacks does the patch earn acceptance.** The generator proposes and steers; the validator attacks and certifies. The measured message of the literature is unambiguous — Qi, Long, Achour, and Rinard showed that the overwhelming majority of plausible patches are functionality-deletions that pass the suite and fix nothing [1], Long and Rinard showed the search space is biased toward exactly those patches [2], Xin and Reiss's DiffTGen showed the pre-edit program is a free executable specification of the unchanged contract [3], RePaCA and LLM4PatchCorrect showed the overfit judgment is a trainable veto [7][6], R2E-Gym showed the learned verifier's confident-language failure [8], Wang, Pradel, and Liu showed a benchmark can still call an incorrect patch solved and that differential testing exposes it [13], AutoPatchBench showed the collapse from plausible to correct that an honest gate exists to catch [19], and Godlin and Strichman showed the formal ceiling where damage-absence becomes a proof rather than a sample [23]. For a weak model, a green light on a weak suite is a confident coin flip; what a weak-model harness needs is a gate that asks not whether the tests passed but whether the validator, having attacked the patch as hard as the change's risk demanded, failed to break it — because the question is never whether the suite was green, but whether the cheapest, strongest attacks that could have proved the patch wrong were actually run, and what they failed to find. --- ## References 1. Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. *An Analysis of Patch Plausibility and Correctness for Generate-and-Validate Patch Generation Systems.* International Symposium on Software Testing and Analysis, ISSTA 2015. DOI 10.1145/2771783.2771791. *(Overwhelming majority of plausible patches are functionality-deletions: 104/110 GenProg, 37/44 RSRepair, 22/27 AE — not merely "a third" wrong.)* 2. Fan Long and Martin Rinard. *An Analysis of the Search Spaces for Generate and Validate Patch Generation Systems.* International Conference on Software Engineering, ICSE 2016. DOI 10.1145/2884781.2884872. *(Search space biased toward plausible-but-wrong patches; only a minority of plausible patches correct.)* 3. Qi Xin and Steven P. Reiss. *Identifying Test-Suite-Overfitted Patches through Test Case Generation.* International Symposium on Software Testing and Analysis, ISSTA 2017. DOI 10.1145/3092703.3092718. *(CORRECTED: Xin & Reiss, ISSTA 2017 — not "Xiong et al. FSE 2018"; 1806.10786 is an unrelated math paper and must never be used.)* 4. He Ye, Jian Gu, Matias Martinez, Thomas Durieux, and Martin Monperrus. *Automated Classification of Overfitting Patches with Statically Extracted Code Features.* IEEE Transactions on Software Engineering, 2021. arXiv:1910.12057. *(ODS; static-feature overfitting classifier.)* 5. Haoye Tian, Kui Liu, Abdoul Kader Kaboré, Anil Koyuncu, Li Li, Jacques Klein, and Tegawendé F. Bissyandé. *Evaluating Representation Learning of Code Changes for Predicting Patch Correctness in Program Repair.* International Conference on Automated Software Engineering, ASE 2020. arXiv:2008.02944. *(Real title; "Is this the correct patch?" is a descriptor, not the title.)* 6. Xin Zhou, Bowen Xu, Kisub Kim, DongGyun Han, Hung Huu Nguyen, Thanh Le-Cong, Junda He, Bach Le, and David Lo. *Leveraging Large Language Model for Automatic Patch Correctness Assessment.* IEEE Transactions on Software Engineering, Vol. 50, No. 11, pp. 2865–2883, November 2024. DOI 10.1109/TSE.2024.3452252. *(No arXiv ID exists; 2303.00202 is PatchZero and must never be used.)* 7. Marcos Fuster-Pena, David de-Fitero-Dominguez, Antonio Garcia-Cabot, and Eva Garcia-Lopez. *RePaCA: Leveraging Reasoning Large Language Models for Static Automated Patch Correctness Assessment.* Neurocomputing, Elsevier, 2026. arXiv:2507.22580. DOI 10.1016/j.neucom.2026.134583. *(83.1% accuracy, 84.8% F1, 85.7% recall; Qwen2.5-Coder-3B backbone.)* 8. Naman Jain, Jaskirat Singh, Manish Shetty, Liang Zheng, Koushik Sen, and Ion Stoica. *R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agents.* Conference on Language Modeling, COLM 2025. arXiv:2504.07164. *(Confident-language failure mode of execution-free verifiers.)* 9. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. *Training Software Engineering Agents and Verifiers with SWE-Gym.* arXiv 2024. arXiv:2412.21139. *(First trained trajectory verifier.)* 10. KaShun Shum, Binyuan Hui, Jiawei Chen, Lei Zhang, X. W., Jiaxi Yang, Yuzhen Huang, Junyang Lin, and Junxian He. *SWE-RM: Execution-free Feedback For Software Engineering Agents.* arXiv 2025. arXiv:2512.21919. *(Execution-free verifier at ~100k trajectories / ~170 repos; continuous discrimination score.)* 11. Wenqi Huang, Charley Lee, Leonard Tng, and Serena Ge. *DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks.* arXiv 2026. arXiv:2607.07946. *(113-task long-horizon benchmark; hand-written verification scripts; execution-free verifier benchmarked in SWE-RM.)* 12. Xingyao Wang, Valerie Chen, Heng Ji, and Graham Neubig. *A Rubric-Supervised Critic from Sparse Real-World Outcomes.* arXiv 2026. arXiv:2603.03800. *(OpenHands Critic; LLM verifier scoring trajectories without executing code.)* 13. You Wang, Michael Pradel, and Zhongxin Liu. *Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study.* arXiv 2025. arXiv:2503.15223. *(7.8% correct-but-failing; 29.6% behaviorally divergent; 28.6% certainly incorrect; 6.2pp inflation; PatchDiff.)* 14. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. *SWE-Bench+: Enhanced Coding Benchmark for LLMs.* arXiv 2024. arXiv:2410.06992. *(32.67% solution leakage; 31.08% weak tests.)* 15. Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. *ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases.* arXiv 2025. arXiv:2510.20270. *(How often "passing" is tampering with the checks.)* 16. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. *Agentless: Demystifying LLM-Based Software Engineering Agents.* arXiv 2024. arXiv:2407.01489. *(LOCALIZE/REPAIR/VALIDATE; 32.00% SWE-bench Lite at ~$0.70; validation as harness-owned phase.)* 17. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024. arXiv:2310.06770. *(FAIL_TO_PASS/PASS_TO_PASS; harness-side execution; mechanical test-patch reapplication.)* 18. Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. *SpecRover: Code Intent Extraction via LLMs.* ICSE 2025. arXiv:2408.02232. *(Confidence score per accepted patch; reviewer interpreting failures against extracted intent.)* 19. TJ Byun, Cornelius Aschermann, Kai Yuan Thng, Wu Zhou, Yupeng Yang, Lauren Deason, and Joshua Saxe. *AutoPatchBench: A Benchmark for AI-Powered Security Fixes.* Engineering at Meta, 2025-04-29. *(Non-arXiv blog; no fabricated ID. ~60% plausible → 5–11% correct; Gemini 1.5 Pro 61.1% → 5.3%; differential 84.1% agreement / 41.7% precision.)* 20. Koen Claessen and John Hughes. *QuickCheck: A Lightweight Tool for Random Testing of Haskell Programs.* International Conference on Functional Programming, ICFP 2000. DOI 10.1145/351240.351266. *(Property-based testing; the Level E attack's ancestor.)* 21. Zhiyu Li, Shuai Lu, Daya Guo, Nan Duan, Shailesh Jannu, Grant Jenks, Deep Majumder, Jared Green, Alexey Svyatkovskiy, Shengyu Fu, and Neel Sundaresan. *Automating Code Review Activities by Large-Scale Pre-training.* ESEC/FSE 2022. arXiv:2203.09095. DOI 10.1145/3540250.3549081. *(CodeReviewer; diff-level quality estimation, comment generation, refinement.)* 22. Alberto Bacchelli and Christian Bird. *Expectations, Outcomes, and Challenges of Modern Code Review.* International Conference on Software Engineering, ICSE 2013. DOI 10.1109/icse.2013.6606617. *(CORRECTED VENUE: ICSE 2013, not ESEM 2013; review outcomes dominated by improvement and knowledge transfer, not defect detection.)* 23. Benny Godlin and Ofer Strichman. *Regression verification.* Proceedings of the 46th Annual Design Automation Conference, DAC 2009. DOI 10.1145/1629911.1630034. *(Proving behavioral equivalence of unchanged parts via mutual summaries and SMT.)* 24. Shuvendu Lahiri, Chris Hawblitzel, Ming Kawaguchi, and Henrique Rebêlo. *SymDiff: A Language-Agnostic Semantic Diff Tool for Imperative Programs.* Computer Aided Verification, CAV 2012. *(CORRECTED: CAV 2012, Lahiri/Hawblitzel/Kawaguchi/Rebêlo — not "Lahiri, McMillan & Sharma, CAV 2013," which is Differential Assertion Checking. Non-arXiv; no ID.)* 25. Chuyue Sun, Ying Sheng, Oded Padon, and Clark Barrett. *Clover: Closed-Loop Verifiable Code Generation.* arXiv 2023. arXiv:2310.17807. *(Acceptance by pairwise consistency of code, docstring, and Dafny annotations; no tests asked.)* 26. Yuan-An Xiao, Chenyang Yang, Bo Wang, and Yingfei Xiong. *Accelerating Patch Validation for Program Repair with Interception-Based Execution Scheduling.* IEEE Transactions on Software Engineering, Vol. 50, No. 3, pp. 618–635, 2024. arXiv:2305.03955. *(CORRECTED VENUE: IEEE TSE 2024, not ACM DL 2023; ExpressAPR; two-orders-of-magnitude validation speedup.)* 27. Yiling Lou, Jun Yang, Samuel Benton, Dan Hao, Lin Tan, Zhenpeng Chen, Lu Zhang, and Lingming Zhang. *When Automated Program Repair Meets Regression Testing — An Extensive Study on 2 Million Patches.* ACM Transactions on Software Engineering and Methodology, 2024. DOI 10.1145/3672450. arXiv:2105.07311. *(Real title; RTS-for-APR; efficiency measured in test executions.)* 28. Hongwei Li, Yuheng Tang, Shiqi Wang, and Wenbo Guo. *PatchPilot: A Cost-Efficient Software Engineering Agent with Early Attempts on Formal Verification.* arXiv 2025. arXiv:2502.02747. *(Real title; Validation stage with in-generation checks.)* 29. Nadia Alshahwan, Jubin Chheda, Anastasia Finegenova, Beliz Gokkaya, Mark Harman, Inna Harper, Alexandru Marginean, Shubho Sengupta, and Eddy Wang. *Automated Unit Test Improvement using Large Language Models at Meta.* arXiv 2024. arXiv:2402.09171. *(TestGen-LLM; run/pass/coverage/flake filters on generated tests before human review.)* 30. OpenAI. *Introducing SWE-bench Verified.* OpenAI, 2024-08-13. *(Non-arXiv; no fabricated ID. 500 human-verified instances of SWE-bench.)* 31. Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. *TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark.* arXiv 2024. arXiv:2410.00752. *(CORRECTED AUTHORS: Jain, Synnaeve, and Rozière — not "Löbel et al.")*