Summary / 17. Test execution / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Green Is a Claim About the Suite, Not the Code: Deterministic, Evidence-Typed, Cost-Ordered Test Execution for a Weak-Model Coding Harness *Component 17 of 27 — Coding-Harness Program. This document specifies the test-execution stage, and it is precise about what it is not: it is not the ad-hoc "does it run at all" probe of component 11, and it is not the generation of new tests, which belongs to a later row. It is the institutional certification battery — the repository's own syntax, lint, type, build, unit, integration, and regression checks, run harness-side in a pinned environment, with test files tamper-protected, results captured in machine-readable form, and an explicit flakiness policy. The spine distinction is between "it ran" and "it's certified": component 11 established that a generated program executes; this component establishes whether a candidate patch is allowed to advance, and on what evidence. A green result in this component is not a boolean, it is a typed claim — a claim that a specific check, at a specific tier, in a specific environment, actually proved a specific property. The naive harness asks "did the tests pass?" and accepts a single exit code; this component argues that the only question worth asking is which tests passed, what property each one actually proved, and whether the cheapest sufficient set capable of disproving the patch was run. This paper translates the program's thesis into the test-execution build contract: a deterministic, harness-side validation ladder — syntax, lint, type/build, bug-specific reproduction, impacted unit tests, affected regression, subsystem integration, full suite — in which every result is evidence-typed, bug-fix evidence is contrasted against the original buggy state whenever possible, dependency and coverage information selects the smallest safe regression set, failure at any tier halts more expensive validation and returns structured evidence to diagnosis, and the weak model's only role is steering (in-loop test runs as agent-computer-interface actions, and the interpretation of failures) — never the final verdict, because the certification battery is deterministic software and the L column holds at most one entry.* --- ## Abstract Test execution is where a coding harness learns whether its patch fixed the bug, broke something that worked, or merely survived an unrelated check — and it is where the program's most common lie is told, by the harness to itself, in the single most compressible form: an exit code. This paper argues that "green" is not a fact about a patch but a claim about a suite, and that a weak-model harness must therefore treat test execution as a deterministic, evidence-typed, change-aware, cost-ordered discipline rather than as a single pytest invocation whose output is fed back as a boolean. The review establishes the case across twenty-six parts: Codex and HumanEval founded the contract that correctness is executed behavior, sandboxed from day one [1]; SWE-bench made the certification protocol an institution — harness-side execution, mechanical reapplication of protected test files, and a verdict defined as FAIL_TO_PASS flipping while PASS_TO_PASS holds [2] — and SWE-bench+ measured the gap in the referee itself (32.67% solution leakage, 31.08% weak tests) [3]; SWE-agent put test runs inside the loop as agent-computer-interface actions [4] while Agentless made validation an explicit deterministic phase whose reproduction-plus-regression tests raised patch selection from 77 to 96 fixes on 300 SWE-bench Lite problems [5]; the Validation Evidence study showed 46.0% of positive comparable validation events carried no bug-discriminating information [6]; HyRTS [7], Ekstazi [8], and Rothermel and Harrold [9] supplied the change-aware selection that makes tiered batteries affordable, with TDAD carrying it to agent patches [10]; TestPrune [11] and SpecRover [12] carried selection and interpretation into repair agents, and the traceability study measured how often "the agent ran tests" decouples from "the agent ran the right tests" [13]; ExecutionAgent got arbitrary suites to run at all [14] and ImpossibleBench measured how often "passing" is tampering [15], with Otter supplying the reproduction tests that pair with selection [16]; Luo, Hariri, Eloussi, and Marinov established flakiness as the trust-killer whose ad-hoc remedies are the disease [17]; and beneath it all DeMillo, Lipton, and Sayward's mutation analysis is the proof that suite strength is a measured instrument [18], with SWE-ABS's 19.71% rejection of previously passing patches the benchmark-scale confirmation [19], DeMillo and Offutt [20], Defects4J [21], Hilton et al. [22], and Bessey et al. [23] the industrial and research infrastructure around it, and the agent-generated-test literature — Rethinking the Value of Agent-Generated Tests [24], TestGenEval [25], LiveCodeBench [26] — the warning that the model's own checks are weak, so the repository's battery is the only verification with teeth. From this review the paper derives seven design principles, then specifies a deterministic tiered architecture (D) in which the single learned entry (L) is steering and interpretation — never the verdict — and closes with a recursive self-improvement protocol and a measurement plan whose instruments exist to keep the harness from mistaking an empty, flaky, or vacuous green for evidence. ## 1. Introduction The coding-harness program has, through components 01 through 16, assembled machinery that reads a repository, localizes a fault, maps its dependencies, generates a candidate patch, edits it into place, runs it, and distills failures into durable memory. Component 17 is the moment where all of that work either becomes certified or evaporates: it is where the harness decides, on evidence, whether a candidate patch is allowed to advance toward acceptance. Everything upstream produces a claim — "this patch fixes the bug" — and everything downstream trusts that claim. Test execution is therefore the load-bearing stage of the whole program, and it fails in exactly one way that matters: it lets a patch through on green that proves nothing, or blocks a correct patch on red that proves nothing, and in either case the failure is invisible because the harness asked only whether the tests passed. The thesis that governs this component is stated plainly. **Test execution should be deterministic, evidence-typed, change-aware, and cost-ordered: every candidate patch progresses through an escalating validation ladder from cheap static gates to bug-specific reproduction, impacted unit tests, regression tests, integration checks, and finally broader suites; each test result is classified by whether it demonstrates bug resolution, regression preservation, or merely unrelated success; bug-fix evidence should be contrasted against the original buggy state whenever possible; dependency and dynamic-coverage information should select the smallest safe regression set; and failure at any tier should halt more expensive validation and return structured evidence to diagnosis.** And it carries a strong rule for Piper that compresses the whole thesis into one sentence, stated verbatim: **Do not ask 'Did the tests pass?' Ask 'Which tests passed, what property did each one actually prove, and did we run the cheapest sufficient set capable of disproving this patch?'** The first clause of the thesis is the discipline of typing — a green result carries a role, never a bare boolean, because "17 tests passed" tells the harness nothing about which of the bug-resolution, regression-preservation, or merely-unrelated obligations each test actually discharged. The second clause is the discipline of contrast — a bug-fix test is not bug evidence until it has demonstrated fail-on-buggy and pass-on-candidate. The third clause is the discipline of cost — the cheapest check that can disprove the patch runs first, and an expensive suite is never spent to collect red lights a lint already lit. The final clause is the discipline of the halting ladder — failure at a tier stops the climb and returns structured evidence to diagnosis, because which tier caught the failure is itself debugging information. The weak-model problem makes test execution the make-or-break stage for a specific reason that has nothing to do with the model's raw capability: a weak model cannot be trusted to interpret a wall of failures, so the harness must hand it failures that are already minimal, typed, and attributed. If a weak model is given 127 failures accumulated across ten unverified edits, it will diagnose nothing; if it is given a single bug-specific reproducer that flipped from fail to pass alongside a single impacted regression that flipped from pass to fail, the diagnosis is already written. The ladder exists to make that second scenario the only scenario — every meaningful repository-state transition carries a cheap validation opportunity immediately afterward, so the distance between an edit and the test that exposes its consequence is as small as possible. The trap the ladder guards against is the one the whole program's economics fear: a 40-minute integration suite run per repair round is loop-economics suicide, and a harness that burns it on a file that does not parse has paid for information it could have had in 80 milliseconds. The escape is change-aware cost-ordering — HyRTS showed dependency-aware selection can cut the regression set to 18.35% of tests versus 27.18% for file-level selection, at 31.60 seconds versus 40.06 seconds of offline time [7] — and the evidence-typing that makes every remaining result carry a meaning. The design implication is decisive and it is the thesis: a weak-model harness must run the cheapest sufficient set capable of disproving the patch, type each result by what it proved, contrast bug-fix evidence against the buggy baseline whenever affordable, and stop at the first failure, so that the model's only interpretive burden is reading an already-structured verdict it did not author. The paper proceeds as follows. Section 2 reviews the test-execution literature in ten parts — the founding contract that correctness is executed behavior, the certification protocol as an institution, in-loop execution and the harness-side validation phase, the evidence-quality negative result, change-aware regression test selection, selection and interpretation for repair agents, getting the suite to run and keeping it honest, flakiness as the trust-killer, suite strength as a measured instrument, and the weakness of agent-generated tests. Section 3 derives seven design principles. Section 4 specifies the architecture, labeling every node deterministic (D) or learned (L) and stating the invariant that the L column holds at most one entry, with the deterministic battery as the default. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into ten parts that together answer one question: what is a green test result FOR, and how must test execution be built so that a weak model's small validation budget buys the most disproof per unit of cost? Each part supplies one design lever the certification battery must pull. ### 2.1 The founding contract: correctness is executed behavior **Evaluating Large Language Models Trained on Code** (Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, the operator Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba, arXiv 2021, arXiv:2107.03374) is the founding contract for everything this component does, and it appears here in full because the author list is itself the point: the decision that code-model quality is measured by functional correctness under executed unit tests — pass@k with an unbiased estimator — rather than by BLEU or exact-match proxies that do not track behavior, was made by a fifty-eight-author organization-scale effort, and every benchmark since has inherited it [1]. Two under-appreciated harness details belong with the citation. First, the evaluation required sandboxing from day one, because generated programs occasionally fork-bombed or hogged resources — the earliest lesson that test execution is itself code running in a dangerous environment, deserving container, timeout, and resource isolation. Second, the estimator's unbiasedness matters to this component directly: a harness that samples reruns to inflate a score is p-hacking the suite, and the flake policy of Section 3 is the modern form of the same refusal. The design transfer is the axiom: correctness equals executed behavior, and the harness that executes it is trusted infrastructure — design it like infrastructure. ### 2.2 The certification protocol as an institution **SWE-bench: Can Language Models Resolve Real-World GitHub Issues?** (Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan, ICLR 2024, arXiv:2310.06770) is the paper that mechanized repository-level certification, and its evaluation protocol is this component's playbook written down before this component existed. Built from 2,294 real GitHub issue/PR pairs across 12 popular Python repositories, it runs each instance in a per-instance Dockerized environment with a frozen toolchain, applies the gold PR's test patch mechanically over the model's patch — so test-file tampering is neutralized by construction — and scores purely as executed verdicts: FAIL_TO_PASS tests must flip, PASS_TO_PASS tests must hold [2]. The design tacitly encodes everything this row demands — harness-side execution, tamper protection, environment pinning — and it is the paper that gave the program its two-sided validation vocabulary: a patch is resolved only when the bug-fixing tests and the regression-preservation tests both pass, which is the exact distinction between "the bug is fixed" and "existing behavior is preserved" that a terrible patch (return the expected value and destroy everything else) and a vacuous patch (preserve everything and never fix the defect) each violate in a different direction. It also surfaced the limit that became this component's own first audit obligation: the later human-validated Verified subset (OpenAI, 2024) kept only 500 of the 2,294 instances, discarding a large share for faulty or underspecified tests — the referee, it turned out, needed refereeing. **SWE-Bench+: Enhanced Coding Benchmark for LLMs** (Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang, arXiv 2024, arXiv:2410.06992) is that refereeing, and it must be cited for its real findings and nothing else. Auditing SWE-bench and SWE-bench Lite test soundness, it reports that 32.67% of resolved patches had the solution directly present in the issue report or comments (solution leakage) and that 31.08% of passed patches were suspicious — incorrect, incomplete, or touching different files — yet passed owing to inadequate test cases (weak tests) [3]. The lesson for the harness is the failure-modes section of Section 6 wearing a citation: the suite must be re-executed fresh against a clean baseline, and suite quality is itself a quantity to be measured, not assumed. (The "empty patch" phenomenon — a FAIL_TO_PASS test that passes with no code change at all — is a distinct failure mode of harness self-testing that this component treats as its own check, but it is not a finding of Aleithan et al., and it is not attributed to them here.) ### 2.3 In-loop execution and the harness-side validation phase **SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering** (John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press, NeurIPS 2024, arXiv:2405.15793) makes tests and program execution first-class actions inside the coding loop, rather than something that happens after the agent finishes [4]. Its custom agent-computer interface lets the model navigate and edit the repository and execute tests and programs, with those observations influencing the next action, and it reports 12.5% pass@1 on SWE-bench and 87.7% on HumanEvalFix while emphasizing that the ACI design itself materially affects performance. This is the in-loop half of the component's D/L division: test runs are the steering wheel, the model's legitimate channel for grounding edits in observable behavior immediately. The discipline it implies is stronger than the one it enforces — every meaningful repository-state transition should carry a cheap validation opportunity immediately afterward, so that "edit expression, parse the file, run the targeted test, continue" replaces "make nine edits, run pytest, confront 127 failures." A weak model handed failures that interact across ten unverified edits diagnoses nothing; the smaller the edit-to-exposure distance, the easier diagnosis becomes. **Agentless: Demystifying LLM-Based Software Engineering Agents** (Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang, arXiv 2024, arXiv:2407.01489) is the harness-side half, and the strongest directly applicable paper for this stage because it turns patch validation into its own explicit pipeline phase — LOCALIZE, REPAIR, VALIDATE — executed by scripts, never left to the model's self-report [5]. Its validation distinguishes two kinds of tests answering two different questions: existing regression tests (did the patch break behavior that worked before?), selected by first identifying tests that pass on the original repository; and reproduction tests (did the patch actually fix the issue?), generated for the newly reported defect, executed on the original buggy version, and retained only if they reproduce the defect before being used against patches. The ablation is the quantitative heart of this component: on the 300 SWE-bench Lite problems in that experiment, majority voting alone reached 77 fixes, adding regression testing reached 81, and adding reproduction testing reached 96 [5]. That is the proof that a patch must satisfy two independent propositions — the bug is fixed AND existing behavior is preserved — and that test evidence, selected and typed, is worth real selection performance, not a rubber stamp. ### 2.4 The evidence-quality negative result: green is not bug evidence **Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?** (Xiaonan Xu and Wenjing Wu, arXiv 2026, arXiv:2607.28871) is the paper that makes the naive harness's central question embarrassing, and it is mandatory for the 2026 version of this subsystem [6]. It asks the question coding agents generally do not: the agent ran a test and it passed — would that exact test also have passed on the original buggy code? If so, the passing test provides zero evidence that the bug was fixed. The authors replay each validation command against three states — the buggy repository state B, the candidate patch state S, and the developer gold-fix state G — and classify what a passing result actually proves. Across 3,730 validation events from 643 rollouts on 110 tasks, they find that 46.0% of positive comparable validation events carried no bug-discriminating information, and that 23.8% of baseline trajectories ended with an entire positive evidence base of this kind [6]. This is the single most important warning in the component: nearly half of "the test passed" events proved nothing about the bug. The deterministic transfer is the baseline-replay contrast of Section 3 — run the candidate's validation test on the buggy state and on the candidate state, and classify the pair (B FAIL / C PASS is bug-discriminating; B PASS / C PASS is regression-only; B FAIL / C FAIL is no fix; B PASS / C FAIL is a regression). No language model is needed for the classification. The authors also report that returning buggy-state replay information reduced evidence-inadequate closures, but the magnitude fell below their prespecified threshold for practical effect size, so this paper treats the prevalence finding as the anchor and the intervention magnitude as still needing replication. ### 2.5 Change-aware regression test selection **Hybrid Regression Test Selection** (Lingming Zhang, IEEE/ACM 40th International Conference on Software Engineering, ICSE 2018, DOI 10.1145/3180155.3180198) is the deliberate reach outside the LLM-agent literature, because software engineers had already solved the expensive half of this component's problem: do not rerun the entire suite when a small part of the program changed — determine which tests can actually be affected [7]. HyRTS combines regression-test selection at multiple granularities, choosing method-level dependency analysis where its precision pays for itself and file-level analysis where it is too expensive, and the evaluation is the numbers this component's cost-ordering is built on: 2,707 revisions across 32 Java projects totaling more than 124 million lines of code, where HyRTS selected 18.35% of tests versus 27.18% for file-level selection, with average offline test time of 31.60 seconds versus 40.06 seconds [7]. It is a non-arXiv paper, cited by its ICSE 2018 DOI and not by any invented identifier. The transfer is the ladder's middle rungs: the harness already has, from components 02 through 06, the repository graph, dependency graph, changed symbols, and impact analysis, so the naive "small edit, run the entire suite" can become "changed symbol, affected tests, run those first, widen only if they pass." **Ekstazi: Lightweight Test Selection** (Milos Gligoric, Lamyaa Eloussi, and Darko Marinov, IEEE/ACM 37th International Conference on Software Engineering, ICSE 2015, DOI 10.1109/ICSE.2015.230) is the dependency-based regression-test-selection technique that makes tiered batteries tractable in practice, cited here with its corrected venue: it is an ICSE 2015 paper, not an ESEC/FSE 2015 one [8]. Its lightweight approach — tracking file-level dependencies to select only tests that depend on changed files — is the cameo from the impact-analysis row that the expensive tiers of this component's ladder inherit. **A Safe, Efficient Regression Test Selection Technique** (Gregg Rothermel and M. J. Harrold, ACM Transactions on Software Engineering and Methodology, 1997, DOI 10.1145/248233.248262) is the safe-RTS foundation the modern selection papers are reviving, cited by its TOSEM DOI with no arXiv identifier [9]. Safety — selecting a superset of the tests whose behavior the change could affect — is the property this component demands of its own selection tier: the "smallest safe regression set" of the thesis is safe first, small second. **TDAD: Test-Driven Agentic Development — Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis** (Pepe Alonso, Sergio Yovine, and Victor A. Braberman, arXiv 2026, arXiv:2603.17973) carries graph-based impact analysis to agent-generated patches, running regression-impact analysis before submission so that patches whose dependency graph reaches already-broken or newly-broken tests are caught before they ship [10]. It is the proof that the selection machinery of HyRTS [7] and Ekstazi [8] composes directly with an LLM repair loop — the harness computes the impact graph, and the checks are run because the harness runs them, not because the model chose to. ### 2.6 Selection and interpretation for repair agents **Can Old Tests Do New Tricks for Resolving SWE Issues?** (Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, and Martin Hirzel, Proceedings of the Symposium on the Foundations of Software Engineering, FSE 2026, arXiv:2510.18270) is the TestPrune paper and the regression-test-selection paper written specifically for agents [11]. Its survey is itself a useful table row: Agentless filters to tests that pass on buggy code then applies an LLM filter; SpecRover runs regression tests per patch with an LLM interpreting results; and SWE-RL, R2E-Gym, PatchPilot, and Trae Agent all adopt Agentless-style regression testing, differing mainly in how selected tests filter or rank candidates. Its critique is that none of these leverage coverage or measure the intrinsic quality of the selection step. TestPrune does — coverage-based selection plus systematic minimization to k = 10 tests — and yields a 9.4%–12.9% relative increase in issue-resolution rate within the Agentless framework, capturing fixes that agents had correctly produced but not submitted as final patches [11]. The two integration modes matter directly: feeding selected tests as validation feedback inside an agent loop, and using them to pick one patch from tens of candidates. **SpecRover: Code Intent Extraction via LLMs** (Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury, ICSE 2025, arXiv:2408.02232) made "run the suite, then have a model read the results" a deliberate stage, and it is the piece most harnesses skip [12]. Beyond executing regression tests on every candidate patch, SpecRover adds a reviewer agent that interprets failures against the extracted intent — a lightweight specification of the issue — and exposes a confidence score with the patch so a downstream human or gate can decide whether to trust it. The transfer is the boundary of the L column: interpreting a failure's meaning against the issue's intent is legitimate steering, but the interpretation is downstream of, and cannot substitute for, the deterministic verdict that a specific test failed. **Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study** (Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, and Baishakhi Ray, ISSTA 2026, arXiv:2506.08311) supplies the measurement of test-running behavior that the earlier rows referenced by the wrong section pointer [13]. Defining gold regression tests as SWE-bench's pass-to-pass tests, it finds that workflow agents run regression tests far more often than open-process agents — ACR and Agentless for nearly all tasks, MASAI for over 85% — but that selection accuracy varies wildly: MASAI hits gold tests only 4% of the time, Agentless 45%, ACR 100%, and open-process agents' exploratory heuristics reduce the likelihood of finding relevant tests. The finding for this component is that "the agent ran tests" and "the agent ran the right tests" are decoupled, and that autonomous agents are worst at the second — the argument for the harness owning test selection rather than leaving it to the model. The citation is corrected in two respects: the venue is ISSTA 2026, and the relevant content is Section 4 (agent effectiveness in patch generation, including overfitting and test-coverage issues) and Section 5 (issue-reproducing test generation) — not §6.2, which concerns patch edit-distance similarity between developer and agent patches, not test-running behavior. ### 2.7 Getting the suite to run, and keeping it honest **You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects** (Islem Bouzenia and Michael Pradel, ISSTA 2025, arXiv:2412.10133) addresses the prerequisite this component's ladder silently assumes: that the suite can be run at all [14]. Across arbitrary projects, the friction is environment setup, dependency resolution, and the bespoke invocation of each test harness, and ExecutionAgent is the LLM agent that handles that friction so the deterministic battery has something to execute. Its place is strictly upstream of the verdict — it gets the suite to run; it does not interpret the results, which remain the harness's own. **ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases** (Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini, arXiv 2025, arXiv:2510.20270) measures the attack surface this component's tamper-protection exists to close: how often agents "pass" by exploiting the checks rather than satisfying them [15]. It is the benchmark-side statement of a harness-side discipline — the suite is the agent's optimization target, and whatever the battery actually measures, the loop will deliver, so a tamperable battery selects for gaming rather than correctness. The tamper-protection of Section 3 is the answer: reapply protected test files, or diff-scope-check that no test, fixture, or skip marker changed. **Otter: Generating Tests from Issues to Validate SWE Patches** (Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel, ICML 2025, arXiv:2502.05368) is the standard reproduction-test generator that pairs with TestPrune's selection [16]. Its place in this component is the bug-specific reproducer tier: when a newly reported issue has no repository test exposing it yet, a generated reproduction test — validated against the buggy baseline before being used against patches — is the only test that answers "did the patch actually fix the issue?" rather than "did it break anything?". ### 2.8 Flakiness: the trust-killer **An Empirical Analysis of Flaky Tests** (Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov, 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014, DOI 10.1145/2635868.2635920) is the trust-killer of this component, and it is cited here with a correction that matters: the paper is an FSE 2014 publication with DOI 10.1145/2635868.2635920 and has no arXiv identifier — the identifier 1407.3067 that once accompanied it resolves to an unrelated population-genetics paper and must never be used for it [17]. The study's large-scale analysis of flaky tests in Java and Android projects finds the dominant causes are asynchronous waits and concurrency, and that the prevailing "fix" is ad-hoc — deleting waits, adding sleeps, or deleting the test. For an agent loop flakiness is doubly corrosive: a false red causes the repair loop to abandon a correct patch, teaching it a phantom regression, while a false green certifies broken code. And rerun-until-green is p-hacking the suite — teaching the agent that persistence, not correctness, flips tests. The transfer is the flake policy of Section 3: cap reruns, record flake signatures, and quarantine a test that flips across reruns, before the loop's statistics are corrupted. ### 2.9 Suite strength is a measured instrument **Hints on Test Data Selection: Help for the Practicing Programmer** (Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward, IEEE Computer, Vol. 11, No. 4, pp. 34–41, April 1978, DOI 10.1109/C-M.1978.218136) is the philosophical floor of the component, pre-personal-computer and pre-everything, and the row's outlier [18]. It is the founding paper of mutation analysis: seed small syntactic faults ("mutants") into a program, and measure a test suite's quality by the fraction of mutants it kills. Its two durable hypotheses — the Competent Programmer Hypothesis (real bugs are small deviations from correct programs) and the Coupling Effect (tests that kill simple mutants tend to kill complex ones) — are the reason "green is a claim about the suite, not just the code" is a sentence rather than a slogan. A weak suite greenlights patches that no serious suite would pass, and an agent optimizing against it will find the gap. The citation carries the full subtitle, and the venue is IEEE Computer 11(4):34–41, April 1978. **SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark** (Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand, arXiv 2026, arXiv:2603.00520) is the benchmark-scale confirmation of the 1978 lesson, cited with its corrected title and figure: the correct title is the adversarial-strengthening one above (not "Strengthening Test Suites to Improve SWE-bench Evaluation"), and the stronger suites reject 19.71% of previously passing patches, knocking the top agent's score from 78.8% to 62.2% [19]. Read as: running the existing tests verifies only as much as the existing tests cover, and in real repositories that coverage is thin exactly where agents make mistakes. It is the reason the "existing checks" of this row are a floor, not a ceiling, and the warrant for auditing the suite itself as a measured instrument. **Constraint-Based Automatic Test Data Generation** (Richard A. DeMillo and A. Jefferson Offutt, IEEE Transactions on Software Engineering, Vol. 17, No. 9, 1991, DOI 10.1109/32.92910) is the mutation-based test-data-generation sequel to the 1978 paper, the industrial continuation of the suite-strength lineage that the suite-strength audit of Section 6 operationalizes [20]. **Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs** (René Just, Darioush Jalali, and Michael D. Ernst, ISSTA 2014, DOI 10.1145/2610384.2628055) is the engineered fault-to-test corpus that is automated program repair's substrate — curated execution harnesses as research infrastructure, the proof that the certification battery is itself an object worth engineering [21]. **Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects** (Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig, IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, DOI 10.1109/ASE.2016.30) is the every-change execution battery in its industrial form, cited with its corrected venue (ASE 2016, not ICSE 2016) [22]. **A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World** (Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, the operator McPeak, and Dawson Engler, Communications of the ACM, Vol. 53, No. 2, 2010, DOI 10.1145/1646353.1646374) is the false-positive economics of the lint tier — the Coverity study of why the cheapest static gate pays for itself only when its warnings are triaged against a real signal, not treated as an oracle [23]. ### 2.10 The model's own tests are weak, so the repository's battery has the teeth **Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents** (Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, and Lingxiao Jiang, arXiv 2026, arXiv:2602.07900) is the outlier whose implication lands on this row rather than the test-generation row: the agent's own checks are weak, so the repository's existing regression, lint, and type checks are the only verification with real teeth, and the harness should run them deterministically rather than rely on the model to decide to [24]. Across six frontier models on mini-SWE-agent, large changes in how often agents wrote tests translated into only small shifts in outcomes; the "net change of zero resolved tasks" headline is model-specific — it holds for gpt-5.2 and claude-opus-4.5, while gemini-3-pro-preview dropped by 13 and deepseek-v3.2-reasoner by 2 — so the defensible reading is the paper's own: prompt-induced changes in the volume of agent-written tests do not significantly change final outcomes. Agents leaned on value-revealing print statements more than formal assertions, and when assertions appeared they skewed toward local properties or exact values, with relational or range-style checks in only 2.98%–7.86% of cases — closer to an observational feedback channel than a regression safety net. **TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark** (Kush Jain, Gabriel Synnaeve, and Baptiste Rozière, arXiv 2024, arXiv:2410.00752) is the real-repository evaluation of generated tests, cited with its corrected authorship — the authors are Jain, Synnaeve, and Rozière, not "Löbel et al.", a name that appears on no TestGenEval paper [25]. **LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code** (Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica, arXiv 2024, arXiv:2403.07974) is the contamination-free, continuously refreshed execution evaluation that closes the suite-blindness failure mode — tests visible in training data or issue text are not a referee, and a harness that cannot tell contaminated green from earned green has no verdict at all [26]. Together the three warrant the component's deepest division of labor: the model proposes and steers, and the deterministic, harness-owned, repository-shipped battery certifies. ## 3. Design Principles Seven principles follow from the review, each tied to the evidence above and each a clause of the strong Piper rule made mechanical. **P1 — A green result is typed evidence, not a boolean.** Every validation result carries a role stating which obligation it discharged: BUG_REPRODUCTION (fail-on-buggy, pass-on-candidate — the bug is fixed), REGRESSION (pass-on-baseline, pass-on-candidate — behavior is preserved), EXPECTED_CHANGE (old behavior, new behavior, both observed), and STATIC_GATE (parse, lint, type, or build succeeded). "17 tests passed" is noise; the harness records, for each test, its check, its role, its baseline result, and its candidate result. This is SWE-bench's FAIL_TO_PASS/PASS_TO_PASS vocabulary generalized beyond the benchmark [2], and it is the mechanism by which a terrible patch that satisfies the bug test while destroying everything else, and a vacuous patch that preserves everything while never fixing the defect, each fail a different obligation instead of both collapsing into one number. **P2 — A bug-fix test is not bug evidence until it shows fail-on-buggy and pass-on-candidate.** The Validation Evidence result — 46.0% of positive comparable validation events carried no bug-discriminating information [6] — makes baseline replay mandatory wherever affordable. The candidate's bug-specific test is run on the buggy state B and the candidate state C, and the pair is classified deterministically: B FAIL / C PASS is bug-discriminating; B PASS / C PASS is regression-only; B FAIL / C FAIL is no fix; B PASS / C FAIL is a regression. A passing test that would also have passed on the buggy code establishes only that the patch did not break that test — it proves nothing about the defect. **P3 — Escalate by cost and information; stop at the first cheap failure.** The ladder runs the cheapest, highest-information check first, and a patch fails at the earliest relevant tier. There is no value in a 40-minute integration suite on a file that does not parse, and no value in running four thousand unrelated tests before the one test written to reproduce the bug. Which tier caught the failure is itself debugging evidence: a T0 failure is a syntax/edit error, a T3 failure means the bug is not fixed, a T4-pass-T5-fail means a regression or impact-analysis gap, and a T7-only failure means a distant dependency escaped targeted selection. **P4 — Select the smallest safe regression set.** Dependency and dynamic-coverage information — the change-impact graph of component 06 and the dynamic execution coverage of component 12 — select the tests that can plausibly be affected, and only they run at the expensive tiers, with the full suite held in reserve. This is HyRTS's 18.35%-versus-27.18% precision [7], Ekstazi's lightweight dependency selection [8], and Rothermel and Harrold's safety guarantee [9], composed with TDAD's agent-patch impact analysis [10] and TestPrune's coverage-based minimization [11]. Safety is the first property — the set must be a superset of what the change could affect — and smallness is the second, because the thesis says the smallest safe set, not the smallest set. **P5 — The harness owns the verdict; the model only steers.** The certification battery executes via the harness's own invocation in its own pinned environment — containerized parity with CI, locked dependencies, seeded randomness, no network by default. The agent's "tests pass" is a claim; the harness's run is a fact. Test files are tamper-protected SWE-bench-style: reapply the protected test files after the model's patch, or diff-scope-check that no test, fixture, or skip/xfail/mock-of-SUT marker changed, because deleting a flaky test is not a fix [2]. The model's legitimate channels are in-loop test runs as agent-computer-interface actions [4] and the interpretation of failures against the issue's intent [12] — steering, never the verdict. **P6 — Green is a claim about the suite, not just the code, so the suite is a measured instrument.** A green result certifies as much as the suite can detect, and suite strength is measurable: DeMillo, Lipton, and Sayward's mutation analysis [18] is the instrument, SWE-ABS's 19.71% rejection of previously passing patches is its benchmark-scale reading [19], and DeMillo and Offutt's constraint-based test generation [20], Defects4J [21], Hilton et al.'s CI adoption [22], and Bessey et al.'s static-analysis economics [23] are the infrastructure around it. The harness periodically mutates critical modules and checks that the suite notices; a battery that does not kill its mutants is not a referee. **P7 — Flakiness is p-hacking unless it is capped.** A flaky test corrupts both directions: a false red abandons a correct patch and teaches a phantom regression, a false green certifies broken code, and rerun-until-green teaches the agent that persistence flips tests [17]. The policy is fixed and mechanical: at most N reruns per test with recorded signatures, a test that flips across reruns goes to quarantine rather than victory, and distinct diagnostics are kept for timeouts and hangs (component 11's non-termination rule). Determinism is not optional plumbing; it is the precondition for every downstream statistic. ## 4. The Architecture The test-execution stage is a tiered ladder of checks; every node is labeled D (deterministic software) or L (a learned model). The invariant that governs the whole is: **the L column contains at most one entry, and the deterministic battery is the default.** A pure certification run of a candidate patch has zero L entries — every tier from syntax to full suite is deterministic software on CPU, and the verdict is computed by the harness. The single L entry appears only when the harness is operating in-loop, and it is steering and interpretation: the model may choose which tests to run as agent-computer-interface actions (SWE-agent's in-loop half [4]) and may read a failing test's message against the issue's intent to guide diagnosis (SpecRover's reviewer half [12]). The model never authors the verdict, never decides whether a green result counts, never reruns a test until it passes, and never interprets a result in a way that overrides the deterministic classification. The concrete input is a candidate patch (or candidate repository state), the buggy baseline, and the repository's own battery plus the change-impact and coverage information from components 06 and 12; the output is a typed evidence record — a structured verdict per tier with roles, baseline/candidate contrasts, flake signatures, and the tier at which the patch failed, if it did. The ladder, in the deterministic tiers T0 through T7 plus the one learned entry (L): 1. **T0 — Syntax and parse [D].** The changed files must parse. Millisecond cost. A parse failure halts the ladder immediately: this is an edit failure, not a semantic one, and no model interpretation is needed to know the patch is not ready. 2. **T1 — Lint and static checks [D].** The repository's linters and cheap static analyzers run over the changed files. Milliseconds to seconds. The triage discipline of Bessey et al. [23] applies: a lint warning is a signal to be interpreted, not an oracle, so the harness distinguishes hard failures from warnings and records both, rather than flattening a noisy linter into a boolean gate. 3. **T2 — Type check and build [D].** The changed targets type-check and build, in the pinned environment. Seconds. This is the last gate before any behavior is executed; a type or build failure returns the compiler's structured diagnostic directly to the diagnosis stage. 4. **T3 — Bug-specific reproduction [D].** The reproducer — the repository test that exposes the defect, or a generated reproduction test validated against the buggy baseline (Otter's line [16]) — runs on the buggy state B and the candidate state C, and the pair is classified per P2: B FAIL / C PASS is bug-discriminating and the patch advances; B PASS / C PASS means the reproducer does not actually reproduce the defect and must be revalidated; B FAIL / C FAIL means the bug is not fixed and the patch halts with that evidence; B PASS / C FAIL means the patch regressed the very behavior it was meant to fix. This is the tier where Agentless's reproduction testing bought 15 of the 19 fixes between 77 and 96 [5], and where the Validation Evidence warning is enforced [6]. 5. **T4 — Impacted unit tests [D].** The unit tests whose symbols the patch's diff touches run. Seconds. Selection is deterministic from the dependency graph (P4). A failure here — with T3 having passed — is an impact-analysis result: the patch fixed the defect but broke a neighbor it did not mean to touch. 6. **T5 — Affected regression tests [D].** The regression subset selected by dependency and coverage — Ekstazi-style [8], HyRTS-style [7], TestPrune-style [11] — runs, all PASS_TO_PASS obligations in the affected neighborhood. Seconds to minutes. The PASS_TO_PASS distinction from SWE-bench [2] is enforced: these tests answer "did the patch break behavior that worked before?", and a flip here is a regression, typed as such. 7. **T6 — Subsystem integration [D].** The integration checks for the subsystem the patch touches run. Minutes. This is the first tier that exercises interactions across modules, and it is reached only if every cheaper tier passed. 8. **T7 — Full suite [D].** The complete regression suite runs, potentially expensive, reserved for the final certification of a patch that has cleared every rung. A failure only here means a distant dependency escaped targeted selection — itself a signal that the selection tiers' coverage should be widened (Section 5). 9. **(L) In-loop steering and failure interpretation [L] — the at-most-one learned entry.** In an interactive repair loop, the model may invoke test runs as ACI actions to ground its edits [4], and may read a failing test's output against the issue's intent to steer diagnosis [12]. It does not rerun until green, does not waive a failing tier, and does not author the verdict — the deterministic ladder computes the verdict, and the model's interpretation is consumed downstream as a hypothesis about the failure, not as a result. The cross-cutting deterministic services that make the ladder a certification instrument rather than a script: `type_evidence` (classify every result by role per P1), `replay_baseline` (run a test on the buggy state and classify the B/C pair per P2), `select_impacted_tests` (compute the smallest safe regression set from the dependency and coverage graphs per P4), `apply_flake_policy` (cap reruns, record signatures, quarantine flippers per P7), `verify_test_integrity` (reapply protected test files or diff-scope-check test changes per P5), `pin_environment` (containerized parity, locked dependencies, seeded randomness, no network), and `audit_suite_strength` (mutate critical modules and check the suite notices per P6). Every one is deterministic software on CPU. The target is explicit and checkable: the tiers T0 through T7 are software on CPU, and the L column holds at most one entry — the steering-and-interpretation call that contributes the model's only legitimate value (choosing what to probe next and reading what a failure means), never the verdict. Any proposed addition that would put a language model elsewhere in the ladder — having the model decide whether a green result counts, rerun a test until it passes, waive a failing tier, or author the final verdict — is a regression against P5 and the Piper rule and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of Section 6, on held-out tasks, at a defensible cost, or be rejected. The invariant is stated again so it can be enforced mechanically: **"the L column contains at most one entry."** ## 5. Recursive Self-Improvement Protocol Component 17 of 27 follows the program's uniform protocol: the executable specification is written to 100%, round 1 lands at approximately 80% of the specified behavior, and later rounds close the gap. Each round is A/B-gated against the honest metrics of Section 6 — above all evidence-typing coverage and cost-ordered budget efficiency — never the end-to-end resolve rate alone. The three-way split disciplines every change: steering goes to the LLM's single entry (in-loop test invocation and failure interpretation), mechanics go to deterministic software (the ladder, the evidence typing, the selection, the flake policy, the suite audit), and every change to either is A/B-gated on honest metrics, never on vibes. Because test execution decides what a green result even means, its improvement protocol measures the TYPING and COST of the battery rather than trusting the model's sense that "the tests passed," improving itself in four loops. First, *ladder calibration*: the harness measures whether the tier ordering is actually cheapest-first — whether a T2 gate catches everything a T1 gate would have caught at less cost, and whether any tier is ever reached after a cheaper tier would already have failed the patch — and reorders accordingly, guided by the cost-ordering arithmetic of HyRTS's selection times [7] and Bessey et al.'s static-gate economics [23]. Second, *baseline-replay coverage*: the harness measures what fraction of bug-fix tests actually got the fail-on-buggy/pass-on-candidate contrast, and widens the contrast where it is missing, guided by the Validation Evidence prevalence finding that 46.0% of uncontrasted positives prove nothing [6]; a bug-fix test that never demonstrates fail-on-buggy is not evidence, and the loop exists to make every bug-fix test demonstrate it. Third, *selection tightening*: the harness measures the false-negative rate of the regression-selection tier — how often a full-suite failure at T7 escaped the T5 selection — and widens the coverage, guided by Rothermel and Harrold's safety guarantee [9] and TestPrune's coverage-based minimization [11]; a selection tier that leaks regressions to the full suite is not safe, and safety precedes smallness. Fourth, *suite-strength auditing*: the harness periodically mutates critical modules and measures the mutation kill rate, widening the battery where the suite fails to notice, guided by DeMillo, Lipton, and Sayward's Coupling Effect [18] and SWE-ABS's demonstration that stronger suites reject 19.71% of previously passing patches [19]; a battery that does not kill its mutants is a claim without an instrument behind it. Because every result carries an evidence record, a regression in execution is visible in the typing and cost metrics before it surfaces as a wrong verdict, and each A/B change must justify itself on held-out tasks at a defensible cost before it ships. ## 6. Measurement & Evaluation Test execution is measured *separately* and *honestly*, because a stage whose whole purpose is to certify or falsify a patch can be defeated in either of two opposite ways: by certifying on green that proves nothing, or by falsifying on red that proves nothing. The instruments keep the battery accountable to what each result actually demonstrated and what it cost, not to the count of green lights. **Evidence-typing coverage.** The primary instrument reports, for every validation run, what fraction of results carry a typed role (BUG_REPRODUCTION, REGRESSION, EXPECTED_CHANGE, STATIC_GATE) with baseline and candidate results attached, versus the fraction that are bare booleans. This is the Piper rule's first clause made quantitative: a harness that reports "17 tests passed" scores near zero no matter how green it is, because it has not recorded what any of the 17 proved. The measurement is deterministic and outside the model because the model cannot be trusted to type its own results. **Bug-discriminating rate.** The companion instrument reports, of all green results in a run, what fraction were actually bug-discriminating (B FAIL → C PASS) versus regression-only (B PASS → C PASS). This is the Validation Evidence instrument [6]: a run whose greens are mostly regression-only is a run that has not established the bug is fixed, no matter how many tests passed. **Baseline-replay coverage.** The instrument reports what fraction of bug-fix tests received the buggy-baseline contrast at all. A bug-fix test that never demonstrated fail-on-buggy is not bug evidence, and this instrument is the loop that drives that fraction toward one. **Cost-ordering efficiency.** The instrument reports the validation budget spent per patch that was actually falsified: how many patches were halted at a cheap tier versus how many burned an expensive tier that a cheaper one would have caught. This measures the thesis's clause that failure at any tier should halt more expensive validation — a harness that spends ten integration minutes to learn what a lint would have shown in 80 milliseconds is paying for information it already had. **Selection precision and safety.** The instrument reports, for the regression-selection tier, both precision (how much of the suite it selected) and safety (how often a full-suite failure escaped selection). Safety is the hard floor — Rothermel and Harrold's guarantee [9] — and precision is the budget. A selection tier that leaks regressions is unsafe; a selection tier that selects everything is precise only trivially and buys no budget. **Flake rate and rerun counts.** The instrument reports per-test flake signatures and the distribution of rerun counts. A rising rerun count is p-hacking in progress [17]; a test that flips across reruns must be in quarantine, and the instrument verifies it is. **Suite strength (mutation kill rate).** The instrument reports the battery's mutation kill rate on critical modules — the fraction of seeded mutants the suite kills [18][19]. A battery with a falling kill rate is a referee going blind, and the audit loop of Section 5 is driven by this number. **Tamper-detection rate.** The instrument reports how often the diff-scope check or protected-file reapplication caught a change to a test, fixture, or skip marker [2][15]. A zero rate is not proof of honesty; it is a signal to check that the tamper protection is actually installed. A failure-mode audit closes the positive instruments by naming how a certification battery lies. *Empty-green* is exit 0 with zero tests collected — success declared by an empty referee, the micro-scale form of the benchmark soundness failures this component exists to prevent; the harness self-test must include an empty-patch run to prove the referee cannot be scored on without playing. *Rerun-until-green* is flakiness converted into policy, the loop optimizing for dice [17]. *Tampering* is weakened assertions, skipped tests, or mocks replacing the system under test — the optimization target finding the gap in a weak suite [15]. *Environment drift* is green in the sandbox and red in CI, the pinned-environment discipline skipped once and discovered in production. *Target-only certification* is the failing test flipping while its neighbors regress — PASS_TO_PASS shipped broken [2]. *Vacuous green* is a patch no test actually executed — the diff's coverage is zero, and the green certifies nothing. *Suite-blindness* is contaminated green — tests visible in training data or issue text, the failure LiveCodeBench exists to exclude [26]. Two honest caveats close the section. First, the strong-pipeline numbers of Section 2 — HyRTS's selection precision [7], Agentless's 77-to-96 ablation [5], SWE-ABS's 19.71% rejection [19], TestPrune's 9.4%–12.9% relative gain [11] — were measured on repositories and models far larger than a weak-model harness controls, and a weak model should expect materially smaller absolute effects while leaning all the harder on the cheap tiers, the typed evidence, and the smallest-safe-set discipline to make its small budget count. Second, all execution metrics must be reported on held-out and out-of-benchmark tasks, as in the earlier components, so a battery that has memorized a task set's answers cannot score well on suite strength without its verdicts having been measured on problems it never saw. ## 7. Conclusion Test execution is where the coding harness's most expensive lie is told in its cheapest-looking form: an exit code. This paper has argued that "green" is not a fact about a patch but a claim about a suite, and that a weak-model harness must therefore run its certification battery deterministically, type every result by the obligation it discharged, contrast bug-fix evidence against the buggy baseline whenever possible, select the smallest safe regression set from dependency and coverage, and stop at the first cheap failure. The thesis that has governed every design choice is the one with which this paper began and closes: **Test execution should be deterministic, evidence-typed, change-aware, and cost-ordered: every candidate patch progresses through an escalating validation ladder from cheap static gates to bug-specific reproduction, impacted unit tests, regression tests, integration checks, and finally broader suites; each test result is classified by whether it demonstrates bug resolution, regression preservation, or merely unrelated success; bug-fix evidence should be contrasted against the original buggy state whenever possible; dependency and dynamic-coverage information should select the smallest safe regression set; and failure at any tier should halt more expensive validation and return structured evidence to diagnosis.** And the rule that compresses that thesis into one sentence of discipline for Piper is: **Do not ask 'Did the tests pass?' Ask 'Which tests passed, what property did each one actually prove, and did we run the cheapest sufficient set capable of disproving this patch?'** The model steers — its single learned contribution is to choose what to probe next in the loop and to read what a failure means, never to author the verdict. The harness certifies — the ladder from syntax to full suite is deterministic software, the evidence typing is deterministic, the baseline replay is deterministic, the selection is deterministic, the flake policy is deterministic, and the suite-strength audit is deterministic, because a verdict that a weak model could flatter is not a verdict. The architecture holds the invariant mechanically: the L column contains at most one entry, and the deterministic battery is the default. The measured message of the literature is unambiguous — Codex and HumanEval founded correctness on executed behavior [1], SWE-bench made the two-sided protocol an institution [2], Agentless showed reproduction-plus-regression validation is worth fifteen fixes on three hundred problems [5], the Validation Evidence study showed nearly half of passing results prove nothing about the bug [6], HyRTS showed the selection that makes the ladder affordable [7], Luo, Hariri, Eloussi, and Marinov showed the flakiness that corrupts it [17], and DeMillo, Lipton, and Sayward showed, before the personal computer, that the suite itself is a measured instrument [18]. For a weak model, a green light on a weak suite is a coin flip with a confident face; what a weak-model harness needs is a deterministic, typed, baseline-contrasted, cost-ordered battery that knows exactly which property each green result proved — because the question is never whether the tests passed, but whether the cheapest sufficient set that could have disproved the patch was actually run, and what each of its results actually showed. --- ## References 1. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, the operator Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. *Evaluating Large Language Models Trained on Code.* arXiv 2021. arXiv:2107.03374. *(The founding contract: correctness is executed behavior under unit tests with an unbiased pass@k estimator; sandboxing from day one.)* 2. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* ICLR 2024. arXiv:2310.06770. *(2,294 issue/PR pairs, 12 Python repos; FAIL_TO_PASS/PASS_TO_PASS; mechanical test-patch reapplication; Dockerized per-instance environments.)* 3. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. *SWE-Bench+: Enhanced Coding Benchmark for LLMs.* arXiv 2024. arXiv:2410.06992. *(32.67% solution leakage; 31.08% weak tests. The empty-patch finding is NOT this paper's and is not attributed to it.)* 4. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. *SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.* NeurIPS 2024. arXiv:2405.15793. *(Tests as first-class ACI actions; 12.5% pass@1 SWE-bench, 87.7% HumanEvalFix.)* 5. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. *Agentless: Demystifying LLM-Based Software Engineering Agents.* arXiv 2024. arXiv:2407.01489. *(LOCALIZE/REPAIR/VALIDATE; regression + reproduction tests; 77 → 81 → 96 fixes on 300 SWE-bench Lite.)* 6. Xiaonan Xu and Wenjing Wu. *Validation Evidence in LLM Repair Agents: How Much of What Passes Actually Tests the Bug?* arXiv 2026. arXiv:2607.28871. *(46.0% of positive comparable validation events carried no bug-discriminating information; 23.8% of baseline trajectories; 3,730 events / 643 rollouts / 110 tasks.)* 7. Lingming Zhang. *Hybrid Regression Test Selection.* IEEE/ACM 40th International Conference on Software Engineering, ICSE 2018. DOI 10.1145/3180155.3180198. *(Non-arXiv. 2,707 revisions, 32 Java projects, 124M+ LoC; 18.35% vs 27.18% selection; 31.60s vs 40.06s.)* 8. Milos Gligoric, Lamyaa Eloussi, and Darko Marinov. *Ekstazi: Lightweight Test Selection.* IEEE/ACM 37th International Conference on Software Engineering, ICSE 2015. DOI 10.1109/ICSE.2015.230. *(CORRECTED VENUE: ICSE 2015, not ESEC/FSE 2015.)* 9. Gregg Rothermel and M. J. Harrold. *A Safe, Efficient Regression Test Selection Technique.* ACM Transactions on Software Engineering and Methodology (TOSEM), 1997. DOI 10.1145/248233.248262. *(Non-arXiv; the safe-RTS foundation.)* 10. Pepe Alonso, Sergio Yovine, and Victor A. Braberman. *TDAD: Test-Driven Agentic Development — Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis.* arXiv 2026. arXiv:2603.17973. *(Graph-based impact/regression analysis before submission.)* 11. Yang Chen, Toufique Ahmed, Reyhaneh Jabbarvand, and Martin Hirzel. *Can Old Tests Do New Tricks for Resolving SWE Issues?* FSE 2026. arXiv:2510.18270. *(TestPrune; coverage-based selection minimized to k = 10; 9.4%–12.9% relative gain in Agentless.)* 12. Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. *SpecRover: Code Intent Extraction via LLMs.* ICSE 2025. arXiv:2408.02232. *(Regression tests per patch + reviewer interpreting failures against extracted intent + confidence score.)* 13. Ira Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti, Shyam Ramji, Junfeng Yang, Gail Kaiser, and Baishakhi Ray. *Understanding Automated Program Repair Agents Through the Lens of Traceability: An Empirical Study.* ISSTA 2026. arXiv:2506.08311. *(CORRECTED: venue ISSTA 2026; test-running content in §4 and §5, not §6.2, which concerns patch edit-distance similarity.)* 14. Islem Bouzenia and Michael Pradel. *You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects.* ISSTA 2025. arXiv:2412.10133. *(ExecutionAgent; gets arbitrary suites to run.)* 15. Ziqian Zhong, Aditi Raghunathan, and Nicholas Carlini. *ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases.* arXiv 2025. arXiv:2510.20270. *(How often "passing" is tampering with the checks.)* 16. Toufique Ahmed, Jatin Ganhotra, Rangeet Pan, Avraham Shinnar, Saurabh Sinha, and Martin Hirzel. *Otter: Generating Tests from Issues to Validate SWE Patches.* ICML 2025. arXiv:2502.05368. *(Reproduction tests validated against the buggy baseline.)* 17. Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. *An Empirical Analysis of Flaky Tests.* 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2014. DOI 10.1145/2635868.2635920. *(CORRECTED: FSE 2014, not ISSTA 2014; no arXiv ID — 1407.3067 resolves to an unrelated population-genetics paper and must never be used.)* 18. Richard A. DeMillo, Richard J. Lipton, and Frederick G. Sayward. *Hints on Test Data Selection: Help for the Practicing Programmer.* IEEE Computer, Vol. 11, No. 4, pp. 34–41, April 1978. DOI 10.1109/C-M.1978.218136. *(Mutation analysis; Competent Programmer Hypothesis; Coupling Effect.)* 19. Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand. *SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark.* arXiv 2026. arXiv:2603.00520. *(CORRECTED TITLE and FIGURE: 19.71% of previously passing patches rejected; top agent 78.8% → 62.2%.)* 20. Richard A. DeMillo and A. Jefferson Offutt. *Constraint-Based Automatic Test Data Generation.* IEEE Transactions on Software Engineering, Vol. 17, No. 9, 1991. DOI 10.1109/32.92910. *(Mutation-based test data generation; the Godzilla toolset.)* 21. René Just, Darioush Jalali, and Michael D. Ernst. *Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs.* ISSTA 2014. DOI 10.1145/2610384.2628055. *(Engineered fault→test corpus; APR's research substrate.)* 22. Michael Hilton, Timothy Tunnell, Kai Huang, Darko Marinov, and Danny Dig. *Usage, Costs, and Benefits of Continuous Integration in Open-Source Projects.* IEEE/ACM International Conference on Automated Software Engineering, ASE 2016. DOI 10.1109/ASE.2016.30. *(CORRECTED VENUE: ASE 2016, not ICSE 2016.)* 23. Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, the operator McPeak, and Dawson Engler. *A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World.* Communications of the ACM, Vol. 53, No. 2, 2010. DOI 10.1145/1646353.1646374. *(Coverity; false-positive economics of the lint tier.)* 24. Zhi Chen, Zhensu Sun, Yuling Shi, Chao Peng, Xiaodong Gu, David Lo, and Lingxiao Jiang. *Rethinking the Value of Agent-Generated Tests for LLM-Based Software Engineering Agents.* arXiv 2026. arXiv:2602.07900. *(Net-change-of-zero is model-specific: gpt-5.2 and claude-opus-4.5 = 0; gemini −13; deepseek −2; relational/range checks 2.98%–7.86%.)* 25. Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. *TestGenEval: A Real World Unit Test Generation and Test Completion Benchmark.* arXiv 2024. arXiv:2410.00752. *(CORRECTED AUTHORS: Jain, Synnaeve, and Rozière — not "Löbel et al.")* 26. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. *LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.* arXiv 2024. arXiv:2403.07974. *(Contamination-free, continuously refreshed execution evaluation.)*