Summary / 17. Test execution / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 17 — Test Execution (the deterministic, evidence-typed, cost-ordered certification stage) > Derived from `papers/17-test-execution.md` — "Green Is a Claim About the Suite, Not the Code: Deterministic, Evidence-Typed, Cost-Ordered Test Execution for a Weak-Model Coding Harness." > This document drives the build of the **test-execution stage**: the deterministic, evidence-typed, change-aware, > cost-ordered certification battery in which a candidate patch meets reality and the harness commits to the VERDICT it > will carry forward. It sits at the point where everything upstream — the decomposed request, the candidate hypothesis > set, the repair — is either certified or falsified: components 01–15 assembled the request, the dependency graph, the > coverage, and the candidate; this component runs the battery that decides what "green" even means for whatever they > produced. The sharp distinction is between a BOOLEAN and an EVIDENCE RECORD: a boolean is an exit code with no > provenance; an evidence record is a per-test `ValidationResult` carrying the obligation it discharged, its baseline and > candidate results, its flake history, and whether it actually discriminated the bug. Naive harnesses run `pytest` and > read a green light; this stage runs a tiered validation ladder — cheap static gates first, then bug-specific > reproduction, then impacted unit tests, then affected regression, then integration, then the full suite — types every > result by role, contrasts bug-fix evidence against the buggy baseline, selects the smallest safe regression set from > dependency and coverage, caps flakiness so rerun-until-green cannot p-hack the loop, tamper-protects the test files so > deleting a flaky test is not a fix, and audits the suite itself so green carries a measured meaning. The output of > component 17 is **not a patch** and **not a boolean** — it is a typed **EVIDENCE RECORD**: a structured verdict per > tier with roles, baseline/candidate contrasts, flake signatures, tamper detections, and the tier at which the patch > failed, if it did. Every tier from syntax to full suite is deterministic software on CPU; the weak model's AT MOST ONE > contribution is a single learned step of STEERING AND INTERPRETATION (invoke test runs in-loop and read a failure > against the issue's intent), and it never authors the verdict. The guiding rule, made mechanical: **Do not ask "Did the > tests pass?" Ask "Which tests passed, what property did each one actually prove, and did we run the cheapest sufficient > set capable of disproving this patch?"** This spec pins the paper's design to exact primitives, the exact > `ValidationResult` schema, the exact evidence-role enum, the exact B/C classification, exact per-tier caps, exact > L-column firing, exact A/B instruments and thresholds, exact ablation mechanics, and an exact build sequence. Where the > paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible test-execution stage** — a deterministic, evidence-typed, change-aware, cost-ordered validation ladder of syntax/parse, lint and static checks, type check and build, bug-specific reproduction, impacted unit tests, affected regression tests, subsystem integration, and the full suite — that turns a candidate patch plus the buggy baseline plus the repository's own battery plus the change-impact and coverage information from components 06 and 12 into a typed **evidence record** certifying or falsifying the patch. It differs from its predecessors in *kind*, not just position: the repair loop (14) asks *"how do I improve this failed patch?"* and loops the **CANDIDATE**; reflection (15) asks *"what did this failure teach?"* and loops the agent's **UNDERSTANDING**; test execution asks *"which tests passed, what property did each one actually prove, and did we run the cheapest sufficient set that could have disproved this patch?"* and certifies the **VERDICT** those stages act on — without which "fixed" and "broken" would collapse into one number the model is free to flatter. The paper's seven principles decide every design choice. **P1 A green result is typed evidence, not a boolean** — every validation result carries a role stating which obligation it discharged (BUG_REPRODUCTION, REGRESSION, EXPECTED_CHANGE, STATIC_GATE) plus its baseline and candidate results, so "17 tests passed" is noise and a terrible patch that fixes the bug while destroying everything else is distinguishable from a vacuous patch that preserves everything while never fixing the defect. **P2 A bug-fix test is not bug evidence until it shows fail-on-buggy and pass-on-candidate** — the candidate's bug-specific test is run on the buggy state B and the candidate state C and the pair is classified deterministically (B FAIL / C PASS = bug-discriminating; B PASS / C PASS = regression-only; B FAIL / C FAIL = no fix; B PASS / C FAIL = regression), because 46.0% of positive comparable validation events carry no bug-discriminating information. **P3 Escalate by cost and information, stop at the first cheap failure** — a patch fails at the earliest relevant tier, and which tier caught the failure is itself debugging evidence. **P4 Select the smallest safe regression set** — dependency and coverage select the tests that can plausibly be affected, safety first, smallness second, the full suite held in reserve. **P5 The harness owns the verdict; the model only steers** — the battery executes in a pinned, tamper-protected environment, and the model's legitimate channels are in-loop test runs and failure interpretation, never the verdict. **P6 Green is a claim about the suite, not just the code, so the suite is a measured instrument** — the harness periodically mutates critical modules and checks the suite notices; a battery that does not kill its mutants is not a referee. **P7 Flakiness is p-hacking unless it is capped** — at most N reruns per test with recorded signatures, a test that flips across reruns goes to quarantine rather than victory, because rerun-until-green teaches the agent that persistence, not correctness, flips tests. **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — the eight ladder tiers (T0–T7), evidence typing and the `ValidationResult` store, baseline replay and B/C classification, dependency-based change-aware selection, the flake policy, tamper protection, environment pinning, and a basic suite-strength (mutation) audit — plus the *bound* of the *one* in-loop steering-and-interpretation call. The heavy pieces (full dynamic-coverage selection, advanced mutation-audit automation, adaptive per-difficulty tier budgets, cross-session flake history) are **not** built in Round 1 and are **not excluded** — later rounds climb toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by **evidence-typing coverage** and its co-signatory **bug-discriminating rate**, with companion **cost-ordering efficiency**, never the end-to-end resolve rate alone and never model self-assessment (§7). The stage lives in `build/17-test-execution/` and writes nothing into the live harness. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–13); **DEFERRED = later rounds** (rows 14–17), **not excluded** but gated by the same §7 A/B protocol. Cross-session flake history is this component's own concern (it does **not** hand it off); every later round of this component still builds within the same `build/17-test-execution/` tree. | # | Component (paper §4) | Classification | |---|---|---| | 1 | Evidence typing + the `ValidationResult` store — `type_evidence` classifies every result by role per P1 and persists it; "17 tests passed" is refused | **BUILD-NOW** | | 2 | Baseline replay + B/C classification — `replay_baseline` runs a test on the buggy state B and the candidate state C and classifies the pair per P2 | **BUILD-NOW** | | 3 | T0 Syntax and parse [D] — the changed files must parse; millisecond gate; failure halts immediately | **BUILD-NOW** | | 4 | T1 Lint and static checks [D] — linters over changed files; hard failures vs warnings, both recorded (triaged, never oracle) | **BUILD-NOW** | | 5 | T2 Type check and build [D] — changed targets type-check and build in the pinned environment; structured compiler diagnostics | **BUILD-NOW** | | 6 | T3 Bug-specific reproduction [D] — the reproducer (repo test or generated reproduction validated against the buggy baseline) replayed on B and C; the pair classified per P2 | **BUILD-NOW** | | 7 | T4 Impacted unit tests + T5 affected regression tests + change-aware selection — `select_impacted_tests` computes the smallest safe regression set from the dependency (06) and coverage (12) graphs per P4 | **BUILD-NOW** | | 8 | T6 Subsystem integration + T7 full suite [D] — integration checks for the touched subsystem, then the reserved full suite | **BUILD-NOW** | | 9 | Flake policy — `apply_flake_policy` caps reruns, records signatures, quarantines flippers per P7 | **BUILD-NOW** | | 10 | Tamper protection + environment pinning — `verify_test_integrity` reapplies protected test files or diff-scope-checks; `pin_environment` locks deps, seeds randomness, no network per P5 | **BUILD-NOW** | | 11 | Suite-strength audit — `audit_suite_strength` mutates critical modules and checks the suite notices per P6 (basic in Round 1) | **BUILD-NOW** | | 12 | (L) In-loop steering and failure interpretation — the model invokes test runs as ACI actions and reads a failing test against the issue's intent; never the verdict | **BUILD-NOW** (bounded stub only) | | 13 | Loop driver + A/B harness (§7 metrics + ablation runner + failure-mode audits) | **BUILD-NOW** | | 14 | **Full dynamic-coverage selection at scale** | **DEFERRED** — Round 1 ships dependency-based selection from components 06/12; TestPrune-style coverage-based minimization to a budget is deferred | | 15 | **Advanced mutation-audit automation (SWE-ABS-style adversarial strengthening)** | **DEFERRED** — Round 1 ships a basic kill-rate audit; systematic suite strengthening is deferred | | 16 | **Adaptive per-difficulty tier budgets** | **DEFERRED** — Round 1 ships FIXED tier cost caps; a variable budget that reorders tiers per task is deferred | | 17 | **Cross-session flake history / persistent quarantine** | **DEFERRED** — Round 1 ships per-session flake signatures; a shared quarantine across repo-sessions is deferred | | 18 | **Learned judgment anywhere in the ladder** — a model deciding whether a green result counts, rerunning a test until it passes, waiving a failing tier, or authoring the final verdict | **DEFERRED**, and rejected unless it beats the deterministic stage on honest metrics | **The cost rule (explicit).** The single learned entry — in-loop steering and failure interpretation — is the AT MOST ONE L-column entry this architecture permits. In Round 1 it is a **bounded stub**: wired in at its firing point (§4), seeded, schema-constrained, never auto-promoted; its full training and calibration is **deferred**. Its cost is bounded before invocation: one call per steering action, temperature zero, schema-constrained, bounded payload, parse-failure fallback that returns to the deterministic stage with the violation fed back (§4 caps). It is kept **only if** the §7 no-L-vs-L ablation shows it improves the KEY measures on held-out tasks; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the typing, replay, selection, flake, tamper, pinning, audit, or verdict path by default — the full deterministic ladder (rows 1–11, 13) plus the *bounded stub* of the single steering call (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a learned selection/scope/judgment, a full training run, or break the at-most-one L-column invariant — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out tasks, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Eight deterministic tiers — T0 through T7 — plus the single learned entry (L). The L column holds AT MOST ONE thing: in-loop steering and failure interpretation (§4), reached only when the harness operates inside a repair loop. A pure certification run of a candidate patch has ZERO L entries — every tier from syntax to full suite is deterministic software on CPU, and the verdict is computed by the harness: ``` CANDIDATE PATCH (C) + BUGGY BASELINE (B) + REPO BATTERY + CHANGE-IMPACT (06) + COVERAGE (12) → T0 syntax & parse [D] → parse gate; failure halts the ladder immediately → T1 lint & static checks [D] → hard failures vs warnings, both recorded → T2 type check & build [D] → structured compiler diagnostics returned → T3 bug-specific reproduction [D] → replay on B and C; B/C pair classified (P2) → T4 impacted unit tests [D] → symbol-touched units from the dependency graph (P4) → T5 affected regression tests [D] → smallest safe regression set (P4) → T6 subsystem integration [D] → integration checks for the touched subsystem → T7 full suite [D] → reserved; a failure only here = selection gap → (L) in-loop steering & failure interpretation [L — the at-most-one entry] ``` The pipeline reasons over **the candidate patch made into a certified or falsified verdict** — the buggy baseline, the repository's own battery, and the change-impact and coverage graphs are the only things the battery runs against, and every result it produces is typed, contrasted, selected, flake-capped, tamper-checked, and audited deterministically. All artifacts land in `build/17-test-execution/`. ### 3.1 The evidence contract: `ValidationResult` and the evidence-role enum (D) **Input:** one executed test (or static check) and its raw outcome. **Output:** a typed `ValidationResult`. **No language model runs here.** Every result the ladder produces is bound deterministically into the typed `ValidationResult`, the single currency the harness records, reports, and A/B-measures (P1). A result that cannot be typed — a bare boolean with no role and no baseline/candidate contrast — is not a result the harness will certify on; it is recorded as a *typing failure*, which is itself a scored defect (§7.1), never silently promoted to green. **The `ValidationResult` schema (exact — EXACTLY these fields):** ```json { "test_id": "…", // the test's (or static check's) identifier "role": "…", // the evidence role it discharged (the paper's "category") "candidate_commit": "…", // the candidate patch state C under test "baseline_result": "…", // result on the buggy state B (present iff a contrast was run) "candidate_result": "…", // result on the candidate state C "duration": 0.0, // wall-clock seconds "exit_code": 0, // the raw process exit code "flaky_history": [ "…" ], // prior flake signatures (§3.11) "affected_by_change": false, // whether the change-impact graph reaches this test (P4) "bug_discriminating": false, // B FAIL / C PASS (P2) — set deterministically, never by a model "stdout_summary": "…", // truncated captured output "failure": null } // structured failure diagnostic, or null on pass ``` `test_id` identifies the check; `role` carries the evidence role (the paper's "category"); `candidate_commit` the candidate state under test; `baseline_result` and `candidate_result` hold the B/C contrast (§3.2); `duration` the wall clock; `exit_code` the raw process code; `flaky_history` the prior signatures (§3.11); `affected_by_change` whether the change-impact graph reaches it (P4); `bug_discriminating` the deterministic B/C classification (§3.2); `stdout_summary` the truncated output; `failure` the structured diagnostic or `null`. **The evidence-role enum (exact — EXACTLY these values):** ``` BUG_REPRODUCTION // fail-on-buggy, pass-on-candidate — the bug is fixed REGRESSION // pass-on-baseline, pass-on-candidate — behavior preserved (PASS_TO_PASS) EXPECTED_CHANGE // old behavior and new behavior, both observed (the intended diff) STATIC_GATE // parse, lint, type, or build succeeded ``` **Caps:** evidence typing is pure deterministic software on CPU — no model types its own results, and the four-role enum is closed: a result that fits no role is a typing failure, never a green light. ### 3.2 Baseline-replay contrast and B/C classification (D) **Input:** a test with both a buggy-state run and a candidate-state run. **Output:** the deterministic B/C classification recorded on `bug_discriminating` and the pair stored on `baseline_result`/`candidate_result`. **No language model runs here.** The Validation Evidence warning — 46.0% of positive comparable validation events carried no bug-discriminating information — makes baseline replay mandatory wherever affordable (P2). The candidate's bug-specific test is run on the buggy state B and the candidate state C, and the pair is classified deterministically: ``` B FAIL / C PASS → bug-discriminating (the fix is real) B PASS / C PASS → regression-only (proves nothing about the bug) B FAIL / C FAIL → no fix (the bug is not fixed) B PASS / C FAIL → regression (the patch broke what it meant to fix) ``` Only the first cell is bug evidence; a passing test that would also have passed on the buggy code establishes only that the patch did not break that test. The classification is arithmetic over the pair, computed by the harness — never by a model reading the output. **Procedure (exact, deterministic):** 1. Run the test on the buggy state B; record `baseline_result`. 2. Run the test on the candidate state C; record `candidate_result`. 3. Classify the pair by table lookup into the four cells above; set `bug_discriminating = true` iff B FAIL / C PASS. **Caps:** baseline replay is pure deterministic software on CPU — the B/C pair is classified by table lookup, and a bug-fix test that never received the B contrast is recorded with `baseline_result` absent and `bug_discriminating` false, which §7.2's baseline-replay-coverage instrument drives toward one. ### 3.3 T0 — Syntax and parse (D) **Input:** the changed files of the candidate patch. **Output:** a parse gate; a parse failure halts the ladder immediately. **No language model runs here.** The changed files must parse. Millisecond cost. This is an edit failure, not a semantic one — no interpretation is needed to know the patch is not ready, and no later tier may run after a T0 failure. **Procedure (exact, deterministic):** 1. Parse each changed file with the language's parser (`ast.parse` for Python). 2. On any parse failure, record `{ role: STATIC_GATE, failure: <diagnostic> }` and halt the ladder. 3. On success, record `{ role: STATIC_GATE, candidate_result: "pass" }` and advance to T1. **Caps:** parse is pure deterministic software on CPU — the model never argues a syntax error into submission; a T0 failure returns the parse diagnostic as `failure` with `role = STATIC_GATE` and halts. ### 3.4 T1 — Lint and static checks (D) **Input:** the changed files. **Output:** hard failures vs warnings, both recorded. **No language model runs here.** The repository's linters and cheap static analyzers run over the changed files. Milliseconds to seconds. The triage discipline of the Coverity economics applies: a lint warning is a signal to be interpreted, not an oracle, so the harness distinguishes hard failures (which halt) from warnings (which are recorded and passed through), rather than flattening a noisy linter into a boolean gate. **Procedure (exact, deterministic):** 1. Run the repository's configured linters over the changed files. 2. Classify each finding as a hard failure (halts) or a warning (recorded, passed through). 3. Record both classes as `ValidationResult`s with `role = STATIC_GATE`; halt only on a hard failure. **Caps:** lint is pure deterministic software on CPU — the model never decides whether a warning is fatal; the harness records both classes and halts only on a hard failure. ### 3.5 T2 — Type check and build (D) **Input:** the changed targets. **Output:** a type/build gate returning the compiler's structured diagnostic. **No language model runs here.** The changed targets type-check and build in the pinned environment (§3.11). Seconds. This is the last gate before any behavior is executed; a type or build failure returns the compiler's structured diagnostic directly to the diagnosis stage with `role = STATIC_GATE`. **Procedure (exact, deterministic):** 1. Type-check and build the changed targets in the pinned environment. 2. On failure, record the compiler's structured diagnostic and halt. 3. On success, record and advance to T3. **Caps:** type/build is pure deterministic software on CPU — the model never reinterprets a compiler error into a pass; the structured diagnostic is the result. ### 3.6 T3 — Bug-specific reproduction (D) **Input:** the bug-specific reproducer and both repository states B and C. **Output:** the B/C pair classified per §3.2. **No language model runs here.** The reproducer — the repository test that exposes the defect, or a generated reproduction test validated against the buggy baseline first (Otter's line) — runs on the buggy state B and the candidate state C, and the pair is classified per P2: B FAIL / C PASS is bug-discriminating and the patch advances; B PASS / C PASS means the reproducer does not actually reproduce the defect and must be revalidated; B FAIL / C FAIL means the bug is not fixed and the patch halts with that evidence; B PASS / C FAIL means the patch regressed the very behavior it was meant to fix. This is the tier where reproduction-plus-regression validation bought fifteen of nineteen fixes in the Agentless ablation, and where the Validation Evidence warning is enforced. **Procedure (exact, deterministic):** 1. Validate the reproducer against the buggy baseline — it must fail on B before it may certify on C. 2. Run it on the buggy state B and the candidate state C. 3. Classify the pair per §3.2 and set `bug_discriminating` accordingly. **Caps:** reproduction is pure deterministic software on CPU — the model never decides whether the reproducer "counts"; the pair is classified arithmetically, and a reproducer that cannot demonstrate fail-on-buggy is not bug evidence until it does (§8 invariant 2). ### 3.7 T4 — Impacted unit tests (D) **Input:** the unit tests whose symbols the patch's diff touches, from the dependency graph (component 06). **Output:** a pass/fail gate over the impacted units. **No language model runs here.** The unit tests whose symbols the patch's diff touches run. Seconds. Selection is deterministic from the dependency graph (P4). A failure here — with T3 having passed — is an impact-analysis result: the patch fixed the defect but broke a neighbor it did not mean to touch. **Procedure (exact, deterministic):** 1. From the dependency graph (component 06), compute the unit tests whose symbols the diff touches. 2. Run those units. 3. A failure (with T3 passed) is typed `REGRESSION` and recorded as an impact-analysis result. **Caps:** impacted-unit execution is pure deterministic software on CPU — the model neither chooses which units to run nor interprets a T4 failure; the set is computed from the dependency graph, and the failure is typed `REGRESSION`. ### 3.8 T5 — Affected regression tests (D) **Input:** the regression subset selected by dependency and coverage (Ekstazi-style, HyRTS-style, TestPrune-style). **Output:** a pass/fail gate over the affected regression set. **No language model runs here.** The regression subset selected by dependency and coverage runs — all PASS_TO_PASS obligations in the affected neighborhood. Seconds to minutes. The PASS_TO_PASS distinction is enforced: these tests answer "did the patch break behavior that worked before?", and a flip here is a regression, typed as such. Selection is `select_impacted_tests` (§3.11): the smallest SAFE set — a superset of what the change could affect — computed deterministically from components 06 and 12. **Procedure (exact, deterministic):** 1. Compute the smallest safe regression set from the dependency (06) and coverage (12) graphs. 2. Run the set. 3. A flip is typed `REGRESSION`; a regression that escapes T5 to T7 is recorded as a selection-safety defect. **Caps:** regression selection and execution are pure deterministic software on CPU — the model never narrows the safe set to make itself look good; a T5 failure is typed `REGRESSION`, and a regression that escapes T5 to T7 is a selection-safety defect scored by §7.2. ### 3.9 T6 — Subsystem integration (D) **Input:** the integration checks for the subsystem the patch touches. **Output:** a pass/fail gate over subsystem integration. **No language model runs here.** The integration checks for the subsystem the patch touches run. Minutes. This is the first tier that exercises interactions across modules, and it is reached only if every cheaper tier passed. **Procedure (exact, deterministic):** 1. Run the subsystem integration checks for the touched subsystem. 2. A failure is typed `REGRESSION` and halts before the full suite. **Caps:** integration is pure deterministic software on CPU — the model never skips a subsystem check it finds inconvenient; a T6 failure is typed `REGRESSION` and halts before the full suite. ### 3.10 T7 — Full suite (D) **Input:** the complete regression suite. **Output:** a pass/fail gate over the full suite. **No language model runs here.** The complete regression suite runs — potentially expensive, reserved for the final certification of a patch that has cleared every rung. A failure only here means a distant dependency escaped targeted selection — itself a signal that the selection tiers' coverage should be widened (§5). **Procedure (exact, deterministic):** 1. Run the complete regression suite. 2. A failure only here is recorded as a selection gap and scored by §7.2's safety instrument. **Caps:** full-suite execution is pure deterministic software on CPU — the model never declares "close enough" on a T7 failure; a T7-only failure is recorded as a selection gap and scored by §7.2's safety instrument. ### 3.11 Cross-cutting deterministic services (D) Six services make the ladder a certification instrument rather than a script, each pure deterministic software on CPU: `select_impacted_tests` (compute the smallest safe regression set from the dependency and coverage graphs per P4 — safety first, smallness second); `apply_flake_policy` (cap reruns at N per test, record signatures, quarantine a test that flips across reruns per P7 — distinct diagnostics kept for timeouts and hangs, component 11's non-termination rule); `verify_test_integrity` (reapply protected test files or diff-scope-check that no test, fixture, or skip/xfail/mock-of-SUT marker changed per P5 — deleting a flaky test is not a fix); `pin_environment` (containerized parity with CI, locked dependencies, seeded randomness, no network by default per P5); `audit_suite_strength` (mutate critical modules and check the suite notices per P6 — a battery that does not kill its mutants is not a referee); and `type_evidence`/`replay_baseline` (the typing and B/C classification of §3.1/§3.2). Every one is software on CPU, and none consults a model. **Procedure (exact, deterministic):** 1. `select_impacted_tests` — intersect the change-impact graph with the test dependency graph to yield the smallest set that is a superset of what the change could affect. 2. `apply_flake_policy` — count reruns against the cap N; record the signature; quarantine a test that flips across reruns. 3. `verify_test_integrity` — recompute protected-file hashes, or diff-scope-check that no test/fixture/skip marker changed; on violation, reapply the protected files and record a tamper detection. 4. `pin_environment` — lock dependencies, seed randomness, and drop network for the run. 5. `audit_suite_strength` — seed mutants into critical modules, run the suite, and record the kill rate. **Caps:** all six services are pure deterministic software on CPU — the model never selects the set, caps the reruns, reapplies the files, pins the environment, audits the suite, or types a result. ### 3.12 The single L entry — in-loop steering and failure interpretation (L) **(L) In-loop steering and failure interpretation [L — the at-most-one learned entry].** In an interactive repair loop, the model may invoke test runs as agent-computer-interface (ACI) actions to ground its edits, and may read a failing test's output against the issue's intent to steer diagnosis (SpecRover's reviewer half). It does **not** rerun until green, does **not** waive a failing tier, does **not** author the verdict, and does **not** decide whether a green result counts — the deterministic ladder computes the verdict, and the model's interpretation is consumed downstream as a hypothesis about the failure, not as a result. This entry is detailed in §4. **The L column holds at most one entry, and the deterministic battery is the default.** Tiers T0–T7 and the six cross-cutting services are deterministic software on CPU; the single learned node is §4, reached only in-loop, and never a model that runs over the whole ladder. Any addition putting a model elsewhere — typing a result, classifying a B/C pair, selecting the regression set, capping reruns, reapplying test files, pinning the environment, auditing the suite, authoring the verdict, rerunning until green, waiving a failing tier, or judging whether green counts — is a regression against P1–P7 and must clear the §7 bar on held-out tasks at defensible cost, or be rejected. **Enforced mechanically: "the L column contains at most one entry."** --- ## 4. The LLM Step (L, AT MOST ONE entry) The **single** learned node in the entire stage: **`steer_and_interpret`** — in-loop steering and failure interpretation. The weak model, operating inside a repair loop, invokes test runs as agent-computer-interface actions to ground its edits (SWE-agent's in-loop half), and reads a failing test's message against the issue's extracted intent to guide diagnosis (SpecRover's reviewer half). This is the STEERING-AND-INTERPRETATION step and nothing more: **choose what to probe next in the loop, and read what a failure means.** The model emits only a STEERING ACTION and an INTERPRETATION — it is **steering, not certifying**. The model is **forbidden from authoring the verdict, deciding whether a green result counts, rerunning a test until it passes, waiving a failing tier, or interpreting a result in a way that overrides the deterministic classification** — those are deterministic and owned by the harness (§3). **Prompt inputs (assembled deterministically, in this exact order):** 1. **`candidate_patch`** — the candidate patch (or candidate repository state) under diagnosis. 2. **`issue_intent`** — the extracted specification/issue intent the patch is meant to satisfy. 3. **`failing_result`** — the deterministic `ValidationResult`(s) the battery just produced, including the tier that failed and the structured diagnostic. 4. **`tier_state`** — which tiers have run and passed, so the model's steering cannot re-argue a tier that already failed. **Input → output schema (exact):** the model emits a JSON object conforming exactly to `{ steering_action, interpretation }`. `steering_action` names a specific test or probe to run next (a concrete ACI invocation), or `null` if the model has nothing to probe; `interpretation` states, in natural language, what the failure means against the issue's intent — a HYPOTHESIS for the diagnosis stage, never a verdict. Neither field may contradict the deterministic result it is handed: an `interpretation` that asserts "green" over a deterministic failure is a schema violation. **Caps:** **ONE call per steering action**, `temperature = 0`, `max_completion_tokens` bounded, schema-constrained; a bounded payload. The model **STEERS only, never certifies, judges, reruns, or waives**. **Gating (exact):** the model may invoke a test run as an ACI action and read a failure against the issue's intent. It does **not** decide whether a green result counts, does **not** rerun a test until it passes, does **not** waive a failing tier, and does **not** author the verdict — those are the ladder's deterministic properties (§3); an output asserting any of them is rejected deterministically and never admitted on the model's say-so (§8 invariants 1 and 3). **Parse-failure fallback (exact):** any JSON parse failure, schema violation (a missing `steering_action`, an `interpretation` that asserts green over a deterministic failure, or an action that waives/re-runs a tier), or empty/malformed output is **not silently retried as a fresh steering call**: the stage **returns to the deterministic stage with the violation fed back** — the malformed steering action is recorded as a violation and the ladder's deterministic verdict stands unmodified — **never a re-prompt, never a second call at the same seed**. The L column is reached **once** per steering opportunity. **Re-invocation on a later round (exact):** steering is re-considered only when a *later round* of the same repair loop is processed, scoped strictly to that round's candidate patch and fed the accumulated deterministic results (§3) as its `tier_state` and `failing_result`. It remains the **single conceptual learned entry** — a bounded, once-per-opportunity call — **never a model that runs over the whole ladder**. **Invariant 1 (§8): the L column contains at most one entry — a bounded, seeded steering-and-interpretation call whose output the deterministic ladder disposes of.** --- ## 5. Implementation Order Twelve phases, each independently testable and reversible, landing in `build/17-test-execution/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Evidence contract (P1)** | `evidence.py` (§3.1 the `ValidationResult` schema + the four-role enum + `type_evidence`) | Run a real test: a `ValidationResult` is produced with a role; a bare boolean is refused and recorded as a typing failure; assert **no LLM consulted**. | Delete `evidence.py`. | | **2. Baseline replay (P2)** | `replay.py` (§3.2 run a test on B and C; classify the pair into the four cells) | A test failing on B and passing on C → `bug_discriminating = true`; passing on both → regression-only; failing on both → no fix; passing on B failing on C → regression; assert **no LLM consulted**. | Delete `replay.py`. | | **3. Static gates T0–T2** | `gates.py` (§3.3–3.5 syntax, lint, type/build; hard-failure-vs-warning split) | A file that does not parse halts at T0; a lint warning is recorded but passed; a type error returns the compiler diagnostic; assert **no LLM consulted**. | Delete `gates.py`. | | **4. Bug reproduction T3** | `reproduce.py` (§3.6 the reproducer replayed on B and C, classified per §3.2) | A reproducer that passes on B does not certify (regression-only); one that fails on B and passes on C advances; assert **no LLM consulted**. | Delete `reproduce.py`. | | **5. Change-aware selection (P4)** | `select.py` (§3.11 `select_impacted_tests` from the 06 dependency graph + 12 coverage) | A change touching symbol S selects exactly the tests whose dependency reaches S; the set is a superset of what the change could affect; assert **no LLM consulted**. | Delete `select.py`. | | **6. Impacted units + regression T4/T5** | `regress.py` (§3.7–3.8 run the selected unit + regression sets; type a flip as `REGRESSION`) | A patch fixing the defect but breaking a neighbor fails T4 with the neighbor typed `REGRESSION`; assert **no LLM consulted**. | Delete `regress.py`. | | **7. Integration + full suite T6/T7** | `suite.py` (§3.9–3.10 subsystem integration then the reserved full suite) | A patch clearing T0–T5 runs T6; a T7-only failure is recorded as a selection gap; assert **no LLM consulted**. | Delete `suite.py`. | | **8. Flake policy (P7)** | `flake.py` (§3.11 `apply_flake_policy`: cap reruns at N, record signatures, quarantine flippers) | A test that flips across reruns goes to quarantine, never victory; a hang hits the non-termination rule; assert **no LLM consulted**. | Delete `flake.py`. | | **9. Tamper + pinning (P5)** | `integrity.py` (§3.11 `verify_test_integrity` + `pin_environment`: reapply protected files / diff-scope-check; lock deps, seed RNG, no network) | A diff touching a test/fixture/skip marker is caught; the environment refuses network and reuses the seed; assert **no LLM consulted**. | Delete `integrity.py`. | | **10. Suite-strength audit (P6)** | `audit.py` (§3.11 `audit_suite_strength`: mutate critical modules, measure kill rate) | Seeded mutants in a critical module are killed or missed and the rate recorded; a battery killing zero mutants reports so; assert **no LLM consulted**. | Delete `audit.py`. | | **11. L stub (steering + interpretation)** | `steer.py` (§4 ONE call + temperature 0 + seeded + schema-constrained + parse-failure fallback) | With a mocked LLM: the call is **one**, temperature 0; output has exactly the steering fields; an interpretation asserting green over a deterministic failure → returns to the deterministic stage, **no second call**; assert the verdict is unchanged. | Delete `steer.py`. | | **12. Loop driver + A/B harness** | `pipeline.py` (§3 wiring T0→…→T7→(L)), `ab.py` (§7 KEY + companion + secondaries + ablation runner + no-L-vs-L + decision) | `pipeline.py` on a fixture with a mocked stub: T0→T7→steer. `ab.py` runs the §7 pipeline-vs-baseline and ablations on held-out fixtures. | Feature flag `off` → live harness untouched; `ab.py` read-only. | The A/B runner (Phase 12) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The pipeline reuses components 06 (dependency graph), 12 (coverage), and 11 (execution substrate / sandbox) for selection and execution; it invents no new execution substrate and no new diagnosis path. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of the earlier components' dependency graph, coverage, and execution substrate)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `re` | §3.1/§3.4 lint-signature and diagnostic parsing templates | CPython stdlib. | **No** | | stdlib `json` | §3.1/§4 the `ValidationResult` schema and the steering schema | CPython stdlib. | **No** | | stdlib `sqlite3` | §3.1/§3.11 the durable `ValidationResult` store and flake-signature ledger | CPython stdlib. | **No** | | stdlib `ast` | §3.3 parse gate; §3.7 symbol-touched unit computation | CPython stdlib. | **No** | | stdlib `hashlib` | §3.11 tamper-protection file hashes (protected test files) | CPython stdlib. | **No** | | stdlib `subprocess` (sandboxed) | §3.3–3.10 running the battery under the pinned, no-network environment | CPython stdlib; sandbox enforced by §3.11 `pin_environment`. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic tiers; the stores | In-repo, pinned interpreter. | **No** | | Component 06 dependency graph | §3.7/§3.8/§3.11 change-aware selection (`select_impacted_tests`) | Built by spec 06; reused. | **No** | | Component 12 coverage | §3.8/§3.11 regression-set selection | Built by spec 12; reused. | **No** | | Component 11 execution substrate / sandbox | §3.3–3.10 the harness's own pinned execution | Built by spec 11; reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single in-loop steering call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all tiers use stdlib `re`/`json`/`sqlite3`/`ast`/`hashlib`/`subprocess` (with the §3.11 sandbox), and the dependency graph, coverage, and execution substrate are **reused wholesale from components 06, 12, and 11**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The DEFERRED rows (§2 rows 14–17) are a coverage-selection scale-up, an adversarial-strengthening automation, an adaptive budget, a persistent quarantine, or a learned judgment — not new third-party Python installs. Any future trained-model download, fine-tuning run, or learned-judgment dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol Test execution is measured *separately* and *honestly*, because a stage whose whole purpose is to certify or falsify a patch can be defeated in either of two opposite ways: by certifying on green that proves nothing, or by falsifying on red that proves nothing. The instruments keep the battery accountable to what each result actually demonstrated and what it cost, not to the count of green lights. **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, 13, 14, 15, and 16** (Python-only, single-repo, no-Docker), each annotated additionally with the repository's own battery (so FAIL_TO_PASS and PASS_TO_PASS can be labeled independently of the harness's run). Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. Held-out tasks feed the battery, the evidence typing, the B/C replay, and the outcome statistics; in-benchmark tasks feed nothing that scores the stage. **Baseline (concrete).** The **naive "run pytest, boolean" harness** — the failure mode this component rejects: on each candidate patch the harness runs `pytest` once, reads a bare exit code, and reports "tests passed" or "tests failed" with **no evidence typing, no baseline replay, no change-aware selection, no flake policy, no tamper protection, no suite-strength audit**, and a green light is whatever the exit code says. The structured battery must beat *this* on the honest measures below, and the ablation (7.3) removes mechanisms against those same measures. ### 7.1 Evidence-typing coverage + bug-discriminating rate — the KEY measures The primary instruments, the direct score of the whole stage. **Evidence-typing coverage** — of every validation run, what fraction of results carry a typed role (BUG_REPRODUCTION, REGRESSION, EXPECTED_CHANGE, STATIC_GATE) with baseline and candidate results attached, versus the fraction that are bare booleans. A harness that reports "17 tests passed" scores near zero no matter how green it is, because it has not recorded what any of the 17 proved. **Bug-discriminating rate** — of all green results in a run, what fraction were actually bug-discriminating (B FAIL → C PASS) versus regression-only (B PASS → C PASS). A run whose greens are mostly regression-only has not established the bug is fixed, no matter how many tests passed. Both are computed deterministically and outside the model, because the model cannot be trusted to type its own results. **State explicitly: "Never the end-to-end resolve rate alone"** — execution must not be judged by whether the eventual patch resolved (component 14's measure) but by whether each green result actually proved the property it was claimed to prove. **State explicitly: "Never introspective fluency or model self-assessment" — a battery scoring highly on the model's own sense that "the tests passed" while its greens are untyped regression-only noise is the thesis's exact failure and scores near zero here.** **These are the KEY measures — the structured battery is adopted only if it STRICTLY beats the naive "run pytest, boolean" baseline on BOTH AND does not regress cost-ordering efficiency (7.2).** ### 7.2 The companion + secondary instruments Report alongside the KEY measures; the secondaries report, do not gate on: 1. **Cost-ordering efficiency** — the companion: how much validation budget was spent per patch actually falsified — how many patches halted at a cheap tier versus how many burned an expensive tier that a cheaper one would have caught. A harness that spends ten integration minutes to learn what a lint would have shown in 80 milliseconds is paying for information it already had. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). **Tokens** = `tokens / instance`. **Secondary instruments (report, do not gate on):** 1. **Baseline-replay coverage** — what fraction of bug-fix tests received the buggy-baseline contrast at all. A bug-fix test that never demonstrated fail-on-buggy is not bug evidence, and this instrument is the loop that drives that fraction toward one. 2. **Selection precision and safety** — for the regression-selection tier, both precision (how much of the suite it selected) and safety (how often a full-suite failure escaped selection). Safety is the hard floor; precision is the budget. A selection tier that leaks regressions is unsafe; a selection tier that selects everything is precise only trivially and buys no budget. 3. **Flake rate and rerun counts** — per-test flake signatures and the distribution of rerun counts. A rising rerun count is p-hacking in progress; a test that flips across reruns must be in quarantine, and the instrument verifies it is. 4. **Suite strength (mutation kill rate)** — the battery's kill rate on critical modules, the fraction of seeded mutants the suite kills. A falling kill rate is a referee going blind. 5. **Tamper-detection rate** — how often the diff-scope check or protected-file reapplication caught a change to a test, fixture, or skip marker. A zero rate is not proof of honesty; it is a signal to check that the tamper protection is actually installed. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add an execution mechanism because it sounds sensible" mechanically. Two passes, everything else held fixed, the delta measured on the honest measures (evidence-typing coverage + bug-discriminating rate + cost-ordering efficiency + cost/tokens + secondaries): 1. **Ablate evidence typing** — record bare booleans only: the direct test of P1. 2. **Ablate baseline replay** — never run the B/C contrast: the direct test of P2 and the 46.0% warning. 3. **Ablate change-aware selection** — always run the full suite (or nothing): the direct test of P4's budget. 4. **Ablate the flake policy** — unlimited reruns, no quarantine: the direct test of P7. 5. **Ablate tamper protection** — no reapplication, no diff-scope check: the direct test of P5. 6. **Ablate environment pinning** — no locked deps, no seed, network open: the direct test of the environment-drift failure mode. 7. **Ablate the suite-strength audit** — never mutate, never measure kill rate: the direct test of P6. 8. **Ablate the single L steering step** — **no-steering-vs-steering** (deterministic-only vs ask) on the KEY measures: the test of whether the bounded stub (§4) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L stub, which must survive the no-steering-vs-steering ablation to be kept), then to any other mechanism. **Never add an execution mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run battery-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured battery **wins iff**: ``` Evidence-typing coverage STRICTLY beats the naive "run pytest, boolean" baseline (KEY) AND bug-discriminating rate STRICTLY beats the naive "run pytest, boolean" baseline (KEY) AND cost-ordering efficiency STRICTLY beats the naive "run pytest, boolean" baseline (companion) AND flake rate not regressed (≤ baseline; rising reruns = a defect to fix) AND tamper-detection rate not regressed (a zero rate signals the protection is not installed) AND cost_battery ≤ 1.5 × cost_baseline AND tokens_battery ≤ 1.5 × tokens_baseline ``` Both KEY measures must **strictly beat the naive "run pytest, boolean" baseline**; the companion must **strictly beat** it; flake and tamper must not regress; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — resolve is component 14's measure, and a battery that improves resolve while leaving evidence typing and bug-discriminating rate flat against the boolean baseline is **not adopted**. **Never introspective fluency or model self-assessment** — a battery that reports the model's confidence that "tests passed" while its greens are untyped regression-only noise scores *worse* and is rejected. If the battery wins it replaces the naive boolean path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways a certification battery can lie) 1. **Empty-green** — exit 0 with zero tests collected: success declared by an empty referee. Caught by the empty-patch self-test run, which must prove the referee cannot be scored on without playing. 2. **Rerun-until-green** — flakiness converted into policy, the loop optimizing for dice. Caught by rising rerun counts and the quarantine rule. 3. **Tampering** — weakened assertions, skipped tests, or mocks replacing the system under test. Caught by the diff-scope check and protected-file reapplication. 4. **Environment drift** — green in the sandbox, red in CI. Caught by the pinned environment and the no-network, seeded, locked-dependency discipline. 5. **Target-only certification** — the failing test flips while its neighbors regress (PASS_TO_PASS shipped broken). Caught by the PASS_TO_PASS regression set and the bug-discriminating rate. 6. **Vacuous green** — a patch no test actually executed: the diff's coverage is zero and the green certifies nothing. Caught by the coverage-typing instrument and the affected-by-change flag. 7. **Suite-blindness** — contaminated green: tests visible in training data or issue text certify nothing. Caught by the suite-strength audit and the held-out reporting discipline. ### 7.6 Two honest caveats **First, the strong-model caveat:** the paper's headline numbers — HyRTS's 18.35%-vs-27.18% selection and 31.60s-vs- 40.06s times, Agentless's 77-to-96 ablation, SWE-ABS's 19.71% rejection, TestPrune's 9.4%–12.9% relative gain — will **NOT** reproduce on a weak model, and were measured on repositories and models far larger than a weak-model harness controls; a weak-model harness should expect materially smaller absolute effects while leaning all the harder on the cheap tiers, the typed evidence, and the smallest-safe-set discipline to make its small budget count. **Second, the held-out caution:** all execution metrics must be reported on held-out and out-of-benchmark tasks, so a battery that has memorized a task set's answers cannot score well on suite strength without its verdicts having been measured on problems it never saw. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/17-test-execution/`; it never writes into the live harness's source. - A **feature flag** (env var `TESTEXEC=17battery|pytestbool`, default `pytestbool`) routes the A/B; the flag alone switches between the structured battery and the naive "run pytest, boolean" baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running old-vs-new is always possible in the same session, repo, model. - The durable `ValidationResult` store and the flake-signature ledger are append-only and versioned; rolling the build back never loses them. A result falsified by later evidence is kept with its role and its B/C pair — never silently dropped and never silently re-typed by a model. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains at most one entry** — in-loop steering and failure interpretation (§4), the one bounded, seeded call choosing what to probe next and reading what a failure means, reached only inside a repair loop, never re-prompted on parse failure. Any phase introducing a model elsewhere (typing a result, classifying a B/C pair, selecting the regression set, capping reruns, reapplying test files, pinning the environment, auditing the suite, authoring the verdict, rerunning until green, waiving a failing tier, or judging whether green counts) is rejected outright. An **empty** L column (the stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **A bug-fix test is not bug evidence until it shows fail-on-buggy AND pass-on-candidate** — the B/C pair is classified deterministically (§3.2), and a reproducer that cannot demonstrate fail-on-buggy is not bug evidence until it does (P2). 3. **The harness owns the verdict** — the model never authors the verdict, never decides whether a green result counts, never reruns a test until it passes, and never waives a failing tier; its interpretation is a downstream hypothesis, not a result (P5). 4. **Test files are tamper-protected** — protected test files are reapplied or diff-scope-checked so no test, fixture, or skip/xfail/mock-of-SUT marker changed; deleting a flaky test is not a fix (P5). 5. **The flake policy is capped** — at most N reruns per test with recorded signatures; a test that flips across reruns goes to quarantine, never victory (P7). 6. **Suite strength is a measured instrument** — the battery's mutation kill rate is audited periodically, and a battery that does not kill its mutants is reported as such, never trusted as a referee (P6). 7. **Selection is safe first, small second** — the selected regression set is a superset of what the change could affect, and a regression that escapes T5 to T7 is a selection-safety defect, never excused (P4). 8. **Evidence typing is complete** — every result carries a role with baseline/candidate contrast where affordable; a bare boolean is a typing failure, never a green light (P1). 9. **All metrics on held-out/out-of-benchmark tasks** — the KEY measures and every secondary are reported on held-out tasks, never tuning or in-benchmark data (§7). --- *End of spec. This document is the build contract; the next document records the A/B outcome.*