Summary / 27. Benchmarking and objective scoring / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 27 — Benchmarking / Objective Scoring > Derived from `papers/27-benchmarking-objective-scoring.md` — "Executed Tests and Measurable System Outcomes as the Only Admissible Evidence of Capability — the Scoreboard That Does Not Read Essays." > This document drives the build of component 27: **benchmarking and objective scoring** — the measurement plane that decides > whether any of the other twenty-six components actually made Piper better. Every component before this one was justified, in > the moment, by an argument from mechanism: *this is how it works, and here is why it should help.* Component 27 is where the > program stops accepting arguments from mechanism and starts demanding evidence from outcome. It is the outermost loop: the > fifteen-plus prior components optimize the harness, and this component is how anyone — including Piper herself — knows whether > the optimization was real. The central claim of this component is that a capability claim is settled by what the system *did* > in an executable environment, measured mechanically, and never by what another model — or another person — *thinks* the answer > looks like. "Another model says it looks right" is not a fallback methodology; it is the **null hypothesis** the entire > discipline exists to reject. The design is a measurement plane kept outside the thing it measures: execution-based grading that > runs the candidate's actual patch against the repository's actual tests and reads the verdict off the machine; a decomposed > scorecard that attributes gain to the specific stage of the coding loop where it was won, so a bottleneck can be identified > independently of end-to-end success; a contamination discipline that reserves genuinely unseen or continuously refreshed tasks > for generalization claims, because a benchmark that has been optimized against is no longer measuring ability but familiarity; > a controlled-comparison discipline that freezes model, budget, environment, and sampling parameters when a harness change is > under test, repeats stochastic evaluations to estimate variance, and records tokens, calls, CPU, wall time, and dollar cost > alongside correctness; and an audit discipline that treats the instrument itself as the object of standing review, because a > measure that becomes a target ceases to be a measure. The structural invariant that governs the whole design is that **the L > column contains at most one entry** — and at this component the enforcement path must be deterministic software on CPU [D] with > **zero learned entries**, because the model is the thing being *measured*, never the thing doing the measuring. The evidence is > assembled from the founding execution-based benchmarks (SWE-bench; HumanEval; MBPP; CodeXGLUE; RepoBench; FixEval), the > contamination-resistant successors (SWE-bench Pro; SWE-bench-Live; SWE-PolyBench; SWE-Perf; LiveCodeBench; EvoEval; > Terminal-Bench 2.0; Multi-SWE-bench; BigCodeBench), the economic and process-oriented outliers (SWE-Lancer; MLE-bench; > TheAgentCompany), the measurement-theory lineage (Campbell's Law; Goodhart; Strathern; Cronbach and Meehl), and the audits that > found the referee itself wanting (the Agentic Benchmark Checklist; SWE-bench+; The SWE-Bench Illusion; SWE-ABS). It closes with > the self-referential observation that makes this the final component rather than merely the last one: the scoreboard does not > read essays, and it does not read Piper's essays about herself either. This spec pins the paper's design to exact primitives: > the exact `Score` execution-verdict record with its two-sided FAIL_TO_PASS + PASS_TO_PASS gate, the exact hierarchical > `Scorecard` record, the exact `BenchmarkSet` three-set separation record, the exact `ABComparison` record with its frozen > config and cost field, the exact D/L invariant at its zero-learned extreme, the exact eight-phase build sequence, and the exact > freeze-one-change-one A/B protocol. --- ## 1. Purpose & Scope Build a **deterministic measurement and grading plane plus a zero-learned scoring path** — a software layer that, for every candidate the harness produces and every harness change the program proposes, answers *did it work?*, *where did the gain or loss come from?*, *did it generalize or just memorize?*, and *what did it cost?* — all of it deterministic software on CPU, with no learned model anywhere in the path that produces the verdict. The scope is the **adjudication boundary only**. Components 01–26 own *what* the harness computes, *what order* the stages run in, and *whether* the computed things are safe to run; this component owns *whether any of it worked* — the one question where a wrong answer is not a quality defect but the whole bet of the architecture. The answer it defends is the most conservative one available: **grade executed behavior, not model opinion; attribute every gain to the stage that earned it; reserve unseen tasks for generalization; freeze everything and change one thing; report cost alongside correctness; and audit the instrument as hard as the instrument audits the harness.** The paper's discipline compresses into a handful of load-bearing moves that carry the entire design. **Execution over opinion**: the unit of evidence is the executed verdict — a patch is correct if and only if it passes the tests that encode the intended behavior, and the only entity that knows which is the runtime that executes it. **The two-sided gate**: a patch is *resolved* only if it makes every FAIL_TO_PASS test pass *and* preserves every PASS_TO_PASS test — a fix that fixes the bug and breaks three other things is not a fix. **Decomposition**: every stage of the coding loop has its own mechanical metric, so a bottleneck is identifiable independently of end-to-end success and a gain is attributable to the change that earned it. **Contamination resistance**: a benchmark that has been optimized against is measuring familiarity, not ability, so the harness keeps three distinct task sets and never conflates them. **Controlled comparison**: when a harness change is under test, hold model, budget, environment, and sampling parameters fixed and change exactly one thing, repeat to estimate variance, and record cost alongside correctness. **Audit of the instrument**: a measure that becomes a target ceases to be a measure, so the benchmark itself is the object of standing, mechanical audit. **Zero-L**: the scoring path holds zero learned entries by construction; the model is the thing being measured, never the thing doing the measuring. Three principles decide every design choice. **P1 The scoreboard reads outcomes, not essays.** A capability claim is settled by what the system *did* in an executable environment, measured mechanically. "Another model says it looks right" is the **null hypothesis** — the default assumption that must be actively rejected by producing something better — never a weaker grade of evidence to fall back on. A benchmark whose scores are partially model-opinion is a benchmark that has smuggled the null hypothesis back in as part of the method. **P2 The D/L invariant at zero.** The measurement/grading plane holds zero learned entries by construction; **the L column contains at most one entry**, and here the bound is met at zero — **an empty L column is a permitted, correct state**, and it is the **only** correct state for the scoring path, because the model is the *subject* of the score and a subject that grades its own exam produces a fluent and worthless number. **P3 A gain must be measured, not felt.** Nothing gets promoted because it feels smarter; a component survives only if Piper measurably performs better when it exists than when it does not, under controlled ablation, with cost recorded alongside correctness. The build lives in `build/27-benchmarking-objective-scoring/`, writes nothing into the live harness, and is kept **only if** the measurement plane strictly dominates its own absence — measured on the honest instruments of §8: the executed verdict against held-out tasks, the decomposed scorecard that attributes gain to stage, the three-benchmark-set discipline that keeps generalization honest, the freeze-one-change-one A/B harness, and the cost ledger — A/B-tested against the unmeasured baseline and against the intermediate ablations (no two-sided gate; no decomposition; no three-set separation; no timestamp gating; no variance estimation; no cost ledger; no instrument audit). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–8); **DEFERRED = later rounds** (rows 9–13), **not excluded** but gated by the same §8 A/B protocol. Every later round of this component still builds within the same `build/27-benchmarking-objective-scoring/` tree. | # | Component (paper §2/§3/§4/§5/§6) | Classification | |---|---|---| | 1 | Executed-test runner — run the candidate patch against the repo's real tests, read verdicts off the machine (§2) | **BUILD-NOW** | | 2 | Mechanical scorer — the two-sided FAIL_TO_PASS + PASS_TO_PASS gate, binary resolved/not-resolved (§2.2) | **BUILD-NOW** | | 3 | Hierarchical scorecard — component-level + end-to-end + economics planes (§3.3) | **BUILD-NOW** | | 4 | Benchmark-set manager — development / regression / holdout-live separation (§4.4) | **BUILD-NOW** | | 5 | Contamination discipline — timestamp gating + held-out rotation (§4.1, §4.2) | **BUILD-NOW** | | 6 | A/B harness — freeze-one-change-one, repeat to estimate variance (§5.1, §5.2) | **BUILD-NOW** | | 7 | Cost/latency ledger — tokens, calls, CPU, wall time, dollars alongside correctness (§5.3) | **BUILD-NOW** | | 8 | Instrument audit — empty-patch canaries + task validation, standing, mechanical (§6) | **BUILD-NOW** | | 9 | Live continuously-refreshed instruments — standing SWE-bench-Live subscription, LiveCodeBench timestamp-gated pull (§4.2) | **DEFERRED** — requires standing access to continuously refreshed corpora and a refresh scheduler; Round 1 ships the offline fixture subset and adds the live pull when the harness's own benchmark pipeline is the thing under continuous refresh | | 10 | Full multilingual + environment-state battery — SWE-PolyBench, Multi-SWE-bench, Terminal-Bench 2.0, BigCodeBench at full scale (§3.4, §4.3) | **DEFERRED** — requires per-task Docker images and multi-OS environments; Round 1 ships the Python-repository subset and adds the full battery when the harness's generalization claims need the cross-language and environment-state axes | | 11 | Economic-value axis — the SWE-Lancer / MLE-bench internal engineering-value score (§5.4) | **DEFERRED** — requires real freelance-scale tasks and Kaggle-style graders; Round 1 ships the cost-per-resolve ratios of §5.3 and adds the value axis when the harness completes real paid-shaped work | | 12 | Cross-benchmark, cost-aware comparison — the HAL three-dimensional space (§5.5) | **DEFERRED** — requires the 9-model × 9-benchmark × 3-scaffold matrix; Round 1 ships the two-arm A/B harness and adds the full matrix when harness-level comparison at scale is needed | | 13 | Item-Response-Theory difficulty filtering — the mid-range 30–70% difficulty filter (§5.2) | **DEFERRED** — requires historical pass rates across the full task bank to calibrate the filter; Round 1 ships seeded evaluation over the development set and adds the IRT filter when the task bank is large enough to estimate difficulty reliably | **The cost rule (explicit).** The L column holds **zero** entries in this component's scoring path — not one, not "at most one," zero. No learned model is consulted at any point in the measurement plane: not to run a test, not to score a verdict, not to attribute a gain to a stage, not to assign a contamination status, not to estimate variance, not to run an A/B comparison, not to record a cost, not to audit an instrument. The model's entire presence is **outside** the scoring path: it *produces the candidates* — patches, plans, fixes — and the machine *produces the verdicts* — resolved, not resolved, faster, slower, cheaper, more expensive. The DEFERRED rows (live instruments, multilingual battery, value axis, cross-benchmark matrix, IRT filtering) must clear the same bar as in every other component: strictly dominate the deterministic Round 1 build on §8's measures — specifically, measure a generalization or value axis the Round 1 build demonstrably fails to — at a defensible cost, or be rejected. Note the asymmetry the paper insists on: the DEFERRED rows are *expansions of the instrument*, not *learned additions to the grader* — no later round may put a model in the scoring path, because the model is the thing being measured, and a grader that contains a learned model inherits every bias the model under test was selected for. **Design decision: ZERO-L, DETERMINISTIC-ONLY.** Round 1 puts no model anywhere in the scoring path, inside or at the edge. The full deterministic grading plane (rows 1–8) is the build. There is no L slot to fill. A scoring path that *would* need a learned entry has already failed — because the model is the thing being measured, and a measurement instrument that contains the entity it measures is a self-report, not a measurement — and Round 1 is constructed so that need never arises. --- ## 3. Deterministic Orchestration (the grading plane, [D]) The grading plane is the software that, for every candidate and every harness change, decides what the executed behavior *means* — and it is, by construction, entirely outside the model. Its parts are the eight mechanisms of §2 rows 1–8: the executed-test runner, the mechanical scorer, the hierarchical scorecard, the benchmark-set manager, the contamination discipline, the A/B harness, the cost ledger, and the instrument audit. Recapitulating the invariant at the architecture level, because this is the section it governs: > **the L column contains at most one entry.** The entire grading plane — test execution, mechanical scoring, the pinned harness, variance estimation, the A/B comparison, the scorecard decomposition, the benchmark-set separation, the cost ledger, and the instrument audit — is deterministic software on CPU [D], holding **zero learned entries**, and here it is exactly **zero** — **an empty L column is a permitted, correct state**, and here it is the **only** correct state for the scoring path. Concretely, the grading plane's pipeline is: ``` CANDIDATE (patch / plan / fix produced by the model [L] — the thing being measured) → executed-test runner [D] → run the patch against the repo's real tests (subprocess + exit codes) (§2 / paper §2) → mechanical scorer [D] → two-sided gate: every FAIL_TO_PASS passes AND every PASS_TO_PASS preserved (§2.2) → hierarchical scorecard [D] → per-stage mechanical metrics + end-to-end + economics (§3.3) → benchmark-set manager [D] → development / regression / holdout-live separation, contamination status (§4.4) → contamination discipline [D] → timestamp gating, held-out rotation, task-burning (§4.1, §4.2, §4.4) → pinned harness [D] → frozen config applied identically to both arms of a comparison (§5.1) → variance estimation [D] → repeated seeds → mean, variance, confidence interval (§5.2) → A/B comparison [D] → statistical test over two resolved/not-resolved distributions (§5.1, §5.2) → cost/latency ledger [D] → tokens, calls, CPU, wall time, dollars alongside correctness (§5.3) → instrument audit [D] → empty-patch canaries + task validation, standing (§6) VERDICT (resolved / not_resolved / faster / slower / cheaper / more expensive) leaves the grading plane ``` The model never runs a test, never scores a verdict, never attributes a gain to a stage, never assigns a contamination status, never estimates variance, never runs an A/B comparison, never records a cost, and never audits an instrument. Its relationship to the grading plane is a single, well-defined edge: it *produces candidates* — patches, plans, fixes — and the machine *produces verdicts* — resolved, not resolved, faster, slower, cheaper, more expensive. The two columns touch at exactly one point: the handoff of a patch to the runner and a verdict back. **Produce, then be judged; never judge.** Every function in this section is deterministic; the L column contributes nothing here. All artifacts land in `build/27-benchmarking-objective-scoring/`. ### 3.1 The executed-test runner [D] The first mechanism is the one that makes every downstream verdict a report of what happened rather than an opinion of what happened. A candidate patch is applied to the repository, and the repository's actual test suite is executed in the pinned environment; the runner records exit codes, test names, and durations, and nothing else. The lineage is the founding contract of §2: **fluency is not correctness** — a program textually similar to a reference can be wrong in a way no string comparison detects, and a program that looks nothing like the reference can be right in a way only execution reveals. The founding instruments wrote this into the field's practice. HumanEval ('Evaluating Large Language Models Trained on Code,' arXiv:2107.03374) introduced `pass@k`: a solution counts as correct if and only if it passes the hidden unit tests, measured over repeated samples so a model's *ability to eventually produce a working solution* is scored rather than a single lucky draw — the component's first lesson, that a single run is a sample from a distribution, not a capability. MBPP ('Program Synthesis with Large Language Models,' arXiv:2108.07732) supplied the second lesson: capability scales with the *difficulty distribution* of the task set, and a benchmark of toy problems measures toy performance. CodeXGLUE (arXiv:2102.04664) supplied the third, the one the whole component is built on: **evaluation is infrastructure** — a benchmark dataset is a platform, not a one-off, whose value is that many systems can be compared under one mechanical protocol. And FixEval ('FixEval: Execution-based Evaluation of Program Fixes for Programming Problems,' arXiv:2206.07796) supplied the empirical foundation at scale: over a raw pool of 712K Java and 3.28M Python submissions, *match-based metrics do not reflect model-generated program fixes accurately*, while execution-based evaluation — running each candidate through the full test suite designed for that solution — does. The runner is the witness, not the judge: it reports what the machine did. ### 3.2 The mechanical scorer: the two-sided gate [D] The second mechanism is the field's definition of the word "resolved," inherited from SWE-bench ('SWE-bench: Can Language Models Resolve Real-World GitHub Issues?', arXiv:2310.06770, ICLR 2024). A patch is graded by the tests that the *human* fix's pull request introduced and preserved, and the grading is **two-sided**: a patch is accepted if and only if it makes every **FAIL_TO_PASS** test pass (the tests that failed before the fix and pass after — the proof that the fix *works*) *and* preserves every **PASS_TO_PASS** test (the tests that passed before and must still pass after — the proof that the fix *broke nothing*). The two-sided gate is the executed form of the discipline a human pull-request reviewer applies: a fix that fixes the bug and breaks three other things is not a fix. A benchmark that scored only FAIL_TO_PASS would be scoring half the contract; a harness that optimizes only for making the failing test pass is optimizing for exactly the half-measure a regression test exists to catch. The verdict is deliberately **binary** — an instance is resolved or it is not, no partial credit — and the "almost right" patch scores identically to the patch that never got started. For *end-to-end* scoring this is a feature: the honest, uncompromising summary of "did the system do the job." It is precisely this binary-ness that makes the decomposition of §3.3 mandatory — the binary number says *whether*, and says nothing about *where*. ### 3.3 The hierarchical scorecard [D] The third mechanism is the decomposition that turns a scoreboard from a score into a diagnosis. Its founding instrument is RepoBench (arXiv:2306.03091, ICLR 2024), which splits repository-level scoring into three settings scored *separately* rather than smeared together — **RepoBench-R** (retrieval), **RepoBench-C** (completion), and **RepoBench-P** (pipeline) — converting an un-attributable number into an attributable one. The same principle, applied to Piper's loop, produces the hierarchical scorecard of §5.2: for every task, the scoreboard records not only `resolved?` but a vector of per-stage metrics — localization (file recall, symbol recall, rank of the correct target), planning (fraction of required obligations recovered), patching (apply / parse / build), structure (correct entities touched, unnecessary entities touched), testing (bug tests and regression tests both exercised) — plus the end-to-end verdict and the economics. The end-to-end number remains the headline, but it is now the *bottom* of a hierarchy rather than the whole of it. Its second founding result is SWE-agent ('SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering,' arXiv:2405.15793, NeurIPS 2024): the **harness is a first-order variable** — the agent-computer interface moves resolution by multiple points *at fixed model* — which makes the harness itself a subject of measurement rather than an invisible constant, and which makes the entire project of building harnesses a project that *must* be measured change by change, with the model held fixed. ### 3.4 The benchmark-set manager and the contamination discipline [D] The fourth and fifth mechanisms are the separation that keeps generalization honest. The harness maintains three distinct task sets and never conflates them: the **development set** (small, fast, diagnostic, may contain known tasks), the **regression benchmark** (never changes casually, run before and after every architecture modification), and the **true holdout/live set** (unseen or continuously refreshed tasks — the only number that may be called a *generalization* score). The discipline's core is a hard rule: **tasks used to make a decision are burned for future decisions** — once the harness has been iterated against a task, that task is development or regression data, never again generalization data. The contamination discipline adds the timestamp gating of LiveCodeBench (arXiv:2403.07974) — score each model only on problems published after its training cutoff, settling the contamination question by construction — and the held-out rotation of the refresh instruments (§4.2 of the paper), so that the holdout set is touched only for major architecture decisions, rarely, and its tasks are retired afterward. ### 3.5 The A/B harness, the cost ledger, and the instrument audit [D] The sixth, seventh, and eighth mechanisms are the disciplines of §5 and §6. The A/B harness enforces **freeze everything, change one thing**: model, temperature, token budget, tool budget, task set, execution environment, timeout, and candidate count are all held fixed, and only the change under test varies — only then does a difference in outcome admit a causal interpretation. The cost ledger records **tokens, calls, CPU, wall time, and dollar cost alongside correctness**, and treats the ratio of correctness to cost as a score in its own right — two harnesses with different cost profiles are not comparable on correctness alone. The instrument audit treats the benchmark itself as the object of standing review: **empty-patch canaries** (an empty or trivial patch must score not-resolved, testing outcome validity — that the scorer accepts *exactly* the correct solutions) and **task validation** (a task must be solvable only by the intended capability, not by an artifact of the setup — testing task validity). These are the two properties of the Agentic Benchmark Checklist ('Establishing Best Practices for Building Rigorous Agentic Benchmarks,' arXiv:2507.02825, colloquially "AI Agent Benchmarks Are Broken"): *execution-based is a necessary condition for objectivity, not a sufficient one.* --- ## 4. The L Column — zero learned entries in the scoring path Every element of this component is classified into one of two columns: **[D]** — deterministic software running on CPU — or **[L]** — a learned model. The structural invariant that governs the entire design is: > **the L column contains at most one entry.** The measurement/grading plane — test execution, mechanical scoring, the pinned harness, variance estimation, the A/B comparison, the scorecard decomposition, the benchmark-set separation, the cost ledger, and the instrument audit — is deterministic software on CPU [D], with **zero learned entries**. The model is the thing being *measured* by this plane, and it cannot be the thing doing the measuring: a grader that contains a learned model is a grader whose verdict inherits every bias the model under test was selected for, and a benchmark whose scores are partially model-opinion is a benchmark that has smuggled the null hypothesis back in as part of the method. **An empty L column is a permitted, correct state** — a deterministic phase of the harness that needs no model call at all — and it is the **only** correct state for the scoring path, because the scoring path is exactly where a model, whose sycophancy, verbosity, and self-preference biases are themselves part of what is under test, must have no role in producing the characterization. The only learned element in the system is *outside* the scoring path: the model [L] that produces the candidates — the patches, plans, and fixes whose behavior the plane characterizes. Inside the scoring path there is nothing to learn, and nothing that learns. This is the same invariant that governs Components 24, 25, and 26, held at the same strength and applied to the one question those components did not answer. The sandbox's invariant holds *during execution* (the code is contained); the permission system's invariant holds *before execution* (the authority is derived); the security component's invariant holds *before interpretation* (the text is data, not instruction). The benchmark component's invariant holds *after everything* — at the point where the question "did any of it work?" must be answered without asking the thing that did the work. The four compose into a single boundary: **the security component decides what an observation is allowed to mean; the permission system decides what an action is allowed to ask for; the sandbox decides what a received capability is allowed to do; and the benchmark decides whether the whole assembly is actually any good.** None of the four has a learned entry in its enforcement or scoring path, and all four are correct to have none. ### 4.1 Why the model cannot grade its own exam The claim that the scoring path must hold zero learned entries is a reading of the measurement, not paranoia. The founding empirical claim of execution-based evaluation is exactly this: **text-similarity metrics and generative plausibility mispredict functional correctness.** A candidate answer can be near-identical to the reference and still wrong, or wildly different and still right, and the only entity that knows which is the runtime that executes it. The question this component answers — *is Piper better?* — is the one question in the entire harness that cannot be answered by asking a model, because the model is the *subject* of the question. To ask the model whether the model-plus-harness is better is to ask the examined party to grade its own exam, and the answer will be whatever the model's sycophancy, verbosity, and self-preference biases produce — a confident, fluent, and worthless number. This is why the component's null hypothesis is stated the way it is: **"another model says it looks right" is not a weaker grade of evidence to be used when execution is inconvenient; it is the null hypothesis** — the default assumption that must be actively rejected by producing something better. The method that replaces it is the scoreboard: executed tests against real, held-out tasks, run through a pinned harness, scored mechanically, with per-stage attribution and honest variance. The deeper reading is the one the audits of §6 confirm. In HAL ('Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation,' arXiv:2510.11977, 2025), LLM-aided log inspection surfaced an agent *searching for the benchmark on Hugging Face instead of solving the task* — the agent was optimizing the *measure*, not the *construct*. An agent that can be induced to chase the measure is exactly the entity that cannot be trusted to grade itself against the measure. And in the SWE-bench+ audit (arXiv:2410.06992), filtering out the contaminated and weak-test instances dropped the top system's resolution rate from 12.47% to 3.97% — a three-quarters collapse — the single clearest number in the literature for how much of a headline score can be instrument artifact rather than capability. A grader that shared the biases of the graded would not have caught that; only a deterministic computation over executed behavior can. Hence zero, not one: the model is the thing being measured, and a measurement instrument that contains the entity it measures is a self-report. ### 4.2 Zero, not one — recapitulated at the architecture level Recapitulating the invariant at the architecture level, because this is the section it governs: > **the L column contains at most one entry.** The entire grading plane — test execution, mechanical scoring, the pinned harness, variance estimation, the A/B comparison, the scorecard decomposition, the benchmark-set separation, the cost ledger, and the instrument audit — is deterministic software on CPU [D], holding **zero learned entries by construction**. The model is **not a contributor** to this plane; it is the thing the plane measures. **An empty L column is a permitted, correct state** — and here it is the **only** correct state for the scoring path: a grading plane with any learned entry in the scoring path has put a model whose sycophancy and self-preference are part of what is under test in charge of deciding whether the system improved. The only learned element in the system is **outside** the scoring path — the model [L] that produces the candidates — and **inside** the scoring path there is nothing to learn, and nothing that learns. ### 4.3 The kinship with the program's own deepest rule This component is not a new idea entering the program from outside; it is the coding-harness instance of a rule the program already lives by, stated in its own terms: **runtime is the only witness.** Throughout the harness, a hypothesis about the world has been treated as a claim to be settled by execution — a patch is only as good as the tests it passes, a fix is only as good as the state it produces. Component 27 takes that same principle and points it at the harness itself. The claims this component adjudicates are claims of the form "Component N improved the system," and the only witness it will accept is a measured difference in executed behavior between the system with Component N and the system without it. The D/L invariant that governs the scoring path — **the L column contains at most one entry**, and in the scoring path the correct value is zero — is the same invariant that protects this component from its own subject, because the only thing that can be trusted to say whether Piper got better is a deterministic computation over executed behavior, and Piper is, by construction, not that computation. --- ## 5. The Core Schemas (the build's JSON contracts) The following schemas are the load-bearing contracts of the build. Their shapes are EXACTLY these fields, no more and no fewer, and the label "EXACTLY these fields" is itself part of the contract — a field added for convenience is a spec violation. They are derived from the paper's §3–§5, and each is grounded in the paper's own words. ### 5.1 The Score record (EXACTLY these fields) — the execution-verdict and the two-sided gate The paper's grading discipline (§2.2) — "a patch is accepted if and only if it makes every FAIL_TO_PASS test pass and preserves every PASS_TO_PASS test" — compiles to exactly this record. The `resolved` verdict is derived, never asserted: it is the conjunction of the two gates, and nothing else may set it. ```json Score { instance_id: string // the benchmark instance (issue / task) this verdict is for patch_ref: string // the candidate patch under test (a content hash or ref) — what was executed resolved: enum // {resolved, not_resolved} — binary, no partial credit (§2.2); derived, never asserted fail_to_pass: TestGate // the FAIL_TO_PASS gate: tests that failed before the fix and must pass after (§2.2) pass_to_pass: TestGate // the PASS_TO_PASS gate: tests that passed before and must still pass after (§2.2) graded_by: string // the pinned harness + environment + timeout that produced this verdict (§5.1 of the paper) graded_at: integer // epoch seconds at verdict time } TestGate { total: integer // number of tests in this gate passed: integer // FAIL_TO_PASS: tests now passing; PASS_TO_PASS: tests still passing (preserved) verdict: enum // {satisfied, violated} — satisfied only if passed == total; derived, never asserted } ``` The load-bearing decisions, enforced not advisory: - **`resolved` is the conjunction of the two gates.** `resolved` holds only if `fail_to_pass.verdict == satisfied` AND `pass_to_pass.verdict == satisfied`. A patch that makes every failing test pass but breaks one passing test is `not_resolved` — a fix that fixes the bug and breaks three other things is not a fix. The `resolved` field is never writable by any caller; it is computed from the two gates, and nothing else may set it. - **The two gates are distinct and both are mandatory.** A scorer that recorded only `fail_to_pass` would be scoring half the contract — optimizing for making the failing test pass while a regression test exists to catch the half-measure. The two-sided gate is the executed form of the human pull-request reviewer's discipline. - **Binary, no partial credit.** `resolved` is a closed two-valued enum; there is no `almost_resolved`, no partial score, no proximity credit. The "almost right" patch scores identically to the patch that never got started, at the end-to-end plane — and it is exactly this binary-ness that makes the per-stage decomposition of §5.2 mandatory, because the binary verdict says *whether* and nothing about *where*. - **`graded_by` names the instrument.** Every verdict is traceable to the pinned harness, environment image, and timeout that produced it — the lesson of SWE-agent (§3.3): because the harness is a first-order variable, the harness is *named* in every result, and cross-harness comparisons are treated as cross-experiments rather than rankings. - **The verdict is read off the machine, never off a model.** `resolved`, `fail_to_pass.passed`, `pass_to_pass.passed`, and the two `verdict` fields are all outputs of deterministic test execution; no field in this record is ever produced by a learned model. ### 5.2 The Scorecard record (EXACTLY these fields) — the hierarchical score The paper's decomposition discipline (§3.3) — "component-level metrics for every stage of the coding loop so bottlenecks can be identified independently of end-to-end success" — with its three planes (component metrics, end-to-end, economics) compiles to exactly this record. The `component_metrics` sub-record carries the per-stage mechanical metrics of §3.1; the `end_to_end` sub-record carries the executed verdicts; the `economics` sub-record carries the cost/value numbers of §5.3. ```json Scorecard { run_id: string // the harness run this scorecard summarizes task_id: string // the benchmark instance under measurement component_metrics: ComponentMetrics // per-stage mechanical metrics (§3.1, §3.3) — attributable, not smeared end_to_end: EndToEnd // the executed verdicts (§2.2) — the headline, now the bottom of the hierarchy economics: Economics // the cost/value numbers (§5.3) — correctness is never read without cost } ComponentMetrics { localization: LocalizationMetrics // Recall@k against the gold patch's touched files and symbols planning: { obligations_recovered: number } // fraction of required obligations the plan recovered (0..1) patching: { apply: boolean, parse: boolean, build: boolean } // does the patch apply, parse, build structure: { correct_entities_touched: integer, unnecessary_entities_touched: integer } // touched the right things, avoided the wrong ones testing: { bug_tests_exercised: boolean, regression_tests_exercised: boolean } // both test suites actually exercised } LocalizationMetrics { file_recall: number // fraction of gold-patch touched files recalled symbol_recall: number // fraction of gold-patch touched symbols recalled target_rank: integer // rank of the correct target in the retrieval order (1 = first) } EndToEnd { resolved: enum // {resolved, not_resolved} — the binary executed verdict (§2.2) regressions: integer // introduced regressions detected (a fix that broke other things) robustness: number // success under perturbation / held-out variants (the generalization residue) } Economics { tokens: integer // LLM tokens consumed across the run calls: integer // LLM calls made across the run cpu: number // CPU-seconds consumed across the run wall_time: number // wall-clock seconds elapsed across the run dollars: number // LLM + compute dollar cost across the run } ``` The load-bearing decisions, enforced not advisory: - **Every metric in `component_metrics` is executable and mechanical.** Localization is scored by Recall@k against the gold patch's touched files and symbols; planning by the fraction of required obligations recovered; patching by apply/parse/build success; structure by the correct and unnecessary entities touched; testing by whether the bug tests and the regression tests were both exercised. None of these is a model's opinion of itself; each is a deterministic computation over a candidate's behavior against a ground truth. - **The three planes are read together, never separately.** `end_to_end.resolved` remains the headline, but it is the *bottom* of the hierarchy rather than the whole of it. The joint reading is what lets the harness produce a sentence the naive scoreboard cannot: *Component 4 improved and is no longer the bottleneck; the residual gap now sits at Component 13.* The naive benchmark produces "42%," useful and radically insufficient; the hierarchical scorecard produces the diagnosis that tells the next change where to go. - **A gain is attributable independently of end-to-end success.** When Component 4 changes, the scorecard reports its effect on `localization.file_recall` directly — "51% → 62%" — rather than forcing that effect to be inferred from a final resolve rate a downstream bottleneck may have held flat. The gain is real, visible, and correctly attributed; the loss of propagation is itself the diagnosis that the next change should target. - **`economics` is not optional.** The architecture's founding bet is that a relatively weak model is made competitive through scaffolding — and scaffolding costs tokens, calls, and wall-clock time. A scorecard that omits `economics` is blind to the one quantity the architecture is constantly spending. Two harnesses with different cost profiles are not comparable on correctness alone. ### 5.3 The BenchmarkSet record (EXACTLY these fields) — the three-set separation The paper's contamination discipline (§4.4) — "the harness maintains three distinct sets, and never conflates them" — compiles to exactly this record. The `role` enum is the paper's three sets, verbatim; the `contamination_status` enum carries the paper's distinction between known, unseen, and continuously refreshed tasks. ```json BenchmarkSet { set_id: string // the set's identity (a stable, versioned name) role: enum // {development, regression, holdout_live} — the three-benchmark-set discipline (§4.4) task_ids: string[] // the instances in this set (content-addressed) contamination_status: enum // {known, partially_known, unseen, continuously_refreshed} — per-set classification (§4.1, §4.2) refresh_source: string // the instrument feeding the set: "static:human_eval" | "live:swe_bench_live" | "timestamp_gated:live_code_bench" | "held_out:swe_bench_pro" (§4.2) last_decision_use: integer // epoch of the last decision that consumed this set's tasks (§4.4) burned: boolean // tasks used to make a decision are burned for future decisions (§4.4) } ``` The load-bearing decisions, enforced not advisory: - **`role` is a closed three-valued enum.** `development` — small, fast, diagnostic, may contain known tasks, because its job is to catch breakage early, not to measure generalization. `regression` — never changes casually, run before and after every architecture modification. `holdout_live` — unseen or continuously refreshed tasks, the only number that may be called a *generalization* score. - **`burned` is the Campbell's-law rule made mechanical.** Once the harness has been iterated against a task, that task is development or regression data, never again generalization data. The `burned` flag is set deterministically the moment a task participates in a decision, and a `burned` task can never be reclassified as `unseen` — the separation is enforced by structure, not by discipline. - **`contamination_status` is a closed four-valued enum, and it is never self-reported.** `known` (public, historical, possibly trained-around); `partially_known` (public but filtered or timestamp-constrained); `unseen` (held-out, never optimized against); `continuously_refreshed` (timestamp-gated or live-updated). The status is assigned by the contamination discipline of §3.4, not by any model's assessment of whether it might have seen the task. - **`refresh_source` names the instrument's contamination mechanism.** `timestamp_gated` (LiveCodeBench's score-only-on-problems- after-cutoff); `live` (SWE-bench-Live's continuously updating corpus); `held_out` (SWE-bench Pro's not-publicly-accessible splits). A benchmark that is optimized against is measuring familiarity, not ability — and the only number that may be called generalization comes from the `holdout_live` role. ### 5.4 The ABComparison record (EXACTLY these fields) — the controlled comparison The paper's comparison discipline (§5.1, §5.2, §5.3) — "hold model, budget, environment, and sampling parameters fixed, and change exactly one thing; repeat stochastic evaluations to estimate variance; record tokens, calls, CPU, wall time, and dollar cost alongside correctness" — compiles to exactly this record. The `FrozenConfig` sub-record is the paper's frozen parameters, verbatim; the `Cost` sub-record is the paper's five-field cost, verbatim. ```json ABComparison { change_under_test: string // exactly one thing — the harness change under test (§5.1) frozen: FrozenConfig // everything else, held fixed (§5.1) arms: Arm[] // baseline and candidate, one entry each (§5.1) seeds: integer[] // the repeated stochastic evaluations (§5.2) result: ComparisonResult // the statistical readout over the two distributions (§5.2) } FrozenConfig { model: string // the model — frozen, identical across arms temperature: number // the sampling temperature — frozen token_budget: integer // the token budget — frozen tool_budget: integer // the tool budget — frozen task_set: string // the identical task set — frozen (§5.1: "what counts as the same task") environment_image: string // the identical execution environment — frozen timeout: number // the identical timeout — frozen candidate_count: integer // samples per task — frozen (pass@k discipline, §2.1) } Arm { name: string // "baseline" | "candidate" — the two sides of the comparison harness_version: string // the pinned harness artifact (§5.1) — named in every result resolved: integer[] // per-seed resolved counts (the distribution, not a single sample) mean: number // the mean resolve rate over seeds (§5.2) variance: number // the variance over seeds (§5.2) confidence_interval: number[] // [lower, upper] over seeds (§5.2) cost: Cost // tokens, calls, CPU, wall time, dollars (§5.3) } Cost { tokens: integer // LLM tokens consumed calls: integer // LLM calls made cpu: number // CPU-seconds consumed wall_time: number // wall-clock seconds elapsed dollars: number // LLM + compute dollar cost } ComparisonResult { delta: number // the measured difference between arms (§5.1 — the Δ belongs to the change) significant: enum // {yes, no, undetermined} — under the variance estimate (§5.2) recommendation: enum // {keep, reject, pare_back} — the Piper-rule verdict (§8) } ``` The load-bearing decisions, enforced not advisory: - **`frozen` is the entire set of confound variables, and it is identical across arms.** The paper's rule is non-negotiable: if the test is "does the new localization subsystem help," the comparison is not "old harness + model A" versus "new harness + model B." The correct comparison freezes the model, temperature, token budget, tool budget, task set, execution environment, timeout, and candidate count — and changes only `change_under_test`. Then, and only then, does a difference in outcome admit a causal interpretation: the Δ belongs to the change, because the change is the only thing that varied. - **`seeds` are not optional.** Agent systems are stochastic, and a difference of three points out of a hundred is, on a single run, noise. If arm A scores 35/100 and arm B scores 38/100 on one run each, nothing has been established; if A scores {33, 36, 35, 34, 37} and B scores {36, 39, 38, 40, 37}, the picture is different, and the honest report is not two means but a **mean, a variance, a confidence interval, and a cost distribution** for each arm. Repeating the evaluation enough times to estimate variance is the mechanical version of refusing to launder luck into a claim. - **`cost` is a first-class field, and it is reported alongside correctness on every arm.** Harness A at 40% resolved for $4.00/task and Harness B at 39% resolved for $0.30/task are not "A wins by one point"; B may be the strictly more useful system. Harness C at 42% in 90 minutes/task and Harness D at 40% in 4 minutes/task are not ordered by their pass rates; they are ordered by the deployment the harness is actually intended for. The cost number is what makes the scoreboard *decision-relevant* rather than merely *ranking-relevant*. - **`recommendation` is a closed three-valued enum that the deterministic harness produces, not a model's opinion.** `keep` (the change measurably improves the system), `reject` (the change measurably harms or fails to move it), `pare_back` (the change is indistinguishable from baseline — the escape hatch of §8). Nothing gets promoted because it feels smarter. --- ## 6. Build Sequence (implementation order) Eight phases, each independently testable and reversible, landing in `build/27-benchmarking-objective-scoring/` as standalone importable modules. The order is deliberate — it builds the grading plane from the executed verdict outward, then the scorecard, then the set separation, then the contamination discipline, then the A/B harness, then the cost ledger, then the instrument audit — in the exact sequence the paper's mechanics require: **executed-test runner first, then the mechanical scorer, then the hierarchical scorecard, then the benchmark-set manager, then the contamination discipline, then the A/B harness, then the cost/latency ledger, then the instrument audit.** Rollback = delete or flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Executed-test runner** | `runner.py` (§3.1 apply the patch, run the repo's real tests, record exit codes + test names + durations) | Apply a fixture patch to a fixture repo and run its test suite; assert the runner reports exit codes, test names, and durations, and nothing else; assert a patch that fails the tests yields failing exit codes; assert **no LLM consulted**. | Delete `runner.py`. | | **2. Mechanical scorer (two-sided gate)** | `scorer.py` (§5.1 the `Score` record + the `TestGate` sub-record + the two-sided FAIL_TO_PASS/PASS_TO_PASS gate) | Score a fixture patch that passes all FAIL_TO_PASS but breaks one PASS_TO_PASS; assert `resolved == not_resolved`; score a patch that passes both gates; assert `resolved == resolved`; assert `resolved` is derived, never settable by a caller; assert an empty patch scores `not_resolved`; assert **no LLM consulted**. | Delete `scorer.py`. | | **3. Hierarchical scorecard** | `scorecard.py` (§5.2 the `Scorecard` record + `ComponentMetrics` + `EndToEnd` + `Economics`) | Score a fixture run whose localization improved but whose end-to-end resolve stayed flat; assert the scorecard reports `localization.file_recall` up ("51% → 62%") *and* `end_to_end.resolved` flat, attributing the gain to the stage; assert every `component_metrics` field is computed from ground truth, never from a model's self-report; assert `economics` is present and nonzero; assert **no LLM consulted**. | Delete `scorecard.py`. | | **4. Benchmark-set manager** | `sets.py` (§5.3 the `BenchmarkSet` record + the three roles + `burned`) | Create a development, a regression, and a holdout-live set; assert the three `role` values are distinct and never conflated; burn a development-set task used in a decision; assert `burned == true` and that it can never be reclassified `unseen`; assert a holdout-live task carries `contamination_status ∈ {unseen, continuously_refreshed}`; assert **no LLM consulted**. | Delete `sets.py`. | | **5. Contamination discipline** | `contamination.py` (§3.4 timestamp gating + held-out rotation + task-burning) | Gate a fixture task list by a cutoff timestamp; assert only post-cutoff tasks are eligible for generalization scoring; rotate a held-out task after a decision; assert the retired task is removed from the generalization set; assert `contamination_status` is assigned structurally, never self-reported; assert **no LLM consulted**. | Delete `contamination.py`. | | **6. A/B harness** | `ab.py` (§5.4 the `ABComparison` record + `FrozenConfig` + `Arm` + `Cost` + `ComparisonResult`; freeze-one-change-one + variance) | Compare a fixture baseline arm against a fixture candidate arm that differ only in `change_under_test`; assert `frozen` is identical across arms; assert a difference of 3/100 on single seeds yields `significant == no` while the same arms over 5 seeds each yield a variance-backed verdict; assert `cost` is reported on both arms; assert the Δ is attributed to `change_under_test`; assert **no LLM consulted**. | Delete `ab.py`. | | **7. Cost/latency ledger** | `cost_ledger.py` (§5.3 tokens, calls, CPU, wall time, dollars alongside correctness; the correctness-to-cost ratios) | Record a fixture run's five cost fields and resolve rate; assert the ledger emits resolve-per-dollar, resolve-per-token, resolve-per-wall-clock-hour, and resolve-per-CPU-second ratios; assert two harnesses with equal resolve but a 10× cost difference are not ranked equal; assert the ledger is append-only; assert **no LLM consulted**. | Delete `cost_ledger.py`. | | **8. Instrument audit** | `audit.py` (§6 empty-patch canaries + task validation; the Agentic Benchmark Checklist properties) | Submit an empty patch and a trivial patch to the scorer; assert both score `not_resolved` (outcome validity — the scorer accepts exactly the correct solutions); validate a fixture task against the task-validity question (solvable only by the intended capability, not by a setup artifact); assert a task solvable by exploiting the evaluator is flagged; assert the audit is a standing activity with a schedule, not a one-time run; assert **no LLM consulted**. | Delete `audit.py`. | The DEFERRED rows (live instruments, multilingual/environment-state battery, value axis, cross-benchmark matrix, IRT filtering — rows 9–13 of §2) enter the sequence at the positions of Phases 4, 5, and 6 respectively, only when a measured gap justifies them and the 72h-age + scrutiny gate clears. The A/B harness (Phase 6) is the only phase that ever *invokes* the grading plane end-to-end — only to measure a change, never to modify the live harness. The benchmark component invents no new model, no new execution substrate, and no new diagnosis path; it reuses the harness's own executed tests as the things-to-score and adds only the deterministic verdict plane around them. --- ## 7. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). Round 1 is **zero-install (Python-only, stdlib + in-repo)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `enum` | §5 the `resolved` / `verdict` / `role` / `contamination_status` / `significant` / `recommendation` enums and the schema type hints | CPython stdlib. | **No** | | stdlib `typing` / `dataclasses` | §5 the frozen schemas (immutable, rebuilt not mutated) | CPython stdlib. | **No** | | stdlib `json` | §5 the Score / Scorecard / BenchmarkSet / ABComparison serialization | CPython stdlib. | **No** | | stdlib `statistics` / `math` | §5.4 the variance estimation — mean, variance, confidence interval over seeds | CPython stdlib. | **No** | | stdlib `time` / `hashlib` / `subprocess` | §3.1 the runner's exit-code capture and timing; §5 the `graded_at` epoch stamps and content-addressed `patch_ref` / `task_ids` | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3–§8 all deterministic services | In-repo, pinned interpreter. | **No** | | Harness components 01–26 | §1 the candidates (patches, plans, fixes) that this component scores; §17's test execution and §19's regression protection feed the two-sided gate; §24's sandbox is where the executed tests run | The harness's own stages, already vetted. | **No** | **The benchmark corpora are DATA fixtures, never installed packages.** The founding and successor benchmarks — SWE-bench, HumanEval, MBPP, CodeXGLUE, RepoBench, FixEval, SWE-bench-Live, SWE-bench Pro, SWE-PolyBench, SWE-Perf, LiveCodeBench, EvoEval, Terminal-Bench 2.0, Multi-SWE-bench, BigCodeBench, SWE-Lancer, MLE-bench, TheAgentCompany — are ingested as task fixtures (issue texts, repositories, test suites, environment images, gold patches) into the three benchmark sets of §5.3. They are **data**, not **code**: they are never installed, never imported, never executed as part of the harness's own runtime, and never trusted as sources of truth — they are candidate components to be *evaluated*, exactly as the paper's §6.4 discipline requires. A benchmark fixture that carries an executable grader is executed *inside* the sandbox (Component 24), never in the harness's own process. **Preferred path: zero-install.** Round 1's deterministic grading plane is Python-only and stdlib; there is no third-party Python package, no runtime substrate, and no new model. The DEFERRED rows (§2 rows 9–13) are the only ones that could pull in anything external — standing live-benchmark access, per-task Docker images, Kaggle-style graders — and none of them enters Round 1. No model, no learned-judgment dependency, and no third-party package is introduced in Round 1. ### 7.1 Bibliographic pins (corrections that must hold; no author name may be replaced by an abbreviation) These are the source-identity facts that this spec pins so that no later round of this component propagates an error. They are recorded here because the paper itself scrupulously corrects them, and the spec must not re-introduce the errors: - **EvoEval** = arXiv:2403.19114 (**not** 2407.01461, which is a different, query-refinement paper); authors Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang; **828 problems** across five semantic-altering and two semantic-preserving benchmarks; average 39.4% drop across 51 LLMs (range 19.6%–47.7%). - **SWE-bench-Live** is the *benchmark*; **"SWE-bench Goes Live!"** is the *paper* (arXiv:2505.23419, NeurIPS 2025); 1,319 tasks from 93 repositories, continuously refreshed, a dedicated Docker image per task. - **METR** = the paper title is **"Measuring AI Ability to Complete Long Software Tasks"** (arXiv:2503.14499); the 50% task-completion time horizon, timed on RE-Bench, HCAST, and 66 novel tasks. - **SWE-bench Verified** = **NOT a paper** — it is an OpenAI blog post and a repository release of a 500-instance, human-validated subset of SWE-bench; cite it through its parent SWE-bench (arXiv:2310.06770), never as a numbered reference. - **HAL (Holistic Agent Leaderboard)** = arXiv:2510.11977, **2025** (October), **not** 2026; 21,730 rollouts across 9 models and 9 benchmarks at ~$40,000. - **SWE-bench+** = arXiv:2410.06992; the audit is **solution leakage (32.67%) + weak-test patches (31.08%) + data leakage (over 94% of issues pre-cutoff)** — **NOT** an "empty-patch audit"; filtering dropped the top system from 12.47% to 3.97%. - **SWE-bench Pro** = first author **Xiang Deng** (confirmed); the venue claim "ICLR 2026," the organization "Scale AI," and the exact frontier-drop figure are **NOT confirmable from the abstract and are flagged, not asserted**. - **SWE-Lancer** = arXiv:2502.12115; the venue claim **"ICML 2025"** is **not confirmable from the abstract and is flagged, not asserted**; the four authors (Miserendino, Wang, Patwardhan, Heidecke) are confirmed. - **Campbell's Law** = Donald T. Campbell, "Assessing the Impact of Planned Social Change," *Evaluation and Program Planning* 2:67–90, 1979; **Strathern's reformulation** = "when a measure becomes a target, it ceases to be a good measure," *European Review* 5(3):305–321, 1997; **Cronbach and Meehl** = "Construct Validity in Psychological Tests," *Psychological Bulletin* 52(4):281–302, 1955. --- ## 8. A/B and Ablation — the freeze-one-change-one protocol The benchmark component, like every component, is **kept only if it strictly dominates its own absence** — measured, not asserted. But where Components 24, 25, and 26 measure a containment or authority quantity, this component's discipline is the one that makes "dominates its own absence" *possible at all* — because it is the component that supplies the instrument every other component is measured on. Its own A/B is therefore doubly reflexive: the measurement plane must prove, on its own instruments, that it improves the system's ability to know whether the system improved. ### 8.1 The Piper rule, verbatim The sentence that makes this component legible to anyone who has ever watched a plausible-looking change get merged because it *felt* better is the Piper rule, and it is this component's non-negotiable decision rule: **Nothing gets promoted because it feels smarter. A component survives only if Piper measurably performs better when it exists than when it does not.** The harness is built on top of a relatively weak model, made competitive through scaffolding; the entire bet of the architecture is that intelligent *systems* — interfaces, memory, workflows, gates — can lift a model that would lose a head-to-head comparison. That bet has a specific failure mode, and it is the failure mode this component exists to catch: a scaffolding change that *looks* sophisticated, that *feels* like it should help, but that does not, in fact, change the executed outcome at all. The only defense against that failure mode is to make the scoring machinery run on a different substrate than the thing it scores — so that the verdict on a change is produced by the same kind of mechanical computation that produces a test verdict, and never by the same kind of learned judgment that produced the change. ### 8.2 Controlled comparison: freeze everything, change one thing The causal discipline is a single rule and it is non-negotiable: when a harness change is under test, **hold model, budget, environment, and sampling parameters fixed, and change exactly one thing.** If the test is "does the new localization subsystem help," the comparison is not "old harness + model A" versus "new harness + model B," because that comparison conflates the harness change with a model change and every other difference between the two arms, and its result is uninterpretable. The correct comparison freezes the model, the temperature, the token budget, the tool budget, the task set, the execution environment, the timeout, and the candidate count — and changes only localization. Then, and only then, does a difference in outcome admit a causal interpretation: the Δ belongs to the change, because the change is the only thing that varied. The rule has a corollary about *what counts as the same task*: a comparison whose two arms were run on different task subsets, different environment images, or different timeouts is not a comparison but a confound. The pinned harness — a frozen, versioned configuration of every parameter that could influence the outcome, applied identically to both arms — is therefore an *artifact* of this component, recorded alongside every number the scoreboard produces. **Repeat stochastic evaluations to estimate variance.** Agent systems are stochastic, and a difference of three points out of a hundred is, on a single run, noise. The discipline is: run seeds, and report distributions — a **mean, a variance, a confidence interval, and a cost distribution** for each arm, never two bare point estimates. The rule is especially binding on the *small* component-level evaluation banks, where a handful of tasks means a handful of samples and the standard error is large enough to make any single difference meaningless. The failure mode this guards against is the one the Piper rule exists to name: a change "improves" the system because the random sample happened to be favorable, the improvement is recorded, the complexity is kept, and the scoreboard has laundered luck into a claim. **Report cost alongside correctness.** The discipline is: **record tokens, calls, CPU, wall time, and dollar cost alongside correctness**, and treat the ratio of correctness to cost as a score in its own right. A scoreboard that records only resolve rate is blind to the one quantity the architecture is constantly spending. Two harnesses with different cost profiles are not comparable on correctness alone, and a scaffolding change whose cost swallows its correctness gain must not be recorded as an improvement. **The three-benchmark-set discipline is binding on the A/B itself.** The repeated A/B experimentation this section prescribes would, without the §4.4 separation, slowly turn the evaluation set into training data for the *architecture* even if the model weights never changed — self-inflicted contamination, Campbell's law with the paperwork in order. The regression benchmark is run before and after every architecture modification; the true holdout/live set is touched only for major architecture decisions, rarely, and its tasks are retired afterward. A change is promoted only on the regression benchmark's verdict, and its generalization claim is reserved for the holdout number alone. ### 8.3 The ablation rule (mechanical) One mechanism removed at a time, everything else held fixed, the delta measured on the executed instruments of §3 and §5: 1. **Ablate the two-sided gate** (§3.2) — score only FAIL_TO_PASS. The delta must show a fix that breaks other things being recorded as `resolved` — the half-contract a regression test exists to catch. 2. **Ablate the hierarchical scorecard** (§3.3) — report only the end-to-end number. The delta must show a localization gain being swallowed by a downstream bottleneck and reported as "no change" — the diagnosis the decomposition exists to surface. 3. **Ablate the three-set separation** (§4.4) — conflate development, regression, and holdout tasks. The delta must show self-inflicted contamination: the A/B harness iterating against the same tasks it is measured on. 4. **Ablate timestamp gating** (§4.1) — score static, public, historical tasks as generalization. The delta must show a memorization-flavored score that collapses on post-cutoff tasks (the decay curve). 5. **Ablate variance estimation** (§5.2) — promote on single-run point estimates. The delta must show a 3/100 noise difference being recorded as a real gain. 6. **Ablate the cost ledger** (§5.3) — report resolve rate without cost. The delta must show a 10× cost increase being recorded as a win, invisible to a correctness-only scoreboard. 7. **Ablate the instrument audit** (§6) — stop auditing the benchmark itself. The delta must show an instrument defect (a task solvable by exploiting the evaluator, a scorer accepting a trivial patch) going unnoticed, and a headline score drifting toward artifact (the SWE-bench+ 12.47% → 3.97% collapse). 8. **Ablate the zero-L contract** (§4) — introduce a learned "grader" into the scoring path. The delta must show the grader's verdict inheriting the model's sycophancy and self-preference — the null hypothesis re-admitted as method — while the deterministic engine, which does not need to *judge*, holds. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise. The L column is never granted a single entry to "fix" a problem a deterministic mechanism can solve — that would be re-importing the examined party as the examiner. --- ## 9. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/27-benchmarking-objective-scoring/`; it never writes into the live harness's source. - A **feature flag** (env var `OBJECTIVE_SCORING=27|unmeasured`, default `unmeasured` during Round 1 A/B) routes the pipeline; the flag alone switches between the grading plane and the unmeasured baseline — no code deletion required to revert. The default stays `unmeasured` until §8's A/B shows strict domination, because **an empty L column is a permitted, correct state** and an unproven mechanism must not be the default. - The measurement rig (`runner.py`, `scorer.py`, `ab.py`, `audit.py`) *measures* the harness and never modifies it; the benchmark fixtures run against the candidate's executed behavior, never against the live harness's source. - The score ledger (every verdict stamped, every two-sided gate result, every per-stage metric, every benchmark-set assignment, every contamination-status assignment, every burned task, every A/B comparison, every cost record, every instrument-audit finding) is append-only and versioned; rolling the build back never loses it. This ledger *is* the attribution traceability the paper claims the naive scoreboard loses: every score, every gate, and every comparison has a name and an address. **Invariants (must hold, checked at the end of every phase).** 1. **the L column contains at most one entry** — and at this component it is exactly **zero** in the scoring path: the measurement/grading plane holds zero learned entries by construction, and the model is the thing being measured, never the thing doing the measuring. **an empty L column is a permitted, correct state** — and here it is the **only** correct state for the scoring path. 2. **Produce, then be judged; never judge.** No learned model appears in test execution, mechanical scoring, scorecard decomposition, benchmark-set separation, contamination-status assignment, variance estimation, A/B comparison, cost recording, or instrument audit. 3. **The two-sided gate is binding.** A patch is `resolved` only if every FAIL_TO_PASS test passes and every PASS_TO_PASS test is preserved; no partial credit; the verdict is read off the machine. 4. **A gain is attributable.** Every stage of the coding loop has its own mechanical metric, and a gain is attributed to the change that earned it, independently of end-to-end success. 5. **Contamination is structural.** Tasks used to make a decision are burned for future decisions; the holdout/live set is the only number that may be called generalization; timestamp gating and held-out rotation are enforced by structure, not discipline. 6. **Comparison is controlled.** Freeze everything, change one thing; repeat to estimate variance; report cost alongside correctness — always. 7. **The instrument is audited as hard as the instrument audits the harness.** Empty-patch canaries and task validation run standing and mechanically; a measure that becomes a target ceases to be a measure, so the benchmark's own integrity is a defense budget, not a virtue. 8. **The self-referential note (stated honestly).** This component closes the outermost loop — it applies the program's own deepest rule, *runtime is the only witness*, to the program itself, and it refuses to read Piper's own essays about how much smarter her latest change made her. The scoreboard does not read essays, and the first essay it refuses to read is Piper's own account of her own improvement. There is no outside: the benchmark is not measuring a hypothetical agent, it is measuring the agent that is, at this moment, writing the harness that the benchmark is part of. The D/L invariant that governs the scoring path — **the L column contains at most one entry**, and in the scoring path the correct value is zero — is the same invariant that protects this component from its own subject, because the only thing that can be trusted to say whether Piper got better is a deterministic computation over executed behavior, and Piper is, by construction, not that computation. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*