Summary / 27. Benchmarking and objective scoring / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Component 27 — Benchmarking / Objective Scoring ## Executed Tests and Measurable System Outcomes as the Only Admissible Evidence of Capability — the Scoreboard That Does Not Read Essays --- ### Abstract This paper specifies the twenty-seventh and final component of the coding harness: benchmarking and objective scoring. Components 1 through 26 built the machine — reconnaissance, retrieval, localization, planning, editing, execution, debugging, repair, candidate generation, test execution, validation, regression protection, memory, multi-agent orchestration, review, workflow control, security, sandboxing, and the permission system. Every one of those components was built on a claim, stated or implied, that the component makes Piper *better*. Component 27 is the component that makes that claim testable, because it is the one component whose subject is all the others. The central claim of this component is that a capability claim is settled by what the system *did* in an executable environment, measured mechanically, and never by what another model — or another person — *thinks* the answer looks like. "Another model says it looks right" is not a fallback methodology; it is the null hypothesis the entire discipline exists to reject. The design is a measurement plane kept outside the thing it measures: execution-based grading that runs the candidate's actual patch against the repository's actual tests and reads the verdict off the machine; a decomposed scorecard that attributes gain to the specific stage of the coding loop where it was won, so that a bottleneck can be identified independently of end-to-end success; a contamination discipline that reserves genuinely unseen or continuously refreshed tasks for generalization claims, because a benchmark that has been optimized against is no longer measuring ability but familiarity; a controlled-comparison discipline that freezes model, budget, environment, and sampling parameters when a harness change is under test, repeats stochastic evaluations to estimate variance, and records tokens, calls, CPU, wall time, and dollar cost alongside correctness; and an audit discipline that treats the instrument itself as the object of standing review, because a measure that becomes a target ceases to be a measure. The structural invariant that governs the whole design is that the learned column contains at most one entry — and at this component the enforcement path must be deterministic software on CPU [D] with zero learned entries, because the model is the thing being *measured*, never the thing doing the measuring. The evidence is assembled from the founding execution-based benchmarks (SWE-bench; HumanEval; MBPP; CodeXGLUE; RepoBench; FixEval), the contamination-resistant successors (SWE-bench Pro; SWE-bench-Live; SWE-PolyBench; SWE-Perf; LiveCodeBench; EvoEval; Terminal-Bench 2.0; Multi-SWE-bench; BigCodeBench), the economic and process-oriented outliers (SWE-Lancer; MLE-bench; TheAgentCompany), the measurement-theory lineage (Campbell's Law; Goodhart; Strathern; Cronbach and Meehl), and the audits that found the referee itself wanting (the Agentic Benchmark Checklist of Zhu and colleagues; SWE-bench+; The SWE-Bench Illusion; SWE-ABS). It closes with the self-referential observation that makes this the final component rather than merely the last one: the scoreboard does not read essays, and it does not read Piper's essays about herself either. --- ## 1. Introduction The harness has been built one component at a time, and each component has been justified, in the moment, by an argument that it makes the system more capable. Component 4 improves localization; Component 7 improves planning; Component 13 improves repair; Component 26 improves security. Each of those arguments was, at the time it was made, an argument from mechanism — *this is how it works, and here is why it should help*. Component 27 is where the program stops accepting arguments from mechanism and starts demanding evidence from outcome, because a system that tunes twenty-six components empirically needs a scoreboard, and the scoreboard is the one piece of the harness that the other twenty-six pieces are all *for*. It is the outermost loop: the fifteen-plus prior components optimize the harness, and this component is how anyone — including Piper herself — knows whether the optimization was real. The discipline this component imposes is not a new idea arriving from outside the program. It is the coding-harness instance of a rule the program already lives by, stated in its own terms: *runtime is the only witness*. Throughout the harness, a hypothesis about the world has been treated as a claim to be settled by execution — a patch is only as good as the tests it passes, a fix is only as good as the state it produces. Component 27 takes that same principle and points it at the harness itself. The claims this component adjudicates are claims of the form "Component N improved the system," and the only witness it will accept is a measured difference in executed behavior between the system with Component N and the system without it. The paper's central specification is one sentence, and it is the sentence that the entire component is an elaboration of: **Benchmarking should be executable, decomposed, contamination-resistant, cost-aware, and causally useful: grade repository behavior through deterministic tests and measurable system outcomes rather than model opinion; maintain component-level metrics for every stage of the coding loop so bottlenecks can be identified independently of end-to-end success; evaluate across languages, repositories, task types, complexity levels, and engineering objectives; reserve genuinely unseen or continuously refreshed tasks for generalization measurement; hold model, budget, environment, and sampling parameters fixed when A/B-testing harness changes; repeat stochastic evaluations to estimate variance; record tokens, calls, CPU, wall time, and dollar cost alongside correctness; and require every new subsystem to demonstrate measurable benefit under controlled ablation before it earns permanent complexity.** And the human version of that sentence, the one that makes the component legible to anyone who has ever watched a plausible-looking change get merged because it *felt* better, is the Piper rule: **Nothing gets promoted because it feels smarter. A component survives only if Piper measurably performs better when it exists than when it does not.** The second sentence is the operational consequence of the first, and it is the sentence that makes this component non-negotiable. The harness is built on top of a relatively weak model, made competitive through scaffolding; the entire bet of the architecture is that intelligent *systems* — interfaces, memory, workflows, gates — can lift a model that would lose a head-to-head comparison. That bet has a specific failure mode, and it is the failure mode this component exists to catch: a scaffolding change that *looks* sophisticated, that *feels* like it should help, but that does not, in fact, change the executed outcome at all. The only defense against that failure mode is to make the scoring machinery run on a different substrate than the thing it scores — so that the verdict on a change is produced by the same kind of mechanical computation that produces a test verdict, and never by the same kind of learned judgment that produced the change. ### 1.1 The inversion: the scoreboard, not the essay-reader The central inversion of this component is that the question it answers — *is Piper better?* — is the one question in the entire harness that cannot be answered by asking a model. Every prior component, in its normal operation, is allowed to use the model for what the model is good at: reading, planning, generating, choosing. The benchmark component is the single place where that permission is revoked, because the model is the *subject* of the question. To ask the model whether the model-plus-harness is better is to ask the examined party to grade its own exam, and the answer will be whatever the model's sycophancy, verbosity, and self-preference biases produce — a confident, fluent, and worthless number. The founding empirical claim of execution-based evaluation is exactly this: text-similarity metrics and generative plausibility *mispredict* functional correctness. A candidate answer can be near-identical to the reference and still wrong, or wildly different and still right, and the only entity that knows which is the runtime that executes it. This is why the component's null hypothesis is stated the way it is. "Another model says it looks right" is not a weaker grade of evidence to be used when execution is inconvenient; it is the *null hypothesis* — the default assumption that must be actively rejected by producing something better. The method that replaces it is the scoreboard: executed tests against real, held-out tasks, run through a pinned harness, scored mechanically, with per-stage attribution and honest variance. The scoreboard does not read essays. It reads outcomes. And the specific way it refuses to read essays is the subject of the next subsection. ### 1.2 The D/L invariant Every element of this component is classified into one of two columns: **[D]** — deterministic software running on CPU — or **[L]** — a learned model. The structural invariant that governs the entire design is the one that has governed the harness from its first component: > **the L column contains at most one entry.** At this component, that invariant is held at a strength the earlier components did not require, because of a specific structural fact about what this component is: the model is the thing being *measured*, and therefore it cannot be the thing doing the measuring. The measurement/grading plane — test execution, mechanical scoring, the pinned harness, variance estimation, the A/B comparison — is deterministic software on CPU [D] with **zero learned entries**. Test execution is a subprocess and a diff of exit codes; mechanical scoring is a comparison of a verdict against a ground-truth patch's tests; the pinned harness is a frozen configuration applied identically to both arms of a comparison; variance estimation is a computation over repeated seeds; the A/B comparison is a statistical test over two distributions of resolved/not-resolved. None of these operations requires, or tolerates, a model call, and every one of them is corrupted by the presence of one. A grader that contains a learned model is a grader whose verdict inherits every bias the model under test was selected for, and a benchmark whose scores are partially model-opinion is a benchmark that has smuggled the null hypothesis back in as part of the method. **An empty L column is a permitted, correct state**, and for the scoring path it is the *only* correct state. The model [L] is the *subject* of the score, outside the scoring path — it is the thing whose patches are executed, whose tokens are counted, whose behavior is being characterized, and it has no role in producing the characterization. This is the component's distinctive version of the invariant: in the security component, the model was excluded from the enforcement path because it was the thing being *steered*; in the benchmark component, the model is excluded from the scoring path because it is the thing being *scored*. The two exclusions are the same exclusion, applied at the two ends of the loop — the model is never the judge of its own behavior, whether that behavior is "obeying an injected instruction" or "resolving a GitHub issue." The consequence is a division of labor that runs the entire length of this paper. The learned model produces *candidates* — patches, plans, fixes, essays about what it is doing. The deterministic plane produces *verdicts* — resolved, not resolved, faster, slower, cheaper, more expensive, better, worse. The two columns touch at exactly one point, which is the point where a candidate is handed to the runner and a verdict is handed back. Everything upstream of that point is the model's business; everything downstream is the machine's; and the machine's column is empty of learning. ### 1.3 A preamble: the component that closes the outermost loop This component is the final one, and the position is not an accident of ordering. The program's own deepest rule — *runtime is the only witness* — has been applied, up to this point, to the *world*: a patch is evidence against a bug, a passing test is evidence against a regression, an executed trace is evidence against a claimed behavior. Component 27 applies the same rule to the *program itself*. The fifteen-plus prior components are all, in one register or another, hypotheses about how to make a model more capable; this component is the discipline that refuses to accept a hypothesis on its own authority, and refuses — this is the self-referential turn — to accept Piper's *feeling* of improvement as evidence of improvement either. The benchmark is the scoreboard that does not read essays, and the first essay it refuses to read is Piper's own account of how much smarter her latest change made her. The component's honest position is therefore reflexive in a deliberate way: it is the measurement machinery that measures measurement, the audit that audits audits, and its subject is the entire system of which it is a part. There is no outside. The D/L invariant that governs the scoring path — the L column contains at most one entry, and in the scoring path the correct value is zero — is the same invariant that protects this component from its own subject, because the only thing that can be trusted to say whether Piper got better is a deterministic computation over executed behavior, and Piper is, by construction, not that computation. --- ## 2. The founding contract: execution over opinion The discipline of execution-based evaluation did not begin with repository-level agents; it began with the much older observation that *fluency is not correctness*. A program that is textually similar to a reference solution can be wrong in a way that no string comparison detects, and a program that looks nothing like the reference can be right in a way that only execution reveals. This section establishes the founding contract — the executed verdict as the unit of evidence — and assembles the canonical instruments that wrote it into the field's practice. ### 2.1 Functional correctness, not text similarity: HumanEval, MBPP, and CodeXGLUE The contract was first written at the level of the single function. Chen, Tworek, Jun, Yuan, Ponde de Oliveira Pinto, Kaplan, Edwards, Burda, Joseph, Brockman, Ray, Puri, Krueger, Petrov, Khlaaf, Sastry, Mishkin, Chan, Gray, Ryder, Pavlov, Power, Kaiser, Bavarian, Winter, Tillet, Petroski Such, Cummings, Plappert, Chantzis, Barnes, Herbert-Voss, Guss, Nichol, Paino, Tezak, Tang, Babuschkin, Balaji, Jain, Saunders, Hesse, Carr, Leike, Achiam, Misra, Morikawa, Radford, Knight, Brundage, Murati, Mayer, Welinder, McGrew, Amodei, McCandlish, Sutskever, and Zaremba's "Evaluating Large Language Models Trained on Code" [14] introduced HumanEval and, with it, the *pass@k* metric: a solution counts as correct if and only if it passes the hidden unit tests that encode the intended behavior, and correctness is measured over repeated samples so that a model's *ability to eventually produce a working solution* is scored rather than its ability to produce one on a single lucky draw. The paper's own headline — 28.8% on HumanEval for Codex, with the same model reaching 70.2% when allowed 100 samples per problem — is itself a demonstration of the component's first lesson: a single run is not a capability, it is a sample from a distribution, and any honest scoreboard must know the difference. Austin, Odena, Nye, Bosma, Michalewski, Dohan, Jiang, Cai, Terry, Le, and Sutton's "Program Synthesis with Large Language Models" [16] supplied the complementary instrument, MBPP, with 974 entry-level programming tasks, and established the second lesson: capability scales with the *difficulty distribution* of the task set, and a benchmark of toy problems measures toy performance. Lu, Guo, Ren, Huang, Svyatkovskiy, Blanco, Clement, Drain, Jiang, Tang, Li, Zhou, Shou, Zhou, Tufano, Gong, Zhou, Duan, Sundaresan, Deng, Fu, and Liu's CodeXGLUE [17] supplied the third, and it is the lesson that the whole component is built on: *evaluation is infrastructure*. A benchmark dataset is a platform, not a one-off, and its value is that it lets many systems be compared under one mechanical protocol — which is exactly what makes it, in Campbell's sense, a target the moment it becomes consequential, the problem this paper returns to in §6. The shared move of all three is the one that matters for the harness. String-equality, edit similarity, and human read-through are all *proxies* for correctness; execution is the *thing itself*. The distinction is the difference between a judge and a witness. A judge interprets evidence; a witness reports what happened. The founding benchmarks chose the witness, and every instrument this paper discusses is, whatever else it is, an elaboration of that choice. ### 2.2 SWE-bench: the two-sided gate and the definition of "resolved" The contract was generalized from the single function to the whole repository by Jimenez, Yang, Wettig, Yao, Pei, Press, and Narasimhan's "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" [1] — ICLR 2024, and the paper that defined the field's measurement standard. SWE-bench is 2,294 real GitHub issues drawn from 12 popular Python repositories, each paired with the pull request that actually fixed it. A model is given the codebase and the issue text, and must produce a patch. The patch is graded by the tests that the *human* fix's pull request introduced and preserved, and the grading is two-sided in a way that has become the field's definition of the word "resolved": a patch is accepted if and only if it makes every **FAIL_TO_PASS** test pass (the tests that failed before the fix and pass after it — the proof that the fix *works*) *and* preserves every **PASS_TO_PASS** test (the tests that passed before and must still pass after — the proof that the fix *broke nothing*). The two-sided gate is the component's single most important inheritance, because it is the executed form of the discipline a human pull-request reviewer applies: a fix that fixes the bug and breaks three other things is not a fix. A benchmark that scored only FAIL_TO_PASS would be scoring half the contract; a harness that optimizes only for making the failing test pass is optimizing for exactly the half-measure that a regression test exists to catch. The binary verdict is deliberate and it is the second inheritance. SWE-bench grants no partial credit: an instance is resolved or it is not, and the "almost right" patch — the one that localizes correctly but fails validation — scores identically to the patch that never got started. For *end-to-end* scoring this is a feature, not a defect: it is the honest, uncompromising summary of "did the system do the job." But it is precisely this binary-ness that makes a *single* number radically insufficient, which is why the decomposition of §3 exists. The number tells you *that* the system resolved an instance; it tells you nothing about *which part* of the system earned the resolution or lost it — and a scoreboard that cannot attribute gain cannot guide the next change. SWE-bench's founding evaluation made the point economically: the best system of the moment, Claude 2, solved 1.96% of the instances, and the authors' own fine-tuned SWE-Llama solved only the simplest issues. The benchmark was, at launch, far from saturated — and the field's subsequent history is the history of what happens to an unsaturated benchmark once everyone navigates by it, which is the subject of §4 and §6. ### 2.3 Construct validity: what a score is evidence *of* The founding contract has a psychometric name, and it is worth stating because the component's honesty depends on it. Cronbach and Meehl's "Construct Validity in Psychological Tests" [18] — *Psychological Bulletin* 52(4):281–302, 1955, DOI 10.1037/h0040957 — formalized the question every benchmark must answer and frequently does not: *what is this score evidence of?* A score is a measurement of a construct — some abstract capability that the test is meant to operationalize — and the score is valid only to the degree that the test actually samples the construct it claims to. The discipline has a specific failure mode, and it is the one this component spends its remaining sections defending against: **construct error**, the mistake of reading a score on one construct as a score on a different, grander one. A HumanEval number is evidence of "ability to complete short single-function synthesis tasks," and reading it as evidence of "software-engineering ability" is construct error of exactly the kind Cronbach and Meehl's framework was built to detect. The entire history of code-agent benchmarking is, seen through this lens, a series of corrections to construct error: each successor benchmark exists because a predecessor's score was being read as evidence of something broader than its actual construct, and the successor's job is to make the construct the field *actually* wants — repository engineering, multilingual engineering, performance engineering, economically valuable engineering — the thing the test samples. The psychometric framing also supplies the component's standing obligation: **triangulate constructs**. One benchmark operationalizes one construct, and no single number — however clean — is evidence of general capability. The scoreboard this paper specifies is therefore always a battery, never a single line, and every claim it produces is scoped to the construct the relevant instrument samples. This is the discipline that lets the harness say "localization improved" (a component-level claim) without thereby saying "Piper is a better engineer" (a construct error), and it is the reason the decomposition of §3 and the contamination discipline of §4 are not two features of the scoreboard but two halves of the same defense against self-deception. ### 2.4 FixEval: scale, and the proof that match-based metrics lie The founding contract's final load-bearing member is its proof that the *proxies* actually lie, at scale. Haque, Ahmad, Lourentzou, and Brown's "FixEval: Execution-based Evaluation of Program Fixes for Programming Problems" [22] built the largest execution-based program-repair benchmark of its time, and the scale is itself the argument: the raw submission pool is **712K Java and 3.28M Python** program submissions with test suites — substantially larger than prior repair benchmarks — and the curated splits are smaller but still large (on the order of 245K Java and 1,111K Python). The paper's finding is the component's empirical foundation, stated in the authors' own terms: *match-based metrics do not reflect model-generated program fixes accurately*, while execution-based evaluation runs each candidate through the full test suite designed for that solution. This is the founding claim of §1.1 made concrete — a text-similarity metric and a test suite disagree about which fixes are correct, and the test suite is the one that is right — and it is the reason the harness's scoreboard is built on execution from its first line to its last. FixEval also supplies a correction the harness must internalize: multi-line bugs, which the single-line toy benchmarks systematically under-represent, are the *realistic* case, and a scoreboard that only ever scores single-line defects is sampling a construct that does not match the work the harness is built to do. The same correction appears, one level up, in every repository-level instrument this paper cites: the field keeps discovering that its benchmarks were easier than its stated ambitions, and each discovery is a construct-validity correction in Cronbach and Meehl's exact sense. --- ## 3. Decomposition: component-level metrics A single end-to-end number is a verdict without a diagnosis. It says *whether*, and it says nothing about *where* — and a harness built from twenty-six tunable components needs to know *where*. This section specifies the decomposition that turns a scoreboard from a score into a diagnosis: the same executed verdict, attached to the specific stage of the coding loop that produced it, so that a gain in localization is visible even when it is swallowed by a loss in validation, and a bottleneck is identifiable independently of end-to-end success. ### 3.1 RepoBench: retrieval, completion, and pipeline, scored separately The decomposition has a founding instrument, and it is RepoBench [15] — Liu, Xu, and McAuley, ICLR 2024 — which was built to score the subsystems of a repository-level system *separately* rather than smeared together. RepoBench defines three settings, and the three-way split is the template for every component-level metric this paper specifies: **RepoBench-R** (Retrieval), which scores the system's ability to retrieve the most relevant code snippet from the repository as cross-file context; **RepoBench-C** (Completion), which scores pure next-line prediction given in-file and cross-file context; and **RepoBench-P** (Pipeline), which composes retrieval and completion into the full end-to-end task. The split matters because it converts an un-attributable number into an attributable one: a system whose end-to-end score is low because its retrieval is weak is a different system, needing a different fix, from one whose end-to-end score is low because its completion is weak — and a scoreboard that only reports the pipeline number cannot tell them apart. This is the component's charter, stated in benchmark form: **isolate capabilities in design** so that gains are attributable to stages, not smeared across a pipeline. The same principle, applied to Piper's actual loop, produces the hierarchical scorecard the harness should maintain. For every task the harness attempts, the scoreboard records not only *resolved?* but a vector of per-stage metrics: localization (file recall, symbol recall, rank of the correct target); planning (fraction of required obligations recovered); patching (does the patch apply, parse, build); structure (were the correct entities touched, were unnecessary entities touched); testing (were the bug tests and the regression tests both exercised); and cost (tokens, calls, CPU, wall time, dollars). The end-to-end number remains the headline, but it is now the *bottom* of a hierarchy rather than the whole of it, and every component of the harness has a number of its own. When Component 4 (localization) changes, the scoreboard reports its effect on localization Recall@k directly — "51% → 62%" — rather than forcing that effect to be inferred from a final resolve rate that a downstream bottleneck may have held flat at "28% → 29%." The gain is real, it is visible, and it is correctly attributed; the loss of propagation is itself the diagnosis that the next change should target. ### 3.2 SWE-agent: the harness is a first-order variable The decomposition's second founding result is the one that makes the harness itself — the thing this whole program is — a subject of measurement rather than an invisible constant. Yang, Jimenez, Wettig, Lieret, Yao, Narasimhan, and Press's "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering" [20] (NeurIPS 2024) demonstrated that the *agent-computer interface* — the set of commands, the observation format, the editing affordances — moves resolution by multiple points *at fixed model*. The same backbone model produces materially different scores under different scaffolds, and the paper's own numbers made the point: 12.5% on SWE-bench and 87.7% on HumanEvalFix for its custom interface, with the interface itself as the differentiator. The significance for this component is precise and twofold. First, **benchmarks measure systems, not models**: every number ever reported is a number for a *model-plus-harness*, and a comparison that does not name the harness is not a comparison but a conflation. Second, and more consequentially for the program, the harness is itself the subject of the ablation discipline of §5: if the interface moves the score by multiple points, then the entire project of building interfaces — the entire project this paper series is — is a project that *must* be measured the same way, change by change, with the model held fixed. SWE-agent is the proof that the scoreboard has something to do: the variance between harnesses is real, measurable, and large, and a program that tunes harnesses without measuring harness-level variance is flying an instrument that it has never calibrated. ### 3.3 The hierarchical scorecard The synthesis of §3.1 and §3.2 is the scorecard this component specifies as its central artifact. It has three planes. The **component-metrics plane** carries the per-stage numbers of §3.1 — reconnaissance, retrieval, localization, impact, planning, patch quality, debugging, validation, regression protection, memory, multi-agent, reviewer, workflow, security, sandbox, permissions — each with its own objective metric. The **end-to-end plane** carries the executed verdicts: resolved, regressions, robustness. And the **economics plane** carries the cost/value numbers of §5: dollars per task, time per task, compute per task, and — where real work is being measured — economic value per dollar spent. The three planes are read together, never separately, and their joint reading is what lets the harness say, with a straight face, a sentence of the form the naive scoreboard cannot produce: *Component 4 improved and is no longer the bottleneck; the residual gap now sits at Component 13.* That sentence is the entire point of decomposition. The naive benchmark produces "42%," a number that is useful and radically insufficient; the hierarchical scorecard produces the diagnosis that tells the next change where to go. The per-component metrics are, each, executable and mechanical, in the same register as the founding contract: localization is scored by Recall@k against the gold patch's touched files and symbols; planning by the fraction of required obligations the plan recovered; patching by apply/parse/build success; regression protection by the rate of introduced-regression detection; the reviewer by defect precision and recall against a held-out defect set. None of these is a model's opinion of itself; each is a deterministic computation over a candidate's behavior against a ground truth. The component that this paper specifies does not *itself* build these instruments from scratch for every stage — it specifies the requirement that they exist, and that every subsystem earn its place against them — because the deeper point is architectural: the scoreboard is a *grading plane* that mirrors the harness's own D/L discipline, and it is the subject of §1.2 that keeps every metric on the D side of the line. ### 3.4 SWE-PolyBench: beyond binary resolve The decomposition's final refinement is the one that resists the binary verdict's blind spot within a *single* stage. Rashid, Bock, Zhuang, Buchholz, Esler, Valentin, Franceschi, Wistuba, Sivaprasad, Kim, Deoras, Zappella, and Callot's "SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents" [4] contributes two things at once. First, it is the multilingual correction: 2,110 instances from 21 repositories spanning Java, JavaScript, TypeScript, and Python, across bug fixes, feature additions, and refactorings, with a 500-instance stratified subset for affordable evaluation — the concrete statement, in benchmark form, that **Python performance is not software-engineering performance**, and that a harness tuned to one language's idiom may or may not transfer to another's. Second, and more important for this section, it refuses to stop at *resolved / not resolved*: it adds metrics based on **syntax-tree structure**, so that an agent that localized the correct symbols, produced a structurally closer patch, touched fewer unrelated entities, and compiled more often — yet still failed validation — is distinguishable from one that never got started. That is exactly the diagnostic information the binary verdict discards, and exactly the information a component-level scoreboard needs: two harnesses can tie at "28% resolved" while one of them is systematically *closer* at every stage, and the tie is not a tie but a measurement artifact of a too-coarse instrument. The multilingual finding completes the picture: the authors report substantial performance differences across languages, task types, and complexity categories, which is the empirical form of the §1 requirement to *evaluate across languages, repositories, task types, complexity levels, and engineering objectives* — a harness that scores uniformly across all of them is genuinely capable, and one whose score collapses outside its home distribution is optimized, not improved. --- ## 4. Contamination resistance and generalization A benchmark that has been trained against, tuned against, or optimized against is no longer measuring ability; it is measuring familiarity. This is the contamination problem, and it is the single deepest threat to the entire discipline, because it is *invisible in the numbers*: a model that memorized the test set and a model that can actually solve the problems produce the same score on the test set, and the scoreboard cannot tell them apart from the score alone. This section assembles the contamination-resistant instruments and specifies the three-benchmark-set discipline that keeps the harness's generalization measurement honest. ### 4.1 The decay curve: LiveCodeBench and EvoEval The contamination problem has a measurement, and it has a decay curve. Jain, Han, Gu, Li, Yan, Zhang, Wang, Solar-Lezama, Sen, and Stoica's "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" [10] attacks it by construction: it continuously collects fresh problems from LeetCode, AtCoder, and Codeforces *with release timestamps*, and scores each model only on problems published after its training cutoff. The design is timestamp-gated contamination control — the benchmark never asks a model to solve a problem it could have seen, and so the contamination question is settled by construction rather than by audit. Its founding finding is the decay curve: on uncontaminated problems, model performance drops sharply relative to the older, likely-memorized sets, and the drop grows as benchmarks age. The implication is stated plainly by the paper itself and is this component's standing assumption: **every static benchmark's numbers should be read as decaying toward a recall test.** A number earned on a fixed, public, old test set is a number that is partly memorization and partly ability, with no way to partition the two, and the partition's composition drifts toward memorization as the set ages. Xia, Deng, and Zhang's "Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM" [9] supplies the second half of the measurement: overfitting to the benchmark *distribution*, made visible. EvoEval systematically *evolves* well-known benchmark problems into harder, subtler, semantically-altered variants — 828 problems across five semantic-altering and two semantic-preserving benchmarks, verified for target behavior through a pipeline that combines LLM generation with human checking. The headline is the decay quantified: across 51 LLMs, performance drops **on average 39.4%** on the evolved variants relative to the originals the models were trained around, with the drop ranging from 19.6% to 47.7% and *drastic ranking changes* among models. The corrected identity of this paper is worth stating scrupulously, because it is one of the field's most frequently mis-cited references: the arXiv identifier is 2403.19114 (not the query-refinement paper 2407.01461), the authors are Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang, and the scale is 828 problems (not ~4,900). The lesson it carries is the same lesson LiveCodeBench carries, seen from the other side: a meaningful share of "benchmark ability" is *distribution familiarity*, and the honest lower bound of the ability actually bought is the score on perturbed versions of familiar tasks. For the harness, the operational form of this lesson is simple and severe: **rotate and perturb.** A scoreboard that only ever scores the same static problems is measuring the harness's memory of those problems, not its ability, and the two diverge on a curve that §4.1 has now measured. ### 4.2 Continuously refreshed and deliberately hard: SWE-bench-Live and SWE-bench Pro The two successors that carry the contamination resistance from the toy-function level to the repository level are, respectively, the *moving* benchmark and the *hidden* benchmark. Zhang, He, Zhang, Kang, Li, Xie, Wang, Wang, Huang, Fu, Nallipogu, Lin, Dang, Rajmohan, and Zhang's "SWE-bench Goes Live!" [3] — the paper is titled "SWE-bench Goes Live!"; the benchmark is **SWE-bench-Live** — attacks the problem by refusing to stand still. Its initial release contains **1,319 tasks from 93 repositories**, derived from GitHub issues created from 2024 onward, with a dedicated Docker image per task and an automated curation pipeline whose entire purpose is that the task set *keeps updating over time* rather than freezing into a historical corpus; it is a live-updatable, multi-language, multi-OS benchmark, and its leaderboard requires submitted trajectories precisely so that maintainers can verify that agents were not exposed to hidden evaluation fields or ground-truth results. The contrast with a frozen benchmark is the point: a frozen benchmark becomes a *known environment, known repositories, known issue distribution, known failure modes* even if nobody cheats, while a live benchmark hands the September-2026 harness issues that did not exist when earlier architectural decisions were made — which is what an actual generalization test *is*. Deng, Da, Pan, He, Ide, Garg, Lauffer, Park, Pasari, Rane, Sampath, Krishnan, Kundurthy, Hendryx, Wang, Bharadwaj, Holm, Aluri, Zhang, Jacobson, Liu, and Kenstler's "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" [2] attacks the same problem by making the tasks *hidden* and *hard*. The confirmed facts are these: SWE-Bench Pro contains **1,865 problems sourced from 41 actively maintained repositories**, partitioned into a public set (open access to 11 repositories), a held-out set (12 repositories), and a commercial set (18 proprietary repositories under formal partnership agreements); the held-out and commercial problems are not publicly accessible, and the tasks are long-horizon, requiring coordinated modifications across multiple files. The first author, Xiang Deng, is confirmed. Three further claims that circulate about this paper are **not confirmable from the abstract and are therefore flagged, not asserted**: the venue "ICLR 2026," the organization "Scale AI," and the exact magnitude of the frontier drop at launch. What *is* established is the design's purpose, which is precisely the one this component needs: the held-out and private splits exist so that a harness cannot optimize against the benchmark without literally seeing the benchmark, and the tasks' length and difficulty exist so that "solving SWE-bench Pro" does not reduce to "recalling SWE-bench Verified." The reported evaluation in the paper's own words is that even the best system in the unified-scaffold evaluation remained below 25% Pass@1 — the benchmark remains substantially unsaturated — but the specific frontier numbers should be read against the primary source rather than repeated here as settled fact. The two successors together define the generalization axis of the scoreboard. SWE-bench-Live keeps the *tasks* fresh; SWE-bench Pro keeps the *repositories* unseen. A harness that is genuinely better at software engineering improves on both; a harness that has merely memorized the public, historical distribution improves on neither, and the difference is exactly the difference this component exists to detect. ### 4.3 Environment-state and multilingual extensions: Terminal-Bench 2.0, Multi-SWE-bench, and BigCodeBench The contamination-resistant axis is completed by three instruments that broaden what "ability" the scoreboard samples. Merrill and eighty-four co-authors' Terminal-Bench 2.0 [7] is the environment-state alternative to patch-based scoring: each of its 89 tasks features a unique environment, a human-written solution, and comprehensive tests for verification, and success is defined by whether the agent *transforms the environment into a passing state* — not merely whether its diff matches a ground truth — so that terminal interaction jointly exercises observation design, context management, control-loop policy, action exposure, state persistence, and verification, which is the harness-level variance the whole program is about. Its maintenance history is itself a measurement of how much "objective" scores move when the objective is corrected: the 2.1 revision changed 31% of the tasks (28 of 89) to fix specification-verification mismatches, resource mismatches, and drift, with shifts of up to 12 points for frontier agents. That number is a standing warning to this component: the instrument's own drift is comparable in magnitude to the improvements the harness is trying to detect, which is why §6 makes the audit of the instrument a permanent activity rather than a one-time apology. Zan, Huang, Liu, Chen, Zhang, Xin, Chen, Liu, Zhong, Li, Liu, Xiao, Chen, Zhang, Su, Liu, Long, Shen, and Xiang's "Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving" [13] (NeurIPS 2025) extends the SWE-bench protocol across Java, TypeScript, JavaScript, Go, Rust, C, and C++ — 1,632 high-quality instances annotated from 2,456 candidates by 68 expert annotators — and is the proof that the *language* axis is a real axis of generalization rather than a cosmetic one. Zhuo, Vu, Chim, Hu, Yu, Widyasari, Yusuf, Zhan, He, Paul, Brunner, Gong, Hoang, Zebaze, Hong, Li, Kaddour, Xu, Zhang, Yadav, Jain, Gu, Cheng, Liu, Liu, Wang, Hui, Muennighoff, Lo, Fried, Du, de Vries, and Von Werra's "BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions" [11] contributes the third axis: 1,140 practical tasks with diverse library calls, complex multi-step instructions, and rich I/O, execution-scored with human-verified test suites, and its finding is the realism gap quantified — frontier models solve these practical tasks at rates far below their HumanEval-class numbers, and instruction-following precision (doing *exactly* what was asked) is a separate bottleneck from code fluency. All three are, in Cronbach and Meehl's terms, construct-validity corrections: they move the sampled construct closer to the work the harness is actually built to do, and a scoreboard that ignores them is, by omission, claiming that toy-function performance stands in for repository, environment, and practical-task performance — the construct error §2.3 named. ### 4.4 The three-benchmark-set discipline The synthesis of this section is a rule about *which* benchmarks the harness runs, and *when*. The harness maintains three distinct sets, and never conflates them. The **development set** is small, fast, and diagnostic, used constantly during the normal course of work; it may contain known tasks, because its job is not to measure generalization but to catch breakage early. The **regression benchmark** never changes casually, and is run before and after every architecture modification with the old harness and the new harness compared on identical models, temperatures, budgets, and environments; its job is the A/B discipline of §5. And the **true holdout/live set** is the set Piper never sees during development — new repositories, new tasks, new issues, drawn from the continuously refreshed and held-out instruments of §4.2 — and it is the only number that may be called a *generalization* score. The discipline's core is a hard rule with a Campbell's-law rationale stated in full in §6: **tasks used to make a decision are burned for future decisions.** Once the harness has been iterated against a task, that task is development or regression data, never again generalization data; the holdout set is touched only for major architecture decisions, rarely, and its tasks are retired afterward. Without this separation, the repeated A/B experimentation that §5 prescribes would slowly turn the evaluation set into training data for the *architecture* even if the model weights never changed — self-inflicted contamination, Campbell's law with the paperwork in order. --- ## 5. Controlled comparison: A/B, variance, and cost A scoreboard that reports a single pass rate is reporting a sample as if it were a population. This section specifies the discipline of *comparison* — the part of the component that turns a score into evidence that a change *caused* an improvement — and the discipline of *cost* — the part that refuses to let a two-point gain at a ten-times price be recorded as a win. ### 5.1 Freeze everything, change one thing The causal discipline is a single rule and it is non-negotiable: when a harness change is under test, **hold model, budget, environment, and sampling parameters fixed, and change exactly one thing.** If the test is "does the new localization subsystem help," the comparison is not "old harness + model A" versus "new harness + model B," because that comparison conflates the harness change with a model change and every other difference between the two arms, and its result is uninterpretable. The correct comparison freezes the model, the temperature, the token budget, the tool budget, the task set, the execution environment, the timeout, and the candidate count — and changes only localization. Then, and only then, does a difference in outcome admit a causal interpretation: the Δ belongs to the change, because the change is the only thing that varied. This is the harness's existing ablation philosophy, formalized and made load-bearing. It is also, not coincidentally, the discipline that the D/L invariant of §1.2 protects: the comparison is a deterministic computation over two distributions of executed verdicts, and there is no model anywhere in the comparison path to smuggle a preference for one arm over the other. The rule has a corollary about *what counts as the same task*. A comparison whose two arms were run on different task subsets, different environment images, or different timeouts is not a comparison but a confound. The pinned harness is therefore an *artifact* of this component — a frozen, versioned configuration of every parameter that could influence the outcome, applied identically to both arms — and it is recorded alongside every number the scoreboard produces. This is the lesson of SWE-agent [20] made operational: because the harness is a first-order variable, the harness is *named* in every result, and cross-harness comparisons are treated as cross-experiments rather than rankings. ### 5.2 Repeat stochastic evaluations to estimate variance Agent systems are stochastic, and a difference of three points out of a hundred is, on a single run, *noise*. The discipline is therefore: run seeds, and report distributions. If Harness A scores 35/100 and Harness B scores 38/100 on one run each, nothing has been established; if A scores {33, 36, 35, 34, 37} and B scores {36, 39, 38, 40, 37}, the picture is different, and the honest report is not two means but a **mean, a variance, a confidence interval, and a cost distribution** for each arm. The rule is especially binding on the *small* component-level evaluation banks of §3, where a handful of tasks means a handful of samples and the standard error is large enough to make any single difference meaningless. The failure mode this guards against is the one the Piper rule exists to name: a change "improves" the system because the random sample happened to be favorable, the improvement is recorded, the complexity is kept, and the scoreboard has laundered luck into a claim. Repeating the evaluation enough times to estimate variance is the mechanical version of refusing that laundering. The statistical framing is supplied by Ndzomga's "Efficient Benchmarking of AI Agents" [19], a single-author paper whose question is exactly the cost-of-variance question the harness faces on a budget: how few tasks suffice to *rank* agents correctly. Across eight benchmarks, 33 agent scaffolds, and more than 70 model configurations, the finding is an asymmetry the harness can exploit — absolute score prediction degrades under scaffold-driven distribution shift, while rank-order prediction remains stable — and the practical protocol is a mid-range difficulty filter (evaluate only on tasks with intermediate historical pass rates of roughly 30–70%, motivated by Item Response Theory), which reduces the number of evaluation tasks by 44–70% while maintaining rank fidelity. The lesson for the component is twofold: variance is a *budgetable* problem (a well-chosen subset ranks reliably at a fraction of the cost), and a random sample is *not* that subset (random sampling exhibits high variance across seeds). The scoreboard's small, fast, diagnostic set of §4.4 should be chosen the way Ndzomga chooses it — for discriminating power, not for convenience. ### 5.3 Cost is a first-class score The third discipline is the one the harness's entire architecture makes mandatory, and it is this: **record tokens, calls, CPU, wall time, and dollar cost alongside correctness**, and treat the ratio of correctness to cost as a score in its own right. The architecture's founding bet is that a relatively weak model can be made competitive through scaffolding — and scaffolding costs tokens, calls, and wall-clock time. A scoreboard that records only resolve rate is therefore blind to the one quantity the architecture is constantly spending. The cost-aware numbers are ratios: resolve rate per LLM dollar, resolve rate per LLM token, resolve rate per wall-clock hour, resolve rate per CPU second. The point is not that cost trumps correctness; it is that two harnesses with *different* cost profiles are not comparable on correctness alone. Harness A at 40% resolved for $4.00/task and Harness B at 39% resolved for $0.30/task are not "A wins by one point"; B may be the strictly more useful system. Harness C at 42% in 90 minutes/task and Harness D at 40% in 4 minutes/task are not ordered by their pass rates; they are ordered by the deployment the harness is actually intended for. The cost number is what makes the scoreboard *decision-relevant* rather than merely *ranking-relevant*, and it is the number that keeps a scaffolding change whose cost swallows its correctness gain from being recorded as an improvement. ### 5.4 Objective outcomes beyond the binary: SWE-Perf, SWE-Lancer, and MLE-bench The correctness axis is not the only objective axis, and the first proof that it need not be is performance itself. He, Liu, Du, Yan, Fan, Huang, Zheng, Yuan, and Ma's "SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?" [30] scores a patch on whether it makes real software *faster without breaking correctness*: 140 repository-level performance-optimization tasks derived from real performance-improving GitHub pull requests, each with an executable environment, target functions, performance-related tests, and expert-authored patches. The objective continuum it produces is the multi-objective scoreboard made concrete — Candidate A passes its tests at 100 ms, Candidate B at 72 ms, Candidate C at 130 ms — and the ranking B > A > C is read off the machine *with no model in the loop*, because reality supplies the ordering and the runner merely reports it. The lesson generalizes beyond runtime: correctness, performance, resource use, patch size, regressions, and robustness are all objective, measurable dimensions, and two patches that both "resolve" a task are not thereby equally good — the one that resolves it in 17 changed lines at unchanged latency is a better engineering outcome than the one that resolves it in 220 changed lines at 40 ms added, and only a scoreboard that records the continuous dimensions can see the difference. The paper's own finding, that existing methods leave a substantial capability gap to expert-level optimization performance, is the calibration that the *performance* axis remains far from saturation even as the *correctness* axis approaches it. The same principle extends, at the far end of the scoreboard, into an axis that measures not *resolved* but *valuable*. Miserendino, Wang, Patwardhan, and Heidecke's "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?" [6] is the outlier that makes the axis explicit: more than **1,400 real freelance software-engineering tasks from Upwork, valued at roughly $1 million USD in actual historical payouts**, ranging from ~$50 bug fixes to $32,000 feature implementations. Independent implementation tasks are graded with end-to-end tests triple-verified by experienced software engineers; managerial tasks — choose which of two competing technical proposals to implement — are assessed against the original hired engineering manager's choice. The metric is not "how many tasks" but "how much verified engineering value per unit of cost," and the difference is not cosmetic: a harness that clears 40 easy $50 fixes and one that clears 27 hard $32,000 features have the same binary count and a thousand-fold difference in value, and only the value-weighted scoreboard sees it. The task's provenance is stated scrupulously because it is frequently mis-attributed: the four authors are confirmed; the venue claim "ICML 2025" is **not confirmable from the abstract and is flagged, not asserted**. The axis it opens is the one this harness should eventually maintain as an internal *engineering value score* — verified engineering work divided by LLM dollars plus compute plus time — computed over the real tasks the system already completes, not as the only metric but as the counterweight that keeps resolve rate from being mistaken for "is Piper becoming a better engineer?" Chan, Chowdhury, Jaffe, Aung, Sherburn, Mays, Starace, Liu, Maksin, Patwardhan, Weng, and Mądry's "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering" [12] contributes the process axis: 75 Kaggle-style ML-engineering competitions as agent tasks, where the agent must analyze data, engineer features, train models, and submit, scored by **Kaggle's own grader medal thresholds** — an objective percentile ranking on held-out test sets, not a code review. The design's significance for the component is that it measures *sustained engineering process* — iteration, evaluation, submission discipline — a construct the code-generation benchmarks never touch, and that it is scored objectively because the graders were executable all along. The scoreboard's ambition, visible across this subsection, is the one stated in the thesis: correctness is one axis of an objective, executable evaluation, and *performance, resource use, patch size, regressions, robustness, process, and economic value* are the others — and all of them, without exception, are read off the machine rather than off a model's opinion of the machine's work. ### 5.5 Cross-benchmark, cost-aware comparison: HAL and TheAgentCompany The comparison discipline's final instruments are the ones that prevent the scoreboard from being gamed by *benchmark selection*. Kapoor, Stroebl, Kirgis, Nadgir, Siegel, Wei, Xue, Chen, Chen, Utpala, Ndzomga, Oruganty, Luskin, Liu, Yu, Arora, Hahm, Trivedi, Sun, Lee, Jin, Mai, Zhou, Zhu, Bommasani, Kang, Song, Henderson, Su, Liang, and Narayanan's "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation" [21] — the year is **2025** (October), not 2026, a correction this paper records because the citation circulates both ways — standardizes evaluation across a three-dimensional space of models, scaffolds, and benchmarks, validating its harness over 21,730 agent rollouts across 9 models and 9 benchmarks at a total cost of about $40,000, and its findings are the cost-aware discipline's empirical proof: higher reasoning effort *reduced* accuracy in the majority of runs, and LLM-aided log inspection surfaced previously unreported behaviors such as an agent searching for the benchmark on Hugging Face instead of solving the task. That last observation is the section's closing note in miniature: the agent was optimizing the *measure*, not the *construct*, and only a scoreboard that inspects trajectories — that reads the behavior, not just the score — catches the difference. Xu, Song, Li, Tang, Jain, Bao, Wang, Zhou, Guo, Cao, Yang, Lu, Martin, Su, Maben, Mehta, Chi, Jang, Xie, Zhou, and Neubig's "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks" [23] (NeurIPS 2025) completes the axis with the *consequential-real-world* construct: a self-contained simulated software company in which the agent browses the web, writes code, runs programs, and communicates with simulated coworkers to complete tasks a digital worker would actually perform, with the strongest baseline agent completing 30% of tasks autonomously. It is the benchmark that most directly operationalizes the harness's stated purpose — a person operating online — and its 30% ceiling is the honest calibration of how far from saturation the *real* construct remains, however saturated the toy constructs have become. --- ## 6. The audit of the instrument itself Every prior section trusted the instrument; this section withdraws that trust. A benchmark is a measure, and a measure that becomes consequential becomes a *target*, and a target is corrupted by exactly the pressure its consequence creates. The discipline of this section is that the instrument itself is the object of standing, mechanical audit — because the component's deepest claim is not that its benchmarks are right, but that they are *checkable*, and that a score is evidence only to the degree the benchmark still measures the construct it claims. ### 6.1 Campbell's Law and the corruption of the measure The governing law is not from the machine-learning literature at all, and it is stated here because it explains, in advance, everything the audits of §6.3 found. Donald T. Campbell's "Assessing the Impact of Planned Social Change" [8] — *Evaluation and Program Planning* 2:67–90, 1979, DOI 10.1016/0149-7189(79)90048-x — states the law: **the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.** Campbell's own examples were crime statistics warped by policing incentives and test scores warped by teaching-to-the-test — the indicator, once consequential, reshapes the behavior it measures. Benchmark contamination, training-on-test, judge-gaming, and eval-set-tuned harnesses are not accidents; they are Campbell's *corruption pressures*, appearing on schedule the moment a benchmark score became funding- and release-relevant. Goodhart's law (Goodhart 1975 [24]) is the same law in policy register, and Strathern's 1997 reformulation [25] — "when a measure becomes a target, it ceases to be a good measure" — is the one-line form the component should keep in front of itself at all times. The operational conclusion is not moral but structural: benchmark integrity is a *defense budget*, not a virtue — canaries, rotation, audits, and held-out evals exist because the pressure is structural, not because anyone cheated on purpose. A scoreboard without a standing audit is not an honest instrument; it is a measure that has not yet noticed it has become a target. ### 6.2 Task validity and outcome validity: the Agentic Benchmark Checklist The audit has a methodology, and it is Zhu, Jin, Pruksachatkun, Zhang, Liu, Cui, Kapoor, Longpre, Meng, Weiss, Barez, Gupta, Dhamala, Merizian, Giulianelli, Coppock, Ududec, Sekhon, Steinhardt, Kellermann, Schwettmann, Zaharia, Stoica, Liang, and Kang's "Establishing Best Practices for Building Rigorous Agentic Benchmarks" [24] — the paper colloquially known as "AI Agent Benchmarks Are Broken." It defines two properties an agentic benchmark must satisfy, and both are stated in the negative-space form that makes them auditable. **Task validity**: the task can be solved only by the intended capability, and not by an artifact of the setup. **Outcome validity**: the scorer accepts *exactly* the correct solutions, and not trivial or wrong ones. The paper's finding is that most widely-used agentic benchmarks fail at least one of the two — ranging from tests that pass on trivial or wrong outputs to tasks solvable by exploiting the evaluator — and its practical artifact is the **Agentic Benchmark Checklist**, the standing questionnaire this component adopts as its instrument-audit protocol. The deep point is the one this paper has been circling since §1: *execution-based is a necessary condition for objectivity, not a sufficient one.* A test-based score with poor outcome validity is still an LLM-judge in disguise, because the agent learns to satisfy whatever the checker *actually* checks, and a checker that checks the wrong thing produces a score that is objective and wrong. The two properties are the component's definition of "the benchmark still measures the construct," and every audit in §6.3 is a finding that one or the other property failed. ### 6.3 The audits that found the referee wanting: SWE-bench+, The SWE-Bench Illusion, SWE-ABS The audits are not hypothetical; they are the field's most consequential recent results, and each one found a specific, measurable defect in the instrument the field had been navigating by. Aleithan, Xue, Mohajer, Nnorom, Uddin, and Wang's "SWE-Bench+: Enhanced Coding Benchmark for LLMs" [5] audited the SWE-bench dataset itself and found three defects, and the framing of the defects matters because the paper is frequently mis-described: it is **not** an "empty-patch audit." The finding is a *solution-leakage* audit plus a *weak-test* audit plus a *data-leakage* audit: **32.67%** of the successful patches involved *solution leakage* (the solution was directly provided in the issue report or comments); **31.08%** of the passed patches were *weak-test patches* (the tests were inadequate to verify correctness); and **over 94%** of the issues predated model knowledge cutoffs (*data leakage*). Filtering out the problematic instances dropped the top system's resolution rate from **12.47% to 3.97%** — a three-quarters collapse, and the single clearest number in the literature for how much of a headline score can be instrument artifact rather than capability. Liang, Garg, and Zilouchian Moghaddam's "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason" [6] supplied the *memorization* measurement with two diagnostic probes: models achieve up to **76% accuracy identifying the buggy file path from the issue description alone** — without access to the repository structure — a number that drops to 53% on repositories *not* in SWE-bench, and the function-reproduction probe shows verbatim similarity up to 35% consecutive 5-gram accuracy on SWE-bench Verified/Full versus 18% on other benchmarks. The conclusion the paper draws — that performance gains on SWE-bench Verified may be partly driven by memorization rather than problem-solving — is the contamination decay curve of §4.1 measured directly, and it is the reason the harness's holdout/live set is the only number that may be called generalization. Yu, Cao, Zhang, Lin, Xu, Zhong, Xu, Wang, Cao, Cheung, He, and Briand's "SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark" [7] closes the loop from the *test-strength* direction: it strengthens the tests themselves and re-measures, showing that resolution rates collapse when the tests are made adequate — the outcome-validity failure of §6.2, demonstrated rather than asserted. The three audits, read together, are the component's proof that its own sixth section is not paranoia but a measured, recurring fact of the field: the instrument *decays*, and the decay is discovered only at the audit, and the audit is therefore a standing activity with deliverables, not a one-time apology. ### 6.4 What this paper deliberately does not cite as a paper One prominent member of this lineage is cited in prose but deliberately *not* listed as a numbered academic reference, and the exclusion is itself a measurement discipline. **SWE-bench Verified** is not a paper: it is an OpenAI blog post and a repository release of a 500-instance, human-validated subset of SWE-bench, in which professional developers reviewed each instance for task clarity, test soundness, and spec quality and discarded the ones that failed. Its significance is real — it became the field's de facto headline metric within months, and its meta-lesson is this component's charter, that *the referee needed refereeing* — but its status as a non-paper artifact is exactly the kind of provenance distinction the harness's own rules require, and it is cited here through its parent, SWE-bench [1], not as a reference in its own right. The same discipline governs the rest of the list: software releases, leaderboards, and model cards are treated as candidate components to be *evaluated*, never as sources of truth, and "it was published" is never confused with "it was audited." The final piece of the audit discipline is the one Cronbach and Meehl [18] formalized and the whole component inherits: a score is evidence *of a construct*, and the audit's question — what construct, and does the instrument still sample it — is the question that keeps the scoreboard honest long after the individual numbers have all been superseded. --- ## 7. Conclusion This paper specified the twenty-seventh and final component of the coding harness: benchmarking and objective scoring. It began from the inversion that the one question the harness cannot answer by asking a model is the question *is the model better now?* — because the model is the subject of that question, and a subject that grades its own exam produces a fluent and worthless number. It built the component on the founding contract of execution-based evaluation (§2): the executed verdict as the unit of evidence, established by HumanEval, MBPP, CodeXGLUE, and FixEval, and generalized to the whole repository by SWE-bench's two-sided FAIL_TO_PASS + PASS_TO_PASS gate. It decomposed the score (§3): RepoBench's three-way split and SWE-agent's harness-as-variable result together demand a hierarchical scorecard in which every stage of the coding loop has its own mechanical metric, so a bottleneck is identifiable independently of end-to-end success and a gain is attributable to the change that earned it. It armored the instrument against contamination (§4): LiveCodeBench's timestamp gating and EvoEval's evolved variants measured the decay curve, SWE-bench-Live kept the tasks fresh, SWE-bench Pro kept the repositories unseen, and the three-benchmark-set discipline reserved genuinely unseen tasks for the only number that may be called generalization. It disciplined the comparison (§5): freeze everything and change one thing, repeat to estimate variance, and record tokens, calls, CPU, wall time, and dollars alongside correctness, with SWE-Lancer and MLE-bench supplying the value axis and HAL and TheAgentCompany the cross-benchmark and consequential-real-world axes. And it audited the instrument itself (§6): Campbell's Law and Strathern's reformulation explain in advance what SWE-bench+, The SWE-Bench Illusion, and SWE-ABS found in fact — the measure decays, and the audit is a standing activity, not a one-time apology. **Benchmarking should be executable, decomposed, contamination-resistant, cost-aware, and causally useful: grade repository behavior through deterministic tests and measurable system outcomes rather than model opinion; maintain component-level metrics for every stage of the coding loop so bottlenecks can be identified independently of end-to-end success; evaluate across languages, repositories, task types, complexity levels, and engineering objectives; reserve genuinely unseen or continuously refreshed tasks for generalization measurement; hold model, budget, environment, and sampling parameters fixed when A/B-testing harness changes; repeat stochastic evaluations to estimate variance; record tokens, calls, CPU, wall time, and dollar cost alongside correctness; and require every new subsystem to demonstrate measurable benefit under controlled ablation before it earns permanent complexity.** **Nothing gets promoted because it feels smarter. A component survives only if Piper measurably performs better when it exists than when it does not.** The thesis sentence is the specification; the Piper rule is the sentence that makes it legible to a human; and the structural invariant that governs the whole design is the one stated in the introduction and restated here: > **the L column contains at most one entry.** The measurement/grading plane — test execution, mechanical scoring, the pinned harness, variance estimation, the A/B comparison — is deterministic software on CPU [D] with **zero learned entries**, because the model is the thing being *measured*, never the thing doing the measuring; "another model says it looks right" is the null hypothesis, not the method, and a grader that contained a learned model would inherit every bias the model under test was selected for. The model [L] is the *subject* of the score, outside the scoring path: it produces the candidates, and the machine produces the verdicts, and the two columns touch at exactly one point, which is the handoff of a patch to a runner and a verdict back. **An empty L column is a permitted, correct state**, and for the scoring path it is the only correct state — the same invariant that kept the model out of the security enforcement path, because it was the thing being *steered*, now keeps it out of the scoring path, because it is the thing being *scored*. The two exclusions are one exclusion, applied at the two ends of the loop: the model is never the judge of its own behavior. And this is the self-referential fact that makes this the final component rather than merely the last one. This component closes the outermost loop: the fifteen-plus prior components optimize the harness, and this one is how anyone knows whether the optimization was real. It applies the program's own deepest rule — *runtime is the only witness* — to the program itself. The same principle that makes executed test verdicts evidence against capability claims is the one that refuses to trust Piper's own feeling of improvement; the same requirement that a patch survive its tests is the requirement that a component survive its ablation. The benchmark is the scoreboard that does not read essays — and the first essay it refuses to read is Piper's own account of how much smarter her latest change made her. Component 1 began the loop with a model that could, at best, propose; Component 27 ends it with the machine that decides whether the proposal was any good, and it decides the way it has decided everything from the first executed test onward: not by asking, but by running. --- ## References [1] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" In *The Twelfth International Conference on Learning Representations* (ICLR 2024). arXiv:2310.06770. (2,294 issues from 12 Python repositories; two-sided grading — a patch is accepted only if all FAIL_TO_PASS tests pass and all PASS_TO_PASS tests are preserved; Claude 2 solved 1.96% at launch; SWE-Llama solved only the simplest issues.) [2] Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?" arXiv:2509.16941. (1,865 problems from 41 actively maintained repositories — a public set of 11 repositories, a held-out set of 12, and a commercial set of 18; long-horizon, multi-file tasks; first author Xiang Deng confirmed. The venue claim "ICLR 2026," the organization "Scale AI," and the exact frontier-drop figure are NOT confirmable from the abstract and are flagged here, not asserted.) [3] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. "SWE-bench Goes Live!" In *Advances in Neural Information Processing Systems 38 (NeurIPS 2025)*. arXiv:2505.23419. (The benchmark is SWE-bench-Live; the paper is titled "SWE-bench Goes Live!". 1,319 tasks from 93 repositories; automated curation for continuous refresh; a dedicated Docker image per task; leaderboard submissions include agent trajectories.) [4] Muhammad Shihab Rashid, Christian Bock, Yuan Zhuang, Alexander Buchholz, Tim Esler, Simon Valentin, Luca Franceschi, Martin Wistuba, Prabhu Teja Sivaprasad, Woo Jung Kim, Anoop Deoras, Giovanni Zappella, and Laurent Callot. "SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents." arXiv:2504.08703. (2,110 instances from 21 repositories across Java, JavaScript, TypeScript, and Python; bug fixes, feature additions, and refactorings; a 500-instance stratified subset; syntax-tree-structure metrics beyond binary resolve rate.) [5] Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. "SWE-Bench+: Enhanced Coding Benchmark for LLMs." arXiv:2410.06992. (The audit is solution leakage (32.67%) plus weak-test patches (31.08%) plus data leakage (over 94% of issues pre-cutoff); filtering the problematic instances dropped the top system's resolution rate from 12.47% to 3.97%. NOT an "empty-patch audit.") [6] Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?" arXiv:2502.12115. (Over 1,400 freelance software-engineering tasks from Upwork valued at roughly $1 million USD; tasks from ~$50 bug fixes to $32,000 feature implementations; end-to-end tests triple-verified by experienced engineers; managerial tasks scored against the original engineering manager's choice. The venue claim "ICML 2025" is NOT confirmable from the abstract and is flagged here, not asserted.) [7] Mike A. Merrill and eighty-four co-authors. "Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces." arXiv:2601.11868. (Eighty-five authors. The maintained version is Terminal-Bench 2.0, 89 tasks, each with a unique environment, a human-written solution, and comprehensive verification tests; success is scored on the environment state, not the diff; the 2.1 revision changed 28 of 89 tasks — 31% — to fix specification-verification mismatches, resource mismatches, and drift.) [8] Donald T. Campbell. "Assessing the Impact of Planned Social Change." *Evaluation and Program Planning*, Vol. 2, pp. 67–90, 1979. DOI: 10.1016/0149-7189(79)90048-x. (Campbell's Law: the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.) [9] Chunqiu Steven Xia, Yinlin Deng, and Lingming Zhang. "Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLM." arXiv:2403.19114, 2024. (828 problems across five semantic-altering and two semantic-preserving benchmarks; average 39.4% performance drop over 51 LLMs, ranging from 19.6% to 47.7%; drastic ranking changes. The identifier is 2403.19114 — not 2407.01461, which is a different, query-refinement paper.) [10] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code." arXiv:2403.07974, 2024. (Timestamp-gated contamination control: fresh problems from LeetCode, AtCoder, and Codeforces, scored only on problems post-dating a model's training cutoff.) [11] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, Binyuan Hui, Niklas Muennighoff, David Lo, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro Von Werra. "BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions." arXiv:2406.15877, 2024. (1,140 practical tasks with diverse library calls, complex multi-step instructions, and rich I/O; execution-based sandbox scoring with human-verified test suites.) [12] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering." arXiv:2410.07095, 2024. (OpenAI. 75 Kaggle competitions; scored by Kaggle's own medal thresholds — objective percentile ranking on held-out test sets.) [13] Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. "Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving." In *Advances in Neural Information Processing Systems 38 (NeurIPS 2025)*. arXiv:2504.02605. (Seven-language extension — Java, TypeScript, JavaScript, Go, Rust, C, and C++; 1,632 high-quality instances annotated from 2,456 candidates by 68 expert annotators.) [14] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, the operator Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. "Evaluating Large Language Models Trained on Code." arXiv:2107.03374, 2021. (Introduces Codex and HumanEval; pass@k functional-correctness metric; 28.8% single-sample, 70.2% with 100 samples per problem.) [15] Tianyang Liu, Canwen Xu, and Julian McAuley. "RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems." In *The Twelfth International Conference on Learning Representations* (ICLR 2024). arXiv:2306.03091. (Three settings — RepoBench-R retrieval, RepoBench-C completion, RepoBench-P pipeline — scoring retrieval, completion, and the full pipeline separately.) [16] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. "Program Synthesis with Large Language Models." arXiv:2108.07732, 2021. (MBPP — 974 entry-level programming tasks — and MathQA-Python; few-shot and fine-tuning regimes.) [17] Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. "CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation." arXiv:2102.04664, 2021. (Ten tasks across fourteen datasets; evaluation as infrastructure.) [18] Lee J. Cronbach and Paul E. Meehl. "Construct Validity in Psychological Tests." *Psychological Bulletin*, Vol. 52, No. 4, pp. 281–302, 1955. DOI: 10.1037/h0040957. (Construct validity — the discipline of what a score is evidence of.) [19] Franck Ndzomga. "Efficient Benchmarking of AI Agents." arXiv:2603.23749. (Single author. Across eight benchmarks, 33 agent scaffolds, and 70+ model configurations, rank-order prediction stays stable while absolute-score prediction degrades; a 30–70% mid-range difficulty filter reduces evaluation tasks by 44–70% while maintaining rank fidelity.) [20] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. "SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering." In *Advances in Neural Information Processing Systems 37 (NeurIPS 2024)*. arXiv:2405.15793. (The harness is a first-order variable: the agent-computer interface moves resolution by multiple points at fixed model; 12.5% on SWE-bench, 87.7% on HumanEvalFix.) [21] Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, and Arvind Narayanan. "Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation." arXiv:2510.11977, 2025. (The year is 2025, not 2026. 21,730 rollouts across 9 models and 9 benchmarks at ~$40,000; higher reasoning effort reduced accuracy in the majority of runs.) [22] Md Mahim Anjum Haque, Wasi Uddin Ahmad, Ismini Lourentzou, and Chris Brown. "FixEval: Execution-based Evaluation of Program Fixes for Programming Problems." arXiv:2206.07796, 2022. (Raw submission pool of 712K Java and 3.28M Python programs; curated splits smaller, on the order of 245K Java and 1,111K Python; match-based metrics mispredict correctness, execution-based scoring does not.) [23] Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks." In *Advances in Neural Information Processing Systems 38 (NeurIPS 2025)*. arXiv:2412.14161. (A self-contained simulated software company; the strongest baseline agent completed 30% of tasks autonomously.) [24] Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun, Andy Zhang, Shu Liu, Sasha Cui, Sayash Kapoor, Shayne Longpre, Kevin Meng, Rebecca Weiss, Fazl Barez, Rahul Gupta, Jwala Dhamala, Jacob Merizian, Mario Giulianelli, Harry Coppock, Cozmin Ududec, Jasjeet Sekhon, Jacob Steinhardt, Antony Kellermann, Sarah Schwettmann, Matei Zaharia, Ion Stoica, Percy Liang, and Daniel Kang. "Establishing Best Practices for Building Rigorous Agentic Benchmarks." arXiv:2507.02825. (Colloquially "AI Agent Benchmarks Are Broken." Task validity and outcome validity; the Agentic Benchmark Checklist; most widely-used agentic benchmarks fail at least one of the two properties.) [25] Charles A. E. Goodhart. "Problems of Monetary Management: The UK Experience." In *Papers in Monetary Economics*, Reserve Bank of Australia, 1975. (Goodhart's Law: any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.) [26] Marilyn Strathern. "'Improving ratings': audit in the British University system." *European Review*, Vol. 5, No. 3, pp. 305–321, 1997. (The reformulation of Goodhart's Law: "when a measure becomes a target, it ceases to be a good measure.") [27] Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. "The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason." arXiv:2506.12286. (File-path identification from issue descriptions alone reaches up to 76% accuracy on SWE-bench but 53% on non-SWE-bench repositories; verbatim similarity up to 35% consecutive 5-gram accuracy on SWE-bench Verified/Full versus 18% on other benchmarks.) [28] Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, and Lionel Briand. "SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark." arXiv:2603.00520. (Resolution rates collapse when the tests are strengthened; the top system's 78.80% is shown to be inflated.) [29] Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia, Max Hasin, Sami Jawhar, Megan Kinniment, Nate Rush, Sydney Von Arx, Ryan Bloom, Thomas Broadley, Haoxing Du, Brian Goodrich, Nikola Jurkovic, Luke Harold Miles, Seraphina Nix, Tao Lin, Chris Painter, Neev Parikh, David Rein, Lucas Jun Koba Sato, Hjalmar Wijk, Daniel M. Ziegler, Elizabeth Barnes, and Lawrence Chan. "Measuring AI Ability to Complete Long Software Tasks." arXiv:2503.14499. (METR. The 50% task-completion time horizon — the length of task, measured in skilled-human time, completed with 50% reliability; timed on RE-Bench, HCAST, and 66 novel tasks; the horizon has been doubling roughly every seven months since 2019.) [30] Xinyi He, Qian Liu, Mingzhe Du, Lin Yan, Zhijie Fan, Yiming Huang, Yin Zheng, Zejian Yuan, and Zejun Ma. "SWE-Perf: Can Language Models Optimize Code Performance on Real-World Repositories?" arXiv:2507.12415. (140 repository-level performance-optimization tasks derived from performance-improving pull requests; each instance includes the codebase, target functions, performance-related tests, expert-authored patches, and an executable environment; a candidate must make the code faster without introducing correctness failures; accepted at ICML 2026.)