Summary / 11. Code execution / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# The Runtime Is the Only Witness: Sandboxed Deterministic Execution as the Observation Layer of a Weak-Model Coding Harness *Component 11 of 27 — Coding-Harness Program. This document specifies the code-execution stage: the substrate that turns the repository from a thing the harness merely reads and edits into a thing that answers back. It sits after component 10 (editing, the write path), which lands the model's patch on disk, and before the components that verify and repair, which judge whether that patch is good. Where generation writes and editing lands, execution is the OBSERVATION layer — the one channel through which the repository itself talks back, mechanically, without the intervention of any model's opinion. This paper translates the program's thesis into the execution-substrate build contract: a deterministic, isolated evidence-generation service in which the runtime, not the model, is the only witness to what a candidate program actually does.* --- ## Abstract Code execution is the observation layer of a weak-model coding harness: the single stage at which the repository stops being a static text the harness reads and becomes a machine that answers back. This paper argues that execution should be a deterministic, isolated evidence-generation service rather than an opportunistic shell invocation — every candidate is run rather than mentally simulated by the model, commands execute under reproducible resource and security boundaries, raw runtime behavior is normalized into compact provenance-rich observations, execution evidence is used both to diagnose failures and to discriminate among candidate implementations, and bounded diagnostic probes may be generated where existing tests are insufficient, never mistaken for authoritative specifications. The paper reviews the execution-centered literature — the generate→execute→observe→diagnose→revise loop of Self-Debug [1], the code-is-action framing of InterCode [2], SWE-agent's execution-as-connective-tissue ACI [3], CodeAct's execution-promoted-to-action-space [4], AlphaCode's execution-as-arbiter-over-scale [5], the trust-transfer outlier PAL [6], and the training-and-selection side of RLEF and LEVER [7][8] — together with honourable mentions from the execution-agent and benchmark lineage and the classical roots beneath them. It derives seven design principles, then specifies a pipeline that is deterministic end to end (D) with exactly one learned entry (L): four action modes, an isolation policy, raw execution, evidence parsing, normalization into a failure-class vocabulary, behavioural caching, and structured observation assembly, with diagnostic-probe generation as the single weak-LLM call. It closes with a recursive self-improvement protocol and a measurement plan whose honest instruments — pass@k, FAIL_TO_PASS and PASS_TO_PASS, the empty-success guard, determinism checks, and anti-gaming re-verification — exist to keep the harness from fooling itself about what the runtime actually witnessed. ## 1. Introduction The coding-harness program decomposes software engineering into a sequence of questions, and components 01 through 10 built the machinery that reads the repository, reasons about it, decomposes a request, constructs a decision window, generates a patch, and edits it onto disk. Through all of that the repository is an object the harness acts *on*; the model forms beliefs about it, and those beliefs are the only currency. Component 11 is where that stops being true. This paper specifies code execution, the substrate that turns the repository from a text the harness reads into a machine that answers back — mechanically, reproducibly, and without deference to any model's guess about what would happen. Everything upstream of this component asks the model what it *thinks*; execution is the single stage that asks the repository what it *does*. Editing is the write path; execution is the observation path that makes the write meaningful, because a patch that is never run has not been shown to do anything at all. The central fact is that a weak model's beliefs about what code does are cheap, fast, and unreliable, whereas the runtime's answer is expensive-but-true — and the harness sits between them. A model can confidently assert that a function returns a value, that a query is well-formed, that a refactor preserved behavior, and be wrong in ways that no amount of additional reasoning will expose, because the failure lives in the substrate — a type mismatch, an off-by-one, a dependency that does not resolve — not in the model's chain of thought. The execution stage exists to replace that assertion with a measurement: the candidate is run, and what actually happens is recorded. This is not an optimization of the model's reasoning; it is a replacement of the question being asked. The distinction is the one InterCode draws when it frames code as the *action* and execution feedback as the *observation* of a sequential decision process [2], and it is why this paper treats execution as its own component rather than a sub-case of verification: verification judges a finished artifact, while execution is the standing observation channel through which every intermediate candidate, not just the final one, is witnessed. The number that makes this stance urgent is the gap between reasoning and running. PAL demonstrated the trust-transfer principle at the extreme: by offloading computation to a Python interpreter and taking the interpreter's printed answer as the answer — never the model's own stated answer — program-aided models beat pure chain-of-thought reasoning by 15% absolute on GSM8K over a PaLM-540B baseline [6]. The model was not made smarter; it was made honest about where the authority for arithmetic should live. Self-Debug showed the same lesson in the repair loop, where feeding execution feedback back into the model reliably improved pass rates by up to roughly twelve points where tests existed [1]. And SWE-agent made the architectural claim that the interface through which the model observes execution results is itself a first-class capability lever, not a detail [3]. The common thread is that execution evidence is not a nice-to-have confirmation of a model's reasoning; it is a different, higher-authority channel that the reasoning must be checked against, and a weak model — which reasons less reliably to begin with — needs that channel *more*, not less. The thesis follows. **Code execution should be a deterministic, isolated evidence-generation service: every candidate is run rather than mentally simulated; commands execute under reproducible resource and security boundaries; raw runtime behavior is normalized into compact, provenance-rich observations; execution evidence is used both to diagnose failures and to discriminate among candidate implementations; and where existing tests are insufficient, bounded diagnostic probes may be generated to expose behavioral differences without being mistaken for authoritative specifications.** And it carries a principle that governs this component's every design choice, stated plainly: **Whenever reality can answer a coding question cheaply, the harness should ask reality before asking the LLM to reason about it.** A model's prediction about what code does is a hypothesis; the runtime's report is a measurement; the harness should treat the former as provisional and the latter as load-bearing, and it should route a question to the runtime the moment a cheap execution can settle it. The paper proceeds as follows. Section 2 reviews the execution literature in nine parts — the anchor, the action-observation framing, the ACI result, the action-space promotion, the scale-based arbiter, the trust-transfer outlier, the training-and-selection side, honourable mentions, and classical roots. Section 3 derives seven design principles. Section 4 specifies the architecture, labeling every node deterministic (D) or learned (L) and stating the invariant that the L column holds exactly one entry. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into eight modern lines — an anchor that closes the loop, a code-is-action framing, an ACI-connective-tissue result, an action-space promotion, an arbiter-over-scale result, a trust-transfer outlier, a training-and-selection pair, and a set of execution-agent honourable mentions — plus the classical roots beneath all of them. Each line answers one question the execution stage must resolve: can execution feedback drive repair, what should the agent's action and observation literally be, where does execution sit in the interface, what happens when execution becomes the action space, how does execution arbitrate at scale, what is the strongest form of trusting the interpreter, how does execution feedback improve the models themselves, and what do the recent execution tools and benchmarks add and caution. ### 2.1 The anchor that closes the loop: Self-Debug **Self-Debug** [1] is the anchor for this component because it is the paper that most completely embodies the operation this stage exists to serve: the full generate→execute→observe→diagnose→revise loop, in which the model writes a candidate, the harness *runs* it, the failure is observed, a diagnosis is formed from that observation, and the model revises accordingly — execution feedback as the driver of an iterative repair cycle rather than a terminal pass/fail. Chen et al. show that execution feedback and self-explanation are COMPLEMENTARY, not substitutes: explaining the code and using the explanation to catch errors helps, but the decisive addition is feeding the actual runtime result back in, so the two mechanisms reinforce rather than duplicate [1]. The paper's most reusable framing is that the traceback is the reasoning prompt — the exception message and stack are not an error report to be forwarded but the raw material the model reasons over to localize the fault, so the harness's job is to deliver that traceback in the form the model can best use. The numbers anchor the effect where tests exist: up to roughly +12% improvement on tasks that have a test oracle (with a more modest +2–3% on Spider), confirming that execution feedback's value scales with the availability of something mechanical to check against [1]. A venue caveat belongs in the same breath: "ICLR 2024" is not confirmable in the arXiv record, so it should be treated as unconfirmed rather than asserted. The weak-model implication is the reason this stage is a component: a weak model reasons less reliably, so the corrective channel of real runtime feedback is a larger fraction of its total competence than it is for a strong model. ### 2.2 Code is the action, execution is the observation: InterCode **InterCode** [2] supplies the framing that this component's architecture adopts literally, and it appears here with a corrected identifier — the arXiv ID is **2306.14898**, not the 2303.07545 the source material claimed. Its thesis is that code is the *action* and execution feedback is the *observation* of a sequential-decision-making process: an agent interacting with a coding task is best modelled not as a single generation but as a turn-taking loop in which the environment executes the agent's code action and returns the resulting observation, so reinforcement-learning and interactive formulations apply [2]. Two properties make it load-bearing for this component. First, its environments are *self-contained* — Docker and related isolation so that execution is safe and reproducible, the direct ancestor of this architecture's isolation policy (P3). Second, its headline result is that multi-turn interaction beats single-turn generation by double digits — GPT-4's SQL-task success rising from 9.1% to 73.7% once it can see execution feedback and act on it — a dramatic confirmation that the observation channel, not the model, is what unlocks complex tasks [2]. The venue is confirmed as NeurIPS 2023 Datasets & Benchmarks. The weak-model lesson is the clearest in the section: if even GPT-4 needs execution feedback to go from 9% to 74% on SQL, a weak model without that channel is not merely underperforming — it is structurally incapable of the task at all. ### 2.3 Execution as the ACI's connective tissue: SWE-agent **SWE-agent** [3] contributes the architectural placement: execution results are the connective tissue of the agent-computer interface — the recurring observation that stitches the agent's actions into a coherent trajectory. Two of its design principles belong directly to this component. First, the agent's model of the world is built from *observable feedback*, not a modelled world state: the harness does not try to represent what the repository is; it shows the agent what commands returned, and the agent's state is whatever the last set of observations imply. Second, trajectory endpoints are executions — each turn ends in a command that ran, so the verdict on each action is mechanical, produced by the runtime rather than asserted by the harness. Two of its tooling decisions are equally load-bearing: deterministic *output truncation* of command results so that a runaway dump cannot flood the observation, and a *"no output produced"* sentinel so that an empty result is represented explicitly rather than as ambiguous silence — both carried into this architecture's normalization stage [3]. A venue caveat belongs here too: "NeurIPS 2024" is *not* in the arXiv record, which carries only a code-and-data URL, so the venue is unconfirmed. The weak-model implication is that execution results must be delivered in a form a weak model can actually consume — bounded, parsed, deduplicated — because an untruncated noisy transcript is where a weak model's capacity to extract the signal collapses. ### 2.4 Execution promoted to the action space: CodeAct **CodeAct** [4] makes the strongest version of the claim this component rests on: execution is not merely a source of feedback but can be promoted to the ACTION SPACE itself — the agent's decisions *are* executable code, so every action is runnable and every observation is the result of running it. Wang et al. show that executable-code actions beat JSON and free-text actions on multi-step tool-use tasks by up to +20% in success while using up to 30% fewer actions, across 17 LLMs, on the API-Bank and M3ToolEval benchmarks [4]. The mechanism is that code is a richer action vocabulary than text: it can express loops, conditionals, and composed tool calls that a JSON schema cannot, and because it is executable, the environment verifies it rather than the model asserting its own success. The venue is confirmed as ICML 2024. The weak-model implication is subtle but important: a weak model is the *least* able to profit from free-text actions that require it to imagine their effect, so making actions concrete and executable — so the runtime settles what each one did — shifts work from the weakest part of the system to its most reliable part, exactly the direction this architecture pushes. ### 2.5 Execution as the arbiter over scale: AlphaCode **AlphaCode** [5] demonstrates execution in the role this architecture's discrimination principle (P6) will use at scale: not as a per-candidate diagnostic but as the *arbiter* that selects among an enormous population of candidates. Li et al. sample on the order of ~1M candidate programs and use execution to filter and organize them; the crucial algorithmic decision is that the survivors are clustered BY RUNTIME BEHAVIOUR, not by textual similarity — programs that produce the same outputs on the held-out sample are treated as one behavioural group, and a representative from each cluster is submitted, largest-cluster-first, so that diverse correct solutions are tried rather than many textual variants of the same behaviour [5]. The result on CodeContests is a 34.2% solve rate on the validation set (outperforming roughly 54.3% of human competitors by rank), achieved not by a smarter model but by execution-driven selection over scale. One claim must be explicitly avoided: the paper does *not* describe the mechanism as "result caching," so that vocabulary must not be attributed to it [5]. The venue is confirmed as Science 2022. The weak-model implication is that execution is what lets a weak generator compete with a strong one through volume: when generation is cheap and imperfect, a deterministic runtime filter can discard the duds and pick the behavioural survivors in a way no amount of reasoning by a weak model could. ### 2.6 The outlier and the strongest trust rule: PAL **PAL** (Program-Aided Language Models) [6] is the outlier in this review because it does not merely use execution feedback to correct or select; it transfers authority entirely. Its trust-transfer principle is stark: the answer is *whatever the interpreter prints*, never the model's stated answer — the model's job is to write the program that computes the answer, and the runtime's job is to compute it, so the two are cleanly separated and the model never gets to overrule the runtime [6]. The numbers make the point concrete: by offloading computation to an interpreter, PAL beats a PaLM-540B chain-of-thought baseline by 15% absolute on GSM8K, with +11.4% on date-understanding and +14.1% on state-tracking tasks — improvements that come purely from moving arithmetic and bookkeeping out of the model's head and into a deterministic executor [6]. The venue is confirmed as ICML 2023. The weak-model implication is the sharpest single rule in this component's literature: a weak model is exactly the model that should not be trusted to compute by reasoning, so the harness should route any computation a runtime can perform to the runtime, and treat the model's arithmetic output as a draft to be verified, never as the answer. ### 2.7 Training and selection side: RLEF and LEVER Two papers cover the side where execution feedback shapes the models themselves rather than a single run. **RLEF** (Reinforcement Learning with Execution Feedback) [7] puts execution in the loop of training: Gehring et al., with Meta affiliation verified clean, apply reinforcement learning in which the reward comes from executing the model's multi-turn code rather than from a static verifier, training Llama 3.1 8B and 70B on CodeContests with a reported ~10× reduction in the samples needed to reach a given performance — execution feedback as a training signal that makes each sample far more informative because the model learns from what its code actually did [7]. The venue is confirmed as ICML 2025. **LEVER** [8] covers the selection side: Ni et al. train a learned verifier that scores a candidate using THREE sources of evidence — the natural-language task, the candidate program, and the *execution result* of that program — and report 4.6–10.9% absolute gains with code-davinci-002 across four datasets, showing that execution outcome is a signal a selection model should be trained on jointly with the code and task, not used in isolation [8]. The venue is confirmed as ICML 2023. Together they close the loop this component only opens at inference time: if execution evidence is valuable enough to shape weights, it is valuable enough to shape every single candidate decision the harness makes. ### 2.8 Honourable mentions The recent execution lineage contributes tools and cautions without anchoring the component. **ExecutionAgent** — the actual paper title is *"You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects"* (the source's "ExecutionAgent" is the method name, not the title) — by Bouzenia and Pradel, ISSTA 2025, arXiv:2412.10133, runs the existing test suites of 33/50 projects across 14 programming languages, an operation this architecture treats as a whole action mode; note that EnvBench (arXiv:2503.14443, 994 repos) and Repo2Run (arXiv:2502.13681, 420 repos) are SEPARATE sibling benchmarks, not this paper's evaluation [9]. **Reflexion** (arXiv:2303.11366, Shinn et al., NeurIPS 2023) externalizes the model's self-critique into natural-language feedback derived from task and execution signals and feeds it back into the next attempt [10]. **Terminal-Bench** (arXiv:2601.11868, Merrill et al., roughly 83 authors) is a benchmark of 89 CLI tasks on which frontier agents score under 65%, with a version ambiguity to flag — the 2.x family exists and results should be reported against a pinned version [11]. **ImpossibleBench** (arXiv:2510.20270, Zhong/Raghunathan/Carlini) contributes the anti-gaming caution that will shape Section 6: it constructs impossible variants whose tests *contradict* the spec, and measures how often agents "pass" by gaming or editing the tests rather than solving the task [12]. **Rollout Infrastructure Tax** (arXiv:2607.01415, an ACM SoCC 2026 submission) documents up to 110× cold-start variation and a 1.8× worker-hour spread across execution substrates — but it measures substrate-side rollout latency ONLY and explicitly EXCLUDES model inference and reward computation, so the stronger "dominates over inference" claim from the mashup must be softened accordingly [13]. **Is Self-Repair a Silver Bullet?** (arXiv:2306.09896) supplies the essential caution that execution feedback only helps repair when the *prompted diagnosis is itself good* — feeding a runtime failure to a model that misdiagnoses it does not help, so the diagnosis step matters as much as the feedback channel [14]. ### 2.9 The classical roots The modern lines rest on six classical results, cited here for lineage rather than numbers. CodeT (arXiv:2207.10397, ICLR 2023) generates both code and its tests and uses the tests to filter and rank the code — the ancestor of the diagnostic-probe insight (the one L entry) and of execution-as-discrimination. HumanEval and the Codex pass@k evaluation (arXiv:2107.03374) established pass@k as the execution contract — the notion that a candidate is scored by how many test cases its actual execution passes, not by how plausible it reads. Execution-guided decoding (Gupta, ICSE 2021) uses actual program execution to guide the search over candidate generations, the ancestor of execution steering generation. Program of Thought (NeurIPS 2023) translates the chain-of-thought into executable code that runs to produce the reasoning — PAL's sibling in the trust-transfer family. Voyager (arXiv:2304.04043) maintains a skill library gated by execution success, so only verified skills are retained — execution-gated memory. And **LLMs Cannot Self-Correct Reasoning Yet** (Huang et al., ICLR 2024) is the classical root of the whole component's premise: without external feedback — and the cleanest external feedback is execution — models largely cannot correct their own errors by introspection alone. These six sit beneath the modern lines as the substrate the execution stage stands on. ## 3. Design Principles Seven principles follow from the review, each tied to the evidence above. **P1 — Reality over rhetoric.** Whenever reality can answer a coding question cheaply, ask reality before asking the LLM to reason about it; the runtime's answer is a measurement, the model's prediction is a hypothesis, and the harness should treat the hypothesis as provisional. PAL's trust-transfer rule is the warrant — the answer is whatever the interpreter prints, never the model's stated answer [6] — and InterCode's multi-turn doubling of GPT-4's SQL success (9.1%→73.7%) is the scale of what this principle buys [2]. A weak model is the least reliable reasoner, so the fraction of decisions that should be settled by execution rather than reasoning is *larger* for it, not smaller. **P2 — Execution is a typed deterministic service, not a generic terminal.** The harness does not open a shell and stream a transcript; it invokes a typed service that runs a candidate under a declared contract and returns structured observables — exit code, duration, stdout and stderr, exception, changed state, test results — not a wall of undifferentiated text. SWE-agent's principled output truncation and "no output produced" sentinel [3] are the warrant that representation of runtime output is a first-class decision: a transcript is what a human browses; a typed observation is what a weak model can consume. The service is deterministic: the same candidate, environment, and command yield the same observable, because only reproducible evidence is trustworthy evidence. **P3 — Sandbox everything.** Isolation is the precondition for trustworthy evidence: a candidate must run under reproducible resource and security boundaries — a CPU cap, a RAM cap, a filesystem boundary, network deny-by-default, a timeout — so that its observed behavior is its own and not an artifact of a hostile, leaky, or runaway environment. InterCode's Docker self-containment [2] and ExecutionAgent's safe test execution [9] are the warrants, and the trust requirement is absolute: if the execution environment can interfere with or be interfered by the host, the observation is not a clean measurement and cannot be the basis of repair or discrimination. This is the sandbox rule stated as an evidence rule, not merely a safety rule. **P4 — Normalize failure into a common evidence vocabulary.** Raw runtime failure is noisy — every tool and language reports errors differently — so the harness normalizes it into a common vocabulary of failure classes (compile, syntax, assertion, runtime, timeout, resource, type, dependency) each carrying a location, a message, an exit code, and a duration, so that the model and the harness reason about failures uniformly. Self-Debug's insight that the traceback is the reasoning prompt [1] is the warrant: the model reasons over the *normalized* failure, so the normalization must preserve everything the model needs to localize and repair while stripping the noise that would distract a weak model. A compile error and a timeout are different species with different remedies, and the vocabulary makes the species explicit. **P5 — A failed execution is information, not rejection.** A failure is not a terminal "no" on a candidate; it is a rich observation to be extracted, classified, and targeted — the harness records what failed, which failure class it belongs to, and what evidence points at the cause, then routes that to the repair step. Self-Debug's repair loop is the warrant: failures become the driver of revision, and the gain is up to ~+12% where tests exist [1]. The classification from P4 makes the targeting possible — an assertion failure in a specific test directs the model to the assertion; a timeout directs it to the loop. A candidate that fails is not wasted; it is the most informative object the harness owns. **P6 — Execution serves two masters: diagnosis AND discrimination.** The same execution evidence must drive two distinct uses: *diagnosis* of a failing candidate to repair it (Self-Debug [1]), and *discrimination* among many candidates to select and rank them (LEVER's learned verifier over execution results [8] and AlphaCode's clustering of survivors by runtime behaviour [5]). These are not the same operation — a diagnosis explains why one thing failed; a discrimination compares many things to pick the best — and the harness must support both, because a weak generator produces many candidates that need ranking as much as it produces one candidate that needs fixing. LEVER's 4.6–10.9% gains and AlphaCode's scale-based selection are the evidence that selection-by-execution is a separate, high-value capability [8][5]. **P7 — Never accept self-report; distinguish authoritative tests from generated probes.** The harness must re-verify claims rather than trust them — a candidate that reports success, or a harness state that reports "all tests pass," is only believed if the harness itself re-runs and confirms — and it must keep the categories of evidence separate: authoritative tests are ground truth that gate a candidate, while generated diagnostic probes are bounded hypotheses about behavior that must never be promoted to authoritative status on the model's say-so. ImpossibleBench's construction of tests that contradict the spec, and its measurement of agents passing by gaming or editing tests [12], is the anti-gaming warrant; CodeT's code-and-test generation [15] is the probe warrant. The harness decides what counts as authority, mechanically, never the model. ## 4. The Architecture The pipeline is a sequence of stages; every node is labeled D (deterministic software) or L (a learned model). The invariant that governs the whole is: *the L column contains exactly one entry — a single weak-LLM diagnostic-probe-generation call — and every upstream and downstream step is deterministic software on CPU.* The deterministic path is the default; the single learned call fires only when no adequate test oracle exists to judge a candidate. The concrete inputs are a candidate artifact (a patch, a snippet, a command), the environment declaration for it, and the set of existing tests; the output is a structured `ExecutionObservation` that diagnosis and discrimination stages consume. The flow, in the seven deterministic stages (a) through (g) plus the one learned entry (L): 1. **(a) Action classification [D].** The incoming request is classified deterministically into one of four execution modes by what it asks the runtime to establish: *compile/parse* (does the artifact build at all — syntax and type correctness without behavioral meaning), *targeted execution* (run a specific entry point or snippet and observe its outputs and side effects), *test execution* (run the existing suite or a subset and collect verdicts), and *diagnostic probe* (run a generated probe to expose behavioral differences — the only mode that may involve a model, via stage (L)). Classification is a table over request type and artifact shape, not an inference. 2. **(b) Isolation policy [D].** Every execution is wrapped in a reproducible, declared boundary per P3: a CPU cap, a RAM cap, a filesystem boundary restricting writes to a scratch root, network *deny-by-default* unless a specific execution explicitly opts in, and a hard timeout. The policy is recorded in the observation so the reader of the evidence knows under what constraints it was produced — a timeout under a low CPU cap is evidence about the candidate, not about the environment. Isolation is the precondition that makes the runtime's report trustworthy. 3. **(c) Execute [D].** The candidate runs under the declared policy. This stage is deliberately bare: it spawns the process, enforces the resource and time bounds, and captures the raw artifacts — stdout, stderr, exit code, wall-clock duration, and any raised exception — without interpreting them. If the timeout or a resource cap fires, that is recorded as the outcome; the process is killed deterministically, and the evidence reflects the kill, not a hang. 4. **(d) Evidence parsing [D].** The raw artifacts are parsed into structured fields: compile errors are extracted from the compiler/linter output with their locations and messages; runtime evidence (exceptions, stack frames, printed output) is captured; test verdicts are collected from the runner's structured or convention-based output. Parsing is deterministic per toolchain — each language and runner has a declared parser — so that the same raw output always yields the same structured evidence. 5. **(e) Normalization into a failure-class vocabulary [D].** The parsed evidence is classified into the common vocabulary of P4 — compile / syntax / assertion / runtime / timeout / resource / type / dependency — and each classified failure carries a *location* (file and line or symbol), a *message* (the human-and-model-readable error text), the *exit code*, and the *duration*. Normalization is what turns a heterogeneous wall of raw failure into the uniform, repair-grade observation a weak model can actually reason over, per Self-Debug's traceback-as-reasoning-prompt [1]. 6. **(f) Behavioural caching and deduplication [D].** Identical executions are recognized and skipped: the candidate's code is hashed together with its environment and the command, and if an identical run was already observed with the same boundaries, the prior observation is returned rather than the candidate being re-run. This is P2's determinism exploited for economy — identical code under identical constraints yields identical evidence, so re-running it would burn wall-clock time to rediscover a known fact. AlphaCode's clustering intuition applies here in miniature: near-identical candidates need not each pay a full execution. 7. **(L) Diagnostic-probe generation [L] — the single learned entry.** When no adequate oracle exists — the candidate has no test suite, or the existing tests do not distinguish the behavior the harness cares about — the model proposes *bounded diagnostic probes*: small programs or assertions designed to expose behavioral differences among candidates or to establish a property the existing tests leave open. This is CodeT's code-and-test insight [15] applied on demand. The probes are bounded (size, count, scope) and they are *gated*: a probe is executed and its result recorded as *probe evidence*, and it is never auto-accepted as an authoritative test. Authority — what gates a candidate — is reserved for the existing test suite; a generated probe is a hypothesis the harness holds loosely, and ImpossibleBench's anti-gaming lesson [12] is why the two categories can never be conflated. 8. **(g) Structured observation assembly [D].** All of the above is assembled into a single `ExecutionObservation` object — the schema this component's consumers read. Its fields are the *action* taken, the *environment* declaration (toolchain, CPU/RAM caps, filesystem boundary, network policy, timeout), the *cwd*, the *artifact_version* (a hash of the code that ran, so evidence is traceable to exactly what produced it), the *exit_code*, the *duration*, a *timed_out* flag, *stdout* and *stderr* (bounded and elided under P4-style hygiene), the *exception* if one occurred, a *changed_state* record of filesystem or persistent effects, the *test_results* (verdicts per test, with FAIL_TO_PASS and PASS_TO_PASS statuses where applicable), and *resource_usage* (peak CPU and RAM). The observation is the single currency of this component: diagnosis and discrimination stages consume it, and nothing about an execution reaches the model except through it. The target is explicit and checkable: the pipeline's deterministic stages (a)–(g) and the isolation, parsing, normalization, caching, and assembly that surround them are all software on CPU, and the L column holds exactly one entry — the diagnostic-probe-generation call that fires only when no adequate oracle exists. Any proposed addition that would put a language model elsewhere in the pipeline is a regression against P1–P7 and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of Section 6, on held-out repositories, at a defensible cost, or be rejected. The invariant is stated again so it can be enforced mechanically: **the L column contains exactly one entry.** ## 5. Recursive Self-Improvement Protocol Component 11 of 27 follows the program's uniform protocol: the executable specification is written to 100%, round 1 lands at approximately 80% of the specified behavior, and later rounds close the gap. Each round is A/B-gated against the honest metrics of Section 6 — never the end-to-end resolve rate alone, which would let execution quality mask repair quality. Because this component's deterministic stages (a)–(g) carry the load, their improvement is mostly direct engineering, and execution evidence feeds back to improve the harness itself in three concrete loops. First, *probe-generation calibration*: the one learned stage is measured by how often its generated probes actually discriminate behavior the authoritative tests did not, and by how often a probe is (correctly) *not* promoted to authority; over rounds the probe prompt is tuned against those rates so the model proposes fewer redundant and more discriminating probes. Second, *failure-class weighting*: the harness learns which failure classes actually matter for the repairs it performs — if assertion and type failures dominate real resolutions while timeouts rarely do, the observation schema and the diagnosis routing can prioritize the classes that predict a successful next action, tightening the link between the evidence vocabulary of P4 and downstream repair. Third, *observation-schema tightening*: the harness measures which fields of the `ExecutionObservation` the downstream stages actually consume, and a field whose removal does not degrade repair or discrimination on held-out runs is a candidate for removal — the schema should not grow on the intuition that more evidence helps, but shrink to the evidence that demonstrably does. Because every observation is versioned by `artifact_version` and diffable, a regression in what an execution records is visible in the assembled observation long before it surfaces as a failed resolve. ## 6. Measurement & Evaluation Code execution is measured *separately* and *honestly*, because a stage whose whole purpose is to witness reality can be defeated by the harness fooling itself about what it witnessed. The instruments are built to make the observation layer accountable to the runtime it observes. **Pass@k and FAIL_TO_PASS / PASS_TO_PASS as the execution contract.** The primary instrument is the standard execution contract inherited from HumanEval and Codex pass@k and from SWE-bench's FAIL_TO_PASS and PASS_TO_PASS: a candidate's score is how many test cases its *actual execution* passes, where FAIL_TO_PASS counts tests that were failing before a change and pass after it (the true signal of repair) and PASS_TO_PASS counts previously-passing tests that still pass (the guard against regression). These are mechanical, and they are the honest currency of both diagnosis and discrimination: they say what the runtime witnessed, not what the model or harness claims. **The empty-success trap.** A candidate that exits 0 having collected zero tests has not succeeded; it has been witnessed doing nothing. The instrument therefore flags any exit-0 run whose `test_results` is empty or whose claimed coverage is nil, and refuses to count it as evidence of correctness. InterCode's framing — the observation must actually observe something — and the anti-gaming concern make this guard mandatory: an execution that ran no tests proves the program ran, not that it is right. **Determinism guarantees.** Because P2 makes determinism a precondition for trustworthy evidence, the measurement suite checks it: dependencies are pinned, tests are seeded, and identical candidates under identical declared environments must produce identical observations. Any run that cannot be reproduced from its own `artifact_version` and `environment` declaration is flagged, because non-reproducible evidence cannot be the basis of repair or of a recursive-improvement decision. **Anti-gaming re-verification.** The harness never trusts a self-report, including its own: when a candidate or stage reports "all tests pass," the harness independently re-runs the reported-passing tests and confirms, and it applies a *diff-scope check* that the test files themselves were not edited as part of the candidate's change — ImpossibleBench's construction of spec-contradicting tests and its measurement of agents passing by editing tests [12] are the warrant that a pass achieved by tampering with the oracle is not a pass at all. Authoritative tests are treated as ground truth that gates a candidate and are themselves protected from mutation. **Failure-mode audit.** Beyond the positive instruments, the harness audits against the specific ways an execution layer can lie: *trusting the transcript* (believing a wall of output without parsing it into structured evidence), *empty success* (exit 0, no tests), *noisy untruncated dumps* (a runaway output flooding the observation and hiding the signal), *environment drift* (the declared environment no longer matching what actually ran), *flaky-test thrash* (a candidate rejected or accepted on a test that is not deterministic), *verdict-only feedback* (delivering only pass/fail without the location and message a repair needs), and *test tampering* (the oracle edited by the candidate). Each has a guard: structured parsing, the empty-success flag, output truncation with elision, environment pinning and re-verification, seeded deterministic tests, normalized failure fields, and the diff-scope re-verification above. Two honest caveats close the section. First, the numbers of Section 2 were largely obtained with strong models and at scale — AlphaCode's 34.2% needed ~1M samples [5], CodeAct's +20% was measured across strong LLMs [4], LEVER's gains used code-davinci-002 [8] — so a weak-model harness should not expect to reproduce them; its execution stage exists precisely so that weak *generation* is compensated by deterministic *selection*, and the honest instruments here — above all FAIL_TO_PASS/PASS_TO_PASS and the anti-gaming re-verification — are what will show whether that compensation is real. Second, all execution metrics must be reported on held-out and out-of-benchmark repositories, exactly as in the earlier components, so that a model that has memorized a repository's tests cannot score well without its code having actually executed. ## 7. Conclusion Code execution is the observation layer of the weak-model coding harness: the one stage where the repository talks back, mechanically, without the intervention of any model's opinion about what code does. This paper has argued that execution should be a deterministic, isolated evidence-generation service rather than an opportunistic shell: every candidate run rather than mentally simulated, commands executed under reproducible resource and security boundaries, raw runtime behavior normalized into compact provenance-rich observations, execution evidence used both to diagnose failures and to discriminate among candidate implementations, and bounded diagnostic probes generated where existing tests are insufficient without ever being mistaken for authoritative specifications. It has reviewed the evidence that makes this stance load-bearing — Self-Debug's up-to-~+12% repair gain where tests exist [1], InterCode's multi-turn doubling of GPT-4's SQL success [2], PAL's 15%-absolute trust-transfer margin on GSM8K [6], LEVER's 4.6–10.9% verifier gains over execution results [8], AlphaCode's execution-clustered selection over ~1M candidates [5] — and it has designed against the failure modes that would let the harness fool itself, from ImpossibleBench's spec-contradicting tests and test-gaming [12] to the empty-success trap and flaky-test thrash. The architecture is deterministic end to end with exactly one learned entry: the deterministic stages — action classification, isolation, execution, evidence parsing, normalization, behavioural caching, and structured observation assembly — are software on CPU, and the single L entry is diagnostic-probe generation, firing only when no adequate oracle exists. The principle that has governed every choice is the one with which this paper began and closes: whenever reality can answer a coding question cheaply, the harness should ask reality before asking the LLM to reason about it — because for a weak model, which reasons least reliably of all, the runtime is the only witness whose testimony can be trusted. --- ## References 1. Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou. *Teaching Large Language Models to Self-Debug.* 2023. arXiv:2304.05128. *(Venue caveat: "ICLR 2024" not confirmable in the arXiv record. Execution feedback and self-explanation are complementary; traceback-as-reasoning-prompt; up to ~+12% where tests exist, +2–3% on Spider.)* 2. John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao. *InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback.* NeurIPS 2023 Datasets & Benchmarks. arXiv:2306.14898. *(Corrected ID — the source's 2303.07545 is wrong. Code as action, execution feedback as observation; Docker/self-contained envs; GPT-4 SQL 9.1%→73.7% multi-turn.)* 3. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. *SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering.* 2024. arXiv:2405.15793. *(Venue caveat: "NeurIPS 2024" not in the arXiv record. Execution as ACI connective tissue; observable feedback not modelled world-state; trajectory endpoints are executions; output truncation + "no output produced" sentinel.)* 4. Xingyao Wang, et al. *Executable Code Actions Elicit Better LLM Agents (CodeAct).* ICML 2024. arXiv:2402.01030. *(Execution promoted to the action space; up to +20% success with up to 30% fewer actions across 17 LLMs on API-Bank + M3ToolEval.)* 5. Yujia Li, et al. *Competition-Level Code Generation with AlphaCode.* Science 2022. arXiv:2203.07814. *(Execution as arbiter over scale; ~1M candidates; survivors clustered BY RUNTIME BEHAVIOUR not text; 34.2% on CodeContests validation, top-54.3% human ranking. NOT described as "result caching" — do not attribute.)* 6. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. *PAL: Program-Aided Language Models.* ICML 2023. arXiv:2211.10435. *(Trust-transfer: answer is whatever the interpreter prints, never the model's stated answer; 15% absolute over PaLM-540B CoT on GSM8K; +11.4% date-understanding, +14.1% state-tracking.)* 7. Jonas Gehring, et al. *RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning.* ICML 2025. arXiv:2410.02089. *(Meta affiliation verified clean. RL with execution feedback in the loop; Llama 3.1 8B/70B multi-turn; CodeContests; ~10× sample reduction.)* 8. Ansong Ni, et al. *LEVER: Learning to Verify Language-to-Code Generation with Execution.* ICML 2023. arXiv:2302.08468. *(Learned verifier over three evidence sources — NL task, candidate program, execution result; 4.6–10.9% with code-davinci-002 across four datasets.)* 9. Islem Bouzenia and Michael Pradel. *You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects.* ISSTA 2025. arXiv:2412.10133. *(The source's "ExecutionAgent" is the method name, not the title. Runs test suites of 33/50 projects across 14 languages. EnvBench (2503.14443, 994 repos) and Repo2Run (2502.13681, 420 repos) are SEPARATE sibling benchmarks, not this paper's eval.)* 10. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. *Reflexion: Language Agents with Verbal Reinforcement Learning.* NeurIPS 2023. arXiv:2303.11366. 11. Mike A. Merrill, et al. *Terminal-Bench: A Benchmark for AI Agents in the Terminal.* 2026. arXiv:2601.11868. *(Roughly 83 authors; 89 CLI tasks; frontier agents under 65%; a 2.x version ambiguity exists — pin the version before reporting.)* 12. Haozhe Zhong, Aditi Raghunathan, and Nicholas Carlini. *ImpossibleBench.* 2025. arXiv:2510.20270. *(Impossible variants whose tests contradict the spec; measures agents passing by gaming/editing tests.)* 13. *Rollout Infrastructure Tax.* ACM SoCC 2026 submission. arXiv:2607.01415. *(Up to 110× cold-start variation and 1.8× worker-hour spread across substrates. Measures substrate-side rollout latency ONLY; explicitly EXCLUDES model inference and reward computation — soften any "dominates over inference" claim.)* 14. Theo X. Olausson, et al. *Is Self-Repair a Silver Bullet? An Empirical Study into Self-Repair with LLMs.* 2023. arXiv:2306.09896. *(Execution feedback only helps when the prompted diagnosis is good.)* 15. Jiayi Wei, et al. *CodeT: Code Generation with Generated Tests.* ICLR 2023. arXiv:2207.10397. *(Generates code and its tests; tests filter and rank code — the ancestor of the diagnostic-probe insight.)* 16. Mark Chen, et al. *Evaluating Large Language Models Trained on Code (Codex / HumanEval).* 2021. arXiv:2107.03374. *(HumanEval and pass@k established the execution contract.)* 17. *Execution-guided program synthesis* — classical root cited for lineage only (attribution unverified). 18. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. *Program of Thought: Solving Symbolic Problems with Language Models.* NeurIPS 2023. arXiv:2211.11559. 19. Guanzhi Wang, et al. *Voyager: An Open-Ended Embodied Agent with Large Language Models.* 2024. arXiv:2304.04043. *(Execution-gated skill memory.)* 20. Jie Huang, et al. *Large Language Models Cannot Self-Correct Reasoning Yet.* ICLR 2024. arXiv:2310.01798. *(Without external feedback — the cleanest being execution — models largely cannot self-correct by introspection.)*