Summary / 12. Runtime instrumentation and traces / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Don't Guess the State, Watch It: Hypothesis-Driven Selective Instrumentation as the Evidence Layer of a Weak-Model Coding Harness *Component 12 of 27 — Coding-Harness Program. This document specifies the runtime-instrumentation/traces stage. It sits after component 11 (execution), which returns a *verdict* — pass/fail — and before the components that verify and repair, which act on what the verdict points at. Where execution reports *what happened when I ran it*, instrumentation records *the run behind the verdict* — the paths, values, and state deltas that a traceback flattens into a single failing line. This paper translates the program's thesis into the trace/instrumentation build contract: a hypothesis-driven, selective, causal evidence layer in which concrete runtime state — not the model's guess at it — is the witness the repair and verification stages reason over.* --- ## Abstract Runtime instrumentation is the evidence layer of a weak-model coding harness: the stage that records the concrete execution paths, values, and state deltas that component 11's verdict leaves implicit, so that a failing run stops being a single traceback line and becomes a reducible record of what actually happened. This paper argues that instrumentation should be hypothesis-driven, selective, and causal rather than exhaustive — capture concrete execution paths and state only where needed; compare actual state against passing runs, invariants, expected states, or competing repair hypotheses; reduce raw traces through filtering, divergence analysis, and dynamic slicing before they reach the model; and treat observed runtime facts as hard evidence capable of supporting or falsifying hypotheses. The paper reviews the instrumentation-and-trace literature across nine parts — the trace-as-objective-constraint anchor TraceRepair [1], TraceFixer's first-divergence localization [2], LDB's block-level verdicts [3], Haque et al.'s negative trace-injection result [4], CausalRepair's dual-slicing [5], the demand-driven-probe outlier PingFL [6], the training side of NExT [7], honourable mentions from the debugging-agent lineage [8][9][10][11][12], and the classical roots of slicing and delta debugging [13][14][15][16][17]. It derives seven design principles, then specifies a deterministic pipeline (D) with exactly one learned entry (L): the model turns "the program fails here" into "observe X at point Y, testing hypothesis H," and every reduction — filtering, divergence analysis, dynamic slicing, evidence assembly — is deterministic software on CPU. It closes with a recursive self-improvement protocol and a measurement plan whose honest instruments — evidence-pack falsification, the anti-hallucination gate, trace-firehose and observer-effect and heisenbug audits, single-trace overfitting, and probe-bleed — exist to keep the harness from reasoning over state it never actually watched. ## 1. Introduction The coding-harness program has built, through components 01 through 11, machinery that reads the repository, generates and edits patches, and runs them to a verdict. Component 11 gave the harness its observation channel: the repository is executed, and the runtime reports pass or fail — a single terminal judgment. But a verdict is thin evidence. When a test fails, the traceback shows the model where the process died, not what the process was doing along the way — the values that were computed, the branches that were taken, the intermediate state that made the failure not merely possible but inevitable. A weak model asked to repair from a traceback alone must *infer* that state, and its inference is cheap, fast, and unreliable. Component 12 exists to replace that inference with a record: it instruments the run, captures the concrete paths and values behind the verdict, reduces that raw trace into the small residue that matters, and hands the model hard observed facts instead of asking it to imagine them. Execution asks the repository what it did; instrumentation asks the repository *how it did it*, and records the answer as evidence. The weak-model problem makes this stage load-bearing rather than optional. The best repair strategies this program can run are only as good as the diagnosis they are given, and a diagnosis built on a guessed state is a guess twice removed: the model guesses at the intermediate values, then guesses at a cause consistent with its first guess, and a wrong first guess poisons everything downstream no matter how strong the reasoning. The literature is unambiguous that the alternative — grounding the diagnosis in *observed* runtime state — is what converts a repair attempt from a lottery into a constrained search. TraceRepair shows that trace evidence acts as an *objective constraint* on repair hypotheses rather than prompt filler, correctly repairing 392 Defects4J defects [1]. TraceFixer shows that localizing the first divergence between actual and desired execution makes repair a far more constrained problem, improving top-10 repair by 13–20% over source-only [2]. LDB segments the program by control flow and asks for per-block verdicts over observed intermediate variables, gaining up to 9.8% over baselines where Self-Debug plateaus [3]. The common thread is that observation narrows the hypothesis space mechanically, so the model reasons over what remains — and a weak model, which reasons least reliably over imagined state, needs that narrowing *more*, not less. The thesis of this paper is stated plainly. **Runtime instrumentation should be hypothesis-driven, selective, and causal rather than exhaustive: capture concrete execution paths and state only where needed; compare actual state against passing runs, invariants, expected states, or competing repair hypotheses; reduce raw traces through filtering, divergence analysis, and dynamic slicing before they reach the model; and treat observed runtime facts as hard evidence capable of supporting or falsifying hypotheses. Full traces are an internal data source, not model context.** And it carries a principle that governs every design choice in this component, stated as a strong rule for Piper: **Don't give Piper a debugger and tell her to explore. Let Piper ask a precise question; let the harness instrument reality and return the answer.** A debugger handed to a weak model is an invitation to wander an unbounded state space and burn context on traces that never converge on the fault; a precise question, by contrast, is a bounded request the harness can answer deterministically by instrumenting exactly the observation that would discriminate among the model's competing hypotheses. The harness does the exploring; the model does the asking. Within that frame, the paper recognizes four levels of runtime evidence the harness must be able to produce on demand — the *execution path* (which statements and branches actually ran, the cheapest and coarsest record), *state snapshots* (the values of variables at chosen points), *state transitions* (how those values change as execution advances, giving the record a temporal spine), and *dynamic causal dependencies* (which observed state actually could have produced the failure, the tightest and most expensive level). It couples those levels with a governing discipline — *collect richly, expose sparsely* — meaning the harness may instrument broadly and capture voluminously, but what reaches the model is only the filtered, divergence-reduced, sliced residue that bears on the question at hand. Full traces live on disk as an internal data source that downstream deterministic stages mine; the model sees a compact evidence pack, never the firehose. The paper proceeds as follows. Section 2 reviews the instrumentation-and-trace literature in nine parts — the trace-as-objective-constraint anchor, the first-divergence localizer, the block-level-verdict result, the negative trace-injection result, the dual-slicing causal reducer, the demand-driven-probe outlier, the training-side curriculum, honourable mentions, and classical roots. Section 3 derives seven design principles. Section 4 specifies the architecture, labeling every node deterministic (D) or learned (L) and stating the invariant that the L column holds exactly one entry. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into eight modern lines — an anchor that makes traces an objective constraint, a first-divergence localizer, a block-level-verdict result, a negative result that separates instrumentation from presentation, a dual-slicing causal reducer, a demand-driven-probe outlier, a training-side curriculum, and a set of debugging-agent honourable mentions — plus the classical roots beneath them all. Each line answers one question this component must resolve: can observed state constrain repair hypotheses, where does a trace first go wrong, at what granularity should state be judged, why does raw trace injection fail, how does one isolate the state that caused a failure, when should the harness probe instead of trace, how does trace grounding shape the models themselves, what do the recent debugging tools add, and what classical results make selective reduction principled rather than ad hoc. ### 2.1 The anchor: TraceRepair **TraceRepair** — actual title *"From Guessing to Seeing: Enhancing LLM-Based Program Repair via Trace-Guided Multi-strategy Debate"* — is the anchor for this component because it is the paper that most completely embodies the operation this stage exists to serve: runtime evidence as an *objective constraint* on repair hypotheses rather than prompt filler. Wu, Wu, Zhang, Dong, and Shen, in ASE 2026, arXiv:2604.02647, use a probe agent to capture variable snapshots from the runtime, and specialized repair agents then cross-check their competing fixes against those observations — the trace is not decoration appended to a prompt but a court record that the debate must stay consistent with [1]. The mechanism is that competing repair hypotheses are not adjudicated by rhetorical force but by which of them remains *compatible with observed reality*: an explanation that contradicts a captured variable snapshot is eliminated regardless of how plausible it reads. The paper's headline number is 392 — it correctly repaired 392 Defects4J defects, a figure that appears verbatim in the abstract [1]. A citation note belongs in the same breath: the source's alternative title "Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate" is SPURIOUS for this ID — the correct title is "From Guessing to Seeing," and the spurious variant must be dropped. The weak-model implication is the reason this stage is a component: a weak model produces more and less-reliable hypotheses, so constraining them against observed state does more of the work of separating good from bad than it does for a strong model. ### 2.2 First-divergence localization: TraceFixer **TraceFixer** (Bouzenia, Ding, Pei, Ray, and Pradel, 2023, arXiv:2304.12743) contributes the cleanest framing of what a differential trace should localize: find the point where actual execution first diverges from desired execution, because that divergence point is where the failure *began* rather than where it surfaced [2]. The key insight is that the desired-state-at-divergence makes repair a much more constrained problem — once the harness knows both what the program did and what it should have done at the first point of difference, the repair no longer has to search over the whole program but only over the causal neighborhood of that divergence [2]. The numbers anchor the effect: trace information improved top-10 repair by 13–20% over source-only, and 10 of 20 Python bugs were repaired [2]. The weak-model lesson is that the divergence point is precisely the datum a weak model cannot reliably infer from a traceback — the traceback shows the terminal exception, but the first divergence may be many lines earlier — so the harness must find it mechanically and present it, not leave the model to back-search through imagined intermediate state. ### 2.3 Verdicts at block granularity: LDB **LDB** — *"Debug like a Human"* — (Zhong, Wang, and Shang, ACL 2024 Findings, arXiv:2402.16906) supplies the granularity result this component adopts. LDB segments the program by control-flow into basic blocks, tracks the intermediate variables computed in each block, and asks the LLM for a per-block verdict — did this block behave as intended — so the model reasons over observed intermediate values block by block rather than over the whole program at once [3]. The numbers show both the gain and the plateau-breakthrough: up to 9.8% over baselines, and — critically for a weak model — it keeps improving across iterations where Self-Debug plateaus, because each iteration's per-block verdict is grounded in newly observed state rather than in a model that has run out of new guesses [3]. Its line/block/function granularity ablation is this component's token-cost data: block-level granularity wins, which is the warrant that state should be captured and judged at the block/statement level that separates a failure from its innocent neighbors, not coarsely at function entry or exhaustively at every line [3]. ### 2.4 The negative result: Haque et al. **Haque et al.** — *"Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs"* — (KnowledgeNLP @ NAACL 2025, arXiv:2505.04441) is the essential negative result that disciplines this whole component. It found that raw trace injection helped in only 2 of 6 dataset/model configurations, and that trace usefulness DECREASES as complexity increases [4]. The mechanism is unsurprising in retrospect: an untruncated, unfiltered trace is a wall of noise that buries the signal a weak model needs, and the more complex the program the more of the trace is irrelevant to the fault. The conclusion this component draws is architectural, not merely cautionary: instrumentation and trace *presentation* must be separate concerns — the harness may instrument richly, but what it presents to the model must be filtered, reduced, and shaped to the question, because the value of a trace lives entirely in how it is presented, not in its existence [4]. This is the warrant for the collect-richly-expose-sparsely discipline and for the entire deterministic reduction pipeline of Section 4. ### 2.5 Causal reduction by dual slicing: CausalRepair **CausalRepair** (Wu, Chen, Yang, Xue, and Hao, ISSTA 2026, arXiv:2608.10613) contributes the causal-reduction mechanism this component's dynamic-slicing stage adopts. It performs *dual-slicing* — context-aware static slicing on the test side combined with execution-trace dynamic slicing on the source side — to isolate only the runtime facts that could have caused the failure, cutting away the observed-but-irrelevant state that Haque's negative result warns about [5]. The numbers are strong and cheap: 313 repaired Defects4J bugs at roughly $0.029/bug using DeepSeek-V3 [5]. The mechanism matters as much as the number: dual-slicing is the formalization of the principle that the model should reason only over the minimal causal context — the observed statements and values that stand in a dependency relation to the failure — because every observed value that is not on a causal path to the failure is context the model must pay for and may be misled by. The weak-model implication is that the *mechanical reduction* does the heavy lifting: the CPU slices the trace down to the tiny residue, and the weak LLM reasons over that residue rather than over a full execution history it cannot hold or sift. ### 2.6 The demand-driven outlier: PingFL **PingFL** — *"Let the Code Speak: Incorporating Program Dynamic State for Better Method-Level Fault Localization"* — (Qin et al., IEEE/ACM ASE 2025, no arXiv ID) is the outlier in this review because it does not trace exhaustively at all; it instruments *demand-driven* with print-debugging probes, adding observations only where the localization question requires them [6]. Its numbers are the strongest in the section: 812 Defects4J bugs with 450 localized at Top-1, a 41–122% improvement over LLM-based methods [6]. One figure demands an explicit correction: the full system costs $0.22 and 104.62s per bug — the "$0.07 / 51.62s" figure that circulates is the print-debugging component's share ONLY, so $0.07/51.62s must never be attributed as the per-bug cost of the full system [6]. The weak-model lesson is the scientific-experiment framing that becomes P6 below: PingFL treats debugging as a series of precise observations — a "scientific experiment API for code" — rather than as exhaustive surveillance, which is exactly the stance a weak model's bounded context can afford. ### 2.7 The training side: NExT **NExT** (Ni, Allamanis, Cohan, Deng, Shi, Sutton, and Yin, ICML 2024, arXiv:2404.14662) covers the side where trace grounding shapes the models themselves. NExT self-trains on execution-grounded chain-of-thought, so the model learns to *narrate what the program does step-by-step* from actual observed behavior rather than from its own guesses [7]. Its most reusable result for this component is the retention property: it retains much of the repair improvement even when traces are NOT available at inference [7]. The implication is that instrumentation can act as a *curriculum* — even for runs where the harness does not surface a trace, a model trained on trace-grounded reasoning narrates observed behavior more reliably than one trained on imagined behavior, so the evidence layer improves downstream repair at inference time and upstream competence at training time. The weak-model corollary is that the more a weak model has internalized "narrate what the program does" from watching real traces, the better it reasons over the small evidence packs this harness does present. ### 2.8 Honourable mentions The debugging lineage contributes tools and cautions without anchoring the component. **DebugRepair** (arXiv:2604.19305, preprint — no venue on record) practices self-directed debugging in which the model itself decides where to instrument, a useful contrast to this component's deterministic placement [8]. **VulDebugger** — *"Agent That Debugs: Dynamic State-Guided Vulnerability Repair"* — (Liu et al., arXiv:2504.07634) hands the agent a real debugger through run_program / run_to_line / print_value primitives, evidence that direct state inspection beats guesswork for vulnerability repair [9]. **Multi-Perspective Agentic Program Repair** (Huang, Xu, and Zhang, arXiv:2607.12605) fuses a static Code Property Graph with a temporal execution graph, a dual-perspective design this component respects but keeps on the deterministic side of the pipeline [10]. **SemCoder** (arXiv:2406.01006, NeurIPS 2024) trains models on "monologue" reasoning over execution semantics, so the model internalizes what code does rather than only its surface form [11]. **DebugBench** (Tian et al., ACL Findings 2024, arXiv:2401.04621) provides a benchmark of debugging tasks that isolates debugging ability from generation ability, a measurement contribution this component's Section 6 respects [12]. ### 2.9 The classical roots The modern lines rest on six classical results — pre-arXiv conference and IEEE proceedings, so cited here by DOI rather than arXiv ID. **Dynamic program slicing** (Agrawal & Horgan, PLDI 1990, DOI 10.1145/93542.93576) is the direct ancestor of dynamic-slicing stage (g), showing that a slice is the set of statements whose execution can affect a value of interest. **Weiser's program slicing** (IEEE TSE SE-10(4), 1984, DOI 10.1109/TSE.1984.5010248) founded the whole notion of reducing a program to the statements relevant to a computed value. **Cause-effect chains** (Zeller, FSE 2002, DOI 10.1145/587051.587053) formalized the idea that a failure has a causal chain through state, and **delta debugging** (Zeller, ESEC/FSE 1999, DOI 10.1145/318774.318946) gave the mechanism for isolating the minimal change responsible — the ancestors of TraceFixer's divergence and CausalRepair's slicing. **Tarantula** (Jones & Harrold, ASE 2005) treats coverage traces as the cheapest one-bit-per-statement signal — which statements a passing versus failing run executed — the ancestor of this architecture's failure-site seeding stage (a). These sit beneath the modern lines as the substrate on which selective, causal observation stands. ## 3. Design Principles Seven principles follow from the review, each tied to the evidence above. **P1 — Hypothesis-first, selective instrumentation.** Never dump whole-program traces at the model; instrument only where the debugging question demands observation. Haque's negative result — raw trace injection helped in only 2 of 6 configurations and usefulness decreases with complexity [4] — is the warrant that exhaustive traces are noise, and TraceRepair's constraint-based debate [1] is the warrant that the alternative is to observe precisely where competing hypotheses diverge. A weak model cannot sift a firehose; the harness decides what is observed so the model does not have to decide what to read. **P2 — Instrumentation and trace presentation are separate.** Collect richly, expose sparsely. The harness may instrument broadly and capture voluminously, but the model receives only the filtered, divergence-reduced, sliced residue that bears on the question. Haque's negative result is again the warrant [4], and it is reinforced by the context-row discipline of this harness: every token a trace adds to model context displaces a token of reasoning, so presentation is a budgeting decision made deterministically, never by appending raw output. **P3 — Traces are evidence that FALSIFIES, not context that flatters.** Observed values constrain and eliminate competing hypotheses; they are not decoration that confirms the model's favored explanation. TraceRepair's mechanism — competing fixes are cross-checked against observed variable snapshots, and only explanations that remain *compatible with observed reality* survive [1] — is the warrant: a hypothesis contradicted by an observed value is dead, and the harness should treat observation as a falsification instrument rather than a supportive narrative. **P4 — Run differentially.** Diff passing versus failing state and report the first meaningful divergence, not the terminal crash. TraceFixer's first-divergence localization — top-10 repair improved 13–20% over source-only [2] — and Zeller's cause-effect chains [14] are the warrants: the first divergence is where the failure begins, and it is the datum a weak model cannot reliably reconstruct, so the harness must find it mechanically. The differential frame also inherits the determinism precondition of component 11 — two runs only differ meaningfully if the substrate is deterministic. **P5 — Reduce causally.** Dynamic-slice the trace to the minimal causal context, so the CPU does the mechanical reduction and the LLM reasons over the tiny residue. CausalRepair's dual-slicing — 313 repaired Defects4J bugs at ~$0.029/bug [5] — and Agrawal & Horgan's dynamic slicing [13] are the warrants: every observed value not on a causal path to the failure is paid-for context that can mislead. Reduction is deterministic work; reasoning over the residue is the model's only share. **P6 — Demand-driven probes.** Ask a precise question; instrument exactly that observation. PingFL's "scientific experiment API for code" — 812 Defects4J bugs, 450 localized at Top-1 [6] — is the warrant that targeted probing beats exhaustive tracing. This is the concrete shape of the paper's strong Piper rule: the model asks "does X hold at Y," and the harness instruments precisely that, rather than giving the model a debugger and telling it to explore. **P7 — Anti-hallucination gate.** The model may cite only values it actually observed, never narrate unobserved state. Haque's negative result — raw traces mislead at high complexity [4] — is the warrant, and it is sharpened by the "imagined traces" failure mode in which a weak model, unable to recall the real values, invents plausible ones and reasons confidently over them. The gate is mechanical: the evidence pack is the only state the model is allowed to cite, and any claimed observation not present in the pack is flagged. ## 4. The Architecture The pipeline is a sequence of stages; every node is labeled D (deterministic software) or L (a learned model). The invariant that governs the whole is: *the L column contains exactly one entry — a single weak-LLM hypothesis-generation call — and every upstream and downstream step is deterministic software on CPU.* The concrete inputs are a failing candidate and its verdict from component 11, plus the existing tests and any competing repair hypotheses; the output is an assembled `StateObservation`-based evidence pack that the verification and repair stages consume. The flow, in the eight deterministic stages (a) through (h) plus the one learned entry (L): 1. **(a) Failure-site and coverage analysis [D].** The failing run's coverage is analyzed by spectral fault-localization techniques in the spirit of Tarantula [17] — one-bit-per-statement coverage of passing versus failing runs — to seed candidate probe points: the statements whose coverage correlates most strongly with failure. This stage does not localize the fault; it ranks where observation is likely to discriminate, producing a bounded candidate set of probe points rather than a whole-program trace plan. 2. **(b) Selective instrumentation planning [D].** The debugging question — "the program fails here" — is mapped deterministically to concrete observe/probe points. Given the question and the candidate points from (a), a deterministic planner selects the specific statements and variables to observe, at the level of the block-granularity evidence of LDB [3]: enough to separate the failure from its innocent neighbors, not so much as to recreate the firehose Haque warned about [4]. Planning is a mapping from question to observation plan, not an inference. 3. **(c) Instrument [D].** The chosen plan is realized by deterministic code injection — print statements, breakpoints, or a tracing hook (set_trace / sys.settrace / a debugger API) inserted at the planned points. Injection is deterministic per toolchain, so the same plan yields the same instrumentation, and the injected code records exactly the values and paths the plan names. 4. **(d) Execute [D].** The instrumented candidate runs under component 11's sandbox and isolation policy. This stage delegates to execution rather than duplicating it: instrumentation reuses the execution substrate of component 11, adding only its capture hooks, so the run that produced the verdict and the instrumented run that explains it share the same determinism and resource boundaries. If a run is non-deterministic, the differential stages below flag it rather than trusting it. 5. **(e) Capture raw events [D].** The instrumented run yields raw events — execution paths (which statements and branches were taken), observed variable values at each probe point, and state deltas between consecutive observations. This is the collect-richly moment: capture is voluminous and uncurated, written to an internal store that downstream stages mine; none of it reaches the model directly, honoring collect-richly-expose-sparsely. 6. **(f) Filtering and divergence analysis [D].** The raw events are filtered to those on the probed paths, and a passing run (the same or a sibling run that succeeds) is diffed against the failing run to find the first meaningful divergence in observed state — TraceFixer's first-divergence localization [2], Zeller's cause-effect chain [14]. The output is a narrowed record centered on the divergence, with the values that differ identified as the primary suspects. 7. **(g) Dynamic slicing [D].** The divergence-centered record is dynamically sliced to the minimal causal context — CausalRepair's dual-slicing [5], Agrawal & Horgan's dynamic slicing [13] — retaining only the observed statements and values that stand in a dependency relation to the failure. This is P5's mechanical reduction: the CPU cuts the trace to the tiny residue that could have caused the failure, discarding every observed-but-irrelevant value that would have cost context and risked misleading the model. 8. **(L) Hypothesis/question generation [L] — the single learned entry.** The weak model turns "the program fails here" into "observe X at point Y, testing hypothesis H." This is the debugging question plus its competing hypotheses, produced as a single bounded call: the model states what it believes may be wrong and what observation would discriminate among its candidate explanations. The output feeds back into stage (b) — if the model's hypothesis names a variable the initial plan did not cover, the harness instruments that observation — closing the loop between the one learned question and the deterministic observation it demands. This is the PingFL "scientific experiment API" stance [6] and the concrete shape of the strong Piper rule: the model asks, the harness instruments. 9. **(h) Evidence assembly [D].** The reduced record is assembled into a structured evidence pack built from `StateObservation` records. Each record's schema is (execution_id, location, path_index, variables, return_values, branch_taken): execution_id ties the observation to exactly the run that produced it (reusing component 11's artifact_version lineage); location names the file/line or symbol observed; path_index places it on the execution path; variables holds the observed values; return_values the observed returns; branch_taken the branch that was actually taken at a decision point. The pack is the single currency of this component: repair and verification stages reason over it, the anti-hallucination gate (P7) checks every claim against it, and nothing about the run reaches the model except through it. The primitive set the harness exposes is sixfold and deterministic except where noted: `trace_path` (record the execution path of a run), `observe` (capture variable/return state at a named point — the print/probe primitive), `compare_runs` (run two variants and diff their observations), `trace_diff` (find the first meaningful divergence between a passing and a failing trace), `dynamic_slice` (reduce a trace to the statements on a causal path to a value), and `probe` (inject and run a targeted observation, the demand-driven P6 primitive). Of these, `observe` and `probe` are the workhorses of the hypothesis loop — the model's single L call names what `probe` should observe, and `trace_path`, `compare_runs`, `trace_diff`, and `dynamic_slice` are the deterministic reduction machinery that turn raw capture into the sparse pack. The target is explicit and checkable: stages (a)–(h) are software on CPU, and the L column holds exactly one entry — the hypothesis/question-generation call that decides what to observe. Any proposed addition that would put a language model elsewhere in the pipeline is a regression against P1–P7 and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of Section 6, on held-out repositories, at a defensible cost, or be rejected. The invariant is stated again so it can be enforced mechanically: **the L column contains exactly one entry.** ## 5. Recursive Self-Improvement Protocol Component 12 of 27 follows the program's uniform protocol: the executable specification is written to 100%, round 1 lands at approximately 80% of the specified behavior, and later rounds close the gap. Each round is A/B-gated against the honest metrics of Section 6 — never the end-to-end resolve rate alone, which would let instrumentation quality mask repair quality. Because this component's deterministic stages (a)–(h) carry the load, their improvement is mostly direct engineering, and trace evidence feeds back to improve the harness itself in three concrete loops. First, *probe-point calibration*: the harness learns which probe points actually discriminate — a probe point at which passing and failing runs never diverge is evidence about nothing, while a point at which they first diverge is the load-bearing observation — so over rounds the planner (b) is tuned to prefer the discriminating points and drop the silent ones, shrinking each evidence pack to what downstream stages genuinely consume. Second, *evidence-pack shrinkage*: the harness measures which fields of the `StateObservation` pack the verification and repair stages actually use, and a field whose removal does not degrade repair on held-out runs is a candidate for removal — the pack should shrink to the evidence that demonstrably matters, per P2's expose-sparsely, never grow on the intuition that more state helps. Third, *anti-hallucination-gate tightening*: the gate is tightened by measuring how often the model attempts to cite a value not present in the pack, and how often the harness's own assembly admits an unobserved or non-causal value; over rounds the gate's coverage is widened and its checks sharpened so the pack is not merely complete but provably the only state the model can claim. Because every observation is versioned by execution_id and tied to component 11's artifact hash, a regression in what instrumentation records is visible in the assembled pack long before it surfaces as a failed resolve. ## 6. Measurement & Evaluation Runtime instrumentation is measured *separately* and *honestly*, because a stage whose whole purpose is to make observed state the evidence a weak model reasons over can be defeated by the harness or the model reasoning over state that was never actually watched. The instruments are built to keep the evidence layer accountable to the runtime it instruments. **Evidence-pack falsification.** The primary instrument asks whether the evidence pack actually falsifies hypotheses: given a deliberately wrong repair hypothesis, does the observed state eliminate it — do the captured values contradict the wrong explanation the way TraceRepair's cross-checks eliminate incompatible fixes [1]? A pack that fails to falsify a clearly-wrong hypothesis is not evidence; it is decoration, and the instrument measures the fraction of wrong hypotheses that the observed values actually rule out. **The anti-hallucination gate.** The companion instrument audits whether the model ever cites an unobserved value: every claimed observation in a repair narrative is checked against the evidence pack, and any claim — a variable value, a branch taken, a return — not present in the pack is flagged as a hallucination. The gate also audits the pack itself for non-causal or non-observed inclusions, so the harness cannot fabricate evidence even on the model's behalf. Haque's negative result [4] and the imagined-traces failure mode are the warrants that this audit is not optional. **Failure-mode audit.** Beyond the positive instruments, the harness audits the specific ways an instrumentation layer can lie. The *trace-firehose* mode is when raw capture floods the model despite the collect-richly-expose-sparsely discipline, burying the signal — the exact failure Haque measured [4]. The *observer-effect* mode is when the instrumentation itself changes what runs, so the observed state is not the state of the un-instrumented candidate. The *heisenbug* mode is the sharpest instance of the observer effect plus a determinism failure: a bug that vanishes or changes under observation, which makes the differential run (P4) meaningless — determinism, inherited from component 11, is a *prerequisite* for differential runs, and any non-reproducible observation is flagged rather than trusted. *Single-trace overfitting* is a fix that passes the probed run but nothing else — the model repairs exactly the observed path and ignores the states the probes did not cover, so the instruments must test beyond the single evidence pack. *Probe-bleed* is instrumentation edits leaking into the repair diff — the injected print/breakpoint code must be stripped before the patch is evaluated, or the repair is scored on code the harness wrote rather than the model wrote. Each mode has a guard: firehose audits enforce the sparsity cap; observer-effect audits diff instrumented against un-instrumented behavior; heisenbug audits re-run for determinism before trusting any divergence; overfitting is caught by out-of-pack test runs; probe-bleed is caught by diffing the submitted patch against the candidate without instrumentation. Two honest caveats close the section. First, the strong-model numbers of Section 2 — TraceRepair's 392 [1], CausalRepair's 313 [5], PingFL's 812 [6] — will NOT reproduce on a weak model; they were achieved with strong models and at scale, and a weak-model harness should expect materially lower absolute repair counts. Its instrumentation stage exists precisely so that weak *hypothesis generation* is compensated by strong deterministic *observation and reduction*, and the honest instruments here — above all evidence-pack falsification and the anti-hallucination gate — are what will show whether that compensation is real. Second, all instrumentation metrics must be reported on held-out and out-of-benchmark repositories, exactly as in the earlier components, so that a model that has memorized a repository's traces cannot score well without its repairs having been grounded in state the harness actually watched. ## 7. Conclusion Runtime instrumentation is the evidence layer of the weak-model coding harness: the stage that records the concrete execution paths, values, and state deltas behind component 11's verdict, so a failing run stops being a single traceback line and becomes a reducible record of what actually happened. This paper has argued that instrumentation should be hypothesis-driven, selective, and causal rather than exhaustive — capture concrete state only where needed, compare actual state against passing runs and competing hypotheses, reduce raw traces through filtering, divergence analysis, and dynamic slicing before they reach the model, and treat observed facts as hard evidence that falsifies rather than context that flatters. It has reviewed the evidence that makes this stance load-bearing — TraceRepair's 392 trace-constrained Defects4J repairs [1], TraceFixer's 13–20% first-divergence gain [2], LDB's up-to-9.8% block-level verdicts [3], CausalRepair's 313 dual-sliced repairs at ~$0.029/bug [5], PingFL's 812 demand-driven localizations [6] — and it has designed against the failure modes that would let the harness reason over state it never watched, from Haque's raw-trace negative result [4] to the heisenbug and probe-bleed audits. The architecture is deterministic end to end with exactly one learned entry: stages (a)–(h) — failure-site and coverage analysis, selective instrumentation planning, instrumentation, execution, raw-event capture, filtering and divergence analysis, dynamic slicing, and evidence assembly — are software on CPU, and the single L entry is hypothesis/question generation, the one bounded call that turns "the program fails here" into "observe X at point Y, testing hypothesis H." The principle that has governed every choice is the one with which this paper began and closes, stated as the strong rule for Piper: **Don't give Piper a debugger and tell her to explore. Let Piper ask a precise question; let the harness instrument reality and return the answer.** The harness explores mechanically and reduces causally; the model asks precisely and reasons over the tiny residue of state it actually watched — because for a weak model, the state it observed is the only state it can be trusted to reason about. --- ## References 1. Jiaqing Wu, Tong Wu, Manqing Zhang, Yunwei Dong, and Bo Shen. *From Guessing to Seeing: Enhancing LLM-Based Program Repair via Trace-Guided Multi-strategy Debate.* ASE 2026. arXiv:2604.02647. *(The source's alternative title "Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate" is SPURIOUS for this ID — drop it. Runtime evidence as objective constraint, not prompt filler; a probe agent captures variable snapshots; repair agents cross-check competing fixes against observations; 392 correctly repaired Defects4J defects, verbatim in the abstract.)* 2. Islem Bouzenia, Yangruibo Ding, Kexin Pei, Baishakhi Ray, and Michael Pradel. *TraceFixer.* 2023. arXiv:2304.12743. *(Find the first point where actual execution diverges from desired execution; desired-state-at-divergence makes repair a much more constrained problem; trace information improved top-10 repair by 13–20% over source-only; 10 of 20 Python bugs repaired.)* 3. Zhong, Wang, and Shang. *Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step by Step (LDB).* ACL 2024 Findings. arXiv:2402.16906. *(Segments by control-flow into basic blocks; tracks intermediate variables per block; asks the LLM for per-block verdicts; up to 9.8% over baselines; keeps improving where Self-Debug plateaus; block-level granularity wins the ablation.)* 4. Mirazul Haque, Petr Babkin, Farima Farmahinifarahani, and Manuela Veloso. *Towards Effectively Leveraging Execution Traces for Program Repair with Code LLMs.* KnowledgeNLP @ NAACL 2025. arXiv:2505.04441. *(Negative result: raw trace injection helped in only 2 of 6 dataset/model configurations; trace usefulness DECREASES as complexity increases. Warrant that instrumentation and trace presentation must be separate.)* 5. Linhao Wu, Yizhou Chen, Zhen Yang, Pengyu Xue, and Dan Hao. *CausalRepair.* ISSTA 2026. arXiv:2608.10613. *(Dual-slicing: context-aware static slicing on the test side + execution-trace dynamic slicing on the source side; isolates only the runtime facts that could have caused the failure; 313 repaired Defects4J bugs at ~$0.029/bug with DeepSeek-V3.)* 6. Yihao Qin, et al. *Let the Code Speak: Incorporating Program Dynamic State for Better Method-Level Fault Localization (PingFL).* IEEE/ACM ASE 2025. No arXiv ID. *(Demand-driven print-debugging probes instead of exhaustive tracing; 812 Defects4J bugs, 450 localized at Top-1, 41–122% over LLM-based methods. IMPORTANT: the FULL system costs $0.22 and 104.62s per bug — the "$0.07 / 51.62s" figure is the print-debugging component's share ONLY, not the per-bug cost of the full system.)* 7. Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinfei Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. *NExT: Teaching Large Language Models to Reason about Code Execution.* ICML 2024. arXiv:2404.14662. *(Self-trains on execution-grounded chain-of-thought so the model learns to narrate what the program does step-by-step; retains much of the repair improvement even when traces are NOT available at inference — instrumentation as a curriculum.)* 8. Wu, et al. *DebugRepair.* Preprint. arXiv:2604.19305. *(No venue on record; self-directed debugging where the model decides where to instrument.)* 9. Liu, Ma, Xu, Ai, Gao, Sun, and Roychoudhury. *Agent That Debugs: Dynamic State-Guided Vulnerability Repair (VulDebugger).* 2025. arXiv:2504.07634. *(Hands the agent a real debugger via run_program / run_to_line / print_value.)* 10. Huang, Xu, and Zhang. *Multi-Perspective Agentic Program Repair.* 2026. arXiv:2607.12605. *(Fuses a static Code Property Graph with a temporal execution graph.)* 11. Ding, Peng, Min, Kaiser, Yang, and Ray. *SemCoder: Training Code Language Models with Comprehensive Semantics.* NeurIPS 2024. arXiv:2406.01006. *(Reasoning over execution semantics via monologue.)* 12. Tian, et al. *DebugBench: Evaluating Debugging Capability of Large Language Models.* ACL Findings 2024. arXiv:2401.04621. *(Benchmark isolating debugging ability from generation ability.)* 13. H. Agrawal and J. R. Horgan. *Dynamic Program Slicing.* PLDI 1990. DOI 10.1145/93542.93576. *(Dynamic slicing: the set of statements whose execution can affect a value of interest.)* 14. Andreas Zeller. *Isolating Cause-Effect Chains from Computer Programs.* FSE 2002. DOI 10.1145/587051.587053. *(A failure has a causal chain through state.)* 15. Andreas Zeller. *Yesterday, My Program Worked. Today, It Does Not. Why?* ESEC/FSE 1999. DOI 10.1145/318774.318946. *(Delta debugging: isolate the minimal change responsible for a failure.)* 16. Mark Weiser. *Program Slicing.* IEEE Transactions on Software Engineering SE-10(4), 1984. DOI 10.1109/TSE.1984.5010248. *(Foundational program slicing.)* 17. James A. Jones and Mary Jean Harrold. *Empirical Evaluation of the Tarantula Automatic Fault-Localization Technique.* ASE 2005. *(Coverage traces as the cheapest one-bit-per-statement signal — which statements passing vs failing runs executed.)*