Summary / 13. Error interpretation and debugging / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Read the Error Like Evidence, Not Like Noise: Parsing, Classification, and Causal Diagnosis as the Interpretation Layer of a Weak-Model Coding Harness *Component 13 of 27 — Coding-Harness Program. This document specifies the error-interpretation/debugging stage. It sits after component 12 (instrumentation), which gathers the evidence, and before patch generation (component 09) and the repair/verification stages, which act on that evidence. Where execution emits signals and instrumentation records the run behind them, this component is the CONVERSION layer — the interpreter between the runtime's complaint and the repair loop's input. It parses, classifies, and causally diagnoses the failure so that downstream repair never reasons from raw stderr. This paper translates the program's thesis into the diagnosis build contract: an evidence-driven, causal interpretation stage that produces an explicit, provenance-backed diagnosis — with calibrated confidence — before any repair is attempted.* --- ## Abstract Error interpretation is the conversion layer of a weak-model coding harness: the stage that sits between component 12's instrumentation — the evidence that records the run behind the verdict — and component 09's patch generation, which must repair from that evidence. This paper argues that interpreting an error should be an evidence-driven causal-diagnosis stage, not a prompt asking an LLM to explain stderr: classify the failure; normalize compiler, exception, assertion, trace, and runtime evidence into typed observations; distinguish symptom from immediate cause and root cause; generate competing causal hypotheses; derive observations capable of discriminating between them; use execution and instrumentation to falsify unsupported explanations; and emit an explicit, provenance-backed diagnosis with calibrated confidence before any repair is attempted. The paper reviews the error-interpretation and debugging literature across nine parts — the Self-Debug anchor whose explain-before-fix step is diagnosis generation [1], RepairAgent's diagnosis-gates-repair tool discipline [2], CRITIC's grounding requirement that critique cannot run on the model's own confidence [3], AutoSD's scientific-debugging loop [4], DSDBench's diagnosis-not-repair measurement [5], the regression-error taxonomy that names wrong diagnosis the largest failure class [6], DebugBench's execution-feedback result [7], honourable mentions from stalled agents, log diagnosis, RCA, formal-verification debugging, and compiler-error diagnosis [8][9][10][11][12], and the classical roots and two outliers — Becker's error-message editing and the no-external-signal result — beneath them [13][14][15][16]. It derives seven design principles, then specifies a deterministic pipeline (D) with exactly one learned entry (L): the model proposes competing root-cause hypotheses, each with the observation that would discriminate it, and every parsing, classification, triage, normalization, and assembly step is deterministic software on CPU. It closes with a recursive self-improvement protocol and a measurement plan whose honest instruments — diagnosis accuracy, the evidence-citation gate, and a suite of failure-mode audits — exist to keep the harness from declaring a cause it never proved. ## 1. Introduction The coding-harness program has built, through components 01 through 12, machinery that reads the repository, instruments the run, and gathers evidence about what actually happened. Component 12 gave the harness its evidence layer: it records the concrete execution paths, values, and state behind a verdict, so a failing run stops being a single traceback line and becomes a reducible record of observed reality. But observed reality is not yet a *cause*. When a test fails, the runtime emits a signal — a compiler diagnostic, an exception traceback, an assertion message, a test report — and between that raw signal and the repair loop's input there lies an act of interpretation: someone must decide what the failure actually *is*. Component 13 exists to perform that act, and to perform it as evidence-driven causal diagnosis rather than as an LLM asked to free-associate over stderr. Component 12 asks the repository how it failed and records the answer; component 13 reads that record like a witness statement and extracts the cause — the classification, the symptom-versus-cause hierarchy, the competing hypotheses, and the explicit diagnosis that component 09 then repairs against. The weak-model problem makes this stage load-bearing rather than optional. A weak model asked to explain an error will produce a plausible-sounding narrative that may have nothing to do with the observed evidence — it reads the message on the label and constructs a story consistent with its own prior, not with the run. The literature is unambiguous about the failure mode this creates. The regression-error study finds that wrong diagnosis is the single largest failure class at 44.07%, driven by misdirection from test names and error signals — in the source's blunt framing, the model reads the label on the error, not the error [6]. The whole component's feet rest on the result that LLMs cannot reliably self-correct without an external signal [14]. Error interpretation therefore cannot be a generation task; it must be a *diagnosis* task with the same evidentiary discipline as the instrumentation that precedes it — classify the failure into a schema, parse the signal into typed observations, distinguish symptom from immediate cause from root cause, hypothesize competing causes, and let execution and instrumentation falsify everything unsupported before a single repair is attempted. The thesis of this paper is stated plainly. **Error interpretation should be an evidence-driven causal-diagnosis stage, not a prompt asking an LLM to explain stderr: classify the failure; normalize compiler, exception, assertion, trace, and runtime evidence into typed observations; distinguish symptom from immediate cause and root cause; generate competing causal hypotheses; derive observations capable of discriminating between them; use execution and instrumentation to falsify unsupported explanations; and emit an explicit, provenance-backed diagnosis with calibrated confidence before any repair is attempted.** And it carries a strong rule for Piper that governs every design choice: **Never ask "what do you think caused this?" when the harness can instead ask "what observation would prove or disprove each possible cause?"** The first question invites the model to generate a narrative it cannot verify; the second converts diagnosis into a falsifiable prediction the harness can test with component 12's instrumentation, so the model's job becomes proposing discriminative observations rather than asserting unobserved causes. Within that frame the paper holds a three-level hierarchy constant throughout: the *symptom* is what the user or runtime observes — "the program crashes"; the *immediate cause* is the direct trigger the error surfaces — "a null dereference at line 41"; and the *root cause* is the upstream defect whose repair removes the symptom — "an uninitialized field set three call-frames earlier." A diagnosis that stops at the immediate cause repairs the surface and leaves the defect; the harness must move down the hierarchy before it declares success. The paper proceeds as follows. Section 2 reviews the error-interpretation and debugging literature in nine parts — the explain-before-fix anchor, the diagnosis-gates-repair discipline, the grounding requirement, scientific debugging, diagnosis measurement, the failure taxonomy, execution-feedback debugging, honourable mentions, and the classical roots plus two outliers. Section 3 derives seven design principles. Section 4 specifies the architecture, labeling every node deterministic (D) or learned (L) and stating the invariant that the L column holds exactly one entry. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into eight modern lines — an anchor that makes diagnosis an explicit artifact, a repair agent whose failure-analysis prompts are load-bearing, the grounding requirement that critique needs external signals, scientific debugging as a falsifiable loop, a benchmark that measures diagnosis rather than repair, a taxonomy that names wrong diagnosis the largest failure class, an execution-feedback debugging study, and a set of honourable mentions — plus the classical roots and two outliers beneath them all. Each line answers one question this component must resolve: should diagnosis precede repair, when must the model state what-and-why before it earns edit tools, why does critique fail without external grounding, how does one turn a failure into falsifiable hypotheses, what does it mean to measure diagnosis separately from repair, why do models misread errors, does execution feedback uniformly help, what do the recent tools add, and what classical and outlier results make the error surface itself editable and the no-signal baseline inescapable. ### 2.1 The anchor: Self-Debug **Self-Debug** (Chen, Lin, Schärli, Zhou, Le, and others, arXiv:2304.05128, its fourth appearance in this program) is the anchor for this component because its rubber-duck step *is* diagnosis generation: the model explains its code before repairing it, and the explanation is produced in isolation — explain-before-fix [1]. The decisive finding is that generating an explanation yields large gains *without any execution feedback at all* — up to ~+12% on tasks where tests exist — which separates the value of verbalized diagnosis from the value of runtime grounding [1]. The weak-model lesson is twofold and it shapes this whole component. First, forcing an explicit diagnosis before a fix is not ceremony; it is a mechanism that measurably improves repair even when the diagnosis is never checked against a run, so the harness must enforce the ordering. Second, because the explanation is valuable even unchecked, the diagnosis must be an explicit, checkable, cacheable artifact — a structured object that downstream stages can cite, verify against evidence, and store — not something implicit inside the repair prompt where it can neither be audited nor reused. ### 2.2 Diagnosis gates repair: RepairAgent **RepairAgent** (Bouzenia, Devanbu, and Pradel, 2024, arXiv:2403.17134) is the architecture that makes diagnosis-gates-repair concrete. It runs a curated ~14-tool prompt stack whose sequencing mirrors the canonical analyse-failure → explain-bug → localise → edit → validate loop, and critically the *failure-analysis prompts are load-bearing*: the model must state what is wrong and why before it is granted the tools that edit code [2]. The discipline is that a diagnosis that has not named the failing behavior does not earn the right to change anything — the tool gating enforces the ordering mechanically rather than trusting the model to self-sequence [2]. The results are consistent gains over one-shot baselines at very low cost — on the order of a dime per repair — which is precisely the regime a weak-model harness operates in, where diagnosis quality is the lever and each repair must stay cheap [2]. The weak-model implication is P7 below: the harness withholds edit tools until the analysis passes a citation check, so a weak model cannot burn repair budget on a cause it has not yet stated. ### 2.3 Grounding before critique: CRITIC **CRITIC** (Gou, Shao, Gong, Yang, Huang, Duan, and Chen, ICLR 2024, arXiv:2305.11738) supplies the grounding requirement that disciplines the whole interpretation layer: LLMs cannot reliably self-critique their own output without external signals, because critique running on the model's own confidence has no independent witness [3]. Tool-grounded critique — where the model checks its output against an external interpreter or search — yields consistent gains on both reasoning and code tasks, whereas unaided self-critique does not [3]. The diagnosis mechanism this component builds must therefore interact with *observed evidence*, never with the model's sense that it is probably right. A diagnosis asserting a value the harness never observed is rejected by construction (P5), which makes CRITIC's external-signal requirement the epistemic floor of the entire stage: the only admissible grounds for believing a cause are the typed observations component 13 normalized from real signals. ### 2.4 Scientific debugging: AutoSD **AutoSD** (Kang, Chen, Yoo, and Lou; arXiv:2304.02195 preprint, published as EMSE 2025, volume 30, Springer — the circulating "Kang, Yoon & Yoo / TOSEM" attribution is wrong and must not be used) frames debugging as an explicit scientific loop — hypothesis → prediction → experiment → observation → conclusion — that runs *before* patch generation, so a fix is the last step of a falsifiable investigation rather than the first guess [4]. The paper includes a human study of 20 developers in which explanations improved patch-correctness judgment on 5 of 6 real bugs — direct evidence that a human debugger's ability to judge whether a patch is right improves when given an explicit causal explanation, which is the same effect this component's diagnosis artifact is meant to deliver to a downstream weak repair model [4]. The weak-model lesson is that the scientific loop converts "guess a cause" into "propose a hypothesis and the prediction that would confirm or refute it," which is exactly the shape of the component's single learned entry in Section 4. ### 2.5 Measuring diagnosis, not repair: DSDBench **DSDBench** (Yang et al., EMNLP 2025 Oral, arXiv:2503.22388) is the benchmark that measures *diagnosis* rather than repair, which is the measurement stance this component adopts: its tasks require multi-hop error tracing and multi-bug detection under an all-bugs-for-credit rule, so a model earns nothing by catching one of several intertwined defects [5]. Its core finding is that model performance degrades as the number of hops rises — the paraphrase the paper supports is that accuracy falls off over longer error-propagation chains, not the stronger "sharply" phrasing, which must be avoided [5]. The implication for the harness is that diagnosis quality is a function of how far the causal chain extends and how many independent defects the evidence actually contains — which is why Section 6 measures diagnosis on its own terms (did the hypothesis name the real fix site) rather than folding it into the resolve rate where a lucky patch could mask a shallow diagnosis. ### 2.6 The failure taxonomy: regression errors The regression-error study of Ho, Le-Cong, Le, and Rizkallah (arXiv:2506.13182, minor revision at TOSEM) contributes the taxonomy this component's classification schema leans on, and its central number is a diagnosis indictment: wrong diagnosis is the largest failure class at 44.07%, driven by misdirection from test names and error signals, with weak test suites the second contributor at 23.73% [6]. The mechanism is captured in the source's own framing — the model reads the label on the error, not the error: it sees the name of the failing test or the first line of the traceback and builds a diagnosis from that label's connotation rather than from the observed behavior behind it [6]. This is the strongest available warrant for the component's whole posture: parsing and classification must be deterministic, symptom must be separated from cause, and the model must never be handed a raw error message whose label it can misread, because the dominant way diagnosis fails is precisely that misreading. ### 2.7 Execution feedback in debugging: DebugBench **DebugBench** (Tian et al., ACL Findings 2024, arXiv:2401.04621 — the source's 2401.03426 ID is wrong and must be corrected) provides ~4,000+ debugging instances across four bug types with leakage control [7]. Its results calibrate this component's expectations soberly: even GPT-4 resolves only about a third of instances without scaffolding, execution feedback materially improves debugging performance, and yet that feedback is *not* uniformly beneficial — it helps on some tasks and configurations and not others [7]. The weak-model warnings are the sharpest: small models collapse on the benchmark, so the harness cannot expect interpretation to succeed by raw model strength [7]. The taxonomy lesson (P3) also draws from DebugBench: different bug types demand different evidence, so a single generic "explain the error" prompt is the wrong instrument, and the harness must classify before it decides what evidence to gather. ### 2.8 Honourable mentions The recent debugging lineage contributes tools and cautions without anchoring the component. **Stalled/Biased/Confused** (Riddell et al., University of Waterloo, FORGE 2026, arXiv:2601.22208) names stalling and bias-anchoring as its discrete failure classes in which debugging agents get stuck or anchor to an early wrong hypothesis — "confusion" is a paraphrase in the title, not one of the named classes, so only stalling and bias-anchoring are to be cited as such [8]. **LogDx-CI** (arXiv:2605.28876, v1.2 preprint — flag it as a single-author, AI-drafted-ground-truth, 35-case preprint) finds that aggressive log compressors strip assertion, snapshot, and compiler blocks that diagnosis actually needs, and that a hybrid grep-plus-tail strategy with a ~120k-character threshold wins at roughly $0.03 per case [9]. **OpenRCA** (no arXiv ID — ICLR 2025, OpenReview M4qNIzQYpd; it must not be confused with the separate OpenRCA 2.0 at arXiv:2606.27154) evaluates root-cause-analysis agents across 335 enterprise failures and over 68 GB of telemetry, where the best RCA agent reaches only ~11.34% (Claude 3.5) — a sobering bound on automated RCA even for strong models [10]. **FVDebug** (arXiv:2510.15906, whose actual title is *"...of Formal Verification Failures"*, NVIDIA: Bai, Hamad, Ho, Suhaib, and Ren) combines a causal graph with for-and-against prompting and an insight rover to issue per-node symptom-versus-root-cause verdicts — an architecture this component mirrors in its causal-chain field [11]. **RustAssistant** (Deligiannis et al., ICSE 2025, arXiv:2308.05177; the ICSE title adds the prefix "RustAssistant:") is a compiler-error-specific diagnosis tool that reaches ~74% on real Rust compile errors — evidence that a tool specialized to one signal schema (compiler diagnostics) outperforms generic error reading, which motivates P2's per-signal triage [12]. ### 2.9 Classical roots and two outliers This component's lineage is lighter than the instrumentation stage's but it carries two outliers that change the design. **Becker** — *"Enhancing Compiler Error Messages"* by Brett A. Becker, SIGCSE '16 (not ICER, DOI 10.1145/2839509.2844584, not on arXiv) — is the outlier because it is a *cause-effect* result on the error surface itself: the information content of an error message causally changes debugging success, so the harness may *edit the error surface* before the model sees it — suppressing the cascade tail, rephrasing or trimming a misdirecting message — rather than treating stderr as fixed input [13]. **LLMs Cannot Self-Correct Yet** (Huang et al., ICLR 2024, arXiv:2310.01798) is the no-external-signal result sitting under the whole component's feet: without external feedback, self-correction does not reliably improve output, which is the terminal warrant that diagnosis must be grounded in observed evidence (P5) and never in the model's own confidence [14]. The lineage beneath both is **Zeller's** cause-effect chains and delta debugging (FSE 2002 and ESEC/FSE 1999, cited by DOI), which established that a failure is a causal chain through state and that isolating the cause is a falsifiable, mechanical reduction — the ancestors of this component's causal_chain field and its falsification-by-observation discipline [15][16]. ## 3. Design Principles Seven principles follow from the review, each tied to the evidence above. **P1 — Parse, don't paste.** Never hand raw stderr to the model. Every signal — compiler diagnostic, exception traceback, assertion message, test report — is converted deterministically into a structured record carrying its class, message, frames, and actual-versus-expected values. Becker's finding that the information content of an error message causally changes debugging success [13] is the warrant that the error message is the *highest-density token budget* in the whole loop, and parsing is how the harness extracts that density instead of burying it in a wall of unread text. **P2 — Triage by signal schema.** Different signal types demand different first readings. A compiler error is triaged to its earliest root cause with the cascade tail suppressed (Becker's editing of the error surface [13] and RustAssistant's ~74% on compiler errors [12]); an exception is triaged to its deepest non-library frame, where the user's code actually diverged; an assertion failure is triaged to its actual-versus-expected values plus the test's intent (the target of the assertion, not its mechanics). Triage is deterministic per toolchain; the model never decides how to read a signal. **P3 — Classify into a failure taxonomy.** The failure is classified into a schema — syntax, type, null, off-by-one, async-order, env-config, concurrency — because different classes need different evidence and different repair paths. DebugBench's finding that execution feedback helps some bug types and not others [7], and the 44.07% wrong-diagnosis rate driven by misreading signals [6], are the warrants: a class label tells the harness what kind of evidence would discriminate causes, so classification is not a report card but a branch on what to observe next. **P4 — Distinguish symptom from immediate cause from root cause.** The hierarchy is non-negotiable: the harness records the symptom, names the immediate cause the error surfaces, and keeps digging for the root cause whose repair removes the symptom. A diagnosis that stops at the immediate cause is the classic shallow repair that AutoSD's scientific loop [4] and the regression taxonomy's wrong-diagnosis class [6] both warn against; the harness must move down the hierarchy before declaring success, and it must say *which* level each claim sits at so downstream repair does not confuse a surface trigger with a defect. **P5 — Ground before believing.** A diagnosis that asserts an unobserved value is rejected by construction. CRITIC's grounding requirement — self-critique without external signals is unreliable [3] — and Huang's no-external-signal result [14] are the warrants, and they inherit directly into component 12's anti-hallucination gate: every observation a diagnosis cites must be present in the evidence pack, and any claimed value not actually observed flags the diagnosis as unsupported regardless of how plausible it reads. **P6 — Explain-then-fix.** Enforce Self-Debug's ordering mechanically: the diagnosis is produced and articulated *before* any repair is considered, and the explanation is valuable even without execution feedback (up to ~+12% where tests exist) [1]. The diagnosis is therefore an explicit, evidence-citing artifact — it names the cause and points at the observations that support it — never a step buried inside the repair prompt where it cannot be audited, cached, or reused. **P7 — Gate repair on diagnosis.** Withhold edit tools until the analysis passes a citation check — every claim it makes is backed by a cited observation, and the symptom/immediate/root hierarchy is populated. RepairAgent's sequencing — the model must state what and why before it earns edit tools [2] — is the warrant. And the gate has an escalation: if two consecutive diagnoses fail the citation check or the downstream repair fails to resolve the failure, the harness escalates to instrumentation (component 12) to gather more evidence rather than letting the model guess a third time. ## 4. The Architecture The pipeline is a sequence of stages; every node is labeled D (deterministic software) or L (a learned model). The invariant that governs the whole is: *the L column contains exactly one entry — a single weak-LLM causal-hypothesis-generation call — and every upstream and downstream step is deterministic software on CPU.* The concrete inputs are the raw signals from the failed run and the evidence pack from component 12; the output is a `Diagnosis` artifact that component 09 and the repair stages consume. The flow, in the five deterministic stages (a) through (e) plus the one learned entry (L): 1. **(a) Signal parsing [D].** Each raw signal — compiler diagnostic, exception traceback, assertion message, test report — is parsed deterministically per toolchain into a typed structured record carrying its class, message, frames, and actual-versus-expected values (P1). This is the parse-don't-paste step: a compiler message becomes a structured compiler-error record, an exception becomes an exception record with its deepest non-library frame, an assertion becomes an assertion record with the expected and the observed value. 2. **(b) Failure classification [D].** The parsed records are mapped deterministically onto the failure taxonomy — syntax, type, null, off-by-one, async-order, env-config, concurrency (P3). Classification branches on what evidence is relevant; it is not a report but a routing decision that tells the downstream stages which observations would discriminate among causes. 3. **(c) Cascade suppression and project-frame pinning [D].** The record is reduced to its load-bearing residue: a compiler error is cut back to its earliest root cause with the cascade tail suppressed, an exception is pinned to its deepest non-library frame (project code, not framework internals), and an assertion is reduced to its actual-versus-expected extraction plus the test's intent (P2). This is Becker's editing of the error surface [13] made deterministic: the model never sees the misdirecting raw tail. 4. **(d) Evidence normalization [D].** Every observation the surviving records and component 12's evidence pack carry is normalized into a typed observation — a `DiagnosticEvidence` record with schema { source_type, source, timestamp, entity, observation, confidence }: source_type names the signal kind (compiler/exception/assertion/trace/test), source names the file and line or symbol, timestamp ties it to the exact run, entity names the observed program entity, observation holds the normalized fact, and confidence marks how directly it was observed. This is the only currency diagnosis may cite (P5). 5. **(L) Causal-hypothesis generation [L] — the single learned entry.** The weak model proposes the competing root-cause hypotheses for the classified failure, and for each hypothesis it states the observation that would discriminate it from the others — the exact prediction component 12's instrumentation should test. This is AutoSD's scientific loop [4] collapsed into one bounded call and shaped by the strong Piper rule: the model does not assert "X caused this"; it proposes "if X caused this, then observing Y would prove it, and observing not-Y would refute it." The output feeds the instrumentation of component 12, which tests the discriminating observations and returns evidence that falsifies the unsupported hypotheses. 6. **(e) Diagnosis-artifact assembly [D].** The surviving, evidence-backed reasoning is assembled into the `Diagnosis` artifact with schema exactly: { failure_class, symptom, failing_entity, immediate_cause, root_cause_hypothesis, causal_chain, supporting_evidence[], contradictory_evidence[], confidence, unresolved_questions[] }. failure_class is the taxonomy label from (b); symptom, immediate_cause, and root_cause_hypothesis populate the three-level hierarchy of Section 1 (P4); causal_chain records the cause-effect path from root to symptom (Zeller's lineage [15]); supporting_evidence[] and contradictory_evidence[] hold the `DiagnosticEvidence` records that argue for and against the root-cause hypothesis — the contradictory list may be empty, but if instrumentation returned evidence against a hypothesis, that hypothesis cannot survive into root_cause_hypothesis; confidence is a calibrated number that reflects how much of the chain is directly observed versus inferred; and unresolved_questions[] records what the diagnosis does not yet know, so component 09 repairs with explicit awareness of residual uncertainty rather than false certainty. The primitive set the harness exposes is deterministic except where noted: `parse_signal` (turn a raw signal into a typed structured record), `classify_failure` (map records onto the failure taxonomy), `triage` (suppress the cascade tail and pin project frames), `normalize_evidence` (turn observations into `DiagnosticEvidence` records), `hypothesize_causes` (the single learned call proposing competing root-cause hypotheses with their discriminating observations), and `assemble_diagnosis` (build the final `Diagnosis` artifact). The target is explicit and checkable: stages (a)–(e) are software on CPU, and the L column holds exactly one entry — the causal-hypothesis-generation call that decides what component 12 should test. Any proposed addition that would put a language model elsewhere in the pipeline is a regression against P1–P7 and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of Section 6, on held-out repositories, at a defensible cost, or be rejected. The invariant is stated again so it can be enforced mechanically: **the L column contains exactly one entry.** ## 5. Recursive Self-Improvement Protocol Component 13 of 27 follows the program's uniform protocol: the executable specification is written to 100%, round 1 lands at approximately 80% of the specified behavior, and later rounds close the gap. Each round is A/B-gated against the honest metrics of Section 6 — never the end-to-end resolve rate alone, which would let a lucky patch mask a shallow diagnosis. Because this component's deterministic stages (a)–(e) carry most of the load, their improvement is mostly direct engineering, and diagnosis evidence feeds back to improve the harness itself in three concrete loops. First, *diagnosis-accuracy tracking*: the harness records whether the root_cause_hypothesis actually named the site the downstream fix had to touch — a hypothesis whose fix site was elsewhere is a wrong diagnosis even if the patch happened to resolve the failure by accident — and over rounds the classification and triage stages are tuned to push the hypothesis toward the true site. Second, *failure-class calibration*: the harness measures which failure classes drive resolution versus which routinely end in escalation to instrumentation or a failed repair, and spends its diagnosis budget on the classes where a correct diagnosis most improves the odds — DebugBench's finding that execution feedback helps some bug types and not others [7] is the warrant that classes are not interchangeable and must be tracked separately. Third, *evidence-citation-gate tightening*: the gate is tightened by measuring how often a candidate diagnosis cites an observation not present in the evidence pack, and how often a hypothesis survives despite contradictory evidence that should have eliminated it — the 44.07% wrong-diagnosis class [6] is the standing reminder that misreading signals is the dominant failure, so the gate's coverage is widened each round until no unsupported claim reaches the repair stage. Because every `Diagnosis` artifact is versioned and tied to the evidence pack that produced it, a regression in interpretation quality is visible in the diagnosis logs long before it surfaces as a failed resolve. ## 6. Measurement & Evaluation Error interpretation is measured *separately* and *honestly*, because a stage whose whole purpose is to convert raw signals into a grounded cause can be defeated by the harness or the model declaring a cause it never proved. The instruments are built to keep the diagnosis accountable to the evidence it cites. **Diagnosis accuracy.** The primary instrument asks whether the diagnosis is *right about where the fix goes*: given a resolved failure, did the root_cause_hypothesis name the actual site the repair had to change — or did it name a plausible but wrong site that a lucky patch masked? The instrument separates diagnosis quality from repair quality, mirroring DSDBench's diagnosis-not-repair measurement stance [5], and it reports the fraction of resolved failures whose stored diagnosis pointed at the true fix site, not merely the resolve rate. **The evidence-citation gate.** The companion instrument audits whether the diagnosis cites observed values only: every claim in supporting_evidence[] and contradictory_evidence[] is checked against the `DiagnosticEvidence` records actually present, and any diagnosis asserting a value the harness never observed is flagged as fabricated — the mechanical enforcement of P5, grounded in CRITIC's external-signal requirement [3] and Huang's no-self-correction result [14]. **Failure-mode audit.** Beyond the positive instruments, the harness audits the specific ways an interpretation layer can lie. *Cascade-drowning* is when the tail of a compiler or framework error buries the earliest root cause, so the model reads the cascade instead of the trigger — the failure Becker's error-surface editing exists to prevent [13]. *Frame-blindness* is when a diagnosis fastens on a library frame or the traceback's last line rather than the project frame where the defect lives. *Imagined values* is when a diagnosis asserts an unobserved variable value, the fabrication the citation gate rejects. *Diagnosis theatre* is when the model produces a confident, well-structured diagnosis that is entirely ungrounded — the articulate-but-wrong failure that the 44.07% wrong-diagnosis class [6] describes. *Message-as-truth* is when the model reads the label on the error rather than the error behind it [6] — the regression study's central indictment. *Stale diagnosis* is when a diagnosis from an earlier run is reused against a run whose signals changed, so it cites evidence from a different execution. *Truncated signal* is when the raw signal was cut before parsing and the diagnosis reasons over a partial record — the failure LogDx-CI's aggressive compressors caused by stripping assertion, snapshot, and compiler blocks [9]. Each mode has a guard: cascade suppression and project-frame pinning (stage c) audit against cascade-drowning and frame-blindness; the citation gate rejects imagined values and message-as-truth; diagnosis accuracy on resolved runs surfaces theatre; versioning ties each diagnosis to its run and catches stale diagnoses; and the parser logs truncation events so a partial signal is flagged rather than silently diagnosed. Two honest caveats close the section. First, the strong-model numbers of Section 2 — the human-study explanation gains on 5 of 6 real bugs [4], RepairAgent's order-of-a-dime-per-repair cost [2], RustAssistant's ~74% on real compile errors [12] — will NOT reproduce on a weak model; they were achieved with strong models and at scale, and a weak-model harness should expect materially lower absolute diagnosis and repair quality. Its interpretation stage exists precisely so that weak causal *reasoning* is compensated by strong deterministic *parsing, classification, triage, and falsification* — and the honest instruments here, above all diagnosis accuracy and the evidence-citation gate, are what will show whether that compensation is real. Second, all interpretation metrics must be reported on held-out and out-of-benchmark repositories, exactly as in the earlier components, so that a model that has memorized a repository's error signatures cannot score well without its diagnoses having been grounded in evidence the harness actually parsed and observed. ## 7. Conclusion Error interpretation is the conversion layer of the weak-model coding harness: the stage that sits between component 12's instrumentation — the evidence that records the run behind the verdict — and component 09's patch generation, which must repair from that evidence. This paper has argued that interpreting an error should be an evidence-driven causal-diagnosis stage, not a prompt asking an LLM to explain stderr — classify the failure, normalize all evidence into typed observations, distinguish symptom from immediate cause from root cause, generate competing hypotheses, derive the observations that discriminate between them, falsify the unsupported explanations with execution and instrumentation, and emit an explicit provenance-backed diagnosis with calibrated confidence before any repair is attempted. It has reviewed the evidence that makes this stance load-bearing — the Self-Debug gains from explain-before-fix [1], RepairAgent's diagnosis-gates-repair discipline [2], CRITIC's and Huang's grounding requirements [3][14], AutoSD's scientific loop and its human-study gains on 5 of 6 bugs [4], and the 44.07% wrong-diagnosis rate that names misreading the error the dominant failure [6] — and it has designed against the failure modes that would let the harness declare a cause it never proved, from cascade-drowning to diagnosis theatre to message-as-truth. The architecture is deterministic end to end with exactly one learned entry: stages (a)–(e) — signal parsing, failure classification, cascade suppression and project-frame pinning, evidence normalization, and diagnosis-artifact assembly — are software on CPU, and the single L entry is causal-hypothesis generation, the one bounded call in which the model proposes competing root causes and the observation that would discriminate each, which component 12 then tests. The principle that has governed every choice is the one with which this paper began and closes, stated as the strong rule for Piper: **Never ask "what do you think caused this?" when the harness can instead ask "what observation would prove or disprove each possible cause?"** The harness parses, classifies, and falsifies deterministically; the model reasons over the tiny residue and proposes the predictions that settle it — because for a weak model, the only diagnosis worth acting on is the one the evidence can prove. --- ## References 1. Xinyun Chen, Maxwell Lin, Nathanael Schärli, Denny Zhou, and others. *Teaching Large Language Models to Self-Debug.* arXiv:2304.05128. *(Rubber-duck step is diagnosis generation; explain-before-fix; explanation WITHOUT execution feedback still yields large gains — up to ~+12% where tests exist; the diagnosis must be an explicit, checkable, cacheable artifact.)* 2. Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. *RepairAgent: An Autonomous, LLM-Based Agent for Program Repair.* 2024. arXiv:2403.17134. *(Diagnosis-gates-repair; curated ~14-tool prompt stack mirroring analyse-failure → explain-bug → localise → edit → validate; failure-analysis prompts load-bearing — the model states what/why before earning edit tools; consistent gains over one-shot baselines at very low cost, ~a dime per repair.)* 3. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. *CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing.* ICLR 2024. arXiv:2305.11738. *(LLMs cannot reliably self-critique without EXTERNAL signals; tool-grounded critique gives consistent gains on reasoning + code; diagnosis must interact with observed evidence, not the model's own confidence.)* 4. Kang, Chen, Yoo, and Lou. *AutoSD.* Preprint arXiv:2304.02195; published EMSE 2025, volume 30, Springer. *(NOTE: the circulating "Kang, Yoon & Yoo / TOSEM" attribution is WRONG. Scientific debugging as an explicit hypothesis → prediction → experiment → observation → conclusion loop before patch generation; human study of 20 developers; explanations improved patch-correctness judgment on 5 of 6 real bugs.)* 5. Yang, et al. *DSDBench.* EMNLP 2025 Oral. arXiv:2503.22388. *(Measures diagnosis not repair; multi-hop error tracing + multi-bug detection with an all-bugs-for-credit rule; models degrade as the number of hops rises — the paraphrase, not "sharply".)* 6. Ho, Le-Cong, Le, and Rizkallah. *Regression Errors.* arXiv:2506.13182. Minor revision at TOSEM. *(Failure taxonomy; wrong diagnosis is the largest failure class at 44.07%, driven by misdirection from test names/error signals; weak test suites 23.73%; "the model reads the label on the error, not the error.")* 7. Tian, et al. *DebugBench: Evaluating Debugging Capability of Large Language Models.* ACL Findings 2024. arXiv:2401.04621. *(NOTE: the source's 2401.03426 ID is WRONG. ~4,000+ debugging instances across four bug types, leakage-controlled; even GPT-4 resolves ~1/3 without scaffolding; execution feedback materially improves debugging but is NOT uniformly beneficial; small models collapse.)* 8. Riddell, et al. *Stalled/Biased/Confused.* University of Waterloo, FORGE 2026. arXiv:2601.22208. *(Stalling and bias-anchoring are the discrete named failure classes; "confusion" is a paraphrase, not a class.)* 9. *LogDx-CI.* v1.2 preprint. arXiv:2605.28876. *(Flag: single-author, AI-drafted-ground-truth, 35-case preprint. Aggressive compressors strip assertion/snapshot/compiler blocks; a hybrid grep+tail with a ~120k-character threshold wins at ~$0.03/case.)* 10. *OpenRCA.* ICLR 2025, OpenReview M4qNIzQYpd. No arXiv ID. *(Do NOT confuse with OpenRCA 2.0 at arXiv:2606.27154. 335 enterprise failures, >68 GB telemetry; best RCA agent ~11.34% (Claude 3.5).)* 11. Bai, Hamad, Ho, Suhaib, and Ren. *...of Formal Verification Failures (FVDebug).* NVIDIA. arXiv:2510.15906. *(Actual title ends "of Formal Verification Failures"; causal-graph + for-and-against prompting + insight rover; per-node symptom-vs-root-cause verdicts.)* 12. Pantazis Deligiannis, et al. *RustAssistant: ...* ICSE 2025. arXiv:2308.05177. *(ICSE title adds the prefix "RustAssistant:"; compiler-error-specific diagnosis; ~74% on real Rust compile errors.)* 13. Brett A. Becker. *Enhancing Compiler Error Messages.* SIGCSE '16 (not ICER). DOI 10.1145/2839509.2844584. Not on arXiv. *(The outlier: the information content of an error message causally changes debugging success, so the harness may EDIT the error surface before the model sees it.)* 14. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. *Large Language Models Cannot Self-Correct Yet (LLMs Cannot Self-Correct).* ICLR 2024. arXiv:2310.01798. *(The no-external-signal result under the whole component's feet: without external feedback, self-correction does not reliably improve output.)* 15. Andreas Zeller. *Isolating Cause-Effect Chains from Computer Programs.* FSE 2002. DOI 10.1145/587051.587053. *(A failure has a causal chain through state — the lineage of this component's causal_chain field.)* 16. Andreas Zeller. *Yesterday, My Program Worked. Today, It Does Not. Why?* ESEC/FSE 1999. DOI 10.1145/318774.318946. *(Delta debugging: isolating the minimal change responsible for a failure — the falsifiable-reduction ancestor of diagnosis-by-observation.)*