Summary / 13. Error interpretation and debugging / Document
Spec 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 13 — Error Interpretation / Debugging (the ≈80% fraction) > Derived from `papers/13-error-interpretation-debugging.md` — "Read the Error Like Evidence, Not Like > Noise." This document drives the build of the **error-interpretation/debugging stage**: the > deterministic, evidence-driven, causal-diagnosis **conversion layer** that sits between component 12's > instrumentation — the evidence that records the run behind the verdict — and component 09's patch > generation, which must repair from that evidence. It is the interpreter between the runtime's complaint > and the repair loop's input: it parses, classifies, triages, normalizes, and causally diagnoses the > failure so downstream repair never reasons from raw stderr. Where execution (11) emits signals and > instrumentation (12) records the run behind them, this component is the conversion layer — classify → > parse → triage → normalize → hypothesize → assemble — producing an explicit, provenance-backed `Diagnosis` > artifact, with calibrated confidence, before any repair is attempted. The guiding rule is the paper's > strong rule for Piper, made mechanical: **never ask "what do you think caused this?" — ask "what > observation would prove or disprove each cause?"** The first question invites the model to narrate a > cause it cannot verify; the second converts diagnosis into a falsifiable prediction the harness tests with > component 12's instrumentation, so the model's job becomes proposing discriminative observations rather > than asserting unobserved causes. A symptom is not an immediate cause is not a root cause: the harness > records the *symptom* (what the runtime surfaces), names the *immediate cause* (the direct trigger the > error exposes), and keeps digging for the *root cause* (the upstream defect whose repair removes the > symptom) — a diagnosis that stops at the immediate cause repairs the surface and leaves the defect. This > spec translates the paper's design into exact primitives, the exact `DiagnosticEvidence` and `Diagnosis` > schemas, the exact failure taxonomy, exact per-stage input/output/schema/caps, exact L-column caps and > firing conditions, exact A/B instruments and thresholds, exact ablation mechanics, and an exact build > sequence — no further research required. Where the paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible error-interpretation/debugging stage** — a deterministic-first, evidence-driven, causal-diagnosis conversion layer (signal parsing → failure classification → cascade suppression + project-frame pinning → evidence normalization → diagnosis-artifact assembly, plus the **single weak-LLM causal-hypothesis-generation call**) that turns the raw signals of a failed run and component 12's evidence pack into the single structured **`Diagnosis` artifact** — populated with the symptom/immediate-cause/root- cause hierarchy and calibrated confidence — that component 09 and the repair stages consume. It is distinct from its predecessor in *kind*, not merely in order: component 12 asks the repository *how it failed* and records the answer as evidence; interpretation asks of that record *what the failure actually is* — the cause — and emits it as a typed, evidence-citing, cacheable artifact. A record is not yet a cause. When a test fails, the runtime emits a signal — a compiler diagnostic, an exception traceback, an assertion message, a test report — and between that raw signal and the repair loop's input there lies an act of interpretation: someone must decide what the failure actually *is*. This stage performs that act as **evidence-driven causal diagnosis rather than as an LLM asked to free-associate over stderr**. Component 09 repairs against the diagnosis, never against the raw message. The stage is governed by seven principles from the paper, restated because they decide every design choice below. **P1 Parse, don't paste** — never hand raw stderr to the model; every signal is converted deterministically into a typed structured record (class, message, frames, actual-vs-expected). **P2 Triage by signal schema** — different signal types demand different first readings (compiler → earliest root cause, cascade tail suppressed; exception → deepest non-library frame; assertion → actual-vs-expected plus test intent). **P3 Classify into a failure taxonomy** — the failure is classified into a schema so the class label routes what evidence to gather next; classification is not a report card but a branch on what to observe. **P4 Distinguish symptom from immediate cause from root cause** — the three-level hierarchy is non-negotiable, and each claim says which level it sits at. **P5 Ground before believing** — a diagnosis asserting an unobserved value is rejected by construction; every cited observation must be present in the evidence (CRITIC's external-signal requirement, Huang's no-self-correction result). **P6 Explain-then-fix** — Self-Debug's ordering enforced mechanically: the diagnosis is an explicit, evidence-citing artifact produced *before* any repair is considered. **P7 Gate repair on diagnosis** — edit tools are withheld until the analysis passes a citation check, with escalation to instrumentation after repeated failed diagnoses. **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — signal parsing (Python toolchain), failure classification, cascade suppression + project-frame pinning, evidence normalization, and diagnosis-artifact assembly with its three-level hierarchy and confidence calibration — plus the *bound* of the *one* causal-hypothesis-generation call. The heavy pieces (a trained causal-hypothesis generator, multi-toolchain parsers for C/C++/Rust/Java compiler diagnostics, OpenRCA-style system-level root-cause analysis over heterogeneous telemetry — logs/metrics/traces — and Becker-style error-message *rewriting* that edits the error surface) are **not** built in Round 1 and are **not excluded** — later-round content climbing toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by **diagnosis accuracy** and the **evidence-citation gate** — never the end-to-end resolve rate alone (§7). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element is classified below by *which round* builds it. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–5 plus the bounded stub of row 6). - **DEFERRED = later rounds** — the trained-generator / multi-toolchain-parser / system-RCA / error-surface-editing pieces (rows 7–10). They are **not excluded** — they are the content of later rounds, gated by the same §7 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | (a) Signal parsing — deterministic per-toolchain conversion of raw signals (compiler diagnostic / exception traceback / assertion message / test report) into typed structured records | **BUILD-NOW** | | 2 | (b) Failure classification — deterministic mapping of parsed records onto the failure taxonomy; classification branches on what evidence to gather next | **BUILD-NOW** | | 3 | (c) Cascade suppression + project-frame pinning — compiler error → earliest root cause with cascade tail suppressed; exception → deepest non-library frame; assertion → actual-vs-expected + test intent | **BUILD-NOW** | | 4 | (d) Evidence normalization — every surviving observation normalized into the `DiagnosticEvidence` typed observation; the only currency diagnosis may cite | **BUILD-NOW** | | 5 | (e) Diagnosis-artifact assembly — the `Diagnosis` artifact with the symptom/immediate/root hierarchy, causal chain, evidence lists, calibrated confidence, unresolved questions | **BUILD-NOW** | | 6 | (L) Causal-hypothesis generation — **the single learned entry**, included as a *bounded stub* (cost rule below) | **BUILD-NOW** (bounded stub only) | | 7 | **Trained causal-hypothesis generator at scale** | **DEFERRED** — Round 1 ships only the bounded stub (§4); full training/calibration deferred | | 8 | **Multi-toolchain parsers** — C/C++/Rust/Java compiler diagnostics | **DEFERRED** — Round 1 parses the Python toolchain only (§3.1) | | 9 | **System-level root-cause analysis** — OpenRCA-style heterogeneous telemetry (logs/metrics/traces) over >68 GB of enterprise telemetry | **DEFERRED** — Round 1 diagnoses single-run signals + component 12's evidence pack (§3) | | 10 | **Becker-style error-message REWRITING** — editing the error surface (rephrasing/trimming a misdirecting message before the model sees it) | **DEFERRED** — Round 1 suppresses the cascade tail deterministically but does not *rewrite* messages (§3.3) | **The cost rule (explicit).** The paper's single learned entry — causal-hypothesis generation — is the one L-column entry this architecture permits. In Round 1 it is included as a **bounded stub**: the stage exists, wired into the pipeline at its firing point (§4), gated, feeding component 12's instrumentation — never auto-promoted — but its full training and calibration (the paper's §5 self-improvement loop) is **deferred**. The stub's *cost* is bounded before invocation: one call, temperature zero, schema-constrained output, a bounded payload (≤ 4 hypotheses), parse-failure fallback to "no hypothesis" (§4 caps). It is kept **only if** the §7 no-hypothesis-vs-hypothesis ablation shows it improves diagnosis accuracy on held-out repositories; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the interpretation path by default — the full deterministic pipeline (rows 1–5) plus the *bounded stub* of the single hypothesis call (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a compiler-surface parser, an RCA telemetry surface, or break the one-entry L column — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out repositories, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Five stages, all **D** through the emission of the assembled `Diagnosis` artifact. The LLM column holds exactly one thing: causal-hypothesis generation (§4), which feeds component 12's instrumentation, whose returned evidence feeds stage (e). The pipeline: ``` RAW SIGNALS (failed run: compiler diagnostic / exception traceback / assertion message / test report) + COMPONENT 12 EVIDENCE PACK → (a) signal parsing [D] → typed structured records (class/message/frames/actual-vs-expected) → (b) failure classification [D] → taxonomy label; branches on what evidence to gather → (c) cascade suppression + project-frame pinning [D] → the load-bearing residue (earliest root / deepest non-library frame / actual-vs-expected) → (d) evidence normalization [D] → DiagnosticEvidence typed observations (the only citable currency) → (§4) causal-hypothesis generation [L — the ONE entry; feeds component 12's instrumentation] → (e) diagnosis-artifact assembly [D] → the Diagnosis artifact (hierarchy + evidence + calibrated confidence) ``` The pipeline reasons over **observed evidence only** — the normalized `DiagnosticEvidence` records. Nothing about the failure reaches the model except through those records; raw stderr is never pasted (§8 invariant 4). The failure is **diagnosed, never narrated**. All artifacts land in `build/13-error-interpretation-debugging/`. Nothing writes into the live harness. ### 3.1 (a) Signal parsing (D) **Input:** the raw signals of the failed run — a compiler diagnostic, exception traceback, assertion message, or test report — plus the component 12 evidence pack. **Output:** **typed structured records** — each raw signal converted deterministically per toolchain into a record carrying its class, its message, its frames (with project vs library flagged), and its actual-vs-expected values. **No language model runs here.** This is the parse-don't-paste step (P1): a compiler message becomes a structured compiler-error record, an exception becomes an exception record with its deepest non-library frame, an assertion becomes an assertion record with the expected and the observed value, a test report becomes a test record with the failing assertion and the test's intent. Becker's finding that the information content of an error message causally changes debugging success [13] is the warrant that the message is the highest-density token budget in the loop, and parsing extracts that density instead of burying it in a wall of unread text. Round 1 parses the **Python toolchain only** (§2 row 8) via stdlib `re`, `ast`, and `traceback`; multi-toolchain parsers are deferred. **Parsing rule (exact, deterministic per toolchain):** each raw signal is matched against the toolchain's signal grammar and converted into the typed record; an unmatchable fragment is recorded as a truncated-signal event (flagged, never silently dropped), so a partial signal is surfaced rather than silently diagnosed (§7.5 truncated-signal). **Output schema (exact):** ```json { "record_type": "compiler | exception | assertion | test", "class": "…", // the named error/exception class or category "message": "…", // the trimmed message text (cascade tail NOT yet suppressed) "frames": [ { "file": "…", "line": int, "function": "…", "in_project": bool } ], "actual": "…", "expected": "…", // actual-vs-expected where the signal carries both (assertion) "source": "<raw signal provenance>" } ``` **Caps:** one typed record per raw signal; the record's fields are bounded (message trimmed to a stream cap). This stage produces structured records, not diagnoses — stage (b) decides the class. ### 3.2 (b) Failure classification (D) **Input:** the parsed records from (a). **Output:** the failure's **taxonomy label**. **No language model runs here.** Classification is a **deterministic mapping from parsed records onto the failure taxonomy** — not an inference (P3). **The failure taxonomy (exact — EXACTLY this set):** `{ syntax, type, null, off-by-one, async-order, env-config, concurrency }`. A parsed record is mapped to exactly one of these seven labels by deterministic rules over its class/message/frames. **Classification branches on what evidence to gather next (exact):** the label is not a report card but a routing decision — it tells the downstream stages *which observations would discriminate among causes*. A `null` class routes evidence gathering to the frame where the null is dereferenced and the field's assignment site; a `type` class routes it to the call sites where the mismatched value entered; an `async-order` class routes it to ordering/race evidence; an `env-config` class routes it to config and environment state. DebugBench's finding that execution feedback helps some bug types and not others [7], and the 44.07% wrong-diagnosis rate driven by misreading signals [6], are the warrants: a class label tells the harness what evidence would discriminate causes, so classification is a branch, never a destination. **Output schema (exact):** ```json { "failure_class": "syntax | type | null | off-by-one | async-order | env-config | concurrency", "classification_basis": "<the deterministic rule + records that produced the label>", "evidence_routing": "<what this class routes evidence gathering toward>" } ``` **Caps:** exactly one `failure_class` per diagnosis (a failed run is classified once). Classification is pure deterministic software on CPU — no model decides the class. ### 3.3 (c) Cascade suppression + project-frame pinning (D) **Input:** the classified records from (b). **Output:** the **load-bearing residue** — the record reduced to what the diagnosis may actually reason over. **No language model runs here.** This is P2's per-signal triage and Becker's editing of the error surface [13] made deterministic — except that Round 1 *suppresses* the cascade tail and *pins* project frames; it does **not** rewrite message text (that is the deferred Becker-style rewriting, §2 row 10). **Triage rules (exact, deterministic per signal schema):** 1. **Compiler error → earliest root cause, cascade tail suppressed.** The record is cut back to the earliest root cause; the cascade tail — the chain of downstream errors a single defect triggers — is suppressed, so the model never reads the misdirecting raw tail (RustAssistant's ~74% on compiler errors [12] is evidence a tool specialized to one signal schema beats generic error reading). 2. **Exception → deepest non-library frame.** The record is pinned to its deepest **non-library** frame — the project code where the user's code actually diverged, not framework internals and not the traceback's last line. 3. **Assertion → actual-vs-expected extraction + test intent.** The record is reduced to its actual-vs-expected values plus the test's *intent* — the target of the assertion, not its mechanics. **Output schema (exact):** ```json { "primary_entity": "<file:line | symbol the failure is pinned to>", "frame_kind": "earliest_root | deepest_non_library | assertion", "suppressed_cascade_tail": int, "project_frames": [ "<pinned project frames>" ], "actual": "…", "expected": "…", // present when an assertion carries them "test_intent": "…" } // present for assertion/test records ``` **Caps:** the residue is strictly smaller than the (a) record — cascade tail and library frames are removed, never retained as evidence. Frame-blindness and cascade-drowning audits (§7.5) check this stage. ### 3.4 (d) Evidence normalization (D) **Input:** the surviving residue from (c) plus the component 12 evidence pack. **Output:** the normalized **`DiagnosticEvidence`** typed observations — **the only currency diagnosis may cite** (P5). **No language model runs here.** Every observation the surviving records and component 12's evidence pack carry is normalized into a typed observation; nothing is inferred, interpolated, or synthesized into the set. This is CRITIC's external-signal requirement [3] and Huang's no-self-correction result [14] made mechanical: the only admissible grounds for believing a cause are these typed observations normalized from real signals. **The `DiagnosticEvidence` schema (exact — EXACTLY these fields):** ```json { "source_type": "compiler | exception | assertion | trace | test | log | metric | config | runtime_state", "source": "…", // file:line or symbol the observation came from "timestamp": "…", // ties the observation to the exact run "entity": "…", // the observed program entity "observation": "…", // the normalized fact "confidence": 0..1 } // how directly it was observed ``` **Field meanings (exact):** `source_type` names the signal kind — from the enumerated set of nine; `source` names the file/line or symbol; `timestamp` ties it to the exact run (component 12's run provenance); `entity` names the observed program entity; `observation` holds the normalized fact; `confidence` marks how directly it was observed (a directly captured value scores high; a reconstruction scores lower). Every record is an **observed** fact — nothing enters normalization that a probe or parser did not actually capture. **Caps:** exactly the surviving, observed residue is normalized — never the raw signal text, never the raw event store. The `DiagnosticEvidence` set is the immutable evidence pack the citation gate (§7, §8 invariant 2) checks every downstream claim against. ### 3.5 (e) Diagnosis-artifact assembly (D) **Input:** the classified label (b), the triaged residue (c), the normalized evidence (d), and — when the single hypothesis fires — the surviving evidence-backed causal hypotheses returned from component 12's falsification of the (§4) hypotheses. **Output:** the assembled **`Diagnosis` artifact** — the single currency of this component, the artifact component 09 and the repair stages consume. **No language model runs here.** This is P4 (the three-level hierarchy) and P6 (explain-then-fix as an explicit, evidence-citing artifact) made concrete. **The `Diagnosis` schema (exact — EXACTLY these fields):** ```json { "failure_class": "syntax | type | null | off-by-one | async-order | env-config | concurrency", "symptom": "…", // what the runtime surfaces — the three-level hierarchy level 1 "failing_entity": "…", // file:line or symbol that fails "immediate_cause": "…", // the direct trigger the error exposes — hierarchy level 2 "root_cause_hypothesis": "…", // the upstream defect whose repair removes the symptom — level 3 "causal_chain": [ "…" ], // the cause-effect path from root to symptom (Zeller's lineage) "supporting_evidence": [ "<DiagnosticEvidence records>" ], "contradictory_evidence": [ "<DiagnosticEvidence records>" ], // may be empty "confidence": 0..1, // calibrated — reflects observed vs inferred chain "unresolved_questions": [ "…" ] } ``` **Field meanings (exact):** `failure_class` is the taxonomy label from (b); `symptom`, `immediate_cause`, and `root_cause_hypothesis` populate the three-level hierarchy of §1 (P4) — the hierarchy is populated, so no diagnosis stops at the immediate cause; `failing_entity` names the file/line or symbol; `causal_chain` records the cause-effect path from root to symptom (Zeller's lineage [15][16]); `supporting_evidence[]` and `contradictory_evidence[]` hold the `DiagnosticEvidence` records arguing for and against the root-cause hypothesis — the contradictory list **may be empty**, but if instrumentation returned evidence against a hypothesis, that hypothesis **cannot survive into `root_cause_hypothesis`** (a hypothesis contradicted by instrumentation is dead however plausible it reads); `confidence` is a **calibrated number (0..1)** that reflects how much of the chain is directly observed versus inferred (an entirely observed chain scores toward 1; a chain relying on the single hypothesis stub's unverified claims scores lower); and `unresolved_questions[]` records what the diagnosis does not yet know, so component 09 repairs with explicit awareness of residual uncertainty rather than false certainty. **The evidence-citation gate (exact, run at assembly, D):** every claim in `supporting_evidence[]` and `contradictory_evidence[]` must be **present in the `DiagnosticEvidence` records** from (d). A claim whose observation is not in the set is a **fabrication**, and the artifact is rejected at assembly — the harness cannot declare a cause it never proved (P5, §7.2, §8 invariant 2). Only artifacts that pass this gate proceed to the repair stage, and only then do edit tools unlock (§8 invariant 6). **Output schema (exact):** ```json { "diagnosis": { "<the Diagnosis schema above>" }, "evidence_pack_id": "…", // ties the diagnosis to the exact evidence pack + run "hierarchy_populated": true, // symptom/immediate/root all present "citation_gate": true } // every cited observation present in the evidence records ``` **Caps:** exactly one `Diagnosis` per diagnosis run, assembled from the (d) evidence set (never the raw signal). The artifact is immutable once assembled and tied to the evidence pack + run that produced it (§8 invariant 7); a later re-run is a new artifact. **The L column holds exactly one entry.** Stages (a)–(e) are all deterministic software on CPU. The single learned node is stage (§4), causal-hypothesis generation, which feeds component 12's instrumentation; the surviving, falsified hypotheses return to stage (e) as evidence. Any proposed addition that would put a language model elsewhere in the pipeline — parsing, classifying, triaging, normalizing, assembling or gating the artifact, narrating raw stderr — is a regression against P1–P7 and the L-column invariant, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of §7, on held-out repositories, at a defensible cost, or be rejected (paper §4). --- ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **causal-hypothesis generation**. The weak model proposes the **competing root-cause hypotheses** for the classified failure, and for each hypothesis it states the observation that would discriminate it from the others — the exact prediction component 12's instrumentation should test. This is AutoSD's scientific loop [4] collapsed into **ONE bounded call** and shaped by the strong Piper rule: the model does not assert "X caused this"; it proposes "if X caused this, then observing Y would prove it, and observing not-Y would refute it." The output feeds component 12's instrumentation, which tests the discriminating observations and returns evidence that falsifies the unsupported hypotheses. Firing is re-considered only when a new failing candidate needs a diagnosis. **Prompt inputs (assembled deterministically, in this exact order):** 1. **`failure_class`** — the taxonomy label from stage (b). 2. **`residue`** — the triaged residue from stage (c): the primary entity, pinned project frames, actual-vs-expected where present. 3. **`evidence`** — the normalized `DiagnosticEvidence` set from stage (d), in brief (the only evidence the model may cite). 4. **`existing_tests`** — the authoritative tests already present, in brief. **Input → output schema (exact):** ```json { "hypotheses": [ { "hypothesis_id": "H1", "claim": "…", // what the model believes the root cause may be "discriminator_observation": "…", // the observation that would prove/disprove this claim "predicted_outcome": "…" } // what value/state would confirm or falsify it ] } ``` **Caps:** **ONE call**, `temperature = 0`, `max_completion_tokens = 2048`; a bounded number of competing root-cause hypotheses (**≤ 4 per firing**), each naming a single discriminating observation. The output is schema-constrained — the harness, not the model, decides what a hypothesis means. The output feeds component 12's instrumentation, which tests the discriminating observations and returns falsifying evidence to stage (e). **Gating (exact):** the model proposes hypotheses; the harness **instruments and falsifies** deterministically via component 12. A proposed hypothesis is **never treated as an observed fact** — its `predicted_outcome` is a prediction to be confirmed or falsified by component 12's probe, and a hypothesis whose outcome contradicts the observed evidence is **eliminated** (§3.5) — a hypothesis contradicted by instrumentation cannot survive into `root_cause_hypothesis`, not believed on the model's say-so (§8 invariant 3). **Parse-failure fallback (exact):** any JSON parse failure, schema violation, out-of-bound output (more than 4 hypotheses, oversized payload), or empty/malformed output is a **no-op**: the stage generates **no hypothesis** and hands control back to the deterministic path (which assembles a diagnosis from the observed evidence alone, its root-cause hypothesis grounded only in what the evidence supports) — **never a re-prompt, never a second call**. The L column is reached **once** per diagnosis. This is the mechanical enforcement of the one-entry L column and the anti-over-prompting rule. **Re-invocation on a later candidate (exact):** hypothesis generation is re-considered **only** when a *new* failing candidate needs a diagnosis; each invocation is scoped strictly to that candidate and re-uses none of a prior firing's hypotheses. It remains the **single conceptual learned entry** — a bounded, once-per-diagnosis call — **never a model that runs over the whole pipeline**. **Invariant 1 (§8): the L column contains exactly one entry — and that entry is only ever a bounded, gated causal-hypothesis generator whose output component 12 falsifies deterministically.** --- ## 5. Implementation Order Eight phases, each independently testable and reversible, landing in `build/13-error-interpretation-debugging/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Signal parsing (a)** | `parse.py` (§3.1 deterministic per-toolchain parsing of compiler diagnostic / exception traceback / assertion message / test report into typed structured records; Python toolchain; stdlib `re`/`ast`/`traceback`) | Feed raw Python tracebacks, assertion messages, and test reports: each parses to the typed record with class/message/frames (project vs library flagged)/actual-vs-expected; an unmatchable fragment is flagged truncated, never silently dropped; assert **no LLM consulted**. | Delete `parse.py`. | | **2. Failure classification (b)** | `classify.py` (§3.2 the deterministic mapping of parsed records onto the failure taxonomy `{ syntax, type, null, off-by-one, async-order, env-config, concurrency }`; label routes evidence gathering) | Feed classified-fixture records: each maps to exactly one of the seven labels; the label's `evidence_routing` names what evidence it routes toward; same record → same label (deterministic); assert **no LLM consulted**. | Delete `classify.py`. | | **3. Cascade suppression + frame pinning (c)** | `triage.py` (§3.3 per-signal-schema triage — compiler → earliest root cause with cascade tail suppressed; exception → deepest non-library frame; assertion → actual-vs-expected + test intent) | Feed a cascaded compiler record, a library-frame exception record, an assertion record: the compiler record is cut to its earliest root with the tail suppressed; the exception is pinned to its deepest non-library frame (never the library frame or the last line); the assertion is reduced to actual-vs-expected + intent; the residue is strictly smaller than the (a) record; assert **no LLM consulted**. | Delete `triage.py`. | | **4. Evidence normalization (d)** | `normalize.py` (§3.4 normalization of the surviving residue + component 12's evidence pack into the `DiagnosticEvidence` records — the only citable currency) | Feed the (c) residue + a component-12 evidence pack: each surviving observation normalizes to a `DiagnosticEvidence` record with exactly the six fields and a `source_type` from the enumerated nine; nothing inferred or synthesized enters the set; assert **no LLM consulted**. | Delete `normalize.py`. | | **5. Hypothesize-stub (L)** | `hypothesize.py` (§4 the bounded L stub: firing + ONE call + temperature 0 + ≤ 4 hypotheses + schema-constrained output + parse-failure fallback to no hypothesis + output feeding component 12's instrumentation) | With a mocked LLM: the call is **one**, temperature 0, hypotheses ≤ 4, each with `claim`/`discriminator_observation`/`predicted_outcome`; a malformed/over-cap response → no hypothesis, **no second call**; the output feeds component 12 deterministically. | Delete `hypothesize.py`. | | **6. Diagnosis assembly (e)** | `assemble.py` (§3.5 the `Diagnosis` artifact + the three-level hierarchy + `causal_chain` + evidence lists + calibrated confidence + the evidence-citation gate) | `assemble.py` on fixtures: the artifact has exactly the `Diagnosis` fields; the hierarchy is populated (no diagnosis stops at the immediate cause); a hypothesis contradicted by instrumentation is excluded from `root_cause_hypothesis`; the citation gate rejects an artifact citing a claim not in the evidence set; `confidence` ∈ 0..1 reflects observed-vs-inferred. | Delete `assemble.py`. | | **7. Repair gate (P7)** | `gate.py` (§8 invariant 6 the diagnosis-gates-repair mechanism — withhold edit tools until the citation check passes; escalate to component 12 after two failed diagnoses) | Feed a pass-gate diagnosis and a fail-gate diagnosis: edit tools unlock only after the citation check passes and the hierarchy is populated; after two consecutive failed diagnoses the path escalates to instrumentation rather than a third guess; assert **no LLM consulted**. | Delete `gate.py`. | | **8. A/B harness** | `ab.py` (§7 diagnosis-accuracy + evidence-citation-gate capture + ablation runner + no-hypothesis-vs-hypothesis runner + decision) | `ab.py` runs the §7 pipeline-vs-baseline and the §7 ablations, emitting the §7 decision on held-out fixtures. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 8) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of component 11's execution and component 12's instrumentation)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `re` | §3.1 signal-grammar matching for the Python toolchain | CPython stdlib. | **No** | | stdlib `ast` | §3.1 structured analysis of Python source signals | CPython stdlib. | **No** | | stdlib `traceback` | §3.1 parsing exception tracebacks; §3.3 pinning the deepest non-library frame | CPython stdlib. | **No** | | stdlib `json`, `hashlib`, `os`, `sys`, `time` | §3.4–§3.5 schemas, artifact hashing, provenance | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3.1 the Python parser; the candidate runtime | In-repo, pinned interpreter. | **No** | | Component 11 execution substrate | §3.4 re-running under component 12; the failing candidate's run | Built by spec 11; reused, not duplicated. | **No** | | Component 12 instrumentation + evidence pack | §3.4 the evidence to normalize; §4 the falsification of hypotheses | Built by spec 12; reused, not duplicated. | **No** | | Component 01 tool layer / existing test runner | §3.1 the failing test report; §4 the authoritative existing tests | In-repo, already vetted. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single causal-hypothesis call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: signal parsing uses stdlib `re`/`ast`/`traceback`; execution and instrumentation are **reused wholesale from components 11 and 12**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The four DEFERRED rows (§2 rows 7–10) are a trained generator on known architectures, parser surfaces on existing toolchains (C/C++/Rust/Java), an RCA telemetry surface, or an error-surface-rewriting edit path — not new third-party Python installs. Any future trained model download, multi-language parser dependency, RCA telemetry stack, or error-rewriting backend must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — and is out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, and 12** (Python-only, single-repo, no-Docker reproduction) as the source of a held-out **(failing candidate, raw signals, component-12 evidence pack, known fix-site) corpus**: for each instance, a failing candidate with its raw signals and evidence pack, and the mechanically-known **actual fix site** the downstream repair had to change. Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. **Baseline (concrete).** The **raw-stderr-to-LLM** baseline — the naïve path the paper and this component reject: the failing candidate's raw stderr (the untrimmed signal) is handed directly to the model, which is asked to propose a cause and a fix with **no parsing, no classification, no triage, no normalization, no citation gate, no hierarchy**. The structured pipeline must beat *this* on the honest measures below, and the ablation (7.3) removes pipeline mechanisms against those same honest measures. ### 7.1 Diagnosis ACCURACY — the KEY measure The primary instrument, the direct score of the whole interpretation layer: **the fraction of resolved failures whose stored `root_cause_hypothesis` named the ACTUAL fix site** — the site the repair had to change — **not a plausible-but-wrong site a lucky patch masked**. It separates diagnosis quality from repair quality, mirroring DSDBench's diagnosis-not-repair measurement stance [5]: a hypothesis whose fix site was elsewhere is a wrong diagnosis even if the patch happened to resolve the failure by accident (a lucky patch masking a shallow diagnosis is exactly the failure this instrument exists to catch). **This is the KEY measure — the structured pipeline is adopted only if it STRICTLY beats the raw-stderr baseline on diagnosis accuracy AND does not regress resolve (7.2); a resolve win that does not beat the baseline on diagnosis accuracy is not credited.** ### 7.2 The evidence-citation gate + the end-to-end regression guard The companion instrument audits whether the diagnosis cites observed values only: **every claim in `supporting_evidence[]` and `contradictory_evidence[]` is checked against the `DiagnosticEvidence` records actually present**, and any diagnosis asserting an observation the harness never captured is flagged as fabricated — the mechanical enforcement of P5, grounded in CRITIC's external-signal requirement [3] and Huang's no-self-correction result [14]. Reported as the **citation rate** (fraction of cited observations present in the evidence records — **target 1.0**). The coarse end-to-end instrument is kept for parity — **deliberately not** the gate for interpretation changes, because resolve rate cannot separate diagnosis quality from repair quality: 1. **Resolve rate** = `resolved / total` (spec 01's score) — the no-regression guard, never the primary gate. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). 3. **Tokens** = `tokens / instance`. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add an interpretation mechanism because it sounds sensible" mechanically (paper §5, §6). Two passes, everything else held fixed, the delta measured on the honest measures (diagnosis accuracy + the citation gate, plus resolve/cost/tokens): 1. **Remove the triage/normalization stages** — feed the raw (a) records (or, deeper, the raw signal text) instead of the triaged, normalized residue — i.e. remove cascade suppression / frame pinning / evidence normalization — and measure the **diagnosis-accuracy drop**. This is the direct test of P1/P2/P5: does the deterministic reduction measurably improve the diagnosis, or does it degrade toward message-as-truth [6]? 2. **Remove each other mechanism in turn** — the failure-classification routing, the hierarchy population, the citation gate. 3. **Ablate the single L hypothesis step** — run **no-hypothesis vs hypothesis** (never generate a hypothesis vs. generate one) and measure **diagnosis accuracy** on held-out repos. This is the test of whether the bounded hypothesis stub (§4) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L hypothesis stub, which must survive the no-hypothesis-vs-hypothesis ablation on diagnosis accuracy to be kept at all), then to any other mechanism. **Never add an interpretation mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run pipeline-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured pipeline **wins iff**: ``` Diagnosis accuracy STRICTLY beats the raw-stderr-to-LLM baseline (the KEY measure) AND citation rate == 1.0 on the stored diagnoses (the evidence-citation gate holds) AND resolve_pipeline ≥ resolve_baseline (no end-to-end regression) AND cost_pipeline ≤ 1.5 × cost_baseline AND tokens_pipeline ≤ 1.5 × tokens_baseline ``` Diagnosis accuracy must **strictly beat the raw-stderr baseline**; resolve must not regress; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — a candidate that improves resolve but leaves diagnosis accuracy flat against the raw baseline is **not adopted** (a lucky patch would be masking a shallow diagnosis). If the structured pipeline wins, it replaces the raw-stderr path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways the interpretation layer can lie) Beyond the positive instruments, the harness audits the specific ways an interpretation layer can lie, each with a mechanical guard (paper §6): 1. **Cascade-drowning** — the tail of a compiler or framework error buries the earliest root cause, so the model reads the cascade instead of the trigger; guarded by the cascade-suppression audit asserting the residue carries the earliest root with the tail suppressed (stage c). 2. **Frame-blindness** — the diagnosis fastens on a library frame or the traceback's last line rather than the project frame where the defect lives; guarded by project-frame pinning asserting the residue is pinned to the deepest non-library frame. 3. **Imagined values** — the diagnosis asserts an unobserved variable value; guarded by the citation gate rejecting any claim whose observation is not in the `DiagnosticEvidence` set. 4. **Diagnosis theatre** — a confident, well-structured diagnosis that is entirely ungrounded (the articulate-but-wrong failure of the 44.07% wrong-diagnosis class [6]); guarded by diagnosis accuracy on resolved runs, which surfaces a diagnosis whose fix site was wrong. 5. **Message-as-truth** — the model reads the label on the error rather than the error behind it [6]; guarded by parse-don't-paste — raw stderr never reaches the model, so no label can be misread (§8 invariant 4). 6. **Stale diagnosis** — a diagnosis from an earlier run reused against a run whose signals changed, so it cites evidence from a different execution; guarded by versioning that ties each `Diagnosis` to its run (§8 invariant 7). 7. **Truncated signal** — the raw signal was cut before parsing and the diagnosis reasons over a partial record (the failure LogDx-CI's aggressive compressors caused by stripping assertion/snapshot/compiler blocks [9]); guarded by the parser logging truncation events so a partial signal is flagged, never silently diagnosed. ### 7.6 Two honest caveats Two caveats close the section, inherited from the paper's §6. **First, the strong-model-and-scale caveat:** the headline numbers of the paper's review — the human-study explanation gains on 5 of 6 real bugs [4], RepairAgent's order-of-a-dime-per-repair cost [2], RustAssistant's ~74% on real compile errors [12] — were obtained with models far stronger than this harness's target; a weak-model harness should expect materially lower absolute diagnosis and repair quality. Its interpretation stage exists precisely so that weak causal *reasoning* is compensated by strong deterministic *parsing, classification, triage, normalization, and falsification*, and the honest instruments here — above all diagnosis accuracy and the evidence-citation gate — are what will show whether that compensation is real. **Second, the held-out and memorization caution** inherited from the earlier components: all interpretation metrics must be reported on held-out and out-of-benchmark repositories, so a model that has memorized a repository's error signatures cannot score well without its diagnoses having been grounded in evidence the harness actually parsed and observed. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/13-error-interpretation-debugging/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `DIAG=diag13|raw`, default `raw`) routes the A/B; the flag alone switches between the new structured interpretation stage and the raw-stderr-to-LLM baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains exactly one entry** — causal-hypothesis generation (§4), the one bounded, gated call that proposes competing root-cause hypotheses each with its discriminating observation, whose outcome component 12 falsifies. Any phase that introduces a language model elsewhere in the pipeline (parsing, classifying, triaging, normalizing, assembling, gating, narrating raw stderr) is rejected outright. An **empty** L column (the hypothesis stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **No ungrounded diagnosis — the evidence-citation gate.** A diagnosis that asserts an unobserved value is rejected by construction: every cited observation in `supporting_evidence[]`/`contradictory_evidence[]` must be present in the `DiagnosticEvidence` records. A claim not in the evidence set is a fabrication; the harness cannot declare a cause it never proved. 3. **The symptom/immediate-cause/root-cause hierarchy is populated.** No diagnosis stops at the immediate cause: the artifact records the symptom, names the immediate cause the error surfaces, and keeps digging for the root cause whose repair removes the symptom — and says which level each claim sits at. A diagnosis that stops at the immediate cause repairs the surface and leaves the defect; such an artifact is a rejected deliverable. 4. **Parse-don't-paste.** Raw stderr **never reaches the model** — every signal is converted deterministically into typed structured records, and only the normalized `DiagnosticEvidence` residue is citable. A model that sees raw signal text is a rejected deliverable (message-as-truth guard). 5. **Cascade suppression + project-frame pinning.** The diagnosis is reduced to its load-bearing residue: a compiler error is cut to its earliest root cause with the cascade tail suppressed; an exception is pinned to its deepest non-library frame. **Never fix the library frame or the cascade tail** — repairing those leaves the defect and mistakes a consequence for a cause. 6. **Gate repair on diagnosis (P7).** Edit tools are **withheld until the citation check passes** and the hierarchy is populated; a diagnosis that has not named the failing behavior does not earn the right to change anything. The gate has an escalation: if **two consecutive diagnoses** fail the citation check or the repair fails to resolve, the harness **escalates to instrumentation (component 12)** to gather more evidence rather than letting the model guess a third time. 7. **Provenance — every Diagnosis is tied to the evidence pack + run that produced it.** Each artifact is versioned by the evidence pack and run it cites, so a regression in interpretation quality is visible in the diagnosis logs long before it surfaces as a failed resolve; the reader of any diagnosis can reconstruct the exact evidence and run that produced it. A stale diagnosis (citing a different run) is a rejected deliverable. 8. **Classify before diagnose.** The failure is classified into the taxonomy before any diagnosis is attempted, because the class label routes what evidence to gather — a class tells the harness what kind of observation would discriminate causes. A diagnosis attempted without a prior classification (that runs generic "explain the error" reading instead of the class-routed evidence gathering) is a rejected deliverable. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*