Summary / 12. Runtime instrumentation and traces / Document
Spec 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 12 — Runtime Instrumentation / Traces (the ≈80% fraction) > Derived from `papers/12-runtime-instrumentation-traces.md` — "Don't Guess the State, Watch It." This > document drives the build of the **runtime-instrumentation/traces stage**: the deterministic, > hypothesis-driven, selective, **causal evidence layer** that turns the run behind component 11's > verdict into a reducible record a weak model can reason over without imagining it. It sits immediately > after component 11 (execution), which returns a *verdict* — pass/fail — and before the components that > verify and repair (13+), which act on what the verdict points at. Where execution reports *what > happened when I ran it*, instrumentation records *the run behind the verdict* — the paths, values, and > state deltas that a traceback flattens into a single failing line. The guiding rule is the paper's > strong rule for Piper, made mechanical: **Don't give Piper a debugger and tell her to explore. Let Piper > ask a precise question; let the harness instrument reality and return the answer.** This spec translates > the paper's design into exact primitives, exact SBFL formula and probe-point caps, exact per-stage > input/output/schema, the exact `StateObservation` schema, exact L-column caps and firing conditions, > exact A/B instruments and thresholds, exact ablation mechanics, and an exact build sequence — no further > research required. Where the paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible runtime-instrumentation/traces stage** — a deterministic-first, hypothesis-driven, selective, causal evidence layer (failure-site + coverage analysis → selective instrumentation planning → instrumentation → execution → raw-event capture → filtering + divergence analysis → dynamic slicing → evidence assembly, plus the **single weak-LLM hypothesis/question-generation call**) that turns a failing candidate and its component-11 verdict into the single structured **evidence pack** — a set of `StateObservation` records — that the diagnosis and repair stages consume. It is distinct from its predecessor in *kind*, not merely in order: component 11 asks the repository *whether it passed*; instrumentation asks the repository *how it got there* and records the answer as evidence. A verdict is thin evidence — when a test fails, the traceback shows where the process died, not the values computed, the branches taken, the intermediate state that made the failure not merely possible but inevitable. A weak model asked to repair from a traceback alone must *infer* that state, and its inference is cheap, fast, unreliable. This stage replaces that inference with a record: it instruments the run, captures the paths and values behind the verdict, reduces the raw trace to the small residue that matters, and hands the model hard observed facts instead of asking it to imagine them. The stage is governed by four disciplines from the paper, restated because they decide every design choice below. **Hypothesis-driven and selective** — never dump whole-program traces at the model; instrument only where the debugging question demands observation (Haque et al. found raw trace injection helped in only 2 of 6 configurations, usefulness *decreasing* as complexity rises [4]). **Collect richly, expose sparsely** — instrument broadly and capture voluminously, but what reaches the model is only the filtered, divergence-reduced, sliced residue bearing on the question; full traces are an internal data source, never model context. **Traces are evidence that falsifies, not context that flatters** — observed values constrain and eliminate competing hypotheses, so a hypothesis contradicted by an observed value is dead however plausible it reads (TraceRepair [1]). **Never let the model narrate state it did not observe** — the anti-hallucination gate is mechanical: the evidence pack is the only state the model may cite, and any claim not in it is flagged (the paper's "imagined-traces" mode). **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — failure-site + coverage analysis (spectral fault localization), selective instrumentation planning, instrumentation, execution (delegating to component 11's sandbox), raw-event capture, filtering + divergence analysis, dynamic slicing, and evidence assembly with its anti-hallucination gate — plus the *bound* of the *one* hypothesis/question-generation call. The heavy pieces (full debugger integration via a VulDebugger-style `run_to_line`/`print_value` agent, a trained hypothesis generator, CausalRepair-style dual-slicing at scale, record/replay rr-style determinism, multi-language instrumentation) are **not** built in Round 1 and are **not excluded** — later-round content climbing toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by **evidence-pack falsification** and the **anti-hallucination gate** — never the end-to-end resolve rate alone (§7). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element is classified below by *which round* builds it. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–9 plus the bounded stub of row 10). - **DEFERRED = later rounds** — the debugger-grade / trained-generator / replay / multi-language pieces (rows 11–15). They are **not excluded** — they are the content of later rounds, gated by the same §7 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | (a) Failure-site + coverage analysis — spectral fault localization (SBFL) over one-bit-per-statement coverage of passing vs failing runs (Tarantula [17]); output = a bounded ranked list of candidate probe points, NOT a whole-program trace plan | **BUILD-NOW** | | 2 | (b) Selective instrumentation planning — deterministic mapping of the debugging question → concrete observe/probe points at block granularity (LDB [3]); probe-point cap ≤ 16 (expose-sparsely) | **BUILD-NOW** | | 3 | (c) Instrument — deterministic code injection (print / breakpoint / `sys.settrace` / debugger hook) at the planned points; same plan → same instrumentation; records exactly the named values/paths | **BUILD-NOW** | | 4 | (d) Execute — run the instrumented candidate under component 11's sandbox and isolation policy (reuse it, do not duplicate); non-deterministic runs are flagged, not trusted | **BUILD-NOW** | | 5 | (e) Capture raw events — execution paths, observed variable values at each probe point, state deltas; voluminous, uncurated, written to an internal store; none reaches the model directly | **BUILD-NOW** | | 6 | (f) Filtering + divergence analysis — diff a passing run against the failing run; find the FIRST meaningful divergence (TraceFixer [2]); output = narrowed record centered on the divergence, differing values = primary suspects | **BUILD-NOW** | | 7 | (g) Dynamic slicing — backward-slice the divergence-centered record to the minimal causal context; discard observed-but-irrelevant values (CausalRepair [5], Agrawal & Horgan [13]) | **BUILD-NOW** | | 8 | (h) Evidence assembly + anti-hallucination gate — assemble the `StateObservation`-based evidence pack; every claimed value must be present in the pack | **BUILD-NOW** | | 9 | A/B harness (§7: evidence-pack falsification + anti-hallucination gate + ablation runner + failure-mode audits + the hypothesis/no-hypothesis instrument) | **BUILD-NOW** | | 10 | (L) Hypothesis/question generation — **the single learned entry**, included as a *bounded stub* (cost rule below) | **BUILD-NOW** (bounded stub only) | | 11 | **Full debugger integration** — a VulDebugger-style agent driven by `run_to_line` / `print_value` primitives [9] | **DEFERRED** — Round 1 instruments via deterministic injection (`sys.settrace`/`trace`/`inspect`, §3.3) | | 12 | **Trained hypothesis/question generator at scale** | **DEFERRED** — Round 1 ships only the bounded stub (§4); full training/calibration deferred | | 13 | **CausalRepair-style dual-slicing at scale** — static slicing on the test side + dynamic slicing on the source side [5] | **DEFERRED** — Round 1 slices dynamically from observed trace data (§3.7) | | 14 | **Record/replay (rr-style) determinism** | **DEFERRED** — Round 1 inherits component 11's determinism and flags non-reproducible runs (§3.4) | | 15 | **Multi-language instrumentation** — C/C++/Rust/Java instrumenters | **DEFERRED** — Round 1 instruments the Python toolchain only (§3.3) | **The cost rule (explicit).** The paper's single learned entry — hypothesis/question generation — is the one L-column entry this architecture permits. In Round 1 it is included as a **bounded stub**: the stage exists, wired into the pipeline at its firing point (§4), gated, fed back into stage (b), never auto-promoted — but its full training and calibration (the paper's §5 self-improvement loop) is **deferred**. The stub's *cost* is bounded before invocation: one call, temperature zero, schema-constrained output, a bounded payload, parse-failure fallback to "no hypothesis" (§4 caps). It is kept **only if** the §7 no-hypothesis-vs-hypothesis ablation shows it improves falsification on held-out repositories; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the observation path by default — the full deterministic pipeline (rows 1–8) plus the *bounded stub* of the single hypothesis call (§4) is the build; the L step is the exception. Each deferred row would add a debugger surface, a second (or fully trained) model, break the one-entry L column, or add an instrumenter surface — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out repositories, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Eight stages, all **D** through the emission of the assembled evidence pack. The LLM column holds exactly one thing: hypothesis/question generation (§4), fed back into stage (b). The pipeline: ``` FAILING CANDIDATE + ITS VERDICT (component 11) + EXISTING TESTS + COMPETING HYPOTHESES → (a) failure-site + coverage analysis [D] → bounded ranked list of candidate probe points (SBFL) → (§4) hypothesis/question generation [L — the ONE entry; feeds (b)] → (b) selective instrumentation planning [D] → ≤ 16 observe/probe points (block granularity) → (c) instrument [D] → deterministic injected probes (print/breakpoint/settrace) → (d) execute [D] → the instrumented candidate under component 11's sandbox → (e) capture raw events [D] → paths, values, state deltas (voluminous, internal) → (f) filtering + divergence analysis [D] → the first meaningful divergence (passing vs failing) → (g) dynamic slicing [D] → the minimal causal context → (h) evidence assembly + gate [D] → the StateObservation-based evidence pack ``` The pipeline reasons over **observed runtime state only**. Nothing about a run reaches the model except through the assembled, gated evidence pack. The candidate is **watched, never guessed at**. All artifacts land in `build/12-runtime-instrumentation-traces/`. Nothing writes into the live harness. ### 3.1 (a) Failure-site + coverage analysis (D) **Input:** the failure-site + coverage analysis results — the failing run's coverage, a passing run's, and the failing tests. **Output:** a **bounded ranked list of candidate probe points** — the statements whose coverage correlates most strongly with failure — **NOT a whole-program trace plan**. **No language model runs here.** This stage seeds *where observation is likely to discriminate*, in the spirit of Tarantula's cheapest one-bit-per-statement coverage signal [17] (paper §4 stage a); it does not localize the fault. **Spectral fault localization (SBFL) — the Ochiai suspiciousness formula (exact).** For each statement `s`, over the collected passing-vs-failing coverage: ``` suspiciousness(s) = failed(s) / sqrt( total_failed × ( failed(s) + passed(s) ) ) ``` where `failed(s)` = failing runs that executed `s`, `passed(s)` = passing runs that executed `s`, and `total_failed` = the total number of failing runs. A statement in every failing run and no passing run scores 1.0; one executed by all runs regardless of outcome scores 0.0 and carries no discriminatory signal. **Output schema (exact):** ```json { "candidate_probe_points": [ { "rank": int, "location": "file:line", "suspiciousness": 0..1.0, "failed_runs": int, "passed_runs": int } ] } ``` **Caps:** the ranked list is **bounded to the top-50 statements** (the probe-point seeding cap); anything below the cap is discarded before planning. This stage produces candidates, not a trace plan — stage (b) and the model's single hypothesis (§4) decide *which* to actually observe. ### 3.2 (b) Selective instrumentation planning (D) **Input:** the debugging question (from §4, when one fires; otherwise a deterministic default over the failing verdict), the (a) candidate list, and the failure's location. **Output:** a concrete **observe/probe plan** — the specific statements and variables to observe, at the **block-granularity** evidence of LDB [3]: enough to separate the failure from its innocent neighbors, not so much as to recreate the firehose Haque warned about [4]. **No language model runs here.** Planning is a **deterministic mapping from question to observation plan**, not an inference (paper §4 stage b). **Planning rule (exact, deterministic, first-match wins over the ranked candidate list):** 1. If the debugging question names a specific variable and point (from §4), **the planner must include that observe/probe point** — the model's hypothesis is honored deterministically, closing the loop between the one learned question and the observation it demands (paper §4 stage L). 2. The planner then fills the plan from the top of the (a) ranked list — highest-suspiciousness statements first — adding each as a probe point with the observed variables named in the plan. 3. **Stop when the plan reaches the probe-point cap (≤ 16)** or the candidate list is exhausted. 4. If no hypothesis fired (the deterministic default), the plan seeds from the (a) list plus the failure-location neighborhood. **Output schema (exact):** ```json { "probe_points": [ { "id": "PP1", "location": "file:line", "block": "…", "variables": [ "…" ], "source": "hypothesis | sbfl | failure_neighborhood" } ] } ``` **Caps:** the number of probe points is **capped at 16 per planning** — the mechanical enforcement of *expose-sparsely*. Planning is pure deterministic software on CPU; the only learned influence is the single hypothesis call of §4, which may add one point and never runs the planner itself. ### 3.3 (c) Instrument (D) **Input:** the observation plan from (b). **Output:** the **instrumented candidate** — the planned probe points realized by deterministic code injection: **print statements / breakpoints / `sys.settrace` / a debugger hook** inserted at the planned points. **No language model runs here.** Injection is **deterministic per toolchain**, so the same plan yields the same instrumentation, and the injected code records **exactly the named values and paths** (paper §4 stage c). Round 1 instruments the **Python toolchain only** (§2 row 15). **Injection mechanics (exact, deterministic):** each probe point `PP` at `file:line` with a named variable set is realized by (a) a **print/observer probe** — an injected statement recording, at that point, the current value of each named variable, the branch last taken at the nearest decision point, and a path marker — or (b) a **`sys.settrace` tracing hook** recording the same at line granularity for the probed region, with (c) **breakpoints / debugger hooks reserved for the deferred interactive surface** (§2 row 11). The injected code is deterministic: identical plan + identical source → identical instrumentation, recording exactly what the plan names — no more, no less; a variable the plan did not name is never recorded. **Output schema (exact):** ```json { "instrumented_artifact": "<SHA-256 of the instrumented code>", "probe_specs": [ { "id": "PP1", "location": "file:line", "method": "print|settrace", "variables": [ "…" ] } ] } ``` **Caps:** instrumentation is deterministic per plan; the probe count is inherited from (b) (≤ 16). The instrumented code is kept **separately from the candidate** (§8 invariant 6 — instrumentation edits never leak into the repair diff). ### 3.4 (d) Execute (D) **Input:** the instrumented candidate from (c). **Output:** the run's raw behavior under instrumentation. **No language model runs here.** This stage **delegates to component 11's sandbox and isolation policy — reusing it, not duplicating it**: instrumentation reuses component 11's execution substrate (subprocess + resource limits + scratch root + network deny-by-default + per-mode timeouts + stream caps), adding only its capture hooks, so the verdict run and the instrumented run share the same determinism and resource boundaries (paper §4 stage d). A failing run and its passing sibling are both executed so (f) has both sides. **Procedure (exact, deterministic):** 1. **Spawn** the instrumented candidate under component 11's declared isolation policy — CPU cap 1.0 core, RAM cap 512 MiB (`RLIMIT_AS`), scratch-root filesystem boundary with the repo read-only, network deny-by-default, per-mode timeouts, 64 KiB stream caps — reusing component 11's sandbox wholesale. 2. **Capture** the raw run plus whatever the injected probes emitted. 3. **Determinism check (inherited):** a non-deterministic run — one that cannot be reproduced from its own artifact hash + environment (per component 11's determinism precondition) — is **flagged, not trusted**; the heisenbug audit (§7) re-runs before any divergence is trusted. **Caps:** no new isolation policy is invented here — component 11's policy is the single source of execution bounds. Round 1's determinism substrate is component 11's (rr-style replay is deferred, §2 row 14). A non-reproducible run is flagged `deterministic: false` and never drives a divergence claim. ### 3.5 (e) Capture raw events (D) **Input:** the instrumented run(s) from (d). **Output:** the **raw event stream** — execution paths (which statements and branches were taken), observed variable values at each probe point, and state deltas between consecutive observations. **No language model runs here.** This is the **collect-richly** moment: capture is voluminous and uncurated, written to an **internal store** downstream stages mine; **none of it reaches the model directly**, honoring *collect-richly-expose-sparsely* (paper §4 stage e) and P2's separation of instrumentation from presentation. **Raw-event record schema (exact, internal):** ```json { "execution_id": "…", "sequence": int, "kind": "path | value | state_delta", "probe_id": "PP1 | trace", "location": "file:line", "payload": { "variables": { "x": "<value>" }, "branch_taken": "<which branch>", "return_value": "<value>", "state_delta": { "from": "…", "to": "…" } } } ``` **Caps:** capture is bounded per-stream (inherited 64 KiB caps); the raw store is written to `build/12-runtime-instrumentation-traces/raw/` and is an internal substrate only. **None of the raw event stream is ever surfaced to the model** — only stage (h) emits what the model may see. ### 3.6 (f) Filtering + divergence analysis (D) **Input:** the raw event streams from (e) — the failing run's and a passing run's. **Output:** a **narrowed record centered on the first meaningful divergence** in observed state, with the differing values identified as the **primary suspects**. **No language model runs here.** This is TraceFixer's first-divergence localization [2] and Zeller's cause-effect chain [14]: find where actual execution first diverges from desired execution, because that divergence point is where the failure *began* rather than where it surfaced (paper §4 stage f). **Procedure (exact, deterministic):** 1. **Filter** the raw events to those on the probed paths — discard events for statements the plan did not probe (the probe-bleed and firehose guards begin here). 2. **Align** the failing run's filtered stream against the passing run's by path/sequence. 3. **Find the FIRST divergence**: the earliest aligned point where an observed value (variable, return, branch, or state delta) differs. The differing value at that first point is the **primary suspect**; the narrowed record is centered on it. **Output schema (exact):** ```json { "first_divergence": { "location": "file:line", "probe_id": "…", "passing_value": "…", "failing_value": "…", "sequence": int }, "centered_record": [ "<filtered events on the probed path around the divergence>" ], "primary_suspects": [ "<the values that differ>" ] } ``` **Caps:** only the **first** meaningful divergence is reported (subsequent divergences are downstream consequences, not causes). The centered record is bounded to the probed path — never the whole trace. If no divergence is found, the stage records that explicitly rather than inventing one; the failing run's own anomalous values become the suspects. ### 3.7 (g) Dynamic slicing (D) **Input:** the divergence-centered record from (f). **Output:** the **minimal causal context** — the observed statements and values in a **dependency relation** to the failure. **No language model runs here.** This is P5's mechanical reduction: the CPU cuts the trace to the tiny residue that could have caused the failure, discarding every observed-but-irrelevant value that would cost context and risk misleading the model (CausalRepair's dual-slicing [5], Agrawal & Horgan's dynamic slicing [13], paper §4 stage g). Round 1 slices **dynamically from the observed trace data**; the full CausalRepair-style dual-slice is deferred (§2 row 13). **Procedure (exact, deterministic):** backward-slice the divergence-centered record from the failing value — walk the recorded stream **backward**, retaining each observed statement and value in a **data or control dependency** with the failing value (a statement computing a value it depends on, or a branch decision determining the path to it), **discarding** every observed value not on such a path — exactly the observed-but-irrelevant state Haque's negative result warns about [4]. **Output schema (exact):** ```json { "slice": [ { "location": "file:line", "value": "…", "relation": "data_dependency | control_dependency | root" } ], "discarded": int } ``` **Caps:** the slice is the minimal causal residue, strictly smaller than the (f) centered record — the last deterministic reduction before assembly. ### 3.8 (h) Evidence assembly + anti-hallucination gate (D) **Input:** the slice from (g) plus run provenance (execution ids, component-11 artifact hashes). **Output:** the assembled **evidence pack** — the single currency of this component — built from **`StateObservation`** records, over which repair and verification reason and the anti-hallucination gate (P7) checks every claim; **nothing about the run reaches the model except through it** (paper §4 stage h). **The `StateObservation` schema (exact — EXACTLY these fields):** ```json { "execution_id": "…", "location": "file:line | symbol", "path_index": int, "variables": { "<name>": "<observed value>" }, "return_values": { "<call site>": "<observed return>" }, "branch_taken": "<which branch at the nearest decision point>" } ``` **Field meanings (exact):** `execution_id` ties the observation to exactly the run that produced it, reusing component 11's `artifact_version` lineage (§8 invariant 7); `location` names the file/line or symbol observed; `path_index` places it on the execution path; `variables` holds the observed values; `return_values` the observed returns; `branch_taken` the branch actually taken at a decision point. Every value in the pack is an **observed** value — nothing is inferred, interpolated, or synthesized into the pack. **The six-primitive set (exact):** `trace_path` (record a run's execution path), `observe` (capture variable/return state at a named point — the print/probe primitive), `compare_runs` (run two variants and diff their observations), `trace_diff` (find the first meaningful divergence between a passing and a failing trace), `dynamic_slice` (reduce a trace to statements on a causal path to a value), and `probe` (inject and run a targeted observation, the demand-driven P6 primitive). Of these, **`trace_path`, `compare_runs`, `trace_diff`, and `dynamic_slice` are the deterministic reduction machinery (D)** of stages (e)–(g); **`observe` and `probe` are the workhorses of the hypothesis loop** — the model's single L call names what `probe` should observe, which (b) plans and (c) injects — so `probe`/`observe` drive the L step while the four reduction primitives stay purely deterministic. **The anti-hallucination gate (exact, run at assembly, D):** every value downstream reasoning may cite must be **present in the pack**, and the pack may contain only **observed** values. Mechanically the gate (1) audits the pack itself, rejecting any field whose value was not captured by a probe (no non-causal or non-observed inclusion — the harness cannot fabricate evidence even on the model's behalf); and (2) tags the pack so any downstream claim — a variable value, a branch taken, a return — is checkable against it; a claim not in the pack is a **hallucination**, flagged by the §7 audit. The pack is the **only state the model may cite** (§8 invariant 2). **Output schema (exact):** ```json { "evidence_pack": [ "<StateObservation records>" ], "primary_suspects": [ "<the differing values from (f)>" ], "execution_ids": [ "…" ], // tied to component 11's artifact hash "gated": true } // anti-hallucination gate passed ``` **Caps:** exactly one evidence pack per diagnosis, assembled from the (g) slice (strictly the minimal causal residue — never the raw event store). The pack is immutable once assembled; a later re-run is a new `execution_id` and a new pack. **The L column holds exactly one entry.** Stages (a)–(h) are all deterministic software on CPU. The single learned node is stage (§4), hypothesis/question generation, fed back into stage (b) and never auto-promoted. Any proposed addition that would put a language model elsewhere in the pipeline — ranking probe points, deciding the divergence, slicing the trace, assembling or gating the pack, narrating raw events — is a regression against P1–P7 and the L-column invariant, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of §7, on held-out repositories, at a defensible cost, or be rejected (paper §4). --- ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **hypothesis/question generation**. The weak model turns "the program fails here" into "observe X at point Y, testing hypothesis H" — the debugging question plus competing hypotheses — in **ONE bounded call**, schema-constrained, temperature zero, fed back into stage (b) (paper §4 stage L). It is PingFL's "scientific experiment API" stance [6] and the concrete shape of the strong Piper rule — the model asks, the harness instruments; it does **not** wander a state space with a debugger. Firing is re-considered only when a new failing candidate needs a diagnosis. **Prompt inputs (assembled deterministically, in this exact order):** 1. **`failure`** — the component-11 verdict: the failing test, failure class, location, message (bounded). 2. **`candidate_points`** — the bounded ranked probe-point list from stage (a) (top-50), so the hypothesis names points SBFL already flagged or a variable the initial plan did not cover. 3. **`existing_tests`** — the authoritative tests already present, in brief. **Input → output schema (exact):** ```json { "debugging_question": "observe X at point Y, testing hypothesis H", "hypotheses": [ { "hypothesis_id": "H1", "claim": "…", // what the model believes may be wrong "observe": { "variable": "X", "location": "Y" }, // what would discriminate it "expectation": "…" } // what value/state would confirm or falsify it ] } ``` **Caps:** **ONE call**, `temperature = 0`, `max_completion_tokens = 2048`; a bounded number of competing hypotheses (**≤ 4 per firing**), each naming a single observation point. The output is schema-constrained; the harness, not the model, decides what a hypothesis means. The output feeds back into stage (b) — if the hypothesis names a variable the initial plan did not cover, the planner instruments that observation (deterministically, §3.2 rule 1) — closing the loop between the one learned question and the deterministic observation it demands. **Gating (exact):** the model proposes what to observe; the harness **instruments and executes** deterministically. A proposed hypothesis is **never treated as an observed fact** — its expectation is a prediction to be confirmed or falsified by the probe, and a hypothesis whose expectation contradicts the observed value is **eliminated** by the evidence, not believed on the model's say-so (§8 invariant 5). **Parse-failure fallback (exact):** any JSON parse failure, schema violation, out-of-bound output (more than 4 hypotheses, oversized payload), or empty/malformed output is a **no-op**: the stage generates **no hypothesis** and hands control back to the deterministic path (which plans from the (a) SBFL list alone) — **never a re-prompt, never a second call**. The L column is reached **once** per diagnosis. This is the mechanical enforcement of the one-entry L column and the anti-over-prompting rule. **Re-invocation on a later candidate (exact):** hypothesis generation is re-considered **only** when a *new* failing candidate needs a diagnosis; each invocation is scoped strictly to that candidate and re-uses none of a prior firing's hypotheses. It remains the **single conceptual learned entry** — a bounded, once-per-diagnosis call — **never a model that runs over the whole pipeline**. **Invariant 1 (§8): the L column contains exactly one entry — and that entry is only ever a bounded, gated hypothesis/question generator whose output the deterministic stages instrument.** --- ## 5. Implementation Order Eight phases, each independently testable and reversible, landing in `build/12-runtime-instrumentation-traces/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Coverage seeding (a)** | `sbfl.py` (§3.1 the Ochiai formula over passing-vs-failing coverage → the bounded top-50 ranked candidate list) | Feed synthetic coverage fixtures: a statement in every failing run and no passing run scores 1.0; one in all runs scores 0.0; the list is ranked descending and **capped at 50** — never a whole-program trace plan; assert **no LLM consulted**. | Delete `sbfl.py`. | | **2. Selective planning (b)** | `plan.py` (§3.2 the question → observation-plan mapping at block granularity; honors a hypothesis-named point; **probe-point cap ≤ 16**; deterministic first-match fill) | Feed the (a) list + a question: the plan is ≤ 16 probe points; a hypothesis-named variable/point is included when one fires; fill is deterministic (same inputs → same plan); the default (no hypothesis) seeds from SBFL + the failure neighborhood; assert **no LLM consulted**. | Delete `plan.py`. | | **3. Instrumentation (c)** | `instrument.py` (§3.3 deterministic injection — print/observer probes + `sys.settrace` hook — over the Python toolchain; records exactly the named values/paths; instrumented artifact kept separate) | Feed a plan + source: same plan + same source → same instrumentation; the injected code records exactly the named variables/branches and nothing else; the instrumented artifact is hashed and kept separate; assert **no LLM consulted**. | Delete `instrument.py`. | | **4. Execute (d)** | `exec_under.py` (§3.4 delegation to component 11's sandbox and isolation policy; determinism check flags non-reproducible runs) | Feed the instrumented candidate: it runs under component 11's declared policy (assert the component-11 isolate module is called, not re-implemented); a non-deterministic run (perturbed twice) is flagged `deterministic: false`, never trusted; assert **no LLM consulted**. | Delete `exec_under.py`. | | **5. Raw capture (e)** | `capture.py` (§3.5 the raw-event stream — path/value/state-delta records — written to an internal store, bounded per stream, **never surfaced to the model**) | Feed an instrumented run: raw events record execution paths, values, and state deltas to the internal store; per-stream bounds hold; assert the raw store is write-only from the model's perspective; assert **no LLM consulted**. | Delete `capture.py`. | | **6. Divergence (f)** | `divergence.py` (§3.6 filter to the probed path; align failing vs passing streams; report the FIRST meaningful divergence with the differing values as primary suspects) | Feed failing + passing fixtures: the first divergence is reported at the correct aligned point with `passing_value` vs `failing_value`; off-path events are filtered; a no-divergence case records that explicitly; assert **no LLM consulted**. | Delete `divergence.py`. | | **7. Dynamic slicing (g)** | `slice.py` (§3.7 backward-slice the divergence-centered record to the minimal causal context; retain only data/control-dependency values; discard observed-but-irrelevant values) | Feed the divergence-centered record: the slice retains only values in data/control dependency with the failure and drops a captured-but-irrelevant value; the slice is strictly smaller than the centered record; assert **no LLM consulted**. | Delete `slice.py`. | | **8. Assembly + gate + L stub + A/B harness** | `assemble.py` (§3.8 the `StateObservation` evidence pack + the anti-hallucination gate), `hypothesis.py` (§4 the bounded L stub: firing + ONE call + temperature 0 + ≤ 4 hypotheses + gating + parse-failure fallback + feed-back into (b)), `ab.py` (§7 feature-flag router + falsification capture + gate audit + ablation runner + hypothesis/no-hypothesis runner + decision) | `assemble.py` on fixtures: the pack has exactly the `StateObservation` fields and only observed values; the gate rejects a non-observed/non-causal inclusion. `hypothesis.py` with a mocked LLM: the call is **one**, temperature 0, hypotheses ≤ 4; the hypothesis feeds (b) deterministically; a malformed/over-cap response → no hypothesis, **no second call**. `ab.py` runs the §7 pipeline-vs-baseline and the §7 ablations, emitting the §7 decision. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 8) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of component 11's substrate)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `sys.settrace` | §3.3 the tracing-hook injection mechanism; §3.5 raw path capture | CPython stdlib. | **No** | | stdlib `trace` | §3.5 line/branch path capture support | CPython stdlib. | **No** | | stdlib `inspect` | §3.2/§3.3 resolving named variables to concrete observation points | CPython stdlib. | **No** | | stdlib `subprocess`, `resource`, `tempfile` | §3.4 execution bounds — **inherited from component 11's sandbox**, reused, not reimplemented | CPython stdlib; component 11 already vetted. | **No** | | stdlib `json`, `hashlib`, `os`, `sys`, `time` | §3.5–§3.8 schemas, artifact hashing, capture plumbing | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3.3 the Python instrumenter; the candidate runtime | In-repo, pinned interpreter. | **No** | | Component 11 execution substrate | §3.4 the sandbox + isolation policy + determinism precondition | Built by spec 11; reused, not duplicated. | **No** | | Component 01 tool layer / existing test runner | §3.4 the passing reference run; §3.6 the differential baseline | In-repo, already vetted. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single hypothesis/question call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: instrumentation uses stdlib `sys.settrace` / `trace` / `inspect`; execution reuses component 11's subprocess/resource/tempfile sandbox wholesale. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1, and nothing is flagged for a later install** — the five DEFERRED rows (§2 rows 11–15) are a debugger surface on an existing interpreter, training/weights on known architectures, scale on the deterministic substrate, a determinism substrate, or instrumenter surface on existing toolchains — not new third-party Python installs. Any future debugger backend, model download, RL-training run, replay substrate, or multi-language instrumenter must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — a debugger integration or learned generator is ops-critical and gets deep scrutiny regardless of stars — and is out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01 and 11** (Python-only, single-repo, no-Docker reproduction) as the source of a held-out **(failing candidate, verdict, repair hypotheses, expected state) corpus**: for each instance, a failing candidate with its component-11 verdict, a set of **deliberately-wrong repair hypotheses** (crafted to be clearly wrong about the observed state) plus one right hypothesis, and the mechanically-known expected state at the divergence. Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. **Baseline (concrete).** The **raw-trace-injection** baseline — Haque et al.'s negative-result path [4] realized: the failing candidate is run with the full raw trace injected directly into the model's context with **no SBFL seeding, no selective planning, no divergence analysis, no dynamic slicing, no anti-hallucination gate**. The structured pipeline must beat *this* on the honest evidence metrics below, and the ablation (7.3) removes pipeline mechanisms against those same honest metrics. ### 7.1 (a) Evidence-pack FALSIFICATION — the KEY measure The key instrumentation instrument, the direct score of stages (a)–(h): on the held-out corpus, given the evidence pack for a failing candidate, **the fraction of deliberately-wrong repair hypotheses that the observed values actually eliminate** — do the captured values contradict the wrong explanation the way TraceRepair's cross-checks eliminate incompatible fixes [1]? **A pack that fails to falsify a clearly-wrong hypothesis is decoration, not evidence**, and this measures the fraction the observed state rules out. This makes P1–P7 falsifiable: if the pack cannot let a weak model discard what observed state contradicts, the stages are not earning their place. **This is the KEY measure — the structured pipeline is adopted only if it STRICTLY beats the raw-trace baseline on falsification rate AND does not regress resolve (7.2); a resolve win that does not beat the baseline on falsification is not credited.** ### 7.2 (b) The anti-hallucination gate + the end-to-end regression guard The companion instrument audits whether the model ever cites a value not in the pack: **every claimed observation in a repair narrative is checked against the evidence pack, and any claim — a variable value, a branch taken, a return — not present in the pack is flagged as a hallucination.** The gate also audits the pack itself for non-causal or non-observed inclusions, so the harness cannot fabricate evidence even on the model's behalf. This is reported as the **gate-citation rate** (fraction of cited values present in the pack — target 1.0). The coarse end-to-end instrument is kept for parity — **deliberately not** the gate for instrumentation changes, because resolve rate cannot separate instrumentation quality from repair quality: 1. **Resolve rate** = `resolved / total` (spec 01's score) — the no-regression guard, never the primary gate. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). 3. **Tokens** = `tokens / instance`. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add an instrumentation mechanism because it sounds sensible" mechanically (paper §5, §6). Two passes, everything else held fixed, the delta measured on the honest metrics (evidence-pack falsification + the gate, plus resolve/cost/tokens): 1. **Remove the reduction stages** — feed the divergence-centered record (or, deeper, the raw probed events) instead of the dynamically-sliced pack, i.e. remove normalization/divergence/slicing one at a time — and measure the **downstream repair-accuracy drop**. This is the direct test of P2/P5: does the reduction measurably improve the repair that consumes the pack, or does the pack degrade toward Haque's firehose [4]? 2. **Remove each other mechanism in turn** — the SBFL seeding, the probe-point cap, the anti-hallucination gate. 3. **Ablate the single L hypothesis step** — run **no-hypothesis vs hypothesis** (never generate a hypothesis vs. generate one) and measure the **falsification rate** on held-out repos. This is the test of whether the bounded hypothesis stub (§4) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest metrics beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the metrics is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L hypothesis stub, which must survive the no-hypothesis-vs-hypothesis ablation on falsification to be kept at all), then to any other mechanism. **Never add an instrumentation mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run pipeline-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured pipeline **wins iff**: ``` Falsification rate STRICTLY beats the raw-trace-injection baseline (the KEY measure) AND gate-citation rate == 1.0 on the model's repair narratives (the anti-hallucination gate holds) AND resolve_pipeline ≥ resolve_baseline (no end-to-end regression) AND cost_pipeline ≤ 1.5 × cost_baseline AND tokens_pipeline ≤ 1.5 × tokens_baseline ``` Evidence-pack falsification must **strictly beat the raw-trace baseline**; resolve must not regress; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — a candidate that improves resolve but leaves falsification flat against the raw baseline is **not adopted** (instrumentation quality would be masking repair quality). If the structured pipeline wins, it replaces the raw-trace path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways the evidence layer can lie) Beyond the positive instruments, the harness audits the specific ways an instrumentation layer can lie, each with a mechanical guard: 1. **Trace-firehose** — raw capture floods the model despite *collect-richly-expose-sparsely*; guarded by an audit that asserts the model's context contains only the assembled pack, never the raw event store (Haque's exact failure [4]). 2. **Observer-effect** — instrumentation changes what runs, so the observed state is not the state of the un-instrumented candidate; guarded by diffing instrumented against un-instrumented behavior. 3. **Heisenbug** — a bug that vanishes or changes under observation, making the differential run (P4) meaningless; determinism, inherited from component 11, is a *prerequisite* for differential runs, and a non-reproducible observation is **flagged rather than trusted** (§3.4) and re-run before any divergence is trusted. 4. **Single-trace overfitting** — a fix that passes the probed run but nothing else; guarded by **out-of-pack test runs** that exercise states the probes did not cover. 5. **Probe-bleed** — instrumentation edits leaking into the repair diff; guarded by **diffing the submitted patch against the candidate without instrumentation** — the injected code is stripped before the patch is evaluated (§8 invariant 6). ### 7.6 Two honest caveats Two caveats close the section, inherited from the paper's §6. **First, the strong-model-and-scale caveat:** the headline numbers of the paper's review — TraceRepair's 392 trace-constrained Defects4J repairs [1], TraceFixer's 13–20% first-divergence gain [2], LDB's up-to-9.8% block-level verdicts [3], CausalRepair's 313 dual-sliced repairs at ~$0.029/bug [5], PingFL's 812 demand-driven localizations [6] — were obtained with models far stronger than this harness's target; a weak-model harness should expect materially lower absolute repair counts. Its instrumentation stage exists precisely so that weak *hypothesis generation* is compensated by strong deterministic *observation and reduction*, and the honest instruments here — above all evidence-pack falsification and the anti-hallucination gate — are what will show whether that compensation is real. **Second, the held-out and memorization caution** inherited from the earlier components: all instrumentation metrics must be reported on held-out and out-of-benchmark repositories, so a model that has memorized a repository's traces cannot score well without its repairs having been grounded in state the harness actually watched. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/12-runtime-instrumentation-traces/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `TRACES=inst12|raw`, default `raw`) routes the A/B; the flag alone switches between the new selective instrumentation stage and the raw-trace-injection baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains exactly one entry** — hypothesis/question generation (§4), the one bounded, gated call that turns "the program fails here" into "observe X at point Y, testing hypothesis H," fed back into stage (b), never re-prompted on parse failure. Any phase that introduces a language model elsewhere in the pipeline (failure-site ranking, planning, instrumentation, execution, capture, divergence, slicing, assembly, the gate) is rejected outright. An **empty** L column (the hypothesis stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **No narration of unobserved state — the anti-hallucination gate.** The evidence pack is the **only state the model may cite**; every claimed observation in a repair narrative is checked against it, and any value not in the pack is flagged as a hallucination. The pack itself may contain only observed values — the harness cannot fabricate evidence even on the model's behalf. 3. **Determinism — differential runs require reproducible execution.** Two runs only differ meaningfully if the substrate is deterministic (inherited from component 11). A non-reproducible observation is flagged, never trusted; the heisenbug audit re-runs before any divergence is accepted. A divergence claim built on a non-deterministic run is a rejected deliverable. 4. **Collect richly, expose sparsely.** The harness may instrument broadly and capture voluminously, but the model sees **only the reduced, gated evidence pack** — never the raw event store. A firehose that reaches the model is a rejected deliverable. 5. **Traces are evidence that FALSIFIES.** A hypothesis contradicted by an observed value is **eliminated** — regardless of how plausible it reads. Observed values constrain and eliminate competing hypotheses; they are not decoration that confirms the model's favored explanation. 6. **Probe-bleed guard.** Instrumentation edits (injected print/breakpoint code) **never leak into the repair diff** — they are stripped before the patch is evaluated, so the repair is never scored on code the harness wrote rather than the model wrote. 7. **Provenance — every StateObservation carries `execution_id` tied to component 11's artifact hash.** Each observation is versioned by the run that produced it and is diffable, so a regression in what instrumentation records is visible in the assembled pack long before it surfaces as a failed resolve; the reader of any observation can reconstruct the exact run and candidate that produced it. 8. **Hypothesis-first selectivity.** Never whole-program traces by default. The harness instruments only where the debugging question demands observation (seeded by SBFL, planned to ≤ 16 probe points, reduced causally). A whole-program trace plan is a rejected deliverable at every stage. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*