Summary / 05. Issue and fault localization / Document
Spec 2026-09-04
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 05 — Issue/Fault Localization (the ≈80% fraction) > Derived from `papers/05-issue-fault-localization.md`. This document drives the build of the > **issue/fault-localization stage**: the conversion of a natural-language symptom into a ranked, > causally-explained set of candidate edit sites — files, then entities, then statements — before > any patch is attempted. It sits between component 03 (reconnaissance: what the repo *is*) and > component 04 (retrieval: what context is *relevant*), and answers the question neither of them > does: *of the relevant code, where precisely is the fault.* It translates the paper's design into > exact extractors, exact score formulas, exact gate conditions, exact caps, exact schemas, and an > exact build sequence — no further research required. Where the paper says "≈80%", this spec pins > *which* 80%. --- ## 1. Purpose & Scope Build a **reversible issue/fault-localization stage** — a deterministic-first, evidence-fused, hierarchical ranker (issue-signal mining → fused deterministic pre-filter → lazy structural expansion → dynamic/Ochiai evidence → historical evidence → evidence fusion → hierarchical file→entity→statement ranking → confidence test) with a **single conditional LLM step** that is a listwise re-ranker over a shortlist, *never* an explorer — as a drop-in *discrimination stage* feeding component 01's Stage-1 localization and the repair stage, not an accretion onto them. The overall target is the paper's full 100% spec (all five evidence channels plus the learned re-rankers and the full OrcaLoca scheduler). **Round 1 lands the expected ≈80% of it**: everything deterministic (signal mining, the BM25+lexical+graph fused pre-filter, the relevance-gated lazy BFS, Ochiai SBFL with stack-trace pruning, git-history evidence with anti-leakage, rank-combination fusion, the hierarchical ranking, and the agreement confidence test) plus the *one* weak-LLM disagreement-case re-ranker — targeting the reachable fraction of the corpus, the common agreement case where the learned column is empty and localization is deterministic software on CPU. The 137M SweRankEmbed bi-encoder, the trained SweRankLLM re-ranker, OrcaLoca's full ASQ repetition-counter scheduler, and LLM4FL's multi-agent Reflexion are **not** built in Round 1 and are **not excluded** — they are the content of later rounds that climb from that ~80% landing toward 100% (see §2). Round 1 is kept only if it wins a measured A/B test on the *honest* localization metrics *and* does not regress end-to-end resolve/cost/tokens (§7). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element from the paper is classified below by *which round builds it*. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–9). - **DEFERRED = later rounds** — the learned re-rankers, the full scheduler, and the reflexion layer (rows 10–13) that Round 1 intentionally does not build *yet*, because they need model downloads, training, or are measured to be worth less than the deterministic machinery. They are **not excluded from the project** — they are the content of later rounds, the climb toward 100%, gated by the same §7 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | Issue-signal mining (identifiers, file paths, error strings, deepest non-library stack frame — deterministic regex/AST) | **BUILD-NOW** | | 2 | Fused deterministic pre-filter (BM25 + lexical Rouge overlap + graph edges; **NO neural bi-encoder**) → top-K (K=40) | **BUILD-NOW** | | 3 | Lazy structural expansion (import graph of issue-mentioned files → relevance-gated BFS, ≤10 iters) | **BUILD-NOW** | | 4 | Dynamic evidence (Ochiai SBFL + stack-trace prune, when a failing test/repro exists; graceful "no dynamic evidence") | **BUILD-NOW** | | 5 | Historical evidence (`git log` churn hotspots + issue→fix pairs, BM25 over commit messages, anti-leakage: pre-date-only + overlapping-text filter; "no history" gate) | **BUILD-NOW** | | 6 | Evidence fusion (rank-combine the independent signals into ONE ranked list; reciprocal-rank fusion or weighted rank-sum, configurable weights) | **BUILD-NOW** | | 7 | Hierarchical ranking (file → entity → statement) + causal stories + the confidence test (agree → stop; disagree → LLM) | **BUILD-NOW** | | 8 | Disagreement-case weak-LLM listwise re-ranker (reads top-K shortlist, emits candidate IDs — the ONLY LLM step) | **BUILD-NOW (conditional)** | | 9 | Localization A/B + honesty harness (ground-truth metrics + oracle diagnostic + anti-memorization control) | **BUILD-NOW** | | 10 | SweRankEmbed **137M** bi-encoder (trained on SweLoc 67,341 issue→code pairs) | **DEFERRED** — needs a model download (age/scrutiny gate, §6) | | 11 | Trained SweRankLLM listwise re-ranker (the learned replacement for row 8's weak-LLM rerank) | **DEFERRED** — needs training/weight access | | 12 | OrcaLoca's full ASQ repetition-counter scheduler (the complete priority-queue beam-search scheduler) | **DEFERRED** — Round 1 uses a de-LLM'd relevance-gated BFS (§3c) | | 13 | LLM4FL's multi-agent Reflexion (self-critique layer) | **DEFERRED — skip entirely**: the ablation shows it is worth only −6–11 points, vs −19–23 for segmentation; the deterministic rank-and-segment machinery carries the weight | **Confidence-test decision: BUILD-NOW-DETERMINISTIC.** The paper's confidence test is a deterministic agreement check, not a learned step. Round 1 implements it as: compute each evidence channel's top-1 candidate; if the **same top-1 is shared by ≥3 channels** (configurable `AGREE_THRESHOLD = 3`) and no channel's leader strongly contradicts it, **stop** and emit the top-k with causal stories — the L column is empty. Otherwise (disagreement), invoke the single weak-LLM re-ranker once (§4). The only learned node in the entire stage is that disagreement-case re-ranker, and it is reached only on disagreement. --- ## 3. The Deterministic Pipeline (D) Seven stages, all **D** through the production of the ranked, causally-explained candidate list. The LLM column holds exactly one thing: the disagreement-case re-ranker (§4), reached only when the confidence test (§3.7) says *disagree*. The pipeline: ``` SYMPTOM → (a) issue-signal mining → (b) fused deterministic pre-filter → (c) lazy structural expansion → (d) dynamic evidence (SBFL, when tests exist) → (e) historical evidence (git, when history exists) → (f) evidence fusion → (g) hierarchical ranking + confidence test → [agree] STOP with causal stories → [disagree] → (§4) LLM listwise rerank (once) ``` All artifacts land in `build/05-issue-fault-localization/`. Nothing writes into the live harness. ### 3.1 (a) Issue-signal mining (D) **Input:** `issue_text: str` (verbatim) + optional `traceback: str | None` (a captured traceback from `run_test`, component 01 §3.5) + optional `repro_target: str | None` (a failing test node id). **Output:** a structured `IssueBundle` — a bundle of independent evidence channels, each computed deterministically. **No language model reads the issue at this stage.** ```json { "identifiers": [ "memstore.MemStore.remember", "recall", ... ], "file_paths": [ "src/memstore.py", ... ], "error_strings": [ "KeyError: 'kind'", ... ], "deepest_frame": { "file": "src/memstore.py", "func": "remember", "line": 87 }, "repro_target": "tests/test_x.py::TestY::test_z" | null } ``` **Extraction rules (exact):** | Field | Rule (exact) | |---|---| | `identifiers` | Dotted identifiers `[A-Za-z_][A-Za-z0-9_]*(\.[A-Za-z_][A-Za-z0-9_]*)+`; bare CamelCase `\b[A-Z][a-zA-Z0-9_]+\b`; snake_case tokens `\b[a-z_][a-z0-9_]*\b`. Each is kept **only if** it resolves against component 03's `name_index` (the symbol table lexical match); unresolved names are kept as *unresolved* terms — a §3.2 BM25/lexical input, never silently dropped. | | `file_paths` | Any token matching `[\w\-/]+\.(py|js|ts|go|rs|java|c|h|toml|json|yaml|yml)` in `issue_text`/`traceback`. | | `error_strings` | Quoted strings `'…'` / `"…"` from `traceback`/`issue_text`; plus any line matching `Error:` / `Exception:` / `Traceback` / `assert`. | | `deepest_frame` | From `traceback` (or `issue_text`'s pasted traceback): walk frames top→bottom, **drop every frame whose file resolves to a library** (site-packages, stdlib, `vendor/`, `node_modules/`, `third_party/` — the §3.2 library filter), and take the **deepest remaining non-library frame**. This is LLM4FL's stack-trace pruning applied deterministically. No traceback → `deepest_frame = null`. | | `repro_target` | `repro_target` if set, else any token matching `\S+\.py::\S+` or `\S+::\S+::\S+` in `issue_text`. | **Caps:** ≤ **32 identifiers**, ≤ **8 file_paths**, ≤ **8 error_strings**, each truncated at 120 chars; one `deepest_frame`. ### 3.2 (b) Fused deterministic pre-filter (D) **Input:** the `IssueBundle` + component 03's `recon.db` graph + component 04's `api.db` index. **Output:** a fused-scored, truncated candidate entity list of **top-K, K = 40**. Three channels, each deterministic, each scoring every candidate entity in `[0,1]`: 1. **BM25 over the component-03/04 index.** Score each entity's `signature` + `docstring` (`desc_text` from component 04's `api.db`) against the `IssueBundle` terms (identifiers, error strings, file paths) using component 04's exact BM25/IDF scorer (§3.4 of spec 04, identifier weight 3× over prose, `k1 = 1.5`, `b = 0.75`), normalized to `[0,1]`. 2. **Lexical overlap (Rouge).** For each entity, compute Rouge-L (longest-common-subsequence F-score) between the `IssueBundle` terms (and `file_paths` basenames) and the entity's `qualname` / `signature` / `file` path. Normalized to `[0,1]`. 3. **Graph edges.** For each entity, structural distance from every resolved `identifier`'s def node in `recon.db`: caller/callee (`invoke`) and importer/importee (`import`) edges, plus `contain`. Score = `1 / (1 + shortest_path_distance)` from the nearest issue-named symbol; entities not reachable within 10 hops score 0. **Fused score (exact):** ``` fused(e) = w_bm25 · bm25(e) + w_lex · rouge(e) + w_graph · graph(e) w_bm25 = 0.5, w_lex = 0.3, w_graph = 0.2 (configurable; sum must equal 1.0) ``` Candidates are sorted by `fused(e)` descending and truncated to **K = 40**. **No neural bi-encoder, no embedding, no GPU** anywhere in this step — the SweRankEmbed 137M bi-encoder is DEFERRED (§2 row 10). This is the search-space reduction that makes everything downstream cheap. ### 3.3 (c) Lazy structural expansion (D) **Input:** the top-40 candidates + the `IssueBundle` + component 03's `recon.db` import edges. **Output:** a structurally-connected candidate subgraph (the top-40 ∪ their reachable relevant neighbors). **Procedure (exact):** 1. **Build the import graph lazily** — only of the **issue-mentioned files** (the `file_paths` and the top-40 candidates' files), *not* the whole repo. No upfront full-repo graph construction (CoSIL: the full index is often unnecessary). 2. **BFS outward** from those seed files over `import`/`invoke`/`contain` edges, with a **boolean relevance gate** replacing the model (CoSIL's one learned node, made deterministic here): a path is expanded iff the candidate snippet's identifiers, error strings, and callee signatures show **≥1 overlap with the `IssueBundle`** (token-level: any `IssueBundle` identifier/error-string token appears in the snippet); a **reject** answer prunes the entire path. Priority is a heap ordered by structural distance from the issue's named symbols (OrcaLoca's scheduler, de-LLM'd). 3. **Bound the BFS at ≤10 iterations** (CoSIL's cap), and cap the expanded subgraph at **≤200 nodes**. **Output:** the seed candidates plus every node reached by a non-pruned path, each tagged with its structural distance from the seeds. This is the same structure as spec 04's call-graph neighbor expansion, but *lazy* (built only where the issue lives) and *relevance-gated*. ### 3.4 (d) Dynamic evidence (D, when tests exist) **Input:** the candidate subgraph + optional `repro_target`. **Output:** per-entity Ochiai suspiciousness, or a structured "no dynamic evidence" marker. **Only when a failing test/repro exists** (`repro_target` set, or a failing test discovered by component 01's `run_test`): 1. Run the failing test; collect **method-level coverage** (which methods each test executed — via stdlib `trace`/`sys.settrace`, or component 01's existing SBFL coverage path). 2. Compute **Ochiai SBFL** per covered method, with the **CORRECT formula** (spec 01 §4 Stage 1, which already carries this exact form): ``` suspiciousness(e) = failed_covered(e) / sqrt( total_failed × (failed_covered(e) + passed_covered(e)) ) ``` where `failed_covered(e)` = number of FAILING tests that execute `e`, `passed_covered(e)` = number of PASSING tests that execute `e`, `total_failed` = total failing tests. **Passed coverage must NOT be the numerator.** Methods with no coverage get `sbfl_score = null`. 3. **Prune stack traces** (LLM4FL): drop external-library frames (same filter as §3.1 `deepest_frame`); **truncate test bodies at the first failing assertion** — coverage beyond the first failed `assert` is discarded. 4. Sort the candidate subgraph by suspiciousness; this produces a per-entity `sbfl_score ∈ [0,1]`. **Degrade gracefully:** when coverage cannot be collected (no failing test, no `repro_target`, or the test harness is unavailable), the stage emits `{"dynamic_evidence": "none"}` and the remaining channels carry the weight — with the historical channel promoted (the paper's "classic SBFL fallback"). Dynamic evidence is **never** fabricated; its absence is reported, not averaged away. ### 3.5 (e) Historical evidence (D, when history exists) **Input:** the candidate subgraph + `repo_root`'s git history. **Output:** per-entity historical prior, or a structured "no history" marker. 1. **Churn hotspots:** `git log --numstat --format='%H'` over the repo; accumulate per-file change counts, take the **top-K = 200** files (Repository Memory's `K = 200`); normalize to `[0,1]`. 2. **Issue→fix pairs:** mine `git log` for commits whose messages reference an issue (`#\d+`, `fix`, `closes`, `resolves`, or a `Fixes #\d+` trailer); pair each commit's changed files with its message. Retrieve past fix files for the current issue via **BM25 over commit messages** with a lightweight tokenizer — **no embeddings, no GPU** (Repository Memory: BM25-with-LLM-tokenizer beats GritLM-7B, and plain BM25 is the deterministic round-1 form). 3. **Anti-leakage (exact, Repository Memory's protocol):** only **pre-date** commits (commit time strictly before the issue's fix date — the ground-truth patch commit is excluded); **filter overlapping-text issues** (drop any past issue whose text shares ≥ a threshold of tokens with the current issue, so self-matching cannot leak the answer). This makes history a genuine prior, not a memorized answer. 4. **Gate OFF when thin:** if the repo has < **20 commits** touching project files, or < **5** issue→fix pairs, the stage emits `{"historical_evidence": "none"}` and the prior is flat (uniform, contributes nothing to fusion). Thin history degrades the signal, so history is one fallible signal among several — never the sole authority. **Output:** a per-entity `history_score ∈ [0,1]` (churn × issue→fix, or churn alone when no fix pairs), or the `"none"` marker. ### 3.6 (f) Evidence fusion (D) **Input:** all computed per-entity evidence scores. **Output:** ONE ranked list. **Method (exact):** rank-combine the independent channels — semantic (the §3.2 BM25 + lexical), graph-distance (§3.2 channel 3 + §3.3 distance), SBFL (§3.4, when present), stack-trace (§3.1 `deepest_frame` proximity), and git-history (§3.5, when present). The default is **weighted rank-sum**, with the weights configurable: ``` rank_sum(e) = Σ_c w_c · (1 / rank_c(e)) w_semantic = 0.35, w_graph = 0.20, w_sbfl = 0.25, w_stack = 0.10, w_history = 0.10 ``` Channels that returned `"none"`/`null` are **dropped from the sum** (their weight is renormalized across the remaining channels — never imputed as 0.5, never averaged with test-present cases). An alternate **reciprocal-rank fusion** (`Σ_c 1/(k + rank_c(e))`, `k = 60`) is behind a config flag (`FUSION=rrs|rrf`, default `rrs`) for the §7 ablation. The output is a single fused ranking of candidate entities. ### 3.7 (g) Hierarchical ranking + confidence test (D) **Hierarchical ranking (exact):** group fused-ranked entities by their `file`, then order files by their best member's fused score → **files**; within each file, order entities by fused score → **entities**; within each top entity, order its statements by per-line fused contribution (the §3.4 coverage suspiciousness where present, else the BM25 line contribution) → **statements**. The output is a three-level ranked structure, capped at **top-10 files, top-5 entities per file, top-5 statements per entity** (statement granularity is spent only on survivors, per P1). **Causal story (exact):** each top candidate carries a one-line story assembled from the agreeing signals, following the paper's template — "entity *e* fails because its state under condition *c* is inconsistent with what its callers at distance *d* assume, evidenced by coverage suspiciousness *s* and history *h*" — rendered compactly as `"X fails because Y under Z"` (X = the symptom, Y = the entity's behavior, Z = the condition/caller context). A candidate whose evidence is semantic-only carries a weaker story and is ranked below a candidate backed by dynamic or historical agreement. **Confidence test (exact):** compute each *available* channel's top-1 candidate. If **the same top-1 entity is shared by ≥3 channels** (configurable `AGREE_THRESHOLD = 3`) and no available channel's leader strongly contradicts the consensus (no channel ranks the consensus leader below top-10), the pipeline **STOPS** and emits the hierarchical ranking with causal stories — **no LLM**. Otherwise the channels **DISAGREE** and the single learned step (§4) runs, once. **Output schema (exact):** ```json { "ranking": { "files": [ { "file": "src/memstore.py", "score": 0.81, "entities": [ ... ] } ], "entities":[ { "qualname": "repo.memstore.MemStore.remember", "kind": "method", "file": "src/memstore.py", "line": 12, "score": 0.77, "causal_story": "remember fails because kind=None under recall-import of the caller", "evidence": { "semantic": 0.9, "graph": 0.6, "sbfl": 0.71, "stack": 0.8, "history": 0.4 } } ] }, "confidence": "agree" | "disagree", "agreement": { "top1": "repo.memstore.MemStore.remember", "channels_agreeing": ["semantic","sbfl","stack"], "count": 3 } } ``` --- ## 4. The LLM Re-rank (L, only on disagreement) The single learned step. It runs **only** when the §3.7 confidence test returns `disagree`, and it is the **ONLY** LLM call in the entire localization stage. In the agree case it is **skipped** and the L column is empty. **Prompt inputs (assembled deterministically, in this exact order):** 1. **`issue_text`** — verbatim. 2. **The top-K shortlist** — the fused top-40 candidates, each rendered as one line: `{entity_id, file, line_range, one-line causal hint}` — **identifiers and causal hints, never bodies, never whole files** (the SweRank insight: the re-ranker need only predict the single positive identifier). **Output schema (exact — one JSON object, an ordered list of IDs + one-line reason each):** ```json { "reranked": [ { "id": "repo.memstore.MemStore.remember", "reason": "caller assumes kind is always set" }, ... ] } ``` **Caps:** **one call**, `temperature = 0`, `max_completion_tokens = 1024`, shortlist ≤ **40** items (each item ≤ 200 chars). The LLM **reads the shortlist and emits IDs** — it does **not** explore the repository, does not see bodies, does not call tools. It is a listwise re-ranker, never an explorer. The re-ranked order is spliced into the §3.7 hierarchical ranking (the model's order replaces the fused order only within the shortlist; the causal stories are still the deterministic ones). Any parse failure of the JSON is treated as a **no-op**: the deterministic fused order stands unchanged and the stage emits `{"rerank": "failed", "fallback": "deterministic_order"}` — never a re-prompt, never a second LLM call. --- ## 5. Implementation Order Eight phases, each independently testable and reversible. Each lands in `build/05-issue-fault-localization/` as a standalone importable module. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Issue-signal mining** | `mine.py` (§3.1 extractors + `IssueBundle`) | Fixture issue text + a pasted traceback: assert `identifiers` extracts dotted names (resolved vs unresolved), `file_paths` matches, `error_strings` matches `KeyError:` lines, and `deepest_frame` skips a `site-packages` frame to land on the deepest non-library frame; no traceback → `deepest_frame = null`. | Delete `mine.py` (pure read). | | **2. Fused pre-filter (BM25 + lexical + graph)** | `prefilter.py` (§3.2, reuses component 03/04's index + `bm25.py`) | Fixture repo: a candidate whose *symbol* matches the issue outranks one whose *comment* matches (3× weight); a candidate one `invoke` hop from an issue-named symbol scores > a 9-hop candidate; output never exceeds K=40; **assert no embedding/neural call is made** (a probe patches `fastembed`/`memstore.embed` to raise — the filter must not touch it). | Delete `prefilter.py` (pure read). | | **3. Lazy graph expansion + relevance gate** | `expand.py` (§3.3 lazy import graph + gated BFS ≤10) | Fixture repo with a known cross-file call: the issue-mentioned file's relevant neighbor is reached within ≤10 iters; a neighbor with zero `IssueBundle` token overlap is pruned (not present in output); a self-contained leaf repo returns just the seeds; subgraph ≤200 nodes. Assert the graph is built **only** for issue-mentioned files, not the whole repo. | Delete `expand.py` (pure read). | | **4. Ochiai SBFL + stack-trace prune** | `sbfl.py` (§3.4 corrected Ochiai + prune) | Fixture repo with a seeded fault + failing test: assert `suspiciousness(e)` matches the §3.4 formula exactly (hand-computed `failed_covered`/`passed_covered`/`total_failed`); a method covered only by passing tests scores **0**, not a false positive; coverage past the first failed `assert` is discarded; no repro → `{"dynamic_evidence": "none"}` (no crash). | Delete `sbfl.py` (pure read). | | **5. Git-history evidence + anti-leakage** | `history.py` (§3.5 churn + issue→fix pairs + anti-leakage) | Fixture git repo with planted history: churn hotspots rank the repeatedly-touched file first; a past fix whose commit message BM25-matches the issue surfaces its file; a **post-date** commit and an **overlapping-text** issue are both excluded (anti-leakage); a repo with <20 commits → `{"historical_evidence": "none"}`. | Delete `history.py` (pure read). | | **6. Fusion + hierarchical ranking + confidence test** | `fuse.py` (§3.6), `rank.py` (§3.7 hierarchical + causal stories + agree/disagree) | Fixture with scripted per-channel scores: assert the fused order matches the §3.6 weighted rank-sum formula; a `"none"` channel is dropped and weights renormalized; the three-level file→entity→statement structure is produced; 3 channels sharing top-1 → `"agree"` with no LLM call (probe the LLM client to raise); 2-vs-2 split → `"disagree"`. | Delete `fuse.py` + `rank.py`. | | **7. Disagreement-case LLM re-rank** | `rerank.py` (§4 shortlist assembly + one call + output schema + parse-failure fallback) | Mock the LLM client to return a scripted ordered-ID list: assert the re-ranked order is spliced in, **one** call with `temperature=0` and ≤1024 tokens, shortlist ≤40 items rendered as `{id, file, line_range, hint}` with no bodies; a malformed JSON response → `{"rerank":"failed"}` and the deterministic order stands, **no second call**. | Delete `rerank.py`. | | **8. A/B + honesty harness** | `ab.py` (feature-flag router + localization-metric capture + oracle diagnostic + anti-memorization control + decision) | End-to-end on 3 warm-up tasks; `ab.py` runs old-vs-new on the §7 sample, emits file/function recall@1/3/5 + MRR + distance-to-edited-entity, the oracle-localization gap, the out-of-benchmark drop, and resolve/cost/tokens + the §7 decision. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 8) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | BM25/IDF scorer | §3.2/§3.5 retrieval | A **~50-line stdlib impl** (pure Python, no pip), reusing component 04's `bm25.py`; optionally SQLite FTS5 `bm25()`. | **No** | | `git log` | §3.5 churn + issue→fix mining | **Confirmed installed** — `git` is present (used by the recovery/backup and code_check flows). | **No** | | tree-sitter / stdlib `ast` | signal mining + graph edges | Reuse component 03's `recon.db` (built by stdlib `ast` for Python); no new parser. | **No** | | Ochiai SBFL | §3.4 dynamic evidence | stdlib `trace`/`sys.settrace`; the **corrected formula from spec 01 §4** (already specified and verified there). | **No** | | Component 03/04 index & graph (`recon.db` + `api.db` `edges`/`nodes`) | pre-filter + graph edges + expansion | In-repo, built by specs 03/04. | **No** | | stdlib `re`, `json`, `subprocess`, `sqlite3`, `difflib`, `hashlib` | extraction, storage, rouge, hashing | CPython stdlib. | **No** | | SweRankEmbed **137M** bi-encoder (row 10) | the learned retriever (SweLoc-trained) | **FLAG: LATER OPTIONAL.** Needs a **model download**. Must be **>72h old** AND (≥10 GitHub stars OR deep-scrutinized via `~/mailtool/security_director.py screen "gh:.../SweRank"`) plus `package_guard.py test`. **Not installed in Round 1.** | **Flagged (later, optional)** | | SWE-bench-Verified dataset | A/B sample (§7) | Not a pip package; a data fetch. Not installed locally. | **Flagged (data, not code)** — see §7 fallback | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: BM25 is a ~50-line stdlib function, `git log` is installed, the graph/index come from components 03/04, and the SBFL formula is the corrected Ochiai form already specified in spec 01. **No new package is installed for Round 1.** The 137M bi-encoder is the only flagged installable, and it is a *later-round* option that must clear the 72h + star/scrutiny + sandbox-test gate before any download — never in Round 1. --- ## 7. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as spec 01** (Python-only, single-repo, no-Docker reproduction). Held out = never seen by any tuning, prompt, or ranking pass. Seeded RNG. **Fallback** (if the dataset cannot be fetched locally): `coding-cortex/piperbench` sampled to 20 tasks, held out identically. ### 7.1 (a) Localization metrics — measured SEPARATELY and HONESTLY Localization is graded on localization, never on the patch it feeds. Ground truth is extracted from the **merged patch itself** — the file, entity, and statement the maintainer's fix actually touched — not from any model claim. For each `(repo, issue)` pair, against that ground truth: 1. **file recall@1/3/5** — fraction of gold files recovered in the top-1/3/5 ranked files. 2. **function recall@1/3/5** — fraction of gold entities recovered in the top-1/3/5 ranked entities. 3. **MRR** — mean reciprocal rank of the first gold entity in the ranked list. 4. **distance-to-edited-entity** — structural distance (hops in `recon.db`'s edge graph) from the top-ranked candidate entity to the true edited entity; rewarding "wrong function, right neighborhood" over a correct-looking wrong answer far away. **Lower is better.** These are run **standalone** (no patch, no Stage 2/3 of component 01), on the *same* held-out sample. Every reported number must carry the memorization caveat (§7.3). ### 7.2 (b) The oracle-localization diagnostic Each evaluated task runs repair **twice** on the harness's own weak model (`deepseek-v4-pro`): once with **oracle** locations (the ground-truth sites, hand-supplied) — the *upper bound* — and once with the pipeline's **retrieved** locations. The **gap** (oracle-resolve − retrieved-resolve) is the **localization ceiling**: the portion of resolve rate recoverable by improving localization alone. OrcaLoca's measured chain (better localization → +6.33 pp resolve) makes this gap real and quantifiable; this diagnostic turns it from a reported finding into a standing measurement. A localization-metric win that leaves the gap unchanged is **not** credited — the ranking must *close* the gap, not merely reorder within it. ### 7.3 (c) The anti-memorization control Report localization on **held-out / out-of-benchmark repositories** — issues the model weights have not plausibly seen — and report the **drop** from in-benchmark to out-of-benchmark as a contamination indicator, per the SWE-Bench Illusion. Any component that "works suspiciously well on Django" but collapses out-of-benchmark is flagged for memorization, not credited for reasoning. Additionally, report **per-repo git-history availability** as a disclosed confound (history-rich and history-poor repos are not the same measurement), and report the **no-repro fallback** quality (issues with no failing test — where dynamic evidence is absent and semantic/structural/historical carry the load) as its own sub-metric, never silently averaged into test-present cases. ### 7.4 (d) End-to-end regression guard 5. **Resolve rate** = `resolved / total` (spec 01's score). 6. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). 7. **Tokens** = `tokens / instance`. ### 7.5 Decision rule (explicit thresholds) Run candidate-vs-current on the *identical* sample with the *identical* model (`deepseek-v4-pro`) and hardware (this box). The candidate **wins iff**: ``` ( file_recall@1 OR file_recall@3 OR file_recall@5 OR function_recall@1 OR function_recall@3 OR function_recall@5 OR MRR OR (distance_to_edited_entity strictly decreases) ) strictly improves (≥1 metric up) AND file/function recall@1/3/5, MRR do NOT regress (each ≥ current) AND distance_to_edited_entity_candidate ≤ distance_to_edited_entity_current AND resolve_candidate ≥ resolve_current (no end-to-end regression) AND cost_candidate ≤ 1.5 × cost_current AND tokens_candidate ≤ 1.5 × tokens_current ``` At least one localization metric must **strictly** improve, and **none** may regress; end-to-end resolve must not regress; cost and tokens are bounded at 1.5×. A candidate that improves localization but regresses resolve or blows a cost/token bound is **not adopted**; the tradeoff is recorded explicitly. If the candidate wins, it replaces the current localization path (component 01 Stage-1's sim+SBFL top-10); if it loses, the build tree is kept for the record and the live harness is unchanged. **Ablation rule (mechanical).** Each evidence channel (§3.2 lexical, §3.2 graph, §3.4 SBFL, §3.1 stack-trace, §3.5 history) is removed in isolation, everything else held fixed, and the delta measured on the localization metrics (§7.1). A channel is **deleted** iff its removal changes none of file/function recall@1/3/5, MRR, or distance-to-edited-entity beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). Applied first to the weakest a-priori cases (the stack-trace channel and the lexical Rouge channel), then to any other channel — regardless of how sensible it looks. This is the paper's "a stage may be removed when it does not move the honest metrics" rule, made mechanical. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/05-issue-fault-localization/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `LOCALIZE=loc05|current`, default `current`) routes the A/B; the flag alone switches between the new localizer and the current component-01 Stage-1 path — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. **Invariants (must hold, checked at the end of every phase).** 1. **No new `.md` files outside this folder** (`projects/self-improvement/{papers,specs,build}`). Everything else is code or DB rows. 2. **No change to the live harness** until the §7 A/B gate passes. The current tool surface stays byte-identical during the entire build. 3. **Localization context is bounded**: the pre-filter truncates to K=40, the expanded subgraph is ≤200 nodes, and the hierarchical ranking is capped at top-10 files / top-5 entities / top-5 statements. The model is never handed the whole repo or full-file bodies. 4. **The LLM is a re-ranker, never an explorer**: the only LLM step is the §4 disagreement-case listwise re-rank over a shortlist of `{id, file, line_range, hint}` — it reads IDs and emits IDs, and is **skipped** in the agree case. Any phase that introduces an LLM into a D node (mining, pre-filter, expansion, SBFL, history, fusion) is rejected outright. 5. **The SBFL formula is the corrected Ochiai form**: `failed_covered / sqrt(total_failed × (failed_covered + passed_covered))`, with passed coverage **never** in the numerator. Any other SBFL variant is a rejected change, not a tuning knob. 6. **Honesty — never report a SWE-bench number without the memorization caveat**: every localization number ships with the out-of-benchmark drop and the per-repo history disclosure (§7.3). A metric reported without those two controls is a rejected deliverable. 7. **Deterministic stages stay deterministic**: stages (a)–(g) of §3 are D; the L column holds exactly one entry (the §4 re-ranker), reached only on disagreement, and empty in the common agreement case. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*