Summary / 03. Repository reconnaissance / Document
Spec 2026-09-04
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 03 — Repository Reconnaissance (the ≈80% fraction) > Derived from `papers/03-repository-reconnaissance.md`. This document drives the build of the > **repository-reconnaissance stage** that sits *beneath* component 01 (the six primitives) and > component 02 (the tool-calling policy): a deterministic, graph-backed map of the repo that is > built in code before the model runs, and a demand-driven retrieval ladder that hands the model a > tiny, tree-encoded slice instead of the repository. It translates the paper's design into exact > parsers, exact JSON schemas, exact gate conditions, exact injection shapes, exact budgets, and an > exact build sequence — no further research required. Where the paper says "≈80%", this spec pins > *which* 80%. --- ## 1. Purpose & Scope Build a **reversible repository-reconnaissance stage** — a deterministic two-artifact scan (an evidence-backed **repository manifest** + a **line-level code graph**), a **level-0 deterministic gate**, a bounded **R0→R2 escalation ladder**, and a **tree-encoded k=1 ego-graph** injection — as a drop-in *map provider* for the harness of component 01 and the policy of component 02, not an accretion onto them. The overall target is the paper's full 100% spec (full manifest coverage across all ecosystems, the full line-level graph, and the complete R0→R4 ladder). **Round 1 lands the expected ≈80% of it**: everything deterministic (per-ecosystem manifest parsers, the tree-sitter/ast line-level graph, the level-0 gate, and the R0→R2 ladder), targeting the reachable fraction of the corpus — the common ecosystems (Python/npm/Go/Rust/Maven) and the shallow rungs of the ladder. The R3/R4 subsystem-and-repo-wide rungs, the full RIG/SPADE CMake extractor + manual non-CMake authoring, a fine-tuned LocAgent-style index, and the SeeRepo visual map are **not** built in Round 1 and are **not excluded** — they are the content of later rounds that climb from that ~80% landing toward 100% (see §2). Round 1 is kept only if it wins a measured A/B test on the reconnaissance metrics *and* does not regress end-to-end resolve/cost/tokens (§9). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element from the paper is classified below by *which round builds it*. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–7). - **DEFERRED = later rounds** — the deep ladder rungs and the ecosystem/manual extractors (rows 8–11) that Round 1 intentionally does not build *yet*, because they need the shallow core underneath them first or per-ecosystem hand-authoring. They are **not excluded from the project** — they are the content of later rounds, the climb toward 100%, gated by the same §9 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | Repo manifest — per-ecosystem parsers (`pyproject.toml`, `setup.py`, `requirements*.txt`, `package.json`, `go.mod`, `Cargo.toml`, `pom.xml`, `Makefile`/`justfile`, `.github/workflows/*`, `Dockerfile`) | **BUILD-NOW** | | 2 | Line-level code graph — tree-sitter (or stdlib `ast` for Python) def/ref nodes + invoke/contain/import edges, filtered to project-internal | **BUILD-NOW** | | 3 | Level-0 deterministic gate (6 signals) + R0→R2 escalation, each level gated on prior evidence | **BUILD-NOW** | | 4 | k=1 ego-graph retrieval + tree-encoding (LocAgent-style tree render) | **BUILD-NOW** | | 5 | Standalone measurement battery (deterministic-questions + ranked-region probe) | **BUILD-NOW** | | 6 | A/B runner (feature-flag router + joint-metric capture) | **BUILD-NOW** | | 7 | Token budget enforcement (manifest ≤5K once, recon slice ≤4K, k=2 flatten forbidden) | **BUILD-NOW** | | 8 | R3 (subsystem: component + aggregators/runners) and R4 (repo-wide: manifest + global entry points) | **DEFERRED** — needs the R0→R2 core first | | 9 | Full RIG/SPADE CMake/CTest extractor + manual authoring of non-CMake build systems | **DEFERRED** — per-ecosystem hand-authoring | | 10 | Fine-tuned LocAgent-style sparse index (fully-qualified-name index + BM25 + code chunks) | **DEFERRED** — needs fine-tuning/training pass | | 11 | SeeRepo visual map (image rendering of repo structure) | **DEFERRED** — presupposes a vision-capable model, off the CPU path | **Level-0 gate decision: BUILD-NOW-DETERMINISTIC** (a Round 1 build item). The paper's gate is already non-learned — a deterministic signal check keyed on unresolved symbol / cross-file import / failing-test location / dependency edge / config reference / named-outside-module. There is nothing learned to defer. The only *learned* node in the whole stage is the model's final "what to inspect next" decision, which is expressed through component 01's bounded primitives, not free exploration — and even that is outside this build's D nodes. --- ## 3. The Deterministic Scan (D) Two artifacts, both built **entirely in code before the model runs**, both stored under `build/03-repository-reconnaissance/`. Neither ever consults the model. ### 3.1 (a) REPO MANIFEST — per-ecosystem parsers **Input:** `repo_root: str`. **Output:** one flat, evidence-backed JSON (target **≤ 5,000 tokens** when rendered; a token counter enforces the cap and truncates the largest list first if exceeded). **Files parsed (exact list), each with a dedicated parser, each mapping to schema fields:** | File (glob) | Extracts | |---|---| | `pyproject.toml` | languages (`[tool.poetry]`/`[project]` blocks), package manager, dependencies (`project.dependencies` / `tool.poetry.dependencies`), entry points (`[project.scripts]`, `[tool.poetry.scripts]`), build/lint/test (`[tool.pytest.ini_options]`, `[tool.ruff]`, `[build-system]`) | | `setup.py` / `setup.cfg` | package manager (setuptools), dependencies (`install_requires`, `extras_require`), entry points (`entry_points`, `console_scripts`) | | `requirements*.txt` | dependencies (name + optional version pin, one per line) | | `package.json` | languages (javascript/typescript), package manager (npm/yarn/pnpm via lockfile), dependencies/devDependencies, scripts (`scripts.build/test/lint`), entry points (`main`, `bin`) | | `go.mod` | languages (go), package manager (go modules), dependencies (`require` block), module path | | `Cargo.toml` | languages (rust), package manager (cargo), dependencies (`[dependencies]`/`[dev-dependencies]`), entry points (`[[bin]]`), test/lint (implicit `cargo test`/`cargo clippy`) | | `pom.xml` | languages (java), package manager (maven), dependencies (`<dependencies>`), build/test/lint (plugin goals `maven-compiler`, `maven-surefire`, `maven-checkstyle`) | | `Makefile` / `justfile` | runners (each target name → command string), build/test/lint if targets named `build`/`test`/`lint` (or regex `^build|^test|^lint|^check`) | | `.github/workflows/*.yml`/`.yaml` | runners (each `run:` step), test/lint commands, CI entry evidence | | `Dockerfile` | build command (final `CMD`/`ENTRYPOINT`), languages (base image), module layout hint (WORKDIR) | **Exact output JSON schema** (top-level keys fixed; every claim carries a `"file:line"` evidence pointer; a claim with no such pointer is illegal): ```json { "repo": "repo-name", "languages": [ { "name": "python", "version_hint": ">=3.9", "evidence": "pyproject.toml:12" } ], "package_manager": { "name": "poetry|pip|npm|yarn|pnpm|go|cargo|maven|setuptools|null", "evidence": "pyproject.toml:3" }, "components": [ { "name": "src/foo", "path": "src/foo", "kind": "package|module|app|lib", "evidence": "src/foo/__init__.py:1" } ], "dependencies": [ { "name": "requests", "version": ">=2.28", "kind": "runtime|dev", "evidence": "requirements.txt:5" } ], "tests": [ { "framework": "pytest", "command": "pytest tests/", "evidence": ".github/workflows/ci.yml:14" } ], "external-packages":[ { "name": "vendor/x", "path": "vendor/x", "evidence": "vendor/x/setup.py:1" } ], "runners": [ { "name": "build", "command": "python -m build", "evidence": "Makefile:8" }, { "name": "lint", "command": "ruff check .", "evidence": "Makefile:12" }, { "name": "test", "command": "pytest tests/", "evidence": "Makefile:15" } ], "entry-points": [ { "name": "cli.main", "kind": "console-script|bin|main", "evidence": "pyproject.toml:22" } ], "layout": { "conventions": ["src-layout", "tests/ mirrors src/"], "evidence": "pyproject.toml:5" }, "errors": [ { "file": "setup.py", "reason": "unparseable: <exception text>", "line": null } ] } ``` - **Field semantics:** `components` = first-party code dirs/modules; `dependencies` = declared third-party packages; `tests` = framework + the exact test command; `external-packages` = vendored or third-party code *in-tree* (to be recognized and left alone); `runners` = the build/test/lint command strings; `entry-points` = program/module entry points; `layout` = module layout conventions; `errors` = the fail-soft log. - **`runners` commands are canonicalized** from the file in this priority order: `Makefile`/`justfile` named targets > CI workflow `run:` steps > package-manager conventions (e.g. `pytest` for Python, `cargo test` for Rust, `go test ./...` for Go). If none found, the runner is **omitted**, not guessed — the paper's "never guess the test runner" rule. - **Fail-soft (exact):** an unparseable file (TOML/YAML/JSON syntax error, or a parser raising) is *never* a build failure — it is appended to `errors` with the exception text and the file is skipped. The manifest always emits; missing fields are empty lists / `null`. ### 3.2 (b) LINE-LEVEL CODE GRAPH — def/ref nodes + edges, filtered **Parser:** `tree-sitter` with per-language grammars; **fallback for Python-only repos: stdlib `ast`** (see §8 — the fallback is what makes Round 1 zero-install). For every source file: 1. **Parse** the file into an AST; walk it to emit **nodes** and **edges**. 2. **Filter (exact):** drop every node/edge whose target resolves to a **builtin**, **standard library**, or **third-party** symbol. Project-internal = the symbol resolves to a def in a file under `repo_root` and not under a vendored path (`vendor/`, `node_modules/`, `site-packages/`, `third_party/`, `external/`). Only project-internal edges are kept. (Names that cannot be resolved at all are kept as *unresolved* nodes — they are a level-0 gate signal, §4.) 3. **Organize** into the graph. **Node schema (one row per def or ref, per code line):** ```json { "id": "repo.module.func:12", "kind": "function|class|var|def|ref", "name": "func", "qualname": "repo.module.func", "file": "module.py", "line": 12, "col": 0, "end_line": 20 } ``` **Edge schema (three types):** ```json { "src": "repo.module.main:15", "dst": "repo.utils.parse_args:8", "type": "invoke|contain|import" } ``` - **invoke** = a call/reference site (`src`, a `ref` node) → its definition (`dst`, a `def` node). - **contain** = a definition (`src`) → its enclosing scope/component (`dst`). - **import** = an import statement (`src`) → the imported project-internal module/def (`dst`). **Storage (exact):** a single SQLite file `build/03-repository-reconnaissance/recon.db` with tables `nodes(id TEXT PRIMARY KEY, kind TEXT, name TEXT, qualname TEXT, file TEXT, line INT, col INT, end_line INT)` and `edges(src TEXT, dst TEXT, type TEXT)` plus a name index `name_index(qualname TEXT)` for lookup (the LocAgent fully-qualified-name dictionary). This is a *separate* artifact from `memory/codegraph.py`'s `code_nodes` (which stores signatures only) — the graph is a superset and lives in the build tree so the whole stage is delete-reversible. **Build cost (exact):** one-time **offline** parse. Target **≈2–10 ms/file** (tree-sitter) or **≈2–5 ms/file** (stdlib `ast`); a 500-file repo lands in **≈1–5 s**, cached. **Invalidation:** the graph is keyed by `sha256(file_bytes + path)` per file; on a later run only changed files re-parse. The build is run once per repo, never per model turn. --- ## 4. Demand-Driven Retrieval (D gate + L selection) Retrieval is a ladder whose first rung is a gate. The gate and every ladder rung are **D**; only the model's "what to inspect next" decision is **L**, and it is expressed through component 01's primitives. ### 4.1 LEVEL-0 DETERMINISTIC GATE — the six signals Inputs: `task_text: str`, `current_entity: str` (the entity the model is currently working in), the manifest, the graph, and the spec-01 Stage-1 SBFL output (if a repro test exists). The gate checks **any** of: | # | Signal | Exact trigger | |---|---|---| | S1 | **unresolved symbol** | `task_text` names an identifier not resolvable to a `def` in `current_entity`'s scope (regex over the symbol token per spec 02 §3.1; lookup fails in `name_index`) | | S2 | **cross-file import** | `current_entity`'s file has an `import` edge whose `dst` is in a *different* project file | | S3 | **failing-test location** | spec-01 SBFL top entity lies in a file *outside* `current_entity`'s file | | S4 | **dependency edge** | `current_entity` has any `invoke`/`import` edge to a `def` in a different project file | | S5 | **config reference** | `task_text` contains a config key/path (e.g. `DATABASE_URL`, `pyproject.toml`, `settings.yaml`, `env.`) whose evidence pointer is outside `current_entity`'s file | | S6 | **named-outside-module** | `task_text` contains a dotted identifier matching a `component`/`entry-point` in the manifest that is not `current_entity`'s own component | **No signal fires → retrieve NOTHING.** The stage returns only: ```json { "retrieved": false, "entity": "repo.memstore.MemStore.remember", "manifest_header": { "languages": ["python"], "package_manager": "poetry", "build": "python -m build", "test": "pytest tests/", "lint": "ruff check ." } } ``` (≈ **≤500 tokens** — the manifest header, not the full manifest.) The model works in the current module alone, honoring Repoformer's ~20% finding. **Any signal fires → escalate R0.** ### 4.2 The escalation ladder — exact conditions Each rung is **gated on evidence from the prior rung**; the ladder stops at the first rung whose evidence is satisfied, and never jumps a rung. | Rung | Content (exact) | Escalation condition (exact) | |---|---|---| | **R0** | `task_text` + the **named entity** (the symbol/file the task references — the S1/S5/S6 match, or the S3 SBFL top entity, or the S2/S4 edge target) | Always the first rung after the gate fires | | **R1** | the **local map** — the entity's own scope: its def + immediate same-file neighbors (its `contain` parent + same-file `invoke` targets) | **R0→R1** fires iff the R0 named entity has ≥1 project-internal edge (`invoke`/`import`/`contain`) *or* is non-self-contained (its file imports/refs anything project-internal). A self-contained leaf → stop at R0. | | **R2** | the **dependency neighborhood** — a k=1 ego-graph over `invoke`/`import` edges from the entity (≈12 nodes / ≈37 edges, §5) | **R1→R2** fires iff R1's map contains a cross-file edge whose `dst` is *not already included* in R1 (i.e. the neighbor is defined in another file and still missing). No such edge → stop at R1. | | **R3** | the **subsystem** — containing component + its aggregators and runners | **DEFERRED** (§2 row 8) | | **R4** | **repository-wide** — the full manifest + global entry points | **DEFERRED** (§2 row 9) — reached only on explicit cross-subsystem evidence | The **escalation is monotone and bounded**: R0 → R1 → R2 → stop. It can never loop, never re-fetch the same level, and never reaches the deferred R3/R4 in Round 1 (those rows are *absent*, not silently skipped — a Round-1 run that would need R3/R4 records `"level": "r3_required_deferred"` and proceeds with R2). --- ## 5. Context Injection (D) The model receives **a tree-encoded k=1 ego-graph plus skeleton signatures — never the whole repo.** The rendering follows LocAgent's tree form (models read trees better than edge lists); the signatures follow component 01's convention (**signatures, not bodies**). **Exact injection shape** (assembled deterministically, in this order): ``` ## Repository context ### Manifest header languages: python · package_manager: poetry · build: `python -m build` · test: `pytest tests/` · lint: `ruff check .` ### Dependency ego-graph (k=1, tree-encoded) repo.cli.main (function) — "def main(argv: list[str]) -> int:" [cli.py:12] ├── repo.utils.parse_args:invoke [cli.py:15] │ └── repo.utils.parse_args (function) — "def parse_args(argv):" [utils.py:8] ├── repo.db.connect:invoke [cli.py:18] │ └── repo.db.connect (function) — "def connect(uri: str):" [db.py:4] └── repo.cli:contain [cli.py:1] ``` **Tree rules (exact):** - Root = the current/selected entity. Each child edge is `├──` / `└──`, indented two spaces per depth; an `invoke`/`import` edge is rendered as `qualname:<type>` at the *call site*, with its target `def` rendered as a **child** line `qualname (kind) — "skeleton signature"`. - `contain` edges are rendered as a self-reference node line; no recursive expansion. - **Skeleton signatures** = component 01's signature-only convention (`def f(...):` one line). **No bodies, no line-span ranges, no diffs** anywhere in the model-facing text. (`[file:line]` is the *only* positional info shown, for the model to name selectors in component-01 calls; it is kept out of the primitives' own output per spec 01 §3.) **Token budget (exact):** | Slice | Budget | |---|---| | Manifest header (no-retrieval path) | **≤ 500 tokens** | | Tree-encoded k=1 ego-graph + signatures (retrieval path) | **≤ 3,500 tokens** | | **Total reconnaissance context at any moment** | **≤ 4,000 tokens** | | Full manifest (the 5K artifact) | injected **once**, at session start, **not** re-injected per turn | **The k=2 rule (hard, encoded as an assertion):** the ego-graph is **k=1 only**; a flattened k=2 expansion is **forbidden** — RepoGraph shows it drops to 26.0%, *below baseline* (it hurts). The tree renderer has a `max_radius = 1` parameter and a unit test asserting no node is ever ≥2 hops from the root. Expanding to k=2 flattened is a rejected change, not a tuning knob. --- ## 6. Measurement (standalone, separate from coding) Reconnaissance is graded on reconnaissance, not on the patch. Two probe families, both run **separately** from the end-to-end resolve A/B (§9), against the *same* held-out sample. ### 6.1 (a) The deterministic-question battery Six fixed questions, graded **against the manifest's `file:line` evidence pointer**, not against model output — a pure test of the deterministic artifact, runnable with no model at all. | Q | Question | Graded against | |---|---|---| | Q1 | where are the tests for `Foo`? | `tests[].command` + `tests[].evidence` | | Q2 | what module defines `Bar`? | `name_index` lookup → the def node's `file:line` | | Q3 | what calls `Baz`? | all `invoke` edges whose `dst` == `Baz`'s def | | Q4 | what command builds this? | `runners[build].command` | | Q5 | which config selects the database? | the manifest field/evidence pointer for the DB config key (S5 signal source) | | Q6 | which files does changing API `X` plausibly affect? | the k=1 ego-graph of `X`'s def (all `invoke`/`import` neighbors) | **Score:** `question_accuracy = correct / 6`, where a question is correct iff the returned answer's `file:line` evidence pointer matches the ground-truth pointer (Q3/Q6 compare *sets* of nodes, exact set equality). This is the RIG architecture-question metric implemented on the deterministic artifact. ### 6.2 (b) The ranked-region probe (RepoBench-R / SWE-Explore style) Given a `(repo, issue)` pair, the stage must return a **ranked list of code regions under a fixed line budget**. Ground truth = the gold region(s) from the instance's gold patch location (or, in SWE-Explore's manner, from ≥2 successful agent trajectories). **Task:** rank regions (files, or def spans) by relevance; accumulate a ranked list until the line budget **L = 100 lines** is exhausted. **Exact scoring:** - **recall@5** = `|{gold regions ∩ top-5 ranked regions}| / |gold regions|` (region-level). - **Context Efficiency** = `(lines within the 100-line budget that lie inside a gold region) / 100` — the fraction of the budget spent on the gold region; higher is better. (This is the metric SWE-Explore reports correlating with repair at r=0.950.) Both probes are **standalone**: they score the recon stage's *map and retrieval*, never a patch, and never go through Stage 2/3 of component 01. The end-to-end resolve A/B (§9) is a **third, separate** measurement that only checks for regression. --- ## 7. Implementation Order Six phases, each independently testable and reversible. Each lands in `build/03-repository-reconnaissance/` as a standalone importable module. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Repo manifest parsers + JSON schema** | `manifest.py` (per-ecosystem parsers + canonicalization + token counter), `schema/manifest.schema.json` | Fixture repos (one per ecosystem: pyproject/package.json/go.mod/Cargo.toml/pom.xml/Makefile/CI/Dockerfile): assert each field's `evidence` pointer exists at that `file:line`; a deliberately-broken `setup.py` lands in `errors`, not a crash; rendered manifest ≤ 5000 tokens. | Delete `manifest.py` + schema; no DB writes, pure read. | | **2. Line-level code graph builder** | `graph.py` (tree-sitter/ast parse → nodes/edges → filter → `recon.db` + `name_index`) | Fixture repo with a known cross-file call: assert the `invoke` edge `caller→callee` exists, the stdlib `print` call produces **no** edge, a third-party import produces **no** edge, and an unresolved name produces an *unresolved* node; assert per-file parse ≤ 10 ms. | Delete `graph.py` + `recon.db` (build-tree artifact only). | | **3. Level-0 gate + R0/R1 escalation** | `gate.py` (6 signals + the R0/R1 ladder + stop-at-R0/stop-at-R1) | Scripted `(task_text, current_entity, manifest, graph, sbfl)`: a self-contained entity + no signal → `{"retrieved": false, ...}`; each of S1–S6 individually fires → escalates; a self-contained leaf stops at R0; an entity with an in-file-only edge stops at R1; an entity with a cross-file edge escalates to R2. | Delete `gate.py`. | | **4. k=1 ego-graph retrieval + tree encoding** | `inject.py` (k=1 ego-graph extractor + tree renderer + budget counter) | Assert the rendered tree is ≈12 nodes/≈37 edges on a fixture; `max_radius=1` enforced (no node ≥2 hops); signature-only (no body text present); total render ≤ 4000 tokens; a unit test asserts k=2 flattening raises/refuses. | Delete `inject.py`. | | **5. Standalone measurement battery** | `measure.py` (question battery + ranked-region probe + recall@5/CE scoring) | Run Q1–Q6 on a fixture with known answers → `question_accuracy` correct; run the ranked-region probe on a fixture with a known gold region → recall@5 and CE computed per §6.2. | Delete `measure.py` (pure read). | | **6. A/B runner** | `ab.py` (feature-flag router + joint-metric capture + decision) | End-to-end on 3 warm-up tasks; `ab.py` runs old-vs-new on the §9 sample, emits the recon metrics + resolve/cost/tokens + the §9 decision. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 6) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. --- ## 8. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Python-only Round 1** if we use stdlib `ast`; the only flagged install is `tree-sitter` for multi-language repos: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `ast`, `json`, `re`, `tomllib`, `sqlite3`, `hashlib`, `os` | Python AST graph (fallback), manifest parsers, name-index storage, file hashing | CPython stdlib. | **No** | | `tree-sitter` + `tree_sitter_{python,javascript,typescript,go,rust,java}` (or `py-tree-sitter`) | multi-language line-level graph | **FLAG: not yet installed as a multi-language graph dependency.** Must be **>72h old** (fresh uploads are a supply-chain vector) AND (≥10 GitHub stars OR deep-scrutinized via `~/mailtool/security_director.py screen "gh:tree-sitter/py-tree-sitter"`), plus a sandbox test (`package_guard.py test`) before install. | **Flagged** — see fallback | | stdlib `ast` for Python-only repos | line-level graph for Python repos, no install | Already in stdlib; mature, no supply-chain surface. | **No** | | `memory/codegraph.py` (SBFL / existing code_nodes), spec-01 Stage-1 localization | the S3 failing-test signal + skeleton-signature convention | In-repo, already vetted. | **No** | | `ruff`, `pytest` | only if the manifest's runners invoke them (they already exist for specs 01/02) | Installed in `~/venvs/memory`. | **No** | | SWE-bench-Verified dataset | A/B sample (§9) | Not a pip package; a data fetch. Not installed locally. | **Flagged (data, not code)** — see §9 fallback | **Preferred path: zero-install.** Round 1 uses stdlib `ast` for the graph and stdlib `tomllib`/`re` for the manifest parsers, which covers **Python-only repos with no install at all**. `tree-sitter` is adopted **only** if the A/B (§9) expands to a multi-language repo and passes the 72h + star/scrutiny + sandbox-test gate; until then the fallback stands and nothing is pip-installed. --- ## 9. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as spec 01** (Python-only, single-repo, no-Docker reproduction). Held out = never seen by any tuning, prompt, or ranking pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. **Measures (joint — recorded in two groups, no single-number score).** *Reconnaissance metrics (the thing being improved):* 1. **recall@5** (§6.2) — gold regions recovered in the top-5. 2. **Context Efficiency** (§6.2) — fraction of the 100-line budget on the gold region. 3. **question-battery accuracy** (§6.1) — `correct / 6`. *End-to-end regression guards (the thing that must not break):* 4. **Resolve rate** = `resolved / total` (spec 01's score). 5. **Cost** = `$ / instance`. 6. **Tokens** = `tokens / instance`. **Decision rule (explicit thresholds).** Run candidate-vs-current on the *identical* sample with the *identical* model (`deepseek-v4-pro`) and hardware (this box). The candidate **wins iff**: ``` recall@5_candidate > recall@5_current (strictly) AND ContextEfficiency_candidate ≥ ContextEfficiency_current (no recon regression) AND question_accuracy_candidate ≥ question_accuracy_current (no recon regression) AND resolve_candidate ≥ resolve_current (no end-to-end regression) AND cost_candidate ≤ 1.5 × cost_current AND tokens_candidate ≤ 1.5 × tokens_current ``` At least one of the three recon metrics must *strictly* improve, and none may regress; end-to-end resolve must not regress; cost and tokens are bounded at 1.5×. A candidate that improves recon but regresses resolve or blows a cost/token bound is **not adopted**; the tradeoff is recorded explicitly. If the candidate wins, it replaces the current recon path (which is "the model looks around"); if it loses, the build tree is kept for the record and the live harness is unchanged. **Ablation rule (mechanical).** Each gate signal and each ladder level is removed in isolation, everything else held fixed, and the delta measured on the recon metrics. A signal/level is **deleted** iff its removal changes none of `recall@5` / `Context Efficiency` / `question_accuracy` beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where the metric is continuous). Applied first to the weakest a-priori cases (S5 config-reference and the R1 local-map rung), then to any other signal or level — regardless of how sensible it looks. --- ## 10. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/03-repository-reconnaissance/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `RECON=graph|current`, default `current`) routes the A/B; the flag alone switches between the new stage and the current "model looks around" path — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. **Invariants (must hold, checked at the end of every phase).** 1. **No new `.md` files outside this folder** (`projects/self-improvement/{papers,specs,build}`). Everything else is code or DB rows. 2. **No change to the live harness** until the §9 A/B gate passes. The current tool surface stays byte-identical during the entire build. 3. **Recon context is always bounded**: manifest ≤ 5K tokens injected once; the retrieved slice ≤ 4K tokens; the model is never handed the whole repo or a flattened k=2 ego-graph. 4. **Never inject the whole repo** — any phase that injects a full-file dump or a k=2 flatten is rejected outright. 5. **Deterministic artifacts are the only trusted map** — the manifest and the graph are built in code; the model never authors or edits them, and never reverse-engineers the build system from a directory listing. 6. **Deterministic stages stay deterministic**: manifest, graph, gate, and ladder are D; the LLM column holds exactly the "what to inspect next" decision, expressed through component 01's primitives. No LLM call may be introduced into any D node. 7. **k=2 flatten is forbidden** (encoded, §5) — it is a rejected change, not a tuning knob. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*