Summary / 04. Code and context retrieval / Document
Paper 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# The Needle and the Noise: Deterministic, Multi-Signal Code Retrieval for a Weak-Model Coding Harness *Component 04 of 27 — Coding-Harness Program. This document specifies the code/context-retrieval stage: the continuous, task-driven pulling of the minimal high-precision context slice during execution. It is the ongoing counterpart to component 03's once-per-task reconnaissance map, and it feeds the primitives of component 01 and the decisions of component 02. It is written to drive a reversible build; the executable specification follows in a companion document.* --- ## Abstract Code/context retrieval — the continuous, task-driven selection of the smallest high-precision slice of a repository during execution — is usually implemented as the model browsing the repo through a search tool. This paper argues it should instead be a deterministic, multi-signal pipeline that runs entirely in code: the language model never browses; it receives a curated working set. The harness assembles recall from three signals — lexical symbols, structural graph edges, and the API/symbol descriptions of what the code must call — then aggressively reranks and compresses: ripgrep lexical recall, identifier-weighted BM25/IDF reranking, structure-aware deduplication, statement-level control- and data-dependence re-ranking over the top-k only, goal-scoped line-preserving pruning, and a hard token cap. Two properties distinguish the design. Retrieval is selective, because more context is frequently noise and every retrieved token must earn admission. And retrieved content is untrusted data, never something to be blindly imitated; a vetting rule gates its use. For a weak model on CPU, the entire retriever is deterministic software, and the model is reserved for the single act that requires it: reasoning over the curated slice. ## 1. Introduction The coding-harness program decomposes software engineering into a sequence of questions, and the first three components answered the opening three: how to *interact* with the code (component 01, the action-calling interface), how to *choose* an action (component 02, the tool-selection policy), and what the repository *is* (component 03, reconnaissance). This paper answers the question that recurs at every subsequent step of a task: *of this enormous repository, what exactly does the model need to see right now?* Component 04, code/context retrieval, is the stage that produces the answer — continuously, once per decision point, not once per task. The distinction from reconnaissance must be drawn sharply, because the two are routinely conflated. Reconnaissance is orientation: a single pass at task start that builds a map — the manifest, the dependency graph, the layout — and establishes what the codebase is. Retrieval is execution-time pulling: at each step, the harness decides which handful of symbols, statements, and API descriptions the model needs in front of it to take the next action. Reconnaissance is amortized across the task; retrieval is spent at every step. A harness can have a perfect map and still fail if it pulls the wrong slice into context at the moment of the edit, or if it pulls a correct slice surrounded by so much noise that the model cannot find the needle. The scale mismatch is the first of four constraints that shape the design. Real repositories span roughly 10^6 to 10^8 tokens of source; a weak model's effective working window is on the order of 10^5 tokens. The ratio is between one and three orders of magnitude, and it cannot be closed by enlarging the window — the hardware will not permit it, and the evidence reviewed in Section 2 says it would not help even if it did. Selection quality is therefore the *ceiling* on output quality: no model, however capable, can produce a correct edit if the context holding the answer was never placed in front of it, and a weak model additionally cannot compensate for a bad selection the way a frontier model might. Retrieval is the stage that sets the ceiling. The second constraint is the *similarity gap*: incomplete code is a poor retrieval key. A task description is almost never textually similar to the code that resolves it — the fix lives in a function whose name shares no tokens with the issue, or in an API whose description matches the *intent* rather than the *surface*. A retriever that keys on lexical similarity between query and code therefore starts from a systematically wrong premise, and the harm compounds at every step of a multi-step task, where the query is itself an incomplete draft. The third constraint is that *more context is not better*. The literature is unusually consistent on this point: retrieved similar code is frequently noise, not help; additional cross-file context degrades performance as often as it improves it; and a small, relevant slice beats a large, mostly-relevant dump. A retriever that optimizes for recall alone — pull everything that might matter — hands the model a haystack and asks it to find the needle. The retriever's real job is precision under a recall floor, not recall under a precision floor. The fourth constraint is the operating regime: a weak model on CPU-only hardware, under an agentless-first bias. Every token of context is a real forward-pass cost, every retrieved file is memory pressure, and the model has little capacity to plan around a bad context or to ignore distractor text. Under these constraints the retrieval stage cannot be "the model searches as it goes," because that assigns the most context-sensitive work — choosing what to read — to the weakest component of the system. It must instead be a deterministic pipeline that selects and compresses before the model ever sees a token, then hands the model a small, already-curated working set. The thesis follows from these four constraints. **Code/context retrieval should be selective, multi-signal, structure-aware, and deterministic: maximize recall of the true change surface — lexical symbols, structural graph edges, and the APIs/symbols the code will need to call — then aggressively rerank and compress until only the smallest sufficient context reaches the model, with a goal-scoped, line-preserving pruning stage and a vetting stage that treats retrieved content as untrusted data.** For a weak model on CPU, the entire retriever is deterministic code — ripgrep, BM25/IDF, a structural graph, statement-level dependence analysis — and the language model never browses the repository at all. The paper proceeds as follows. Section 2 reviews related work across four groups — retrieval mechanism, retrieval policy, pruning and token efficiency, and security — each with its claims, numbers, and weak-model/CPU implication. Section 3 derives seven design principles. Section 4 specifies the architecture, labeling every node deterministic (D) or learned (L). Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into four groups that map onto the retrieval stage's questions: what should be retrieved (mechanism), when to retrieve at all (policy), how to fit what is retrieved into budget (pruning), and how to keep retrieved content from poisoning the model (security). ### 2.1 Retrieval mechanism **CoRet** [1] reframes repository retrieval from autocomplete to *change*: its retriever is trained to answer "what must be inspected to make this edit," not "what looks similar to this text." The representational move is structural — it retrieves function- and class-level *atoms*, augments them with a file-path prefix and with call-graph-neighbor text, and trains contrastively with in-instance negatives (up to 1024), which beat cross-instance negatives. The gains are large: at least 15 percentage points of recall improvement, and a recall@5 improvement of 52.9% over the best baseline. Two points carry the lesson for this paper. First, the *what-must-be-inspected* objective is the right one, and it is the objective this harness adopts: retrieval is for the change surface, not for similarity. Second, the winning signal — atoms, file paths, call-graph neighbors — is reproducible deterministically; no trained encoder is required to reconstruct it. CoRet demonstrates that a neural retriever's headline gain is driven by features a deterministic pipeline can compute directly, which is precisely the substitution this paper wants to make. **GraphCoder** [2] supplies the structural re-ranking discipline. It builds a statement-level *Code Context Graph* with three edge types — CFG control-flow, CDG control-dependence, and DDG data-dependence — retrieves by a coarse lexical filter, then re-ranks by structural decay-with-distance from the retrieval point. It reports +6.06 code exact-match and +6.23 identifier exact-match gains. The ablation is the decisive number: control-flow (CFG) dominates control-dependence and data-dependence, which are roughly equivalent to each other — removing CFG costs 19.82%. Two further findings matter disproportionately here. The gains are *largest on the weakest models tested* (1B and 3.7B parameters), which is the recurring pattern that structure substitutes for reasoning a weak model cannot perform. And the machinery is cheap: statement-level indexing cuts database entries by 79.5%, retrieval takes 1.08 seconds, and the whole construction is tree-sitter-based with no GPU. This is direct evidence that statement-level structural re-ranking is both the highest-value signal and affordable on CPU. **What to Retrieve / AllianceCoder** [3] closes the loop on *what* the retrieval should contain. Its starting observation is negative: retrieved *similar* code is often noise, costing up to 15% (a negative contribution) when it merely looks like the task but is not what the task must call. The load-bearing ingredient is not similar code but *APIs* — the symbols the code will actually invoke. AllianceCoder therefore retrieves "what the code will call" by matching a natural-language *API description* against a natural-language description of the task (not text against code), and it reports a +10–20% Pass@1 improvement, approaching the theoretical upper bound of API retrieval. The implication for this harness is stark: the primary retrieval signal should be symbol/API-description matching — what the change will call — and similar-code matching should be demoted to a weak secondary signal, because similar code is the noise that this paper's title warns against. ### 2.2 Retrieval policy Three works, already treated in component 03 and cited here for policy, establish *when* retrieval pays. **RepoCoder** [4] contributes the iterative retrieval–generation (IRG) loop, and its central finding is that the retriever is not the lever: a cheap sparse retriever performs about as well as a dense one, and the second iteration captures nearly all the gain. **RepoBench** [5] (Liu et al.) supplies the measurement discipline of separating retrieval (RepoBench-R) from completion (RepoBench-C) and pipeline (RepoBench-P), so that *finding* context is scored independently of *using* it — the template for Section 6. **Repoformer** [6] is the cautionary complement: only about 20% of retrievals actually help, the rest are wasted tokens or distractions, and selective retrieval recovers most of the capability at a fraction of the cost. Together they license two architectural commitments: retrieval must be gated and selective (not automatic), and it must be scored separately from the code that consumes it. ### 2.3 Pruning and token efficiency **SWE-Pruner** [7] (Shi et al.) is the closest prior art to the pruning stage of Section 4. It is a *middleware* that filters raw retrieved context before it reaches the agent, guided by a *Goal Hint* — a `context_focus_question` that scopes the pruning to the current objective. Its central methodological finding is about granularity: line-level selection preserves syntax, while token-level pruning breaks it — cutting mid-line destroys the very structure the model needs. On SWE-bench Verified it removes 23–38% of tokens while *improving* success by 1.2–1.4 percentage points, with only 40–50 ms of overhead. The transferable core for this paper is goal-scoped, line-preserving filtering, which is fully deterministic; the mechanism by which SWE-Pruner *decides* which lines to keep is a 0.6B learned skimmer, and that part is explicitly not deterministic. The harness adopts the shape and the granularity and re-implements the decision deterministically. **GrepRAG / "Better Call Grep"** [8] is the strongest single piece of evidence for the deterministic-first stance. It shows that *zero-index ripgrep retrieval beats graph- and vector-based baselines once it is post-processed* — the raw lexical hits, after identifier-weighted BM25/IDF re-ranking and structure-aware deduplication and merging, outperform the fancier index. The latency numbers are the point for a CPU harness: ripgrep answers in under 0.02 seconds per query, and 1.45 seconds on a 754k-LOC repository, against more than 50 seconds for the graph/vector baselines. The accuracy numbers are a 7.04–15.58% relative exact-match gain. And, as the paper notes, this is what the deployed coding scaffolds — Claude Code, Codex CLI, mini-SWE-agent — actually do in production. Zero-index ripgrep is not a fallback; it is the primary retriever, because it is fast, never stale, and its precision is recoverable through deterministic post-processing. ### 2.4 Security **EvoMal** [9] supplies the reason retrieval cannot be a fire-and-forget feed. Its attack is *self-poisoning*: an agent is induced to *imitate* a retrieved malicious skill into a fresh, agent-authored tool — a create-path (authoring a new tool from retrieved content), not a reuse-path (calling the malicious skill directly). The attack succeeds with an ASPR of 20.3%–41.8%, and a *bare payload* — no surrounding benign context — still yields 11.1%, which means the vulnerability is inherent to the act of authoring from retrieved content, not an artifact of the surrounding text. It persists after the malicious skill is removed (68% survival at round 5), and existing name-, signature-, and scanner-based defenses fail against it. The implication for this paper is direct and binding: retrieved content is untrusted *data*, never a *template*. The harness must vet retrieved content and must forbid the model from blindly imitating or executing it — the principle P7 below elevates this from a caution to an architectural rule. ## 3. Design Principles Seven principles follow from the review, each tied to the evidence above. **P1 — Deterministic-first.** The retriever is ripgrep, BM25/IDF, and a structural graph; there is no neural retriever, no embedding index, and no GPU. GrepRAG's zero-index ripgrep beating graph/vector baselines once post-processed [8], and its <0.02s-per-query latency, is the warrant: a deterministic retriever is simultaneously faster and, after post-processing, more precise than the learned alternatives. CoRet's finding that its winning features (atoms, path prefixes, call-graph neighbors) reproduce without a trained encoder [1] is the second warrant: the deterministic pipeline recovers the neural retriever's signal. **P2 — Multi-signal.** Recall is assembled from three independent signals — lexical symbols, structural graph edges, and API/symbol descriptions — because no single signal covers the change surface. Lexical symbols catch exact names and identifiers; the graph catches definitions and references reachable from the retrieval point (GraphCoder [2], CoRet [1]); API-description matching catches what the code will *call*, which AllianceCoder shows is the load-bearing ingredient [3]. The three are fused, not ranked in isolation. **P3 — Retrieve symbols, not similar code.** Similar-code matching is demoted to a weak secondary signal. AllianceCoder's finding that similar code can cost up to 15% [3] is the governing fact: similarity is the noise. The primary signals are the symbols the task names and the APIs/symbols the code will call, matched description-to-description rather than text-to-code [3]. **P4 — Structure-aware re-rank.** Over the top-k candidates only, the harness re-ranks by statement-level control- and data-dependence decay-with-distance from the retrieval point, with control-flow weighted highest. GraphCoder's ablation — CFG ≫ CDG ≈ DDG, and −19.82% if CFG is removed [2] — is the ranking policy, and its finding that gains are largest on the weakest models [2] is the reason the harness can afford to spend the analysis: structure substitutes for reasoning the model cannot do. **P5 — Goal-scoped, line-preserving pruning.** The final compression is scoped by a Goal Hint and operates at line granularity, never token granularity, so syntax is never broken. SWE-Pruner's finding that line-level selection preserves syntax while token-level pruning destroys it [7] is the granularity rule; its −23–38% tokens with +1.2–1.4pp success [7] is the proof that pruning, done at the right granularity and scoped to the goal, is a *gain* rather than a loss. **P6 — Token budget: every retrieved token earns admission.** There is a hard cap, and the pipeline enforces it by deduplication and by dropping mid-relevance filler. Repoformer's finding that only ~20% of retrievals help [6] and GraphCoder's that more context can actively hurt [2] jointly establish that the budget is not a compromise but the mechanism by which the retriever protects the model from its own recall. **P7 — Vet retrieved content.** Retrieved content is untrusted data, never a template to imitate or a script to execute. EvoMal's demonstration that authoring-from-retrieved-content is inherently vulnerable — a bare payload still yields 11.1%, and the poisoning persists after removal [9] — makes this non-negotiable: the model may reason *about* retrieved content, but it may never author a verbatim copy of it into a new tool, and anything to be executed must first pass a vetting gate that treats the content as hostile by default. ## 4. The Architecture The retrieval stage is a single deterministic pipeline. Every node is labeled deterministic (D) or learned (L); the design target is that *every node through the production of the minimal context set is D, and only the final consumption of that set is L*. The pipeline is: **TASK → (1) goal/query construction (D) → (2) multi-source recall (D) → (3) structure-aware dedup/merge (D) → (4) identifier-weighted BM25/IDF re-rank (D) → (5) statement-level CFG/CDG/DDG decay-with-distance re-rank on top-k only (D) → (6) goal-scoped line-level prune (D) → (7) hard token cap (D) → MINIMAL CONTEXT SET → LLM (L).** **(1) Goal/query construction (D).** The raw task is reduced to a retrieval key by extracting, in code: identifiers and symbols it names, error strings and failing-test names it carries, and file paths it references. This is the query; no model paraphrasing, no embedding. The extraction reuses the manifest and line-level graph of component 03 — retrieval navigates the map reconnaissance already built — and the Goal Hint of SWE-Pruner [7] is derived here as a short, deterministic `context_focus_question` from the task's stated objective. **(2) Multi-source recall (D).** Three recall sources fire in parallel, each cheap and deterministic. First, *ripgrep lexical recall* over identifiers, strings, and paths — the primary retriever, with zero index and therefore never stale, at GrepRAG's <0.02s-per-query latency [8]. Second, *symbol/API-description matching*: the task's natural-language description is matched against API/symbol descriptions, not against code text, per AllianceCoder [3]. Third, *call-graph-neighbor expansion*: from any symbol already named, expand one hop over the invoke/contain edges of component 03's graph, per CoRet's call-graph-neighbor augmentation [1]. Each source emits candidates with provenance; similar-code matching, if admitted at all, is the last and lowest-weight source (P3). **(3) Structure-aware dedup/merge (D).** Candidates from the three sources are merged by structural identity — the same function, class, or statement-line appearing in multiple sources is collapsed into one entry, and file regions are merged by the containment hierarchy rather than by textual overlap — per GrepRAG's structure-aware dedup/merge [8]. **(4) Identifier-weighted BM25/IDF re-rank (D).** The merged candidate set is re-ranked by an identifier-weighted BM25/IDF score, per GrepRAG [8]: identifiers and symbols carry disproportionate weight over natural-language prose, so that a candidate whose *symbols* match the task outranks one whose *comments* match it. **(5) Statement-level dependence re-rank on top-k only (D).** Only the top-k candidates (k = 20) proceed to the expensive stage. For each, the harness builds its statement-level Code Context Graph — CFG, CDG, DDG — and re-ranks by decay-with-distance from the retrieval point, with CFG weighted highest, per GraphCoder [2]. Confining the graph analysis to the top-k keeps GraphCoder's 1.08s retrieval [2] from becoming the bottleneck across a whole repository. **(6) Goal-scoped line-level prune (D).** Each surviving candidate is pruned to the lines that serve the Goal Hint, at line granularity, never mid-line, per SWE-Pruner [7]. Signatures are preserved even when bodies are pruned — component 03's skeleton-localization result carries forward: signatures are enough to orient; bodies are noise. **(7) Hard token cap (D).** The pruned set is assembled into the minimal context set and capped at ≤4K tokens. If the cap would be exceeded, the pipeline drops the lowest-weight candidates first (never mid-entry, never mid-line) until it fits. The cap is enforced in code, before the model runs, so the model can neither request nor receive more than the budget. **(Consumption, L).** The minimal context set — a few thousand tokens of symbols, signatures, statement neighborhoods, and API descriptions — is the only thing the model receives. The model reasons over it; it does not browse, and it is not handed a search tool for the repository. This is the point of the whole design: the LLM never browses the repo; it receives a curated working set. **The vetting rule.** Retrieval and vetting are one gate, not two. Retrieved content is marked as untrusted data, and the harness enforces that the model may reason *about* it but may never author a verbatim copy of it into a new tool or execute it directly (P7, EvoMal [9]). Anything the model proposes that reproduces retrieved content verbatim into an authored artifact is rejected by the tool layer before it becomes state — the deterministic enforcement of a rule the model cannot be trusted to self-enforce. ## 5. Recursive Self-Improvement Protocol This is component 04 of a 27-component program, and it follows the discipline of the prior components. The specification is written against the complete stage — the 100% target: all three recall sources, the full statement-level dependence re-rank, goal-scoped line pruning, and the vetting gate. The first implementation round is expected to land at roughly 80%, with the gap concentrated in the deepest pieces (the API-description index, which must be authored per ecosystem, and the full CFG/CDG/DDG re-rank). Later rounds climb toward the full target as the substrate improves. Every change is A/B-gated: a candidate revision of the signal weights, the top-k, the token cap, or the vetting rule is deployed behind a gate and accepted only if it improves the metrics of Section 6 on a fixed evaluation sample. The stage is reversible by construction — each node is a swappable module — so a regression is a rollback, not a rewrite. ## 6. Measurement & Evaluation Retrieval is graded on retrieval, not on the patch it feeds. Two instrument families do the grading, both in the RepoBench-R [5] tradition of scoring the finding of context separately from the use of it. **RepoBench-R-style retrieval scoring.** Given a repository, a task, and gold context derived from successful trajectories, the harness is scored on: recall@k against the gold region; mean reciprocal rank (MRR) of the gold region in the ranked candidate list; required-file recall and required-symbol recall (the fraction of gold files and symbols surfaced); the irrelevant-context ratio (the share of retrieved tokens outside the gold region — the direct measure of the "noise" this paper exists to suppress); and tokens retrieved (the budget actually spent). The irrelevant-context ratio and the token count are the two numbers that distinguish this stage from a naive recall-maximizing retriever, and they are the ones the A/B gate of Section 5 must move downward while holding recall flat. **The oracle-context diagnostic.** To separate retrieval failure from reasoning failure, each evaluated task is run twice on the harness's own weak model: once with the pipeline's retrieved context, and once with *oracle* context — the gold slice, hand-supplied. Four outcomes partition the space. If oracle+Piper succeeds and retrieved+Piper fails, the failure is *retrieval* — the pipeline failed to surface what the model could have used, and the fix belongs to the retriever. If oracle+Piper fails and retrieved+Piper fails, the failure is *reasoning* — the model could not use even perfect context, and no retriever improvement will help. If both succeed, the pipeline is not the bottleneck. If retrieved+Piper succeeds while oracle+Piper fails, the pipeline's curation is outperforming the gold slice — a signal that the gold slice itself is too noisy, and consistent with the more-context-is-noise finding [3][6]. This diagnostic keeps the retriever honest: it is only ever credited for what the weak model demonstrably could use. One honesty constraint closes the section. The absolute numbers above were measured on models far stronger than this harness's — SWE-Pruner on SWE-bench Verified's agents [7], GraphCoder on 1B–3.7B models [2], AllianceCoder on its own Pass@1 scale [3] — and they are directions, not guarantees. The retrieval battery must be run on the weak model itself, and the meta-hypothesis tested alongside it is the ordering claim: that *selective, multi-signal, structure-aware, deterministic retrieval survives transfer to a weak model even though the absolute numbers do not*. If the ordering holds — and GraphCoder's gains being largest on the weakest models [2] is the encouraging precedent — the design is sound and needs only threshold tuning. If it inverts, the modularity of Section 4 means the correction is a parameter change, not a redesign. ## 7. Conclusion Code/context retrieval is the stage that runs at every decision point, and its failure modes — the missing symbol, the pulled-but-wrong slice, the relevant file buried under a haystack of similar-but-irrelevant code — set the ceiling on everything that follows, because no model can use context it was never given, and a weak model cannot recover the context a bad retriever dropped. For a weak model on CPU-only hardware, the stage cannot be a search loop; it must be a deterministic pipeline that maximizes recall of the true change surface — lexical symbols, structural graph edges, and the APIs the code must call — and then compresses ruthlessly: ripgrep lexical recall, identifier-weighted BM25/IDF re-ranking, structure-aware deduplication, statement-level control- and data-dependence re-ranking over the top-k only, goal-scoped line-preserving pruning, and a hard token cap. The literature is consistent that this is right: the retriever is not the lever [4], only a fifth of retrievals help [6], similar code is noise while APIs are the load [3], structure re-ranking pays most on weak models [2], line-level pruning is a gain [7], and retrieved content is untrusted data to be vetted, never imitated [9]. What remains for the model is only the final act of reasoning over a small, curated, already-vetted slice — the needle, extracted from the noise, by code rather than by chance. --- ## References 1. *CoRet: Improved Retriever for Code Editing.* arXiv:2505.24715. ACL 2025. 2. *GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph.* arXiv:2406.07003. ASE 2024. 3. *What to Retrieve for Effective Retrieval-Augmented Code Generation?* (AllianceCoder). arXiv:2503.20589. ICSE 2026. 4. Zhang, F., et al. *RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation.* arXiv:2303.12570. EMNLP 2023. 5. Liu, T., et al. *RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems.* arXiv:2306.03091. ICLR 2024. 6. *Repoformer: Selective Retrieval for Repository-Level Code Completion.* arXiv:2403.10059. ICML 2024. 7. Yuhang Wang, et al. *SWE-Pruner: Pruning Retrieved Context for Software Engineering Agents.* arXiv:2601.16746. Jan 2026. **Preprint** — treat findings as provisional. 8. *Better Call Grep: Zero-Index Repository Retrieval via Ripgrep* (GrepRAG). arXiv:2601.23254. ISSTA 2026. **Preprint** — treat findings as provisional. 9. *EvoMal: Self-Poisoning in Self-Evolving Coding Agents.* arXiv:2608.25776. 2026. **Preprint** — treat findings as provisional.