Summary / 09. Patch and code generation / Document
Spec 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 09 — Patch / Code Generation (the ≈80% fraction) > Derived from `papers/09-patch-code-generation.md`. This document drives the build of the > **patch/code-generation stage**: the narrow, named semantic-transformation step in which, for the > first time in the program, the pipeline's accumulated precision is cashed into a concrete artifact — > a bounded set of candidate code changes. It sits after component 08 (context construction, which > decided, for the single next decision, exactly what of everything the pipeline knows the model sees) > and before the components that execute, validate, and repair the generated candidates (10–11 and > 17–18). It is the **FIRST stage whose output IS the artifact**: every prior component — interaction, > action selection, repository modeling, retrieval, localization, impact analysis, planning, and > context construction — existed to bound this one, and every later component exists to check it. It > translates the paper's design into exact typed intent-package schemas, exact deterministic > edit-representation classification, exact schema-constrained candidate-generation caps, exact > mechanical-application and rollback rules, exact syntax/lint-gate mechanics, exact normalization and > deduplication identity keys, exact L-column caps, exact A/B instruments and thresholds, exact > ablation mechanics, and an exact build sequence — no further research required. Where the paper says > "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible patch/code-generation stage** — a deterministic-first, typed-input/typed-output generation engine (intent-package assembly → edit-representation selection → mechanical edit application → syntax/lint gate → normalization + deduplication) wrapped around a **single bounded weak-LLM candidate-generation call**, as a drop-in *artifact-producing* stage that converts the change obligation and validated context of components 05–08 — the target location and fault (05), the impact obligations (06), the change obligation + acceptance criteria + required validation (07), and the decision window (08) — into the **bounded, normalized, syntactically valid candidate set** handed to the downstream execution and validation components (10–11, 17–18). It is distinct from its predecessors in *kind*, not merely in order: every component before it narrowed *what the model must decide*; generation is where the model is finally asked to *emit the code change* those components exist to scope, and everything that turns that emission into safe repository bytes is deterministic software on CPU. The overall target is the paper's full 100% spec. **Round 1 lands the expected ≈80% of it**: everything deterministic (intent-package assembly, edit-representation selection, mechanical edit application, the syntax/lint gate, normalization + deduplication) plus the *one* bounded candidate-generation call — targeting the reachable fraction of the design where the L column holds exactly one entry and every upstream and downstream step is deterministic software on CPU. The heavy learned / trained-artifact / extra-model-call / GPU pieces (a trained verifier for candidate ranking, a trained generator via RL, a GRPO-trained small editor model for adaptive edit-representation choice, and a learned edit-representation chooser) are **not** built in Round 1 and are **not excluded** — they are the content of later rounds that climb from that ~80% landing toward 100% (see §2). Round 1 is kept only if it wins a measured A/B test on the *honest* generation metrics — led by **resolve rate at a fixed candidate budget / tokens-per-resolved** and **edit-success rate** — *and* does not regress end-to-end resolve/cost/tokens (§7). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element from the paper is classified below by *which round* builds it. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–7). - **DEFERRED = later rounds** — the trained-artifact / extra-model-call / GPU pieces (rows 8–11) that Round 1 intentionally does **not** build *yet*, because each adds a trained artifact, an extra model call against the L-column invariant, or a GPU/training dependency, and each belongs to the *ranking or representation-selection* side — which this component explicitly hands to the downstream validation components (10–11, 17–18) or defers to a later round (paper §2.7, §4). They are **not excluded from the project** — they are the content of later rounds, the climb toward 100%, gated by the same §7 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | (a) Intent-package assembly — gather the change obligation, target code, requirement, dependency context, constraints, and validation obligations (from 05/06/07/08) into a typed intent package | **BUILD-NOW** | | 2 | (b) Edit-representation selection — deterministic classification of the change's scope → cheapest reliable format; search/replace default, whole-file rewrite the gated exception (P4) | **BUILD-NOW** | | 3 | (c) Candidate generation — **the single learned call**, see §4 | **BUILD-NOW** (the L step; one call, N bounded candidates) | | 4 | (d) Mechanical edit application — deterministic transformer: locate search block, verify unique match, replace, reparse (P3) | **BUILD-NOW** | | 5 | (e) Syntax / lint gate — reject candidates that do not parse; roll back to pre-application state (P7) | **BUILD-NOW** | | 6 | (f) Normalization + dedup — normalize whitespace/comments for comparison; collapse near-identical candidates so the valid set contains distinct candidates | **BUILD-NOW** | | 7 | A/B harness + the single bounded candidate-generation call (§4) on the assembled intent package | **BUILD-NOW** (measured by §7, never a default that runs alone) | | 8 | A trained **verifier for candidate ranking** (SWE-Gym — scores agent *trajectories*, selection reported as Best@k, 32.0% Verified / 26.0% Lite, 32B continues to improve at k=8) | **DEFERRED** — adds a trained artifact + an extra model call; ranking is the explicit job of the downstream validation components (10–11, 17–18), not of generation; gated by §7 on held-out instances once Round 1 is in place | | 9 | A trained **generator via RL** (SWE-RL — reward = difflib.SequenceMatcher similarity between predicted and oracle merged-PR patch, no execution feedback; Llama3-SWE-RL-70B at 41.0% on SWE-bench Verified) | **DEFERRED** — a trained generator needing RL training on open-repository software-evolution records + (for 70B-class) real compute; the Round 1 generator is the harness's own weak model behind the deterministic stages, which carry the load precisely because the learned call underperforms | | 10 | A **GRPO-trained small editor model** for adaptive find-replace/whole-file selection (SWE-Edit — GRPO on Qwen3-8B with an adaptive find-replace/whole-file-rewrite policy, +12.5pp edit success, 8B editor to GPT-5-nano parity) | **DEFERRED** — a trained-weights artifact replacing a deterministic stage that Round 1 already implements (b); needs GRPO training; not a drop-in; gated by §7 ablation once it exists | | 11 | A **learned (vs deterministic) edit-representation chooser** — a model that decides search/replace vs whole-file per candidate | **DEFERRED** — a second model call in the pipeline against the L-column invariant; the deterministic scope classification (§3.2) already captures the search/replace-vs-whole-file decision at zero model cost, mirroring the adaptive policy SWE-Edit trains | **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 does not put a model anywhere in the generation path by default. It builds the full deterministic pipeline (rows 1–2, 4–6) and the **single** candidate-generation call (§4) that is the one genuinely learned decision in the stage. The deterministic path is the default; the L step is the *only* learned entry. The trained verifier, the RL-trained generator, the GRPO-trained editor, and the learned representation chooser are all candidates for later rounds precisely because each would add a trained artifact or a *second* model to the pipeline, break the one-entry L column, or replace a deterministic stage that the ablation (§7) may already prove sufficient. Any later-round candidate must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out instances, at a defensible cost, or be rejected. The cost of the deterministic choice is paid once, in the schemas (§3); the cost of a misplaced learned call is paid every generation, in L-column violations and budget spent on a mechanism the deterministic stages may already beat. ## 3. The Deterministic Pipeline (D) Six deterministic stages through the emission of the valid candidate set, plus a seventh row that is the **single learned node**. The pipeline labels every node D (deterministic software) or L (a learned model); the invariant that governs the whole is: *the L column contains exactly one entry — a single bounded candidate-generation call — and every upstream and downstream step is deterministic software on CPU* (paper §4). The flow, from intent to candidate set: ``` CHANGE OBLIGATION + VALIDATED CONTEXT (05 target + fault, 06 impact obligations, 07 change obligation + acceptance criteria + required validation, 08 decision window) → (a) intent-package assembly [D] → typed intent package {target, requirement, dependencies, constraints, validation} → (b) edit-representation selection [D] → cheapest reliable format for the scope → (§4) candidate generation [L — the ONE entry, the sole learned call] → (d) mechanical edit application [D] → applied candidate (or reported failure) → (e) syntax / lint gate [D] → applied candidates that parse (rollback on failure) → (f) normalization + deduplication [D] → the valid candidate set (distinct, normalized) ``` The candidate set is rebuilt from authoritative evidence for **every** generation — the intent package is assembled fresh from the current change obligation, never accumulated across generations (§8 invariant 4). All artifacts land in `build/09-patch-code-generation/`. Nothing writes into the live harness. ### 3.1 (a) Intent-package assembly (D) **Input:** the change obligation and its validated context, already assembled by the earlier components — the target location and fault (05), the impact obligations (06), the change obligation + acceptance criteria + required validation (07), and the decision window (08). **Output:** one typed **intent package** with structured `target`, `requirement`, `dependencies`, `constraints`, and `validation` fields. **No language model runs here.** Assembly is deterministic collection and normalization, **not** composition — nothing invents new intent; this stage only gathers and normalizes the intent the earlier components already determined. This is P5 made concrete (paper §3 P5, §2.6): the generator receives programmer intent plus dependency context plus constraints, not a bare buggy snippet. **Intent-package schema (exact):** ```json { "target": { "entity": "str", // the resolved symbol/entity name (05) "file": "str", // absolute path "span": [int, int] }, // [start_line, end_line] of the localized change site (05) "requirement": "str", // the stated requirement the change must satisfy (07) "dependencies": [ { "name": "str", // a dependency of the current change "signature": "str" } ], // its signature (name, params, return) — never a body "constraints": [ "str" ], // impact obligations the change must preserve (06) "validation": [ "str" ] } // required test / validation-target ids the change must satisfy (07) ``` **Field caps (exact):** `requirement` ≤ **1,024** chars (verbatim from 07); `dependencies` ≤ **16** entries, each `signature` ≤ **512** chars; `constraints` ≤ **16** strings, each ≤ **512** chars; `validation` ≤ **16** test ids, each ≤ **256** chars. `span` is the exact `[start_line, end_line]` from component 05's localization, never re-derived here. **Procedure (exact, deterministic):** 1. **Collect:** pull the change target + span from 05, the impact obligations as `constraints` from 06, the change obligation + requirement + acceptance criteria + required validation as `requirement`/`validation` from 07, and the dependency context from 06/08. 2. **Normalize:** map each piece onto the schema above — resolve the target entity and span, tag the requirement, truncate each dependency/constraint/validation entry to its field cap, and record provenance (where each field came from, so the audit can trace it). 3. **No composition:** a field is either present because an earlier component produced it or absent; nothing is invented here, and no model is consulted to fill a gap. 4. **Bound:** an entry that exceeds its field cap is truncated to the cap and flagged for the §7 audit — never dropped silently, never sent to the model in an over-cap field. **Caps:** the assembled intent package is bounded by the field caps above — ≤ ~4 KB in total by construction. The package is rebuilt per generation; it is never accumulated across generations (§8 invariant 4). ### 3.2 (b) Edit-representation selection (D) **Input:** the assembled intent package (a). **Output:** the **edit representation** — the cheapest reliable format in which candidates over the target span will be expressed. **No language model runs here.** This is P4 executed structurally (paper §3 P4, §2.2): the edit format is a measured variable, not a fixed convention, and the default is **search/replace over a scoped span** — the cheapest representation that applies and reparses reliably (paper §2.1 Agentless, §2.8 Diff-XYZ, §2.9 "Can It Edit?") — with **whole-file rewrite admitted only when the localized span cannot express the change**. **Scope-classification rule (exact — deterministic, not a model choice):** The change is classified **localized-edit** iff its target span (a) is a contiguous region whose extent can be bounded by the search blocks the edit format requires — concretely, iff the edit touches **≤ 40 contiguous lines** **and** the surrounding file is stable (the change does not span a module boundary). Otherwise the change is classified **structural-rewrite** (a larger region or file). The classification reads the span and the change obligation; it never asks a model which format "feels" right, mirroring the adaptive find-replace/whole-file decision SWE-Edit trains into a small model (paper §2.2) — here made deterministic. **Format-selection rule (exact):** 1. **Default = search/replace over the scoped span:** the candidate is expressed as an ordered list of `{search, replace}` block pairs, where each `search` block is exact source text present in the target region and each `replace` block is its corrected counterpart. This is the default for every `localized-edit` classification (paper §2.1 — Agentless's search/replace-over-localized-snippets format). 2. **Whole-file rewrite is a gated exception:** admitted **only** for a `structural-rewrite` classification — a change the localized span cannot express. A whole-file candidate must carry the *complete* replacement file; it is never the default and never chosen for a localized edit. **Caps:** one representation per generation, fixed by the classification before any candidate is generated. The selection is logged (classification + chosen format) for the §7 format-robustness probe and ablation instruments. ### 3.3 (c) Candidate generation (L — the ONE entry) The single learned node; the full spec is §4. **Input:** the intent package (a) and the selected edit format (b). **Output:** N bounded repair candidates in that format. It emits N bounded candidates from the intent package (a), at `temperature > 0` for diversity, in **ONE logical call that produces N candidates** — reached **once** per generation and never re-prompted on parse failure (§8 invariant 3). This is P1 and P2 together (paper §3 P1/P2, §4): the call is narrow (emit well-formed edits over known spans from a typed intent package), and it is sampled to produce a bounded set rather than one greedy draw. It is the deliberate **only** learned step because every other operation in this stage is something deterministic software does more reliably and more cheaply than a weak model. **Inline caps (the governing numbers; §4 is authoritative):** ONE call; `temperature = 0.8`; `max_completion_tokens = 4096`; N ≤ **10** candidates, default **4** (configurable under budget); schema-constrained output (`{search, replace}` block pairs for search/replace, full-file replacement for whole-file). Parse failure → **no-op fallback** (single greedy candidate or empty set + a `regenerate` flag) — never a re-prompt loop, never a second unconstrained call. **Ranking is not this stage's job:** generation produces the valid candidate set and stops; ranking among the valid candidates, and any repair of them, is the explicit job of the downstream validation and repair components (10–11, 17–18), not of this stage (paper §4). Any addition that would put a second model call here — a learned edit-representation chooser, a learned patch ranker inside generation, a model-in-the-loop repair of malformed output — is a regression against P1–P7 and the L-column invariant, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of §7, on held-out instances, at a defensible cost, or be rejected. --- ### 3.4 (d) Mechanical edit application (D) **Input:** each candidate from (c) in the selected edit format. **Output:** each candidate applied to the repository bytes exactly as the edit format specifies — **locate the search block, verify it matches uniquely, replace it, reparse** — or a **reported failure**. **No language model runs here.** This is P3 made concrete (paper §3 P3, §2.3): the harness, not the model, performs the byte-level transformation, so a correct semantic decision is not lost to a formatting slip, and the model never emits repository bytes directly as free-form text. Application is deterministic and auditable: the **same candidate always produces the same repository change or a reported failure**. **Procedure (exact, deterministic):** 1. **Locate:** for each `{search, replace}` pair, find the `search` block in the target file's target span. 2. **Verify unique match:** a `search` block that matches **zero** times, or matches **more than once**, is a **reported failure** — the candidate is rejected, never applied ambiguously. A non-unique match triggers a clean rejection, never silent corruption (paper §2.9 SWE-agent's edit-lint-and-revert guard). 3. **Replace:** apply the `replace` block at the uniquely matched location. 4. **Reparse:** reparse the modified region/file; reparse failure → **reported failure** (handed to the syntax/lint gate (e), which rolls back). **Caps:** application is byte-exact and idempotent per candidate; each candidate is applied against the **same pre-generation snapshot** of the repository (never against a prior candidate's applied state, so candidates remain independent for validation). A candidate that cannot be located or uniquely matched is rejected with a structured failure reason (logged for the §7 format-robustness probe). ### 3.5 (e) Syntax / lint gate (D) **Input:** each applied candidate from (d). **Output:** the **applied candidates that parse**; a candidate that does not parse is **rejected and the repository is rolled back to its pre-application state**. **No language model runs here.** This is P7 executed structurally (paper §3 P7, §2.9): the gate runs deterministic syntax and lint checks on each applied candidate **before** it ever consumes downstream validation resources, so the valid set handed onward is syntactically sound **by construction** — a format failure is impossible downstream, which keeps the semantic/mechanical distinction clean (paper §4). **Procedure (exact):** for each applied candidate, run a deterministic syntax check (parse the modified file with the language's parser — stdlib `ast` for Python, component 03's tree-sitter for other languages) **and** a lint check (component 03's existing lint surface; Round 1 uses the deterministic checks only, no learned linter). A candidate that **fails to parse** is rejected and the repository is rolled back to its pre-application snapshot. A candidate that parses but trips a hard-lint gate (an error-level finding that changes the program's meaning — unused-import, undefined name) is rejected likewise; warnings do not reject. Rejection is always **apply-then-parse with rollback on failure**, so a malformed edit never persists and never reaches validation (paper §2.9). **Caps:** the gate rejects on parse failure or hard-lint failure, deterministically; every rejection is logged with its reason (for the §7 edit-success and format-robustness instruments). The gate runs before any candidate is normalized (f) or handed to validation; the valid set it emits is syntactically sound by construction. ### 3.6 (f) Normalization + deduplication (D) **Input:** the applied candidates that passed the syntax/lint gate (e). **Output:** a **normalized, deduplicated, syntactically valid candidate set** containing distinct candidates. **No language model runs here.** This is the paper's normalization made concrete (paper §2.1 — Agentless normalizes before its final majority-vote selection; §4 stage 6): survivors are normalized — whitespace and comments are ignored for comparison — and near-identical candidates are collapsed, so the valid set handed to the validation components (10–11, 17–18) contains distinct candidates rather than N copies of the same edit. **Normalization rule (exact):** for comparison, two candidates are **equivalent** iff their normalized forms are identical, where normalization strips leading/trailing whitespace per line, collapses internal runs of whitespace to a single space, and removes comment lines — exactly the surface differences Agentless ignores (extra spaces, newlines, comments; paper §2.1). **Deduplication rule (exact):** within the applied set, candidates that normalize to the same form are **collapsed to a single representative** (the first in the applied order). Candidates that normalize to distinct forms are kept as distinct. The deduplicated set is the component's **only output** — the valid candidate set — and is what downstream validation sees. **Caps:** ≤ **1** representative per normalized form; dedup runs after the syntax gate (e) and before the set is emitted to validation, so validation never pays N× for one effective edit. The normalized, deduplicated, syntactically valid candidate set is the sole product of this stage; generation ends here — it does not run tests, does not decide further actions, and does not choose among its own outputs (paper §1, P1). ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **candidate generation**. The weak model emits N bounded repair candidates in the selected edit format, from the intent package, at a temperature greater than zero for diversity, in **ONE logical call that produces N candidates** (paper §2.1 — the model's output is a distribution to be aggregated, never one greedy shot; P2). This is the sole entry in the L column; every upstream and downstream step is deterministic software on CPU (paper §4). It fires on **every** generation whose intent package is non-empty (the deterministic path has already decided the format and bounded the span; this is the residual inference problem P1 leaves the model — "transform this known location so it satisfies this stated requirement" — deliberately narrow). **Firing condition (exact):** the step fires **iff** the intent package (a) has a non-empty `requirement` and a valid `target.span`. A generation with no requirement or no valid target span is a deterministic failure — no candidate call is made (the change obligation was not assembled by 07/05; the stage reports the missing obligation and stops). The step never fires to explore, run tests, or choose among its own outputs. **Prompt inputs (assembled deterministically, in this exact order):** 1. **`intent_package`** — the assembled package from (a), verbatim: `target` (entity, file, span), `requirement`, `dependencies` (signatures only), `constraints`, `validation` (test ids). 2. **`edit_format`** — the selected representation from (b): `search_replace` (and the instruction to emit `{search, replace}` block pairs scoped to the span) **or** `whole_file`. 3. **`n_candidates`** — the bounded sample count for this call. **Input → output schema (exact):** ```json { "candidates": [ { "search": "exact source text present in the target span", "replace": "its corrected counterpart" }, { "search": "…", "replace": "…" } ] } ``` For `whole_file`, each candidate is expressed as a complete-file replacement and the schema's per- candidate object is the full replacement text (the representation chosen in (b) fixes which shape the schema constrains). The output is schema-constrained — search/replace blocks with exactly the fields the deterministic applicator (d) expects — so the model cannot drift into an unappliable format. **Candidate-record schema (exact — the model must emit nothing outside it):** ```json { "search": str, // search/replace: exact source text to locate; whole-file: the full replacement "replace": str } // the corrected text to substitute (may be empty for a pure deletion) ``` **Caps:** **ONE call**, `temperature = 0.8` (a build constant, configurable via the build config), `max_completion_tokens = 4096`, N ≤ **10** candidates (default **4**, configurable under the §7 budget — the budget that governs N is set by cost and diversity, never by the hope that one more blind draw succeeds, P2). `temperature > 0` makes repeated draws from the same intent produce distinct candidates (the diversity the candidate mechanism exists to buy). The output is schema-constrained; the harness, not the model, decides what the candidates mean. **Parse-failure fallback (exact):** any JSON parse failure, schema violation, or malformed search/replace block is a **no-op**: the stage emits a **single greedy candidate** (a one-shot generation at `temperature = 0`) **or an empty set with a `regenerate` flag** — and hands control back to the deterministic path. **NEVER a re-prompt in a loop, NEVER a second unconstrained call.** The L column is reached **once** per generation; a failed parse is not retried with a "please try again" call. This is the mechanical enforcement of the one-entry L column and the anti-over-prompting rule (§8 invariant 3). **Re-invocation on the next generation (exact):** candidate generation is re-considered **only** when the *next* generation assembles its own intent package; each invocation is scoped strictly to that generation's package and re-uses none of a prior generation's candidates. It remains the **single conceptual learned entry** — a bounded, once-per-generation, per-package call — **never a model that runs over the whole run** and never a loop that re-derives what the deterministic stages already settled (paper §4). **Invariant 3 (§8): the L column holds exactly one entry.** ## 5. Implementation Order Eight phases, each independently testable and reversible. Each lands in `build/09-patch-code-generation/` as a standalone importable module. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Intent-package assembly** | `intent.py` (§3.1 collection + normalization + bounding; pulls target/span from 05, constraints from 06, requirement/validation from 07, dependencies from 06/08) | Feed fixtures of 05/06/07/08 outputs: every piece lands as a schema-valid `{target:{entity,file,span}, requirement, dependencies:[{name,signature}], constraints:[], validation:[]}` package; field caps enforced (requirement ≤1024 chars, ≤16 deps/signatures ≤512 chars, ≤16 constraints ≤512 chars, ≤16 validation ids); an over-cap entry truncates + flags; assert **no composition** (no invented field) and **no LLM consulted** (probe the LLM client to raise). | Delete `intent.py` (pure read). | | **2. Edit-representation selection** | `represent.py` (§3.2 scope classification + format selection; search/replace default, whole-file the gated exception) | Feed intent packages with varying spans: a ≤40-contiguous-line span is classified `localized-edit` and selects search/replace; a >40-line or module-spanning change is classified `structural-rewrite` and admits whole-file; **no** localized edit ever selects whole-file; selection is logged with its classification; assert **no LLM consulted**. | Delete `represent.py`. | | **3. Candidate generation (L)** | `generate.py` (§4 firing condition + ONE call + output schema + N cap + parse-failure fallback) | Mock the LLM client: assert the step fires **only** on a non-empty intent package (requirement + valid span); a missing requirement/span → deterministic failure with **no call**; the call is **one**, `temperature=0.8`, `max_completion_tokens=4096`, N ≤ 10 (default 4), prompt carries the intent package + edit format + n_candidates; output schema validated; a malformed / schema-violating response → **single greedy candidate or empty set + regenerate flag** with **no second call, no re-prompt loop**; well-formed candidates pass through to (d). | Delete `generate.py` (no live-harness change). | | **4. Mechanical edit application** | `apply.py` (§3.4 locate + unique-match verify + replace + reparse; same candidate → same change or a reported failure) | Feed candidates over a fixture file: a `search` block found uniquely applies; a search matching zero or >1 times → reported failure (never ambiguous apply); a candidate whose reparse fails → reported failure; application is against the pre-generation snapshot (candidates stay independent); assert determinism (same candidate → same result) and **no LLM consulted**. | Delete `apply.py`. | | **5. Syntax / lint gate** | `gate.py` (§3.5 deterministic syntax + lint checks; reject on parse failure / hard-lint; apply-then-parse rollback to pre-application state) | Feed applied candidates over a fixture: a candidate that does not parse is rejected and the file is rolled back to its pre-application snapshot; an error-level hard-lint (undefined name, unused import) rejects; a warning does not; every rejection is logged with its reason; the valid set emitted parses by construction; assert **no LLM consulted**. | Delete `gate.py`. | | **6. Normalization + deduplication** | `dedup.py` (§3.6 normalize whitespace/comments for comparison; collapse near-identical candidates) | Feed applied candidates: two candidates differing only in whitespace/newlines/comments collapse to one representative; two genuinely distinct candidates stay distinct; dedup runs after the gate and before emission to validation; the emitted set contains distinct candidates only. | Delete `dedup.py`. | | **7. Candidate-set end-to-end** | `pipeline.py` (§3.1→3.6 wired: intent → representation → generation → apply → gate → dedup → valid candidate set) | End-to-end on 3 warm-up edits over a fixture repo: `pipeline.py` emits a normalized, deduplicated, syntactically valid candidate set; a malformed candidate never reaches the output (clean rejection + rollback); generation is reached **once** per run (mock LLM client asserts one call); the L column holds one entry. | Delete `pipeline.py`. | | **8. A/B harness** | `ab.py` (feature-flag router + resolve-at-fixed-budget / tokens-per-resolved capture + edit-success capture + candidate-diversity capture + ablation runner + format-robustness probe + decision) | End-to-end on the §7 sample: `ab.py` runs generated-candidate-set-vs-single-greedy on the §7 sample, emits resolve at fixed budget, tokens-per-resolved, edit-success rate, candidate diversity, end-to-end resolve/cost/tokens, the ablation report, the format-robustness probe, and the §7 decision. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 8) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `json`, `re`, `difflib` | intent-package schemas (§3.1), edit-application search/replace matching (§3.4), candidate dedup identity (§3.6) | CPython stdlib. | **No** | | stdlib `ast` / component 03 tree-sitter | syntax gate parsing (§3.5), target-span grounding (§3.1) | Reuse component 03's parser (already built/flagged there); no new parser. | **No** | | Component 05 localization (`target.span`) | §3.1 the change target + span; §3.2 scope classification | In-repo, built by spec 05. | **No** | | Component 06 impact DAG + downstream obligations | §3.1 `constraints` (impact obligations to preserve) | In-repo, built by spec 06. | **No** | | Component 07 plan node + change obligation + acceptance criteria + required validation | §3.1 `requirement` + `validation` | In-repo, built by spec 07. | **No** | | Component 08 decision window | §3.1 dependency context + the window the change is generated from | In-repo, built by spec 08. | **No** | | Component 03 lint surface | §3.5 the deterministic lint checks | In-repo, already vetted. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single candidate-generation call | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: the generator reuses component 05's localization, component 06's impact obligations, component 07's change obligation, and component 08's decision window; the single LLM call (§4) goes to the harness's own weak model. **No new package is installed for Round 1, and nothing is flagged for a later install** — the four DEFERRED rows (§2 rows 8–11: the SWE-Gym-trained candidate verifier, the SWE-RL-trained generator, the SWE-Edit GRPO-trained editor, and the learned edit-representation chooser) are training/weights on known architectures, not new third-party installs; any future model download, RL-training run, or GPU dependency they imply must clear the 72h + star/scrutiny + sandbox-test gate before use, and is out of Round 1 scope entirely. ## 7. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as spec 01** (Python-only, single-repo, no-Docker reproduction). Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. **Baseline (concrete).** The **single greedy shot** — one candidate generated at `temperature = 0` from the bare target (no intent package, no candidate sampling, no edit-format selection), applied and validated directly. The bounded-candidate set must beat *this* on resolve at a fixed budget, and the ablation (7.4) removes generation mechanisms against the *honest* metrics below. ### 7.1 (a) Resolve rate at a fixed candidate budget / tokens-per-resolved — the KEY measure The key generation instrument, inherited from the paper's §6: for a **bounded generation budget** (N candidates, a fixed token ceiling), the candidate set must **STRICTLY BEAT a single greedy shot on resolve** — the set resolves strictly more than any one of its members alone would — **with NO cost regression**, measured as resolve per generation token consumed (tokens-per-resolved must not rise). This is the instrument that makes P2 falsifiable (paper §6): if sampling N candidates resolves the same as a single greedy draw at higher cost, the candidate mechanism is not earning its tokens, regardless of how elegant the diversity rationale reads on paper. **This is the KEY measure — an edit-success or diversity win that does not beat the single greedy shot on resolve at fixed cost is not credited.** Agentless's forty-patches-per-bug lift is the external reference for what this instrument should show. ### 7.2 (b) Edit-success rate — the honest generation-format score The **fraction of generated candidates that apply cleanly and parse** — the direct score of the mechanical-application (d) and syntax-gate (e) stages, and the metric SWE-Edit reports (93.4% rising to 96.9% with its decomposition). **The bounded-candidate set must STRICTLY IMPROVE edit-success rate** versus the single greedy shot. It isolates format and application reliability from semantic correctness: a low edit-success rate means the channel between the model's decision and the repository bytes is failing, which is a harness problem, not a reasoning problem. ### 7.3 (c) Candidate diversity Do the N candidates actually differ, or are they near-duplicates? A candidate set that collapses to one effective edit (post-dedup, §3.6) has paid N× the generation cost for no diversity. The instrument measures whether §3.6's normalization is merging genuine duplicates or destroying real alternatives, and whether `temperature > 0` sampling is producing distinct repairs of the same intent. A deduplicated set of size **≥ 2** is required for the candidate mechanism to earn its cost; a set that dedups to size 1 against N>1 generated is a failing diversity result. ### 7.4 Decision rule (explicit thresholds) Run candidate-vs-current on the *identical* sample with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The candidate **wins iff**: ``` Resolve-at-budget STRICTLY beats the single greedy shot (the KEY measure) AND edit-success rate STRICTLY improves AND candidate diversity: deduplicated set size ≥ 2 at N > 1 AND resolve_candidate ≥ resolve_current (no solve-rate regression) AND cost_candidate ≤ 1.5 × cost_current AND tokens_candidate ≤ 1.5 × tokens_current ``` Resolve-at-fixed-budget must **strictly beat the single greedy shot** and edit-success must **strictly improve**; resolve must not regress; cost and tokens are bounded at 1.5×. A candidate that improves resolve but leaves the set beating the single greedy shot by nothing at higher cost is **not adopted**. If the candidate wins, it replaces the current path (single greedy shot); if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Ablation rule (mechanical) The instrument that enforces the "never add a generation mechanism because it sounds sensible" discipline mechanically (paper §5, §6). Two passes, everything else held fixed, the delta measured on the honest metrics (resolve-at-budget / tokens-per-resolved and edit-success rate, plus resolve/cost/tokens): 1. **Remove each mechanism in turn** — candidate sampling (drop to one greedy shot), the intent package (feed a bare target instead), edit-format selection (force whole-file rewrites), the syntax/lint gate (forward all candidates to validation). 2. **Re-measure with the mechanism absent.** A mechanism is **DELETED** iff its removal does **not** degrade the honest metrics beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the metrics is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (candidate sampling), then to any other mechanism — regardless of how sensible it looks. **Never add a generation mechanism because it sounds sensible.** ### 7.6 Format-robustness probe (P4/P7 load-bearing check) The instrument that verifies the chosen edit representation and the rollback-on-failure gate are load-bearing, not decorative (paper §6, §2.9): 1. **Verify representation reliability:** search/replace candidates over localized spans must apply **more reliably** than whole-file rewrites of the same change — measure both on the sample. 2. **Verify malformed-format rejection:** deliberately bury a malformed format in a candidate — a missing `search` block, a non-unique match, an off-by-one context line — and confirm it causes a **clean rejection by the syntax gate with rollback**, never silent corruption of the repository. If a malformed candidate corrupts the repository or whole-file rewrites apply more reliably than search/replace, the chosen representation or the gate fails the probe and the mechanism is re-examined against 7.4. ### 7.7 Two honest caveats Two caveats close the section, inherited from the paper's §6. **First, the strong-model-transfer caveat:** the headline numbers of the paper's review — Agentless's 32.00% on a GPT-4-class base, SWE-Edit's +2.1pp with reasoning-model families, AlphaCodium's 19→44% for GPT-4, SWE-RL's 41.0% for a 70B model — were all obtained with strong models, and a weak model should not be expected to reproduce them. This architecture's deterministic stages carry the load precisely because the single learned call will underperform, and the edit-success and tokens-per-resolved instruments are the ones that will show whether the surrounding harness is doing so. **Second, the held-out and memorization caution** inherited from the earlier components: a model that has memorized a repository's shape — or, worse, a benchmark's reference patches — will score well in-benchmark without the intent package or the candidate mechanism having done constructive work, so all generation metrics must be reported on held-out and out-of-benchmark instances, exactly as the planning, localization, and context-construction metrics were. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/09-patch-code-generation/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `GEN=gen09|single`, default `single`) routes the A/B; the flag alone switches between the new generation stage and the current single-greedy-shot path — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. **Invariants (must hold, checked at the end of every phase).** 1. **No new `.md` files outside this folder** (`projects/self-improvement/{papers,specs,build}`). Everything else is code or DB rows. 2. **No change to the live harness** until the §7 A/B gate passes. The current tool surface stays byte-identical during the entire build. 3. **The L column holds exactly ONE entry** — candidate generation (§4), reached once per generation, never re-prompted in a loop on parse failure. Any phase that introduces a language model elsewhere in the pipeline (intent-package assembly, edit-representation selection, mechanical application, the syntax gate, normalization) is rejected outright. 4. **Generation is a constrained phase** — no tools, no exploration, no test execution, and no self-selection of its own outputs. The stage produces its bounded candidate set and stops; ranking among candidates and any repair of them is the explicit job of the downstream validation components (10–11, 17–18). The candidate set is rebuilt per generation, never accumulated. 5. **Semantic/mechanical separation** — the harness performs byte-level application (locate, verify unique match, replace, reparse); the model never emits repository bytes directly as free-form text. The LLM decides what the corrected code ought to mean; the harness turns that decision into safe repository bytes. 6. **Cheapest-reliable representation is the default** — search/replace over a scoped span is the default edit format; whole-file rewrite is a gated exception admitted only when the change's scope (b) genuinely requires a structural rewrite. A mechanism or format that fails the §7.5 ablation is deleted, never kept because it sounds sensible. 7. **A deterministic syntax/lint gate rejects malformed candidates before they reach validation** — every candidate is apply-then-parse with rollback on failure, so a format failure is impossible downstream by construction; no malformed edit ever persists or silently corrupts the repository. 8. **Generation mechanisms are ablated, not accumulated** — a mechanism that does not move the honest metrics (resolve-at-budget / tokens-per-resolved, edit-success rate) is **removed**, never kept because it sounds sensible. Learned additions (verifier, RL-trained generator, GRPO editor, learned representation chooser) must clear the §7 bar on held-out instances at a defensible cost before they may enter the pipeline. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*