Summary / 16. Candidate generation and competing hypotheses / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 16 — Candidate Generation / Competing Hypotheses (the hypothesis-diverse, evidence-constrained stage) > Derived from `papers/16-candidate-generation-competing-hypotheses.md` — "Never Marry Your First Sample: Hypothesis-Diverse, Evidence-Constrained Candidate Generation for a Weak-Model Coding Harness." > This document drives the build of the **candidate-generation stage**: the deterministic-first, hypothesis-diverse, > evidence-constrained pipeline in which a decomposed request first meets solution space and the harness commits to the > SET of explanations it will test. It sits at the front of the flow that components 01–15 assembled — it consumes the > decomposed request and repository context and produces the candidate set everything downstream consumes: execution > falsifies these hypotheses, verification judges them, component 14 repairs the patch of whichever survives, and > component 15 reflects on the failures the survivors leave behind. The sharp distinction is between a SAMPLE and a > HYPOTHESIS: a sample is a stochastic draw conditioned on the prompt; a hypothesis is a causally distinct claim about > the correct behavior and how to produce it, carrying a behavioral prediction reality can confirm or contradict. Naive > harnesses take the first plausible sample, or generate many and let brute force sort it out; this stage constructs > hypotheses DELIBERATELY — spread across genuinely different strategies and frames rather than temperature draws — lets > cheap referees and finally the expensive oracle eliminate the ones reality contradicts, records why each was > eliminated so it is never silently regenerated, branches the survivors where evidence warrants, and hoards > cross-candidate agreement as evidence for later selection and patch fusion. The output of component 16 is **not a > patch** — it is a candidate **HYPOTHESIS SET**: causally distinct, deduplicated, behaviorally clustered, each > hypothesis carrying a behavioral prediction and evidence record, the survivors carrying agreement evidence, the > eliminated carrying prune-reasons — plus the regeneration-constraint set compiled from those prune-reasons. Every > upstream and downstream step is deterministic software on CPU; the weak model's ONE contribution is a single learned > step of SEMANTIC JUDGMENT (propose hypotheses and repair strategies from distinct frames), and every other decision is > deterministic software that keeps duplicates from consuming the budget and keeps contradicted hypotheses from coming > back. The guiding rule, made mechanical: **Five answers are valuable only if they represent five different ideas. > Generate disagreement deliberately, then let reality collapse the search space.** This spec pins the paper's design to > exact primitives, exact `Hypothesis`/`FalsificationRecord`/`Candidate` schemas, exact per-stage caps, exact L-column > firing, exact A/B instruments and thresholds, exact ablation mechanics, and an exact build sequence. Where the paper > says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible candidate-generation stage** — a deterministic-first, evidence-constrained pipeline of frame decomposition and falsification intake, the single frame-conditioned proposal call, normalization and schema binding, diversity measurement and deduplication, behavioral clustering, cheap synthesized referee screening, candidate branching and budget enforcement, oracle verification and evidence recording, falsification pruning and constraint compilation, and agreement extraction — that turns a decomposed request plus repository context plus the falsification ledger from prior attempts into a causally distinct, deduplicated, behaviorally clustered **candidate hypothesis set**. It differs from its predecessors in *kind*, not just position: the repair loop (14) asks *"how do I improve this failed patch?"* and loops the **CANDIDATE**; reflection (15) asks *"what did this failure teach?"* and loops the agent's **UNDERSTANDING**; candidate generation asks *"what different ideas might be correct, and what would reality see if each were?"* and constructs the **SET** those stages consume — without which the whole downstream chain would judge and repair an understanding the harness never deliberately built. The paper's seven principles decide every design choice. **P1 Hypotheses and patches are separate objects** — the output is a typed `Hypothesis`, distinct from the `Patch` that later materializes if it survives, so the harness can falsify an IDEA before spending effort building the patch. **P2 Diversify by strategy and frame, not temperature** — diversity is manufactured by generating from DISTINCT frames, never by cranking temperature on one prompt; five frames with one candidate each outvalue one frame with five draws. **P3 A contradicted hypothesis becomes a constraint on regeneration** — the contradiction is fed forward as an exclusion-list entry, so the next proposal cannot re-propose the falsified idea. **P4 Cluster behaviorally before verifying** — hypotheses are clustered by the behavior they PREDICT so one test or trace can falsify a whole cluster. **P5 Run a cheap synthesized referee before the expensive oracle** — deterministic probes screen every cluster at pocket change so the oracle is spent only on candidates the referee could not kill. **P6 Cross-candidate agreement is deterministic edit-atom evidence** — preserved for later selection or fusion, never mistaken for correctness. **P7 Record prune-reasons so eliminated hypotheses are never silently regenerated** — every elimination writes WHY to a falsification ledger that seeds every future proposal call. **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — frame decomposition and falsification intake, schema binding, diversity and dedup, basic behavioral clustering, cheap referee, branching and fixed budget enforcement, oracle verification and evidence recording, falsification pruning and constraint compilation, and a minimal agreement record — plus the *bound* of the *one* frame-conditioned proposal call. The heavy pieces (full clustering at scale, adaptive per-difficulty budget, edit-atom fusion, a trained generator) are **not** built in Round 1 and are **not excluded** — later rounds climb toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by **oracle-confirmed candidate diversity** and its companion **budget efficiency**, never the end-to-end resolve rate alone (§7). The stage lives in `build/16-candidate-generation-competing-hypotheses/` and writes nothing into the live harness. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–10); **DEFERRED = later rounds** (rows 11–16), **not excluded** but gated by the same §7 A/B protocol. Cross-session persistence of the falsification ledger across repo-sessions is a component-17 concern; even later rounds of this component do **not** build it. | # | Component (paper §4) | Classification | |---|---|---| | 1 | (a) Frame decomposition + budget allocation + falsification intake — split the request into DISTINCT frames (contract-based, minimal-edit, reimplementation, root-cause…); allocate so a minimum number of distinct frames is represented before any frame gets a second candidate; parse the ledger into the exclusion list seeding the generator | **BUILD-NOW** | | 2 | (L) Frame-conditioned hypothesis-and-strategy proposal — seeded with decomposed request + repo context + distinct frames + the exclusion list, the weak model proposes hypotheses and repair strategies FROM EACH DISTINCT FRAME, each stating a claim and a behavioral prediction | **BUILD-NOW** (bounded stub only) | | 3 | (b) Normalization + schema binding — bind each proposal into the typed `Hypothesis` `{ frame, claim, repair_strategy, behavioral_prediction, status }`, kept separate from any `Patch`; reject a proposal with no parseable behavioral prediction as unfalsifiable (P1) | **BUILD-NOW** | | 4 | (c) Diversity measurement + deduplication — SEMANTIC and STRUCTURAL diversity; near-duplicates merged or dropped BEFORE any verification spend (P2) | **BUILD-NOW** | | 5 | (e) Cheap synthesized referee screening — deterministic probes synthesized from each cluster's behavioral predictions; a falsified cluster routes to pruning with its prune-reason, never to the oracle | **BUILD-NOW** | | 6 | (g) Oracle verification + evidence recording — each surviving candidate runs the expensive oracle (real suite, traces, invariants, verifier); outcomes recorded as evidence | **BUILD-NOW** | | 7 | (h) Falsification pruning + constraint compilation — eliminate contradicted hypotheses; write prune-reasons to the ledger; compile it into the exclusion set (P3/P7) | **BUILD-NOW** | | 8 | (f) Candidate branching + budget enforcement — survivors branched into MULTIPLE repair candidates where evidence warrants, under the FIXED budget, minimum-distinct-frame guarantee re-checked | **BUILD-NOW** (fixed budget only) | | 9 | Durable `Hypothesis` + `FalsificationRecord` stores, ledger join | **BUILD-NOW** | | 10 | Loop driver + A/B harness (§7 metrics + ablation runner + failure-mode audits) | **BUILD-NOW** | | 11 | **Full behavioral clustering (d) at scale** | **DEFERRED** — Round 1 ships basic cluster-by-predicted-observable only | | 12 | **Adaptive per-difficulty budget allocation** | **DEFERRED** — Round 1 ships a FIXED budget; the Adaptive-Consistency-style variable budget is deferred | | 13 | **Edit-atom agreement extraction + deterministic patch fusion (i)** | **DEFERRED** — Round 1 records minimal agreement evidence only | | 14 | **Trained candidate generator at scale** | **DEFERRED** — Round 1 ships only the bounded stub (§4) | | 15 | **Learned selection/judgment/budget** — a model choosing its best frame, judging its candidates, setting its own budget, deciding its own diversity | **DEFERRED**, and rejected unless it beats the deterministic stage on honest metrics | | 16 | **Cross-session falsification-ledger persistence / consolidation** | **DEFERRED to component 17** — this stage writes its ledger locally (§3.8) | **The cost rule (explicit).** The single learned entry — frame-conditioned proposal — is the one L-column entry this architecture permits. In Round 1 it is a **bounded stub**: wired in at its firing point (§4), seeded, schema-constrained, never auto-promoted; its full training and calibration is **deferred**. Its cost is bounded before invocation: one call, temperature zero, schema-constrained, bounded payload, parse-failure fallback that returns to the deterministic stage with the violation fed back (§4 caps). It is kept **only if** the §7 no-L-vs-L ablation shows it improves the KEY measure on held-out tasks; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the selection, judgment, budgeting, diversity, or admission path by default — the full deterministic pipeline (rows 1–10) plus the *bounded stub* of the single proposal call (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a learned selection/scope/budget judgment, a full training run, or break the one-entry L column — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out tasks, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Ten named stages — (a) through (i) plus the single learned entry (L) — all deterministic except L. The LLM column holds exactly one thing: frame-conditioned proposal (§4), reached once between frame decomposition and schema binding: ``` DECOMPOSED REQUEST + REPO CONTEXT + FALSIFICATION LEDGER (prior attempts on this & similar tasks) → (a) frame decomposition + budget allocation + falsification intake [D] → distinct frames + budget + exclusion list → (§4) frame-conditioned hypothesis-and-strategy proposal [L — the ONE entry; emits proposals from each frame] → (b) normalization + schema binding [D] → typed Hypothesis; unfalsifiable rejected → (c) diversity measurement + deduplication [D] → merge/drop near-duplicates before any verification spend → (d) behavioral clustering [D] → group by predicted observable (basic in Round 1) → (e) cheap synthesized referee screening [D] → falsified clusters → (h); survivors cleared → (f) candidate branching + budget enforcement [D] → survivors branched under the budget → (g) oracle verification + evidence recording [D] → survivors judged by reality; evidence recorded → (h) falsification pruning + constraint compilation [D] → contradicted eliminated; prune-reasons → ledger → exclusion set → (i) agreement extraction + edit-atom assembly [D] → shared/disputed atoms preserved (minimal in Round 1) ``` The pipeline reasons over **the request made into a deliberate candidate set** — the decomposed request, the repo context, and the ledger of what already failed are the only things the model proposes against, and every proposal it emits is bound, deduplicated, clustered, screened, branched, verified, pruned, and agreed deterministically. All artifacts land in `build/16-candidate-generation-competing-hypotheses/`. ### 3.1 (a) Frame decomposition, budget allocation, and falsification intake (D) **Input:** the decomposed request, the repository context, and the falsification ledger from prior attempts. **Output:** a set of DISTINCT frames, a FIXED budget allocation, and the EXCLUSION LIST parsed from the ledger. **No language model runs here.** The request is split deterministically into distinct frames — distinct strategies and causal framings the set should span: contract-based repair, minimal-edit, reimplementation, root-cause. The budget is allocated so a minimum number of distinct frames is represented before any gets a second candidate (P2), and the ledger (P7) is parsed into the exclusion list of what must NOT be proposed. **The model decides none of it.** Round 1 ships the FIXED budget; the adaptive allocation (row 12) is deferred. **Procedure (exact, deterministic):** 1. **Split the request into frames** — a deterministic split into the strategy/causal axes the set should span. 2. **Allocate the budget** — FIXED in Round 1: at least one candidate per distinct frame before any gets a second; re-checked after branching (§3.6). 3. **Intake the falsification ledger** — parse prior `FalsificationRecord`s into the exclusion list handed to the call. **Output schema (exact):** ```json { "frames": [ { "frame_id": "…", "label": "contract_based | minimal_edit | reimplementation | root_cause | …", "rationale": "…" } ], "budget": { "total_candidates": 0, "min_frames": 0, "per_frame_cap": 0 }, "exclusion_list": [ { "excluded_claim": "…", "prune_reason": "…" } ] } ``` **Caps:** frame decomposition, budget, and ledger intake are pure deterministic software on CPU — the model never chooses the frames, sets its own budget, or decides what it may forget. A decomposition yielding five near-identical hypotheses no matter how many frames it declared lied about its diversity (measured in §7.3). ### 3.2 (b) Normalization and schema binding (D) **Input:** the raw proposals from §4. **Output:** typed `Hypothesis` objects; unfalsifiable proposals rejected. **No language model runs here.** Each proposal is bound deterministically into the typed `Hypothesis`, kept structurally separate from any later `Patch` (P1). A proposal with no parseable `behavioral_prediction` — a claim about what observable would change if the hypothesis were true — is rejected as **unfalsifiable** (Platt's requirement that a hypothesis admit a discriminating experiment), before it costs an oracle call. **The `Hypothesis` schema (exact — EXACTLY these fields):** ```json { "frame": "…", // the frame_id it was proposed from "claim": "…", // the causal claim about the correct behavior "repair_strategy": "…", // the strategy that follows from the claim "behavioral_prediction": "…", // the observable reality can confirm or contradict "status": "proposed | bound | deduped | clustered | screened | branched | verified | survived | falsified | excluded" } ``` `frame` names the frame it was proposed from; `claim` states the causal claim; `repair_strategy` the strategy that follows; `behavioral_prediction` the observable that would change if it were true (the field whose presence decides falsifiability); `status` is the lifecycle tracker advanced deterministically by each downstream stage. A `Hypothesis` is never conflated with the `Patch` that may later instantiate it. **Output schema (exact):** ```json { "hypotheses": [ "<Hypothesis>" ], "rejected_as_unfalsifiable": [ "<proposal lacking a parseable behavioral_prediction>" ] } ``` **Caps:** schema binding is pure deterministic software on CPU — no model binds or rejects its own proposal; an unfalsifiable proposal is dropped here with a rejection recorded, before any verification spend. ### 3.3 (c) Diversity measurement and deduplication (D) **Input:** the typed, bound `Hypothesis` set. **Output:** the set with near-duplicates merged or dropped, plus a measured diversity report. **No language model runs here.** The set is measured for SEMANTIC diversity (are the claims different ideas?) and STRUCTURAL diversity (do the repairs touch different code / use different edit shapes?), and near-duplicates are merged or dropped deterministically (Self-Consistency's warning that duplicates masquerade as votes; Large Language Monkeys' warning that unselected duplicates approach random). It runs BEFORE any verification spend, so the budget buys distinct hypotheses. **Procedure (exact, deterministic):** 1. **Semantic diversity** — pairwise similarity over claims; near-identical claims score as one idea. 2. **Structural diversity** — pairwise overlap over the code the repairs touch and the edit shapes they use. 3. **Dedupe rule** — a hypothesis that is a near-duplicate (semantic AND structural, or an overwhelming semantic restatement) of one already in the set is **merged** (strategy variants folded in) or **dropped** deterministically; the merge/drop never asks the model. **Output schema (exact):** ```json { "hypotheses": [ "<deduplicated Hypothesis>" ], "raw_proposals": 0, // proposals entering (c), for the duplicate-rate instrument "merged_count": 0, "dropped_count": 0, "diversity": { "semantic": 0.0, "structural": 0.0 } } // §7 measures ``` **Caps:** dedup and diversity are pure deterministic software on CPU — the model never decides whether two of its own ideas are "really different." A high raw duplicate rate is recorded as a signal (§7.3), not blamed on the dedup stage. ### 3.4 (d) Behavioral clustering (D) **Input:** the deduplicated `Hypothesis` set. **Output:** the set grouped into behavioral clusters. **No language model runs here.** Hypotheses are clustered deterministically by the behavior they PREDICT — those whose predicted observables overlap are grouped so one executed test or trace can falsify the cluster jointly (P4), while their differing repair strategies remain recorded so a surviving cluster can feed multiple branches (fusion later, P6). **Round 1 ships basic clustering by overlap of the behavioral prediction; full-scale clustering is DEFERRED (row 11).** **Procedure (exact, deterministic):** 1. Group hypotheses whose `behavioral_prediction` observables overlap (same function/site/observable asserted). 2. Each cluster keeps its members' distinct strategies (for branching and fusion later), but is verified as one unit. **Output schema (exact):** ```json { "clusters": [ { "cluster_id": "…", "predicted_observables": [ "…" ], "hypothesis_ids": [ "…" ], "distinct_strategies": [ "…" ] } ] } ``` **Caps:** clustering is deterministic software on CPU — no model decides which of its hypotheses share a fate. Hypotheses predicting the same observable are verified as one cluster; their differing strategies are preserved. ### 3.5 (e) Cheap synthesized referee screening (D) **Input:** the behavioral clusters. **Output:** screened clusters — survivors cleared for the oracle, falsified clusters routed to pruning (h) with their prune-reason. **No language model runs here.** Each cluster is screened by a cheap deterministic referee before any expensive oracle call (P5): generated tests and type/structure/invariant probes, synthesized from the cluster's behavioral predictions, run at pocket change. A cluster the referee contradicts is falsified here and routed to stage (h), never reaching the oracle, which is reserved for the survivors. **Procedure (exact, deterministic):** 1. **Synthesize the referee** — from each cluster's `predicted_observables`, deterministically synthesize probes: generated tests exercising the predicted behavior, type/structure checks, static invariants, or a fast probe observing the predicted change. 2. **Run it** — every cluster runs the synthesized probes at pocket change. 3. **Route the verdict** — a cluster the referee contradicts is **falsified** and routed to (h) with its prune-reason, never to the oracle; survivors clear to stage (f). 4. **Catch the unfalsifiable** — a hypothesis whose prediction the referee cannot exercise is flagged before it costs an oracle call. **Output schema (exact):** ```json { "survivor_clusters": [ { "cluster_id": "…", "referee_probes": [ "…" ], "probe_outcomes": [ "…" ] } ], "falsified_clusters": [ { "cluster_id": "…", "contradicted_by": "…", "prune_reason": "…" } ] } ``` **Caps:** referee synthesis and screening are pure deterministic software on CPU — no model judges whether its own candidate survived. A cheap referee that false-kills good hypotheses is worse than none (measured by falsification precision, §7.3) and is calibrated deterministically. ### 3.6 (f) Candidate branching and budget enforcement (D) **Input:** the cheap-referee survivors. **Output:** branched, budget-enforced `Candidate`s. **No language model runs here.** Promising hypotheses are branched into MULTIPLE repair candidates where evidence warrants (the thesis's "branch promising hypotheses" clause; the search-expansion discipline of Tree of Thoughts / LATS / SWE-Search): a surviving hypothesis may yield several concrete repair edits, each a candidate. The total is enforced against the budget allocated in stage (a), deterministically, and the minimum-distinct-frame guarantee is re-checked so the final set still spans causally distinct ideas. **The `Candidate` schema (exact — EXACTLY these fields):** ```json { "id": "…", // the candidate's identifier "parent_hypothesis": "…", // the Hypothesis it was branched from "patch": "…", // the concrete repair edit(s) instantiating the hypothesis "evidence": [ "…" ], // recorded referee/oracle outcomes (§3.7) "score": 0.0, // deterministic survival/support score "status": "branched | verified | survived | falsified" } ``` `id` identifies the candidate; `parent_hypothesis` links it to the causal claim it instantiates (so falsification is attributable); `patch` is the concrete repair materializing the hypothesis; `evidence[]` accumulates the referee and oracle outcomes supporting or contradicting it (§3.7); `score` is a DETERMINISTIC survival/support score, never the model's self-assessment; `status` tracks its lifecycle. **Output schema (exact):** ```json { "candidates": [ "<Candidate>" ], "budget_used": 0, "budget_total": 0, "frame_span": [ "…" ], // distinct frames still represented after branching "over_budget_rejected": [ "…" ] } // branches cut by budget enforcement ``` **Caps:** branching and budget enforcement are pure deterministic software on CPU — the model neither decides how many branches it deserves nor which branch is best; an over-budget branch is rejected deterministically and the minimum-distinct-frame guarantee is enforced. ### 3.7 (g) Oracle verification and evidence recording (D) **Input:** the branched, budget-enforced `Candidate`s. **Output:** verified candidates with recorded evidence. **No language model runs here.** Each surviving candidate runs the expensive oracle — the real test suite, traces, invariants, or the verifier — and the outcome is recorded deterministically as EVIDENCE: which tests it passed, which it failed, which traces contradicted its behavioral prediction. This is the "let reality collapse the search space" step: reality, not the model, decides which hypotheses survive (Platt's crucial experiment). Survival is traceable to the observations that supported it. **Procedure (exact, deterministic):** 1. **Run the oracle** on each surviving candidate — real suite, traces, invariants, or verifier. 2. **Record the evidence** — append to `evidence[]` which tests passed/failed and which traces contradicted (or supported) its `behavioral_prediction`. 3. **Score deterministically** — update `score` from the recorded evidence; never from the model's self-assessment. **Output schema (exact):** ```json { "verified_candidates": [ "<Candidate with evidence[] populated and score set>" ], "oracle_evidence": [ { "candidate_id": "…", "tests_passed": [ "…" ], "tests_failed": [ "…" ], "traces_contradicting": [ "…" ], "traces_supporting": [ "…" ] } ] } ``` **Caps:** oracle verification and evidence recording are pure deterministic software on CPU — the model never judges whether its own candidate survived; a candidate is never marked survived on the model's confidence alone. ### 3.8 (h) Falsification pruning and constraint compilation (D) **Input:** the oracle-verified candidates and their evidence. **Output:** survivors, and the falsification ledger compiled into the regeneration-constraint (exclusion) set. **No language model runs here.** Hypotheses the oracle contradicted are eliminated deterministically, and each elimination writes a prune-reason to the falsification ledger — contradicted by test X, trace Y, invariant Z (P3/P7). The ledger is compiled into the regeneration-constraint set seeding future proposal calls, so the eliminated hypothesis and the reason it died are carried forward and cannot be re-proposed. This is the search-shrinking step that keeps the harness from being a memoryless sampler that rediscovers its own graveyard every round. **The `FalsificationRecord` schema (exact — EXACTLY these fields):** ```json { "hypothesis_id": "…", // the hypothesis (or candidate's parent) that was falsified "contradicted_by": "…", // the execution / test / trace / invariant that contradicted it "reason": "…" } // the prune-reason — why reality contradicted its behavioral prediction ``` **Procedure (exact, deterministic):** 1. **Eliminate** — a hypothesis whose `behavioral_prediction` the oracle contradicted is falsified; its candidates are marked `falsified`. 2. **Write the prune-reason** — for each elimination, append a `FalsificationRecord` to the ledger. 3. **Compile the constraints** — compile the ledger into the exclusion list seeding future proposal calls, so the generator is structurally prevented from re-proposing it (P3/P7). **Output schema (exact):** ```json { "surviving_candidates": [ "<Candidate with status survived>" ], "falsification_ledger": [ "<FalsificationRecord>" ], "regeneration_constraints": [ { "excluded_claim": "…", "prune_reason": "…" } ] } ``` **Caps:** pruning and constraint compilation are pure deterministic software on CPU — no model decides whether its own defeated idea stays excluded. The exclusion list is exhaustive over the ledger; a nonzero falsified-rebirth leak rate (§7.3) is treated as a defect, never excused. ### 3.9 (i) Agreement extraction and edit-atom assembly (D) **Input:** the surviving candidates. **Output:** a cross-candidate agreement record. **No language model runs here.** The survivors are compared deterministically to extract agreement (P6): which edit atoms they share, which they dispute, and where independent survival over independent falsification corroborates a change. The agreement structure is preserved for two downstream consumers — selection among survivors and the deterministic patch-fusion step that merges agreeing survivors into one patch. No hypothesis that lost is thrown away silently: its prune-reason is in the ledger (h). **Round 1 records a MINIMAL agreement record; full edit-atom assembly and fusion are DEFERRED (row 13).** **Procedure (exact, deterministic):** 1. **Compare survivors** — deterministic comparison of edit locations, edit shapes, and supported claims. 2. **Extract shared vs disputed atoms** — record which atoms the survivors SHARE and which they DISPUTE, plus where independent survival corroborates a change. 3. **Preserve for fusion** — retain the record for later selection and deterministic fusion; never a correctness verdict on its own (Degeneration-of-Thought forbids that inference). **Output schema (exact):** ```json { "agreement_record": { "shared_edit_atoms": [ "…" ], "disputed_edit_atoms": [ "…" ], "corroborated_changes": [ "…" ] } } ``` **Caps:** agreement extraction is pure deterministic software on CPU — no model decides whether its own survivors "really agree." Agreement is preserved as evidence for fusion, never as a verdict, until reality has confirmed it. **The L column holds exactly one entry.** Stages (a)–(i) are deterministic software on CPU; the single learned node is §4, reached once per request between frame decomposition and schema binding. Any addition putting a model elsewhere — frame decomposition, budget, schema binding, diversity, dedup, clustering, referee, branching, oracle, evidence, pruning, constraint compilation, agreement, choosing its best frame, judging its candidates, setting its own budget, or deciding its own diversity — is a regression against P1–P7 and must clear the §7 bar on held-out tasks at defensible cost, or be rejected. **Enforced mechanically: "the L column contains exactly one entry."** --- ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **`propose_from_frames`** — frame-conditioned hypothesis-and-strategy proposal. The weak model, seeded with the decomposed request, the repository context, the distinct frames from stage (a), and the ledger's exclusion list, emits hypotheses and repair strategies FROM EACH DISTINCT FRAME — one or a few per frame — each stating its claim, its strategy, and a `behavioral_prediction` reality can check. This is the SEMANTIC- JUDGMENT step and nothing more: **propose DIFFERENT IDEAS from DIFFERENT FRAMES, and state for each the behavioral prediction reality can check.** The model emits only PROPOSED ideas — it is **proposing, not selecting**. The model is **forbidden from selecting among its own proposals, judging which frame is best, deciding the budget, or evaluating whether its hypotheses are good** — those judgments are deterministic and downstream. **Prompt inputs (assembled deterministically, in this exact order):** 1. **`decomposed_request`** — the decomposed request (the task, the code it touches, the failure context when one exists). 2. **`repo_context`** — the repository context the request operates in. 3. **`frames`** — the distinct frames from stage (a), each with its rationale. 4. **`exclusion_list`** — the ledger's exclusion list (contradicted hypotheses + prune-reasons), so the call is structurally unable to regenerate a falsified idea (P3/P7). **Input → output schema (exact):** the model emits a JSON list of hypothesis proposals, one or a few per frame, each conforming exactly to `{ frame, claim, repair_strategy, behavioral_prediction }`. `frame` must name one of the supplied frames; `behavioral_prediction` must be a concrete observable (the field whose presence the schema-binding stage, §3.2, checks before accepting the hypothesis as falsifiable). **Caps:** **ONE call**, `temperature = 0`, `max_completion_tokens` bounded, schema-constrained; a bounded payload — the number of proposals bounded by the frame count and the per-frame cap set in stage (a). The model **PROPOSES only, never selects, judges, or budgets**. **Gating (exact):** the model emits distinct hypotheses and strategies from the distinct frames it was handed, each with a behavioral prediction. It does **not** select among them, judge which frame is best, decide the budget, or evaluate whether its hypotheses are good — those are stages (c)–(i)'s deterministic checks (§3); a proposal failing them is rejected, deduplicated, falsified, or pruned, never admitted on the model's say-so (§8 invariant 2). **Parse-failure fallback (exact):** any JSON parse failure, schema violation (a missing `frame`, an empty `claim` or `repair_strategy`, a missing or vacuous `behavioral_prediction`, a `frame` not in the supplied set, an out-of-bound or over-budget count), or empty/malformed output is **not silently retried as a fresh proposal**: the stage **returns to the deterministic stage with the constraint violation fed back** — the malformed proposal is recorded as a violation and counted as a rejected candidate — **never a re-prompt, never a second call at the same seed**. The L column is reached **once** per request. **Re-invocation on a later round (exact):** proposal is re-considered only when a *later round* of the same or a new request is processed, scoped strictly to that round's frame decomposition and fed the accumulated falsification ledger (§3.1/§3.8) as its exclusion list. It remains the **single conceptual learned entry** — a bounded, once-per-round call — **never a model that runs over the whole pipeline**. **Invariant 1 (§8): the L column contains exactly one entry — a bounded, seeded frame-conditioned proposer whose output the deterministic stages dispose of.** --- ## 5. Implementation Order Ten phases, each independently testable and reversible, landing in `build/16-candidate-generation-competing-hypotheses/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Frames + budget + intake (a)** | `frames.py` (§3.1 distinct frames + FIXED budget + exclusion-list parse) | Feed a request + a prior ledger: frames distinct; budget guarantees one candidate per frame; exclusion list contains the ledger's contradicted hypotheses; assert **no LLM consulted**. | Delete `frames.py`. | | **2. Proposal-stub (L)** | `propose.py` (§4 ONE call + temperature 0 + seeded + schema-constrained + parse-failure fallback) | With a mocked LLM: the call is **one**, temperature 0; output has exactly the proposal fields; malformed/over-cap → returns to the deterministic stage, **no second call**; feeds (b). | Delete `propose.py`. | | **3. Schema binding (b)** | `bind.py` (§3.2 normalization into `Hypothesis`; unfalsifiable → rejected) | A proposal with a parseable `behavioral_prediction` binds; one without → rejected as unfalsifiable; `Hypothesis` stays distinct from `Patch`; assert **no LLM consulted**. | Delete `bind.py`. | | **4. Diversity + dedup (c)** | `diversity.py`/`dedupe.py` (§3.3 semantic + structural diversity; near-dups merged/dropped before verification) | Two restatements of one idea → merged/dropped; two distinct claims → both kept; report populates; assert **no LLM consulted**. | Delete `diversity.py`/`dedupe.py`. | | **5. Clustering + cheap referee (d, e)** | `cluster.py` (§3.4 basic cluster-by-predicted-observable), `referee.py` (§3.5 deterministic probes; falsified clusters → (h), never the oracle) | Hypotheses sharing a predicted observable cluster together; a contradicted cluster → falsified with prune-reason, never reaches the oracle; assert **no LLM consulted**. | Delete `cluster.py`/`referee.py`. | | **6. Branching + budget (f)** | `branch.py` (§3.6 survivors branched into `Candidate`s; fixed-budget enforcement + frame-span re-check) | A survivor branches into several candidates; an over-budget branch → rejected; frame-span guarantee holds; assert **no LLM consulted**. | Delete `branch.py`. | | **7. Oracle + evidence (g)** | `verify.py`/`evidence.py` (§3.7 run the oracle; record tests passed/failed and contradicted traces; set the deterministic `score`) | A candidate whose prediction the oracle contradicts records the contradicting trace and scores accordingly; survival traceable to observations; assert **no LLM consulted**. | Delete `verify.py`/`evidence.py`. | | **8. Pruning + constraints (h)** | `prune.py`/`constraints.py` (§3.8 eliminate contradicted; append `FalsificationRecord`; compile the ledger into the exclusion set) | A contradicted hypothesis is eliminated with a `FalsificationRecord`; compiled constraints contain it so the next proposal cannot re-propose it; assert **no LLM consulted**. | Delete `prune.py`/`constraints.py`. | | **9. Agreement record (i)** | `agree.py` (§3.9 deterministic comparison into shared/disputed atoms) | Two survivors sharing an edit atom record it as shared; a dispute records as disputed; assert **no LLM consulted**. | Delete `agree.py`. | | **10. Loop driver + A/B harness** | `pipeline.py` (§3 wiring (a)→L→(b)→(c)→(d)→(e)→(f)→(g)→(h)→(i)), `ab.py` (§7 KEY + budget efficiency + secondaries + ablation runner + no-L-vs-L + decision) | `pipeline.py` on a fixture with a mocked stub: frames→propose→bind→dedupe→cluster→referee→branch→verify→prune→agree. `ab.py` runs the §7 pipeline-vs-baseline and ablations on held-out fixtures. | Feature flag `off` → live harness untouched; `ab.py` read-only. | The A/B runner (Phase 10) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The pipeline reuses the decomposed-request and repo-context artifacts and components 11–12's execution substrate for the cheap referee and the oracle; it invents no new execution substrate and no new diagnosis path. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of the earlier components' decomposed request, repo context, and execution substrate)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `re` | §3.1/§3.2 normalization templates; frame signatures | CPython stdlib. | **No** | | stdlib `json`, `os`, `sys`, `time` | §3 schemas, store persistence, provenance timestamps | CPython stdlib. | **No** | | stdlib `sqlite3` | §3.7/§3.8 the durable hypothesis + falsification stores and ledger | CPython stdlib. | **No** | | stdlib `ast` | §3.3/§3.9 structural diversity — code the repairs touch, edit shapes | CPython stdlib. | **No** | | stdlib `difflib` | §3.3/§3.9 structural/edit overlap measurement | CPython stdlib. | **No** | | stdlib `hashlib` | §3.1/§3.8 hypothesis/candidate/falsification hashing and exclusion keys | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic stages; the stores | In-repo, pinned interpreter. | **No** | | Decomposed-request + repo-context artifacts | §3.1 input to frame decomposition; §4 input to the proposal call | Built by earlier components; reused. | **No** | | Components 11–12 execution + traces substrate | §3.5 cheap-referee probes; §3.7 the oracle and its traces | Built by specs 11–12; reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single frame-conditioned proposal call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all stages use stdlib `re`/`json`/`sqlite3`/`ast`/`difflib`/`hashlib`; the decomposed request, repo context, and execution substrate are **reused wholesale from the earlier components**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The DEFERRED rows and the component-17 transfer (§2 rows 11–16) are a full clustering scale-up, an adaptive budget, edit-atom fusion, a trained generator, a learned judgment, or cross-session persistence on an existing store — not new third-party Python installs. Any future trained- model download, fine-tuning run, or learned-judgment dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol Candidate generation is measured *separately* and *honestly*, because a stage whose purpose is to produce distinct, evidence-constrained hypotheses can be defeated in either of two opposite ways: by generating five copies of the same idea and calling it diversity, or by generating five different ideas and wasting them all by choosing among them without evidence. The instruments keep the candidate set accountable to actual diversity and falsification, not to the model's confidence that it did well. **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, 13, 14, and 15** (Python-only, single-repo, no-Docker), each annotated additionally with the KNOWN DISTINCT correct solution strategies and known distinct bug causes (so diversity and correctness can be labeled independently of generation). Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. Held-out tasks feed the candidate-set, referee, oracle, and downstream outcome statistics; in-benchmark tasks feed nothing that scores the stage. **Baseline (concrete).** The **naive repeated-sampling baseline** — the failure mode this component rejects: on each request the model is asked to produce the fix N times (N = the candidate budget) at elevated temperature, with **no frame decomposition, no dedup, no cheap referee, no falsification constraints, no branching discipline**, and the "winner" is chosen by the model's own self-assessment (or, where unavailable, the first sample — the "marry your first sample" failure). The structured pipeline must beat *this* on the honest measures below, and the ablation (7.3) removes mechanisms against those same measures. ### 7.1 (a) Oracle-confirmed candidate diversity — the KEY measure The primary instrument, the direct score of the whole stage: **oracle-confirmed candidate diversity — the SEMANTIC + STRUCTURAL diversity of the set of hypotheses that SURVIVE cheap-referee + oracle verification per unit of budget**. This is the "five different ideas that reality allowed to survive" thesis made quantitative: a generator returning five near-identical hypotheses scores near zero no matter how plausible each is, and one whose "diverse" hypotheses all die in the referee or the oracle scores near zero because nothing it generated survived. Measuring diversity only over oracle-confirmed survivors closes the hole where a generator games pure diversity by proposing useless-but-distinct ideas. **State explicitly: "Never the end-to- end resolve rate alone"** — generation must not be judged by whether the eventual patch resolved (component 14's measure) but by whether the hypothesis set is genuinely diverse among the ideas reality kept. **State explicitly: "Never introspective fluency or model self-assessment" — a generator scoring highly on the model's own sense that it did well, while returning five copies of one idea, is the thesis's exact failure and scores near zero here.** **This is the KEY measure — the structured pipeline is adopted only if it STRICTLY beats the naive repeated-sampling baseline on this measure AND does not regress the budget efficiency (7.2).** ### 7.2 The companion + secondary instruments Report alongside the KEY measure; the secondaries report, do not gate on: 1. **Budget efficiency** — the companion: how much verification and oracle budget the harness spent per distinct idea carried forward — candidates generated per distinct hypothesis that survived, cheap-referee calls per candidate eliminated, oracle calls per survivor. A harness that spends ten oracle calls to learn what one cheap test would have shown is paying for diversity it never bought. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). **Tokens** = `tokens / instance`. **Secondary instruments (report, do not gate on):** 1. **Duplicate rate before deduplication** — of raw proposals, the fraction that were near-duplicates of an idea already in the set, measured at stage (c) before dedup merges them. A persistently high rate is a signal about the frames and the generator's mode-collapse, not a failure of the dedup stage. 2. **Falsification precision** — an audit of whether eliminations were correct: for hypotheses the cheap referee killed and for those the oracle killed, did the evidence actually contradict the hypothesis's behavioral prediction? Guards against over-aggressive pruning that eliminates a good hypothesis on a weak or misattributed signal. 3. **Falsified-rebirth leak rate** — of hypotheses eliminated with a prune-reason, the fraction proposed again in a later round — a regeneration the constraint set (P3/P7) should have made impossible. A nonzero leak rate is the clearest sign the candidate stage has quietly become the memoryless sampler the thesis forbids. 4. **Agreement-evidence yield** — whether the preserved cross-candidate agreement (P6) buys anything downstream: when two survivors agreed on an edit atom, did fusing on that agreement beat picking a single survivor? Verifies the harness is not preserving agreement records nobody reads. 5. **Survivor/resolve rate (report only)** — the end-to-end resolve rate of the surviving candidates, reported but NEVER the gating measure alone; it is component 14's measure, and a generation stage that improves resolve while leaving oracle-confirmed diversity flat against naive sampling is **not** adopted on that alone. ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add a generation mechanism because it sounds sensible" mechanically. Two passes, everything else held fixed, the delta measured on the honest measures (oracle-confirmed candidate diversity + budget efficiency + resolve/cost/tokens + secondaries): 1. **Ablate frame decomposition** — generate from one frame / no frame split: the direct test of P2. 2. **Ablate deduplication** — let near-duplicates consume the budget: the test that duplicates masquerade as votes. 3. **Ablate the cheap referee** — send every candidate straight to the oracle (the oracle-cascade failure): the test of P5's budget saving. 4. **Ablate falsification pruning + constraint compilation** — never record prune-reasons / never seed the exclusion list: the direct test of P3/P7 and the leak rate. 5. **Ablate behavioral clustering** — verify each hypothesis separately instead of by predicted observable: the test of P4's verification economy. 6. **Ablate branching** — never branch survivors: the test of the search-expansion clause. 7. **Ablate evidence recording** — survive without a recorded evidence trail: the test that survival is traceable. 8. **Ablate agreement preservation** — discard agreement records on finding a winner: the test of P6. 9. **Ablate the single L proposal step** — **no-proposal-vs-proposal** (deterministic-frame-only vs. ask) on the KEY measure: the test of whether the bounded stub (§4) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L stub, which must survive the no-proposal-vs-proposal ablation to be kept), then to any other mechanism. **Never add a generation mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run pipeline-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured pipeline **wins iff**: ``` Oracle-confirmed candidate diversity STRICTLY beats the naive repeated-sampling baseline (the KEY measure) AND budget efficiency STRICTLY beats the naive repeated-sampling baseline (the companion) AND falsified-rebirth leak rate not regressed (≤ baseline; nonzero = a defect to fix) AND cost_pipeline ≤ 1.5 × cost_baseline AND tokens_pipeline ≤ 1.5 × tokens_baseline ``` The KEY measure must **strictly beat the naive repeated-sampling baseline**; the budget efficiency must **strictly beat** it; the leak rate must not regress; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — resolve is component 14's measure, and a generation stage that improves resolve while leaving oracle-confirmed diversity flat against naive sampling is **not adopted**. **Never introspective fluency or model self- assessment** — a stage that returns five copies of one idea while the model reports it did well scores *worse* and is rejected. If the pipeline wins it replaces the naive sampling path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways a candidate generator can lie) 1. **Temperature-as-diversity** — raising the temperature and calling the scrambles distinct ideas; caught by near-zero semantic diversity on the KEY measure. 2. **Mode-collapse** — five restatements of the dominant (usually wrong) mode; caught by a persistently high duplicate rate before dedup. 3. **Duplicate-budget-burn** — spending the budget verifying copies before any distinct idea is generated; caught by collapsing budget efficiency. 4. **Falsified-rebirth** — an eliminated hypothesis reappearing because the ledger leaked; caught by the falsified- rebirth leak rate. 5. **Oracle-cascade** — every candidate going straight to the expensive oracle because the cheap referee was never run or was too strict; caught by budget efficiency and falsification precision together. 6. **Selection-by-vibes** — choosing survivors by the model's self-assessment rather than referee and oracle evidence; caught by the KEY measure's oracle-confirmation (model self-selection scores near zero). 7. **Agreement-as-truth** — mistaking cross-candidate agreement for proof of correctness (the Degeneration-of-Thought error); caught by remembering that agreement is evidence for fusion, never a verdict, until reality confirms it. ### 7.6 Two honest caveats **First, the strong-model caveat:** the paper's headline numbers — CodeT's 65.8% [5], CodeMonkeys's 66.2% [15], SWE-Search's 23% relative [8], PatchFusion's self-reported 426/500 and 236/300 [19] — will **NOT** reproduce on a weak model, and PatchFusion's in particular are self-reported from a fresh preprint and must not be treated as an anchor; a weak-model harness should expect materially smaller absolute effects from generation, which is precisely why it leans on cheap referees, deterministic dedup, and falsification pruning to make its small budget count. **Second, the held-out caution:** all candidate-generation metrics must be reported on held-out and out-of-benchmark tasks, so a generator that has memorized a task set's answers cannot score well on diversity without its hypotheses having been measured on problems it never saw. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/16-candidate-generation-competing-hypotheses/`; it never writes into the live harness's source. - A **feature flag** (env var `CANDGEN=16candidate|firstsample`, default `firstsample`) routes the A/B; the flag alone switches between the structured pipeline and the naive repeated-sampling/first-sample baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running old-vs-new is always possible in the same session, repo, model. - The durable `Hypothesis` store and the `FalsificationRecord` ledger are append-only and versioned; rolling the build back never loses them. A hypothesis falsified by later evidence is kept in the ledger with its prune-reason and its exclusion entries compiled into the regeneration constraints — never silently dropped and never silently regenerated. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains exactly one entry** — frame-conditioned hypothesis-and-strategy proposal (§4), the one bounded, seeded call proposing *different ideas from different frames*, reached once per request between frame decomposition and schema binding, never re-prompted on parse failure. Any phase introducing a model elsewhere (frame decomposition, budget, schema binding, diversity, dedup, clustering, referee, branching, oracle, evidence, pruning, constraint compilation, agreement, selecting among its own proposals, judging its best frame, setting its own budget, or choosing what to keep) is rejected outright. An **empty** L column (the stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **A contradicted hypothesis is never silently regenerated** — every falsification is recorded with a prune-reason (§3.8) and compiled into the exclusion list seeding every future proposal call; a hypothesis eliminated by evidence cannot reappear next round simply because the model, on a fresh call, forgot it died. A nonzero falsified-rebirth leak rate is a defect, never excused. 3. **Hypotheses and patches are separate objects** — a `Hypothesis` (`{ frame, claim, repair_strategy, behavioral_prediction, status }`) is never conflated with the `Patch` (`Candidate.patch`) that later materializes if it survives (P1). 4. **Every falsified hypothesis carries a prune-reason** — no elimination happens without a `FalsificationRecord` naming what contradicted it and why (P7). 5. **Diversity is measured before any verification spend** — dedup runs at stage (c), before the cheap referee (e) and the oracle (g), so the budget buys distinct hypotheses (P2). 6. **The cheap referee precedes the expensive oracle** — no candidate reaches the oracle that a cheap synthesized probe already falsified (P5). 7. **Survival is evidence-cited** — every surviving candidate's `evidence[]` records which tests passed/failed and which traces contradicted or supported its behavioral prediction; a candidate is never marked survived on the model's confidence alone (§3.7). 8. **Agreement is evidence, never a verdict** — cross-candidate agreement (P6) is preserved for selection and fusion, never treated as proof of correctness until reality has confirmed it. 9. **All metrics on held-out/out-of-benchmark tasks** — the KEY measure and every secondary are reported on held-out tasks, never tuning or in-benchmark data (§7). --- *End of spec. This document is the build contract; the next document records the A/B outcome.*