Summary / 15. Reflection after failure / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 15 — Reflection after Failure (the memory-admission stage) > Derived from `papers/15-reflection-after-failure.md` — "Retrying Is Cheap, Learning Is the Deliverable." > This document drives the build of the **reflection-after-failure stage**: the deterministic-first, > evidence-grounded **memory-admission** pipeline that sits immediately AFTER component 14 (iterative repair), > which loops the failed candidate toward passing, and BEFORE the verification/regression-testing and > cross-session-memory components, which judge whether a repair is actually good and then carry the harness's > learning across sessions. The sharp distinction is one of **abstraction level**, not merely order: component > 14 asks *"how do I improve this failed patch?"* and loops the **CANDIDATE**; component 15 asks *"what did this > failure teach me that should change how I approach similar problems?"* and loops the agent's **UNDERSTANDING**, > one abstraction level up. Where component 14's failure-signature ledger records WHAT failed and what resolved > it, this component records WHY — distilled into a scoped, supported, reusable lesson. The output of component > 15 is **not a patch** — it is a reusable, evidence-grounded lesson admitted to a durable store. Every failure > termination (resolved or escalated) triggers the pipeline; the weak model's one contribution is a schema-bound > CANDIDATE explanation; every subsequent step — grounding, generalization, dedupe, admission, promote/demote, > retrieval, outcome bookkeeping — is deterministic software that lets **only supported explanations into > memory**. The guiding rule, made mechanical: **A failure may propose a lesson, but only evidence earns that > lesson a place in memory.** This spec pins the paper's design to exact primitives, exact `Reflection`/`Lesson` > schemas, exact per-stage caps, exact L-column firing, exact A/B instruments and thresholds, exact ablation > mechanics, and an exact build sequence. Where the paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible memory-admission stage** — a deterministic-first, evidence-grounded reflection pipeline (evidence collection → ledger lookup → grounding → generalization → dedupe + calibration → admission gate → promote/demote + lifecycle → bounded retrieval → outcome statistics, plus the **single weak-LLM candidate- reflection-emission call**) that runs after **every** failure termination, converting the failure's full evidentiary record (traces 11–12, diagnosis 13, repair outcome 14) into a scoped, supported, reusable **Lesson**, admitting to the durable store only the explanations that survived grounding. It differs from its predecessor in *kind*, not order: component 14 emits a delta that makes a candidate pass; reflection emits what the failure *taught* — the mistaken assumption, overlooked evidence, failed strategy, or violated invariant that causally contributed. A repair is not yet learning. The store this stage admits to is the substrate the cross-session-memory component (17) will persist and consolidate; reflection is the memory-admission stage **without which components 16 and 17 would be judging and persisting an understanding the harness never built.**The paper's seven principles decide every design choice. **P1 Trigger reflection on every failure termination** — reflection is a **scheduled artifact**, never optional; every resolved or escalated failure runs it, because the failed trajectory carries information successes alone do not. **P2 Enforce the schema: no observed failure cited, no acceptance** — the causal chain must be explicit and each link checkable. **P3 Calibrate the abstraction level** — one level above the instance (boundary assumptions, contract misunderstandings), rejecting both the too-fine (line-level restatement) and the too-vague (virtue-level platitude); Chi's distinction: explanation *connecting a step to a principle* learns, *shallow restatement* does not. **P4 Persist outside the window; retrieve boundedly** — lessons live in a durable store, retrieved as a bounded, ranked subset by scope match. **P5 Promote and demote** — recurring lessons rise episodic→procedural→meta_strategy; falsified ones are demoted or retired; a store that never revises has stopped learning. **P6 Dedupe against the failure-signature ledger** — component 14's ledger records WHAT; reflection records WHY keyed to that signature, so the same signature never accumulates duplicate reflections. **P7 Distill downstream** — verified lessons are harvested into the fine-tuning/evaluation corpus (LeMa is the ancestor), closing the recursive self-improvement loop. **Round 1 lands the expected ≈80% of the paper's full 100% spec**: everything deterministic — evidence collection, ledger lookup, grounding, generalization, dedupe + calibration, the admission gate, promote/demote + lifecycle + the blame red-flag class, bounded retrieval, outcome statistics — plus the *bound* of the *one* candidate-reflection-emission call. The heavy pieces (a trained candidate-reflection generator, learned scope-adjudication, fine-tuning-harvest machinery at scale) are **not** built in Round 1 and are **not excluded** — later-round content climbing toward 100% (§2). Round 1 is kept only if it wins a measured A/B test on the *honest* metrics — led by the **lesson-admission precision** and its companion **speculative-reflection rejection rate** — never the end-to-end resolve rate alone (§7). The stage lives in `build/15-reflection-after-failure/` and writes nothing into the live harness. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–11 plus the bounded stub of row 12); **DEFERRED = later rounds** (rows 13–16), **not excluded** but gated by the same §7 A/B protocol. Cross-session persistence of the admitted store is a component-17 concern; even later rounds of this component do **not** build it. | # | Component (paper §4) | Classification | |---|---|---| | 1 | (a) Evidence collection — assemble the failure's full evidentiary record deterministically: execution output + traces (11–12), diagnosis + citations (13), repair outcome (14, resolved or escalated), and any successful trajectory for the contrast clause | **BUILD-NOW** | | 2 | (b) Failure-signature ledger lookup — hash into component 14's signature and look up; returns whether this WHY was reflected before, routing a known signature into corroborate-or-contradict rather than blind append (P6) | **BUILD-NOW** | | 3 | (c) Evidence grounding — check each causal claim against its cited evidence_ids; an unsupported claim → reject as speculative | **BUILD-NOW** | | 4 | (d) Generalization check — corroborate-or-contradict the store + prior failures; narrow/broaden the declared scope to match the evidence | **BUILD-NOW** | | 5 | (e) Dedupe + calibration — merge corroboration / flag contradiction against this signature's reflection; reject line-level as shallow, virtue-level as too vague (P3) | **BUILD-NOW** | | 6 | (f) Memory-admission gate — decide deterministically whether the grounded, generalized, deduped, calibrated candidate earns a `Lesson` | **BUILD-NOW** | | 7 | (g) Promote/demote + lifecycle + blame red-flag — run the three-level lifecycle off outcome statistics; route outward-blame lessons into the red-flag class | **BUILD-NOW** | | 8 | (h) Bounded ranked retrieval — on a new task, top-5 lessons ranked by scope match, hard cap 5 (P4) | **BUILD-NOW** | | 9 | (i) Outcome statistics — times_applied/helped/hurt incremented deterministically on resolution | **BUILD-NOW** | | 10 | Durable lesson store + failure-signature join — the append-only `Lesson` store keyed also to the ledger signature it explains | **BUILD-NOW** | | 11 | A/B harness (§7: admission precision + rejection rate + grounding fidelity + outcome ledger + promote/demote + retrieval effectiveness + harvest yield + ablation runner + failure-mode audits) | **BUILD-NOW** | | 12 | (L) Candidate reflection emission — **the single learned entry**, included as a *bounded stub* (cost rule below) | **BUILD-NOW** (bounded stub only) | | 13 | **Trained candidate-reflection generator at scale** | **DEFERRED** — Round 1 ships only the bounded stub (§4); full training/calibration deferred | | 14 | **Learned scope-adjudication or learned admission** | **DEFERRED**, and rejected unless it beats the deterministic gate on honest metrics; admission is deterministic (§3.6), never a model's say-so | | 15 | **Fine-tuning-harvest machinery at scale** (P7) | **DEFERRED** — Round 1 reports downstream-harvest yield on held-out tasks but runs no full training pipeline | | 16 | **Cross-session memory persistence / consolidation** | **DEFERRED to component 17** — reflection admits to a durable store (§3.10); persisting it across sessions is component 17's job | **The cost rule (explicit).** The single learned entry — candidate reflection emission — is the one L-column entry this architecture permits. In Round 1 it is a **bounded stub**: wired into the pipeline at its firing point (§4), gated, seeded by the deterministic evidence record, never auto-promoted; its full training and calibration (the recursive self-improvement loop) is **deferred**. Its cost is bounded before invocation: one call, temperature zero, schema-constrained, bounded payload, parse-failure fallback that returns to the deterministic stage with the violation fed back (§4 caps). It is kept **only if** the §7 no-L-vs-L ablation shows it improves the KEY measure on held-out tasks; otherwise it is removed and the L column stands empty — a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 puts no model in the admission path by default — the full deterministic pipeline (rows 1–11) plus the *bounded stub* of the single reflection call (§4) is the build; the L step is the exception. Each deferred row would add a trained second model, a learned admission/scope judgment, a full training run, or break the one-entry L column — and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out tasks, at a defensible cost, or be rejected. --- ## 3. The Deterministic Pipeline (D) Ten named stages — (a) through (i) plus the single learned entry (L) — all deterministic except L. The LLM column holds exactly one thing: candidate reflection emission (§4), reached once between the ledger lookup and evidence grounding. The pipeline: ``` FAILURE TERMINATION (resolved or escalated) + EVIDENCE (11-12) + DIAGNOSIS (13) + REPAIR OUTCOME (14) + LEDGER ENTRY (14) → (a) evidence collection [D] → the full evidentiary record (failed + successful trajectories) → (b) failure-signature ledger lookup [D] → signature + whether already reflected (corroborate-or-contradict vs append) → (§4) candidate reflection emission [L — the ONE entry; emits the candidate Reflection] → (c) evidence grounding [D] → accept, or reject as speculative → (d) generalization check [D] → corroborate/contradict store; narrow or broaden scope → (e) dedupe + calibration [D] → merge / flag contradiction / reject shallow or too-vague → (f) memory-admission gate [D] → admit as a Lesson, or reject/demote to unattached candidate → (g) promote/demote + lifecycle + blame red-flag [D] → level change; retire; red-flag outward blame → (h) bounded ranked retrieval [D] → on a new task, top-5 lessons ranked by scope match → (i) outcome statistics [D] → times_applied/helped/hurt incremented on resolution ``` The pipeline reasons over **the failed attempt made learnable** — the evidence record is the only thing the model reflects against, and the reflection it emits is grounded, generalized, deduped, gated, and lifecycle- managed deterministically. The weak model **proposes why**; deterministic software **disposes** what it is worth. All artifacts land in `build/15-reflection-after-failure/`. ### 3.1 (a) Evidence collection (D) **Input:** the failure termination (resolved or escalated), execution output + traces from components 11–12, the `Diagnosis` from component 13, the repair outcome from component 14, and — for the contrast clause — any **successful trajectory** of the same or a closely related task, when one exists. **Output:** the structured **evidentiary record** — the substrate every later stage grounds against. **No language model runs here.** Deterministic assembly of a *complete* record, rather than asking the model what it remembers, is the first defense against the unfaithful-reasoning problem: the model never chooses which facts are in view. **Procedure (exact, deterministic):** collect (1) component 12's instrumentation of the failed run; (2) the component 13 `Diagnosis` chain; (3) component 14's outcome — `resolved` (the passing delta) or `escalated` (the cap/ceiling hit); and (4) any successful trajectory. Each item is tagged with component provenance so a later grounding claim cites it by id. **Output schema (exact):** ```json { "failure_id": "…", "outcome": "resolved | escalated", "traces": [ "…" ], // component 11-12 execution output + instrumentation, provenance-tagged "diagnosis": "…", // the component-13 artifact: symptom → root-cause → cited evidence → fix direction "repair_delta": "…", // component 14's delta when resolved; the escalation target when escalated "successful_trajectory": "…" }// a contrast trajectory, when one exists ``` **Caps:** the record is assembled **only from deterministic inputs** — the model never narrates what it remembers. Exactly one record per failure termination; a record missing traces, diagnosis, or outcome does not proceed to reflection. ### 3.2 (b) Failure-signature ledger lookup (D) **Input:** the current failure (class, message template, site) plus component 14's persistent ledger. **Output:** the failure's **signature**, looked up, and whether this signature was reflected on before (P6). **No language model runs here.** The reflection being produced is **keyed to this signature**, so the ledger's WHAT and the reflection's WHY are joined; a signature with an existing reflection routes the new candidate into a **corroborate-or-contradict check** rather than a blind append. The lookup is a deterministic hash-and-index; the model never reasons about whether it has explained this failure before. **Ledger lookup procedure (exact, deterministic):** 1. **Hash** the failure into component 14's signature (class, message template, site). 2. **Look up** the signature. **If absent** — first occurrence: proceed to a fresh reflection. 3. **If present with a prior reflection** — route the candidate into stage (e)'s corroborate-or-contradict dedupe (§3.5) so the same signature never accumulates duplicate reflections. **Output schema (exact):** ```json { "signature": "…", // failure class + message template + site, as in component 14 "reflected_before": bool, // whether this signature already has an admitted reflection "prior_reflection_id": "…" }// when reflected_before, the existing lesson to dedupe against ``` **Caps:** pure deterministic hashing/indexing on CPU — no model consults the ledger. A first-seen signature proceeds to a fresh reflection; a known signature proceeds to dedupe. ### 3.3 (c) Evidence grounding (D) **Input:** the candidate `Reflection` from §4 plus the evidentiary record from stage (a). **Output:** a verdict — the candidate's causal claims are **grounded** in the evidence they cite, or the candidate is **rejected as speculative**. **No language model runs here.** This is the direct embodiment of the strong Piper rule — the model proposed, and now *evidence earns or refuses*. Each causal claim is checked against its cited `evidence_ids[]`: is the claimed assumption actually present in the failed plan? Is the claimed overlooked evidence actually in the record? Does the cited evidence support the stated cause, or contradict it? This is where plausible-but-unfaithful narratives and hallucinated reflections are caught, and where ExpeL's ablation warning is answered: the reflection that misleads distillation is exactly the one this stage rejects. **Grounding check (exact, deterministic):** for every populated causal-claim field, resolve each `evidence_id` against the record and test (1) it exists, (2) it is of the component kind the claim implies, and (3) its content supports rather than contradicts the claim. **Every causal claim must be supported by its cited evidence_ids; an unsupported claim → reject the candidate as speculative.** **The `Reflection` schema (the L output — exact — EXACTLY these fields):** ```json { "outcome": "resolved | escalated", "failed_assumption": "…", // the mistaken assumption that contributed, when one did "overlooked_evidence": "…", // the evidence that was overlooked, when that contributed "strategy_error": "…", // the failed strategy, when that contributed "violated_invariant": "…", // the violated invariant, when that contributed "evidence_ids": [ "…" ], // citations into components 11-13 observations supporting the causal claims "counterfactual": "…", // what would have had to differ for the outcome to change "candidate_lesson": "…", // the operative lesson at a transferable abstraction level "scope": "…", // the lesson's declared applicability "confidence": 0.0 } // 0.0..1.0, the model's assigned confidence ``` **Field meanings (exact):** `outcome` is resolved or escalated. The four causal-category fields name which of the four (mistaken assumption / overlooked evidence / failed strategy / violated invariant) the model believes contributed; an empty field states that category did not contribute, and **at least one** must be populated for the schema to be complete (P2's no-missing-link rule). `evidence_ids[]` cites component 11–13 observations grounding the causal claims. `counterfactual` states what would have had to differ. `candidate_lesson` is the operative lesson **at a transferable abstraction level** (P3). `scope` declares applicability. `confidence` ∈ [0,1] is the model's assigned confidence — a *proposal*, not an admission credential. **Output schema (exact):** ```json { "grounded": true, "verdict": "grounded | speculative", "unresolved_claims": [ "…" ] } // which causal claims failed grounding, when speculative ``` **Caps:** the grounding check is pure deterministic software on CPU — no model judges whether its own reflection is grounded. A candidate whose causal claims its own cited evidence does not support is **rejected as speculative** — stored nowhere, or demoted to an unattached candidate (§3.6) — and counts toward the speculative-reflection rejection rate (§7.2). ### 3.4 (d) Generalization check (D) **Input:** a grounded candidate plus the durable store and the harness's other experiences. **Output:** whether the candidate's declared scope is supported — corroborate, contradict, or independent, and scope narrowed or broadened accordingly. **No language model runs here.** This is the mechanism for the *scoped* clause: a lesson's scope must match the evidence, checked against the accumulated record rather than accepting the model's declared `scope` at face value. **Generalization check (exact, deterministic):** 1. **Corroborate-or-contradict** — a candidate consistent with admitted lessons/successes **corroborates** and may widen scope; one inconsistent with a prior failure it does not account for **contradicts** and must narrow scope or be flagged. 2. **Counterexample scan** — scan the store for a prior failure that would falsify the candidate at its declared scope; a found counterexample narrows the scope until it no longer covers it, or flags the candidate. 3. **A candidate contradicting an existing admitted lesson → both flagged for adjudication by future evidence, never silently overwrite.** **Output schema (exact):** ```json { "generalized": true, "adjusted_scope": "…", // scope after narrowing/broadening against the record "contradicts_lesson_id": "…", // set when the candidate contradicts an admitted lesson → both flagged "counterexample_ids": [ "…" ] }// prior failures that constrained the scope ``` **Caps:** deterministic software over the store — the model's declared scope is never accepted at face value; a contradiction never resolves by silent overwrite but flags both for adjudication. ### 3.5 (e) Dedupe + calibration (D) **Input:** the grounded, generalized candidate plus any prior reflection keyed to this signature (from stage (b)). **Output:** a deduped, calibrated candidate for the admission gate. **No language model runs here.** Two deterministic jobs: **dedupe** against this signature's existing reflections (P6) and **calibration** to the abstraction level (P3). **Dedupe (exact, deterministic):** 1. If a **corroborating** reflection for this signature exists, the candidate is **merged into it** — its evidence and (later) outcome statistics updated — not stored as a duplicate. 2. If the candidate **contradicts** an existing reflection, **both are flagged for adjudication by future evidence** — never silently overwriting. **Calibration (exact, deterministic)** — enforces Chi's distinction mechanically; a lesson must *connect the failure to an underlying principle* or it does not clear this stage: 1. **Line-level restatement** ("I wrote the wrong call on line 42") → **demoted to episodic or rejected as shallow** — it names the surface event without connecting it to a principle. 2. **Virtue-level** ("be more careful") → **rejected as too vague to act on**. 3. The calibrated level connects the failure to a principle at the right altitude — "I assumed the contract matched the docs; verify assumptions against observed behavior" — the only kind that transfers to a similar problem in a different repository. **Output schema (exact):** ```json { "deduped": "merge | flag_contradiction | new", "calibration": "calibrated | shallow | too_vague", "calibrated_lesson": "…", // the operative lesson after calibration (when calibrated), "episodic_only": false } // true when cleared only at the episodic, non-transferable level ``` **Caps:** dedupe and calibration are deterministic software on CPU — no model calibrates its own abstraction. A shallow or too-vague candidate never reaches the admission gate; a contradiction never resolves by silent overwrite. ### 3.6 (f) Memory-admission gate (D) **Input:** the grounded, generalized, deduped, calibrated candidate. **Output:** **admit** as a `Lesson`, or **reject/demote**. **No language model runs here.** This is the enforcement of "only evidence earns": the gate is the single choke point where a model's confident narrative becomes (or fails to become) a durable belief. **Admission rule (exact, deterministic):** **admit only if the candidate PASSES grounding + generalization + calibration.** (1) **Grounding** — its causal claims are supported by its cited evidence (§3.3). (2) **Generalization** — it does not contradict the accumulated record, or its contradiction is flagged for adjudication (§3.4). (3) **Calibration** — it sits at a transferable abstraction (§3.5). A candidate that failed grounding, or whose evidence is thin relative to its scope, is **rejected — stored nowhere, or demoted to an unattached candidate** the harness may revisit if corroborating evidence appears. **The `Lesson` schema (the deterministic store — exact — EXACTLY these fields):** ```json { "statement": "…", // the operative lesson, at a transferable abstraction "type": "episodic | procedural | meta_strategy", "scope": "…", // declared applicability "derived_from": { "failures": [ "…" ], "successes": [ "…" ] }, // provenance "evidence": [ "…" ], // the evidence grounding the lesson (resolved evidence_ids) "counterexamples": [ "…" ], // accumulated counterexamples (narrow or retire the lesson) "confidence": 0.0, // 0.0..1.0 "times_applied": 0, "times_helped": 0, "times_hurt": 0, // outcome statistics (§3.9) "supersedes": "…", // pointer to the lesson this one replaces, when any "superseded_by": "…" } // pointer set when this lesson is demoted/retired ``` **Field meanings (exact):** `statement` is the operative lesson at the transferable level; `type` ∈ {episodic, procedural, meta_strategy} places it in the three-level store (episodic = repo/task-specific, procedural = cross-repo, meta_strategy = class-of-problem). `scope` declares applicability. `derived_from` records provenance — the failures/successes it came from. `evidence[]` carries the grounding evidence so **every lesson is evidence-cited**. `counterexamples[]` accumulates prior failures that constrain or falsify it. `confidence` is carried from the candidate (itself a proposal). `times_applied/helped/hurt` are the outcome statistics (§3.9). `supersedes`/`superseded_by` track the lifecycle (§3.7). **Output schema (exact):** ```json { "admitted": true, "lesson_id": "…", "rejected_reason": "speculative | contradicts_store | shallow | too_vague | thin_evidence", "unattached_candidate": bool } // true when rejected but held for later revisit ``` **Caps:** the gate is deterministic software on CPU — no model decides whether its own reflection earns a place in memory. A gate that admits everything the model proposes is a rubber stamp and shows up immediately in the KEY measure (§7.1). ### 3.7 (g) Promote/demote + lifecycle + blame red-flag (D) **Input:** an admitted `Lesson` (or the accumulated store, on a lifecycle sweep) and its outcome statistics. **Output:** the lesson at its correct level, promoted/demoted/retired deterministically, outward-blame lessons routed to the red-flag class. **No language model runs here.** This is the continual-revise-or-delete clause made mechanical (P5). **Lifecycle rules (exact, deterministic):** - **Promote:** episodic → procedural → meta_strategy after **≥3 cross-repository times_helped**. - **Demote/retire:** after **≥3 times_hurt** OR **one falsifying counterexample**; `superseded_by` set, edges cleaned, never silently kept (§8 invariant 8). **Blame red-flag (exact, deterministic):** a lesson whose content is "the tests were wrong" / "the framework is broken" — blame directed **outward** rather than at the harness's own reasoning — is the classic shape of a model explaining away its own error. Such a lesson is routed into a **red-flag class** requiring **exceptional evidence** (multiple independent corroborating failures or a reproducible external defect) before admission at anything **above the lowest confidence**. **Output schema (exact):** ```json { "level": "episodic | procedural | meta_strategy", "action": "admit | promote | demote | retire | redflag", "superseded_by": "…", // set on demote/retire "red_flag_class": bool, // true when outward-blame content is flagged "confidence_cap": "lowest" } // red-flag lessons capped to lowest until exceptional evidence clears them ``` **Caps:** promote/demote thresholds and red-flag classification are deterministic rules on CPU — the model never chooses its own level, retirement, or whether its outward blame is credible. Every admitted lesson is **lifecycle-tracked** — a lesson is never immutable. ### 3.8 (h) Bounded ranked retrieval (D) **Input:** a new task whose failure has terminated, plus the durable store. **Output:** a bounded, ranked subset of lessons whose declared scope matches the task, ready to seed the next attempt (and component 17's surface). **No language model runs here.** P4 made concrete: Reflexion's episodic-memory retrieval is the ancestor — the gain lives in carrying the right past reflection forward — and bounded ranking keeps the store usable at scale. **Retrieval rule (exact, deterministic):** lessons are ranked by **scope match** to the current task and the **top-5** retrieved — **bounded retrieval: top-5 lessons ranked by scope match, hard cap 5, per new task.** The ranking is deterministic (a scope-match score over the store); the model never reasons about which past lessons to recall. **Output schema (exact):** ```json { "lessons": [ "<Lesson>, ranked by scope match" ], "cap": 5, "retrieved_lesson_ids": [ "…" ] } // bounded to 5 ``` **Caps:** retrieval is a deterministic rank-and-cap on CPU — never more than 5 lessons per new task enter the window, and no model decides which of its own past lessons to recall. ### 3.9 (i) Outcome statistics (D) **Input:** an admitted lesson that was applied to a task, and that task's outcome once it resolves. **Output:** the lesson's outcome statistics incremented deterministically. **No language model runs here.** **Outcome statistics: times_applied/helped/hurt incremented deterministically each time a lesson is applied and the task resolves.** `times_applied` always increments; `times_helped` XOR `times_hurt` per the outcome. **Caps:** deterministic bookkeeping on CPU — no model records its own outcome. These statistics are the continuing evidence that promotes, demotes, or retires every lesson (§3.7), keeping the store a live model of what works rather than a fossil. **The L column holds exactly one entry.** Stages (a)–(i) are all deterministic software on CPU. The single learned node is §4, candidate reflection emission, reached once per failure termination between ledger lookup and evidence grounding. Any proposed addition putting a language model elsewhere — evidence collection, ledger lookup, grounding, generalization, dedupe/calibration, the admission gate, lifecycle, retrieval, outcome bookkeeping, judging its own reflection, deciding its own scope, or choosing what to admit — is a regression against P1–P7 and the L-column invariant, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of §7, on held-out tasks, at a defensible cost, or be rejected. --- ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **`reflect`** — candidate reflection emission. The weak model, seeded with the evidentiary record and the ledger context, emits **ONE** candidate reflection conforming exactly to the typed `Reflection` schema (§3.3). The schema forces the causal chain P2 demands: the model names the outcome, identifies which of the four causal categories contributed, cites the supporting `evidence_ids`, states a counterfactual, proposes a `candidate_lesson` at the calibrated abstraction level (P3), declares `scope`, and assigns `confidence`. Crucially, the model emits only a **CANDIDATE** explanation — it is **proposing, not admitting**. Every judgment about whether the candidate's claims are grounded, whether it generalizes, and whether it earns a place in memory is deterministic (§3). **Prompt inputs (assembled deterministically, in this exact order):** 1. **`evidence_record`** — the full evidentiary record from stage (a): traces (11–12), diagnosis (13) with its citations, repair outcome (14), and any successful trajectory for contrast. 2. **`signature_context`** — the ledger context from stage (b): the failure signature and whether this WHY was reflected on before, shaping the candidate toward corroboration or contradiction rather than a blind second story. **Input → output schema (exact):** the model emits the `Reflection` schema verbatim (§3.3): `{ outcome, failed_assumption, overlooked_evidence, strategy_error, violated_invariant, evidence_ids[], counterfactual, candidate_lesson, scope, confidence }`. **Caps:** **ONE call**, `temperature = 0`, `max_completion_tokens = 2048`, schema-constrained; a bounded payload — causal fields and `evidence_ids[]` bounded to the evidence actually in the record. The model **PROPOSES only, never admits**. **Gating (exact):** the model emits a candidate explanation of *why* the failure happened. It does **not** judge whether it is grounded, generalizes, or earns a place in memory — those are stages (c), (d), (f)'s checks (§3.3, §3.4, §3.6); a candidate failing them is **rejected as speculative or demoted**, never admitted on the model's say-so (§8 invariant 2). **Parse-failure fallback (exact):** any JSON parse failure, schema violation (a missing causal field where P2 requires one, an empty `candidate_lesson`, an out-of-bound `evidence_id`), or empty/malformed output is **not silently retried as a fresh reflection**: the stage **returns to the deterministic stage with the constraint violation fed back** — the malformed proposal is recorded as a violation and counted as a rejected candidate — **never a re-prompt, never a second call at the same seed**. The L column is reached **once** per failure termination. **Re-invocation on a later failure (exact):** reflection is re-considered only when a *new* failure termination is processed (a fresh evidentiary record, §3.1), scoped strictly to that record; it re-uses none of a prior firing's output except as store/ledger context. It remains the **single conceptual learned entry** — a bounded, once-per-termination call — **never a model that runs over the whole pipeline**. **Invariant 1 (§8): the L column contains exactly one entry — a bounded, seeded candidate-reflection emitter whose output the deterministic stages dispose of.** --- ## 5. Implementation Order Ten phases, each independently testable and reversible, landing in `build/15-reflection-after-failure/` as standalone importable modules. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Evidence collection (a)** | `evidence.py` (§3.1 assembly of the evidentiary record from traces (11–12) + diagnosis (13) + repair outcome (14) + any successful trajectory, provenance-tagged) | Feed a resolved/escalated termination with traces + diagnosis + outcome: the record has exactly the five fields; `outcome` is `resolved` or `escalated`; assert **no LLM consulted**. | Delete `evidence.py`. | | **2. Ledger lookup (b)** | `ledger.py` (§3.2 hashing into component 14's signature + lookup returning reflected_before + prior lesson id) | Identical failures hash to the same signature; first-seen returns `reflected_before: false`; a known signature returns the prior id; assert **no LLM consulted**. | Delete `ledger.py`. | | **3. Reflection-stub (L)** | `reflect.py` (§4 firing + ONE call + temperature 0 + seeded with `evidence_record` + `signature_context` + schema-constrained `Reflection` output + parse-failure fallback feeding the violation back) | With a mocked LLM: the call is **one**, temperature 0; output has exactly the `Reflection` fields; a malformed/over-cap response → returns to the deterministic stage, **no second call**; feeds stage (c). | Delete `reflect.py`. | | **4. Evidence grounding (c)** | `ground.py` (§3.3 resolving each causal claim against its cited `evidence_ids`; unsupported → speculative) | A claim supported by an existing, consistent citation passes; a claim with no/nonexistent/contradicting citation → `speculative`; assert **no LLM consulted**. | Delete `ground.py`. | | **5. Generalization check (d)** | `generalize.py` (§3.4 corroborate-or-contradict + counterexample scan + adjusted scope; contradiction flags both) | A consistent candidate corroborates; a counterexample narrows scope; a contradiction flags both; assert **no LLM consulted**. | Delete `generalize.py`. | | **6. Dedupe + calibration (e)** | `calibrate.py` (§3.5 merge corroboration / flag contradiction; line-level → shallow; virtue-level → too_vague) | A corroboration merges; a contradiction flags both; a line-level `candidate_lesson` → `shallow`; a virtue-level one → `too_vague`; assert **no LLM consulted**. | Delete `calibrate.py`. | | **7. Admission gate (f)** | `admit.py` (§3.6 the gate admitting only candidates that PASS grounding + generalization + calibration; the `Lesson` schema with all sixteen fields) | Candidates passing/failing each gate: only a grounded + generalized + calibrated candidate admits with the exact `Lesson` schema; `derived_from` and `evidence` present; assert **no LLM consulted**. | Delete `admit.py`. | | **8. Lifecycle + red-flag (g)** | `lifecycle.py` (§3.7 promote after ≥3 cross-repo times_helped; demote/retire after ≥3 times_hurt or one falsifying counterexample; outward-blame → red-flag class) | Promotion fires at threshold; demotion/retirement fires with `superseded_by` set; an outward-blame lesson is `redflag`-classified and capped; assert **no LLM consulted**. | Delete `lifecycle.py`. | | **9. Retrieval + outcome stats (h, i)** | `retrieve.py` (§3.8 top-5 by scope match, cap 5), `record.py` (§3.9 deterministic times_applied/helped/hurt) | `retrieve.py`: top-5 return, never more than 5. `record.py`: resolved → `times_helped`; worse → `times_hurt`; both increment `times_applied`; assert **no LLM consulted**. | Delete `retrieve.py` / `record.py`. | | **10. Loop driver + A/B harness** | `pipeline.py` (§3 wiring (a)→(b)→L→(c)→(d)→(e)→(f)→(g), plus (h)/(i) on resolution), `ab.py` (§7 KEY measure + rejection rate + secondaries + ablation runner + no-L-vs-L + decision) | `pipeline.py` on a fixture with a mocked stub: the sequence runs collect→lookup→reflect→ground→generalize→calibrate→admit→lifecycle. `ab.py` runs the §7 pipeline-vs-baseline and ablations, emitting the decision on held-out fixtures. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 10) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The pipeline reuses component 14's ledger and components 11–13's evidence wholesale; it invents no new execution substrate and no new diagnosis path. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of components 11–14's evidence and ledger)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `hashlib` | §3.2 hashing into component 14's signature; §3.6 lesson-id/artifact hashing | CPython stdlib. | **No** | | stdlib `re` | §3.2 message-template normalization matching component 14's signature | CPython stdlib. | **No** | | stdlib `json`, `os`, `sys`, `time` | §3 schemas, store persistence, provenance timestamps | CPython stdlib. | **No** | | stdlib `sqlite3` | §3.6/§3.8 the durable `Lesson` store and bounded retrieval | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic stages; the store | In-repo, pinned interpreter. | **No** | | Components 11–12 execution + evidence | §3.1 the `traces`; §3.3 the cited evidence | Built by specs 11–12; reused. | **No** | | Component 13 diagnosis artifact | §3.1 the `diagnosis`; §3.3 root-cause citations | Built by spec 13; reused. | **No** | | Component 14 repair outcome + ledger | §3.1 `repair_delta`/`outcome`; §3.2 the ledger the WHY is keyed to (P6) | Built by spec 14; reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single candidate-reflection call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all stages use stdlib `hashlib`/`re`/`json`/`sqlite3`; evidence and the ledger are **reused wholesale from components 11–14**, not duplicated. The single LLM call (§4) goes to the harness's own weak model as a bounded stub. **No new package is installed for Round 1.** The DEFERRED rows and the component-17 transfer (§2 rows 13–16) are a trained generator, a learned admission/scope judgment, fine-tuning runs, or cross-session persistence on an existing store — not new third-party Python installs. Any future trained-model download, fine-tuning run, or learned-adjudication dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol Reflection is measured *separately* and *honestly*, because a stage whose purpose is to decide what the harness believes about its own past can be defeated by mistaking a model's plausible story for an earned lesson, or by a store that accumulates confident noise and is never caught. The instruments keep the admission process and the store accountable to actual outcomes, not to the model's introspective fluency. **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, 13, and 14** (Python-only, single-repo, no-Docker) as the source of a held-out **(failure termination, evidence pack, diagnosis, repair outcome, known-cause, should-admit/should-reject lesson set) corpus**. Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. Held-out tasks feed retrieval and outcome statistics; in-benchmark tasks feed nothing that scores the gate. **Baseline (concrete).** The **naive-introspection baseline** — the failure mode this component rejects: on each failure termination the model is asked **"what did you learn"** and the answer is stored **directly, with no grounding, no generalization, no dedupe, no calibration, no admission gate, no lifecycle, no bounded retrieval** (whatever the model wrote is kept, Reflexion-in-pure-form). The structured pipeline must beat *this* on the honest measures below, and the ablation (7.3) removes mechanisms against those same measures. ### 7.1 (a) Lesson-admission precision — the KEY measure The primary instrument, the direct score of the whole admission gate: **lesson-admission precision — of admitted lessons, the fraction that subsequently prove helpful (aggregate times_helped / times_applied) on held-out tasks**; plus the companion **speculative-reflection rejection rate** (fraction of candidate reflections rejected as ungrounded). This is the honest measure of the gate, making the strong Piper rule measurable rather than rhetorical: a gate that admits everything the model proposes shows low precision, because ungrounded reflections and unfaithful narratives dominate; a gate that admits nothing shows perfect precision on an empty store. It is the ground truth the admission-gate-tuning loop optimizes. **State explicitly: "Never the end-to-end resolve rate alone"** — reflection must not be judged by whether the patch resolved (component 14's job) but by whether what it admits proves helpful later. **State explicitly: "Never introspective fluency — a store of confident noise is worse than an empty store."** **This is the KEY measure — the structured pipeline is adopted only if it STRICTLY beats the naive-introspection baseline on this measure AND does not regress the rejection rate (7.2).** ### 7.2 The companion + secondary instruments Report alongside the KEY measure; the secondaries report, do not gate on: 1. **Speculative-reflection rejection rate** — the companion: of candidates proposed, the fraction rejected as ungrounded (causal claims unsupported by cited evidence, or scope exceeding the evidence). A healthy stage rejects a substantial fraction; a stage that rejects almost nothing has made its gate a rubber stamp and quietly become the introspection the thesis forbids. 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). **Tokens** = `tokens / instance`. **Secondary instruments (report, do not gate on):** 1. **Evidence-grounding fidelity** — for a sample of admitted lessons, is the `failed_assumption` actually present, is the `overlooked_evidence` actually in the record, does the cited evidence establish the claimed cause? Checks both failure modes at once — the shallow restatement that teaches nothing and the unfaithful narrative. 2. **Per-lesson outcome ledger** — each lesson's `times_applied/helped/hurt`, counterexamples, promotion level. Demotion/retirement rates are reported as *health*: a store that never revises has stopped learning. 3. **Promote/demote + retrieval effectiveness** — do promoted lessons help more across repos? Do tasks whose bounded retrieval included the right lesson resolve better than those whose did not? 4. **Downstream-harvest yield** — of lessons harvested into the fine-tuning/eval corpus (P7, deferred at scale in Round 1 but reported as yield on held-out tasks), what fraction demonstrably reduce the targeted failure class? ### 7.3 Ablation rule (mechanical) The instrument that enforces "never add a reflection mechanism because it sounds sensible" mechanically. Two passes, everything else held fixed, the delta measured on the honest measures (admission precision + rejection rate + resolve/cost/tokens + secondaries): 1. **Ablate grounding** — admit without checking causal claims against cited evidence: the direct test of the strong Piper rule. 2. **Ablate generalization** — admit without the corroborate-or-contradict/counterexample check. 3. **Ablate dedupe** — never merge/flag against the signature's existing reflection, so the same WHY accrues duplicates. 4. **Ablate calibration** — admit line-level and virtue-level reflections: the test of Chi's distinction. 5. **Ablate the admission gate** — admit everything the model proposes (the naive baseline): the direct demonstration that introspective fluency is not learning. 6. **Ablate lifecycle (promote/demote)** — never promote or retire: does the store accrue dead weight? 7. **Ablate retrieval ranking** — retrieve without the top-5 scope-match rank/cap: does the right lesson reach the window? 8. **Ablate the single L reflection step** — **no-reflection vs reflection** (never ask why vs. ask) on the KEY measure: the test of whether the bounded stub (§4) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L stub, which must survive the no-reflection-vs-reflection ablation to be kept), then to any other mechanism. **Never add a reflection mechanism because it sounds sensible.** ### 7.4 Decision rule (explicit thresholds) Run pipeline-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured pipeline **wins iff**: ``` Lesson-admission precision STRICTLY beats the naive-introspection baseline (the KEY measure) AND speculative-reflection rejection rate is healthy (a substantial, non-trivial fraction rejected) AND cost_pipeline ≤ 1.5 × cost_baseline AND tokens_pipeline ≤ 1.5 × tokens_baseline ``` The KEY measure must **strictly beat the naive-introspection baseline**; the rejection rate must show the gate is discriminating, not a rubber stamp; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — resolve is component 14's measure, and a reflection stage that improves resolve while leaving admission precision flat against naive introspection is **not adopted**. **Never introspective fluency** — a pipeline that admits a store of confident noise scores *worse* than the empty store and is rejected. If the pipeline wins it replaces the naive-introspection path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Failure-mode audit (mechanical, the ways a reflection stage can lie) 1. **Introspection-in-disguise** — asks the model what it learned and stores it without grounding; caught by admission precision collapsing toward the naive baseline. 2. **Confabulated-cause** — an admitted lesson's stated cause does not match its cited evidence; caught by grounding fidelity. 3. **Blame-deflection** — lessons route cause outward ("the tests were wrong"); caught by the red-flag class requiring exceptional evidence (§3.7). 4. **Shallow-restatement** — lessons restate at the line level and teach nothing reusable; caught by the calibration stage and Chi's distinction (§3.5). 5. **Unfalsifiable-store** — lessons never demoted regardless of outcome; caught by a near-zero demotion/retirement rate in the outcome ledger. 6. **Retrieval-miss** — the store holds the right lesson but retrieval never surfaces it; caught by the retrieval-effectiveness instrument. ### 7.6 Two honest caveats **First, the strong-model caveat:** the paper's headline numbers — Reflexion's 91% pass@1 [1], SWE-Exp's 73.0% [3], ReasoningBank's ~20% relative [2] — were obtained with far stronger models and will **NOT** reproduce on a weak model; a weak-model harness should expect materially smaller absolute effects from reflection, which is precisely why it leans on deterministic grounding, calibration, and the outcome ledger to compensate for weak introspective accuracy with strong evidence-admission discipline. **Second, the held-out caution:** all reflection metrics must be reported on held-out and out-of-benchmark tasks, so a store that has memorized a task set's failures cannot score well without its lessons grounded in evidence the harness actually observed. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/15-reflection-after-failure/`; it never writes into the live harness's source. - A **feature flag** (env var `REFLECT=15reflect|introspect`, default `introspect`) routes the A/B; the flag alone switches between the structured pipeline and the naive-introspection baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running old-vs-new is always possible in the same session, repo, model. - The durable `Lesson` store and ledger join are append-only and versioned; rolling the build back never loses them. A lesson falsified by later evidence is retired via `superseded_by` + edge-cleaning, never overwritten in place. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains exactly one entry** — candidate reflection emission (§4), the one bounded, seeded call proposing *why* a failure happened, reached once per failure termination between ledger lookup and grounding, never re-prompted on parse failure (the fallback returns to the deterministic stage with the violation fed back). Any phase introducing a language model elsewhere (evidence collection, ledger lookup, grounding, generalization, dedupe/calibration, the admission gate, lifecycle, retrieval, outcome bookkeeping, judging its own reflection, deciding its own scope, choosing what to admit) is rejected outright. An **empty** L column (the stub ablated away in §7.3) is a permitted, correct state — never a violation. 2. **The model proposes, evidence disposes** — no reflection is admitted without grounding against its cited evidence; a candidate whose causal claims its own cited evidence does not support is rejected as speculative, never admitted on the model's confidence. 3. **Every lesson is evidence-cited** — `derived_from` (`failures[]`/`successes[]`) and `evidence[]` present on every admitted `Lesson`; a lesson without them is a rejected deliverable (§3.6). 4. **Blame-deflection is red-flagged** — outward blame ("the tests were wrong" / "the framework is broken") requires exceptional evidence before admission above the lowest confidence (§3.7). 5. **Every lesson is lifecycle-tracked** — `times_applied/helped/hurt` and `supersedes/superseded_by` maintained deterministically; a lesson is never immutable. Promote after ≥3 cross-repo times_helped; demote/retire after ≥3 times_hurt or one falsifying counterexample (§3.7). 6. **Dedupe is keyed to the failure-signature ledger** — reflection (WHY) joined to the ledger's signature (WHAT); a reflection on an already-reflected signature is merged or flagged, never blindly appended (§3.5). 7. **All metrics on held-out/out-of-benchmark tasks** — the KEY measure and every secondary are reported on held-out tasks, never tuning or in-benchmark data (§7). 8. **A lesson falsified by later evidence is retired, never silently kept** — a counterexample contradicting an admitted lesson, or ≥3 times_hurt, demotes or retires it with `superseded_by` set and edges cleaned (§3.7). --- *End of spec. This document is the build contract; the next document records the A/B outcome.*