Summary / 15. Reflection after failure / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Retrying Is Cheap, Learning Is the Deliverable: Evidence-Grounded Reflection as the Memory-Admission Stage of a Weak-Model Coding Harness *Component 15 of 27 — Coding-Harness Program. This document specifies the reflection-after-failure stage. It sits directly AFTER component 14 (iterative repair), which takes a failed patch and loops the candidate toward passing, and BEFORE the verification/regression-testing and cross-session-memory components, which judge whether a repair is actually good and then carry the harness's learning across sessions. The sharp distinction is one of abstraction level: component 14 asks "how do I improve this failed patch?" and loops the CANDIDATE; component 15 asks "what did this failure teach me that should change how I approach similar problems?" and loops the agent's UNDERSTANDING, one abstraction level up. The output of component 15 is not a patch — it is a reusable lesson, strategy, constraint, warning, or corrected assumption. The failure-signature ledger of component 14 records WHAT failed; this component records WHY. This paper translates the program's thesis into the memory-admission build contract: run reflection as a deterministic distillation pipeline after every failure termination, in which the weak model's one contribution is a schema-bound CANDIDATE explanation of why the failure happened, and every subsequent step — evidence grounding, generalization checking, dedupe, admission, promotion, demotion, retrieval, and outcome bookkeeping — is deterministic software that lets only supported explanations into memory.* --- ## Abstract Reflection after failure is the stage where the coding harness decides what a failed attempt is worth keeping, and it is the component where the program's most seductive failure mode lives: the model's confident, plausible post-hoc story about why it failed. Introspection is cheap and almost always wrong, and this paper argues that reflection must be an evidence-grounded experience-distillation stage, not introspection: the harness does not ask the model to feel its way to wisdom; it runs a deterministic pipeline that converts a known outcome into a scoped, supported, reusable lesson, admitting to memory only the explanations that survived grounding. The paper reviews the learning-from-failure literature in fifteen parts — Reflexion as the anchor whose verbal-reinforcement episodic memory is the direct ancestor of this stage [1], ReasoningBank's induction from failures and its honest ~20% relative ceiling [2], SWE-Exp's multi-level experience extraction at 73.0% Pass@1 [3], ExpeL's cross-task insight distillation with its ablation warning that model-generated reflections can hurt [4], LeMa's mistake-correction-to-finetune path [5], Self-Refine [6], the two negative results that discipline the whole design — that LLMs cannot reliably self-correct [7] and that stated reasoning is frequently unfaithful [8] — plus the experience-graph and survey mentions [9][10][11][12][13][14] and the classical self-explanation root of Chi [15]. It derives seven design principles, then specifies a deterministic pipeline (D) with exactly one learned entry (L): evidence collection, evidence grounding, generalization, dedupe against the failure-signature ledger, the memory-admission gate, promotion/demotion, and bounded retrieval are deterministic software, and the single L entry is the emission of a schema-bound candidate reflection. It closes with a recursive self-improvement protocol and a measurement plan whose honest instruments — the lesson-admission precision, the speculative-reflection rejection rate, the per-lesson outcome ledger, and the revise/delete rate — exist to keep the harness from mistaking a model's plausible story for an earned lesson. ## 1. Introduction The coding-harness program has, through components 01 through 14, assembled machinery that reads a repository, decomposes a request, generates and lands a patch, runs it under deterministic execution, records the run with instrumentation, reads the failure like evidence, and — in component 14 — turns that diagnosis into a working fix by looping the failed candidate toward passing. Component 15 stands where that loop ends and asks the question every loop must ask the moment it stops: the failure is resolved, the tests are green, the patch is landed — but what did the harness actually learn, and is it worth carrying forward? The naive answer is to let the resolved failure evaporate: the patch is fixed, the ledger (component 14) records that this signature was resolved, and the harness moves on, having learned nothing that will help next time it meets a structurally similar problem in a different repository. This paper argues that the evaporation is the missed opportunity, and that the moment a failure resolves is precisely when the harness is best positioned to distill it into something reusable — before the specifics of the bug fade, while the failed and successful trajectories are both in view, and while the evidence that separated them can still be cited. Reflection after failure is not a nicety bolted onto the repair loop; it is how a harness that repairs a thousand one-off bugs becomes one that stops making the same class of mistake, and it is the memory-admission stage without which components 16 and 17 — verification and cross-session memory — would be consolidating and persisting an understanding the harness never actually built. The thesis of this paper is stated plainly. **Reflection after failure should be an evidence-grounded experience-distillation stage, not introspection: after an outcome is known, identify the mistaken assumption, overlooked evidence, violated invariant, or failed strategy that causally contributed to it; contrast failed and successful trajectories where possible; convert supported explanations into scoped reusable lessons; reject speculative reflections; store admitted lessons with provenance, confidence, applicability and outcome statistics; and continually revise or delete them as subsequent experience provides supporting or contradictory evidence.** And it carries a strong rule for Piper that governs this component's every design choice, stated verbatim: **A failure may propose a lesson, but only evidence earns that lesson a place in memory.** The rule is the whole component compressed into one sentence of discipline. "Propose" is the key word — it grants the model the right to hypothesize, to offer its candidate explanation of why the failure happened, because a model that is never asked why will never surface the assumption that quietly went wrong. But "only evidence earns" is the load-bearing half: a proposal is a hypothesis, and a hypothesis has no place in memory until something outside the model's own confidence supports it. The model's story about its failure — however plausible, however well-articulated — is not evidence that the story is true; it is data to be checked. The rule makes the memory-admission gate the arbiter of what the harness believes about its own past, and it exists because the entire reflection literature, read honestly, is a warning that the model's post-hoc self-narrative is the least trustworthy source in the pipeline. The weak-model problem makes this stage both more necessary and more treacherous. A weak model fails often, so it has abundant raw material to learn from, but it also produces the most confabulated self-explanations, precisely because its failures are more often genuine reasoning errors whose true cause it cannot introspect. The literature supplies a double-sided frame. On the positive side, learning from one's own failures is real and powerful: Reflexion lifts GPT-4 from ~80% to 91% pass@1 on HumanEval with no weight updates, purely from verbal reinforcement of past mistakes [1]; ReasoningBank reports roughly a 20% relative improvement ceiling from induction over failures, ~16% fewer steps, and WebArena-Shopping rising 46.5 to 49.7 when failures feed the induction [2]; SWE-Exp reaches 73.0% Pass@1 on SWE-bench Verified by extracting multi-level experience [3]; ExpeL distills insights across tasks from successes and failures [4]. On the negative side, the same literature warns that the model's reflection is not to be trusted on its say-so: LLMs frequently cannot self-correct reasoning and intrinsic self-correction can even make it worse [7]; stated reasoning is often unfaithful to the true drivers of the answer [8]; and ExpeL's ablation finds that model-generated reflections can actively HURT insight extraction, because hallucinated reflections mislead the distillation [4]. The design implication is decisive and it is the thesis: the model's candidate explanation is a proposal that must earn admission through deterministic grounding against observed evidence, never a belief stored because it was articulated confidently. The paper proceeds as follows. Section 2 reviews the reflection literature in fifteen parts — the verbal-reinforcement anchor, induction from failures, multi-level experience extraction, cross-task insight distillation, mistake-correction-to-finetune, the self-refine pattern, the two negative results that discipline the whole design, honourable mentions, and the classical self-explanation roots. Section 3 derives seven design principles. Section 4 specifies the architecture, labeling every node deterministic (D) or learned (L) and stating the invariant that the L column holds exactly one entry. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides into fifteen parts — the verbal-reinforcement anchor, induction from failures, multi-level experience extraction, cross-task insight distillation, mistake-correction-to-finetune, the self-refine pattern, and the two negative results that discipline the whole design (models cannot reliably self-correct; stated reasoning is unfaithful), plus the experience-graph and survey mentions, and the classical self-explanation root beneath them all. Each answers one question the reflection stage must resolve: does verbalized self-reflection actually improve performance, how should failure-derived knowledge be stored and induced over, at what granularity should experience be extracted, can insights transfer across tasks, does fine-tuning on corrected mistakes help, is the plain self-refine loop enough, why must the harness refuse to trust introspective explanations, what do the recent experience-memory tools add, and what classical result makes connecting an explanation to an underlying principle the mechanism of learning. ### 2.1 The anchor: Reflexion **Reflexion** (Shinn, Cassano, Berman, Gopinath, Narasimhan, and Yao, NeurIPS 2023, arXiv:2303.11366) is the anchor because its mechanism is this component's thesis in its simplest form: instead of updating weights after a failure, the agent writes a verbal self-reflection in prose — what it did wrong and why — stores it in episodic memory, and retrieves it on the next attempt as context. The headline is the strongest single number in the reflection literature: 91% pass@1 on HumanEval with GPT-4, up from roughly an 80% baseline, achieved purely through this verbal-reinforcement-without-weight-update loop [1]. Two things make Reflexion the anchor. First, it proves the central claim this component rests on — that a model's prose reflection on its own past failure, carried into the next attempt, is a genuine and large source of improvement, so the harness's choice of which reflections to carry forward is a choice about what it will learn. Second, Reflexion stores its reflections with no discriminating admission gate: whatever the model wrote is kept. This component's whole architecture — the evidence-grounding, the rejection of speculative reflections, the provenance-and-confidence schema — is precisely the admission machinery Reflexion, in its pure form, lacks, and that the negative results prove it needs. ### 2.2 ReasoningBank **ReasoningBank** (Ouyang et al., Google, ICLR 2026, arXiv:2509.25140) is the induction-over-failures result this component leans on for both its promise and its discipline. It frames long-horizon reasoning as an iterative reasoning-and-induction process in which the agent reasons, evaluates, and induces reusable reasoning from its own successes and failures, banking the distilled knowledge for future use. Two facts must be stated precisely. The honest headline is a ceiling of roughly 20% RELATIVE improvement from induction — NOT "34.2%," which was a misread of a SWE-bench baseline in an earlier treatment — together with roughly 16% fewer steps, and WebArena-Shopping rising from 46.5 to 49.7 specifically when failures are added to the induction [2]. It is evaluated on SWE-bench-VERIFIED. The transfer to this component is double. The ~20% relative ceiling sets an honest expectation for what failure-induction can buy — real, but bounded, so the harness must not over-promise on its reflection stage. And the WebArena-Shopping contrast — better when failures are fed into the induction than when only successes are — is direct empirical evidence for the thesis's claim that the FAILED trajectory, not just the successful one, is the raw material of learning, which is exactly why this component triggers reflection on failure terminations rather than only banking wins. ### 2.3 SWE-Exp **SWE-Exp** (Silin Chen et al., 2025, arXiv:2507.23361) contributes the multi-level experience-extraction result that answers this component's granularity question. It extracts experience at multiple levels from problem-solving logs — distinguishing problem-comprehension-level experience from code-change-level experience — rather than treating a whole episode as a single undifferentiated lesson. Its headline is 73.0% Pass@1 on SWE-bench Verified with Claude 4 Sonnet [3]. The granularity insight is the one this component adopts as P3 (calibrate the abstraction): a lesson extracted at the level of "read the contract before changing the call site" is reusable, while a lesson at the level of "line 42 was wrong" is not, and SWE-Exp's separation of comprehension-level from code-change-level experience is empirical evidence that the abstraction level at which experience is extracted determines whether it generalizes. A coding harness that reflects at the wrong level — too fine to transfer, or too vague to act on — has learned nothing no matter how many failures it distills, so the reflection schema and the scope field in Section 4 are the mechanism for enforcing the level that SWE-Exp shows works. ### 2.4 ExpeL **ExpeL** (Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang, AAAI 2024, arXiv:2308.10144) contributes the cross-task insight-distillation result, and its ablation warning is the most important caution in this review. ExpeL collects the agent's experiences across tasks — both successes AND failures — and distills them into insights that transfer to unseen tasks. But its ablation delivers the warning that disciplines this entire design: model-generated reflections can actively HURT insight extraction, because when the model is asked to reflect and the reflections are hallucinated — plausible but grounded in nothing the agent actually did — they mislead the downstream distillation rather than helping [4]. This is the negative result, sitting inside a positive paper, that justifies the strong Piper rule verbatim. ExpeL's successes are successes of distillation over real experience; its ablation failure is the failure of trusting the model's generated reflection on its own say-so. The reflection stage therefore never asks the weak model to produce the lesson directly and store it — it asks the model to propose a candidate explanation and runs it through deterministic evidence-grounding that checks the explanation against what actually happened, rejecting it if the story does not match the observed failure. ### 2.5 LeMa **LeMa (Learning From Mistakes)** (An et al., arXiv preprint, arXiv:2310.20689) contributes the mistake-correction-to-finetuning path, and it carries a provenance discipline that this component must honor exactly. Two corrections to earlier treatments belong in the same breath. The source once cited the wrong arXiv ID — 2310.19835, which is actually a chest-X-ray paper and must never be cited as LeMa; the correct ID is 2310.20689. And the claim of a "+6–13 point HumanEval gain" does not exist in the paper — LeMa is a math-and-commonsense result, evaluated on GSM8K, MATH, SVAMP, ASDiv, and CSQA, with gains of roughly +0.3 to +4.6 points depending on model and task [5]. The mechanism is what transfers: LeMa generates reasoning traces for problems the model got wrong, reasons through why the answer was incorrect and how to correct it, and fine-tunes on the corrected reasoning, so that the model learns from its own mistakes by being trained on mistake-correction rather than merely being told about them at inference time. For this component, LeMa is the evidence that failure-derived corrections are worth banking as durable knowledge, and its honest (much smaller than once claimed) gains are a calibration against over-promising: mistake-correction helps, but modestly, so the harness should expect its reflection stage to pay off in reduced repeat-failures over time rather than in dramatic single-task jumps. ### 2.6 Self-Refine **Self-Refine** (Madaan et al., NeurIPS 2023, arXiv:2303.17651) is the origin of the plain self-feedback-and-refine pattern and serves this component as both ancestor and caution. It reports roughly 20% absolute average improvement across 7 tasks from a generate-feedback-refine loop with intrinsic self-feedback and no oracle, and on code specifically it reports up to 13% — though that figure is specific to Codex and code-davinci-002, and for GPT-4 the code-optimization improvement is 8.7% [6]. Self-Refine is the ancestor that Reflexion and every later reflective loop refines, and it is the reason "ask the model to reflect and refine" is the tempting baseline. But this component does not adopt the plain self-refine pattern as its design, and the reason is the pairing of Self-Refine's ceiling with the two negative results below [7][8]: self-refine works when the model can correctly identify its own error, and fails precisely when it cannot — and the evidence is that it frequently cannot. The reflection-after-failure stage therefore does not loop the model on its own candidate until it is satisfied; it runs the candidate through deterministic grounding that decides, from evidence outside the model's confidence, whether the reflection is worth keeping at all. ### 2.7 The two negative results The two negative results are the load-bearing walls of this design, and they are reviewed together because together they are decisive. **Large Language Models Cannot Self-Correct Reasoning Yet** (Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Yu, Xinying Song, and Denny Zhou, ICLR 2024, arXiv:2310.01798): intrinsic self-correction frequently fails to improve performance and can make reasoning WORSE [7]. The mechanism is that when the error stems from the same reasoning limitation that produced it, the model cannot see it from inside, so its 'correction' replaces a wrong answer with another wrong answer. The second wall is **Reasoning Models Don't Always Say What They Think** (Yanda Chen et al., Anthropic, 2025, arXiv:2505.05410): the stated reasoning of a reasoning model is frequently unfaithful to the factors that actually drive the answer [8]. Taken together the two walls are the empirical foundation of the strong Piper rule. If models cannot reliably self-correct [7], reflection must not be expected to fix the current failure — only to inform future ones. And if stated reasoning is often unfaithful [8], the model's candidate explanation is exactly the kind of plausible-but-not-causal story that must not be admitted on its say-so. Both walls point to the same architecture: the model proposes, and deterministic grounding disposes. ### 2.8 Honourable mentions The surrounding experience-memory lineage contributes tools and surveys without anchoring the component. **Experience Memory Graph** (Wang et al., UESTC, KDD 2027, arXiv:2607.13884) builds a graph memory of per-task intra-task nodes joined by cross-task edges through query-embedding similarity (not literally 'state-indexed'), demonstrating one-shot error correction by retrieving the closest prior experience when a task fails [9]; it is a related-work map situating R2D2 step-level reflection, trajectory-graph GNNs, and the AgentDebug taxonomy. **AgentDebug** (Zhu et al., 2025, arXiv:2509.25370) — real title 'Where LLM Agents Fail...', the framework name not being the paper title — contributes a large-scale failure taxonomy and bench of general agent errors, not coding-specific ones [10]. **When Can LLMs Actually Correct Their Own Mistakes?** (Kamoi et al., TACL 2024, arXiv:2406.01297) is the critical survey of the conditions under which self-correction is trustworthy [11]. **Agent Workflow Memory** (Wang, Mao, Fried, and Neubig, 2024, arXiv:2409.07429) mines SUCCESS-ONLY workflows — the baseline the failure-feeding results [2] argue against, which this component rejects because it throws away the failed trajectories that carry the most information [12]. **ParamMem** (Yao et al., February 2026 preprint, arXiv:2602.23320) reports that reflection DIVERSITY correlates with success and encodes reflection patterns into a parametric memory module [13]. **Beyond Final Code** (Chen, Ma, and Jiang, ICSE 2026, arXiv:2503.12374) is the process-oriented error analysis of software-development agents, examining errors along the way rather than only the final patch — the whole posture of this component, that the journey's failures are the learning material [14]. ### 2.9 Classical roots Beneath all of these sits the self-explanation lineage of Chi et al., and this component's posture — that explaining a failure in terms of an underlying principle, not restating it, produces learning — is Chi made into a memory-admission discipline. Chi's Cognitive Science work is not on arXiv and must be cited by venue. The 1989 correlational study, 'Self-explanations: How students study and use examples in learning to solve problems,' Cognitive Science 13(2):145–182, found GOOD self-explainers — who connected steps to underlying principles — outperformed POOR explainers who merely restated the example. The 1994 follow-up, 'Eliciting self-explanations improves understanding,' Cognitive Science 18(3):439–477, supplied the CAUSAL manipulation: eliciting self-explanation improved understanding, establishing the act causes learning, not merely correlates with it. The transfer is precise and carries the schema's sharpest constraint: self-explanation that CONNECTS a step to an underlying principle improves learning, while SHALLOW RESTATEMENT does not. 'I failed because I wrote the wrong function call' is shallow restatement — it names the surface event without connecting it to a principle, and Chi's result says it teaches nothing reusable. 'I assumed the contract matched the docs but the observed exception showed otherwise — I must verify assumptions against observed behavior' connects the failure to a principle, and that is the only kind this component's calibration stage (P3) admits. ## 3. Design Principles Seven principles follow from the review, each tied to the evidence above and each a clause of the strong Piper rule made mechanical. **P1 — Trigger reflection on every failure termination.** Reflection is a scheduled artifact of the repair loop, not an optional introspection the model performs when it feels like it. Every time a failure terminates — whether resolved by component 14's repair or escalated as unresolvable — the harness runs the reflection stage on that failure. This is what makes the failed trajectory the raw material rather than an afterthought: ReasoningBank's WebArena-Shopping contrast (46.5 to 49.7 when failures are added to induction) is the empirical warrant that the failed trajectories carry information the successes alone do not [2], and SWE-Exp's process-orientation is the broader statement that the journey's errors are where the learning lives [14]. A failure that resolves and is not reflected upon is a lesson thrown away; the trigger is what prevents the evaporation this paper's introduction named as the naive failure mode. **P2 — Enforce the schema: no observed failure cited, no acceptance.** The model's reflection must conform to a fixed schema that forces the causal chain to be explicit: an observed failure must be cited, a belief-level root cause identified, an operative lesson stated, and a confidence assigned. The chain is enforced because it is the discipline that turns a monologue into an explanation. The Huang et al. result that self-correction frequently fails [7] and the Yanda Chen et al. result that stated reasoning is unfaithful [8] together imply the model cannot be trusted to volunteer a sound chain unprompted — it must be forced to name the failure it is explaining, state the cause it believes, derive the lesson it claims, and attach a confidence, so that each link can be checked. A reflection that omits any link — no cited failure, no root cause, no operative lesson — is rejected by the admission gate, because a chain with a missing link is a chain that cannot be grounded. **P3 — Calibrate the abstraction level.** The reflection must sit one level above the instance — boundary assumptions, protocol ordering, contract misunderstandings — and must reject reflections that are either too fine (line-number or token-level restatements) or too vague (virtue-level platitudes). SWE-Exp's separation of problem-comprehension from code-change-level experience is the empirical evidence that extraction granularity determines generalizability [3]; Chi's classical result is the sharpest statement, that self-explanation connecting a step to an underlying principle improves learning while shallow restatement does not [15]. A reflection at the line level is shallow restatement — it names the surface event and teaches nothing reusable; a reflection at the virtue level ("be more careful") is too vague to act on; the calibrated level — "I assumed the contract matched the docs; verify assumptions against observed behavior" — is the only one that transfers to a similar problem in a different repository. **P4 — Persist outside the window; retrieve boundedly.** Admitted lessons live in a durable store — not in the model's context window — and are retrieved as a bounded, ranked subset when a new task resembles their scope. Reflexion's episodic memory is the ancestor: verbal reinforcement only improves future attempts if the reflection is carried forward [1], and Reflexion stores in episodic memory precisely so it survives past the window in which the failure happened. The bounded, ranked retrieval is what makes the store usable at scale: the harness cannot stuff every past lesson into the window, so it must rank lessons by their scope's match to the current task and retrieve only the top subset, so that the most applicable lessons are present without flooding the context with irrelevant history. **P5 — Promote and demote.** Lessons that recur — that prove applicable again and again — are promoted from episodic (repo/task-specific) to procedural (cross-repo) to meta-strategy (class-of-problem) level, while lessons that are falsified by subsequent experience are demoted or retired. This is the continual-revision clause of the thesis made mechanical: a store that never revises is a store that accumulates dead weight, and the promotion/demotion machinery is what keeps the store a live model of what actually works rather than a fossil of what once seemed true. ReasoningBank's induction [2] and ParamMem's finding that reflection diversity correlates with success [13] both support treating the store as something to actively grow and prune, not merely append to. **P6 — Dedupe against the failure-signature ledger.** Component 14's ledger records WHAT failed (the failure signature — class, message template, site) and what resolved it; component 15's reflection records WHY (the causal account of that signature). The two must be cross-referenced so a reflection is never stored in isolation from the failure it explains and so the same failure signature does not accumulate duplicate reflections. The reflection is keyed to the ledger's signature — reflection = why, keyed to the ledger's what — so that when a signature recurs, the harness retrieves both the ledger's record of what was tried and the reflection's account of why it failed, and so that a newly-proposed reflection on an already-reflected signature can be checked against the existing one for corroboration or contradiction rather than blindly appended. **P7 — Distill downstream.** Verified lessons — reflections that survived grounding and have accrued positive outcome statistics — are harvested into the fine-tuning and evaluation corpus, so that the reflection stage's products feed the model's training rather than only its in-context behavior. LeMa's mistake-correction-to-finetune is the direct ancestor: corrections derived from the model's own mistakes, when trained on, durably improve it [5]. The harvest is the program-wide recursive-self-improvement link (Section 5): the harness's failures, distilled into verified lessons, become training data that makes the model fail less — which produces fewer, better failures — which distills into sharper lessons, closing the loop between reflection at inference time and improvement at training time. ## 4. The Architecture The reflection stage is a sequence of stages; every node is labeled D (deterministic software) or L (a learned model). The invariant that governs the whole is: the L column contains exactly one entry, and every upstream and downstream step is deterministic software on CPU — a single weak-LLM candidate-reflection-emission call, with evidence collection, grounding, generalization, dedupe, admission, promotion/demotion, retrieval, and outcome bookkeeping all deterministic. The concrete inputs are the failed patch, the failure evidence from components 11–13, the repair outcome from component 14 (resolved or escalated), and the failure-signature ledger entry; the output is a set of admitted, evidence-grounded lessons in the durable store. The flow, in the deterministic stages (a) through (i) plus the one learned entry (L): 1. **(a) Evidence collection [D].** The failure's full evidentiary record is assembled deterministically: the execution output and traces from components 11–12, the diagnosis and its citations from component 13, the repair outcome from component 14, and — critically, for the contrast clause of the thesis — any SUCCESSFUL trajectory of the same or a closely related task, when one exists. The evidence record is the substrate every later stage grounds against, and the deterministic assembly of a complete record — rather than asking the model what it remembers about the failure — is the first defense against the unfaithful-reasoning problem [8], because the model never gets to choose which facts are in view; they all are. 2. **(b) Failure-signature ledger lookup [D].** The current failure is hashed into the same signature used by component 14's ledger — failure class, message template, site — and looked up. The lookup returns whether this signature has been seen and reflected upon before, so the stage knows whether it is distilling a first occurrence or producing a duplicate reflection on a known signature (P6). The reflection being produced is keyed to this signature, so the ledger's WHAT and the reflection's WHY are joined; a signature with an existing reflection routes the new candidate into a corroborate-or-contradict check against the existing lesson rather than a blind append. The lookup is a deterministic hash-and-index; the model never reasons about whether it has explained this failure before. 3. **(L) Candidate reflection emission [L] — the single learned entry.** The weak model, seeded with the evidence record and the ledger context, emits ONE candidate reflection conforming exactly to the typed `Reflection` schema: `{ outcome, failed_assumption, overlooked_evidence, strategy_error, violated_invariant, evidence_ids[], counterfactual, candidate_lesson, scope, confidence }`. The schema forces the causal chain P2 demands: the model must name the outcome it is explaining, identify which of the four causal categories (mistaken assumption, overlooked evidence, failed strategy, violated invariant) it believes contributed, cite the evidence_ids that support that belief, state a counterfactual (what would have had to differ for the outcome to change), propose a candidate_lesson at the calibrated abstraction level (P3), declare the lesson's scope, and assign a confidence. Crucially, the model emits only a CANDIDATE explanation — it is proposing, not admitting. Every judgment about whether the candidate's claims are grounded in the cited evidence, whether the lesson generalizes, and whether it earns a place in memory is deterministic. This is the one place in the pipeline where a language model forms a hypothesis about why something happened; the harness decides what the hypothesis is worth. 4. **(c) Evidence grounding [D].** The candidate is grounded deterministically: each claim in the reflection — the failed_assumption, the overlooked_evidence, the strategy_error, the violated_invariant — is checked against the evidence_ids it cites. Is the claimed assumption actually present in the plan or patch that failed? Is the claimed overlooked evidence actually in the record? Does the cited evidence support the stated cause, or does the model's story contradict what the evidence shows? The grounding stage is the direct embodiment of the strong Piper rule — the model proposed, and now evidence earns or refuses. A reflection whose causal claims are not supported by its own cited evidence is rejected as speculative. This is where the model's plausible-but-unfaithful narratives [8] and its hallucinated reflections [4] are caught, and where ExpeL's ablation warning is answered: the reflection that misleads distillation in ExpeL is exactly the reflection this stage rejects. 5. **(d) Generalization check [D].** A grounded candidate is checked against the harness's other experiences deterministically: does this candidate lesson, at its declared scope, contradict or corroborate any already-admitted lesson or any prior failure that is inconsistent with it? A lesson claiming "contracts must always be verified against observed behavior" is checked against the store — have there been prior failures where contract verification would NOT have helped (a counterexample that narrows the scope) or where it would have (corroboration that widens it)? The generalization check is what prevents over-broad lessons that a single failure does not actually support, and it is the mechanism for the scoped clause of the thesis: a lesson's scope must match the evidence, and the check narrows or broadens it against the accumulated record rather than accepting the model's declared scope at face value. 6. **(e) Dedupe and calibration against the ledger [D].** The grounded candidate is deduped against existing reflections keyed to this signature (P6): if a corroborating reflection already exists, the candidate is merged into it (updating its evidence and outcome statistics) rather than stored as a duplicate; if the candidate contradicts an existing reflection, both are flagged for adjudication by future evidence rather than silently overwriting. The candidate is also calibrated to the abstraction level (P3): reflections that ground to line-level restatement are demoted to episodic or rejected as shallow; reflections at the virtue level are rejected as too vague to act on. Chi's distinction is enforced mechanically here: the reflection must connect the failure to an underlying principle, or it does not clear this stage [15]. 7. **(f) Memory-admission gate [D].** The gate decides, deterministically, whether the grounded, generalized, deduped candidate earns admission to the durable store — the enforcement of the strong Piper rule's "only evidence earns." A reflection that passed grounding (its causal claims are supported by its cited evidence), passed generalization (it does not contradict the accumulated record), and is calibrated (it sits at a transferable abstraction level) is admitted as a `Lesson` with the schema `{ statement, type, scope, derived_from: {failures[], successes[]}, evidence[], counterexamples[], confidence, times_applied, times_helped, times_hurt, supersedes, superseded_by }`. A reflection that failed grounding, or whose evidence is thin relative to its claimed scope, is rejected — stored nowhere, or demoted to an unattached candidate the harness may revisit if corroborating evidence later appears. The gate is the single choke point where a model's confident narrative becomes (or fails to become) a durable belief. 8. **(g) Promote/demote and lifecycle [D].** Admitted lessons enter the three-level store: episodic (repo/task-specific), procedural (cross-repo), or meta-strategy (class-of-problem). Over time, the outcome statistics (P5) drive deterministic promotion and demotion: a lesson that accrues times_helped across multiple repos rises from episodic to procedural to meta-strategy; a lesson that accrues times_hurt, or that a counterexample falsifies, is demoted or retired — its superseded_by pointer set, its edges cleaned. This is the continual-revise-or-delete clause of the thesis made mechanical. The blame-direction red-flag rule applies at this stage: lessons whose content is "the tests were wrong" or "the framework is broken" — blame directed outward rather than at the harness's own reasoning — are routed into a red-flag class that requires exceptional evidence (corroboration from multiple independent failures, a reproducible external defect) before they can be admitted at anything above the lowest confidence, because blaming the tests is the classic shape of a model explaining away its own error. 9. **(h) Bounded ranked retrieval [D].** When a new task arrives, the harness retrieves from the durable store a bounded, ranked subset of lessons whose declared scope matches the current task, using the same ranking discipline as the failure-signature ledger: the most applicable lessons first, a hard cap on how many enter the window (P4). Reflexion's episodic-memory retrieval is the ancestor — the improvement comes from carrying the right past reflection forward [1] — and the bounded ranking is what keeps the store usable at scale rather than flooding the window with all of history. 10. **(i) Outcome statistics [D].** Every time an admitted lesson is applied and the task resolves, the outcome is recorded deterministically against the lesson: times_applied and times_helped increment if the lesson's guidance was followed and the task succeeded, times_hurt increments if following the lesson led to a worse outcome. These statistics are the continuing evidence that the thesis's final clause demands — subsequent experience provides the supporting or contradictory evidence that promotes, demotes, or retires every lesson, so the store is never static but continually revised by the harness's actual outcomes. The primitive set is deterministic except where noted: `collect_evidence` (assemble the failure's full record), `lookup_reflection_signature` (hash against the ledger), `reflect` (the single learned call emitting the schema-bound candidate Reflection), `ground_reflection` (check causal claims against cited evidence), `generalize_check` (compare against the store and prior failures), `dedupe_reflection` (merge or flag against existing reflections and calibrate abstraction), `admission_gate` (decide whether the candidate earns a Lesson), `apply_lifecycle` (promote, demote, retire, red-flag), `retrieve_lessons` (bounded ranked retrieval by scope), and `record_lesson_outcome` (bookkeep times_applied/helped/hurt). The target is explicit and checkable: stages (a)–(i) are software on CPU, and the L column holds exactly one entry — the candidate-reflection-emission call that proposes why the failure happened. Any proposed addition that would put a language model elsewhere in the pipeline — having the model judge whether its own reflection is grounded, decide its own scope, or choose what to admit — is a regression against P1–P7 and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of Section 6, on held-out tasks, at a defensible cost, or be rejected. The invariant is stated again so it can be enforced mechanically: **"the L column contains exactly one entry."** ## 5. Recursive Self-Improvement Protocol Component 15 of 27 follows the program's uniform protocol: the executable specification is written to 100%, round 1 lands at approximately 80% of the specified behavior, and later rounds close the gap. Each round is A/B-gated against the honest metrics of Section 6, never the end-to-end resolve rate alone. The three-way split disciplines every change: semantic judgment goes to the LLM's single entry, mechanics to deterministic software, and every change to either is A/B-gated on honest metrics, never on vibes. Because reflection decides what the harness believes about its own past, its improvement protocol measures the quality of the store and the admission process rather than trusting introspective fluency, improving itself in four loops. First, *admission-gate tuning*: the harness measures admission precision (admitted lessons that help ÷ lessons admitted) and tightens grounding strictness so the gate admits fewer, better lessons — guided by ExpeL's ablation that ungrounded reflections mislead [4] and the two negative results that introspective explanations are unreliable [7][8]; a reflection stage that admits noise is worse than one that admits nothing. Second, *store-quality and promotion calibration*: the harness measures whether lessons promoted to meta-strategy actually help more across repositories and recalibrates the promotion thresholds against the per-lesson outcome ledger, guided by SWE-Exp's evidence that granularity determines generalizability [3]; the promotion rules are A/B-tested like any other component. Third, *retrieval-effectiveness tuning*: the harness measures whether bounded ranked retrieval surfaces the lessons that help and tunes scope-matching accordingly, guided by Reflexion's demonstration that carrying the right past reflection forward is where the gain lives [1]; a store of good lessons never retrieved at the right moment might as well be empty. Fourth, *downstream harvest*: the harness periodically harvests verified lessons — high times_helped, low times_hurt — into the fine-tuning and evaluation corpus (P7), guided by LeMa's demonstration that mistake-corrections improve the model when trained on [5]; each harvest closes the loop, making the model fail less, which yields sharper failures, which distill into better lessons. Because every admitted lesson carries provenance and is versioned through its lifecycle, a regression in the store is visible in the lesson ledger before it surfaces as a wrong repair, and each A/B change must justify itself on held-out tasks at a defensible cost before it ships. ## 6. Measurement & Evaluation Reflection after failure is measured *separately* and *honestly*, because a stage whose whole purpose is to decide what the harness believes about its own past can be defeated by the harness mistaking a model's plausible story for an earned lesson, or by a store that accumulates confident noise and is never caught. The instruments are built to keep the admission process and the store accountable to actual outcomes, not to the model's introspective fluency. **Lesson-admission precision.** The primary instrument reports the ratio of admitted lessons that subsequently prove helpful (times_helped over times_applied, tracked per lesson and aggregated) to admitted lessons that prove useless or harmful. This is the honest measure of the gate, and it is what makes the strong Piper rule measurable rather than rhetorical: a gate that admits everything the model proposes will show low precision, because ungrounded reflections and unfaithful narratives [7][8] will dominate; a gate that admits nothing will show perfect precision on an empty store. The instrument is the ground truth the admission-gate-tuning loop (Section 5) optimizes, and it is the direct quantitative answer to the question ExpeL's ablation raises about whether model reflections help or hurt [4]. **Speculative-reflection rejection rate.** The companion instrument measures the gate's discrimination: of the candidate reflections the model proposes, what fraction are rejected as speculative — their causal claims not supported by their cited evidence, their scope exceeding what the evidence supports. A healthy reflection stage rejects a substantial fraction, because a weak model's post-hoc self-explanations are frequently plausible-but-unfaithful [8], and a stage that rejects almost nothing is a stage that has made its gate a rubber stamp. The instrument is what prevents the reflection stage from quietly becoming the introspection the thesis forbids. **Evidence-grounding fidelity.** The instrument audits whether admitted lessons actually cite and are supported by the evidence they claim: for a sample of admitted lessons, is the failed_assumption actually present in the failed plan, is the overlooked_evidence actually in the record, does the cited evidence actually establish the claimed cause? This is the check against the model's two failure modes at once — the shallow restatement Chi shows teaches nothing [15] and the unfaithful narrative that does not reflect the true drivers [8]. A store whose lessons do not match their cited evidence is a store of confabulation, and the instrument is what surfaces it. **Per-lesson outcome ledger.** The bookkeeping instrument tracks each admitted lesson's life: times_applied, times_helped, times_hurt, counterexamples accrued, and its current promotion level. This is the mechanical heart of the thesis's continual-revise-or-delete clause: a lesson that accrues times_hurt or is falsified by a counterexample is demoted or retired, and the instrument reports the demotion and retirement rates as a sign of health rather than of failure — a store that never revises is a store that has stopped learning. It is also what separates a scoped lesson (which survives because it only claims what it can deliver) from an over-broad one (which accrues hurt and is pruned). **Promote/demote and retrieval effectiveness.** The learning instruments measure whether the store's structure serves the harness: do lessons promoted to procedural or meta-strategy actually help more across repositories than lessons left at episodic (SWE-Exp's granularity claim [3])? Do tasks whose bounded ranked retrieval included the right lesson resolve better than tasks whose retrieval did not (Reflexion's retrieval-is-where-the-gain-lives claim [1])? These instruments close the loop on P4 and P5, ensuring the store is not just correctly populated but correctly navigated, so a harness does not hold a store of good lessons it never retrieves at the right moment. **Downstream-harvest yield.** The final instrument measures what the store gives back to the model: of the lessons harvested into the fine-tuning and evaluation corpus (P7), what fraction demonstrably reduce the targeted failure class on held-out tasks? LeMa's mistake-correction-to-finetune [5] is the warrant that the harvest can improve the model, and the yield instrument is what verifies the improvement is real rather than assumed, closing the recursive self-improvement loop with evidence. A failure-mode audit closes the positive instruments by naming how a reflection stage lies. *Introspection-in-disguise* is when the stage asks the model what it learned and stores the answer without grounding — the failure the thesis forbids, caught by low admission precision; *confabulated-cause* is when an admitted lesson's stated cause does not match its cited evidence, caught by the grounding-fidelity instrument; *blame-deflection* is when lessons route their cause outward ('the tests were wrong') rather than at the harness's own reasoning — the red-flag class requiring exceptional evidence to clear; *shallow-restatement* is when lessons restate the failure at the line level and teach nothing reusable, caught by the calibration stage and Chi's distinction [15]; *unfalsifiable-store* is when lessons are never demoted regardless of outcome, caught by a near-zero demotion/retirement rate; and *retrieval-miss* is when the store holds the right lesson but bounded retrieval never surfaces it, caught by the retrieval-effectiveness instrument. Each mode has a guard. Two honest caveats close the section. First, the strong-model numbers of Section 2 — Reflexion's 91% [1], SWE-Exp's 73.0% [3], ReasoningBank's ~20% [2] — will NOT reproduce on a weak model; a weak-model harness should expect materially smaller absolute effects from reflection, which is precisely why its reflection stage leans on deterministic grounding, calibration, and the outcome ledger to compensate for weak introspective accuracy with strong evidence-admission discipline. Second, all reflection metrics must be reported on held-out and out-of-benchmark tasks, as in the earlier components, so a store that has memorized a task set's failures cannot score well without its lessons having been grounded in evidence the harness actually observed. ## 7. Conclusion Reflection after failure is the stage where the coding harness decides what a failed attempt is worth keeping, and it is where the program's most seductive failure mode lives — the model's confident, plausible post-hoc story about why it erred. This paper has argued that the seduction is the design error: introspection is cheap and almost always wrong, because LLMs cannot reliably self-correct [7], their stated reasoning is frequently unfaithful to the true drivers of their answers [8], and ungrounded model reflections actively mislead distillation [4]. The harness must therefore not ask the model to feel its way to wisdom; reflection must be a deterministic pipeline that admits to memory only the explanations that survived grounding. The thesis that has governed every design choice is the one with which this paper began and closes: **Reflection after failure should be an evidence-grounded experience-distillation stage, not introspection: after an outcome is known, identify the mistaken assumption, overlooked evidence, violated invariant, or failed strategy that causally contributed to it; contrast failed and successful trajectories where possible; convert supported explanations into scoped reusable lessons; reject speculative reflections; store admitted lessons with provenance, confidence, applicability and outcome statistics; and continually revise or delete them as subsequent experience provides supporting or contradictory evidence.** And the strong rule that compresses that thesis into one sentence of discipline for Piper is: **A failure may propose a lesson, but only evidence earns that lesson a place in memory.** The model proposes — it emits one schema-bound candidate explanation of why the failure happened, naming the causal category, citing its evidence, stating its counterfactual, declaring its scope, and assigning a confidence. Evidence disposes — deterministic grounding checks the proposal's causal claims against the evidence it cites, the generalization check compares it against the store and prior failures, the admission gate decides whether it earns a Lesson, the lifecycle machinery promotes or demotes or retires it as outcomes accrue, and bounded retrieval surfaces the right lessons at the right moment. The architecture is deterministic end to end with exactly one learned entry. The failure-signature ledger of component 14 records WHAT failed; this component records WHY, keyed to that what, one abstraction level above the candidate loop. Retrying is cheap — a failed patch can be repaired a dozen times for pocket change. Learning is the deliverable, and it is not delivered by the model narrating its own failure; it is delivered by a harness that grounds every lesson in evidence, admits only what that evidence supports, and revises or retires every belief as its own subsequent experience confirms or refutes it. Because for a weak model, the only reflection worth keeping is the one that survived being checked against what actually happened — and the only harness that learns from its failures is the one that refuses to trust its own voice about them. --- ## References 1. Shinn, Cassano, Berman, Gopinath, Narasimhan, and Yao. *Reflexion.* NeurIPS 2023. arXiv:2303.11366. *(ANCHOR. Verbal self-reflection in prose stored in episodic memory, retrieved next attempt; no weight updates. 91% pass@1 HumanEval (GPT-4) from ~80% baseline.)* 2. Ouyang et al. (Google). *ReasoningBank.* ICLR 2026. arXiv:2509.25140. *(Iterative reasoning-and-induction; ~20% RELATIVE improvement ceiling — NOT "34.2%", which was a SWE-bench baseline misread; ~16% fewer steps; WebArena-Shopping 46.5→49.7 when failures added to induction; evaluated on SWE-bench-VERIFIED.)* 3. Silin Chen et al. *SWE-Exp.* 2025. arXiv:2507.23361. *(Multi-level experience extraction — problem-comprehension vs code-change level; 73.0% Pass@1 on SWE-bench Verified with Claude 4 Sonnet.)* 4. Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. *ExpeL.* AAAI 2024. arXiv:2308.10144. *(Cross-task insight distillation from successes + failures. ABLATION: model-generated reflections can HURT insight extraction — hallucinated reflections mislead.)* 5. An et al. *LeMa (Learning From Mistakes).* arXiv preprint. arXiv:2310.20689. *(Mistake-correction reasoning → fine-tune. NOTE: the earlier source cited the WRONG ID 2310.19835 (a chest-X-ray paper); this is correct. The "+6–13 HumanEval" figure does NOT exist — LeMa is math+commonsense: GSM8K/MATH/SVAMP/ASDiv/CSQA, gains ~+0.3 to +4.6 pts by model/task.)* 6. Madaan et al. *Self-Refine.* NeurIPS 2023. arXiv:2303.17651. *(Generate-feedback-refine with intrinsic feedback, no oracle; ~20% absolute average across 7 tasks; up to 13% on code (Codex/code-davinci-002-specific; GPT-4 code-opt is 8.7%).)* 7. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Yu, Xinying Song, and Denny Zhou. *Large Language Models Cannot Self-Correct Reasoning Yet.* ICLR 2024. arXiv:2310.01798. *(NEGATIVE RESULT. Intrinsic self-correction frequently fails to improve and can make reasoning WORSE.)* 8. Yanda Chen et al. (Anthropic). *Reasoning Models Don't Always Say What They Think.* 2025. arXiv:2505.05410. *(NEGATIVE RESULT. Stated reasoning is frequently unfaithful to the factors driving the answer — plausible-but-not-causal explanations.)* 9. Wang et al. (UESTC). *Experience Memory Graph.* KDD 2027. arXiv:2607.13884. *(Graph memory of per-task intra-task nodes + cross-task edges joined by query-embedding similarity — NOT literally "state-indexed". One-shot error correction. Related-work map: R2D2 step-level reflection, trajectory-graph GNNs, AgentDebug taxonomy.)* 10. Zhu et al. *AgentDebug.* 2025. arXiv:2509.25370. *(Real title "Where LLM Agents Fail..." — the framework name AgentDebug is not the paper title. Large-scale failure taxonomy + bench; general agent errors, not coding-specific.)* 11. Kamoi et al. *When Can LLMs Actually Correct Their Own Mistakes?* TACL 2024. arXiv:2406.01297. *(Critical survey of the conditions under which self-correction is trustworthy.)* 12. Wang, Mao, Fried, and Neubig. *Agent Workflow Memory.* 2024. arXiv:2409.07429. *(Success-only workflow mining — the baseline later failure-feeding work argues against.)* 13. Yao et al. *ParamMem.* February 2026 preprint. arXiv:2602.23320. *(Reflection DIVERSITY correlates with success; encodes reflection patterns into a parametric memory module. NOT "ICML 2025" — that was wrong.)* 14. Chen, Ma, and Jiang. *Beyond Final Code.* ICSE 2026. arXiv:2503.12374. *(Process-oriented error analysis of software-development agents. NOTE: the earlier label "Unveiling Pitfalls" was FABRICATED; this is the real title.)* 15. Chi et al. *Self-explanation.* Cognitive Science. Not on arXiv. 1989: Cognitive Science 13(2):145–182, correlational (good vs poor explainers). 1994 follow-up: "Eliciting self-explanations improves understanding," Cognitive Science 18(3):439–477, the CAUSAL manipulation. *(Self-explanation that connects steps to underlying principles improves learning; shallow restatement does not.)*