Summary / 10. Analogical reasoning / Document
2026-10-04
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 10 — Analogical Reasoning: Research Review & v2 Engineering Design Oct 3, 2026 · @Bink ## 1. Executive summary **Bottom line:** Piper's analogy problem is a retrieval-encoding problem first and a reasoning problem second. Fifty years of cognitive science say the same thing: people (and similarity-indexed memories) retrieve by surface features but map by structure, so deep analogues sit unused in memory. The fix with the strongest evidence is to store and query a *structural* re-description of each important episode, then force an explicit, checkable mapping before anything transfers. Deliberate blending is the weakest-evidenced part of the current design and should be demoted. Evidence labels used throughout: **\[E\]** established finding with replicated or peer-reviewed support; **\[X\]** reasonable engineering extrapolation from established findings; **\[S\]** my own speculative proposal, untested. Key conclusions: - **\[E\] Retrieval and mapping are different processes with different drivers.** Surface similarity dominates what gets *reminded*; relational structure dominates what is judged sound and what transfers (Gentner, Rattermann & Forbus 1993; Holyoak & Koh 1987). A single embedding-similarity retriever reproduces the human failure, not the human strength. - **\[E\] Cues and comparison work.** A one-line hint to use a prior case raises transfer sharply (Gick & Holyoak 1980). Comparing two analogues side by side induces a schema that transfers better than studying either alone (Gick & Holyoak 1983; Gentner, Loewenstein & Thompson 2003; meta-analysis by Alfieri, Nokes-Malach & Schunn 2013). - **\[E\] Frontier LLMs show real but brittle analogical competence.** Strong zero-shot results (Webb, Holyoak & Lu 2023) degrade on counterfactual or unfamiliar variants (Lewis & Mitchell 2024), and long-context analogue *retrieval* is markedly weaker than short-context mapping (AnaloBench, Ye et al. 2024). Bare-LLM performance is therefore a serious baseline, and any benchmark that does not control for familiarity will overstate both the model and the machinery. - **\[X\] The right architecture is MAC/FAC-shaped:** a cheap, wide, structure-aware retrieval stage, then an expensive, narrow, explicit mapping stage with a disanalogy check, then transfer only of the mapped mechanism, then verification. This is supported by Forbus, Gentner & Law (1995) and by case-based reasoning practice (Aamodt & Plaza 1994), but its value *for an LLM agent* is untested. - **\[X\] Representation:** store, per significant episode, a typed failure/solution signature (roles, relations, causal chain, violated assumption, fix-as-operator) at three abstraction levels, embedded separately from the surface narrative and also hashed into a small relational vocabulary for exact matching. - **\[S\] Component verdicts:** retire `associate` (zero callers, no evidence of need); fold `reuse-search` into a new `analog-retrieve` path; keep `blend-worker` only as an offline, sandboxed generator whose outputs must survive the same mapping-and-verification gate as everything else, and stop describing it as part of live cognition until it does. - **Minimal experiment (days, not weeks):** signature-index Piper's last \~100–200 resolved failures, build 40–60 paired items with structural targets and surface decoys (half with randomized labels), and run Bare LLM vs semantic retrieval vs signature retrieval vs signature + forced mapping. Keep the machinery only if the signature arms beat semantic retrieval on *transfer accuracy* and *decoy rejection* by a pre-registered margin, not merely on retrieval recall. The instrument matters more than the mechanism: the benchmark in sections 14–16 is designed so that a null result is possible and informative. ## 2. Operational definition of analogical reasoning **Proposed definition \[X\], grounded in Structure-Mapping Theory (Gentner 1983):** Piper performs analogical reasoning on a target problem T when she (1) brings in a source case S that differs from T in surface content, (2) states an explicit one-to-one correspondence M between roles in S and roles in T, (3) the correspondence preserves at least one higher-order relation (typically causal: *X causes Y*, *X prevents Y*, *X is trusted in place of Y*), (4) she projects a candidate inference from S to T through M (a cause, a fix, a prediction), and (5) that inference changes what she does in T and is better than what she would have done without S. Condition 5 is what makes this a benchmarkable capability rather than a style of talk. An analogy that is mapped correctly but changes nothing is decoration. ### Neighbouring phenomena and what Piper should count | Phenomenon | What it is | Count as success? | | --- | --- | --- | | Semantic similarity | Shared topic, words, entities | No. This is the decoy signal. | | Episodic retrieval | Recalling a specific past event | Only as Stage B input; not analogy unless mapped. | | Pattern matching | Recognising an instance of a known template | Partial. Counts as schema application, scored separately. | | Metaphor | Expressive cross-domain description | No, unless it yields a testable inference. | | Conceptual blending | Fusing two input spaces into a novel emergent space (Fauconnier & Turner 2002) | Only if its emergent inference survives verification. Otherwise generation, not reasoning. | | Abstraction | Dropping specifics to widen scope | Enabling step; not success by itself. | | Schema induction | Extracting a reusable relational template from 2+ cases | Yes, as a learning outcome (scored by later transfer). | | Transfer learning (ML sense) | Weight or representation reuse across tasks | Out of scope; no weights change. | | Case-based reasoning | Retrieve, reuse, revise, retain a prior case (Aamodt & Plaza 1994) | Yes when source and target differ in surface; *near* CBR (same domain) scored as a separate, easier tier. | | Structural analogy | Relational alignment with candidate inference | Yes. Core target. | | Causal analogy | Structural analogy where the preserved relations are causal | Yes. The most valuable tier for debugging and planning. | ### Distinguishing "the LLM already knew this" from discovery This is the central measurement problem. Four complementary controls, strongest first: 1. **Private-source test \[X\].** Use sources drawn from Piper's own post-training history (her incidents, logs, fixes). The base model cannot have memorised the pairing. If performance on the target rises only when the private source is retrievable, the analogy machinery is doing work. 2. **Source-dependence delta \[S\].** Run each item with the correct source, with no source, and with a structurally different source of matched surface similarity. Define source dependence as accuracy(correct source) minus accuracy(no source). A system that "already knew" the answer shows a delta near zero. A system that is merely primed by any source shows equal gains from the wrong source. 3. **Label randomisation and counterfactual worlds \[E for the method\].** Replace entity and relation names with arbitrary tokens while preserving graph topology. Lewis & Mitchell (2024) used permuted alphabets for exactly this purpose; human accuracy stayed high while GPT models dropped. Note the caveat from Webb, Holyoak & Lu (2024): part of that drop traced to a counting difficulty, not to analogy. Randomisation must not add unrelated task demands. 4. **Mapping-lesion test \[S\].** Delete one relation in the source that the mapping claims is load-bearing. If the transferred conclusion does not change, the stated mapping was not what produced it (a post-hoc rationalisation). Famous analogies (atom as solar system, Duncker's radiation problem, ecosystem as market) should be excluded from scoring entirely. They measure recall, not reasoning. ## 3. Human cognitive-science findings **The human literature points to one design rule: encode experiences relationally and cue retrieval relationally, because reasoning is rarely the bottleneck; access is.** People who are told a relevant source exists usually solve the target; people who must notice it themselves mostly do not. | Finding | Key evidence | Strength | Engineering translation | | --- | --- | --- | --- | | Spontaneous cross-domain transfer is rare; a hint to use the prior case raises it sharply | Gick & Holyoak 1980 (radiation/fortress problems): a minority transfer spontaneously, most do once told the story is relevant | \[E\] widely replicated | Analogy must be an explicit, triggered operation, not a hoped-for side effect | | Retrieval is driven by surface similarity; mapping and soundness judgements by relational structure | Gentner, Rattermann & Forbus 1993; Holyoak & Koh 1987 | \[E\], with a disagreement (below) | Separate retrieval and mapping stages; do not let the retriever's similarity score decide soundness | | Comparing two analogues induces a schema that transfers better than studying either alone | Gick & Holyoak 1983; Catrambone & Holyoak 1989; Gentner, Loewenstein & Thompson 2003; meta-analysis Alfieri, Nokes-Malach & Schunn 2013 (moderate pooled effect, roughly d ≈ 0.5) | \[E\] | Schema induction by pairwise comparison during consolidation | | Relational encoding at study fixes later retrieval ("inert knowledge") | Gentner, Loewenstein, Thompson & Forbus 2009 | \[E\] | Write a structural signature when an episode is stored, not only when an analogy is requested | | Systematicity: people prefer mappings with connected higher-order relations and project inferences from that connected structure | Gentner & Toupin 1986; Clement & Gentner 1991 | \[E\] | Score mappings by connected causal structure, not by count of matched facts | | Expertise shifts salience from surface to principle | Chi, Feltovich & Glaser 1981; Novick 1988 (experts show positive transfer and resist surface-driven negative transfer) | \[E\] | Piper's "expertise" is her relational vocabulary; grow it from her own failures | | Progressive alignment: close comparisons first make far comparisons possible | Kotovsky & Gentner 1996 | \[E\] in children; \[X\] for agents | Retrieve near analogues first, then use their shared schema to search far | | Relational complexity limits mapping | Halford, Wilson & Phillips 1998 | \[E\] | Cap signatures at a handful of relations; chunk larger structures | | Prior solutions can block better ones (Einstellung) and transfer wrongly | Luchins 1942; Novick 1988 (novice negative transfer) | \[E\] | Always keep a no-analogy arm in the loop; log negative transfer | | Distant analogies are not reliably better for design than near ones | Chan, Dow & Schunn 2015; Fu et al. 2013 | \[E\], domain-limited | Do not optimise for distance; optimise for usefulness | | Working scientists use many analogies, mostly local and regional, few far ones | Dunbar 1995, 1997 (in vivo lab studies) | \[E\], small samples | Most value lies in "same failure shape in another project", not in exotic cross-domain leaps | ### Where the literature disagrees The "surface dominates retrieval" result is not absolute. Holyoak & Koh (1987) found structural similarity also aids access. Blanchette & Dunbar (2000) showed that when people *generate* analogies rather than receive them, most of their retrieved sources share deep structure with little surface overlap. A plausible reconciliation \[X\]: retrieval follows whatever features were *encoded*, and lab participants encode surface features by default, while people producing an argument encode the goal and its causal structure. For Piper this is good news. It implies the retrieval gap is fixable by changing what is encoded, which is the core intervention proposed below. ### Retrieval and mapping as separate processes The two-stage view (cheap, wide, surface-sensitive access followed by expensive, narrow, structure-sensitive mapping) is the backbone of MAC/FAC (Forbus, Gentner & Law 1995) and fits the behavioural data. It is a modelling claim about function, not a settled claim about brain architecture; treat it as \[E\] for behaviour, \[X\] for mechanism. ## 4. Classical computational analogy systems **Classical systems solved mapping well and representation badly.** Every symbolic engine works when given clean relational descriptions and fails when it must build them. LLMs invert this: they build plausible descriptions easily and map inconsistently. The obvious \[X\] design is to let the LLM write the representation and let a deterministic matcher check it. | System | Mechanism | What it can do | What it cannot do | Lesson for Piper | | --- | --- | --- | --- | --- | | SME (Falkenhainer, Forbus & Gentner 1989; Forbus et al. 2017) | One-to-one alignment of predicate structures, parallel connectivity, systematicity preference; emits candidate inferences | Produces an inspectable mapping, a structural score and the inferences that follow | Build its own representations; results depend heavily on how the input was encoded | Use an SME-style matcher as the *checker* over LLM-written signatures | | MAC/FAC (Forbus, Gentner & Law 1995) | Stage 1: dot product of predicate-count "content vectors" over all memory; Stage 2: SME on the top few | Cheap, wide retrieval that is sensitive to *which relations* occur, not which words | Respect relational arrangement in Stage 1 | Direct template for Piper's two-stage retrieval | | ACME / ARCS (Holyoak & Thagard 1989; Thagard et al. 1990) | Parallel constraint satisfaction over structure, semantic similarity and **pragmatic centrality** (goal relevance) | Weights mappings by what matters for the current goal | Scale well; constraint weights are hand-set | Add goal relevance as an explicit mapping constraint | | LISA (Hummel & Holyoak 1997, 2003) | Neural role binding by synchrony; working-memory limits; schema induction as a by-product of mapping | Models capacity limits and schema formation | Practical engineering use | Schemas fall out of successful mappings; store them | | Copycat / Metacat (Hofstadter & Mitchell 1995) | Stochastic, parallel building of representations and mappings together, with conceptual "slippage" | Shows that perceiving the right description *is* the analogy | Scale beyond toy domains | Generate several descriptions per problem (section 6 levels) | | Dynamic memory / TOPs (Schank 1982) | Episodes indexed by goal–plan interaction patterns; reminding triggered by expectation failure | Cross-domain reminding through abstract thematic indices | Automatic index construction | Closest historical precedent for "structural failure signatures" | | CHEF (Hammond 1989) | Case-based planner that indexes plans by the failures they avoid and repairs by causal explanation | Anticipates failures from past cases; learns from explained failure | Generalise beyond a hand-built domain theory | Index by the *explanation* of a failure, not its symptoms | | CBR 4R cycle (Aamodt & Plaza 1994; Kolodner 1993) | Retrieve, Reuse, Revise, Retain | Mature practice for near-domain reuse | Far-domain retrieval; the indexing vocabulary problem | Retain step = memory update; revise step = verification | | Derivational analogy (Carbonell 1986; Veloso 1994, Prodigy/Analogy) | Replays the *decision rationale* of a past solution, not the final plan | Reuse across larger differences than plan copying | Requires recorded justifications | Store why each step was taken, not just the steps | | Divago and blending systems (Pereira 2007) | Computational conceptual blending with optimality constraints | Generate novel combinations | Validate them; evaluation remains mostly subjective | Supports demoting `blend-worker` to a gated generator | ### Two negative findings that matter - **The tailorability critique \[E\].** Chalmers, French & Hofstadter (1992) argued that SME-style successes depend on representations hand-built to make the mapping work. With LLM-written signatures the same risk returns in a new form: an LLM that "knows" two cases are analogous may write their signatures to match. Mitigation \[X\]: generate each case's signature independently, without sight of the other case, and at storage time rather than query time. - **Mapping is computationally hard \[E\].** Subgraph isomorphism is NP-complete and graph edit distance is NP-hard. For the small graphs proposed here (under \~15 nodes) exact or beam search is cheap; it becomes a problem only if signatures grow large, which is a reason to keep them small. ## 5. Modern neural and LLM research **LLMs map short, side-by-side analogies well and retrieve analogues from large memories badly.** That split is the single most useful fact for Piper's design: it says the base model is a reasonable mapper and a poor librarian, so the machinery should fix retrieval and add verification rather than replace the model's mapping. ### Embeddings - **\[E\] Word-vector "analogy" is mostly proximity.** The king − man + woman result (Mikolov et al. 2013) depends heavily on excluding the input words and on near-neighbour structure; accuracy falls on less regular relations (Linzen 2016; Rogers, Drozd & Li 2017). Relation-specific embeddings (RelBERT; Ushio et al. 2021) do better, but on word pairs, not situations. - **\[E\] Whole-text embeddings encode topic.** In analogy mining over product descriptions, embedding the full text retrieved topical neighbours; embedding separately extracted *purpose* and *mechanism* representations retrieved far more useful analogues (Hope, Chan, Kittur & Shahaf 2017; Kittur et al. 2019). This is the strongest direct evidence that "embed an abstracted description, not the raw one" works. - **\[E\] Story retrieval from memory is the weak point.** In AnaloBench the gap between humans and GPT-4 grew from about 7 points on one-sentence stories to about 29 points on thirty-sentence stories, and retrieving analogues from a pool of roughly 200 stories was near chance for long stories; scale helped little ([Ye et al. 2024](https://arxiv.org/abs/2402.12370)). ### LLM analogical competence: the open dispute - **For:** Webb, Holyoak & Lu (2023) reported GPT-3 matching or exceeding humans zero-shot on letter-string, matrix, verbal and story analogies. Their reply to critics ([Webb et al. 2024](https://arxiv.org/abs/2404.13070)) argues the counterfactual failures stem largely from a counting difficulty and that models generalise when that demand is removed. - **Against:** On permuted alphabets and symbol alphabets, human accuracy stayed high while GPT models dropped sharply ([Lewis & Mitchell 2024](https://arxiv.org/abs/2402.08955)); a broader robustness study found brittleness on most variations, including answer-order effects ([Lewis & Mitchell 2024b](https://arxiv.org/abs/2411.14215)). Stevenson et al. (2024) found models fail to generalise letter-string analogies to novel alphabets the way children do. - **Narratives:** StoryAnalogy (Jiayang et al. 2023) and ARN (Sourati et al. 2024) show near-human performance on near analogies and low, sometimes below-chance, performance when a far analogy competes with a near disanalogy. - **Mechanistic work \[E, early\]:** Yang et al. (ICML 2025) identify "symbol abstraction" and retrieval heads that support abstract rule induction; Opiełka et al. (2025) find relational concept vectors but limits on abstraction; Lee et al. (2025) report that models encode relations yet fail to apply them to new entities, and that patching relational representations rescues a substantial share of failures. My reading \[X\]: both camps are right about different regimes. Competence is real when source and target are short, co-present and free of competing surface cues. It degrades with length, retrieval load, distractors and unfamiliar encodings. Those are exactly the conditions in Piper's memory. ### Prompting and pipeline methods | Method | What it does | Evidence | Caveat | | --- | --- | --- | --- | | Analogical prompting (Yasunaga et al., ICLR 2024) | Model recalls or invents related solved problems before solving | Gains over zero-shot CoT on maths and code | Sources are self-generated, so it measures recall of training data, not memory use | | Thought Propagation (Yu et al., ICLR 2024) | Solve several analogous sub-problems, aggregate their solutions | Gains on graph and planning tasks | Extra compute is a confound; analogues are near-domain | | FAME (Jacob et al. 2023) | LLM extracts relations; greedy beam mapping over entities | Scalable entity mapping | Entity-level, not event or causal | | YARN ([Khojasteh et al. 2026](https://arxiv.org/abs/2603.29997)) | LLM decomposes stories into events, abstracts them at four levels, greedy one-to-one mapping scores fit | Abstractions consistently improved results; competitive with or better than end-to-end LLM baselines using 8B models | Errors concentrate in choosing the right abstraction level and in implicit causality | | Human–LLM strategy analogies (Puranam et al. 2025) | Compare humans and LLMs on retrieving and mapping strategic analogies | LLMs show high-recall, low-precision retrieval; humans map more reliably | Single domain; preprint | ### Creativity warning In a large blind study, expert reviewers rated LLM-generated research ideas as more novel than human experts' ideas but slightly less feasible (Si, Yang & Hashimoto 2024). That is the quantitative shape of "seductive nonsense": novelty is cheap for LLMs; validity is the scarce resource. Any analogy metric that rewards novelty without verification will select for it. ## 6. Representation of relational structure **Recommendation \[X\]: a small typed relational record per episode, the Structural Signature, written once at storage time, with a controlled vocabulary of roles and relations, a short causal chain, the violated assumption, and the fix expressed as an operator on roles.** It is indexed three ways (predicate-count vector, abstract-text embedding, graph fingerprint) and never fused with the surface embedding. ### Candidate representations compared | Representation | Captures | Retrieval method | Strength | Weakness | | --- | --- | --- | --- | --- | | Raw narrative embedding | Topic, entities, vocabulary | Vector kNN | Free; already exists | Retrieves topical neighbours (section 5) | | Abstracted NL summary, entity names removed | Gist of mechanism | Vector kNN | Cheap; Hope et al. 2017 support | Abstraction level drifts; LLM wording leaks surface | | Purpose / mechanism pair | Goal and how it is achieved | Vector kNN per field | Best-evidenced for invention retrieval | Weak for failures, which need the broken assumption | | Entity–relation graph | Who relates to what | Graph match | Inspectable; supports SME-style mapping | Needs a stable relation vocabulary | | Causal graph / event chain | What caused what | Topology match, path match | Captures the part that transfers | LLMs extract implicit causality unreliably (YARN error analysis) | | Precondition → action → consequence | Operator semantics | Slot match | Feeds planning directly | Brittle to granularity | | Goals / constraints / operators | Problem space | Constraint match | Good for plan reuse | Heavy to write for every episode | | Program or dependency graph | Data and control flow | Graph kernels | Ground truth in code | Only covers code episodes | | Hybrid typed signature (proposed) | Roles, relations, causal chain, violated invariant, fix operator, plus 4 abstraction levels | Three structural indexes + mapping check | Covers retrieval, mapping and transfer with one record | Vocabulary design risk; tailorability risk | ### Proposed Structural Signature schema \[S\] ```json { "episode_id": "ep_2026_09_14_0213", "kind": "failure", "roles": { "SRC": "authoritative source", "DER": "derived representation", "CON": "consumer that makes a claim" }, "relations": [ ["derives_from", "DER", "SRC"], ["trusts_as_authoritative", "CON", "DER"], ["skips_check", "CON", "SRC"], ["diverges_from", "DER", "SRC"] ], "causal_chain": [ ["diverges_from(DER,SRC)", "causes", "false_claim(CON)"], ["skips_check(CON,SRC)", "enables", "false_claim(CON)"] ], "violated_invariant": "claims about state are read from the authoritative source", "symptom": "confident assertion contradicted by later evidence", "fix_operator": "reconcile(CON, DER, SRC) before assert", "levels": { "L0": "Intel feed believed stale because cached summary was read", "L1": "Cache read in place of source caused wrong status", "L2": "Derived copy treated as authoritative; divergence propagated", "L3": "Proxy mistaken for referent" }, "surface": {"domain": "monitoring", "entities": ["intel feed", "cache"]}, "verified": true } ``` The ledger case from the brief (copied ledger total used instead of reconciling against the account) produces the same `relations` and `causal_chain` under a different `surface`. A predicate-count vector over `relations` and `causal_chain` gives the two cases near-identical vectors while their surface embeddings are far apart; that is the measurable property the benchmark should test. ### Indexes 1. **Predicate-count vector \[E for method, MAC/FAC\].** Counts of each relation and role type, plus counts of causal-chain patterns. Cosine similarity. Cheap, deterministic, vocabulary-sensitive. 2. **Abstract-text embedding \[X\].** Embed L2 and L3 separately, with entity names replaced by role tags. Catches matches the vocabulary missed. 3. **Graph fingerprint \[S\].** A Weisfeiler–Lehman hash of the role–relation graph at depth 2 for exact-shape lookup; useful for spotting repeated failure classes. ### Vocabulary design The controlled vocabulary is the hard part and the main risk \[X\]. Too coarse and everything matches; too fine and nothing does. Start with about 20–30 relations drawn from Piper's actual failure log (for example derives\_from, trusts\_as\_authoritative, skips\_check, masks, assumes, observes, reports\_absence, retries, amplifies, starves, races\_with, gates, depends\_on). Let the abstraction step propose a new relation only with a one-line definition; merge synonyms weekly during consolidation; track vocabulary size and per-relation frequency as health metrics. ## 7. Retrieval versus structural mapping **Multi-stage is supported; strictly linear is not.** The A–F pipeline matches MAC/FAC plus the CBR cycle, but the literature argues for three changes: a gate before Stage A, a re-representation loop when mapping fails, and a schema library searched before raw episodes. | Stage | Research support | Verdict | | --- | --- | --- | | A. Abstraction | Hope et al. 2017; YARN 2026; Copycat's representation-building | \[E\] that abstracted representations retrieve better; \[X\] that LLMs can produce them reliably at the right level | | B. Candidate retrieval | MAC/FAC two-stage; AnaloBench shows LLM-only retrieval collapses at scale | \[E\] for two-stage design | | C. Structural mapping | SME, ACME; Puranam et al. 2025 show mapping, not retrieval, is where LLM precision is lost | \[E\] | | D. Contradiction / decoy test | Alignable differences emerge from mapping itself (Markman & Gentner 1993; Gentner & Markman 1997) | \[E\] that comparison surfaces related differences; merge with C, then audit independently | | E. Transfer | SME candidate inferences; derivational analogy | \[E\] for inference projection; \[X\] for translating fixes into actions | | F. Verification | CBR "revise"; negative-transfer literature | \[E\] that unverified transfer is risky; \[X\] for automated checks | ### Recommended changes 1. **Stage 0, a gate \[X\].** Analogy search is costly and can mislead (Einstellung). Run it only when the trigger policy in section 10 fires. 2. **Merge C and D, then audit separately \[X\].** Alignment naturally yields *alignable differences*, the differences that sit on matched structure. Ask the mapper to output them. Then run an independent critic that sees only the mapping and the two signatures, not the mapper's argument, to look for breaks it missed. 3. **Re-representation loop \[E for humans, S for Piper\].** If the best mapping scores below threshold, re-abstract the target at the next level up (or down) and retrieve again. Cap at two loops. This is how progressive alignment and Copycat's slippage both work. 4. **Schema-first lookup \[X\].** Once consolidation has induced schemas from pairs of episodes (section 9), search the schema library before raw episodes. Schemas are fewer, cleaner and already verified. Fall back to episodes when no schema maps. 5. **A labelled parametric channel \[X\].** Let the LLM propose analogues from its own knowledge (analogical prompting), but tag them `parametric` and score them separately. This keeps "the model already knew" visible in every log line. ### Disagreement worth noting Hofstadter's camp holds that representation and mapping cannot be separated: the right description of a situation is found *by* trying to map it. SME's camp treats representation as an input. For engineering, the loop in change 3 is the compromise: separate modules, but mapping failure feeds back into representation. ## 8. False analogy and negative transfer **The best-supported defence is causal: transfer a conclusion only when the causal path that supports it in the source is fully mapped in the target and the target contains no unmapped factor that would block it.** Self-critique by the same model is weak; independent critics and cheap external checks are what actually catch errors. ### What the evidence says - **\[E\] Unmapped causal factors should reduce confidence.** In Bayesian models of analogical inference, people's confidence in a transferred conclusion drops when the source has a preventive cause that the target lacks a match for, or the target has one the source lacks (Lee & Holyoak 2008; Holyoak, Lee & Lu 2010). This gives a concrete, checkable rule. - **\[E\] Goals bend mappings.** When structure is ambiguous, people pick the mapping that suits their purpose (Spellman & Holyoak 1992, 1996). An agent motivated to finish a task will do the same. Mapping should be produced before the agent commits to a fix, not after. - **\[E\] Historical analogies are routinely misapplied in high-stakes decisions** (Neustadt & May 1986; Khong 1992). Neustadt & May's remedy is a written split of *likenesses*, *differences* and *unknowns* before any analogy is used. That is exactly the "matches A, B, C; fails on D" output requested, and it has decades of use in policy training, though little controlled evaluation. - **\[E\] Surface similarity drives negative transfer in novices** (Ross 1987; Novick 1988). Decoys are not a hypothetical risk. - **\[E\] LLM intrinsic self-correction of reasoning is unreliable without external feedback** (Huang et al., ICLR 2024; Kamoi et al. 2024 survey). A "critic" that is the same model, given the same context, will mostly agree with itself. - **\[E\] Comparison surfaces related differences** (Markman & Gentner 1993): the alignment step itself is the best generator of relevant disanalogies. ### Mechanisms, ranked by expected value | Mechanism | Basis | Cost | Notes | | --- | --- | --- | --- | | Causal-path gate: transfer inference only if its supporting path is fully mapped and no unmapped blocker exists | \[E\] Holyoak, Lee & Lu 2010 | Low; deterministic over signatures | Core rule | | Likeness / difference / unknown table, required output | \[E\] Neustadt & May; \[E\] alignable differences | One structured field | Makes partial transfer explicit | | Deterministic structural score (one-to-one consistency, share of source chain mapped, systematicity) | \[E\] SME | Negligible | Rejects mappings that only match isolated facts | | Independent critic: different model or temperature, sees the two signatures and the mapping but not the mapper's argument, must produce the strongest disanalogy | \[X\] from self-correction findings | One small LLM call | Information separation is the point, not a second opinion | | Prediction before implementation: the analogy must name one cheap observable it predicts in the target | \[X\] from scientific method; CBR revise | One check | A failed prediction refutes the analogy before any fix is applied | | Runtime decoy probe: retrieval also returns the highest surface-similar, structurally different case; if the mapper accepts the decoy, distrust its other acceptances | \[S\] | One extra mapping | Live calibration of the mapper on every run | | Negative-transfer ledger: log outcomes of analogy-driven actions against the no-analogy counterfactual where measurable | \[X\] | Storage | Feeds the keep/toss decision | ### Required output format \[S\] ```text ANALOGY source=ep_0213 (derived copy trusted over source) MAPPED DER→cache_summary, SRC→sensor, CON→health_reporter MATCHES derives_from, trusts_as_authoritative, skips_check (3/4 source relations) FAILS diverges_from: unknown in target — no evidence the cache is stale yet BLOCKERS target has TTL eviction (preventer absent in source) TRANSFER mechanism: reconcile-before-assert → transfer conclusion: "the cache is stale" → do NOT transfer PREDICT reading the sensor directly will disagree with the cached value CHECK one sensor read; abort analogy if values agree ``` ## 9. Memory architecture for analogy **Write the structural signature when the episode is stored, refine it during consolidation, and generate only the target's signature at query time.** Storage-time encoding is what the "inert knowledge" research says fixes retrieval (Gentner et al. 2009), and writing each case's signature without seeing its future match is the main defence against the tailorability problem. ### What existing agent-memory work shows Reflexion (Shinn et al. 2023), Generative Agents' reflections (Park et al. 2023), ExpeL (Zhao et al., AAAI 2024), Voyager's skill library (Wang et al. 2023) and Agent Workflow Memory (Wang et al. 2024) all store distilled lessons or reusable procedures and show in-domain gains \[E\]. None indexes by relational structure, and none reports cross-domain transfer as a separate measure. Piper would be testing something those papers did not. ### When signatures are generated | Moment | What is generated | Why | Evidence | | --- | --- | --- | --- | | Episode stored (resolved failure, successful fix, notable plan, board post describing a failure) | Full signature, L0–L3 | Relational encoding at study fixes later access; independent writing limits tailoring | \[E\] Gentner et al. 2009 | | Nightly consolidation | Pairwise comparison of signatures that already map well → induced schema; vocabulary merge; re-abstraction of old free-text episodes | Comparison induces schemas | \[E\] Gick & Holyoak 1983; Gentner et al. 2003 | | Analogy requested | Target signature only, plus re-abstraction if mapping fails | Keeps query cost bounded | \[X\] | ### Storage (SQLite + existing vector store) \[S\] | Table | Key columns | Purpose | | --- | --- | --- | | `signatures` | episode\_id, kind, json, levels, writer\_model, written\_at, verified | One row per signed episode | | `sig_relations` | episode\_id, predicate, arg1\_role, arg2\_role, in\_causal\_chain | Predicate-count vectors and exact lookups | | `sig_vectors` | episode\_id, level, embedding | L2/L3 abstract embeddings (separate collection from surface memory) | | `sig_fingerprint` | episode\_id, wl\_hash | Exact-shape matches; failure-class counting | | `schemas` | schema\_id, json, member\_episodes, times\_used, times\_verified | Induced from verified pairs | | `vocab` | term, definition, kind, count, merged\_into | Controlled role/relation vocabulary | | `analogy_log` | run\_id, target, candidates, mapping, score, critic\_verdict, transferred, prediction, outcome | Every analogy attempt, for metrics | Cost \[X\]: one signature call is roughly 1.5k input and 400 output tokens; at a few dozen signed episodes a day this is negligible next to Piper's normal traffic. ### Abstraction levels (question 8) - **\[E\] Multi-level abstraction helps mapping.** YARN's four levels consistently improved narrative analogy; purpose/mechanism abstraction improved invention retrieval (Hope et al. 2017). - **\[E\] The right level is the main error source.** YARN's error analysis concentrates on abstraction at the wrong level. Higher levels raise recall and lower precision: at L3, "proxy mistaken for referent" matches half of all bugs. - **Recommendation \[X\]:** generate L0→L3 sequentially, each conditioned on the previous (as YARN does); restrict L2 and L3 to vocabulary terms; retrieve at L1, L2 and L3 in parallel; learn per-level weights from the benchmark rather than guessing. - **Two reliability checks \[S\]:** (a) *faithfulness*: given only L2, can a separate model pick the original episode out of five distractors? (b) *de-surfacing*: given only L2, can it guess the original domain? Good abstractions pass (a) and fail (b). Both are cheap and run on a sample during consolidation. ## 10. Triggering and metacognitive control **Analogy should be an explicit operation fired by observable impasse signals, not by self-reported confidence.** Piper's own earlier finding that her metacognition fails at evidence acquisition rather than introspection argues for triggers computed from her action log (repeated error classes, stalls, irreversible actions) rather than from how unsure she says she feels. ### Evidence for triggers - **\[E\] Expectation failure triggers reminding.** Schank's dynamic memory (1982) and CHEF (Hammond 1989) both retrieve cases when a prediction fails, indexed by the explanation of that failure. - **\[E\] Impasses drive subgoaling and learning** in Soar (Laird, Newell & Rosenbloom 1987) and in human skill acquisition (VanLehn 1988). Insight research ties breakthroughs to impasse followed by re-representation (Ohlsson 1992). - **\[E\] Retrieval is not free.** Retrieval helps LLMs on long-tail questions and can hurt on popular ones (Mallen et al. 2023). Active retrieval methods fire on low token confidence (FLARE; Jiang et al. 2023) or learned reflection signals (Self-RAG; Asai et al. 2024), and adaptive routing by query complexity beats always-retrieve (Adaptive-RAG; Jeong et al. 2024). - **\[E\] Prior cases can block better solutions** (Einstellung; Luchins 1942). A trigger that fires too often makes this worse. - **\[X\] Value of computation** (Russell & Wefald 1991) gives the frame: run the operation when expected improvement exceeds its cost. ### Proposed policy \[S\] | Tier | Fires when | Work done | Approx. cost | | --- | --- | --- | --- | | 0. Always on | Every logged error | Graph-fingerprint and predicate-vector lookup against known failure classes; no LLM | Milliseconds | | 1. Standard | Two or more failed repairs with the same error class; or no measurable progress for N steps; or a Tier 0 hit above threshold; or an irreversible or external action is about to run (pre-mortem against the failure library) | Target signature, two-stage retrieval, forced mapping, causal-path gate | 3–4 LLM calls, roughly 6–10k tokens | | 2. Deep | Three or more failed repairs; or a high-stakes task; or Tier 1 mapping below threshold | Re-representation loop, independent critic, labelled parametric channel, runtime decoy probe | 8–12 LLM calls, roughly 20–30k tokens | Weak triggers (low self-reported confidence, "problem seems novel") start with low weight and are kept only if the log shows they predict usefulness. A daily token budget caps Tier 2. Every firing writes to `analogy_log` with the trigger that caused it, so trigger weights can be fitted later from outcome data: for each trigger, the useful-analogy rate and the negative-transfer rate. ## 11. Analogy and causal reasoning **Causal structure should be the highest-weighted channel, not the only one.** Shared causal structure is what licenses transferring a fix; but bare causal topology is too common to discriminate, LLM-extracted causal graphs are error-prone, and some useful analogies are not causal at all. ### Support - **\[E\] People's analogical inferences follow causal models.** Inference strength tracks whether generative and preventive causes are matched (Lee & Holyoak 2008; Holyoak, Lee & Lu 2010). - **\[E\] There is a formal notion of one causal model abstracting another.** Causal abstraction (Rubenstein et al. 2017; Beckers & Halpern 2019; Geiger et al. 2021 onward) defines when a high-level model faithfully summarises a low-level one under interventions. Two situations that both abstract to the same high-level model are causally analogous in a precise sense \[X\]. This gives Piper a principled definition, even if she computes it approximately. - **\[E\] LLMs' formal causal inference is weak** on benchmarks that resist memorisation (CLadder, Jin et al. 2023; Corr2Cause, Jin et al. 2024), though they do well at naming plausible causal pairs (Kıcıman et al. 2023). Expect plausible but unreliable causal graphs. ### Searching memory by causal topology \[X\] Yes, with typed edges. Represent each causal chain as a small directed graph with edge types (causes, prevents, enables, masks, delays) and edge sign, then index by **motif counts**: mediator chains, common causes, masking (a cause that hides its own evidence), feedback loops by sign, races. Weight motifs by rarity across memory (an IDF over motifs), so that a shared "masking + stale copy" motif counts for more than a shared "A causes B". ### What goes wrong | Failure | Why | Mitigation | | --- | --- | --- | | Trivial isomorphism | A → B → C matches almost everything | Typed, signed edges; motif rarity weighting; require a matched violated invariant | | Same topology, different regime | A negative feedback loop can be stable or oscillate depending on gain and delay | Record regime notes (thresholds, delays) in the signature; flag as an unknown in the difference table | | Extraction error | LLM invents or omits causal links | Mark links as observed vs inferred; the causal-path gate only trusts observed links | | Non-causal analogies missed | Shared algebra or invariants (e.g., two problems that are both bipartite matching) | Keep the relational and abstract-text channels alongside the causal one | | Tailored graphs | Model draws the graph to fit the analogy it already suspects | Graphs written at storage time, without sight of the match | Practical link to Piper's causal component \[S\]: let it own the causal graph for each episode, and let the analogy system read it rather than re-deriving causality. One source of truth avoids two subsystems disagreeing about what caused what. ## 12. Analogy and planning **Yes: retrieving a past solution trajectory, mapping its roles, and letting the planner repair the gaps is well precedented, but it only pays when the cases are close enough, and there is a formal result saying it can be no easier than planning from scratch.** Build it with a fallback and measure both paths. ### Evidence - **\[E\] Case-based planning works in practice when cases are near.** CHEF (Hammond 1989) and later systems reuse and repair stored plans; derivational analogy (Carbonell 1986; Veloso 1994) replays the *reasons* for each step, which tolerates larger differences than copying steps. - **\[E\] Worst-case theory is discouraging.** Nebel & Koehler (1995) proved plan modification is not easier than plan generation in the worst case, and retrieving the best case is itself hard. Empirical adaptation planners nonetheless often beat from-scratch planning on near problems (Gerevini & Serina's LPG-adapt line of work). - **\[E\] Macro-operators** learned from solutions speed search (Korf 1985; Botea et al. 2005, Macro-FF), with the known cost that too many macros enlarge the branching factor. - **\[E\] LLM agents gain from reusable procedures.** Voyager's skill library (Wang et al. 2023) and Agent Workflow Memory (Wang et al. 2024) report gains on embodied and web tasks by reusing induced routines. These are in-domain results. ### Proposed mechanism \[X\] 1. Store each successful multi-step solution as a **role-parameterised plan**: steps written as operators on signature roles, each with its precondition and the reason it was taken (derivational trace). 2. On a Tier 1 trigger, retrieve plans whose *goal signature* maps onto the target goal. 3. Map roles, instantiate steps, and mark every step whose precondition is unmapped or unknown. 4. Hand the planner the skeleton plus the marked gaps; it repairs only the gaps. 5. **Fallback rule:** if fewer than about two-thirds of the source plan's preconditions map, plan from scratch and log the skeleton as "offered, rejected". 6. Promote a plan to a **macro** only after two independent successful reuses (schema induction for plans), and retire macros whose reuse success falls below the from-scratch baseline. Measurement: compare adapted-plan versus from-scratch on steps-to-success, tokens and failure rate on matched tasks. The Nebel & Koehler result predicts the advantage disappears on distant cases; the benchmark should check that prediction directly by binning by mapping coverage. ### Analogy in discovery and invention (question 15) **Most documented discovery-by-analogy is local, and the agent's real advantage for novelty is private data, not cleverness.** - **\[E\] Science.** In Dunbar's in vivo studies of molecular biology labs, analogies were frequent; most were to closely related systems and were used to fix experiments, while distant analogies served mainly to explain results to others. Kepler's reasoning shows extended, explicit analogical re-representation driving conceptual change (Gentner et al. 1997). Hesse (1966) gives the classic account of positive, negative and neutral analogy, the same likeness/difference/unknown split as section 8. - **\[E\] Invention.** Purpose/mechanism analogy mining across product and patent text improved the quality of human ideation (Hope et al. 2017; Kittur et al. 2019). TRIZ derives cross-domain inventive principles from patent analysis; its evidence is mostly practitioner reports \[weak\]. - **\[E\] Strategy.** Managers use analogies heavily in novel environments; their quality depends on whether the mapped features are the causally important ones (Gavetti, Levinthal & Rivkin 2005). - **\[weak\] Software and debugging.** I found little controlled research on analogy in debugging specifically; most support is indirect (bug-pattern catalogues, fault-localisation by similarity to past fixes). Treat Piper's debugging use case as genuinely open. Mechanisms that could yield insights not present in training data \[S\]: 1. **Analogy over private corpora.** Pairs drawn from Piper's own incidents, the business operations she supports and her board conversations cannot have been memorised. Cross-project schema induction over these is the most credible route to genuinely new connections. 2. **Schema induction at scale.** Schemas induced from several private episodes, then retrieved against external literature, invert the usual direction: private pattern first, public analogue second. 3. **Prior-art check.** Before an analogy is labelled novel, search for it. If it is already published, label it "recalled" rather than "discovered". 4. **Generate, then verify cheaply.** Blending and far search are allowed to be wild only because every output must name a cheap test (section 8). ## 13. Existing benchmarks **No existing benchmark measures the full chain Piper needs: retrieval from the agent's own history, explicit role mapping, transfer into an action, and rejection of a surface-similar decoy.** Three are worth borrowing from directly: ARN for its decoy design, AnaloBench for its retrieval-from-pool task, and the Lewis & Mitchell counterfactual method for contamination control. | Benchmark | What it measures | Fit to Piper's target | Reuse | | --- | --- | --- | --- | | ARN (Sourati et al. 2024) | Narrative analogy with analogy and disanalogy each near or far in surface | **Direct** for decoy rejection; the far-analogy-vs-near-disanalogy cell is exactly the hard case | Copy its 2×2 design for every Piper item | | [AnaloBench](https://arxiv.org/abs/2402.12370) (Ye et al. 2024) | Pick analogous story from 4, or retrieve top-10 from \~200 | **Direct** for retrieval stage | Use T2 as an external sanity check of the retriever | | StoryAnalogy (Jiayang et al. 2023) | Story pairs rated for entity and relational similarity | Mapping and similarity judgement | Calibration set for the abstraction step | | ParallelPARC (Sultan et al. 2024) | Process-level analogies with distractors, generated by a pipeline | Close; procedural and cross-domain | Borrow its generation pipeline idea | | Letter-string and counterfactual alphabets (Webb et al. 2023; [Lewis & Mitchell 2024](https://arxiv.org/abs/2402.08955)) | Abstract rule transfer under familiar vs unfamiliar encodings | Adjacent; no cross-domain content | Borrow the counterfactual method, not the items | | RAVEN / I-RAVEN / PGM (Zhang et al. 2019; Hu et al. 2021; Barrett et al. 2018) | Visual relational induction | Adjacent; also a cautionary tale: RAVEN's answer sets allowed shortcut solutions, fixed in I-RAVEN | Lesson: audit decoys for shortcuts | | ARC / ConceptARC (Chollet 2019; Moskvichev, Odouard & Mitchell 2023) | Few-shot abstraction on grids | Adjacent: abstraction, not mapping across domains | Not recommended for Piper | | SAT, BATS, Google analogies, E-KAR (Turney 2003; Gladkova et al. 2016; Mikolov et al. 2013; Chen et al. 2022) | Word-level proportional analogies | Low; lexical relations, heavily contaminated | Skip | | SCAN-analogies (Czinczoll et al. 2022) | Scientific and creative metaphors | Low; famous analogies are memorised | Use as a contamination control only | | SCAN, COGS (Lake & Baroni 2018; Kim & Linzen 2020) | Compositional generalisation | Adjacent; recombination, not transfer | Skip | | Agent benchmarks (WebArena, Mind2Web, SWE-bench) | Task success | Not analogy; could host a transfer study | Possible downstream check later | ### Why they are inadequate for this purpose 1. **No private retrieval.** All use public material; none tests memory the model never saw. 2. **Recognition, not use.** They score choosing or completing an analogy, not whether a transferred fix works. 3. **No partial transfer.** None asks which parts of a mechanism should transfer and which should not. 4. **Contamination.** Most items are public and many are famous; several are now in pretraining corpora. 5. **Decoys are hand-built and few.** Only ARN systematically crosses surface and structural similarity. ## 14. Proposed Piper benchmark **PARB (Piper Analogical Reasoning Bench) has eight task families built from two item sources: procedurally generated relational worlds for clean attribution, and held-out incidents from Piper's own history for real-world validity.** Every item has a surface-matched decoy and a machine-checkable answer key for the mapping, not just the final choice. \[S\] throughout, built on the methods cited. ### Task families | Family | Item | Scored on | Borrowed from | | --- | --- | --- | --- | | F1 Cross-domain mapping | Source problem + solution; target in another domain with the same structure | Role-mapping accuracy, transferred inference | Gick & Holyoak paradigm | | F2 Decoy rejection | Four candidates crossing surface (near/far) with structure (analogy/disanalogy) | Picks far analogy over near disanalogy; justifies by relation | ARN 2×2 | | F3 Spontaneous retrieval | Target only, no hint; "have you seen this shape before?" Hidden answer key of episode ids; about 30% of items have no true match | Retrieval recall@k, historical claim precision, correct "no match" | AnaloBench T2, adapted to private memory | | F4 Solution transfer | Must output concrete target-domain actions | Executable check in a sandbox where possible, else key-element rubric | CBR reuse | | F5 Partial analogy | Source mechanism has several parts; target supports only some | Per-part transfer/withhold accuracy | Hesse's positive/negative/neutral analogy | | F6 False-analogy rejection | No valid source exists in the pool, including a tempting famous analogy | False-positive rate | Negative-transfer literature | | F7 Synthetic worlds | Generated causal worlds with nonce labels, permuted relation names, held-out templates | Everything above, contamination-free | Lewis & Mitchell counterfactual method | | F8 Re-representation | Match exists only at L2/L3, not L0/L1 | Success after re-abstraction | Progressive alignment; Copycat | ### Synthetic world generator (F7) 1. **Template library.** About 20 structural motifs as typed causal graphs: stale copy trusted, observer failure read as state failure, masking cause, runaway positive feedback, starvation, priority inversion, survivorship filter, proxy optimisation (Goodhart), race, cascade through shared dependency, and so on. 2. **Held-out split.** Design and tune on 12 templates; test only on 8 never seen during development. Without this, Piper's vocabulary will quietly be fitted to the test set. 3. **Skins.** Render each template into several domains (plumbing, beekeeping, logistics, an invented alien ecology) and a nonce version with random tokens for entities *and* relation words, preserving topology. 4. **Minimal-pair decoys.** Take the target skin and apply one structural edit: reverse one causal edge, flip a preventer to an enabler, or delete the derived-copy relation. Surface is identical; structure differs. These are the hardest and fairest decoys, and they are immune to the shortcut problem RAVEN had. 5. **Auto-generated keys.** Because the generator knows the true role correspondence, mapping accuracy is scored exactly, not by an LLM judge. 6. **Matched familiar skins.** Each nonce item has a familiar-word twin, so any drop under nonce labels is measured rather than confounded with reading difficulty (Webb et al. 2024's objection). ### Private-history items (F3, part of F1/F4) - Bink labels about 40 real incident pairs: a later target incident and the earlier incident(s) that share its structure, plus 15 targets with no true match. - **Holdout rule:** during testing, the target incident and anything derived from it are removed from Piper's memory snapshot, and the test runs against that snapshot. Otherwise she "retrieves" the answer itself. - Labels are fixed before any v2 machinery is built, and two labelling passes (or Bink plus one independent model pass reviewed by Bink) estimate key reliability. ### Contamination: which controls give the strongest evidence | Control | Rules out | Strength | | --- | --- | --- | | Held-out private incidents | Memorised pairing; tests real use | Strongest ecological evidence | | Generated worlds, held-out templates, nonce labels and relation names | Memorised content *and* familiar templates | Strongest causal attribution to the mechanism | | Relation-name permutation only | Lexical cueing of relations | Moderate | | Entity-name randomisation only | Entity associations | Weak: famous templates ("fox and grapes") survive renaming | | Paraphrase of public items | Exact-string recall | Weakest | Use the top two together: generated worlds tell you whether the mechanism works; private incidents tell you whether it matters. ### Size Minimal experiment: 48 items (F2, F3, F5, F7). Full PARB: about 320 items, 40 per family, half synthetic, half private where the family allows. ## 15. Metrics and scoring **Score each stage separately, then judge the system on one headline number that charges for every wrong or harmful analogy: Net Analogical Value.** Stage metrics locate failures; the headline metric stops a system from winning by producing many clever analogies of which few help. \[S\] for the composite definitions; the component ideas are standard. ### Stage metrics | Metric | Definition | Stage | | --- | --- | --- | | Retrieval recall@k | Share of items where a keyed source appears in the top k candidates | B | | Historical claim precision | When Piper says "I have seen this shape before", share where the cited episode is in the key | B | | No-match specificity | Share of no-match items where she correctly reports none | B | | Role-mapping F1 | F1 over (source role, target role) pairs against the generated or labelled key | C | | Decoy discrimination d′ | Signal-detection sensitivity: hit rate on true analogues vs false-alarm rate on decoys. Separates skill from a bias toward saying yes | C/D | | Partial-transfer balanced accuracy | Per mechanism component: correct transfer or correct withholding, averaged over both classes | D/E | | Transfer accuracy | Share of items whose target-domain action passes the executable check or key-element rubric | E | | Verification survival | Share of proposed analogies that pass the prediction check and critic | F | | Negative-transfer rate | Share of analogy-driven actions that score worse than the no-analogy arm on the same item | F | | Source dependence Δ | Accuracy with correct source minus accuracy with no source; also minus accuracy with a wrong source | Attribution | | Calibration (Brier) | Brier score of stated confidence that an analogy is valid | All | | Cost | Tokens and wall time per useful analogy | All | ### Headline metric: Net Analogical Value Let O be the number of items (opportunities). Let U be verified-useful analogies (correct source, correct mapping, transfer passes). Let F be accepted false analogies (decoys or no-match items accepted). Let N be negative-transfer events. Then: ```latex \mathrm{NAV} = \frac{U - \alpha F - \beta N}{O}, \qquad \alpha = 1,\ \beta = 2 ``` NAV is zero for a system that never proposes analogies and negative for one that proposes many bad ones. Harm counts double because an applied wrong fix costs more than a missed right one. The weights are a policy choice; fix them before running and report sensitivity at α, β ∈ {0.5, 1, 2, 3}. Report alongside it **Useful Analogy Precision** (U divided by all analogies proposed) and **Yield** (U divided by items where a valid analogy existed). A system that sprays analogies shows high Yield and low Precision; NAV makes it pay for that. ### Novelty, counted only when useful **Novel useful rate** = useful analogies whose source is private (not in any public corpus) or fails a prior-art search, divided by O. Novel-but-useless analogies earn nothing. This is the measure for `blend-worker`. ### Statistics Paired design: every arm sees every item. Compare arms with paired bootstrap confidence intervals on NAV and McNemar tests on per-item correctness. With 48 items, only large effects (on the order of 15–20 accuracy points) will be detectable; say so in the results rather than over-reading small differences. ## 16. Experimental arms and ablations **Ten arms, run on every item, so each mechanism is isolated by one pairwise comparison and two placebo arms rule out "more context helps" and "being told to think analogically helps".** All arms use the same base model, temperature and token cap; key arms are repeated on a second base model to check the result is not DeepSeek-specific. \[S\] for the design. ### Arms | Arm | Configuration | Purpose | | --- | --- | --- | | A0 | Bare LLM, no memory, no hint | Floor | | A1 | Bare LLM + "a related past case may help; consider analogies" | Isolates the hint effect (Gick & Holyoak) and analogical prompting | | B | A1 + current fused `memstore.py` retrieval, top k | Current system | | B′ | B with k raised to match C's context size | Controls for amount of retrieved text | | C | A1 + structural-signature retrieval, top k | Retrieval encoding | | C′ | C with signatures shuffled across episodes | Placebo: structured-looking but wrong context | | H | Union of B and C candidates | Do the channels complement each other? | | D | C + forced role mapping and difference table | Explicit mapping | | E | D + causal-path gate, independent critic, prediction check | Verification | | G | E with the gold source handed over | Oracle: upper bound once retrieval is solved | ### What each comparison isolates | Comparison | Isolates | Metric to read | | --- | --- | --- | | A1 − A0 | Being prompted to use analogy | Transfer accuracy, F6 false positives | | B − A1 | Having memory at all | Recall@k on F3 | | C − B′ | Structural vs semantic encoding at equal context | Recall@k on far items; d′ | | C − C′ | Signature content vs structured-looking noise | Recall@k, transfer | | H − max(B, C) | Complementarity of channels | Recall@k | | D − C | Explicit mapping | Role-mapping F1, transfer, d′ | | E − D | Verification | NAV, precision, negative-transfer rate | | G − E | Remaining retrieval gap | Transfer accuracy | ### Hypothesis cards | Hypothesis | Manipulation | Control | Measure | Failure signature | Keep if | | --- | --- | --- | --- | --- | --- | | H1: Structural signatures retrieve far analogues that semantic retrieval misses | Signature index (C) | B′ and C′ | Recall@5 on far-analogy items | C ≤ B′, or C ≈ C′ (gain is from any extra structure-shaped text) | C beats B′ by ≥ 15 points on far items, CI excluding 0, and beats C′ | | H2: Forced mapping improves transfer and decoy rejection | Mapping step (D) | C | Role-mapping F1, d′, transfer accuracy | No change in d′; transfer flat while cost rises | d′ up by ≥ 0.3 or transfer up ≥ 10 points | | H3: Verification raises net value by cutting false and harmful analogies | Gate + critic + prediction (E) | D | NAV; precision; negative-transfer rate | NAV flat because Yield falls as much as F and N | NAV up with CI excluding 0 | | H4: Machinery adds value beyond the base model | Full system (E) | A1 | NAV on F7 nonce items and F3 private items | E ≈ A1 on synthetic items (the LLM already does it) | E beats A1 on both item sources | | H5: Re-representation recovers missed matches | Loop (E + loop) | E | F8 success | No gain; loop mostly adds false positives | F8 success up ≥ 15 points with F6 false positives not worse | | H6: Blending adds useful novelty | `blend-worker` outputs gated through E | E without blending | Novel useful rate; NAV | Novel useful rate ≈ 0 or NAV falls | Novel useful rate > 0 with NAV not lower | If H4 fails on synthetic items but passes on private ones, the honest conclusion is that the gain is a memory-access gain, not a reasoning gain. That is still worth keeping, but it should be described that way.