Summary / 10. Analogical reasoning / Document
2026-10-04
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 10 — Analogical Reasoning ## Synthesized Research Review, Benchmark, and v2 Engineering Package **Date:** 2026-10-03 **Status:** Research synthesis for implementation and experimental validation **Source basis:** Three supplied research reports. This document reconciles their overlap, preserves material disagreements, and selects a single testable engineering plan. No additional external verification was performed for this synthesis. ### Evidence labels - **[E] Established:** supported in the supplied research by replicated, peer-reviewed, or longstanding findings. - **[X] Engineering extrapolation:** a reasonable design inference from established findings, but not directly validated for Piper. - **[S] Speculative:** a proposed Piper-specific mechanism, metric, or threshold that must earn its place experimentally. --- # 1. Executive conclusion Piper's analogical-reasoning problem is primarily an **encoding and retrieval problem**, and only secondarily a mapping problem. The supplied research converges on the same pattern: 1. Humans and LLM-like similarity systems tend to **retrieve by surface features**. 2. Once an appropriate source is present, both humans and modern LLMs can often **map deeper relational structure**. 3. Therefore, Piper's current semantic/fused retrieval is unlikely to reliably surface the far analogues that matter most. 4. Her dormant `blend-worker`, `reuse-search`, and `associate` paths do not currently constitute a live analogical-reasoning system. 5. The most defensible intervention is to store a **structural re-description** of important episodes, retrieve against that representation, force an explicit source→target mapping, expose disanalogies, and verify the transferred inference before acting. The core v2 loop should therefore be: ```text observable impasse / opportunity ↓ target structural signature ↓ schema-first + multi-channel retrieval ↓ explicit source→target mapping ↓ likeness / difference / unknown analysis ↓ causal-path transfer gate ↓ target-domain solution or prediction ↓ cheap verification / sandbox test ↓ outcome logged back to memory ``` The most important experimental question is not whether Piper can *say* an analogy. It is: > **Does explicit analogical machinery increase verified useful transfer beyond the same base model with ordinary memory retrieval, after controlling for hints, context volume, memorized examples, and surface similarity?** A null result is acceptable and informative. --- # 2. Operational definition ## 2.1 What counts as analogical reasoning **[X] Piper performs analogical reasoning when she:** 1. retrieves or is given a source case that differs from the target in surface content; 2. explicitly aligns source roles and relations with target roles and relations; 3. preserves at least one higher-order relational structure, preferably causal, functional, or dependency structure; 4. projects a candidate inference, solution, prediction, or plan through that mapping; 5. identifies where the analogy does **not** hold; 6. transfers only what survives the structural comparison; and 7. produces a measurable improvement, correct prediction, or correctly justified non-transfer on the target. Condition 7 matters. A structurally elegant analogy that changes nothing is explanatory decoration, not demonstrated competence. ## 2.2 What does not count by itself | Phenomenon | Count as analogy? | Reason | |---|---:|---| | Semantic similarity | No | Shared topic or vocabulary is not relational mapping. | | Episodic recall | No | Remembering a case is only input to analogy. | | Keyword/pattern matching | No | Surface correspondence is the primary decoy signal. | | Metaphor | Usually no | Counts only if it produces a testable relational inference. | | Conceptual blending | No, by default | Generates combinations rather than preserving structure. | | Abstraction | No | Necessary representation step, not transfer. | | Schema induction | Supporting capability | Counts when the induced schema later transfers. | | Case-based reasoning | Sometimes | Counts when reuse depends on explicit structural alignment. | | Structural analogy | Yes | Core capability. | | Causal analogy | Yes | Highest-value subtype for debugging and planning. | --- # 3. What the research says ## 3.1 Retrieval and mapping are different problems **[E]** The strongest repeated finding across the supplied literature is the retrieval–mapping dissociation. People frequently fail to spontaneously retrieve a distant but structurally relevant source. Once the source is cued, they can often map it successfully. Surface similarity strongly influences reminding; relational structure strongly influences soundness and transfer. **Engineering implication [X]:** Piper should have separate instrumentation for: - **Can she retrieve the right source?** - **Can she map it once supplied?** - **Can she transfer the right part?** - **Can she reject an attractive but wrong source?** A single end-to-end score hides the failure location. ## 3.2 Comparison induces schemas **[E]** Comparing two structurally similar examples can induce an abstract schema that transfers better than studying either example alone. **Engineering implication [X]:** consolidation should not merely summarize individual episodes. It should periodically compare already-verified analogues and derive reusable schemas. ## 3.3 Expertise is partly representation **[E]** Experts tend to encode problems by underlying principles rather than surface features. **Engineering implication [X]:** Piper's analogue of expertise is not just a larger memory. It is a better **relational vocabulary** and better structural encoding at storage time. ## 3.4 Far analogy remains brittle for LLMs **[E, as reported in the supplied research]** Modern LLMs can perform strongly on many short, familiar, co-present analogies, yet degrade on far analogies, unfamiliar encodings, distractors, long-context retrieval, and contamination-resistant variants. **Engineering implication [X]:** use the LLM as a representation and mapping component, but do not assume its implicit analogy generation is reliable enough to be the system. --- # 4. Central design decision: represent structure explicitly ## 4.1 Structural Signature The proposed atomic memory object is a **Structural Signature (SS)**. **[S] Recommended contents:** - typed roles rather than surface names; - typed relations; - causal or dependency chain; - violated invariant or assumption; - observed symptom; - fix expressed as an operator over roles; - surface-domain metadata; - verification status; - several abstraction levels. Example: ```json { "episode_id": "ep_2026_09_14_0213", "kind": "failure", "roles": { "SRC": "authoritative source", "DER": "derived representation", "CON": "consumer that makes a claim" }, "relations": [ ["derives_from", "DER", "SRC"], ["trusts_as_authoritative", "CON", "DER"], ["skips_check", "CON", "SRC"], ["diverges_from", "DER", "SRC"] ], "causal_chain": [ ["diverges_from(DER,SRC)", "causes", "false_claim(CON)"], ["skips_check(CON,SRC)", "enables", "false_claim(CON)"] ], "violated_invariant": "claims about state are read from the authoritative source", "symptom": "confident assertion contradicted by later evidence", "fix_operator": "reconcile(CON, DER, SRC) before assert", "levels": { "L0": "Intel feed believed stale because cached summary was read", "L1": "Cache read in place of source caused wrong status", "L2": "Derived copy treated as authoritative; divergence propagated", "L3": "Proxy mistaken for referent" }, "surface": { "domain": "monitoring", "entities": ["intel feed", "cache"] }, "verified": true } ``` A financially framed event such as "copied ledger total trusted instead of reconciling against the account" should encode to approximately the same L2/L3 structure despite very different surface vocabulary. That is precisely the property the benchmark must test. ## 4.2 Abstraction ladder **[X] Use multiple levels rather than one canonical abstraction.** ```text L0 — literal episode L1 — implementation pattern L2 — system mechanism L3 — abstract relational pattern ``` Example: ```text L0: Python health check crashed because an environment value was empty. L1: Input assumption caused monitoring failure. L2: Measurement mechanism failed and absence of evidence was read as evidence of absence. L3: Observer failure mistaken for observed-state failure. ``` Too little abstraction preserves surface bias. Too much abstraction causes everything to match everything. The right level should therefore be learned empirically. --- # 5. Storage-time encoding, not query-time invention **[E/X] The structural signature should normally be written when an important episode is stored**, before Piper knows what future case it might be compared against. This matters for two reasons: 1. relational encoding improves later retrieval; 2. it reduces the **tailorability problem**—the risk that an LLM presented with two known-to-be-related cases writes convenient matching representations after the fact. Recommended generation policy: | Moment | Work | |---|---| | Important episode stored | Generate SS + L0–L3 + fix/operator representation | | Nightly consolidation | Compare strongly mapped episodes, induce schemas, merge vocabulary synonyms | | Analogy query | Generate target signature only; optionally re-represent if mapping fails | | Legacy memory | Backfill signatures selectively, starting with resolved failures/successes | --- # 6. Structural vocabulary The controlled vocabulary is likely the hardest engineering problem. **[S] Start with approximately 20–30 relations grounded in Piper's actual incidents**, for example: ```text derives_from trusts_as_authoritative skips_check masks assumes observes reports_absence retries amplifies starves races_with gates depends_on enables prevents causes invalidates blocks propagates reconciles_with ``` Rules: - a new relation requires a short definition; - synonyms are merged during consolidation; - relation frequency is logged; - a vocabulary term that is never useful in successful mappings is a candidate for removal; - the benchmark, not aesthetic preference, decides whether the vocabulary is too coarse or too fine. --- # 7. Retrieval architecture ## 7.1 Do not collapse every retrieval signal into one fused score The supplied reports differ slightly here. The strongest synthesis is: > **Keep retrieval channels separate, label their provenance, then let structural mapping determine soundness.** Recommended channels: ### Channel S0 — exact structural fingerprint A Weisfeiler–Lehman or similar graph fingerprint for exact/repeated shapes. ### Channel S1 — predicate/role vector MAC/FAC-like counts of structural predicates and causal motifs. ### Channel S2 — abstract-text embeddings Separate embeddings for L2 and L3 descriptions with surface entities removed or role-substituted. ### Channel S3 — schema library Verified induced schemas searched before raw episodes. ### Channel M — ordinary semantic memory Existing `memstore.py` / fused semantic retrieval remains useful for near analogues and ordinary context. ### Channel P — parametric model knowledge The LLM may propose an analogy from pretrained knowledge, but the source must be labelled `parametric` and scored separately. The runtime candidate pool is the **union**, not an opaque fusion. This permits attribution: ```text candidate source=ep_1042 found_by=[S1,S2] semantic_rank=42 structural_rank=2 ``` That is experimentally much cleaner than a fused relevance score. --- # 8. Mapping architecture Once candidates exist, mapping should become explicit. ## 8.1 Required mapper output ```text SOURCE ROLE TARGET ROLE ----------- ----------- SRC sensor DER cache_summary CON health_reporter MATCHED RELATIONS - derives_from - trusts_as_authoritative - skips_check UNMATCHED SOURCE RELATIONS - diverges_from TARGET-ONLY FACTORS - TTL eviction ``` ## 8.2 Deterministic checks Where possible, score: - one-to-one role consistency; - share of source causal path mapped; - share of target path explained; - systematicity / connectedness; - unmatched blockers or preventers; - contradictory edge signs. An LLM may propose the mapping, but deterministic checks should decide whether the claimed mapping is structurally coherent. --- # 9. Re-representation loop There is a real theoretical disagreement in analogy research: - one camp treats representation as input to mapping; - another treats finding the right representation as part of analogy itself. For Piper, use a bounded loop. ```text target SS at current level ↓ retrieve + map ↓ mapping below threshold? ├── no → continue └── yes → re-abstract one level up/down and retry ``` **[S] Cap at two re-representation attempts.** The loop is evaluated separately in the F8 benchmark family. If it only increases false positives, remove it. --- # 10. False analogy and negative transfer Piper's own phrase "seductive nonsense" is the correct failure class. The most important rule is: > **Do not transfer a conclusion unless the structural or causal support for that conclusion is itself mapped.** ## 10.1 Required likeness / difference / unknown record Every accepted analogy should expose: ```text ANALOGY source=ep_0213 MAPPED DER→cache_summary, SRC→sensor, CON→health_reporter MATCHES derives_from, trusts_as_authoritative, skips_check FAILS diverges_from: unknown in target BLOCKERS target has TTL eviction TRANSFER reconcile-before-assert → YES "cache is stale" → NO PREDICT direct sensor read will disagree with cache CHECK perform one source read; abort analogy if prediction fails ``` ## 10.2 Defence stack, ranked 1. **Causal-path gate** — deterministic, low cost. 2. **Explicit likeness / difference / unknown table**. 3. **Deterministic structural score**. 4. **Independent critic** that sees signatures and mapping but not the mapper's argumentative prose. 5. **Prediction before implementation**. 6. **Cheap sandbox or external check**. 7. **Negative-transfer ledger**. 8. Optional runtime decoy probe for calibration. Same-model self-critique is not treated as sufficient verification. --- # 11. Triggering analogy search Analogy should be an explicit cognitive operation, not something Piper performs only when the base model happens to think of it. But it should not fire constantly. ## 11.1 Prefer observable impasse signals Given Piper's already-observed metacognitive calibration weakness, the trigger should prefer **behavioural signals** over self-reported confidence. High-value triggers: - repeated failed repairs; - repeated error class; - no measurable progress for N steps; - exact/near structural fingerprint hit; - imminent irreversible or external action; - planning search explosion; - causal model contradiction; - recurring symptoms across superficially different tasks. Self-reported "low confidence" or "this feels novel" can remain weak features until outcome data proves they predict useful analogy. ## 11.2 Tiered policy | Tier | Trigger | Work | |---|---|---| | 0 | Every logged error | fingerprint + predicate lookup only; no LLM | | 1 | repeated/stalled/high-risk or strong structural hit | target signature, retrieval, forced mapping, causal gate | | 2 | deeper impasse/high stakes/mapping failure | re-representation, independent critic, parametric channel, optional decoy probe | Every invocation logs **why it fired** so trigger policies can later be fit from actual useful-analogy and negative-transfer rates. --- # 12. Memory schema Recommended SQLite additions: ```text signatures( episode_id, kind, signature_json, writer_model, written_at, verified ) sig_relations( episode_id, predicate, arg1_role, arg2_role, in_causal_chain ) sig_vectors( episode_id, level, embedding_ref ) sig_fingerprint( episode_id, wl_hash ) schemas( schema_id, schema_json, member_episodes, times_used, times_verified ) vocab( term, definition, kind, count, merged_into ) analogy_log( run_id, target_id, trigger, candidate_ids, candidate_channels, mapping_json, structural_score, critic_verdict, transferred_elements, predicted_observable, verification_result, outcome_delta, token_cost ) ``` The analogy log is not optional. A live mechanism without invocation and outcome telemetry risks recreating the exact current problem: impressive code that never participates in cognition. --- # 13. Analogy and causal reasoning Causal structure should be the **highest-weighted structural channel, not the only one**. Two cases with approximately isomorphic typed causal graphs are strong candidates even when vocabulary differs. But causal analogy can fail because of: - missing confounders; - incorrect source causal graphs; - abstraction-level mismatch; - interventions valid in one domain but invalid in the other; - extra preventive or enabling factors in the target. Therefore: ```text causal topology match + role/relation consistency + unmapped blocker check = candidate causal analogy ``` Do not equate graph isomorphism with guaranteed transferable action. --- # 14. Analogy and planning Analogical planning is a natural downstream use. Represent a successful plan as: ```text initial state → operator 1 → intermediate state → operator 2 → ... → goal state ``` Then: 1. retrieve a structurally similar prior plan; 2. map source roles to target roles; 3. project the plan skeleton; 4. identify unmatched preconditions/operators; 5. give the skeleton to the planner for repair; 6. execute or simulate; 7. record outcome. Repeated verified transfers can be compiled into **macro-actions** or schemas. The key metric is not "planner used an old plan." It is whether analogy improves: - plan success; - number of search expansions; - tool calls; - tokens; - wall time; - error rate. --- # 15. Existing Piper components: disposition ## `associate` **Recommendation [S]: retire unless a live caller and unique measurable function can be demonstrated.** Zero-use code should not remain part of Piper's claimed cognition. ## `reuse-search` **Recommendation [S]: replace or refactor into `analog-retrieve`.** It should become a front-end over the labelled structural/semantic channels, returning candidates with provenance. ## `blend-worker` **Recommendation [S]: demote from "analogy machinery" to a sandboxed novelty generator.** It may generate candidate cross-domain connections, but every output must pass the same mapping, causal gate, disanalogy, and verification pipeline. The benchmark decides whether it survives. ## fused `memstore.py` retrieval **Recommendation:** retain as the ordinary semantic channel, but do not treat its score as analogical soundness. --- # 16. Piper Analogical Reasoning Bench (PARB) No supplied public benchmark simultaneously tests: - retrieval from Piper's private history; - far structural retrieval; - explicit role mapping; - surface-matched decoy rejection; - partial transfer; - executable transfer; - contamination resistance; - live usefulness. Therefore build a Piper-specific suite. ## 16.1 Task families | Family | Task | Primary score | |---|---|---| | F1 Cross-domain mapping | source + target share structure across domains | role mapping + transferred inference | | F2 Surface decoy rejection | far structural match competes with near surface mismatch | decoy discrimination | | F3 Spontaneous retrieval | target only; retrieve from private history or say no match | recall@k + claim precision + specificity | | F4 Solution transfer | adapt source fix into target actions | executable transfer accuracy | | F5 Partial analogy | only part of source mechanism transfers | transfer/withhold balanced accuracy | | F6 False-analogy rejection | no valid source exists | false-positive rate | | F7 Synthetic worlds | nonce entities/relations, held-out structures | contamination-resistant end-to-end score | | F8 Re-representation | match becomes visible only at another abstraction level | recovery after abstraction loop | ## 16.2 Synthetic generator **[S] Template library:** roughly 20 structural motifs, for example: - stale derived copy trusted over source; - observer failure interpreted as state failure; - masking cause; - runaway positive feedback; - starvation; - priority inversion; - survivorship filter; - proxy optimization / Goodhart pattern; - race condition; - cascade through shared dependency. Design split: ```text development templates: 12 held-out test templates: 8 ``` For each template: - render several ordinary-domain skins; - render one or more nonce-label skins; - optionally permute relation names; - generate a minimal-pair decoy by one structural edit; - retain an auto-generated mapping key. Minimal-pair decoy edits include: - reverse a causal edge; - flip `prevents` ↔ `enables`; - delete a dependency; - change a mediator to a confounder; - change copied state to independent state. Surface wording should remain as close as possible while structure changes. ## 16.3 Private-history items Create a curated private set from Piper incidents. Recommended MVP: - about 25–30 true-match targets; - about 15 no-match targets; - fixed labels before v2 implementation; - target event and derivatives removed from the memory snapshot during testing. Private incidents provide ecological validity. Synthetic worlds provide causal attribution. Use both. --- # 17. Contamination controls Ranked strongest to weakest: 1. **Held-out private Piper incidents**. 2. **Generated worlds using held-out structural templates plus nonce entity/relation labels**. 3. Relation-name permutation. 4. Entity-name randomization alone. 5. Public-item paraphrase. Classic famous analogies should not contribute to the main score. ## 17.1 Source-dependence test For selected items run: ```text correct source no source surface-matched wrong source ``` Define: ```text Source Dependence Δ = accuracy(correct source) - accuracy(no source) ``` Also compare correct source against wrong-source condition. A system that already knew the answer has Δ≈0. A system that is merely primed by *any* example improves equally with the wrong source. ## 17.2 Mapping-lesion test Delete or alter one relation that the system says is load-bearing. If the transferred conclusion is unchanged, the explicit mapping may be post-hoc explanation rather than causal support for the answer. --- # 18. Metrics Do not collapse all diagnostics into one score. Score stages separately. | Metric | Definition | |---|---| | Retrieval recall@k | correct source appears in top-k | | Historical claim precision | claimed prior episode is actually keyed as a match | | No-match specificity | correctly says none when no valid source exists | | Role-mapping F1 | source↔target role correspondences against key | | Decoy discrimination d′ | sensitivity to true analogues vs false acceptances | | Partial-transfer balanced accuracy | correct transfer and correct withholding | | Transfer accuracy | target-domain action/solution passes check | | Verification survival | proposed analogy survives critic/prediction/test | | Negative-transfer rate | analogy-driven action performs worse than control | | Source Dependence Δ | correct-source accuracy minus no-source/wrong-source | | Calibration | Brier score for analogy-validity confidence | | Cost | tokens / calls / wall time per useful analogy | --- # 19. Headline metric: Net Analogical Value **[S]** Let: - \(O\) = opportunities; - \(U\) = verified-useful analogies; - \(F\) = accepted false analogies; - \(N\) = negative-transfer events. Then: ```text NAV = (U - αF - βN) / O ``` Initial pre-registered policy: ```text α = 1 β = 2 ``` A harmful applied analogy costs more than a false suggestion that is rejected before action. Report sensitivity at several weight choices rather than hiding dependence on the policy. Also report: ```text Useful Analogy Precision = U / proposed analogies Yield = U / valid-analogy opportunities Novel Useful Rate = useful private/novel analogies / O ``` Novelty without verified usefulness earns nothing. This gives `blend-worker` a fair but difficult metric. --- # 20. Experimental arms The synthesis recommends the stronger multi-arm design rather than only A/B/C/D/E. All arms use the same base model, temperature, tool access, and token cap. | Arm | Configuration | Purpose | |---|---|---| | A0 | bare LLM; no memory; no hint | floor | | A1 | bare LLM + "consider related past cases" hint | isolates prompting/reminding | | B | A1 + current semantic/fused memory | current Piper memory contribution | | B′ | B with context size matched to C | controls for extra retrieved text | | C | A1 + structural-signature retrieval | tests structural encoding | | C′ | C with signatures shuffled across episodes | placebo for "structured-looking context" | | H | union of B and C candidates | tests complementarity | | D | C + forced role mapping + difference table | tests explicit mapping | | E | D + causal gate + independent critic + prediction check | tests verification | | G | E with gold source handed to system | oracle upper bound once retrieval is solved | Optional: - **E+R:** E plus re-representation loop. - **E+Bld:** E plus gated `blend-worker`. - repeat key arms on a second base model. --- # 21. What each comparison tells us | Comparison | Question | |---|---| | A1 − A0 | Does merely reminding the model to think analogically help? | | B − A1 | Does Piper memory help at all? | | C − B′ | Does structural encoding beat semantic retrieval at equal context? | | C − C′ | Is the structural content real signal or placebo? | | H − max(B,C) | Are semantic and structural retrieval complementary? | | D − C | Does forced mapping improve transfer/decoy rejection? | | E − D | Does verification increase net value? | | G − E | How much remaining error is retrieval versus mapping/transfer? | | E+R − E | Does re-representation recover misses without inflating false positives? | | E+Bld − E | Does blending create verified novelty rather than noise? | --- # 22. Pre-registered hypotheses ## H1 — Structural signatures improve far retrieval **Manipulation:** C **Control:** B′ and C′ **Measure:** recall@5 on far items, d′ **Failure signature:** C≤B′ or C≈C′ **Keep if:** C beats B′ by ≥15 percentage points on far items with uncertainty excluding zero, and beats C′. ## H2 — Forced mapping improves use, not just explanation **Manipulation:** D **Control:** C **Measure:** mapping F1, d′, transfer accuracy **Failure signature:** mapping prose increases but transfer/decoy metrics do not **Keep if:** d′ rises materially (initial target ≥0.3) or transfer accuracy rises ≥10 points. ## H3 — Verification reduces seductive nonsense **Manipulation:** E **Control:** D **Measure:** NAV, precision, negative-transfer rate **Failure signature:** NAV flat or lower because verification kills useful yield without enough safety gain **Keep if:** NAV improves with paired uncertainty excluding zero. ## H4 — The machinery adds value beyond the base model **Manipulation:** E **Control:** A1 **Measure:** NAV on private items and nonce synthetic items **Failure signature:** E≈A1 on synthetic and private items **Keep if:** E beats A1 on both item sources. Interpretation nuance: - passes private, fails synthetic → primarily a **memory-access improvement**; - passes synthetic, fails private → structural mechanism works but is not integrated with real memory; - passes both → evidence for a genuine useful analogical subsystem. ## H5 — Re-representation helps **Manipulation:** E+R **Control:** E **Measure:** F8 recovery, F6 false positives **Keep if:** F8 rises ≥15 points with no meaningful false-positive increase. ## H6 — Blending adds verified novelty **Manipulation:** E+Bld **Control:** E **Measure:** novel useful rate + NAV **Keep if:** novelty increases and NAV does not decline. --- # 23. Minimal viable experiment Do not build the full architecture first. ## MVP data - structurally sign roughly **100–200 resolved Piper failures/successes** if available; - if that volume is not yet clean enough, start smaller but keep the benchmark fixed; - build **48 benchmark items** concentrated on F2, F3, F5, and F7; - include both private-history and synthetic items; - randomize/nonce-label roughly half the synthetic items. ## MVP code 1. structural-signature extraction; 2. SQLite tables for signatures/relations; 3. predicate-vector index; 4. L2/L3 embedding index; 5. optional graph fingerprint; 6. retrieval channel provenance; 7. explicit mapping output; 8. likeness/difference/unknown fields; 9. causal-path transfer gate; 10. analogy log. ## MVP arms At minimum run: ```text A0 bare LLM A1 analogy-hinted LLM B current semantic memory C structural retrieval D structural retrieval + forced mapping E mapping + verification ``` If cheap, include B′ and C′ immediately because they rule out two important confounds. ## MVP success criterion Do **not** keep v2 because retrieval recall increased. Keep it only if structural/mapping arms improve at least one downstream outcome such as: - transfer accuracy; - decoy discrimination; - partial-transfer accuracy; - NAV; without an unacceptable cost or negative-transfer increase. --- # 24. Implementation sequence ## Phase 0 — measure current reality - instrument `blend-worker`, `reuse-search`, `associate`; - confirm actual call counts; - establish current A0/A1/B benchmark baseline; - freeze benchmark keys. ## Phase 1 — structural representation - implement signature JSON; - create initial relation vocabulary; - sign a sample of historical episodes; - manually inspect agreement and failure modes; - add L0–L3 abstraction. ## Phase 2 — structural retrieval - predicate vectors; - L2/L3 embeddings; - graph fingerprint; - candidate provenance; - schema table, initially empty. ## Phase 3 — mapping - source→target role table; - deterministic one-to-one and path checks; - matched/unmatched relations; - mapping confidence. ## Phase 4 — verification - causal-path gate; - likeness/difference/unknown; - independent critic; - one cheap prediction where possible; - sandbox/external check where available. ## Phase 5 — consolidation - induce schemas from repeated verified mappings; - merge relation synonyms; - record successful fix operators; - develop macro-actions for plan reuse. ## Phase 6 — live triggering - Tier 0 fingerprint lookups; - Tier 1 impasse-triggered analogy; - Tier 2 deep/re-representation path; - token budget and trigger outcome fitting. ## Phase 7 — optional creativity channel - reintroduce `blend-worker` only through the verified pipeline; - score novelty only after usefulness. --- # 25. Primary failure modes | Failure | Symptom | Mitigation | |---|---|---| | Signature too surface-like | structural retrieval mirrors semantic search | entity stripping, controlled roles, nonce test | | Signature too abstract | everything looks analogous | level-specific retrieval, precision penalties | | Tailored signatures | source/target magically match after comparison | write episode SS at storage time, independently | | Vocabulary fragmentation | same relation gets many names | consolidation merge process | | Vocabulary collapse | generic predicates dominate | frequency/discrimination monitoring | | Mapping rationalization | answer unchanged when mapping is lesioned | mapping-lesion test | | Critic agreement loop | critic always approves mapper | information-separated or different model critic | | Over-triggering | cost rises, negative transfer increases | observable trigger policy + NAV | | Under-triggering | dead machinery again | opportunity→invocation telemetry | | Benchmark shortcut | high score via artifacts rather than structure | minimal-pair decoys, held-out templates | | Contamination | classic items produce inflated score | private + generated held-out worlds | | Pretty-analogy spam | lots of clever output, little utility | precision, NAV, negative-transfer penalty | | Verification gaming | system learns check rather than task | held-out checks and executable outcomes | --- # 26. Decision rules for Component 10 After the MVP, classify the result rather than forcing a success story. ## Result A — structural retrieval wins, mapping adds little Interpretation: Piper had a **memory access problem**, not a mapping problem. Action: keep signatures/retrieval; simplify mapper. ## Result B — retrieval flat, forced mapping wins Interpretation: current memory already surfaces adequate candidates; explicit relational discipline prevents bad transfer. Action: keep mapping/disanalogy; reconsider structural indexing. ## Result C — retrieval and mapping both win Interpretation: explicit analogy machinery provides measurable added capability. Action: proceed to schemas, plan transfer, and live triggers. ## Result D — only verification wins Interpretation: Piper can already generate analogies, but needs protection against false transfer. Action: retain causal/disanalogy/verification gate; avoid unnecessary retrieval complexity. ## Result E — no arm beats the hinted bare LLM Interpretation: the explicit machinery is currently cost without value. Action: do not describe it as cognition. Remove or redesign it. ## Result F — private items improve but synthetic items do not Interpretation: **memory access**, not general structural reasoning. Still useful, but label it accurately. --- # 27. Updated description for Piper if v2 is not yet validated Until the machinery produces measured gains, Piper should not describe dormant components as active cognition. A defensible current description is: > Analogical reasoning currently happens mostly implicitly inside the base model and through ordinary memory retrieval. Dedicated analogy components exist but have not yet demonstrated live use or measurable added value. The current engineering effort is building a structural-signature retrieval and verification pipeline designed to test whether explicit machinery improves cross-domain transfer beyond the base model. Once v2 passes the benchmark, the description can be updated to reflect the measured mechanism. --- # 28. Recommended "Latest / Baseline / Live use" measurements For the cognition scoreboard: ```text 10 Analogical reasoning Latest — PARB NAV: [value] Transfer accuracy:[value] Decoy d′: [value] Recall@5 far: [value] Negative transfer:[value] Baseline — A0 NAV: [value] A1 NAV: [value] B current memory: [value] Live use — last 7 days opportunities: [count] Tier 0 hits: [count] Tier 1 runs: [count] Tier 2 runs: [count] useful analogies: [count] false accepts: [count] negative transfers:[count] median token cost: [value] ``` This makes "used sparingly" measurable rather than rhetorical. --- # 29. Selected references from the supplied research The following were repeatedly identified in the source material as especially relevant: - Gentner, D. (1983). *Structure-Mapping: A Theoretical Framework for Analogy*. Cognitive Science, 7(2), 155–170. DOI: 10.1207/s15516709cog0702_3. - Falkenhainer, B., Forbus, K. D., & Gentner, D. (1989). *The Structure-Mapping Engine*. Artificial Intelligence. - Forbus, K. D., Gentner, D., & Law, K. (1995). MAC/FAC analogical retrieval. - Gick, M. L., & Holyoak, K. J. (1980, 1983). Analogical problem solving and schema induction. - Holyoak, K. J., & Koh, K. (1987). Surface and structural similarity in analogical transfer. - Gentner, D., Rattermann, M. J., & Forbus, K. D. (1993). Roles of similarity in transfer. - Markman, A. B., & Gentner, D. (1993); Gentner & Markman (1997). Alignable differences. - Aamodt, A., & Plaza, E. (1994). Case-Based Reasoning: foundational 4R cycle. - Carbonell, J. (1986); Veloso, M. (1994). Derivational analogy / plan reuse. - Hammond, K. (1989). CHEF and failure-driven case-based planning. - Hofstadter, D., & Mitchell, M. (1995). Copycat / representation and mapping. - Hope, T., Chan, J., Kittur, A., Shahaf, D. and related work on purpose/mechanism abstraction for analogy mining. - Webb, T., Holyoak, K., & Lu, H. (2023, 2024). LLM analogical reasoning and subsequent debate. - Lewis, M., & Mitchell, M. (2024). Counterfactual / unfamiliar encoding tests for analogy. - Ye et al. (2024). AnaloBench. arXiv:2402.12370. - Yasunaga et al. (ICLR 2024). Analogical prompting. - Yu et al. (ICLR 2024). Thought Propagation. - YARN (2026). Narrative abstraction and mapping. arXiv:2603.29997. - Hatalis et al. (2025). Case-Based Reasoning for LLM agents. arXiv:2504.06943. - Hummel & Holyoak (1997, 2003). LISA. - Lee & Holyoak (2008); Holyoak, Lee & Lu (2010). Causal constraints on analogical inference. - Shinn et al. (2023). Reflexion. - Park et al. (2023). Generative Agents. - Zhao et al. (2024). ExpeL. - Wang et al. (2023). Voyager. - Wang et al. (2024). Agent Workflow Memory. - Huang et al. (ICLR 2024) and later surveys on limits of intrinsic LLM self-correction. --- # 30. Final engineering recommendation Do not start by reviving the dormant analogy tools. Start by building the **instrument**: 1. freeze PARB; 2. measure A0/A1/B; 3. encode structural signatures independently at storage time; 4. add a labelled structural retrieval channel; 5. force explicit mapping; 6. add causal/disanalogy verification; 7. score with downstream transfer metrics and NAV; 8. keep or toss each mechanism by pre-registered comparisons. If this works, Piper gains something materially different from "the LLM can make analogies": she gains a memory-linked, inspectable, falsifiable mechanism for finding the same causal or relational shape in a different place and deciding exactly what is safe to transfer.