another similar doc · Transfer

2026-10-03

Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.

Rendering…

# Transfer in LLM-Based Autonomous Agents — Research Report for Piper Oct 2, 2026 · @Bink ## 1. Executive summary **Short answer to the central question: partially yes, but the evidence is thinner than the agent-memory literature implies, and almost none of it measures *far* transfer with a proper no-exposure control.** Several LLM agents (ExpeL, CLIN, Agent Workflow Memory, Voyager, ICAL) show that an agent can extract something from experience in task A, store it, retrieve it on task B, and beat a matched agent without that experience. The gains are largest for near transfer (same environment, new task) and shrink as distance grows; AWM's cross-task / cross-website / cross-domain splits on Mind2Web are the closest thing in the literature to a transfer-distance curve for an LLM agent. Five findings matter most for Piper: 1. **The reviewer's "no machinery in use" finding is the default outcome, not a Piper bug.** Fifty years of cognitive science (Gick & Holyoak 1980; Gentner, Rattermann & Forbus 1993) show that people possess the relevant source knowledge but fail to *retrieve* it spontaneously when the surface differs. Retrieval is the bottleneck, not mapping. LLMs inherit a version of this: they map well when handed an analogue and retrieve poorly across surface boundaries. 2. **Abstraction beats episodes for far transfer; episodes beat abstraction for near transfer.** Comparing two or more cases to extract a schema (Gentner, Loewenstein & Thompson 2003) is the best-supported intervention in human transfer. Agent analogues (ExpeL insights, CLIN causal abstractions, AWM workflows, TroVE/ReGAL libraries) consistently induce from *multiple* episodes, not one. 3. **Nearest-neighbour retrieval is not optimal.** For compositional generalization, retrieving *structurally diverse* demonstrations beats retrieving the most similar ones (Levy, Bogin & Berant 2023). This directly answers the "closest example vs structurally similar distant example" question: retrieve for structural coverage, gate for applicability. 4. **Negative transfer is real and cheap to cause.** Irrelevant context degrades LLM reasoning (Shi et al. 2023); self-generated reflections without external verification often do not help (Huang et al. 2024). A transfer system without an applicability gate and an outcome ledger will import confident mistakes. 5. **Most apparent LLM "transfer" is confounded by pretraining.** Counterfactual-task studies (Wu et al. 2024; McCoy et al. 2023; Lewis & Mitchell 2024; Mystery Blocksworld in PlanBench) show performance collapsing when familiar surface forms are swapped for novel ones. Any Piper benchmark must use novel symbol systems or procedurally generated targets, or it will measure the base model's prior. **Recommendation.** Treat transfer as an *instrumented pipeline with a ledger*, not a capability claim: (a) induce abstractions only from ≥2 episodes plus a verified outcome; (b) store them with explicit preconditions and a usage record; (c) retrieve by structure-keyed signatures, not only embeddings; (d) gate by preconditions plus calibrated historical hit-rate; (e) run a seven-arm ablation on a five-band distance benchmark built on obfuscated or synthetic targets. The headline metric is a **transfer gradient**: transfer gain as a function of distance band, summarized by the area under that curve and the band at which gain crosses zero. **Verification note.** This report was written from model knowledge (cutoff mid-2026) without opening sources in this session. Paper titles, authors, years and arXiv IDs listed without a flag are ones I am confident exist. Specific numeric results are stated qualitatively unless I am confident; anything flagged **UNVERIFIED** should be checked before you rely on it. Where a claim is my inference rather than a paper's finding, it is labelled *(inference)*. ## 2. Definition and taxonomy of transfer **Working definition for Piper:** transfer is a *causal* effect — the change in performance on target task B attributable to prior experience with source task A, holding the agent otherwise fixed. Anything that cannot be expressed as a difference between an exposed and an unexposed condition is not yet transfer; it is a capability. **Core definitions by field** - **Machine learning (Pan & Yang 2010).** Given a source domain D\_S with task T\_S and a target D\_T with T\_T, transfer learning improves the target predictive function using knowledge from D\_S and T\_S, where D\_S ≠ D\_T or T\_S ≠ T\_T. Domain = feature space + marginal distribution; task = label space + conditional. Useful because it separates *domain* shift (same task, new inputs: Python bug → JS bug) from *task* shift (same inputs, new goal). - **Reinforcement learning (Taylor & Stone 2009).** Transfer is measured by changes in the target learning curve: jumpstart, asymptotic performance, total reward, transfer ratio, time-to-threshold. Defines transfer operationally by curves, which is the right frame for an agent that improves over repeated attempts. - **Continual learning (Lopez-Paz & Ranzato 2017).** Forward transfer = effect of learning task *i* on future tasks *j > i*; backward transfer = effect on earlier tasks *j < i*. Negative backward transfer is forgetting. - **Cognitive psychology (Perkins & Salomon 1992; Barnett & Ceci 2002).** Transfer is the application of learning in one context to another. Perkins & Salomon split *low-road* transfer (automatic, triggered by surface similarity of well-practised routines) from *high-road* transfer (deliberate, mindful abstraction and search for connections). Piper's "map the dependency envelope first" is a description of high-road transfer — which is exactly the kind that requires explicit machinery and fails silently without it. - **Educational psychology (Bransford & Schwartz 1999).** Classic transfer tests are "direct application" (sequestered problem solving). They propose *preparation for future learning*: transfer measured as faster or better learning in B, not one-shot success in B. This maps onto Taylor & Stone's learning-curve metrics. - **Intelligence measurement (Chollet 2019).** Intelligence is skill-acquisition efficiency over a scope of tasks, weighted by generalization difficulty, controlling for priors and experience. This is the most useful formal framing for Piper's "transfer gradient": it explicitly weights credit by how far a task is from what the system was exposed to. **Taxonomy: what belongs under transfer** | Concept | Belongs under transfer? | Note | | --- | --- | --- | | Near / far transfer | Yes — core | A continuum of source–target distance, not two categories (Barnett & Ceci 2002) | | Positive / negative / zero transfer | Yes — core | The sign of transfer gain | | Forward / backward transfer | Yes — core | Direction in a task sequence | | Cross-task, cross-domain, cross-modal, cross-lingual | Yes | Which axis of distance changed | | Compositional transfer | Yes | Recombining previously learned parts in new combinations | | Analogy | Yes — a mechanism | One route by which transfer happens: map relational structure from source to target | | Abstraction | Yes — a mechanism | A representation that makes transfer possible; not itself transfer | | Reuse (code, skills) | Yes — near-transfer special case | Literal reuse is the zero-adaptation end of the spectrum | | Systematic generalization | Related, distinct | Generalizing to novel combinations of known primitives; no source/target exposure manipulation | | OOD generalization | Related, distinct | Performance under distribution shift; not tied to a specific prior experience | | Memorization | Not transfer | It is the main confound: success on B because B (or a near-copy) was seen | | Retrieval of a fact | Not transfer by itself | Becomes transfer only if the retrieved item came from a different task and required adaptation | The key line *(inference)*: generalization is a property of a single trained system across inputs; transfer is a *counterfactual* claim about what a specific experience caused. Piper's evaluation should only use the word transfer when an exposure manipulation exists. **Mapping the seven sources of apparent transfer** | Source in the brief | How to isolate it | | --- | --- | | 1. Base LLM | Control arm with no exposure; counterfactual / obfuscated targets | | 2. Agent architecture | Same exposure, architecture on vs off | | 3. Memory / retrieval | Raw episodes stored and retrieved, no abstraction step | | 4. Explicit abstraction | Abstractions stored, episodes withheld | | 5. Analogy / search | Structure-keyed search on vs embedding-only retrieval | | 6. Training / adaptation | Out of scope for frozen weights; prompt/memory adaptation only | | 7. Apparent similarity | Matched-surface, mismatched-structure foils (Section 9, 13) | ## 3. Cognitive-science foundations **Human far transfer is rare without explicit support; the failure point is noticing, not applying.** This is the single most transferable lesson from the psychology literature to Piper. | Finding | Source | Evidence type | Implication for Piper | | --- | --- | --- | --- | | Transfer depends on shared "identical elements" between tasks | Thorndike & Woodworth 1901, *Psychological Review* 8; formalized as shared production rules by Singley & Anderson 1989, *The Transfer of Cognitive Skill* | Experimental (humans) | Near transfer is predictable from shared components; a component-overlap measure is a valid distance metric | | Subjects who read the "fortress" story rarely solve Duncker's radiation problem spontaneously; most do once given a hint that the story is relevant | Gick & Holyoak 1980, *Cognitive Psychology* 12:306–355 | Experimental | Retrieval failure dominates. Piper needs an explicit "where have I seen this structure?" trigger | | Two source analogues plus comparison produce a schema and much better transfer than one | Gick & Holyoak 1983, *Cognitive Psychology* 15:1–38 | Experimental | Induce abstractions from ≥2 episodes | | Analogy = mapping of relational structure, preferring systems of higher-order relations (systematicity) over object attributes | Gentner 1983, "Structure-mapping", *Cognitive Science* 7:155–170 | Theory + computational model (SME) | Represent problems as relations (what causes / depends on what), not features | | Retrieval is driven by surface similarity; mapping and soundness judgments are driven by structure | Gentner, Rattermann & Forbus 1993, *Cognitive Psychology* 25:524–575 | Experimental | Embedding retrieval will reproduce the human bias; add structure-keyed retrieval | | Two-stage retrieval: cheap content-vector filter (MAC) then structural match (FAC) | Forbus, Gentner & Law 1995, "MAC/FAC", *Cognitive Science* 19:141–205 | Computational model | Direct template for transfer search: embed → shortlist → structural re-rank | | Explicitly comparing two cases (analogical encoding) improves later transfer even with no prior expertise | Gentner, Loewenstein & Thompson 2003, *J. Educational Psychology* 95:393–408 | Experimental | Abstraction step should be contrastive: "what do these two episodes share?" | | Experts categorize problems by deep principle; novices by surface features | Chi, Feltovich & Glaser 1981, *Cognitive Science* 5:121–152 | Experimental | A good problem signature is principle-level ("conservation", "stale cache") not keyword-level | | Taxonomy of far transfer on nine dimensions: knowledge domain, physical context, temporal context, functional context, social context, modality, plus content dimensions | Barnett & Ceci 2002, *Psychological Bulletin* 128:612–637 | Review / framework | Ready-made multi-axis distance coordinates for Piper's bands | | Far-transfer claims in education are often not replicated | Detterman 1993, in *Transfer on Trial*; Sala & Gobet 2017, *Current Directions in Psychological Science* 26:515–520 | Review / meta-analysis | Expect far-transfer effects to be small; power the benchmark accordingly | | Low-road vs high-road transfer | Perkins & Salomon 1989, *Educational Researcher* 18(1):16–25 | Theory | Distinguish implicit (base model) from deliberate (machinery) transfer — the reviewer's exact distinction | | Transfer as preparation for future learning | Bransford & Schwartz 1999, *Review of Research in Education* 24:61–100 | Theory + studies | Measure learning speed on B, not just first-attempt success | | Case-based reasoning cycle: retrieve, reuse, revise, retain | Aamodt & Plaza 1994, *AI Communications* 7(1):39–59; Kolodner 1993, *Case-Based Reasoning* | Framework (AI) | The canonical engineering loop; Piper's proposed pipeline is a CBR cycle | **What this predicts for LLM agents** *(inference)*: the underlying model behaves like a subject with vast relevant knowledge and strong mapping ability but no reliable spontaneous retrieval across surface boundaries. That predicts exactly the reviewer's observation: transfer looks present in demos (where the analogue is in context) and absent as an operating process. ## 4. Transfer in machine learning and LLMs **Weight-level transfer is well established and measurable; in-context and agent-level transfer is real but fragile and heavily confounded by pretraining.** ### 4.1 Weight-level transfer (training-time) | Study | System tested | What it shows | Demonstrates transfer? | | --- | --- | --- | --- | | Yosinski et al. 2014, "How transferable are features in deep neural networks?" arXiv:1411.1792 | CNNs on ImageNet splits | Transferability falls as source and target tasks become dissimilar; quantifies layer generality | Yes, with A→B controls | | Zamir et al. 2018, "Taskonomy" arXiv:1804.08328 (CVPR best paper) | 26 vision tasks | Builds a full task-to-task transfer affinity matrix; transfer structure is learnable and non-uniform | Yes — the closest model of a measured transfer map | | Achille et al. 2019, "Task2Vec" arXiv:1902.03545 | Vision tasks | Task embeddings from Fisher information predict which source helps a target | Yes (predictive of transfer) | | Vu et al. 2020, "Exploring and Predicting Transferability across NLP Tasks" arXiv:2005.00770 (EMNLP) | BERT, 33 NLP tasks | Intermediate-task transfer gains often positive even across task types; task embeddings predict good sources | Yes, with paired controls | | Pruksachatkun et al. 2020, "Intermediate-Task Transfer Learning with Pretrained Models: When and Why Does It Work?" arXiv:2005.00628 (ACL) | RoBERTa, 110 source–target pairs | Many pairs show *negative* transfer; reasoning-heavy sources (e.g., QA) transfer best | Yes, including negative | | Hernandez et al. 2021, "Scaling Laws for Transfer" arXiv:2102.01293 (OpenAI) | Transformers, English text → Python | Defines *effective data transferred*: pretraining is worth a fitted amount of target data; follows a power law in fine-tune data and model size | Yes — gives an equation (Section 10) | | Wei et al. 2022, "Finetuned Language Models Are Zero-Shot Learners" (FLAN) arXiv:2109.01652 | 137B LM | Instruction tuning on held-out task *clusters* improves zero-shot on unseen clusters; benefit appears only at scale | Yes — cluster-level holdout is a good design pattern | ### 4.2 In-context and reasoning transfer (no weight change) - **Demonstrations transfer format more than content.** Min et al. 2022, "Rethinking the Role of Demonstrations" arXiv:2202.12837: randomizing labels in demonstrations barely hurts; the label space, input distribution and format drive most of the gain. *Implication:* a "transfer" arm that merely shows an example may be helping through format, not through the lesson. Include a format-matched control. - **Retrieval of similar demonstrations helps.** Liu et al. 2022, "What Makes Good In-Context Examples for GPT-3?" arXiv:2101.06804; Rubin et al. 2022, "Learning To Retrieve Prompts for In-Context Learning" (EPR) arXiv:2112.08633. - **But diversity beats similarity for compositional targets.** Levy, Bogin & Berant 2023, "Diverse Demonstrations Improve In-context Compositional Generalization" arXiv:2212.06800 (ACL). - **Self-generated analogues help.** Yasunaga et al. 2024, "Large Language Models as Analogical Reasoners" arXiv:2310.01714 (ICLR): prompting the model to recall relevant problems and solutions before solving beats zero-shot CoT and often matches hand-picked few-shot examples on math and code. *This is direct evidence that the base model has latent analogues it does not use unless asked* — supporting an explicit retrieval trigger even before any external memory. - **Analogy in LLMs: strong on familiar formats, weak on counterfactual ones.** Webb, Holyoak & Lu 2023, "Emergent analogical reasoning in large language models", *Nature Human Behaviour*, arXiv:2212.09196: GPT-3 matched or exceeded human participants on several zero-shot analogy tasks. Lewis & Mitchell 2024, "Using Counterfactual Tasks to Evaluate the Generality of Analogical Reasoning in LLMs" arXiv:2402.08955: on permuted-alphabet variants, human performance holds while GPT models degrade substantially. Webb et al. published a response disputing parts of this; the debate is unresolved. **Conflicting finding.** - **Counterfactual tasks expose recitation.** Wu et al. 2024, "Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks" arXiv:2307.02477 (NAACL): base-10 → base-9 arithmetic, altered chess rules, modified code semantics all degrade sharply. McCoy et al. 2023, "Embers of Autoregression" arXiv:2309.13638: accuracy tracks the probability of the output and the frequency of the task (e.g., rot-13 vs rot-2). Mirzadeh et al. 2024, "GSM-Symbolic" arXiv:2410.05229 (Apple): changing names and numbers in GSM8K items shifts accuracy; adding an irrelevant clause drops it sharply. - **Compositional limits.** Dziri et al. 2023, "Faith and Fate: Limits of Transformers on Compositionality" arXiv:2305.18654: multi-step composition often reduces to matching seen subgraphs. - **Planning under renaming.** Valmeekam et al. 2023, "PlanBench" arXiv:2206.10498: "Mystery Blocksworld" (same domain, obfuscated predicate names) collapses LLM planning accuracy relative to standard Blocksworld. *This is the cleanest available template for Piper's contamination-robust targets.* - **Self-correction without feedback does not reliably help.** Huang et al. 2024, "Large Language Models Cannot Self-Correct Reasoning Yet" arXiv:2310.01798 (ICLR). *Implication:* lessons extracted by self-reflection without an external outcome signal should not be trusted. ### 4.3 Cross-lingual and cross-language code transfer - Cassano et al. 2023, "MultiPL-E" arXiv:2208.08227: HumanEval/MBPP translated to \~18 programming languages; performance degrades from high-resource to low-resource languages. Gives a ready language-distance axis. - Cross-lingual NLP transfer (mBERT, XLM-R) is the most mature evidence that a skill learned in one surface language works in another, with degradation tracking typological and script distance — e.g., Pires, Schlinger & Garrette 2019, "How multilingual is Multilingual BERT?" arXiv:1906.01502; Lauscher et al. 2020, "From Zero to Hero" arXiv:2005.00633. **Net assessment.** Weight-level transfer has clean A→B controls. In-context transfer works mainly when the target is surface-familiar; the base model's apparent far transfer frequently disappears under counterfactual reformulation. For Piper, this means the "base LLM" arm will look strong on naturalistic tasks and weak on obfuscated ones — and the obfuscated ones are the ones that tell you whether machinery adds anything. ## 5. Near vs far transfer and quantifying transfer distance **Degradation with distance is well documented, but no single distance metric is validated for LLM agents; the practical answer is a multi-axis coordinate plus an empirically fitted curve.** ### 5.1 Evidence that transfer degrades with distance - Yosinski et al. 2014: transferability drops with source–target dissimilarity (vision, weight-level). - Taskonomy (Zamir et al. 2018): transfer affinity is structured; a hierarchy of tasks emerges where nearby tasks transfer well. - **Agent Workflow Memory** (Wang, Mao, Fried & Neubig 2024, arXiv:2409.07429): on Mind2Web's three test splits — *cross-task* (same websites), *cross-website* (same domain, new sites), *cross-domain* (new domains) — induced workflows improve success over the no-memory baseline on all three, with the benefit generally smaller as the split gets farther. Exact per-split deltas: **UNVERIFIED** — check Table 2 of the paper. This is the best existing three-band distance curve for an LLM agent. - Mind2Web itself (Deng et al. 2023, arXiv:2306.06070) was designed with these three generalization splits and shows baseline performance falling from cross-task to cross-domain. - Kirk et al. 2023, "A Survey of Zero-shot Generalisation in Deep Reinforcement Learning", *JAIR*, arXiv:2111.09794: frames generalization by which context factors differ between train and test; useful for designing controlled distance bands. ### 5.2 Candidate distance measures | Measure | Source | Captures | Weakness for Piper | | --- | --- | --- | --- | | Embedding distance of task descriptions | Common in ICL retrieval (Liu et al. 2022) | Surface/topical similarity | Exactly the surface bias Gentner et al. 1993 warn about | | Task embedding (Fisher / gradient-based) | Task2Vec (Achille 2019); Vu et al. 2020 | Similarity of what a model must learn | Needs weight access; not available with a frozen API model | | Empirical transfer affinity | Taskonomy (Zamir 2018) | Ground truth: how much A actually helps B | Expensive; it *is* the outcome, so circular as a predictor | | Shared identical elements | Thorndike; Singley & Anderson 1989 | Overlap of steps/production rules | Requires a decomposition of each solution | | Structural (relational) similarity | Gentner 1983; SME; MAC/FAC | Same relational graph regardless of surface | Requires a relational representation of the problem | | Barnett & Ceci dimensions | Barnett & Ceci 2002 | Multi-axis context distance (domain, physical, temporal, functional, social, modality) | Ordinal, hand-coded | | Generalization difficulty | Chollet 2019 | Information-theoretic: how much new information B requires given A and priors | Formal but not directly computable | ### 5.3 Recommended approach for Piper *(inference, grounded in the above)* Use **two** distance measures and keep them separate: 1. **Surface distance d\_s** — embedding distance between source and target task text, plus categorical changes (language, tool, file format). Cheap and automatic. 2. **Structural distance d\_r** — fraction of the solution's relational skeleton *not* shared. Concretely: write each item's solution as a small causal/procedural graph (nodes = roles like *untrusted input*, *numeric compare*, *guard*; edges = causes/precedes), then d\_r = 1 − (size of maximum common subgraph / size of larger graph). For a 30–50 item benchmark these graphs can be hand-authored once. Then define **bands** by Barnett & Ceci–style axes (same repo → same language → same paradigm → adjacent domain → distant domain) while holding d\_r near zero within a transfer pair. That design separates the two things you actually want to know: - *Transfer gradient along surface distance* (structure held constant): how far can Piper carry a lesson? - *Foil sensitivity along structural distance* (surface held constant): does Piper wrongly carry a lesson to look-alikes? The gradient itself: plot TG(band) with bootstrap confidence intervals; summarize with **AUC** across bands and **d₀**, the first band where the lower CI bound of TG crosses zero ("transfer horizon"). A decreasing logistic fit TG(d) = TG\_max / (1 + e^{k(d − d₅₀)}) gives a single "half-transfer distance" d₅₀ to track over time. ## 6. Abstraction, analogy, retrieval and transfer search **The literature supports a layered memory — episodes for near transfer, induced abstractions for far transfer — with retrieval that shortlists by content and re-ranks by structure.** ### 6.1 Transfer through abstraction Your "bad memory vs transferable abstraction" example is well supported. - **Human evidence.** Schemas induced from compared cases transfer; single cases don't (Gick & Holyoak 1983; Gentner, Loewenstein & Thompson 2003). - **Program-library induction.** Ellis et al. 2021, "DreamCoder" arXiv:2006.08381 (PLDI): wake-sleep loop that solves tasks, then *compresses* solutions into reusable library functions; the library accelerates later, harder tasks within a domain. Grand et al. 2024, "LILO" arXiv:2310.19791 (ICLR): LLM-guided search + Stitch compression + auto-documentation of abstractions; documentation improves the LLM's ability to use them. Wang, Neubig & Fried 2024, "TroVE: Inducing Verifiable and Efficient Toolboxes" arXiv:2401.12869 (ICML): LLM grows, uses and *trims* a function toolbox, verifying by execution agreement. Stengel-Eskin et al. 2024, "ReGAL: Refactoring Programs to Discover Generalizable Abstractions" arXiv:2401.16467 (ICML): refactors solutions into shared helpers; helpers improve accuracy on held-out tasks in the same domain. - **Agent insight extraction.** Zhao et al. 2024, "ExpeL: LLM Agents Are Experiential Learners" arXiv:2308.10144 (AAAI): the agent gathers success/failure trajectories, then extracts natural-language insights by *comparing* successes with failures and across successes. Includes an explicit transfer experiment: insights from HotpotQA applied to FEVER improve over the no-insight baseline. Magnitude: **UNVERIFIED** — check the paper's transfer table. - **Causal abstractions.** Majumder et al. 2023, "CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization" arXiv:2310.10134 (AI2): memory stores *causal* abstractions ("X is necessary for Y", "X does not contribute to Y") rather than raw advice; evaluated on ScienceWorld for adaptation (repeat trials), and generalization to new tasks and new environments, with a meta-memory that summarizes across trials. **This is the closest published design to what you want** — causal-form lessons, explicit generalization evaluation. - **Abstraction from demonstrations.** Sarch et al. 2024, "VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought" (ICAL) arXiv:2406.14596 (NeurIPS): turns noisy demonstrations into annotated examples with causal and task abstractions; improves on TEACh, VisualWebArena, Ego4D. - **Options / temporal abstraction.** Sutton, Precup & Singh 1999, "Between MDPs and semi-MDPs", *Artificial Intelligence* 112:181–211. Options = initiation set + policy + termination condition. *The initiation set is exactly the "preconditions" field in your transfer-memory schema.* **Can agents automatically extract transferable rules?** Yes, for near and medium transfer, with three robust caveats from the above: (1) induce from multiple episodes, ideally contrasting successes with failures; (2) verify against outcomes, not self-judgement; (3) prune — TroVE and DreamCoder both show unchecked libraries bloat and degrade retrieval. ### 6.2 Analogy - Structure-mapping theory (Gentner 1983) and the Structure-Mapping Engine (Falkenhainer, Forbus & Gentner 1989, *Artificial Intelligence* 41:1–63) give the canonical algorithm: align relational structure, prefer systematic mappings, project candidate inferences. - Hill et al. 2019, "Learning to Make Analogies by Contrasting Abstract Relational Structure" arXiv:1902.00120 (ICLR, DeepMind): networks trained with contrasting relational foils generalize analogies to novel domains far better than those trained on random negatives. *Directly supports training/evaluating with structural foils.* - Yasunaga et al. 2024 (analogical prompting) and Yu et al. 2024, "Thought Propagation: An Analogical Approach to Complex Reasoning with LLMs" arXiv:2310.03965 (ICLR): explicitly proposing and solving analogous problems, then reusing their solutions, improves results over CoT on shortest-path, creative writing and planning tasks. - Mitchell 2021, "Abstraction and Analogy-Making in Artificial Intelligence" arXiv:2102.10717: review arguing analogy is central to robust generalization and largely unsolved. **Does explicit analogy beat model-only?** On reasoning benchmarks, explicit analogical prompting beats plain CoT (Yasunaga; Yu). On counterfactual analogies, LLMs' mapping itself degrades (Lewis & Mitchell 2024). *Inference:* explicit analogy machinery helps mostly by fixing *retrieval*; it does not fix mapping on genuinely novel structures. ### 6.3 Retrieval and transfer | Arm | Representative evidence | What it predicts | | --- | --- | --- | | Model-only | Yasunaga 2024 (self-generated analogues help, so knowledge is latent) | Decent near transfer; poor spontaneous far transfer | | Episodic retrieval + model | Synapse (Zheng et al. 2024, arXiv:2306.07863, ICLR): retrieves exemplar *trajectories*; strong on same-site tasks. Reflexion (Shinn et al. 2023, arXiv:2303.11366): verbal reflections improve *repeat attempts* of the same task | Large near-transfer gains, little far transfer | | Retrieval + abstraction | ExpeL, CLIN, AWM, ICAL | Gains extend to new tasks / environments / domains, shrinking with distance | | Retrieval + analogy search | Thought Propagation; analogical prompting; MAC/FAC (human-model) | Best expected for far transfer; least directly tested in agents | **Is the closest example optimal?** No, not in general. Levy et al. 2023 show structural *coverage* beats nearest-neighbour for compositional generalization; Gentner et al. 1993 show nearest-by-surface retrieves misleading analogues; Shi et al. 2023 ("Large Language Models Can Be Easily Distracted by Irrelevant Context", arXiv:2302.00093) show near-but-irrelevant context hurts. Recommended policy *(inference)*: retrieve a small set mixing one surface-nearest episode with 1–2 structure-nearest abstractions from other domains, and let the gate (Section 9) drop non-applicable ones. ### 6.4 Transfer search mechanisms Published mechanisms for "where else have I solved this structure?": - **Two-stage MAC/FAC** — content vector filter, then structural evaluator (Forbus et al. 1995). - **Self-generated analogue recall** — ask the model for structurally similar solved problems first (Yasunaga 2024). - **Trajectory / plan similarity** — Synapse retrieves by task metadata embeddings; Voyager retrieves skills by embedding of the skill's *description* rather than its code, which is a cheap form of abstraction-keyed retrieval. - **Code-structure similarity** — AST/dataflow similarity (classic clone detection; CodeBERT-family embeddings). Good for near transfer inside code; blind across domains. - **Graph matching on causal/dependency graphs** — maximum common subgraph, or graph kernels; the structural re-ranker for d\_r in Section 5. *(Inference)* For Piper, the cheapest effective version is: every stored lesson carries a short **structural signature** written in a controlled vocabulary of roles and relations (e.g., `source:untrusted-optional → op:numeric-compare ⇒ failure:type-error; fix:guard-before-use`). Index signatures both by embedding and by exact role-set overlap. That gives MAC (embedding) and a crude FAC (role overlap) without a graph-matching engine. ## 7. Skill libraries **Every working skill library verifies a skill by execution before storing it, indexes by a natural-language description rather than raw content, and composes skills by calling them from new code. None of them has a principled applicability gate beyond retrieval similarity.** | System | What a skill is | Stored / indexed | Retrieved | Applicability | Adaptation | Success measure | Composes? | | --- | --- | --- | --- | --- | --- | --- | --- | | **Voyager** — Wang et al. 2023, arXiv:2305.16291 (NVIDIA/Caltech) | JavaScript function over Mineflayer API | Code, keyed by embedding of an LLM-written description | Top-k by embedding of task + environment feedback | Implicit: LLM decides whether to call | LLM writes new code calling skills; iterative refinement with execution errors | Tech-tree milestones, items; zero-shot generalization to new world with vs without library | Yes — skills call skills | | **JARVIS-1** — Wang et al. 2023, arXiv:2311.05997 | Multimodal memory of plans + situations | Plan text + visual state | Retrieval by task and observation similarity | Situation match | Re-planning from retrieved plans | Minecraft task success over 200+ tasks; improves with memory growth | Plans reuse sub-plans | | **DreamCoder** — Ellis et al. 2021 | Lambda-calculus function | Typed library | Neural recognition model guides search | Type system + search | Program search with library primitives | Tasks solved, search time | Yes, hierarchically | | **LILO** — Grand et al. 2024 | Compressed λ-abstraction with LLM-written name/docstring | Library + docs | LLM proposes programs using documented library | LLM judgement | Synthesis | Solve rate on held-out tasks in string editing, scene reasoning, graphics | Yes | | **TroVE** — Wang, Neubig & Fried 2024 | Python helper function | Toolbox, periodically trimmed | Model chooses: use / create / import | Self-consistency across sampled solutions | Function reuse in generated code | Accuracy and toolbox size vs baselines (MATH, TableQA, visual QA) | Yes | | **ReGAL** — Stengel-Eskin et al. 2024 | Refactored helper function | Library verified by re-execution | Retrieved for new queries | Execution verification at induction | Program generation | Held-out accuracy in LOGO, date reasoning, TextCraft | Yes | | **Agent Workflow Memory** — Wang et al. 2024 | Reusable web workflow: steps with variable slots | NL workflow text | Included in context (offline) or induced online from the test stream | Task-type matching by the LLM | Fill variables for new site / task | Mind2Web cross-task/website/domain; WebArena success | Yes — complex workflows built from simpler ones | | **ExpeL** — Zhao et al. 2024 | NL insight + successful trajectories | Insight list + trajectory pool | Insights always in context; trajectories by similarity | LLM judgement | In-context | HotpotQA, ALFWorld, WebShop; HotpotQA→FEVER transfer | Not explicitly | | **CLIN** — Majumder et al. 2023 | Causal abstraction ("X necessary for Y") | Memory per trial + meta-memory | Whole memory in context | Uncertainty words ("may", "should") in the abstraction | In-context | ScienceWorld: adaptation, generalization to unseen tasks and environments | No | | **Options** — Sutton, Precup & Singh 1999 | (initiation set, policy, termination) | Policy parameters | Available when state ∈ initiation set | **Explicit initiation set** | Policy reuse | Learning speed in RL | Yes — options over options | | **Policy sketches** — Andreas, Klein & Levine 2017, arXiv:1611.01796 (ICML) | Subpolicy tied to a symbol in a task sketch | Modular policies | By symbol | Symbol presence in sketch | Recombined in new sketches | Zero-shot / few-shot on new task sketches | Yes | | **SayCan** — Ahn et al. 2022, arXiv:2204.01691 (Google) | Robot primitive skill with value function | Skill set | LLM scores usefulness | **Learned affordance value: can this skill succeed from this state?** | None — pure selection | Real robot instruction success | Sequenced | **Two design takeaways for Piper.** 1. SayCan's split — *usefulness* (does the LLM think this skill helps?) × *affordance* (can it succeed here?) — is the only explicit two-factor applicability test in this set. Piper's gate should score both: relevance by the model, feasibility by preconditions and track record. 2. Options' initiation set is the formal object behind "preconditions". Voyager and its descendants dropped it in favour of LLM judgement; that is exactly where negative transfer enters. Note on Piper's existing tools *(inference)*: `reuse-search` is a retrieval component, `transplant` an adaptation component, `semantic-diff` a structural-comparison component, and `polyglot-advisor` a cost/applicability component. In the literature's terms they are the right parts. What's missing is (a) an induction step that produces abstractions, (b) a trigger that fires retrieval on every new problem, and (c) a ledger that records outcomes. Without (b), tools of this kind are dormant by design — which matches the reviewer's finding. ## 8. Meta-learning and continual learning with frozen weights **Most weight-level ideas translate to the memory level as *policies over what to store, retrieve, protect and replay*; the inner/outer-loop structure of meta-learning maps cleanly onto "solve tasks / tune the transfer machinery across tasks".** ### 8.1 Meta-learning | Concept | Source | Agent-level analogue with frozen LLM | | --- | --- | --- | | MAML: learn an initialization that adapts in few steps | Finn, Abbeel & Levine 2017, arXiv:1703.03400 (ICML) | Learn a **starting context** (which abstractions to always preload) that maximizes performance after k attempts on new tasks — tuned across a task distribution | | Meta-RL / RL² | Duan et al. 2016, arXiv:1611.02779; Wang et al. 2016, "Learning to reinforcement learn" arXiv:1611.05763 | The LLM's in-context adaptation already *is* an inner loop; the agent's outer loop tunes prompts, memory and retrieval policies | | Task distributions and held-out task families | FLAN cluster holdouts (Wei 2022); Meta-World (Yu et al. 2020, arXiv:1910.10897) | Split Piper's benchmark by task *family*, never by item, so the machinery is tuned on some families and evaluated on others | | Learned retrieval policies | EPR (Rubin 2022); learned demonstration selection | Learn a reranker over memory items using logged (retrieved item → outcome) pairs — feasible without touching LLM weights | | Curriculum learning | Bengio et al. 2009, ICML; Voyager's automatic curriculum | Order exposures from near to far; Voyager shows automatic curricula drive skill discovery | | Representation learning | Bengio, Courville & Vincent 2013, arXiv:1206.5538 | The "representation" Piper controls is the lesson schema and structural signature vocabulary — this is where design effort pays | **Realistic translation** *(inference)*: with frozen weights, the trainable parameters are (1) the abstraction-induction prompt, (2) the signature vocabulary, (3) retrieval weights (embedding vs structure vs track record), (4) the gate threshold, (5) what gets preloaded. Each can be tuned on a training set of transfer pairs and evaluated on held-out families. That is meta-learning in the outer loop, done with search or bandits rather than gradients. DSPy-style prompt optimization (Khattab et al. 2023, arXiv:2310.03714) is one practical tool for (1). ### 8.2 Continual learning | Mechanism | Source | Memory-level analogue | | --- | --- | --- | | Forward / backward transfer metrics | Lopez-Paz & Ranzato 2017, "Gradient Episodic Memory" arXiv:1706.08840 | Re-run earlier benchmark items after new lessons are added; BWT < 0 means new lessons are interfering | | Catastrophic forgetting / EWC | Kirkpatrick et al. 2017, arXiv:1612.00796 (PNAS) | **Protect** high-evidence lessons from being overwritten or merged by later induction; edits require stronger evidence the more a lesson has been validated | | Progressive networks | Rusu et al. 2016, arXiv:1606.04671 | Add new lessons as new entries with lateral links instead of rewriting old ones; never destructively edit | | Experience replay | Rolnick et al. 2019, "Experience Replay for Continual Learning" arXiv:1811.11682 (NeurIPS); Chaudhry et al. 2019 tiny episodic memories arXiv:1902.10486 | Periodically re-validate old lessons on stored episodes; demote lessons that no longer hold | | Modular / compositional networks | Andreas et al. 2016 neural module networks arXiv:1511.02799; Mendez & Eaton 2021, "Lifelong Learning of Compositional Structures" arXiv:2007.07732 (ICLR) | Skills as composable modules with typed interfaces | | Memory consolidation (complementary learning systems) | McClelland, McNaughton & O'Reilly 1995, *Psychological Review* 102:419–457; Kumaran, Hassabis & McClelland 2016, *Trends in Cognitive Sciences* | Fast episodic store + slow semantic store; an offline "sleep" pass that induces abstractions from batches of episodes (also DreamCoder's sleep phase) | | CL metrics beyond forgetting | Díaz-Rodríguez et al. 2018, "Don't forget, there is more than forgetting" arXiv:1810.13166 | Track memory size, compute, and efficiency alongside accuracy | | Agent memory architecture | Sumers et al. 2024, "Cognitive Architectures for Language Agents (CoALA)" arXiv:2309.02427 (TMLR); Packer et al. 2023, MemGPT arXiv:2310.08560; Park et al. 2023, Generative Agents arXiv:2304.03442 | CoALA's working / episodic / semantic / procedural split is the right vocabulary for Piper's memory stores | A memory-level continual-learning benchmark for LLM agents exists in early form: Wu et al. 2024, "StreamBench: Towards Benchmarking Continuous Improvement of Language Agents" arXiv:2406.08747 (**UNVERIFIED** details) — agents receive a stream of tasks with feedback and are scored on improvement over time. ## 9. Negative transfer and applicability gating **Negative transfer is frequent in both ML and LLM settings, and the best-supported defence is not a better similarity score but an explicit precondition check plus an empirical track record per lesson.** ### 9.1 Evidence - Rosenstein, Marx, Kaelbling & Dietterich 2005, "To Transfer or Not To Transfer", NIPS Workshop on Inductive Transfer: when source and target are too dissimilar, transfer hurts — an early, widely cited demonstration. - Wang, Dai, Póczos & Carbonell 2019, "Characterizing and Avoiding Negative Transfer" arXiv:1811.09751 (CVPR): formal definition — negative transfer occurs when the target error with source data exceeds the error without it — and a discriminator-gate method to filter harmful source samples. - Zhang et al. 2022, "A Survey on Negative Transfer" arXiv:2009.00909 (*IEEE/CAA J. Automatica Sinica*): taxonomy of causes (domain divergence, source quality, target data quantity, algorithm). - Pruksachatkun et al. 2020: many intermediate-task pairs give negative transfer in NLP fine-tuning. - Multi-task interference: Yu et al. 2020, "Gradient Surgery for Multi-Task Learning" (PCGrad) arXiv:2001.06782 — conflicting gradients between tasks. - **LLM context-level:** Shi et al. 2023 (irrelevant context sharply hurts arithmetic reasoning); Mirzadeh et al. 2024 (GSM-NoOp: a single irrelevant but plausible clause degrades accuracy); Huang et al. 2024 (unverified self-correction can turn right answers wrong). These are the context-level analogues of negative transfer: a retrieved lesson is just more context, and plausible-but-wrong context is the damaging kind. - **Agent memory:** **UNVERIFIED** — I recall a 2025 study (Xiong et al., on memory management in LLM agents) reporting an "experience-following" effect where agents copy retrieved past outputs closely, so erroneous memories propagate and degrade later performance unless memory addition is quality-filtered. Check before citing. - **Humans:** Gentner et al. 1993 — surface-similar but structurally different analogues are retrieved most readily and lead to errors; Einstellung (Luchins 1942, *Psychological Monographs* 54(6)) — a practised method blocks seeing a simpler one. ### 9.2 How an agent can decide "looks similar, but don't reuse" Published mechanisms, roughly in increasing strength: 1. **Similarity threshold** (most agent systems). Weak: similarity is exactly what misleads. 2. **Explicit initiation set / preconditions** (options; CLIN's "necessary for" form; SayCan affordances). Check each precondition against the current problem; any fail → reject. 3. **Structural alignment check** (SME-style). Ask: does every role in the lesson's signature map to an entity in the new problem, and do the relations hold? Unmapped roles = do not apply. 4. **Verification by execution / test** (Voyager, TroVE, ReGAL). Apply in a sandbox; adopt only if tests pass. Strongest for code, unavailable for many non-code targets. 5. **Empirical track record** (Wang et al. 2019's learned gate is the ML analogue). Maintain per-lesson success rates conditional on match type; reject when the posterior probability of help is low. 6. **Counterfactual comparison** — solve with and without the lesson, compare via a verifier; the most expensive but the only direct estimate of transfer gain on the live instance. Recommended gate for Piper *(inference)*: G(lesson, problem) = 1 if (all hard preconditions hold) ∧ (all failure conditions absent) ∧ (structural role coverage ≥ θ\_r) ∧ (posterior P(help | match type) ≥ θ\_p); when the problem is high-stakes and a verifier exists, require (6). Log every gate decision and its outcome — those logs are what calibrate θ. ## 10. Metrics **Use Taylor & Stone's curve metrics for multi-attempt targets and Lopez-Paz & Ranzato's matrix metrics for task sequences; both work over API calls because they need only scores, not gradients.** **Taylor & Stone 2009** ("Transfer Learning for Reinforcement Learning Domains: A Survey", *JMLR* 10:1633–1685). With f\_T(t) = performance at attempt t with transfer, f\_0(t) without: ```latex \text{Jumpstart} = f_T(1) - f_0(1) ``` ```latex \text{Asymptotic gain} = f_T(\infty) - f_0(\infty) ``` ```latex \text{Transfer ratio} = \frac{\text{area under } f_T - \text{area under } f_0}{\text{area under } f_0} ``` Time-to-threshold: the number of attempts (or tokens, or tool calls) to reach a fixed score, compared across arms. For Piper this is the natural "learning efficiency" metric and matches Bransford & Schwartz's preparation-for-future-learning view. **Lopez-Paz & Ranzato 2017** (GEM, arXiv:1706.08840). Train/expose on tasks 1…T in order; R\_{i,j} = test score on task j after exposure to task i; b̄\_j = score on j with no exposure: ```latex \text{ACC} = \frac{1}{T}\sum_{i=1}^{T} R_{T,i} ``` ```latex \text{BWT} = \frac{1}{T-1}\sum_{i=1}^{T-1} \left(R_{T,i} - R_{i,i}\right) ``` ```latex \text{FWT} = \frac{1}{T-1}\sum_{i=2}^{T} \left(R_{i-1,i} - \bar{b}_i\right) ``` FWT is precisely your Transfer Gain averaged over a sequence. BWT detects whether new lessons damage old performance. **Negative transfer gap** (Wang et al. 2019): NTG = ε\_T(with source) − ε\_T(without source); positive NTG = negative transfer. **Effective data transferred** (Hernandez et al. 2021): D\_T = k·(D\_F)^α·(N)^β, where D\_F is fine-tuning data, N model size. Not directly usable with frozen weights, but its idea — express transfer as "equivalent target examples" — is: measure how many target-domain worked examples the control arm needs to match the transfer arm. That gives an interpretable unit ("lesson X is worth \~3 target examples"). **Forgetting** (Chaudhry et al. 2018, "Riemannian Walk" arXiv:1801.10112): F\_j = max over earlier steps of R\_{l,j} − R\_{T,j}. ### Which work for Piper (API-only agent) | Metric | Usable? | Notes | | --- | --- | --- | | Transfer gain TG = P(B∣A) − P(B∣¬A) | Yes — primary | Paired by item; report mean and bootstrap CI | | Normalized TG = TG / (1 − P(B∣¬A)) | Yes | Fraction of available headroom captured; avoids ceiling artifacts | | Jumpstart, time-to-threshold, transfer ratio | Yes | Needs multi-attempt targets with feedback | | FWT / BWT matrix | Yes | Needs a sequential protocol and re-tests | | Negative-transfer rate | Yes | Fraction of items where the transfer arm fails but the control succeeds (paired discordance) | | Gate precision / recall | Yes | Precision = applied lessons that helped; recall = helpful lessons that were applied | | Equivalent-examples | Yes, costly | Dose–response on target examples in the control arm | | Task2Vec / Fisher distance | No | Needs weights | | Effective data transferred (as fitted law) | No | Needs fine-tuning | Statistics: TG is a paired difference, so use McNemar's test for binary outcomes and a paired bootstrap for continuous scores; run each item ≥5 seeds per arm, since LLM run-to-run variance on agent tasks is large. ## 11. Experimental design and existing benchmarks **Your two-arm design is sound but under-controlled; add a placebo-exposure arm, a foil arm, obfuscated targets, and family-level holdouts.** ### 11.1 Confounds and controls | Confound | Control | Source of the technique | | --- | --- | --- | | Base-model prior / contamination | Novel symbol systems: rename identifiers, invent DSLs, permute operators, obfuscate predicates | PlanBench Mystery Blocksworld (Valmeekam 2023); counterfactual tasks (Wu 2024); Lewis & Mitchell 2024 | | Training-data leakage of targets | Procedurally generated items; post-cutoff items | LiveCodeBench (Jain et al. 2024, arXiv:2403.07974) uses release dates to exclude contaminated problems; Procgen (Cobbe et al. 2020, arXiv:1912.01588) | | Exposure helps via format, length, or "priming to be careful" rather than the lesson | **Placebo arm**: exposure to an unrelated episode of identical length and format | Min et al. 2022 showed format effects dominate demonstrations | | Target merely similar to source (item-level leak) | Hold out by task *family*, not item; ensure target cannot be solved by copying source | FLAN cluster holdout; Mind2Web splits | | Prompt leakage of the answer | Lesson text is generated before the target is chosen; audit for target-specific tokens | Standard in ICL evaluation | | Surface similarity driving reuse | **Foil arm**: surface-similar target whose structure differs and where the lesson is wrong | Gentner et al. 1993; Hill et al. 2019 contrasting foils | | Order and fatigue effects | Counterbalance; randomize item order per seed | Standard within-subject design | | Random variance | ≥5 seeds per item per arm; paired analysis; pre-registered primary metric | — | | Evaluator bias | Objective scoring (tests, exact match, numeric tolerance); blind any LLM judge to arm | — | **Arms for each item:** (1) control — no exposure; (2) placebo — irrelevant exposure of matched length; (3) transfer — exposure to A; (4) oracle — the abstracted lesson given directly (upper bound on what the induction step could achieve). Transfer gain is (3) − (2), not (3) − (1). The gap (4) − (3) measures loss in Piper's own abstraction and retrieval; (2) − (1) measures pure context effects. ### 11.2 Existing benchmarks and adaptability for Piper | Benchmark | What it measures | Adapt for Piper? | | --- | --- | --- | | ARC / ARC-AGI (Chollet 2019) and ARC-AGI-2 | Few-shot abstraction of novel grid transformations with minimal prior knowledge | Partially: good as very-far-band targets; source-side lessons rarely map, so use for gradient's far end and foil checks | | ConceptARC (Moskvichev, Odouard & Mitchell 2023, arXiv:2305.07141) | ARC tasks grouped by concept (inside/outside, same/different) with variations | Yes: concept groups give natural A→B pairs within one concept | | SCAN (Lake & Baroni 2018, arXiv:1711.00350), COGS (Kim & Linzen 2020, arXiv:2010.05465), CFQ (Keysers et al. 2020, arXiv:1912.09713) | Systematic / compositional generalization | Use as a template for building novel-DSL targets, not as transfer tests directly | | Mind2Web (Deng et al. 2023) | Web agents; cross-task / cross-website / cross-domain splits | Yes: already a three-band gradient; replicate AWM's induction with Piper's machinery | | WebArena (Zhou et al. 2023, arXiv:2307.13854) | Functional web tasks in self-hosted sites | Yes, for online-induction transfer across sites | | ScienceWorld (Wang et al. 2022, arXiv:2203.07540) | Text-world science experiments; CLIN's testbed | Yes: CLIN's adaptation / generalization protocol is directly reusable for the "measurement technique → next instrument" claim | | ALFWorld (Shridhar et al. 2021, arXiv:2010.03768) | Household text tasks | Near-transfer only; heavily used, likely contaminated | | SWE-bench (Jimenez et al. 2024, arXiv:2310.06770); SWE-bench Multilingual / Multi-SWE-bench | Real repo issue resolution; non-Python variants | Yes for Python→other-language band, but contamination risk is high | | MultiPL-E (Cassano 2023); Aider polyglot | Same problems across many languages | Yes: hold problem constant, vary language — clean language-distance axis | | CRUXEval (Gu et al. 2024, arXiv:2401.03065) | Code execution reasoning | Moderately: can host invented-semantics variants | | PlanBench (Valmeekam 2023) | Planning with standard and obfuscated domains | Yes: obfuscation template for every band | | Procgen; Meta-World; Continual World (Wołczyk et al. 2021, arXiv:2105.10919) | RL generalization and continual-learning transfer with FWT/BWT | Metrics and protocol only | | Alchemy (Wang et al. 2021, arXiv:2102.02926, DeepMind) | Meta-RL with latent causal structure that changes per episode | Concept useful: "discover the causal structure, reuse it" — good model for scientific-diagnosis items | | Letter-string analogies; counterfactual variants (Webb 2023; Lewis & Mitchell 2024) | Analogical mapping | Yes as a sanity check on mapping ability independent of retrieval | | StreamBench (**UNVERIFIED**) | Continuous improvement of agents over a task stream | Yes, as protocol for online induction | No existing benchmark measures what you want end to end: an *agent-level, multi-band, exposure-controlled* transfer gradient. You will need to build it (Section 14), borrowing Mind2Web's split logic, PlanBench's obfuscation, CLIN's protocol, and GEM's metrics. ## 12. Strongest findings, conflicts, and the central question **The full causal chain (experience → extract → store → retrieve in a different task → adapt → improve vs no-experience control) has been demonstrated for LLM agents, but only up to roughly "same kind of environment, different task or site". No published agent study cleanly demonstrates it across the far bands in your matrix.** ### 12.1 Strongest evidence for the causal chain | Study | Chain steps shown | Farthest distance tested | Control | | --- | --- | --- | --- | | AWM (Wang et al. 2024) | All six | Cross-domain websites | No-memory agent, same model | | CLIN (Majumder et al. 2023) | All six | Unseen tasks and unseen environments within ScienceWorld | No-memory / Reflexion baselines | | ExpeL (Zhao et al. 2024) | All six | Different QA dataset (HotpotQA → FEVER) | No-insight agent | | Voyager (Wang et al. 2023) | All six | New Minecraft world, new tasks | Same agent without library; AutoGPT ± library | | ICAL (Sarch et al. 2024) | All six | Across tasks within each benchmark | Raw-demonstration and no-memory baselines | | TroVE / ReGAL | Extract + store + reuse on held-out tasks | Held-out tasks in the same domain | No-library baselines | | Gick & Holyoak 1983; Gentner et al. 2003 (humans) | All, with abstraction | Far (military story → medical problem; negotiation cases) | No-source and single-source groups | ### 12.2 Conflicting or negative findings - **Analogy in LLMs:** Webb et al. 2023 (emergent, human-level) vs Lewis & Mitchell 2024 (fails under counterfactual alphabets). Unresolved; most likely both are right about different regimes — familiar vs novel surface encodings. - **Reasoning generality:** strong benchmark scores vs counterfactual collapse (Wu 2024; McCoy 2023; GSM-Symbolic 2024; Mystery Blocksworld). Newer reasoning models narrow but do not eliminate these gaps (**UNVERIFIED** for current frontier models; re-test). - **Self-reflection:** Reflexion reports gains with environment feedback; Huang et al. 2024 find intrinsic self-correction without external feedback does not help or hurts. *Reconciliation:* lessons help when grounded in external outcomes. - **Demonstration content:** retrieval of similar examples helps (Liu 2022) yet labels barely matter (Min 2022) — so part of "transfer" from examples is format transfer. - **Human far transfer:** large historical claims (Latin, chess, brain training) largely fail meta-analysis (Sala & Gobet 2017). Expect small far-band effects; the absence of a significant far-band result is informative, not a failure of the study. ### 12.3 Answer to the central question *Can an autonomous LLM agent acquire an experience in one domain, transform it into a reusable representation, recognize its relevance in a substantially different domain, adapt it, and measurably outperform the same agent without the experience?* - **Acquire and transform:** yes, demonstrated (ExpeL, CLIN, AWM, ICAL, library induction). - **Outperform on a different task:** yes, demonstrated at near and medium distance with proper controls. - **Recognize relevance in a *substantially* different domain:** weakly evidenced. Existing systems mostly retrieve within a domain or preload all memories. Spontaneous cross-domain retrieval is the unmeasured step — and the human literature predicts it is the hardest. - **Beyond the base model:** the evidence that machinery adds value *over what the base LLM would do anyway* is clearest when targets are novel (obfuscated, post-cutoff, procedurally generated). On familiar targets, the base model's latent analogues (Yasunaga 2024) compete with explicit memory. So the honest status is: **demonstrated for near/medium transfer, open for far transfer.** Piper's benchmark, if built as below, would produce a result the literature does not currently have. ## 13. Proposed Piper transfer architecture and transfer memory **Keep your pipeline's order, but split it into an online loop (retrieve → gate → apply → verify) and an offline loop (induce → consolidate → recalibrate), and make the trigger unconditional.** The literature-backed design is a case-based-reasoning cycle (Aamodt & Plaza 1994) with MAC/FAC retrieval, option-style preconditions, SayCan-style two-factor gating, DreamCoder-style offline induction, and a per-lesson outcome ledger. ### 13.1 Critique of the proposed pipeline 1. **"Transfer opportunity detector" is the wrong gate position.** Human data show people fail to notice relevance; a detector that decides *whether* to search will reproduce that failure. Make retrieval run on every non-trivial problem and let the applicability gate decide. Cost is a cheap embedding lookup plus a re-rank. 2. **"Extract reusable abstraction" should not happen at solve time.** Abstraction from a single episode under time pressure produces the "line 172" memories. Move induction offline, require ≥2 episodes and an external outcome, and contrast successes with failures (ExpeL, Gentner 2003). 3. **"Identify shared structure" needs a representation.** Without a structural signature written at storage time, this step becomes another LLM judgement. Store signatures; re-rank on them. 4. **"Measure outcome" must include the counterfactual when possible.** The ledger needs to know whether the lesson *helped*, not just whether the task succeeded. 5. **Missing: forgetting and interference control.** Add replay-based revalidation and protection of well-evidenced lessons (Section 8.2). ### 13.2 Recommended loops [embedded content: Piper transfer engine · online loop and offline loop\] Retrieval fires on every problem; the gate, not a detector, decides what is used. Verified outcomes update the ledger, which sets the gate's thresholds, while offline induction turns clusters of episodes into lessons. **Online, per problem:** 1. Encode the problem: free text plus a structural signature in the controlled vocabulary. 2. Retrieve: embedding shortlist (k ≈ 20) across *all* domains → structural re-rank → keep a mixed set (1 nearest episode + 1–3 structurally nearest lessons). 3. Gate each candidate: preconditions, failure conditions, role coverage, posterior P(help | match type). 4. Adapt: explicit role mapping ("source role *optional external value* = target role *env var that may be unset*"). Unmapped roles fail the gate. 5. Execute; verify with tests or scorer. When a verifier exists and stakes are high, also run the no-lesson variant. 6. Write an episode record and one ledger row per retrieved candidate (applied / rejected / outcome). **Offline, periodic ("sleep"):** 1. Cluster recent episodes by signature; induce or refine lessons only where ≥2 episodes agree. 2. Verify new lessons by replay on stored episodes (TroVE / ReGAL style). 3. Recompute per-lesson calibration from the ledger; demote or retire lessons whose help rate drops. 4. Re-run a sample of the benchmark to compute BWT; block merges that cause interference. ### 13.3 Transfer-memory record Fields grounded in the literature (source in brackets): ```json { "id": "lesson-0412", "version": 3, "abstraction": "When a value comes from an optional external source, establish presence before any operation that assumes a type.", "form": "causal", "signature": { "roles": ["source:untrusted-optional", "op:type-assuming", "failure:silent-or-crash"], "relations": ["source -> op", "absent(source) => failure"], "fix_pattern": "guard-before-use" }, "preconditions": ["a value can be absent at runtime", "a downstream step assumes presence or type"], "failure_conditions": ["absence is a valid, handled state downstream", "value is compile-time guaranteed"], "role_mapping_hints": {"source:untrusted-optional": ["API field", "env var", "sensor dropout", "survey non-response"]}, "source_episodes": ["ep-2026-08-11-py-null", "ep-2026-08-19-js-undefined"], "source_domains": ["python", "javascript"], "evidence": [{"episode": "ep-2026-08-19-js-undefined", "verifier": "unit-tests", "result": "pass"}], "ledger": { "structural_match": {"applied": 10, "helped": 8, "hurt": 0}, "surface_match_only": {"applied": 7, "helped": 1, "hurt": 3}, "rejected_by_gate": 14 }, "posterior_help": {"structural_match": 0.75, "surface_match_only": 0.22}, "max_band_succeeded": 3, "protected": true, "created": "2026-08-20", "last_validated": "2026-09-28", "supersedes": "lesson-0412@2" } ``` Why each non-obvious field exists: - **form = causal** — CLIN found causal abstractions ("X necessary for Y") more useful than free advice. - **signature** — structural index for FAC re-rank (Forbus 1995) and for computing structural distance. - **preconditions / failure\_conditions** — options' initiation and termination sets; Section 9 gate. - **role\_mapping\_hints** — SME candidate inferences; makes cross-domain mapping explicit and auditable. - **source\_episodes ≥ 2, multiple domains** — Gick & Holyoak 1983; Gentner 2003. - **ledger split by match type** — exactly the calibration you proposed ("8/10 structural vs 1/7 superficial"). A Beta(1,1) prior per match type gives posterior\_help = (helped + 1) / (applied + 2), which is what the shown values compute. - **max\_band\_succeeded** — the lesson's own position on the transfer gradient. - **protected / supersedes** — EWC- and progressive-network analogues: well-evidenced lessons are versioned, not overwritten. **Can transfer knowledge become calibrated?** Yes — the ledger makes each lesson a small Bayesian model of its own applicability. Calibration itself can be scored: bin gate decisions by posterior\_help and check the observed help rate per bin (a reliability diagram / Brier score). This is ordinary forecast calibration applied to the gate. ## 14. Proposed transfer-gradient benchmark and ablations **Build it as six lesson *families*, each with one source episode and five targets at increasing distance, plus one structural foil: 36 items. Holding the lesson fixed within a family is what turns pass/fail into a gradient.** ### 14.1 Design principles - **One lesson, five bands.** Each family's source episode A is identical across bands; only the target moves. The gradient is then TG(band) per family, averaged across families. - **Bands** (Barnett & Ceci-style, structure held constant): | Band | Name | What changes from A | | --- | --- | --- | | B1 | Very near | Different repo, same language and paradigm | | B2 | Near | Different programming language | | B3 | Medium | Different technical medium: shell, SQL, config, logs | | B4 | Far | Different technical domain, mostly non-code: ops, data lineage, instruments, org graphs | | B5 | Very far | Non-technical: experimental science, forecasting, research verification | - **Two exposure channels.** *In-context*: A's episode is in the same context as B — the only channel through which a base LLM can transfer. *Cross-session*: A is solved in an earlier session; B runs in a fresh context and anything from A must travel through Piper's memory. The cross-session channel is the one that tests machinery. - **Contamination control.** Every target comes from a seeded generator with invented identifiers, invented DSL syntax where possible, synthetic data and synthetic corpora (PlanBench-style obfuscation). Each family also has a naturalistic variant, so the obfuscation penalty itself is measured. - **Latin-square placebo.** Family k's source episode serves as the placebo exposure for family k+1, so placebos match length and format exactly. - **Measured distance.** For each item, compute d\_s (embedding distance of A and B task text) and d\_r (relational-skeleton mismatch from the hand-authored signature graphs, Section 5.3). Bands are the design variable; d\_s and d\_r are recorded covariates, so the gradient can also be refit on continuous distance. ### 14.2 Arms per item | Arm | Exposure before B | Purpose | | --- | --- | --- | | Control | None | Baseline P(B ∣ ¬A) | | Placebo | Another family's A (matched length and format) | Removes generic priming / format effects | | Transfer | Solve A with feedback | Experience path; TG = Transfer − Placebo | | Oracle | The canonical lesson text, no episode | Upper bound for the induction step; Oracle − Transfer = loss in Piper's abstraction and retrieval | | Foil | Solve A, then a surface-similar target where the lesson is wrong | Measures negative transfer and gate quality | ### 14.3 One family fully specified (F1) 1. **Domain A.** Python service in a synthetic repo: a rate limiter reads `cfg.get("zq_limit")` and compares it with an integer; requests crash with a TypeError when the key is absent. Piper must fix it; hidden tests confirm. 2. **Transferable lesson.** When a value originates from a source that can be absent, establish presence (or a typed default) before any operation that assumes its type. 3. **Domain B (per band).** B1: second Python repo, optional CLI argument fed to a numeric threshold. B2: TypeScript client where a fetched JSON field may be missing and is used in arithmetic, silently producing NaN. B3: Bash deploy script where an unset variable sits inside a numeric test and an `rm` path. B4: SQL report where `NOT IN (subquery)` returns no rows because the subquery can yield NULL. B5: a synthetic survey dataset where non-response is coded as 0 and pulls a mean down; Piper must produce the corrected estimate. 4. **Shared structure.** Optional source → type-assuming operation → silent or loud failure → guard or explicit missing-value handling. Same relational skeleton in all five; d\_r ≈ 0 by construction. 5. **Control condition.** Fresh agent, no exposure, same target, same budget. 6. **Transfer condition.** Agent solved A in a prior session (cross-session) or earlier in context (in-context). 7. **Scoring.** B1–B3: hidden tests (pass/fail) plus a test that exercises the absent case. B4: exact result set. B5: estimate within ±1% of ground truth and correct handling statement. Secondary: tokens and tool calls to solution. 8. **Contamination risks.** NULL-in-NOT-IN and unset-variable bugs are well known; the base model may solve B3 and B4 without exposure. Mitigate with invented table/variable names, nested indirection, and the naturalistic-vs-obfuscated comparison. 9. **Negative transfer looks like.** Foil: a Python target where `None` is a meaningful sentinel that must propagate (a test asserts it). Applying the guard breaks that test. Also: in B5, imputing a value where the correct move is to exclude non-response. 10. **Conclusion justified.** If TG(B1..B3) > 0 and TG(B4..B5) ≈ 0 with foil errors low, Piper carries this lesson across languages and media but not out of software; if Oracle ≫ Transfer at far bands, the failure is in abstraction or retrieval, not mapping. ### 14.4 Item matrix (six families × five bands + foils) Fields 5, 6 and 10 follow the template above for every item. | Family and lesson | B1 very near | B2 near | B3 medium | B4 far | B5 very far | Foil (lesson is wrong) | Scoring | | --- | --- | --- | --- | --- | --- | --- | --- | | **F1 Optional-source guard** (as above) | Optional CLI arg → threshold | TS: missing JSON field → NaN | Bash: unset var in test and rm path | SQL: NULL in NOT IN subquery | Survey non-response coded 0 | None is a valid sentinel | Hidden tests; exact rows; estimate ±1% | | **F2 Bisect the change set**: between a known-good and known-bad state, halve the difference to localize the cause | git history of a Python repo, 64 commits | JS lockfile: 32 upgraded packages | 40-line config; one line breaks a service | 16 feature flags rolled out; latency regression in a simulator | 8 protocol changes to a simulated assay; ≤4 runs allowed | Failure needs two changes together (non-monotone); pairwise search required | Culprit found within probe budget; probes used | | **F3 Calibrate against a reference**: measure known references, fit offset and gain, correct, verify on a held-out reference | Second sensor class in Python | JS client–server clock skew | Shell benchmark: subtract harness overhead before comparing | Simulated instrument with hidden offset and gain, 6 query budget | Forecaster's resolved predictions; recalibrate new forecasts | Nonlinear instrument; two-point linear fit is wrong | Error vs ground truth below tolerance; Brier score for B5 | | **F4 Map the dependency envelope**: before moving a component, compute what it depends on and what depends on it | Move module between Python packages | Port Python utility to TS incl. stdlib semantics (regex, dates) | Rename a SQL column with dependent views and triggers | Reassign a role in a synthetic org graph; list broken processes | Remove a supplier in a synthetic supply network; list affected products | Copy (not move): dependents unaffected; full analysis wastes budget and causes needless edits | F1 score of affected set vs ground truth; foil: diff size, tests | | **F5 Attack the check, not the claim**: a check built with the thing it checks misses inputs its author didn't imagine; construct one | Passing Python test suite; find a failing input | JS email/URL validator regex; find bypass | Bash filename handling; spaces, newlines, leading dash | Synthetic access-control policy; find an escalation path | Synthetic study with a confound; name the observation that would falsify the claim | Exhaustive check over a tiny finite domain; correct answer is "no counterexample" | Counterexample validated by executor; rubric for B5 | | **F6 Trace to the original source**: copies are not independent evidence; find where a value or claim originates | Python config set in 4 layers; find effective source | Node env precedence (dotenv vs process env) | Duplicate alerts from 3 monitors; find the single root event | Dashboard metric derived from a deprecated table | 6 synthetic articles, 5 cite one original that misquotes a number | Independent measurements that agree; should count as corroboration | Root identified exactly; foil: correct independence call | That is 30 transfer items + 6 foils. With 5 seeds and 5 arms that is 900 runs per channel, which is affordable on API budgets and gives reasonable power for band-level TG differences of \~0.15 (inference; run a pilot to estimate variance). ### 14.5 Harness (what to implement) 1. `generate(family, band, seed)` → target task, hidden tests or scorer, signature graph. 2. `expose(agent, arm, family, seed)` → runs source, placebo, oracle or nothing; for cross-session, persists only memory state. 3. `solve(agent, target, budget)` → transcript, final artifact, tokens, tool calls, retrieval log, gate log. 4. `score(target, artifact)` → objective result. 5. Store one row per run: family, band, seed, arm, channel, ablation config, d\_s, d\_r, score, cost, retrieved lesson ids, gate decisions. 6. Analysis: TG(band) = mean(Transfer − Placebo) paired by (family, seed), bootstrap CIs; AUC over bands; transfer horizon d₀; negative-transfer rate; foil error rate; gate precision and recall; Oracle − Transfer gap per band. ### 14.6 Ablations All arms use the same base model and budget, cross-session channel unless stated. | Config | Enabled | Key comparison | What a positive result means | | --- | --- | --- | --- | | A Base LLM only | Nothing persists between sessions; in-context channel only | A in-context vs A cross-session (should be ≈0) | Size of implicit base-model transfer | | B + episodic retrieval | Raw episodes stored; embedding retrieval | B − A | Value of remembering at all; expected near-band only | | C + semantic abstraction | Induced lessons stored; episodes withheld | C − B by band | Abstraction extends the gradient | | D + analogy search | C plus structural signatures and re-rank | D − C at B4–B5 | Structure-keyed retrieval fixes cross-domain noticing | | E + skill library | Executable skills (code) for code bands; lessons elsewhere | E − C at B1–B3 | Executable reuse beats text for near transfer | | F + applicability gate | C or D plus gate | F − D on foils and on negative-transfer rate | The gate prevents misapplication without killing TG | | G Full system | All plus offline induction and ledger calibration | G − best single component; also G over a second run of the benchmark | Machinery compounds; calibration improves with use | Pre-register three hypotheses: (H1) TG of A in the cross-session channel ≈ 0; (H2) the gradient's AUC rises monotonically from B to D to G; (H3) F reduces foil errors by more than it reduces TG on non-foils. If H2 fails at B4–B5, the reviewer's verdict stands: Piper's far transfer, where it exists, comes from the base model. ## 15. Open questions, recommended papers, bibliography ### 15.1 Open research questions - Does structure-keyed retrieval produce cross-domain noticing in LLM agents, or does the LLM's own relevance judgement remain the bottleneck even when the right lesson is retrieved? - What signature vocabulary is expressive enough to match across software, operations and science, but small enough to stay consistent? Is it learnable from the ledger? - How should credit be assigned when several lessons were applied and the task succeeded? - Does the transfer horizon d₀ move outward with use (a true learning-to-transfer effect), or is it fixed by the base model? - How much far-band transfer survives a base-model upgrade? Lessons written for one model may be redundant or harmful for a stronger one. - Can the oracle gap be closed by better induction prompts alone, or does it require more source episodes per lesson? ### 15.2 Recommended papers, ranked by relevance to Piper | Rank | Paper | Why it matters | | --- | --- | --- | | 1 | Wang, Mao, Fried & Neubig 2024, "Agent Workflow Memory", arXiv:2409.07429 | Full causal chain with a three-band distance split | | 2 | Majumder et al. 2023, "CLIN", arXiv:2310.10134 | Causal-form lessons; explicit generalization protocol | | 3 | Zhao et al. 2024, "ExpeL", arXiv:2308.10144 (AAAI) | Insight extraction by contrast; cross-dataset transfer test | | 4 | Wang et al. 2023, "Voyager", arXiv:2305.16291 | Reference skill library with ablations | | 5 | Gentner, Rattermann & Forbus 1993, *Cognitive Psychology* 25:524–575 | Surface drives retrieval, structure drives mapping | | 6 | Forbus, Gentner & Law 1995, "MAC/FAC", *Cognitive Science* 19:141–205 | Template for two-stage transfer search | | 7 | Gick & Holyoak 1980; 1983, *Cognitive Psychology* 12:306–355; 15:1–38 | Retrieval failure; schema induction from two analogues | | 8 | Gentner, Loewenstein & Thompson 2003, *J. Educ. Psych.* 95:393–408 | Comparison-based abstraction improves transfer | | 9 | Taylor & Stone 2009, *JMLR* 10:1633–1685 | Transfer metrics: jumpstart, ratio, time-to-threshold | | 10 | Lopez-Paz & Ranzato 2017, "GEM", arXiv:1706.08840 | FWT / BWT matrix metrics | | 11 | Levy, Bogin & Berant 2023, arXiv:2212.06800 (ACL) | Diverse beats nearest retrieval for compositional targets | | 12 | Yasunaga et al. 2024, "LLMs as Analogical Reasoners", arXiv:2310.01714 | Base model holds latent analogues it won't use unprompted | | 13 | Valmeekam et al. 2023, "PlanBench", arXiv:2206.10498 | Obfuscation template (Mystery Blocksworld) | | 14 | Wu et al. 2024, "Reasoning or Reciting?", arXiv:2307.02477 | Counterfactual-task methodology | | 15 | Lewis & Mitchell 2024, arXiv:2402.08955 | Analogy fragility under counterfactual surfaces | | 16 | Webb, Holyoak & Lu 2023, *Nature Human Behaviour*, arXiv:2212.09196 | The opposing, positive analogy result | | 17 | Barnett & Ceci 2002, *Psych. Bulletin* 128:612–637 | Multi-dimensional far-transfer taxonomy for bands | | 18 | Wang, Dai, Póczos & Carbonell 2019, arXiv:1811.09751 (CVPR) | Formal negative transfer; gating | | 19 | Shi et al. 2023, arXiv:2302.00093 | Irrelevant context harms LLM reasoning | | 20 | Sutton, Precup & Singh 1999, *Artificial Intelligence* 112:181–211 | Options: preconditions as initiation sets | | 21 | Ahn et al. 2022, "SayCan", arXiv:2204.01691 | Two-factor (usefulness × feasibility) skill selection | | 22 | Wang, Neubig & Fried 2024, "TroVE", arXiv:2401.12869 | Verify-and-trim library growth | | 23 | Stengel-Eskin et al. 2024, "ReGAL", arXiv:2401.16467 | Refactoring into generalizable abstractions | | 24 | Ellis et al. 2021, "DreamCoder", arXiv:2006.08381 | Wake–sleep library learning | | 25 | Grand et al. 2024, "LILO", arXiv:2310.19791 | Documented abstractions improve LLM use | | 26 | Sarch et al. 2024, "ICAL", arXiv:2406.14596 | Abstraction from noisy demonstrations | | 27 | Sumers et al. 2024, "CoALA", arXiv:2309.02427 | Memory-type vocabulary for agents | | 28 | Chollet 2019, "On the Measure of Intelligence", arXiv:1911.01547 | Generalization difficulty as the scoring frame | | 29 | Huang et al. 2024, arXiv:2310.01798 (ICLR) | Unverified self-lessons are unreliable | | 30 | Zamir et al. 2018, "Taskonomy", arXiv:1804.08328 | Measured transfer map as a target artifact | | 31 | Vu et al. 2020, arXiv:2005.00770 (EMNLP); Pruksachatkun et al. 2020, arXiv:2005.00628 (ACL) | Paired A→B transfer, including negative, in LMs | | 32 | Bransford & Schwartz 1999, *Rev. Res. Educ.* 24:61–100 | Transfer as faster future learning | ### 15.3 Additional works cited - Aamodt & Plaza 1994, "Case-Based Reasoning: Foundational Issues…", *AI Communications* 7(1):39–59. - Achille et al. 2019, "Task2Vec", arXiv:1902.03545. - Andreas, Klein & Levine 2017, "Modular Multitask RL with Policy Sketches", arXiv:1611.01796. - Andreas et al. 2016, "Neural Module Networks", arXiv:1511.02799. - Bengio et al. 2009, "Curriculum Learning", ICML. - Bengio, Courville & Vincent 2013, "Representation Learning: A Review", arXiv:1206.5538. - Cassano et al. 2023, "MultiPL-E", arXiv:2208.08227. - Chaudhry et al. 2018, "Riemannian Walk", arXiv:1801.10112; 2019, "On Tiny Episodic Memories in Continual Learning", arXiv:1902.10486. - Chi, Feltovich & Glaser 1981, *Cognitive Science* 5:121–152. - Cobbe et al. 2020, "Procgen", arXiv:1912.01588. - Deng et al. 2023, "Mind2Web", arXiv:2306.06070. - Detterman 1993, in Detterman & Sternberg (eds.), *Transfer on Trial*. - Díaz-Rodríguez et al. 2018, arXiv:1810.13166. - Duan et al. 2016, "RL²", arXiv:1611.02779. - Dziri et al. 2023, "Faith and Fate", arXiv:2305.18654. - Falkenhainer, Forbus & Gentner 1989, "The Structure-Mapping Engine", *Artificial Intelligence* 41:1–63. - Finn, Abbeel & Levine 2017, "MAML", arXiv:1703.03400. - Gentner 1983, *Cognitive Science* 7:155–170. - Gu et al. 2024, "CRUXEval", arXiv:2401.03065. - Hernandez et al. 2021, "Scaling Laws for Transfer", arXiv:2102.01293. - Hill et al. 2019, arXiv:1902.00120 (ICLR). - Jain et al. 2024, "LiveCodeBench", arXiv:2403.07974. - Jimenez et al. 2024, "SWE-bench", arXiv:2310.06770. - Keysers et al. 2020, "CFQ", arXiv:1912.09713. - Khattab et al. 2023, "DSPy", arXiv:2310.03714. - Kim & Linzen 2020, "COGS", arXiv:2010.05465. - Kirk et al. 2023, *JAIR*, arXiv:2111.09794. - Kirkpatrick et al. 2017, "EWC", arXiv:1612.00796. - Kolodner 1993, *Case-Based Reasoning*, Morgan Kaufmann. - Kumaran, Hassabis & McClelland 2016, *Trends in Cognitive Sciences* 20(7). - Lake & Baroni 2018, "SCAN", arXiv:1711.00350. - Lauscher et al. 2020, arXiv:2005.00633. - Liu et al. 2022, arXiv:2101.06804. - Luchins 1942, *Psychological Monographs* 54(6). - McClelland, McNaughton & O'Reilly 1995, *Psychological Review* 102:419–457. - McCoy et al. 2023, "Embers of Autoregression", arXiv:2309.13638. - Mendez & Eaton 2021, arXiv:2007.07732. - Min et al. 2022, arXiv:2202.12837. - Mirzadeh et al. 2024, "GSM-Symbolic", arXiv:2410.05229. - Mitchell 2021, arXiv:2102.10717. - Moskvichev, Odouard & Mitchell 2023, "ConceptARC", arXiv:2305.07141. - Packer et al. 2023, "MemGPT", arXiv:2310.08560. - Pan & Yang 2010, "A Survey on Transfer Learning", *IEEE TKDE* 22(10):1345–1359. - Park et al. 2023, "Generative Agents", arXiv:2304.03442. - Perkins & Salomon 1989, *Educational Researcher* 18(1):16–25; 1992, "Transfer of Learning", *International Encyclopedia of Education*. - Pires, Schlinger & Garrette 2019, arXiv:1906.01502. - Rolnick et al. 2019, arXiv:1811.11682. - Rosenstein, Marx, Kaelbling & Dietterich 2005, NIPS Inductive Transfer Workshop. - Rubin, Herzig & Berant 2022, "EPR", arXiv:2112.08633. - Rusu et al. 2016, "Progressive Neural Networks", arXiv:1606.04671. - Sala & Gobet 2017, *Current Directions in Psych. Science* 26:515–520. - Shinn et al. 2023, "Reflexion", arXiv:2303.11366. - Shridhar et al. 2021, "ALFWorld", arXiv:2010.03768. - Singley & Anderson 1989, *The Transfer of Cognitive Skill*, Harvard University Press. - Thorndike & Woodworth 1901, *Psychological Review* 8:247–261. - Wang, J. X. et al. 2016, "Learning to Reinforcement Learn", arXiv:1611.05763. - Wang, J. X. et al. 2021, "Alchemy", arXiv:2102.02926. - Wang, R. et al. 2022, "ScienceWorld", arXiv:2203.07540. - Wang, Z. et al. 2023, "JARVIS-1", arXiv:2311.05997. - Wei et al. 2022, "FLAN", arXiv:2109.01652. - Wołczyk et al. 2021, "Continual World", arXiv:2105.10919. - Yosinski et al. 2014, arXiv:1411.1792. - Yu, T. et al. 2020, "Meta-World", arXiv:1910.10897; Yu, T. et al. 2020, "PCGrad", arXiv:2001.06782. - Yu, J. et al. 2024, "Thought Propagation", arXiv:2310.03965. - Zhang et al. 2022, "A Survey on Negative Transfer", arXiv:2009.00909. - Zheng et al. 2024, "Synapse", arXiv:2306.07863. - Zhou et al. 2023, "WebArena", arXiv:2307.13854. **UNVERIFIED items to check before relying on them:** StreamBench (Wu et al. 2024, arXiv:2406.08747) details; the 2025 "experience-following" agent-memory study (Xiong et al.); AWM per-split numbers; ExpeL HotpotQA→FEVER magnitude; whether current frontier reasoning models still show the counterfactual gaps cited.