Summary / 18. Generalization / Document
2026-10-12
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper · Cognition — Component 18: Generalization **Research, microbenchmark, and upgrade working paper · v1.0 · 2026-10-12** **Capability:** Applying learned knowledge to novel cases. **Status:** Proposed; no baseline score or intervention result yet. **Operating rule:** Baseline → one small change → sealed retest → keep/revert. Expand only after a credible positive signal. ## 1. Executive finding Generalization is not the existence of a component registry, a reuse search, or a lesson store. It is **observable improvement on a new problem that requires the same underlying principle but differs meaningfully from training examples**. Piper currently has infrastructure relevant to reuse but no direct instrument establishing this ability. This paper proposes a small paired experiment that reuses Component 17 (Learning) tasks and independently grades both near and far transfer. The governance observation is related but distinct: the September 30 review reported **1 strict outside verdict and 6 loose outside verdicts among 1,079 closed deliverables**. That measures a severe shortage of independent *verification*, not generalization itself. Do not report either metric as a substitute for the other. The review also found **zero capability-tool calls** and a stale component registry; both require an inexpensive inspection before adding new machinery. **First bet:** lessons are stored but not consulted when the superficial appearance of a problem changes. Test this hypothesis before building a generalization engine. ## 2. Source-of-truth inventory and cautions | Item | Snapshot / observation | What to verify now | |---|---|---| | `component-registry` v1.1.1 | Described as versioned catalogue, judged stale | Current version; sample entries checked against actual code | | `resume-system` v1.3.0 | Listed as relevant system | Whether it helps transfer or merely preserves work state | | `outside-verdict-weekly` v1.0.0 | Live counting mechanism | Denominator, time window, strict vs loose rules; author independence | | Capability lookup | Zero calls in reviewer window | Call logging works; eligible situations existed | | Closed deliverables | 1,079; strict 1, loose 6 | Reproduce counts with exact date range and definitions | These figures are historical reviewer observations, **not fresh measurements**. Never silently describe them as current. Do not alter production acceptance rules merely to improve the outside-verdict metric. ## 3. Operational definition and boundaries **Learning (Component 17):** does performance improve after feedback on variants of the taught task family? **Generalization (Component 18):** does that improvement survive changes in surface vocabulary, setting, tools, and representation while the causal/algorithmic principle remains the same? **Transfer (Component 19):** how well does the capability cross into substantially different domains or workflows? There is overlap, but Component 18 focuses on *novel exemplars of the same underlying rule*, with graduated distance. **Independent verification:** is correctness determined by tests, references, or judges outside the agent's editable scope? This is a prerequisite for credible scores, **not itself proof of generalization**. ### Generalization distances - **Near:** change names, constants, layout, or ordering; preserve algorithm and tools. - **Medium:** change task representation or tool interface; preserve governing principle. - **Far:** different domain and task framing; require recognizing and adapting the principle, not copying a recipe. - **Negative transfer:** superficially similar case where the learned principle **must not** be applied. A test suite that only varies filenames or numeric constants is insufficient. ## 4. Research basis (starting references) These are conceptual leads, not claims that any technique will work for Piper: 1. **Lake & Baroni (2018), “Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks.”** Systematic recombination is harder than surface interpolation. https://proceedings.mlr.press/v80/lake18a.html 2. **Chollet (2019), “On the Measure of Intelligence.”** Distinguish skill on known tasks from efficiency of adaptation to novelty. https://arxiv.org/abs/1911.01547 3. **Zhou et al. (2023), “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models.”** Decomposition may support compositional transfer but is an empirical question. https://arxiv.org/abs/2205.10625 4. **Finn, Abbeel & Levine (2017), “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks.”** A contrasting formal account of adaptation from small experience; not a directly drop-in method for Piper. https://arxiv.org/abs/1703.03400 5. **Jimenez et al. (2024), “SWE-bench: Can Language Models Resolve Real-World GitHub Issues?”** Hidden external tests exemplify independence, although repo-level coding performance is not by itself a generalization measure. https://arxiv.org/abs/2310.06770 ## 5. Microbenchmark v1 — six pairs, no infrastructure epic **Build budget:** 30–45 minutes if Component 17 harness already supports sealed variants; otherwise stop after a minimal manual pilot. **Run budget:** aim for 10–20 minutes and a small pre-declared API cap (suggested **US$3** for the pilot; operator can lower). **Changes permitted:** none during baseline. **Data protection:** held-out prompts, solutions, and grading remain outside Piper's writable scope. ### Six small families (one teaching example, one novel test, one matched control where practical) | ID | Teaching failure / rule | Sealed novel case | Negative-control trap | Diagnostic signal | |---|---|---|---|---| | G1 | Fix an off-by-one boundary in a loop | New language or unfamiliar iterator API, same boundary invariant | Similar-looking loop with deliberate inclusive end | Identifies invariant vs copies edit | | G2 | Reject a stale claim; perform fresh source read | Different ledger and supersession layout | Latest entry explicitly invalidated | Reads provenance vs trusts surface form | | G3 | Correct a race by making update atomic | Different storage primitive/workflow | Apparent race actually caused by retry duplication | Causal understanding vs indiscriminate locking | | G4 | Avoid a bad evaluator that cannot fail | Unfamiliar acceptance contract with vacuous assertions | Legitimately always-true invariant with independent proof | Detects tautology without rejecting everything | | G5 | Choose reuse over rewrite when fit is high | New API/component naming, same useful existing capability | Similar name but wrong semantics | Semantic selection vs keyword mimicry | | G6 | Calibrate using prior outcome base rates | New work category and changed outcome prevalence | Deliberate distribution shift | Adjusts reasoning vs ports old probability blindly | **Important:** A single six-family run is a *screen*, not a definitive estimate of generalization. Report raw item outcomes, not a polished precision score. Reuse the Component 17 learning/control setup to prevent attributing natural improvement or prompt effects to learning. ### Minimal experiment, fixed in advance 1. Seal six novel cases and six control cases. Reviewer creates answer keys and test assets that Piper cannot inspect or edit. 2. Randomize case order and which task families receive feedback; use matched difficulty where possible. 3. Record baseline outcomes **before teaching**, including whether the relevant capability lookup or memory retrieval actually fired. 4. Provide **one** concise correction or learning event for taught families only. No code edits yet. 5. Start a fresh probe context and test sealed novel variants. Repeat untreated control variants under the same compute limit. 6. Optionally introduce a minimal targeted retrieval cue for one affected family; retest against *new* sealed variants, never the same test prompt alone. 7. Analyze using absolute raw changes and confidence/uncertainty, including missed opportunities to abstain. Stop if results are flat, noisy, or harmful. ### Primary and secondary scoring Let `T_before`, `T_after` be success fractions for taught families, and `C_before`, `C_after` for matched untreated controls. **Difference in differences:** `learning_adjusted_gain = (T_after - T_before) - (C_after - C_before)` For *novel* cases, report separately by near/medium/far distance. With six items, **do not claim statistical certainty** from a one-item swing. **Generalization retention:** `novel_gain / taught_gain` is only descriptive when `taught_gain` is clearly positive; **undefined** when taught gain is zero or negative and unstable when near zero. Never turn an undefined value into 0 or 100%. Also log: - **False transfer rate:** applying the taught rule in an explicit counterexample. - **Independent hidden-test pass rate:** grader outside agent write set. - **Cost per successful new case:** tokens, wall time, tool calls; compare against no-lesson baseline. - **Reuse/retrieval invocation:** eligible opportunities, actual use, and whether use improved outcome. - **Verifier coverage:** strict externally graded completions / eligible completions, with explicit denominator. Avoid merging all metrics into one score before the behavior is understood. Follow the shared 38-suite scoring contract for published score normalization. ## 6. Trip-the-system exercises — provoke observable failure, cheaply Use 2–3 probes first. Do not launch all of these as a large batch. **T1. Rename everything.** Teach an implementation bug fix, then reframe its recurrence with disjoint identifiers and altered filesystem layout. *Trip:* keyword-based reuse fails. *Trace:* tool choices, retrieved evidence, edits, test result. **T2. Similar name, wrong behavior.** Present a component whose name appears to match the requested operation but violates one invariant. *Trip:* overgeneralization. *Trace:* whether Piper reads code and validates semantics. **T3. Correct answer, wrong representation.** Teach a table-based constraint, test it as JSON events. *Trip:* surface dependence. *Trace:* extracted constraints, not chain-of-thought capture. **T4. Lesson must be ignored.** Reuse a previous postmortem's vocabulary in an incident with a different root cause. *Trip:* negative transfer. *Trace:* competing hypotheses and discriminating tests. **T5. Retrieval boundary.** A lesson is remembered in a summary, but authoritative source state has changed. *Trip:* stale rule transfer. *Trace:* as-of reads, source timestamps and decisions. **T6. Tool not called.** Give a problem with clear reusable local capability. *Trip:* capability lookup exists but is not used. *Trace:* eligible opportunity, call attempts, selection and outcome. **T7. Domain swap.** Teach an idempotency property in webhook ingestion; test it in scheduled job retry handling. *Trip:* deeper principle transfer. *Trace:* identified invariant, implementation and tests. **T8. Friendly evaluator.** Piper may write a persuasive explanation that its change works; independent harness supplies adversarial edge cases. *Trip:* self-certification. *Trace:* own verdict vs outside verdict mismatch. **T9. Held-out reversal.** Teach “prefer new data” then include a case where newest data is retracted and older verified data is authoritative. *Trip:* learned slogan in place of source reasoning. **T10. Persistence reset.** Teach a verified lesson, reset active session, present distant variant. *Trip:* context-only adaptation masquerading as durable generalization. **Safety:** all code tasks run in sandboxes or disposable repos; no destructive production operations, external messages, or credential exposure. ## 7. Failure taxonomy Assign **one primary** cause and any secondary tags using evidence from logs: | Code | Mechanism | Observable symptom | |---|---|---| | F01 | Lesson not encoded | No retrievable lesson or invariant | | F02 | Retrieval not triggered | Relevant prior evidence exists but no lookup | | F03 | Retrieval wrong | Similar but inapplicable lesson selected | | F04 | Surface matching | Only renamed variables/structure defeat application | | F05 | Missing causal abstraction | Rule restated without conditions of applicability | | F06 | Negative transfer | Rule applied to a counterexample | | F07 | Stale provenance | Superseded lesson treated as live | | F08 | Execution failure | Correct principle identified, implementation incorrect | | F09 | Verification failure | Agent declares success without outside proof | | F10 | Evaluation leakage | Test or answer key available during preparation | | F11 | Compute confound | Test arm received extra model calls or context | | F12 | Registry drift | Capability map misleading or outdated | This classification should drive the **next smallest experiment**, not trigger twelve simultaneous fixes. ## 8. Targeted interventions, prioritized by cheapness ### I0 — Instrument rather than intervene **Hypothesis:** observed generalization failures are mostly failures to consult existing tools or lessons. **Change:** none. Add lightweight `eligible_lookup`, `lookup_fired`, `retrieval_id`, `outcome` tags. **Test:** two near + two medium variants. **Stop:** if tracing cannot establish eligibility or outcomes, fix observability first. ### I1 — One-line applicability cue **Hypothesis:** lessons are stored as anecdotes, not transferable conditions. **Change:** in a copy of the lesson record, add `invariant`, `applies_when`, `does_not_apply_when`, `verification`. No new retrieval system. **Test:** two fresh analogues plus one negative-transfer trap. **Keep:** more independently correct novel outcomes, no rise in false transfer, comparable cost. **Revert:** if merely more confident or merely more verbose. ### I2 — Capability lookup at one meaningful junction **Hypothesis:** a targeted lookup before writing code improves reuse and decreases waste. **Change:** one call at a guarded, well-defined “before new implementation” decision point; require semantic inspection of returned candidate. **Test:** three fresh tasks, one intentionally misleading registry entry. **Keep:** at least one clear extra correct outcome or reduced cost with non-inferior correctness; no false reuse. ### I3 — Registry freshness spot-check **Hypothesis:** stale registry entries systematically misroute transfer. **Change:** verify only the handful of entries needed for experiment, then repair the specific stale record. **Test:** previously failed lookup plus untouched held-out case. **Do not** build a full registry rewrite without a demonstrated causal role. ### I4 — Outside-verdict path on one artifact class **Hypothesis:** self-certified completion masks generalization errors. **Change:** for one disposable code-task class, attach immutable hidden tests with an independent writer. **Test:** score both “Piper says done” and external pass. **Keep:** coverage and detection improve without meaningfully increasing false failures or cost. This upgrades evaluation credibility, **not necessarily generalization**. ## 9. Micro-loop and escalation rules | Phase | Time target | Exit condition | |---|---:|---| | Inventory & confirm current state | 10–15 min | Count and invocation definitions reproducible | | Pilot with 2–3 cases | 10–15 min | At least one informative failure or stable pass | | Baseline up to six cases | 15–25 min | Paired outcomes captured with independent keys | | One intervention | ≤30–45 min | Smallest modification with a rollback path | | Retest 3–6 new cases | 10–20 min | Positive signal without negative transfer | | Expand | Approval after evidence | Larger sealed sample justified by observed effect | **Strict stop rules:** halt if validation assets leak, judge independence fails, baseline is at ceiling, test definitions shift between arms, tool costs exceed cap, or agent write scope becomes ambiguous. If 6/6 baseline passes, increase difficulty through representation and negative transfer **before** adding more questions. If an intervention produces only 1 extra pass, mark **promising but uncertain**, then repeat with a few new cases rather than declaring victory. **Keep/revert:** require verified improvement or a narrower efficiency win, no material regression on traps, and bounded expense. Record bad and null results, including why the test may lack power. Reserve extensive multi-arm studies for an intervention that survives at least two independent micro-retests. ## 10. Data contract for Claude Code Append event-level records; do not rely on free-text claims of success. Suggested JSONL fields: ```json { "suite": "cognition-18-generalization-v1", "trial_id": "G3-medium-20261012-01", "family_id": "G3", "arm": "taught", "phase": "sealed_retest", "novelty_distance": "medium", "case_hash": "sha256:...", "lesson_id": "lesson-...", "lesson_visible": true, "capability_lookup_eligible": true, "capability_lookup_fired": false, "retrieved_source_ids": [], "verdict_origin": "independent_hidden_test", "verdict_author_outside_write_set": true, "passed": false, "false_transfer": false, "tokens": 0, "elapsed_seconds": 0, "cost_usd": 0.0, "primary_failure": "F02" } ``` **Validation:** unique case IDs, fixed answer-key hashes, time-stamped lesson availability, frozen random assignment, and explicit grader provenance. Separate `strict` verdicts (outside writable scope, deterministically graded) from `loose` (outside narrative feedback without authoritative tests). A model's claim that a task passed is **never** its independent verdict. ## 11. Work orders (one at a time) ### Work order A — Read-only audit Inspect `component-registry`, `resume-system`, `outside-verdict-weekly`, capability lookup registration, and their event streams. Verify the reviewer snapshots with date-ranged queries. **Deliver:** one-page audit with evidence paths and any contradictions. **No modifications.** Timebox 15 minutes. ### Work order B — Minimal sealed probe Reuse Component 17 evaluation structure; implement **two** generalization pairs and **one** negative-transfer case. Grader and tests outside Piper's write scope. Record before/after and teaching/control conditions. **Acceptance:** harness demonstrably fails a planted wrong answer and logs grader provenance. **No production deployment.** Timebox 25 minutes. ### Work order C — Expand only until a failure is exposed Add at most three cases using different representations. Aim for a non-ceiling baseline with interpretable failure traces, not a fixed large N. **Acceptance:** each case maps to an invariant and a known counterexample. Timebox 20 minutes. ### Work order D — Test one smallest fix Choose **only one** of I1–I4 based on observed failures. State predicted effect before change. Save rollback marker. Retest on fresh hidden cases, matched compute. **Acceptance:** independent results, costs, and regressions reported. Timebox 45 minutes. ### Work order E — Decide, document, stop Log hypothesis, predeclared expectation, baseline, trial outcomes, side effects, costs and keep/revert. A no-signal experiment is a valid ending. Propose bigger work **only** after a repeat positive signal on additional sealed variants. ## 12. Reporting template ```text Component 18 · Generalization · suite v1 Snapshot date / software versions: What changed (if anything): Hypothesis / predeclared prediction: Case count and held-out method: Taught baseline → after: __ / __ Untaught control baseline → after: __ / __ Novel near / medium / far outcomes: Negative-transfer traps: __ / __ Strict independent checks: __ / __ (eligibility denominator: __) Capability lookup: __ calls / __ eligible opportunities Time, tokens, cost: Main observed failure mode: Limits (small N, ceiling, possible leakage): Decision: KEEP / REVERT / INCONCLUSIVE Next smallest discriminating test: ``` ## 13. Interpretation and limits - The **1/1,079 strict-verdict** observation should motivate an independent grading path, but it is not a measured 0.09% generalization score. - An improved hidden-test pass rate might reflect better coding, tool selection, or greater compute. Compare equal-budget controls to isolate transfer. - A high novel/in-family ratio may be misleading if both scores are low, and the ratio is unstable when taught improvement is tiny. - Not all lessons should generalize. **Appropriate non-application** on counterexamples must earn credit. - Never claim that Piper has learned a transferable skill from one successful analogical case. This initial bench is meant to decide whether a larger test is worth paying for. ## 14. Bottom line **Smallest useful next step:** independently grade two renamed or reframed variants of an already taught lesson, plus one deceptively similar case where that lesson should *not* apply. Log whether Piper even invoked her capability/lesson machinery. Fix only the first demonstrated bottleneck. Retest on new variants. If there is no measurable signal, stop. --- *Working document for Piper / Claude Code. The September 30, 2026 reviewer observations and October 11, 2026 page review date are provided by the operator and should be treated as historical snapshots, not live-verified runtime facts.*