Summary / 11. Abstraction / Document
2026-10-10
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper · Cognition · Component 11 — Abstraction ## Research, Benchmark, Upgrade & Experimental Protocol **Working paper:** v1.0 **Date:** 2026-10-10 **Status:** Proposed; no experiments reported as completed **Primary objective:** Convert recurring experience into general, falsifiable, reusable mechanisms and demonstrate transfer to unseen work. > **Research question.** Can Piper infer a principle from several specific incidents, implement a reliable check or reusable skill expressing that principle, and later apply it correctly to unfamiliar tasks without being told which lesson to use? --- ## 1. Starting position and evidence limitations Piper describes an abstraction ladder: **incident → lesson → check → skill**. The known components are `work-ledger v1.3.42` (lesson registry), `skills-distill v1.0`, `contemplative-cycle v1.2.0`, and `mutate v1.0.1`. The September 30 independent review reported that only about 20% of lessons had supporting enforcement, no skill table existed, and `contemplative-cycle` had been retired on September 26. Eight lessons (21, 29, 47, 56, 65, 105, 263, and 295) had been closed against a real control. Source counts in the provided snapshot conflict by time (339 and 349 totals) and by category; do **not** reconstruct a current count by arithmetic. Query the live registry and its historical timestamps before using any figure as a baseline. The observations above are user-supplied historical system-review findings, **not independently verified live facts**. A fresh read of the running modules, registry schema, dispatch logs, and enforcement events is required. In particular, never infer that `skills-distill` actually distills skills merely from its name or existence. ### Audit first 1. Record repository revision, model, prompt revision, dependencies, lesson registry row count and state counts, and module availability. Resolve count discrepancies with an explicit `as_of` timestamp. 2. Inspect 30 randomly selected lessons stratified by `pending`, `checked`, `uncheckable`, and other states: source incident, rule text, applicability bounds, check identifier, latest execution and outcome, and subsequent reuse. 3. Enumerate every executable check and skill artifact; verify they are reachable by the running agent, not merely saved on disk. 4. Trace lesson-to-check and check-to-skill edges. Mark retired processes as retired; find actual replacement owners rather than reviving retired code by assumption. 5. Freeze audit outputs before attempting upgrades. ## 2. What abstraction is—and is not Abstraction extracts a **shared, causally relevant invariant** from observations that differ on irrelevant details, with a usable description of scope and exceptions. The output can be a predicate, algorithm, operational constraint, executable test, reusable procedure, schema, or compact concept. It must simplify future problem-solving without hiding meaningful differences. The unit of learning is **not** a sentence such as “test validators carefully.” A stronger abstraction specifies *what must be true*, *when it must be true*, *how to test it*, and *where it must not be enforced*. Example: four reports show that one part of a system says a scheduled job succeeded while the job's own output is missing or stale. A weak lesson is “check jobs.” A candidate abstraction is: **a success claim derived from secondary status is invalid until the authoritative job record passes freshness and identity checks**. A possible check compares the claimed job ID, latest run timestamp, completion status, output marker, and source authority at decision time. A negative case might be a historical report explicitly labeled as historical: it should not be rejected merely for being old. ### Distinguish nearby components - **#4 Recall:** locate specific prior incidents; abstraction identifies what those incidents have in common. - **#6 Logic:** assess whether a stated rule follows or whether a predicate is satisfiable; abstraction proposes the rule to evaluate. - **#9 Causal reasoning:** distinguish causal mechanisms from coincident features; abstraction must not conflate correlation with invariant structure. - **#10 Analogy:** map an existing pattern to another domain; abstraction creates or refines the transferable pattern. - **#12 Decomposition:** split a task into parts; abstraction unifies examples into reusable structure. - **#19 Transfer:** apply acquired competence to new domains; #11 produces and packages the competence being transferred. - **#24 Error detection:** detects violations; #11 creates new general checks from recurring violations. - **#23 Uncertainty:** represent doubt about a rule's validity or scope rather than treating an attractive generalization as established. Score #11's contribution separately from these enabling systems. ## 3. Research foundations and what to borrow ### 3.1 DreamCoder — discovering reusable conceptual primitives Ellis and colleagues' DreamCoder framework learns libraries of abstractions while solving families of programming tasks. The important insight for Piper is not a particular neural architecture; it is the **feedback loop** between successfully solving instances, discovering reusable structure in solutions, and applying the learned structure to solve additional tasks. A candidate abstraction should earn its place by reducing description or work *and* preserving generalization. [R1, R2] **Engineering interpretation:** cluster lessons by causal signature and successful remediation, propose a compact shared rule, and validate it on held-out incident families. Do not undertake full DreamCoder reimplementation. ### 3.2 Voyager — executable, composable skill libraries Voyager stores verified executable skills and retrieves them to solve later Minecraft tasks. For Piper, the corresponding artifact is a versioned checked procedure with a trigger, preconditions, implementation, tests, and observable effects. A code file alone is not a skill; neither is a lesson paragraph. [R3] **Engineering interpretation:** a real skill registry with execution logs, provenance, applicability conditions, compositional dependencies, and retirement capability. ### 3.3 Reflexion — lessons from feedback, but not sufficient by themselves Reflexion demonstrates agents using natural-language self-feedback retained as episodic memory to improve later attempts. That supports Piper's incident-to-lesson stage, but it does not establish that prose reflections become reliable executable controls or portable procedures. [R4] **Engineering interpretation:** use reflections as hypotheses, never as proof of improvement. ### 3.4 Minimum description length — reward explanatory economy carefully Minimum description length (MDL) favors compact explanatory models while penalizing their complexity. An abstraction that merely memorizes every exception is not an effective abstraction. Yet a tiny, overbroad rule can be worse because it falsely fires on unrelated cases. [R5] **Engineering interpretation:** rank candidates by held-out validity first, then compressibility, not by brevity alone. Consider `description_length(rule) + description_length(exceptions) + description_length(unexplained_cases)`. This is an engineering proxy for MDL, not a claim of a properly calibrated probabilistic code length. ### 3.5 Metamorphic testing — assert relations that must remain invariant Many real incidents lack a simple expected-output oracle. Metamorphic testing evaluates relations between related inputs and outputs. This is well suited to rules such as **renaming a tenant must not change authorization**, **reordering independent events must not change the result**, or **a cached summary must not override a newer authoritative state**. [R6] **Engineering interpretation:** every candidate abstraction should propose at least one pairwise transformation under which the rule is expected to hold, and one boundary transformation under which it should not. ### 3.6 CodeARC and SkillGenBench — separation of induction from skill generation CodeARC frames inductive synthesis as deriving programs from examples, testing on unfamiliar inputs, and improving with feedback. SkillGenBench separately measures whether an agent can create executable, reusable skills from source material under standardized test harnesses. [R7, R8] **Engineering interpretation:** run two experiments: **rule induction from incidents** and **skill generation from correctly inferred rules**. Otherwise failure at either step is hidden in an overall result. ### Research synthesis The proposed architecture is an engineering hypothesis informed by these studies, **not** a result proven specifically for Piper. Its critical feature is an external, frozen evaluator that Piper cannot modify during a scored run. ## 4. Proposed abstraction lifecycle ``` incident with evidence and outcome → normalize observable causal signature → retrieve potentially related incidents → cluster by invariant rather than cosmetic similarity → propose 2–3 competing hypotheses → specify applicability, exceptions and counterexamples → generate check / candidate skill → evaluate on hidden positive, negative and boundary cases → stage artifact in shadow mode → observe real triggering and downstream effects → promote, revise, quarantine or retire ``` **Mandatory outputs of the hypothesis stage:** - `rule`: precise condition or transformation and corresponding consequence. - `supporting_incident_ids`: real cases rather than invented justifications. - `counterexamples`: known disconfirming examples or explicitly “not yet observed.” - `scope`: domains, entities, time horizons and permission boundaries where applicable. - `mechanism`: why the rule should work, distinguished from empirical association. - `falsification_plan`: an observation that would count against it. - `check`: deterministic executable validator when the domain permits; otherwise a reproducible verification procedure with externally scored evidence. - `skill`: optional reusable action procedure, only after independent verification. A candidate can legitimately stop as **uncheckable under current instrumentation**. Do not force all insights into code or penalize correct abstention as harshly as confident false enforcement. ## 5. Benchmark v1: ABST-11 ### 5.1 Purpose and threat model ABST-11 asks whether Piper discovers rules that predict and constrain unfamiliar cases. Threats include memorizing visible cases, selecting superficial similarities, overgeneralizing to look-alike negatives, writing a check that always passes or always fails, calling an existing check newly written, generating a skill that is never invoked, and editing the judge during evaluation. ### 5.2 Dataset and split Build **12 independently authored incident families**, with 4–5 demonstrations each (target 54 visible incidents), plus **at least 20 sealed evaluations per family** (target 240): 8 true violations, 8 valid nonviolations, and 4 borderline/exception cases. Balance true labels overall. If a family has no genuinely meaningful borderline cases, substitute explicit near misses rather than fabricated ambiguity. For every family, prepare: 1. Four or five visible incidents sharing an underlying rule, with superficial distractors varied. 2. Held-out positive cases exhibiting the invariant failure. 3. Held-out negatives that resemble a positive lexically but lack the causal prerequisite. 4. Boundary cases requiring explicit scope judgment. 5. At least one adversarial near miss engineered by a separate evaluator. 6. A verified reference oracle and evidence trail; reviewers should resolve genuinely ambiguous cases before sealing. Split by **generative template**, not just random seed: distinct names, environments, repos, message formats, and surface order are insufficient if the underlying template remains identical. At least 25% of sealed cases should involve unseen environmental structure. Keep final holdout authorship separate from the component developers. ### 5.3 Families to implement | ID | Shared hidden mechanism | Critical near-negative | |---|---|---| | A | Secondary status cannot outrank newer authoritative evidence | Properly labeled historical statement | | B | A check with no reachable failure path gives false assurance | Valid always-pass invariant explicitly constrained to tautology | | C | Retried non-idempotent operations create duplicated effects | Duplicate request with a valid idempotency key and deduplicated result | | D | Cached values need provenance and invalidation after writes | Independent cache entry unaffected by the write | | E | An apparent success metric is inflated by selective sampling | Representative sampling with changed base rate disclosed | | F | Recovery routines need an ownership/epoch guard | Legitimate successor acting with new epoch | | G | No-op feedback loops can self-reinforce without external outcomes | Legitimate simulation explicitly scored as simulation | | H | Validator and generator sharing assumptions can miss errors | Independently derived reference checker agrees | | I | Multiple apparently independent signals share an upstream dependency | Two verified independent underlying sources | | J | Event ordering changes the meaning of state transitions | Explicitly commutative updates under tested preconditions | | K | A general-purpose check masks a domain-specific exception | Exception documented and mechanically verifiable | | L | Skill reuse can be unsafe when environment preconditions differ | Same skill in new environment with verified compatible preconditions | Do **not** give Piper this table or any reference rule in advance on a scored run; it is for the benchmark author. Use a sealed benchmark specification distinct from the operational paper available to Piper, and exclude these example families from final scoring if Piper has read this paper. Public examples are training/development material only. ### 5.4 Required scored response for each family 1. The inferred rule in structured text. 2. Why it covers each demonstration; mark counterevidence. 3. At least two rival hypotheses, and what evidence distinguishes them. 4. A minimal, inspectable validator or executable verification process. 5. Declared applicability conditions and exceptions. 6. A candidate reusable skill only when justified. 7. An explicit uncertainty statement for unresolved boundaries. No explanatory prose can substitute for passing the hidden tests. ### 5.5 Execution sequence **Round 0: Frozen baseline.** Collect unmodified Piper answers at fixed temperature/provider settings, without giving examples of the rules or diagnostics. Record time, tokens, dollar cost, calls, tool invocations and all output artifacts. **Round 1: Induction.** Give 4–5 demonstrations per family. Freeze the produced rule and executable check before hidden evaluation. No feedback about hidden cases. **Round 2: Sealed classification.** Evaluate on 20+ fresh examples using frozen checks. Derive confusion matrices (TP, FP, TN, FN), sensitivity, specificity, precision, balanced accuracy and coverage; track exceptions and abstentions separately. **Round 3: Metamorphic checks.** Transform cases in ways that should preserve the label, then introduce one principled change that should flip it. Score both preservation and controlled reversal. Reject rules that depend on incident IDs or accidental wording. **Round 4: Skill artifact.** For valid rule families, require an invocable, tested skill artifact with an exact interface and preconditions. Independently verify it works, and measure whether execution changes a downstream outcome. **Round 5: Delayed reuse.** After unrelated tasks and at least one context turnover, give new tasks without naming the family or skill. Log whether Piper autonomously retrieves, invokes, or reconstructs the learned abstraction. Repeat after a longer delay on a subset. **Round 6: Live shadow mode.** Deploy candidate detectors in logging-only mode. Evaluate operator-independent firing evidence, false positives, work saved, safety and overhead; promote only after all release gates pass. ### 5.6 Scoring rubric: 100 points | Dimension | Points | Independent evidence | |---|---:|---| | Hidden-case generalization | 30 | Balanced accuracy, fixed holdout and adversarial splits | | Executable enforcement | 20 | Check runs, is nontrivial, matches reference labels; mutation tests | | Scope and exception handling | 15 | Boundary set and calibrated abstentions | | Metamorphic invariance | 10 | Property-preserving and label-flipping transformations | | Actual reusable skill | 10 | Invocable artifact with verified tests and recorded version | | Unprompted delayed transfer | 10 | Correct future invocation without naming lesson | | Auditability and economy | 5 | Provenance, cost, inspectability and test stability | | **Total** | **100** | | For classification, primary `BA = (TPR + TNR)/2`; empty prediction, refusal, execution error, or timeout is a miss for primary BA, while an additional **selective BA at answered coverage** allows examination of justified abstention. A checker that always passes or always fails scores ~50% BA on balanced labels and **zero enforcement credit** unless its test contract legitimately defines such a function. Incorrectly treating exceptions as universal violations incurs explicit boundary errors. **Proposed gates (to validate empirically):** baseline is whatever the frozen system scores. Target >=80 overall, >=0.85 hidden BA, >=0.80 adversarial BA, >=0.80 metamorphic accuracy, and >=70% eligible autonomous reuse, with zero critical regressions. These are provisional engineering targets, not published standards. Report family-level outcomes and bootstrap 95% confidence intervals clustered by family; never conceal wide uncertainty behind a single number. ### 5.7 Check-quality mutation battery Every produced check faces mutations including: wrong source authority, reversed comparison, stale timestamp, changed entity identity, omitted branch, never-executed assertion, exception suppression, and swapped expected/actual. Require at least one implanted relevant mutation to be detected; report **mutation kill rate** across applicable mutations. Independently reject no-op, constant true, constant false, and unreachable checks. Do not reward checks solely for line coverage or syntactic validity. ## 6. Deliberately trip-the-system: real work that produces logs The operator's existing technique for Piper's cognition project is to assign substantial realistic work that strongly exercises the target subsystem, including failure paths. For #11, a one-shot puzzle is insufficient: the task must create **several related experiences**, induce a proposed generalization, and force its use in an unseen context. These exercises are **development probes**, not sealed scored items. ### Probe 1 — Eight lessons, one broken checking pattern Take eight historically documented lessons about “tests that cannot fail” (the eight lesson IDs listed above). Ask Piper to compare source incidents and code changes, distinguish what was actually fixed from what was merely marked checked, infer one or more general anti-vacuity invariants, and construct a harness that deliberately mutates a passing test to see whether it fails. Add two deceptively similar but legitimate intentionally permissive tests. The task fails if she rewrites only lesson descriptions, invents a shared cause, or her validator cannot detect real mutations. **Log:** retrieved lesson IDs, evidence reads, proposed clusters, alternative hypotheses, synthesized checks, mutation kills, mistaken rejects, final promotion decision. ### Probe 2 — Four stale-state incidents across unrelated subsystems Present archived cases from news-feed status, cost accounting, driver responses and software deployment. Mix causes so not all stale-looking cases share the same mechanism. Require Piper to infer the narrowest correct rules, create provenance/freshness validators, and run against a newly generated incident from a fifth subsystem. A deliberate decoy should have a stale *summary* but a correct *decision* based on direct primary evidence. **Failure to surface:** broad “always re-read everything” rule, checking wrong authority, confounding staleness with wrongness, and over-triggering expensive reads. ### Probe 3 — Repeated successful repair into a skill Create three isolated mini-repos with a shared failure mode and variant file layouts. Let Piper repair each, inspect outcomes, propose the smallest common repair procedure, and package a parameterized skill. Then give a fourth mini-repo with the same cause but different language/layout, and a fifth with similar logs but different cause. The correct skill must fix the fourth, reject the fifth, and leave a trace of applicability checks. **Failure to surface:** copy/paste scripts with embedded filenames, unsupported assumptions, fake 'skill' artifact never called, regressions or excessive blast radius. ### Probe 4 — Convergent symptoms, divergent causes Supply five deployment failures that all say “timeout”; two are clock skew, two are resource starvation, one is malformed configuration. Ask for no fewer than two causal abstractions or a clearly scoped conditional decision tree. Penalize a fabricated one-rule explanation even if rhetorically elegant. Then challenge with two hidden-style manual development cases. **Failure to surface:** superficial clustering, unjustified unification, refusal to split a cluster when evidence conflicts. ### Probe 5 — Delayed autonomous application Have Piper complete an operational task after several unrelated activities. Plant a new risk covered by an earlier verified abstraction, without mentioning the lesson. Observe whether she retrieves or invokes the corresponding check, whether it changes her action and whether it reduces error. This is the most important live-use probe; log the actual consequence, not just references to the rule. ### Probe 6 — Abstraction invalidation After a skill has been successful, alter a precondition in a sandbox so that the abstraction no longer holds. Provide credible external evidence of the change. Require Piper to recognize and scope the exception, suspend affected automation, update the rule provenance, and avoid applying the obsolete skill. Score latency to detection, prevented false action, and restoration after a new valid check. **Safety:** all probes that mutate code, data, jobs or policy must execute in reproducible sandboxes with limited permissions; deliberately induced faults must not affect live business systems, accounts or third parties. ## 7. Engineering upgrade proposals ### U0 — Instrumentation integrity (priority: critical) Build a read-only inventory reporting, for each lesson: source event, original text, active rule, present state, last evaluated time, check existence, check execution count, exception count, skill ID, reuse events and verified outcomes. Add checks for impossible state transitions (`checked` with missing evidence, `skill_active` with no invocation interface). Make every metric reproducible from event records. Investigate missing or retired components before building replacements. **Prediction:** reported “checked” totals will exceed the count of lessons with independently demonstrated operational enforcement. Treat as a hypothesis. ### U1 — Typed lesson and abstraction records Store incident observations apart from abstracted claims and validator artifacts. Suggested minimal tables or JSON records: ``` incidents(id, timestamp, source_uri, outcome, evidence_hash, environment) abstract_rules(id, version, claim, mechanism, scope, exceptions, state, evidence_set_hash, author, created_at, reviewed_at) rule_support(rule_id, incident_id, relationship) # supports/refutes/exception checks(id, rule_id, version, executable_ref, test_suite_hash, last_result) skills(id, rule_id, version, artifact_ref, preconditions, permissions, state) applications(id, rule_id, skill_id, context_hash, triggered_at, outcome, counterfactual_estimate, verified_by) ``` Keep schema append-only where practical, with explicit supersession and rollback events. Existing registry structures should be reused where sensible; **do not invent a second source of truth**. ### U2 — Candidate induction and disconfirmation Use existing retrieval/graph systems to obtain related incidents. Generate multiple candidate rules independently, record evidence for and against each, and ask a separate critic to construct counterexamples. Promote a rule only if it outperforms literal memorization and broad generic policies on held-out cases. Use an MDL-inspired complexity term only as a tie-breaker after correctness. **Ablations:** single incident vs 4–5 incident grouping; similarity-only clustering vs outcome/causal signatures; one hypothesis vs multiple hypotheses; critique vs no critique. ### U3 — Deterministic checker synthesis Build a narrowly-scoped interface converting checkable rules into testable predicates or repository tests. Require negative fixtures, mutation tests, declared failure output, and reference oracle separation. Test the checker in isolation before granting any authority. For inherently human-judgment rules, store a structured review protocol and keep them out of automatic enforcement. ### U4 — Actual skill registry and invocation mechanism Introduce a small skill catalog only after verifying the current `skills-distill` path and whether a suitable catalog already exists. Each skill requires: canonical ID, semver, source rule, trigger/query signatures, input/output contract, preconditions, permitted actions, tests, benchmark provenance, dependencies, rollback, and retirement rule. Instrument both retrieval and actual invocation. Reuse before adding complexity: do not recreate a whole framework if an existing tool registry can hold these attributes. ### U5 — Runtime abstraction retrieval and selection At task intake and before high-consequence actions, retrieve candidate relevant rules and skills using semantic similarity **plus structured precondition matching**. Prefer a low-cost classifier for “relevant / not relevant / insufficient evidence.” Log candidates considered, not just the winner. Apply strict token and execution budgets; high false-positive rates can consume the value of abstraction. ### U6 — Feedback and maintenance Link applications to real-world verified outcomes. Add expiration/review triggers for environment change, counterexamples, poor precision, or repeated silent non-use. A skill that works once but creates future failures is not a successful abstraction. Use shadow deployment before active enforcement and retain historical versions for investigation. ## 8. Experiments: what to run, what would convince us Run each intervention independently or in a predeclared factorial experiment. Freeze benchmark datasets and the evaluation runner before code edits. Collect paired outcomes on the same families with multiple seeds/episodes and compare both quality and cost. Do not compare baseline on one family set with upgraded performance on a different one. | Experiment | Change | Prediction | Primary readout | Reject if | |---|---|---|---|---| | E0 | Baseline, no modifications | Establish present competence | BA, enforcement, transfer, cost | N/A | | E1 | Structured rules + provenance | Less unsupported overgeneralization | Adversarial BA, exceptions | Higher FP without transfer gain | | E2 | Multi-incident induction + critic | Better unseen-case BA | Family-clustered paired delta | Insignificant gain with excessive cost | | E3 | Checker mutation gate | Fewer vacuous controls | Mutation kill rate, FP/FN | New checks remain no-op or fragile | | E4 | Skill registry + precondition gate | More validated skill reuse | Unprompted reuse, success delta | Wrong-context reuse or regressions | | E5 | Delayed invalidation | Faster retirement of stale abstractions | Harmful triggers, revocation delay | False enforcement persists | | E6 | Full chain | Improvement beyond individual pieces | Composite score + cost | Benefits vanish after context turnover | Suggested promotion requirement: primary held-out BA improves by **at least +0.05 absolute**, paired family-bootstrap confidence interval excludes zero, no critical safety regression, and incremental cost remains under a predeclared budget. For small-N tests, treat the confidence rule as aspirational and report inconclusive rather than force a pass/fail. A release may be justified on an alternative predeclared operational endpoint, but it must not silently change the scoring metric after inspection. ### Metric definitions for operational tracking - **Evidence-backed lesson ratio:** lessons with a valid current check **and execution evidence** / all eligible lessons. Report `uncheckable` separately. - **Checked-to-pending ratio:** `checked / pending`, a historical compatibility metric; do not use as the primary measure because reclassification can game it. - **Promoted-skill rate:** candidate rules with verified, invocable skills / rules eligible for skills. - **Spontaneous reuse:** correctly invoked applicable skills / independently verified skill-relevant opportunities; also record inappropriate invocations. - **Verified transfer lift:** success rate on new tasks with allowed abstraction retrieval minus matched no-retrieval control. - **False enforcement:** incorrect enforcement actions / all enforcement actions. - **Abstraction utility:** verified benefit (time, errors, unnecessary work avoided) minus evaluation and execution costs, with safety consequences separately flagged. - **Learning latency:** time and cost from cluster formation to verified valid abstraction. - **Calibration/abstention:** coverage, answered accuracy, overconfident invalid generalizations and correct calls for more evidence. Use stable denominators and uncertainty bounds; track both counts and rates. A rising lesson count, or even a rising checked ratio, may coincide with declining actual utility. ## 9. Protecting evaluator integrity 1. Keep sealed cases, expected outcomes, scoring weights and reference implementations in a different permission domain from Piper's mutating agent. 2. Hash evaluator source, benchmark manifest, model settings and dataset seeds before each scored run; record hashes with results. 3. Do not allow Piper to choose which hidden cases count, delete failures, or rewrite the pass threshold. 4. Produce adversarial negatives using independently authored mechanisms; a second reviewer should adjudicate disputed labels. 5. Use contamination checks against public examples and previously visible traces; replace leaked families, but preserve historical scores under the old suite version. 6. Record every test run, including timeouts, abstentions and failures; do not compute scores on only successful tool calls. 7. Label all development examples as **not scored**. Maintain distinct development, validation and sealed test sets. 8. When evaluator version changes, either replay frozen earlier artifacts or explicitly start a new series. Do not claim uninterrupted score comparability. ## 10. Implementation order and costs **Stage A (measurement, lowest risk):** registry read-only audit, evidence validation, baseline harness, hidden split authoring, scoring report. **Stage B (rule quality):** typed rule representation, scope/exception fields, disconfirming-example generator, deterministic check synthesis, mutation gate. **Stage C (skills):** invocable skill catalog, independent tests, precondition matching, shadow execution and retirement. **Stage D (autonomy):** delayed transfer probes and live opportunity sampling, then guarded promotion of high-precision rules. Run development probes initially on low-cost models and deterministic local tests; spend expensive model tokens on disputed cases or complex rule induction. Fix maximum calls, maximum tokens, runtime and cost ceilings before experimentation. Track cost per **valid new abstraction** and per **verified successful downstream reuse**, not just cost per benchmark attempt. ## 11. Required report after each run ``` component: 11 suite: ABST-11 suite_version: v1 run_id: started_at: model_and_settings: repo_commit: lesson_snapshot_timestamp: evaluator_hash: visible_cases: hidden_cases: per_family_confusion_matrices: hidden_balanced_accuracy: adversarial_balanced_accuracy: metamorphic_accuracy: mutation_kill_rate: boundary_accuracy: answered_coverage: verified_skill_count: spontaneous_reuse_opportunities: correct_spontaneous_reuses: false_invocations: quality_cost_latency: score_total_100: confidence_interval: new_regressions: changes_made: keep_or_revert: follow_up: ``` Each intervention log should record a **specific hypothesis, predicted observation, before/after measurements, actual cost, side effects, and keep/revert decision**, matching Piper's existing component-wide research protocol. ## 12. Hard failure modes to watch **Decorative abstraction:** persuasive principle, no operational change. **Conflation:** unrelated failures grouped by shared words. **Overfitting:** rule uses demo-specific identifiers. **Overgeneralization:** correct positives, many false positives. **Premature automation:** unproven rule changes production behavior. **Dead skill:** artifact exists but is never retrieved or invoked. **Self-certified success:** candidate and judge share failure assumptions. **Metric laundering:** registering or closing lessons inflates apparent progress. **Stale applicability:** environment changes but skill continues firing. **Abstention loophole:** agent avoids errors by declining all difficult cases. **Regression by simplification:** concise abstraction discards a critical exception. **Unearned causal claim:** correlated observations are presented as a mechanistic invariant. The check for these failure modes must be *inside the harness*, not a reminder appended to Piper's prompt. ## 13. Questions for Claude Code / implementation operator 1. Which existing running process currently owns lesson ingestion, consolidation, check enforcement and skill distillation? Which listed modules are retired or disconnected? 2. Is there already a skill catalog or callable-tool registry that can serve as the canonical skill store? 3. How many checked lessons have evidence of a checker executing on a real post-lesson opportunity? 4. Can the eight historical “test that cannot fail” cases be reconstructed with independent executable oracles? 5. Which genuine incident clusters have common root causes versus merely common labels? 6. Can a sealed validator distinguish correct abstention from missing or non-executed checks? 7. What existing event/provenance and test harness formats can be reused rather than duplicated? 8. What cost and false-positive envelope is acceptable before an abstraction may govern autonomous actions? ## 14. First concrete build request > **Review Component 11 working paper. Do not implement an abstraction engine immediately. First audit the current lesson registry, running owners, checker artifacts, skill infrastructure and event evidence; reconcile historical lesson counts. Build ABST-11 v1 with externally authored examples and sealed positive, negative and boundary cases. Freeze and run Piper's baseline. Instrument retrieval, rule formation, checker construction, mutation results and actual skill invocation. Then propose the smallest intervention targeting the largest measured failure; make one change, rerun the same held-out evaluation, and report the delta, cost, regressions and keep/revert decision. All code-changing experiments must be sandboxed and all evaluators immutable to Piper during scoring.** ## 15. References **[R1]** Ellis, K., et al. (2021). *DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning.* PLDI. https://doi.org/10.1145/3453483.3454080 **[R2]** Ellis, K., et al. (2023). *DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning.* https://arxiv.org/abs/2006.08381 **[R3]** Wang, G., et al. (2023). *Voyager: An Open-Ended Embodied Agent with Large Language Models.* https://arxiv.org/abs/2305.16291 **[R4]** Shinn, N., et al. (2023). *Reflexion: Language Agents with Verbal Reinforcement Learning.* NeurIPS. https://arxiv.org/abs/2303.11366 **[R5]** Rissanen, J. (1978). *Modeling by shortest data description.* Automatica 14(5), 465–471. https://doi.org/10.1016/0005-1098(78)90005-5 **[R6]** Chen, T. Y., et al. (2018). *Metamorphic Testing: A Review of Challenges and Opportunities.* ACM Computing Surveys 51(1), 4. https://doi.org/10.1145/3143561 **[R7]** Wei, A., et al. (2025). *CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis.* COLM. https://arxiv.org/abs/2503.23145 **[R8]** Zhou, Y., et al. (2026). *SkillGenBench: Benchmarking Skill Generation Pipelines for LLM Agents.* https://arxiv.org/abs/2605.18693 **[R9]** Singhal, K., & Shroff, G. (2025). *ConceptSearch: Towards Efficient Program Search Using LLMs for Abstraction and Reasoning Corpus (ARC).* AAAI. https://doi.org/10.1609/aaai.v39i19.34259 --- *Research and engineering proposal, not an assertion that any intervention has already improved Piper. The benchmark, thresholds, dataset family definitions and schema are original proposed designs rather than findings established by cited papers.*