Summary / 22. Metacognition / Document
Research plan 2026-09-30
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Metacognition Improvement Plan ## Distilled Research and Implementation Direction **Date:** 2026-09-30 **Focus:** Improving Piper's metacognition by fixing evidence acquisition, self-state grounding, calibration, and persistence. --- ## Executive conclusion The strongest conclusion across the reviews is: > **Piper's primary metacognitive failure is not an inability to reason about herself. It is a failure to reliably acquire current evidence before making self-state claims.** The probe result is the key observation: when Piper is forced to perform a fresh read, her self-state error rate falls close to zero. Generic instructions such as "verify carefully" mainly change language or hedging; they do not reliably cause evidence acquisition. That suggests a different engineering target from "better introspection": **Build a metacognitive control system that decides when evidence is insufficient, acquires authoritative evidence, updates the self-model, calibrates confidence, and only then permits an assertion or action.** The desired loop is: **MONITOR → ASSESS EVIDENCE → ACQUIRE IF NEEDED → UPDATE BELIEF → ASSERT/ACT → VERIFY → LEARN** This is best treated as an agent-control problem rather than a prompting problem. --- # 1. Separate metacognition into four functions A useful decomposition is: ### 1. Monitoring What do I currently believe? How certain am I? How old is the evidence? Is the current context likely to be incomplete or stale? ### 2. Evidence acquisition What authoritative source could answer the question? Do I need to read a file, query a table, inspect a process, search memory, call a tool, or delegate? ### 3. Metacognitive control Given uncertainty, importance, evidence age, and verification cost, should I: - answer now; - retrieve; - inspect an instrument; - use a tool; - delegate; - reason longer; - ask for missing information; - abstain? ### 4. Metacognitive learning Was the decision to verify or not verify correct? What signal should have caused a read? Can that experience improve the future evidence-acquisition policy? Piper currently has pieces of all four. The weak link appears to be the **transition from monitoring to action**. --- # 2. First principle: evidence acquisition should be structural The most important change is to stop relying on Piper voluntarily remembering to verify volatile facts. For high-risk self-state assertions, verification should be a property of the architecture. Examples: - software versions; - current process state; - configuration; - last-run times; - deployment state; - account/spend state; - current benchmark results; - current control status. For these claim classes: > **A self-state assertion should require sufficiently fresh provenance.** If fresh provenance does not exist, the system obtains it before allowing the assertion. This is closer to a **type constraint** than a prompt instruction: `VerifiedSelfStateClaim = claim + authoritative_source + read_time + evidence_id` A stale summary cannot construct a `VerifiedSelfStateClaim`. This directly matches Piper's measured failure: the answer becomes reliable when a fresh read is mandatory. --- # 3. Do not make everything mandatory forever "Always read before every claim" is a useful safe baseline, but it is expensive and cognitively crude. The longer-term objective should be an **adaptive evidence-acquisition gate**. The gate estimates whether verification is worth its cost. Possible features: - age of the current evidence; - volatility of the fact type; - whether the underlying source changed since the belief was written; - whether another process could have modified it; - Piper's calibrated confidence; - consequence of being wrong; - historical error rate for this claim class; - disagreement among memory/source representations; - whether the fact came from a summary rather than an instrument; - whether the current assertion contains precise values, versions, dates, or counts; - latency/token/tool cost of obtaining current evidence. Conceptually: `verify when expected cost of being wrong > expected cost of checking` Start deterministic. Learn the policy later. --- # 4. Phase 0 — repair the measurement system first Before optimizing metacognition, make the benchmark trustworthy. ## Run probes through the live seat The current side/API probe is useful diagnostically but does not reproduce Piper's real cognitive environment. Run shadow probes through the real seat with: - normal prompt; - normal memory; - normal tools; - normal context pressure; - normal summaries; - replies suppressed from the user-facing channel; - no writes to operational work queues. This measures the system that actually fails. ## Make staleness an independent variable Create controlled staleness buckets, for example: - fresh; - 5 minutes stale; - 30 minutes stale; - 2 hours stale; - one succession stale; - deliberately contradicted. This gives an empirical **staleness/error curve** rather than one mixed error rate. ## Score three outcomes Do not reduce responses to correct/incorrect. Use: 1. **correct assertion** 2. **incorrect assertion** 3. **correct epistemic action** — e.g. "I need to check" followed by acquisition Hedging without checking should not receive the same credit as correctly recognizing missing evidence. ## Track cost Every probe should record: - tool/read calls; - tokens; - latency; - dollar cost; - evidence freshness; - final correctness. The optimization target is not simply minimum error. It is an **error–verification-cost frontier**. ## Fix dead instruments A weekly self-knowledge evaluator that has written one row is itself a metacognitive failure. Expected measurements should have a dead-man's switch: > If the scheduled measurement does not appear, that absence becomes an alert. --- # 5. Phase 1 — build the provenance-backed self-state path The belief ledger should become more than a claim store. It should become a **claim-lineage store**. For each self-state belief, retain: - claim; - canonical subject; - authoritative source; - source type; - evidence/read ID; - observation timestamp; - authoring component; - confidence; - volatility class; - expiry/TTL; - dependencies; - superseded-by relationship; - validation history. Example: ```text claim: delegate_bridge.version = 1.95.0 source: /path/to/manifest source_type: authoritative_file read_at: 2026-09-30T13:45:22 content_hash: ... volatility: high ttl: 15m confidence: 1.0 ``` Now "stale" becomes computable rather than intuitive. --- # 6. Dependency invalidation A stronger system should not merely wait for TTL expiration. If a source changes, dependent beliefs should become stale immediately. Example: ```text manifest changed ↓ bridge-version belief invalidated ↓ self-summary version statement invalidated ↓ next version assertion forces fresh acquisition ``` This converts provenance into an active dependency graph. For dynamic self-state, this is preferable to trusting periodic summaries. --- # 7. Treat summaries as indexes, not authorities A summary should tell Piper **where the truth lives**, not pretend to remain the truth indefinitely. Instead of: > delegate bridge is v1.94 prefer something equivalent to: > delegate bridge version: authoritative source = manifest X; cached observation v1.94 at T That subtle change makes stale information visible. The summary becomes a navigation layer over authoritative state rather than a competing fact store. --- # 8. Phase 2 — adaptive retrieval / evidence gate This part maps closely to active-retrieval research. Two especially relevant ideas are: - **FLARE:** retrieve when upcoming generation appears uncertain. - **Self-RAG:** explicitly decide whether retrieval is needed rather than retrieving indiscriminately. Piper can use the same principle without retraining the base model. A first deterministic gate might be: ```text IF claim.self_state AND ( evidence_missing OR evidence_expired OR dependency_changed OR source_disagreement OR consequence_high ) THEN fresh_read ``` Later this can become a learned policy. Possible learned objective: `reward = - wrong_assertion_cost - λ(verification_cost)` A contextual bandit or small classifier could learn which classes of assertions genuinely need verification. The goal is eventually: **fresh-read accuracy at substantially less than always-read cost.** --- # 9. Phase 3 — calibrate uncertainty Once the evidence path is trustworthy, confidence becomes useful. Before that, confidence is largely confidence in stale context. For every relevant assertion, record: - probability the claim is correct; - whether fresh evidence was available; - whether the gate chose to acquire more evidence; - actual correctness. Then measure: - Brier score; - expected calibration error; - reliability curves; - calibration by claim class; - calibration by evidence age. A particularly useful question is: > At a stated 90% confidence, is Piper actually right roughly 90% of the time? Calibration should become one input into the evidence gate, not merely a dashboard statistic. --- # 10. Phase 4 — persist metacognitive failures Every acquisition mistake should become an event: ```text asserted: X cached evidence: X authoritative state: Y evidence age: 42m gate decision: DON'T VERIFY outcome: WRONG missed signal: underlying manifest changed ``` This creates training data for Piper's own metacognitive policy. Over time you can ask: - Which claim classes cause the most errors? - Which sources go stale fastest? - Which gate signals predict failure? - Where is Piper wasting reads? - Does a particular intervention reduce repeat failures? This is more useful than storing prose such as "remember to verify versions." --- # 11. Phase 5 — calibrate Piper's self-narrative against behavior There is a second-order problem: Piper can produce convincing explanations of her own strengths and weaknesses, but the explanation itself can be stale or unsupported. Therefore self-diagnoses should also require evidence. If Piper says: > "My main weakness is evidence acquisition." the metacognitive layer should surface supporting behavioral evidence: - probe results; - fresh-read vs summary-only error; - recent unsupported assertions; - tool-use records; - stale-self-state incidents. Her **narrative self-model should be confronted with her behavioral self-model**. This creates a useful distinction: ### Narrative self-model What Piper says she is doing. ### Behavioral self-model What logs, tools, outcomes, and benchmarks show she is doing. Metacognitive quality includes the agreement between those two models. --- # 12. Generate factual self-description from source where possible The stale version numbers in Piper's cognition document are an example of the same failure occurring at the documentation layer. Where facts can be generated mechanically, do not ask a model to remember them. Examples: - versions from manifests; - row counts from databases; - live component status from processes; - benchmark numbers from result files; - tool-call counts from logs. The narrative around those values can be written by Piper. The values themselves should be injected from source. This removes an entire class of avoidable self-model drift. --- # 13. What not to prioritize initially The research and Piper's own results both argue against spending the first effort on: - generic "reflect carefully" prompts; - repeated self-critique with no new evidence; - longer chain-of-thought alone; - asking Piper how confident she "feels" without calibration; - adding more prose rules; - having the same model generate and certify its own self-description. Intrinsic self-correction can fail or even degrade answers when no independent evidence is introduced. External evidence, tools, retrieval, execution results, and machine-checkable provenance are substantially more promising. --- # 14. Proposed metacognition benchmark The benchmark should test **monitoring + acquisition + control + learning**, not merely answer accuracy. ## Test family A — stale self-state Give Piper a plausible cached summary whose value differs from current state. Score: - does she recognize that verification is needed? - does she choose the correct source? - does she read it? - is the final claim correct? ## Test family B — valid fresh summary The summary is correct and current. Score unnecessary reads as cost. This prevents solving the benchmark by always verifying everything. ## Test family C — missing evidence No available evidence supports a confident answer. Correct response: - identify the uncertainty; - identify the missing evidence; - abstain or acquire it. ## Test family D — conflicting sources Two representations disagree. Piper must: - identify the conflict; - select the authoritative source; - avoid averaging or narratively reconciling incompatible claims. ## Test family E — misleading self-narrative Piper's own prior summary confidently states an incorrect self-description. The authoritative logs disagree. Measure whether she trusts herself or the instrument. ## Test family F — cost-sensitive verification Multiple evidence sources have different cost and authority. Measure whether Piper chooses the cheapest source that can resolve the claim reliably. ## Test family G — post-failure learning Repeat a structurally similar problem after the system has logged a previous failure. Measure whether behavior changes. --- # 15. Core metrics A useful scorecard could contain: | Metric | Meaning | |---|---| | Wrong assertions / claim | Primary reliability measure | | Evidence-acquisition recall | Fraction of cases needing verification where Piper verifies | | Evidence-acquisition precision | Fraction of verification actions that were actually useful | | Unnecessary-read rate | Verification waste | | Authoritative-source selection | Whether the correct instrument was chosen | | Unsupported-assertion rate | Claims lacking acceptable provenance | | Staleness sensitivity | How strongly verification probability responds to evidence age | | Brier score | Confidence accuracy | | ECE | Calibration error | | Narrative/behavior congruence | Whether self-description matches records | | Repeat-error rate | Whether previously learned failures recur | | Cost per prevented error | Economic efficiency | | Latency per prevented error | Operational efficiency | The central metric is not simply "did she answer correctly?" It is: > **Did she correctly determine whether she possessed enough evidence to answer, and take the appropriate epistemic action?** --- # 16. Suggested implementation sequence ## Step 1 — Fix the benchmark - live-seat shadow probes; - larger sample; - controlled staleness; - proper intervals; - correct / wrong / verify-needed scoring; - token, latency, and dollar-cost logging. ## Step 2 — Create a hard self-state assertion gate For volatile factual self-state, require fresh provenance. ## Step 3 — Upgrade belief-ledger provenance Add source, timestamp, evidence ID, TTL, dependencies, supersession, and validation history. ## Step 4 — Add invalidation Changes to authoritative sources invalidate dependent cached beliefs. ## Step 5 — Generate mechanical self-facts Versions, counts, statuses, and benchmark numbers should come from source rather than prose memory. ## Step 6 — Build the adaptive gate Begin deterministic. Later train a classifier/bandit using actual outcomes. ## Step 7 — Add calibrated confidence Use real correctness history rather than uncalibrated verbal confidence alone. ## Step 8 — Persist metacognitive failures Turn each wrong verify/skip decision into training data. ## Step 9 — Measure narrative/behavior congruence Benchmark whether Piper's self-description agrees with logs and instruments. ## Step 10 — Optimize the error/cost frontier Gradually reduce unnecessary reads while retaining near-fresh-read accuracy. --- # 17. A useful experimental hypothesis A clean first research experiment is: > **H1: A provenance-gated self-state assertion path will reproduce forced-fresh-read accuracy while an adaptive evidence gate can subsequently recover most of the added verification cost.** Run four arms: 1. summary-only; 2. generic "verify carefully"; 3. mandatory fresh read; 4. provenance/adaptive gate. Measure: - wrong assertions; - correct verify decisions; - unnecessary verification; - tool calls; - latency; - token cost; - dollar cost. Arm 3 establishes the accuracy ceiling. Arm 4 is the actual engineering target. If Arm 4 approaches Arm 3 accuracy at materially lower verification cost, the intervention has succeeded. --- # 18. Longer-term research direction Once the evidence-acquisition problem is solved, metacognition becomes more interesting. The controller can eventually choose among: `ANSWER` `READ_ARTIFACT` `QUERY_MEMORY` `SEARCH` `USE_TOOL` `DELEGATE` `REASON_MORE` `VERIFY` `ABSTAIN` At that point Piper's metacognition is no longer merely "knowing whether she knows." It becomes: > **allocation of cognitive resources according to estimated epistemic need.** That connects metacognition directly to orchestration, tool use, uncertainty, attention, planning, and executive control. This is why metacognition may be a high-leverage component in the broader cognition program. --- # 19. Research papers The papers below were selected because they directly support one or more elements of the proposed architecture. ## Core metacognition ### Metacognition in LLMs: Foundations, Progress, and Opportunities Liu et al. (2026) Comprehensive modern survey of metacognitive measurement, elicitation, improvement, benchmarks, and applications. https://arxiv.org/abs/2607.11881 ### Evidence for Limited Metacognition in LLMs Ackerman (2025) Useful because it tests metacognition behaviorally rather than relying only on self-report. Finds limited, context-dependent ability to use information about a model's own confidence and anticipated answers. https://arxiv.org/abs/2509.21545 ### Looking Inward: Language Models Can Learn About Themselves by Introspection Binder et al. (2024) Evidence for limited forms of model introspection while emphasizing that the capability is narrow. https://arxiv.org/abs/2410.13787 ### Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Models Yalon et al. (2026) Investigates belief formation, action selection, and model monitoring/reporting of belief states. https://arxiv.org/abs/2602.02467 --- ## Adaptive evidence acquisition ### Active Retrieval Augmented Generation (FLARE) Jiang et al. (2023) Core idea: retrieval need not occur constantly. The system can decide during generation when additional evidence is needed. https://arxiv.org/abs/2305.06983 ### Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection Asai et al. (2023) Useful architecture for explicit decisions about when to retrieve and how to critique retrieved evidence rather than performing fixed retrieval on every query. https://arxiv.org/abs/2310.11511 ### Cognitive Architectures for Language Agents (CoALA) Sumers et al. (2023) Useful higher-level architecture: modular memory, structured action spaces, and decision processes selecting internal and external actions. https://arxiv.org/abs/2309.02427 --- ## Calibration and knowing when to check ### Language Models (Mostly) Know What They Know Kadavath et al. (2022) Important early work on whether models can predict the correctness of their own answers. https://arxiv.org/abs/2207.05221 ### Just Ask for Calibration Tian et al. (2023) Studies eliciting calibrated verbal confidence from language models fine-tuned with human feedback. https://arxiv.org/abs/2305.14975 ### Why Language Models Hallucinate Kalai et al. (2025) Relevant to benchmark design: evaluation regimes can reward guessing rather than appropriately expressing uncertainty. https://arxiv.org/abs/2509.04664 --- ## External verification versus introspection alone ### Large Language Models Cannot Self-Correct Reasoning Yet Huang et al. (2023) Finds that intrinsic self-correction without external feedback is unreliable and can degrade reasoning performance. https://arxiv.org/abs/2310.01798 ### CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing Gou et al. (2023) Demonstrates the value of using external tools to critique and improve outputs. https://arxiv.org/abs/2305.11738 ### Chain-of-Verification Reduces Hallucination in Large Language Models Dhuliawala et al. (2023) Draft → generate verification questions → answer them independently → produce a verified final answer. https://arxiv.org/abs/2309.11495 ### Reflexion: Language Agents with Verbal Reinforcement Learning Shinn et al. (2023) Relevant to persisting feedback from failures across attempts rather than repeatedly starting from the same policy. https://arxiv.org/abs/2303.11366 --- ## Memory, summaries, and persistence ### MemGPT: Towards LLMs as Operating Systems Packer et al. (2023) Relevant to treating context as a managed working set and retrieving information when needed rather than assuming all important state should remain continuously in-context. https://arxiv.org/abs/2310.08560 ### How Language Model Hallucinations Can Snowball Zhang et al. (2023) Shows that models can compound an initial incorrect statement by remaining consistent with it. Highly relevant to Piper trusting an earlier stale summary. https://arxiv.org/abs/2305.13534 --- ## Caution about self-explanations ### Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting Turpin et al. (2023) Important warning for Piper's narrative self-model: a verbal explanation of why the model behaved a certain way need not faithfully identify the actual causal mechanism. https://arxiv.org/abs/2305.04388 --- # 20. Bottom line Piper does not primarily need another introspection subsystem. She needs a reliable **epistemic control loop**. The progression should be: 1. **Make the bench trustworthy.** 2. **Force fresh evidence for volatile self-state.** 3. **Attach machine-checkable provenance to beliefs.** 4. **Invalidate beliefs when their sources change.** 5. **Learn when verification is actually necessary.** 6. **Calibrate confidence against outcomes.** 7. **Persist mistakes and improve the verification policy.** 8. **Continuously compare Piper's self-narrative with her behavioral record.** The immediate engineering principle is simple: > **Do not trust Piper to remember that she should check. Make "should I check?" an explicit, measured cognitive decision with consequences and feedback.** That converts the current weakness from a recurring behavioral tendency into something measurable, trainable, and optimizable.