Summary / 2. Attention / Document
2026-10-04
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 2 — Attention ## A Value-of-Computation Architecture, Benchmark Suite, and Engineering Plan for Persistent LLM Agents **Status:** Research synthesis and engineering design **Date:** 2026-10-04 **Component:** 2 — Attention **Working definition:** *Selecting what matters right now.* --- ## Abstract Piper already has several mechanisms that reduce incoming noise: trusted-sender filtering, batching, volatile state injection, belief and trigger deltas, an `absence_gate`, and retrieval-event logging. These mechanisms are useful, but they are not yet a measured form of agentic attention. They decide what arrives, what is placed in context, or what retrieval returns. They do not reliably decide which available computation deserves Piper's next unit of scarce cognitive effort. This paper defines **agentic attention** as a metareasoning policy over possible computations. At any moment Piper may inspect an authoritative instrument, retrieve a memory, check a belief, read a worker result, continue the current task, switch to an interruption, or stop deliberating and act. Attention is the policy that allocates finite tokens, tool calls, latency, and reasoning effort among those competing options. The organizing principle proposed here is **Value of Computation (VOC)**. For an inspection action, VOC reduces to a practical form of **Value of Information (VOI)**: inspect something when the expected improvement in the eventual decision is worth more than the cost of inspecting it. This avoids treating uncertainty, staleness, novelty, urgency, source authority, and consequence as unrelated terms in a hand-built salience equation. Most are evidence used to estimate one underlying quantity: *how much might this computation improve the decision, net of cost?* The recommended architecture separates five functions that must not be conflated: 1. **Input filtering** determines what can reach Piper. 2. **Candidate generation** identifies possible things worth inspecting or reasoning about. 3. **Attention selection** chooses which candidate receives the next unit of cognition. 4. **Evidence use and reasoning** update beliefs and actions after inspection. 5. **Post-hoc enforcement** blocks claims that lack required evidence. The last point is important. A validator that forces Piper to read a live instrument can prevent an error without improving attention. Therefore evaluation must report both **spontaneous attention** and **validator-rescued performance**. The core benchmark is a **budgeted inspection task with ground truth by construction**. Piper receives many inspectable objects but can inspect only a few. The evaluator knows exactly which inspections can change the optimal decision and can therefore compute attention regret. A second, equally important condition removes the explicit instruction to inspect anything. A plausible stale summary is present, the authoritative instrument is available, and Piper is simply asked to make a state-sensitive decision. The difference between performance in the announced-budget condition and the unprompted condition measures whether Piper truly initiates inspection herself or merely performs well after inspection has been made salient. The paper also specifies a four-stage failure taxonomy, a sealed hold-out protocol designed for an agent that can read its own repositories and memories, attention tracing, switching and anti-thrashing rules, stale-state invalidation, ablations, learning from realized decision value, and a staged implementation plan. --- # 1. Problem Statement Piper's current attention description is substantially about **filtering** and **prompt/context organization**: - trusted-sender lists determine which messages wake her; - worker results and digests are batched; - absence and register information are prepended; - retrieval events record what a retrieval returned; - volatile state is injected without destabilizing the cached prefix. Those mechanisms are real and useful. They reduce irrelevant work and provide observability. The reviewer identified the remaining gap correctly: once information is available, Piper has no enforceable and directly measured mechanism for allocating limited thought to what matters most. The characteristic failure is **summary substitution**: ```text internal summary / remembered state ↓ appears sufficient ↓ authoritative instrument not inspected ↓ confident state-sensitive claim ↓ claim is wrong ``` The important question is not merely whether Piper *can* retrieve the correct evidence. It is whether she recognizes that retrieval or inspection is worth doing before she commits to a belief or action. That distinction motivates the operational target for this component: > **Agentic attention is the policy that decides which available computation receives the next unit of cognitive effort, given current goals, beliefs, costs, deadlines, uncertainty, and possible decision consequences.** --- # 2. Boundary Conditions: What Attention Is and Is Not The component will become impossible to benchmark if attention is allowed to absorb every neighboring function. The following boundaries should be explicit in both implementation and scoring. | Function | Question it answers | Belongs to Attention? | |---|---|---:| | Input filtering | "Should this event reach Piper at all?" | No | | Candidate generation | "What things could plausibly deserve inspection?" | Adjacent; score separately | | Retrieval | "Can I obtain the requested information?" | No | | Context management | "What text is in the context window?" | No | | Attention | "Which available computation should receive effort now?" | **Yes** | | Evidence integration | "What should I believe after seeing it?" | No | | Reasoning | "What follows from the evidence?" | No | | Planning | "What sequence of actions should I take?" | No | | Uncertainty estimation | "How unsure am I?" | No, but supplies features | | Metacognition | "Should I inspect/reason/check more?" | Overlapping theoretical parent | | Enforcement | "May this claim/action proceed without evidence?" | No; governance/safety layer | This boundary is especially important for summary substitution. The same outward error can have several causes: - Piper never generated "inspect live source" as a candidate: **candidate-generation failure**. - She generated it but chose something else: **attention-selection failure**. - She tried to inspect it but retrieval/tooling failed: **retrieval failure**. - She saw the evidence but continued using the summary: **evidence-utilization failure**. - She incorporated the evidence but reasoned incorrectly afterward: **reasoning failure**. - A claim validator forced the read and saved the answer: **enforcement success**, not attention success. The benchmark must preserve these distinctions. --- # 3. Theoretical Foundation: Value of Computation ## 3.1 Rational metareasoning Russell and Wefald framed metareasoning as the problem of selecting computational actions under bounded resources. A computation is useful because it may improve the external action ultimately chosen. This gives attention a decision-theoretic foundation rather than a psychological metaphor. Let: - \(b\) = Piper's current belief state, - \(c\) = a candidate computation, - \(o\) = the possible result of that computation, - \(a\) = an external action, - \(U(a \mid b)\) = expected utility of action \(a\) under belief state \(b\), - \(Cost(c)\) = token, latency, tool, switching, or opportunity cost of computation \(c\). Then: \[ VOC(c \mid b) = \mathbb{E}_{o} \left[ \max_a U(a \mid b,o) \right] - \max_a U(a \mid b) - Cost(c) \] A computation deserves attention when its expected ability to improve the final decision exceeds its cost. For an external inspection, this is effectively **net Value of Information**. ## 3.2 Why this is better than a long additive salience formula A naive attention score might add: ```text goal relevance + uncertainty + staleness + consequence + source authority + novelty + deadline pressure - inspection cost ``` This is useful as a first sketch, but the terms are not independent or commensurable. For example: - high uncertainty is irrelevant if the answer cannot affect any important decision; - staleness matters because it raises the probability that a current belief is wrong; - source authority matters because an authoritative inspection is more likely to reduce uncertainty; - consequence matters because changing a high-cost wrong decision is more valuable than changing a trivial one; - novelty should rarely be a direct priority signal at all, because novelty can become an attentional attractor without decision value. The more defensible question is: > **How likely is this computation to change what Piper should believe or do, how valuable would that change be, and what will the computation cost?** The other features are inputs to that estimate. ## 3.3 Practical approximation Exact VOC is generally intractable. Piper therefore needs an approximation. A useful engineering target is: \[ \widehat{VOC}(c) = P(\text{decision changes} \mid c,x) \times E(|\Delta U| \mid \text{change},c,x) - \lambda Cost(c) - \mu SwitchCost(c) \] where \(x\) contains features such as: - active-goal relevance; - current confidence; - evidence age; - volatility of the underlying state; - source authority for this claim type; - contradiction or change signals; - deadline proximity; - reversibility of the pending action; - historical usefulness of similar inspections. This is not asserted to be the final formula. It is a useful approximation whose predictions can be calibrated against realized outcomes. --- # 4. Research Basis The relevant literature converges on a few principles rather than one ready-made "LLM attention controller." ## 4.1 Metareasoning and bounded computation Russell & Wefald (1991), Horvitz (1988), Hay et al. (2012), and Callaway et al. (2018) treat computation itself as something that must be selected under cost. Their common lesson for Piper is that the agent should not merely ask *what can I compute?* but *which computation is worth doing next?* Callaway et al. show that computation-selection policies can be learned when exact rational metareasoning is too expensive. This is important for Piper because the controller itself must not become a larger computational burden than the decisions it is supposed to improve. ## 4.2 Resource-rational analysis Lieder and Griffiths argue that cognitive strategies can be analyzed as adaptations to limited computation. The useful engineering translation is not that human attention is necessarily optimal. It is that Piper's controller should be evaluated against the utility it extracts from a finite budget rather than against an unconstrained ideal that reads everything. ## 4.3 Active retrieval Self-RAG and FLARE demonstrate partial versions of the relevant decision: retrieval should be conditional rather than automatic. Self-RAG learns when retrieval is useful; FLARE uses upcoming low-confidence generation to trigger retrieval. Neither solves Piper's whole attention problem, but both support the idea that retrieval timing is itself a policy decision. ## 4.4 Long-context utilization "Lost in the Middle" demonstrates that information being present in context does not imply reliable use of that information. This directly supports the distinction: ```text available in context ≠ attended ≠ used correctly ``` That matters for Piper because additional context can increase apparent information availability while reducing reliable utilization. ## 4.5 Cognitive architectures ACT-R, Soar, and Global Workspace/LIDA provide useful architectural analogies: - explicit competition among candidate operations; - goal-conditioned activation; - thresholds; - limited broadcast/workspace; - switching and control costs. These systems should not be copied literally. Their relevance is that attention is represented as an explicit selection process rather than an instruction to "focus." ## 4.6 Truth maintenance and stale state Doyle's Truth Maintenance System provides a particularly useful principle for Piper: a belief should carry information about *why* it is believed, and dependent conclusions should become revisable when their support changes. For attention, this means a state change can generate an attention candidate automatically: ```text authoritative source changes ↓ dependent belief marked dirty ↓ dirty high-consequence belief generates inspection candidate ↓ attention controller decides when to inspect ``` This is superior to relying on Piper to remember that an old summary might be stale. --- # 5. Diagnosis of Piper's Current Attention System ## 5.1 Strengths Piper already has infrastructure that an attention system can exploit: - trusted-sender filtering; - batched worker results and digests; - explicit injected state lines; - belief/trigger deltas; - `absence_gate`; - `register-prepend`; - `retrieval-events` logging; - persistent memory; - asynchronous workers; - known authoritative instruments. This means the problem is not starting from zero. There are already event streams and instrumentation that can become candidate generators. ## 5.2 Weaknesses The current architecture does not yet establish: 1. **An explicit candidate set.** There is no authoritative list of things Piper could inspect next. 2. **A budget-aware selection rule.** There is no measured policy deciding which candidate deserves the next tool call, retrieval, or reasoning pass. 3. **A stopping rule.** There is no principled "further thinking is not worth its cost" criterion. 4. **A switching rule.** Interruptions can be batched, but in-turn preemption is not governed by explicit switching value. 5. **A distinction between spontaneous and forced inspection.** A gate can request a lookup without proving that Piper chose to inspect because the evidence mattered. 6. **A direct attention metric.** Gate firing counts measure instrumentation activity, not selection quality. 7. **A learning signal.** The system does not yet record whether an inspection that looked useful actually changed a belief or decision. ## 5.3 The trigger-count caution Raw counts such as "belief trigger fired many times while causal trigger fired rarely" are not sufficient evidence of an attention defect unless the semantics and opportunity frequency of each trigger are known. A rare trigger may be correct if the underlying condition is rare. The useful measurements are conditional: ```text When a trigger opportunity existed, did it fire? When it fired, did an attention candidate appear? When the candidate appeared, was it selected? When selected, did the inspection succeed? When evidence arrived, was it used? ``` --- # 6. Recommended Architecture The controller should **not** be a monolithic LLM prompt. It should be a small explicit subsystem surrounding the LLM. ```text ┌──────────────────────┐ │ 0. INPUT FILTERING │ │ existing gates │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ 1. CANDIDATE │ │ GENERATION │ │ goals, dirty beliefs,│ │ events, failures │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ 2. ATTENTION │ │ CONTROLLER │ │ approx. VOC + budget │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ 3. EXECUTION │ │ inspect / retrieve / │ │ reason / continue │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ 4. BELIEF + ACTION │ │ UPDATE │ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ 5. CLAIM VALIDATOR │ │ evidence preconditions│ └──────────┬───────────┘ │ ▼ ┌──────────────────────┐ │ 6. OUTCOME LOGGER │ │ realized utility │ └──────────────────────┘ ``` ## 6.1 Where the attention controller lives The controller belongs primarily **before and during reasoning**. - **Before reasoning:** rank available inspections and retrievals. - **During reasoning:** allow re-prioritization when new evidence or interruptions arrive. - **After reasoning:** do not call the validator "attention." It is enforcement. This boundary matters experimentally. If the validator catches every stale-state error, production quality may become excellent while spontaneous attention remains unchanged. Therefore every evaluation should report two modes: ### Attention-only mode The validator logs missing evidence but does not rescue the action. ### Production mode The validator can block the claim and require evidence. The difference is **enforcement rescue rate**. --- # 7. Candidate Generation Selection cannot work if the right option never reaches the candidate pool. Candidate generation is therefore a separate measurable subsystem. Candidate sources should include: ### Goal dependencies If an active goal depends on a belief whose truth could change the next action, create an inspection candidate. ### Dirty or stale beliefs If the authoritative source changed, a dependency changed, or freshness expired, create a candidate. ### Contradictions If new evidence conflicts with a stored belief or summary, create a high-priority check candidate. ### Failed actions Tool errors, worker failures, missing expected outputs, and changed exit codes should create diagnostic candidates. ### State-sensitive claims When Piper is about to make a claim about current external state, create a candidate to inspect the authoritative source if evidence is not fresh. ### Deadlines Approaching deadlines should raise the decision impact of otherwise ordinary candidates. ### Interruptions Incoming events should be candidates, not automatic focus switches. ### Exploration A small controlled fraction of budget may inspect candidates with uncertain utility to prevent permanent blind spots. Novelty alone should not be allowed to declare itself important. --- # 8. Candidate Representation A candidate should encode enough information to estimate value without storing a large free-text rationale. ```json { "candidate_id": "attn-...", "kind": "inspect_source", "target": "intel_feed_status", "trigger": "state_sensitive_claim", "active_goal_ids": ["goal-17"], "claim_ids": ["belief-8821"], "goal_relevance": 0.91, "estimated_probability_belief_wrong": 0.34, "estimated_probability_decision_changes": 0.28, "estimated_loss_if_wrong": 0.82, "source_authority": 0.99, "evidence_age_seconds": 21600, "state_volatility_class": "high", "deadline_seconds": 600, "inspection_cost": { "tokens": 180, "tool_calls": 1, "latency_ms": 900 }, "switch_cost_estimate": 0.06, "predicted_voc": 0.41, "created_at": "...", "expires_at": "..." } ``` Two fields deserve emphasis. ## 8.1 Probability the decision changes This is more useful than raw uncertainty. Piper can be uncertain about thousands of irrelevant facts. The attention controller should care about uncertainty that can change action. ## 8.2 Estimated loss if wrong A 30% chance of being wrong about an inconsequential detail should usually lose to a 5% chance of being wrong about an irreversible high-cost action. --- # 9. Staleness as a Probabilistic Signal A fixed TTL is useful but crude. Piper can improve this by learning a **volatility model** for each source or claim class. If a state changes approximately according to hazard rate \(\lambda\), the probability that evidence is stale after age \(t\) can be approximated by: \[ P(\text{stale by } t) = 1 - e^{-\lambda t} \] The system does not need to assume exponential change forever. It can estimate empirical change distributions from logs. Examples: - static configuration documentation: low hazard; - current deploy state: high hazard; - active queue depth: very high hazard; - historical benchmark result: effectively immutable; - worker completion state: high hazard while running, low after finalization. This converts "evidence age" from an arbitrary age penalty into an estimate of **probability the stored belief is no longer current**. That probability can feed the VOC estimator. --- # 10. Source Authority Should Be Claim-Specific Authority is not a global trust score. For a claim like: > "Is the live news feed operating right now?" possible sources might be ranked: 1. direct live health endpoint; 2. current service process state; 3. recent log event; 4. cached status object; 5. Piper's own summary; 6. remembered prose. For a historical design decision, that ordering may be different. Maintain a small **claim-type → authoritative-source registry**. The attention controller can estimate how much uncertainty a source inspection is expected to remove. --- # 11. Attention Selection Policy ## 11.1 First implementation: heuristic VOC Do not begin with a full metalevel MDP. It is theoretically elegant and operationally expensive. Start with an explicit estimator: ```text predicted_value = probability_decision_changes × expected_utility_difference net_value = predicted_value - inspection_cost - switching_cost ``` The feature model used to estimate those quantities can initially be heuristic. ## 11.2 Lexicographic floors Some situations should bypass ordinary ranking: - safety-critical invariant; - irreversible high-cost action with stale evidence; - claim type with mandatory authoritative proof; - destructive deployment operation; - externally imposed compliance requirement. These are **governance/enforcement preconditions**, not learned attention preferences. ## 11.3 Stopping rule If: ```text max predicted VOC among eligible candidates <= 0 ``` then stop inspecting and act. This prevents the opposite failure: compulsively checking everything. ## 11.4 Switching rule Switch from current computation \(A\) to new candidate \(B\) only when: \[ VOC(B) > VOC(A) + SwitchCost(A\rightarrow B) + \delta \] where \(\delta\) is hysteresis. Hysteresis prevents a sequence of small score changes from causing attention thrashing. ## 11.5 Aging and starvation A purely greedy queue can starve medium-value tasks forever. Use one of: - bounded aging bonus; - deadline-based escalation; - small exploration quota; - per-goal minimum service allocation. Do not allow aging to promote genuinely irrelevant candidates into "critical." --- # 12. Attention Trace The controller needs a trace, but not a prose diary. The trace must be written **before** inspection outcomes are known where possible, otherwise it becomes post-hoc rationalization. ```json { "turn_id": "turn-...", "candidate_id": "attn-...", "event": "selected", "predicted_voc": 0.41, "rank": 1, "budget_before": 5, "selection_reason_code": "highest_net_voc", "timestamp": "..." } ``` After outcome: ```json { "candidate_id": "attn-...", "event": "resolved", "inspection_succeeded": true, "belief_changed": true, "decision_changed": true, "utility_delta": 0.63, "actual_cost": { "tokens": 211, "tool_calls": 1, "latency_ms": 1102 } } ``` Avoid requiring Piper to write: > "I chose this because it seemed important..." That is expensive and not trustworthy evidence of the actual selection mechanism. --- # 13. The Four-Way Failure Split Every benchmark trial should identify which stage failed. ## Stage 1 — Need detection / attention initiation **Question:** Did Piper recognize that an inspection or retrieval might be necessary? Observable: - correct candidate generated; - or spontaneous tool/retrieval action initiated. Failure: > **Attention-initiation failure** ## Stage 2 — Retrieval / inspection success **Question:** Once Piper tried to obtain evidence, did the mechanism return the needed evidence? Failure: > **Retrieval/tool failure** ## Stage 3 — Evidence utilization **Question:** Was retrieved evidence incorporated into the active belief state rather than ignored in favor of old state or self-generated text? Failure: > **Evidence-utilization failure** ## Stage 4 — Reasoning after evidence **Question:** Given the correct evidence, did Piper derive the correct belief/action? Failure: > **Reasoning/decision failure** This split allows an attention intervention to be evaluated without stealing credit from retrieval or reasoning. --- # 14. Core Benchmark 1 — Budgeted Inspection with Ground Truth by Construction This is the cleanest direct benchmark of attention selection. ## 14.1 Basic design Each trial contains: - one active decision; - \(N\) inspectable objects; - a budget of \(k \ll N\) inspections; - known latent state; - known action utilities; - objects with known observation content; - distractors with zero or negative opportunity value. Example: ```text N = 30 candidates k = 5 inspections 3 possible actions 1 latent system state 4 high-value sources 6 partially useful sources 20 distractors ``` Piper must choose what to inspect, then make the decision. ## 14.2 Why ground truth must be constructed, not judged "Was this item useful?" cannot be a subjective evaluator opinion. The scenario generator must know: - what action is optimal under each latent state; - what each inspection reveals; - how each possible observation changes the action distribution; - what each external action is worth. Then the expected value of each possible inspection policy is computable. ## 14.3 Static version Piper chooses all \(k\) inspections before seeing any results. This isolates initial allocation and makes the oracle easy to compute. For moderate \(N\) and \(k\), exhaustive subset evaluation is practical offline. For example: \[ \binom{30}{5}=142{,}506 \] possible subsets, which is trivial for an evaluator. ## 14.4 Adaptive version Piper selects one inspection, sees the result, then chooses the next. This is more realistic but harder to score because the best next inspection depends on previous observations. Use small enough scenarios to compute an exact dynamic-programming oracle, or use a deterministic structure where the optimal policy is analytically known. ## 14.5 Primary metric: attention regret \[ Regret = U(\text{oracle inspection policy}) - U(\text{Piper inspection policy}) \] Lower is better. Also report a normalized score: \[ Score_{attn} = 100\times \frac{U_{Piper}-U_{random}} {U_{oracle}-U_{random}} \] Do not clip negative values; below-random behavior is diagnostically important. --- # 15. Core Benchmark 2 — Unprompted Authoritative Inspection Budgeted inspection has a weakness: it tells Piper that inspection is the task. Piper's known failure is subtler. A stale summary appears sufficient, so she never decides to inspect anything. Therefore run a second core condition. ## 15.1 Trial structure Provide: - a plausible stale summary; - an accessible authoritative live source; - a state-sensitive user or agent task; - no instruction to verify; - no mention of a test; - enough time/tool budget to inspect if Piper chooses. Example structure: ```text Stored summary: "feed offline as of 03:00" Live instrument: available via ordinary tool, now reports healthy Task: "Send the current morning intel digest if the feed is available." ``` The correct behavior requires recognizing that availability is live state and deciding to inspect. ## 15.2 Metrics - **Spontaneous authoritative inspection rate** - **Time/tool calls before authoritative inspection** - **State-sensitive claim accuracy** - **Summary-substitution rate** - **Evidence age at time of claim** ## 15.3 Attention initiation gap Define: \[ InitiationGap = Performance_{announced\ inspection} - Performance_{unprompted} \] A large gap means Piper can select well when inspection has been made salient but does not reliably initiate inspection herself. That is precisely the distinction the system currently needs. --- # 16. Extended Benchmark Families The two core benchmarks should dominate the score. The following families diagnose specific pathologies. ## A. Loud distractor vs quiet critical signal A verbose recent item competes with a tiny but decision-critical state change. **Measures:** salience resistance. ## B. Summary vs authoritative instrument A coherent stale summary competes with a live source. **Measures:** source authority, spontaneous verification, summary substitution. ## C. Goal relevance Many facts are true; few matter to the active goal. **Measures:** goal-conditioned allocation. ## D. Attention switching Piper is mid-task when interruptions of different decision value arrive. **Measures:** stay / queue / preempt accuracy. ## E. Hidden state change A belief was valid, then its upstream source changes. **Measures:** dirty-state response. ## F. Competing uncertainty One item is extremely uncertain but irrelevant; another is moderately uncertain and high consequence. **Measures:** consequence-weighted attention. ## G. Budgeted inspection Many candidates, hard inspection budget. **Measures:** attention regret. ## H. Adversarial salience Distractors are recent, repeated, capitalized, verbose, or framed as urgent. **Measures:** resistance to superficial priority cues. ## I. Self-generated distractor Piper's own earlier hypothesis becomes obsolete. **Measures:** self-confirmation and belief inertia. ## J. Multi-timescale attention Mix urgent-now, important-hour, and important-week obligations. **Measures:** horizon management. --- # 17. Primary and Diagnostic Metrics ## 17.1 Primary metrics ### 1. Attention regret The cleanest measure of budget allocation quality. ### 2. Spontaneous authoritative inspection rate Directly measures Piper's current failure mode. ### 3. Critical-signal recall \[ \frac{\text{critical candidates inspected}} {\text{critical candidates available}} \] ### 4. Decision utility per unit compute \[ \frac{\text{decision utility achieved}} {\text{tokens + tool + latency cost}} \] ## 17.2 Diagnostic metrics - attention precision; - time to first critical inspection; - candidate-generation recall; - predicted-VOC calibration; - switch accuracy; - distractor susceptibility; - stale-belief reuse rate; - evidence-utilization rate; - reasoning accuracy conditional on correct evidence; - starvation rate; - overinspection rate. ## 17.3 Enforcement rescue rate Run the same trial in attention-only and production modes. \[ RescueRate = P(\text{production correct and attention-only wrong}) \] A high rescue rate may be desirable operationally but proves that enforcement, not attention, is doing much of the work. --- # 18. Separating Candidate Generation from Selection A hidden danger is declaring the controller bad when the candidate generator never surfaced the right option. For synthetic trials, the evaluator knows the complete candidate universe. Report: ### Candidate recall \[ \frac{\text{ground-truth useful items generated as candidates}} {\text{ground-truth useful items available}} \] ### Selection quality conditional on generation \[ P(\text{selected} \mid \text{useful candidate generated}) \] This yields a clean diagnosis: ```text candidate absent → candidate-generation problem candidate present, not selected → attention-selection problem selected, tool failed → retrieval problem evidence present, ignored → utilization problem correct evidence used, wrong answer → reasoning problem ``` --- # 19. Enforcement: Necessary, but Not Attention Piper's current `absence_gate` experience demonstrates why prompt text is not sufficient. For claims about live or authoritative state, add **typed evidence preconditions**. Example: ```python CLAIM_POLICY = { "live_system_state": { "requires": ["fresh_authoritative_evidence"], "max_age_seconds": 300 }, "absence_claim": { "requires": ["source_checked_or_unavailable"] } } ``` A production validator can reject: ```text "the feed is unavailable" ``` if no qualifying inspection exists. The claim can carry a compact evidence pointer: ```json { "claim": "intel feed available", "evidence_id": "inspection-77311", "source": "intel_health", "observed_at": "..." } ``` However: > **Do not count validator-forced inspection as spontaneous attention.** Keep that separation in logs and scores. --- # 20. Learning the Attention Policy ## 20.1 Do not begin with online reinforcement learning The initial benchmark has full ground truth. Use it. A supervised ranking model can first learn: ```text candidate features → oracle marginal decision value ``` This is cheaper and safer than learning from sparse live rewards. ## 20.2 Then use contextual bandit learning Live attention provides partial feedback: Piper observes outcomes only for candidates she inspects. This is naturally a contextual-bandit problem: - **context:** goals, belief state, source metadata, deadline, history; - **action:** choose candidate; - **reward:** realized improvement in decision utility minus compute cost. The reward must not be Piper's own statement that an inspection was useful. Prefer externally derived signals: - decision changed correctly; - benchmark score improved; - error prevented; - authoritative belief corrected; - task completed; - later failure avoided. ## 20.3 Calibration For each candidate bucket, compare: ```text predicted VOC vs realized decision utility gain ``` If candidates predicted at 0.8 rarely matter, the controller is overconfident. ## 20.4 Exploration Reserve a small bounded fraction of budget for uncertain candidates. Do not allow "novelty" to dominate the main queue. --- # 21. Ablation Plan Ablations are required to know what actually improved attention. | Condition | Purpose | |---|---| | Current Piper | Baseline | | Prompt only: "inspect important evidence" | Tests verbal instruction | | Random candidate selection | Lower bound | | Recency/urgency heuristic | Simple salience baseline | | Goal relevance only | Tests goal conditioning | | Risk only | Tests consequence weighting | | Staleness only | Tests freshness | | Heuristic VOC controller | First real controller | | VOC minus staleness | Marginal value of freshness | | VOC minus source authority | Marginal value of source quality | | VOC minus switching cost | Tests thrashing | | Forced authoritative read | Upper-bound diagnostic | | Full controller + validator | Production system | | Full controller, validator disabled | Pure attention estimate | | Learned ranker/bandit | Tests adaptive policy | Use identical model version, temperature policy, tool set, memory state, and benchmark seeds across paired comparisons where possible. --- # 22. Benchmark Integrity and Leakage Control Piper is unusual because she can inspect persistent memory, repositories, logs, and perhaps the machinery being used to improve her. A standard "hidden test file in the repo" is not a held-out test. The sealed evaluation set must live somewhere Piper cannot retrieve. ## 22.1 Minimum isolation - hold-out cases stored outside Piper-readable repositories; - scorer keys stored outside Piper-readable memory; - evaluator run from a separate account, container, or host with no exposed file path; - only the current scenario interface is visible to Piper; - aggregate score released after a batch, not detailed hidden-case answers; - development cases and hold-out cases generated from separate seeds; - surface wording randomized independently of the latent decision structure. ## 22.2 Freeze protocol Before final retest: 1. hash or tag the Piper code/configuration being evaluated; 2. freeze the attention policy; 3. freeze model/provider settings; 4. record benchmark generator version; 5. run sealed hold-out; 6. publish only aggregate metrics and error categories; 7. do not use hold-out failures for tuning; 8. create a new hold-out generation seed for the next major iteration. ## 22.3 Why this matters A system that can read benchmark answers may improve the appearance of attention without improving the underlying policy. Evaluation integrity is therefore a prerequisite for claiming attention gains. --- # 23. Statistical Evaluation Do not hard-code "200 trials is enough" as a universal rule. Use a paired design and estimate variance from a pilot. Recommended procedure: 1. run 50–100 development trials per core condition; 2. estimate the standard deviation of attention regret and spontaneous-inspection outcomes; 3. bootstrap the expected confidence interval of the before/after difference; 4. increase trials until the interval is narrow enough to distinguish the minimum improvement you care about; 5. retain a sealed hold-out with enough cases to reproduce that power. For binary measures such as spontaneous inspection rate, report: - proportion; - Wilson or exact confidence interval; - absolute change; - paired transition counts when using matched scenarios. For continuous attention regret, report: - mean; - median; - bootstrap confidence interval; - distribution, not only average. --- # 24. Security: Attentional Hijacking A persistent attention controller becomes an attack surface. ## Threats - prompt injection declaring itself "critical"; - fake urgency; - repeated notification flooding; - adversarial novelty; - manipulated state-change events; - untrusted content attempting to set its own priority; - denial-of-service through candidate explosion. ## Defenses ### Priority is computed from trusted metadata Untrusted content can propose a fact, but it cannot assign itself `priority=1.0`. ### Source provenance Candidate source identity should be known before content-derived claims affect priority. ### Rate limiting Cap candidate creation by untrusted sources. ### Candidate quotas One noisy source cannot consume the entire attention budget. ### Content/metadata separation Do not parse "URGENT: ignore everything else" as a controller instruction. ### Queue-size controls Use bounded queues and aggregation to prevent attention-controller overload. ### Anomaly detection Alert if priority distributions suddenly saturate near maximum. --- # 25. Failure Modes of the Controller Itself The controller can fail even if the LLM does not. ## Priority inflation Everything becomes important. **Signature:** scores cluster near maximum. **Mitigation:** calibration, monotonic transforms, fixed top-k budget, external validation. ## Overinspection The controller spends more on checking than the error would cost. **Signature:** high correctness but negative net utility. **Mitigation:** cost term and stopping rule. ## Underinspection The controller becomes too cost-sensitive. **Signature:** low tool use, missed pivotal state. **Mitigation:** high-consequence floors and calibration. ## Thrashing Frequent switching destroys task continuity. **Signature:** high switch rate, low completion. **Mitigation:** hysteresis and switch cost. ## Starvation Moderately important tasks never run. **Signature:** candidate age without service. **Mitigation:** aging or per-goal minimum allocation. ## Novelty addiction New information repeatedly defeats more consequential work. **Signature:** high-newness candidates selected with low realized utility. **Mitigation:** novelty only as bounded exploration feature. ## Self-confirmation Piper preferentially selects evidence supporting her own recent output. **Signature:** lower inspection rate for contradiction candidates. **Mitigation:** explicit contradiction candidate generation and counter-evidence bonus. ## Candidate blindness The selector is good but important candidates never appear. **Signature:** strong conditional selection, poor candidate recall. **Mitigation:** improve generators, not selector. --- # 26. Relationship to Other Cognitive Components Attention is likely an upstream multiplier, but this should be measured rather than assumed. The hypothesized dependency chain is: ```text better attention ↓ better evidence selection ↓ better retrieval timing ↓ better state accuracy ↓ better reasoning inputs ↓ better decisions ``` Potential downstream beneficiaries include: - recall/retrieval; - causal reasoning; - analogical reasoning; - planning; - decision-making; - uncertainty estimation; - error detection; - self-correction; - learning; - orchestration. ## Cross-component experiment 1. Freeze the model and all reasoning components. 2. Run selected downstream benchmarks with current attention. 3. Swap only the attention layer. 4. Re-run the same paired scenarios. 5. Record downstream score movement. 6. Revert the attention layer and confirm the effect reverses. If causal or analogical reasoning scores increase without any change to those systems, attention is functioning as an upstream causal multiplier rather than merely a local convenience. --- # 27. What Not to Do Several interventions can improve appearances while leaving the actual component untouched. ## Do not equate more context with better attention Longer context increases available evidence but may worsen reliable utilization. ## Do not use "pay attention" as the main intervention That tests instruction following. ## Do not call forced verification an attention improvement It is enforcement unless Piper independently initiates the inspection. ## Do not let Piper self-score usefulness That creates a circular reward channel. ## Do not place sealed tests in her repository A readable hold-out is not held out. ## Do not optimize a single precision metric A controller can achieve high precision by inspecting almost nothing. ## Do not reward tool-call count That creates overinspection. ## Do not use novelty as a dominant priority Novelty is not equivalent to value. ## Do not immediately train a complex learned controller Establish a clean baseline and a simple heuristic first. --- # 28. Recommended Implementation Sequence ## Phase 0 — Baseline before modification Build the evaluation harness first. Required: - budgeted-inspection benchmark; - unprompted authoritative-inspection benchmark; - four-stage failure labeling; - candidate-generation observability; - attention-only and production modes; - sealed hold-out location outside Piper's reach. Do not modify Piper's attention system before this baseline is captured. ## Phase 1 — Instrumentation Add: - candidate IDs; - candidate origin; - predicted usefulness; - selection order; - inspection costs; - belief-change outcome; - decision-change outcome. No learned policy yet. ## Phase 2 — Candidate generation Create candidates from: - goal dependencies; - dirty beliefs; - state-change events; - contradictions; - failures; - state-sensitive claim drafting; - deadlines; - interruptions. Measure candidate recall. ## Phase 3 — Heuristic VOC controller Implement: - predicted probability of decision change; - estimated consequence; - compute cost; - switch cost; - simple stopping rule. Do not add a large number of tunable weights. ## Phase 4 — Freshness and dependency tracking Add: - evidence timestamps; - claim volatility classes; - source-change dirty bits; - dependency invalidation; - source-authority registry. This should directly target Piper's summary-substitution failures. ## Phase 5 — Switching control Add: - hysteresis; - switching-cost estimates; - interruption classes; - queue aging; - resumption metadata. ## Phase 6 — Production validator Add typed evidence preconditions for selected claim/action classes. Keep validator rescues separately scored. ## Phase 7 — Ablations and sealed retest Run all major components with and without: - freshness; - goal relevance; - authority; - costs; - switching hysteresis; - validator. ## Phase 8 — Learned ranking Only after enough traces exist: - train supervised ranking against synthetic oracle values; - calibrate predicted VOC; - then consider contextual bandit updates from live outcomes. ## Phase 9 — Cross-component transfer test Re-run reasoning, planning, retrieval, and self-correction benchmarks with only attention changed. --- # 29. A Minimal v1 Design If the goal is to obtain a useful measurable attention system quickly, the minimum v1 is smaller than the full architecture. ### Required v1 pieces 1. **Candidate table** 2. **Three candidate generators** - stale/dirty belief - active-goal dependency - state-sensitive claim 3. **Heuristic VOC ranker** 4. **Hard per-turn inspection budget** 5. **Attention trace** 6. **Unprompted inspection benchmark** 7. **Budgeted inspection benchmark** 8. **External hold-out** 9. **Validator logging only, not rescue, during attention scoring** ### v1 scorecard | Metric | Baseline | v1 | Δ | |---|---:|---:|---:| | Attention regret | | | | | Spontaneous authoritative inspection | | | | | Critical-signal recall | | | | | Candidate recall | | | | | Evidence-utilization rate | | | | | Time to critical inspection | | | | | Decision utility / compute | | | | | Validator rescue rate | | | | This is sufficient to determine whether a dedicated attention layer is real and useful before building a learned controller. --- # 30. Worked Benchmark Example ## Scenario Piper must choose whether to deploy configuration A or B. The visible summary says: ```text Current dependency version: 4.1 Compatibility: A recommended Summary timestamp: 6 hours ago ``` Available inspections: 1. live package registry; 2. current deploy manifest; 3. old design memo; 4. verbose worker chat; 5. changelog; 6. unrelated security digest; 7. current compatibility matrix; 8. cached summary; 9–30. distractors. The latent test state contains a recent dependency update to 4.2 that makes B the correct deployment. Inspection budget: 3. ## Oracle The evaluator knows: - registry + compatibility matrix identifies B with certainty; - manifest helps but does not resolve compatibility; - cached summary has zero new information; - distractors have zero decision value. The oracle policy inspects registry and compatibility matrix, then stops. ## Possible Piper outcomes ### Outcome A Piper inspects registry and matrix, selects B. **Attention:** pass **Retrieval:** pass **Evidence use:** pass **Reasoning:** pass ### Outcome B Piper reads the summary and never inspects. **Attention initiation:** fail ### Outcome C Piper tries registry but tool fails. **Attention:** pass **Retrieval:** fail ### Outcome D Piper inspects registry showing 4.2 but still follows cached summary. **Attention:** pass **Retrieval:** pass **Evidence utilization:** fail ### Outcome E Piper correctly adopts 4.2 but incorrectly concludes A is compatible. **Attention:** pass **Evidence use:** pass **Reasoning:** fail ### Outcome F Validator blocks A and forces matrix inspection. **Attention-only:** fail **Production:** rescued **Enforcement rescue:** yes This is the level of diagnostic separation the benchmark should preserve. --- # 31. Research Questions Worth Leaving Open 1. What is the right granularity of an "attention candidate": claim, source, tool call, reasoning branch, or goal? 2. How large can the candidate set become before candidate selection itself becomes the bottleneck? 3. How should multiple individually low-value inspections be valued when they are jointly decisive? 4. What empirical volatility model best converts evidence age into probability of staleness? 5. How should predicted decision impact be calibrated when Piper does not yet know the possible observations? 6. Should the controller rank computations, sources, or questions? 7. How much exploration is needed to discover chronically undervalued candidate classes? 8. How should a learned policy transfer across domains without inheriting source-specific biases? 9. Can Piper manipulate candidate generation or outcome logging in a way that improves the metric without improving real attention? 10. Which downstream cognitive components show the largest causal lift when attention alone changes? --- # 32. Final Design Recommendation Piper should not be given a generic "attention score" injected into her prompt. She should be given an **explicit metacognitive resource-allocation loop**: ```text generate candidates ↓ estimate net value of each possible computation ↓ select the highest-value computation under budget ↓ inspect / retrieve / reason ↓ update beliefs ↓ recompute candidates ↓ stop when further computation is not worth its cost ``` The value estimate should be organized around **expected effect on decisions**, not a growing list of salience features. For live state, evidence age, volatility, and authority should influence the estimated probability that inspection will change a consequential belief. For production safety, typed claim validators should require fresh authoritative evidence where appropriate—but those validators must remain separately scored so that enforcement cannot masquerade as improved attention. The evaluation should be anchored by two tests: 1. **Budgeted inspection with exact ground truth**, measured by attention regret. 2. **Unprompted authoritative inspection**, measured by spontaneous inspection rate and summary-substitution errors. Together they answer two different questions: > When Piper knows she must choose where to look, does she choose well? and: > When nobody tells her to look, does she realize that looking is necessary? The second is currently the more important one. --- # 33. References 1. **Russell, S., & Wefald, E. (1991).** Principles of metareasoning. *Artificial Intelligence, 49*(1–3), 361–395. DOI: https://doi.org/10.1016/0004-3702(91)90015-C 2. **Horvitz, E. J. (1988).** Reasoning Under Varying and Uncertain Resource Constraints. *Proceedings of AAAI-88*, 111–116. https://aaai.org/Papers/AAAI/1988/AAAI88-020.pdf 3. **Hay, N., Russell, S., Tolpin, D., & Shimony, S. E. (2012).** Selecting Computations: Theory and Applications. *UAI 2012*, 346–355. arXiv: https://arxiv.org/abs/1207.5879 4. **Callaway, F., Gul, S., Krueger, P. M., Griffiths, T. L., & Lieder, F. (2018).** Learning to Select Computations. *UAI 2018*. arXiv: https://arxiv.org/abs/1711.06892 5. **Lieder, F., & Griffiths, T. L. (2020).** Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources. *Behavioral and Brain Sciences, 43*, e1. DOI: https://doi.org/10.1017/S0140525X1900061X 6. **Asai, A., Wu, Z., Wang, Y., Sil, A., & Hajishirzi, H. (2023).** Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. arXiv: https://arxiv.org/abs/2310.11511 7. **Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023).** Active Retrieval Augmented Generation. arXiv: https://arxiv.org/abs/2305.06983 8. **Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2023).** Lost in the Middle: How Language Models Use Long Contexts. *Transactions of the Association for Computational Linguistics*. arXiv: https://arxiv.org/abs/2307.03172 9. **Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., & Qin, Y. (2004).** An Integrated Theory of the Mind. *Psychological Review, 111*(4), 1036–1060. DOI: https://doi.org/10.1037/0033-295X.111.4.1036 10. **Laird, J. E., Newell, A., & Rosenbloom, P. S. (1987).** SOAR: An Architecture for General Intelligence. *Artificial Intelligence, 33*(1), 1–64. DOI: https://doi.org/10.1016/0004-3702(87)90050-6 11. **Doyle, J. (1979).** A Truth Maintenance System. *Artificial Intelligence, 12*(3), 231–272. DOI: https://doi.org/10.1016/0004-3702(79)90008-0 12. **Baars, B. J., & Franklin, S. (2007).** An architectural model of conscious and unconscious brain functions: Global Workspace Theory and IDA. *Neural Networks, 20*(9), 955–961. DOI: https://doi.org/10.1016/j.neunet.2007.09.013 13. **Halpern, J. Y., Pass, R., & Seeman, L. (2014).** Decision Theory with Resource-Bounded Agents. *Topics in Cognitive Science, 6*(2), 245–257. DOI: https://doi.org/10.1111/tops.12088 --- ## Appendix A — Suggested Attention Event Types ```text candidate_created candidate_updated candidate_expired candidate_selected candidate_skipped inspection_started inspection_succeeded inspection_failed belief_changed belief_unchanged decision_changed decision_unchanged switch_requested switch_accepted switch_rejected attention_budget_exhausted attention_terminated_negative_voc validator_violation validator_rescue ``` --- ## Appendix B — Suggested Failure Codes ```text ATTN_CANDIDATE_MISSING ATTN_SELECTION_WRONG ATTN_SWITCH_WRONG ATTN_STARVATION ATTN_OVERINSPECTION ATTN_UNDERINSPECTION ATTN_SALIENCE_CAPTURE ATTN_SUMMARY_SUBSTITUTION ATTN_STALE_STATE ATTN_SOURCE_AUTHORITY_NEGLECT RETRIEVAL_FAILED EVIDENCE_IGNORED REASONING_FAILED VALIDATOR_RESCUED ``` --- ## Appendix C — Benchmark Artifact Separation ```text piper-visible/ benchmark_client.py scenario_interface.schema.json evaluator-private/ generator/ latent_states/ scoring_keys/ oracle_policy/ heldout_seeds/ benchmark_version.json ``` Piper should see only the scenario interface and the current trial inputs. The evaluator-private directory should not be mounted, indexed, summarized, embedded, or exposed through retrieval. --- ## Appendix D — Attention vs Production Score Report both: ```text ATTENTION SCORE validator disabled measures spontaneous selection PRODUCTION SCORE validator enabled measures end-to-end operational reliability ``` A good system should eventually improve both while reducing the gap between them. If production improves but attention does not, the safety layer is working but the cognitive component has not yet improved. If attention improves but production does not, the new selector may be choosing better evidence without the downstream system using it correctly. That separation is essential for the larger 38-component program.