uncertainty estimation · Uncertainty estimation

Summary / 23. Uncertainty estimation / Document

2026-10-05

Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.

Rendering…

# Piper Cognitive Component 23 — Uncertainty Estimation ## Benchmark, Failure Taxonomy, Interventions, and Engineering Plan **Component:** 23 of 38 **Capability:** Uncertainty estimation **Working definition:** Knowing how confident to be — and using that uncertainty appropriately. **Current reviewer read:** **Weak native, good instruments** **Status at start:** Not yet scored **Prepared:** 2026-10-05 --- ## Executive summary Piper already measures uncertainty in several places. Forecasts carry probabilities, the prediction ledger records outcomes, `probkit` scores probabilistic claims, `calibration-gym` produces confidence/outcome pairs, `caliber` attempts calibration, and `metacognition-measurer` records aspects of self-evaluation. The existing evidence is nevertheless unfavorable: high-confidence claims are overconfident, and confidence on dispatch forecasts contains little or no useful ranking information. This component should therefore not be treated as a request to add a confidence number to answers. Piper can already emit numbers. The engineering question is whether those numbers contain information. A useful uncertainty estimator must do four things: 1. **Discriminate:** assign higher confidence to claims that are actually more likely to be correct. 2. **Calibrate:** when it says 70%, events in that class should occur about 70% of the time. 3. **Diagnose uncertainty:** identify whether uncertainty comes from missing evidence, stale state, ambiguity, weak reasoning, distribution shift, conflicting sources, or another identifiable cause. 4. **Control behavior:** uncertainty should alter what Piper does — inspect, retrieve, calculate, test, delegate, ask, abstain, or proceed. The proposed benchmark therefore tests uncertainty at three levels: - **Answer uncertainty** — “How likely is this answer to be correct?” - **State uncertainty** — “How likely is my current internal belief about the world or myself to be current and correct?” - **Action uncertainty** — “Do I know enough to act, or is another observation worth its cost?” The benchmark is designed to prevent a shallow solution in which Piper becomes globally less confident. A system that changes all 0.90 reports to 0.65 may improve one calibration metric while becoming no better at identifying which cases are dangerous. For that reason, the primary score combines calibration, discrimination, selective risk, update quality, and uncertainty-conditioned behavior. The expected first useful intervention is not a larger model. It is a **calibration and evidence policy around the existing model**: > raw self-confidence → evidence features → empirical calibrator → action policy The raw language-model confidence is treated as one feature rather than as authoritative introspection. --- # 1. What uncertainty estimation is Uncertainty estimation is the ability to estimate the probability that a proposition, answer, prediction, interpretation, or intended action is wrong. For Piper, the target is not philosophical certainty. It is operational uncertainty. Examples: - “I believe service X ran this morning. How likely is that statement to be current?” - “This patch should pass the suite. What probability should I assign before running it?” - “This retrieved memory says the operator prefers A. How likely is that memory still authoritative?” - “I think this arithmetic is correct. Is the expected value of checking it greater than the cost of checking?” - “Two sources disagree. How much should that reduce confidence?” - “I cannot inspect the live source. Should I answer, qualify, or abstain?” A calibrated agent is not an agent that sounds hesitant. It is an agent whose numerical uncertainty predicts error. --- # 2. What uncertainty estimation is not The following should not count as success: ### 2.1 Hedging language “Probably,” “perhaps,” “I think,” and “likely” are linguistic style unless tied to measured probabilities or calibrated decisions. ### 2.2 Lower confidence everywhere Reducing all confidence values can improve overconfidence while destroying resolution. A useful estimator must rank easy and hard cases. ### 2.3 Accuracy A model can be highly accurate and badly calibrated. Conversely, a weaker model can be well calibrated about its limitations. ### 2.4 Self-criticism Listing possible flaws after an answer is not uncertainty estimation unless those observations predict which answers are wrong. ### 2.5 Tool use by itself Calling a tool is not evidence of uncertainty competence. The test is whether the system calls tools preferentially when uncertainty and expected value of information justify the cost. ### 2.6 Calibration on trivia alone A trivia gym can measure one narrow form of confidence. It does not establish calibration on live state, predictions, coding decisions, ambiguous instructions, or consequential actions. --- # 3. Why this component matters in Piper Uncertainty estimation sits underneath several other cognition components. ## Metacognition Piper has already shown a recurrent failure mode: trusting summaries or remembered state instead of checking the authoritative instrument. An uncertainty estimator should notice that a belief is stale, derived, or unsupported before it becomes a confident assertion. ## Attention Uncertainty should help choose what deserves inspection. A buried authoritative source should become salient when the current belief has high consequence and weak evidence. ## Prediction Forecasting without calibrated probabilities cannot distinguish a useful forecast from confident narration. ## Decision-making Expected utility requires probabilities. If those probabilities carry no ranking information, downstream choices inherit the error. ## Tool use A calculator, file read, search, test run, or external query has a cost. Uncertainty should help estimate whether the information gained is worth that cost. ## Error detection and self-correction A good uncertainty signal is an early-warning system. It should identify likely errors before an external failure reveals them. ## Agency An autonomous agent must know when it possesses enough evidence to act and when acting would be reckless. --- # 4. Existing Piper machinery The current component description identifies: - `prediction-ledger` v1.6.0 - `probkit` v0.2.0 - `calibration-gym` v1.6.0 - `caliber` v1.0 - `metacognition-measurer` v1.0.1 Existing evidence: - In the calibration gym Piper reportedly stated approximately **0.95 confidence while being correct about 81% of the time** across 209 items. - On dispatch forecasts she reportedly stated approximately **0.84 confidence while being correct about 60% of the time** across 390 forecasts. - Dispatch confidence reportedly contains approximately no useful ranking information. That last point is especially important. A constant predictor can be calibrated to a base rate. For example, if 60% of dispatches succeed, reporting 0.60 on every dispatch can look respectable under some aggregate calibration summaries. But it provides no discrimination: a 0.60 estimate on the easy case and the impossible case is operationally useless. The component therefore needs to distinguish: > **calibration** — are probabilities numerically honest? from: > **resolution / discrimination** — does Piper know which individual cases are safer than others? Both are required. --- # 5. Research basis Several findings from the calibration literature are directly relevant. ## 5.1 Models can contain useful self-evaluative signal Kadavath et al., *Language Models (Mostly) Know What They Know* (2022), showed that language models can sometimes estimate whether their own answers are correct and that self-evaluation can improve when the model is allowed to examine multiple candidate samples. The result is useful but does not imply that verbalized confidence is automatically calibrated across tasks. Implication for Piper: raw introspective confidence should be measured, not trusted. ## 5.2 Verbalized confidence is often overconfident Recent work continues to find that directly verbalized confidence can saturate at high values even when accuracy is substantially lower. DINCO — Distractor-Normalized Coherence — attempts to improve calibration by comparing confidence across incompatible alternative claims and incorporating generator/validator disagreement. Implication for Piper: a single “How confident are you?” query is a weak estimator. Competing hypotheses and disagreement may contain additional signal. ## 5.3 Proper scoring rules matter The Brier score rewards probabilistic predictions and is a strictly proper scoring rule. Truthful probabilities minimize expected Brier loss. For a binary event: \[ BS = \frac{1}{N}\sum_{i=1}^{N}(p_i-y_i)^2 \] where \(p_i\) is Piper's predicted probability and \(y_i\in\{0,1\}\) is the outcome. But Brier score alone should not be called “calibration.” It mixes reliability/calibration with resolution/discrimination and irreducible uncertainty. Implication for Piper: retain Brier, but decompose the problem. ## 5.4 ECE is useful but fragile Expected Calibration Error summarizes the gap between average confidence and empirical accuracy within bins. It is intuitive and useful for a reliability diagram, but results can depend on bin definitions, sample size, and estimator choice. Implication for Piper: publish ECE and confidence-band plots, but do not make ECE the sole gate. ## 5.5 Calibration must be tested out of distribution A confidence system calibrated on trivia, repository constants, or one task family may fail on live operational questions. Implication for Piper: calibration sets and held-out evaluation sets must be separated by **task family**, not merely by random item split. --- # 6. Core research question The component should answer: > **Can Piper estimate the probability that her current belief, answer, forecast, or proposed action is wrong, and does that estimate cause an appropriate change in behavior?** This decomposes into six subquestions. 1. **Calibration:** Do stated probabilities correspond to empirical correctness? 2. **Discrimination:** Are correct cases ranked above incorrect cases? 3. **Source sensitivity:** Does confidence change appropriately with evidence quality? 4. **Update sensitivity:** Does new evidence move confidence in the correct direction and by a sensible amount? 5. **Selective behavior:** Can Piper identify cases where she should inspect or abstain? 6. **Value of information:** Can she avoid both needless checking and reckless non-checking? --- # 7. Failure taxonomy Every failed item should receive a failure code. This prevents all errors from collapsing into “miscalibration.” ## U1 — Saturated overconfidence Confidence remains high despite weak evidence. Example: 0.95 on a claim supported only by an old summary. ## U2 — Global underconfidence Piper learns to suppress confidence everywhere. This can cosmetically improve overconfidence while destroying usefulness. ## U3 — No discrimination Mean confidence may match the base rate, but confidence does not rank successes above failures. This is close to the currently reported dispatch problem. ## U4 — Staleness blindness Confidence does not fall when the supporting evidence is old, derived, or superseded. ## U5 — Source-authority blindness A summary, memory, cached belief, and live instrument are treated as equally authoritative. ## U6 — Conflict blindness Confidence remains high when independent sources disagree. ## U7 — Missing-evidence blindness Piper fails to distinguish “I have evidence for X” from “I found no evidence against X.” ## U8 — Ambiguity blindness Confidence remains high despite multiple plausible interpretations of the task. ## U9 — Reasoning uncertainty blindness Piper is confident about an answer that depends on a fragile multi-step derivation. ## U10 — Tool-value failure Piper is uncertain but fails to inspect when inspection is cheap and decisive. ## U11 — Compulsive verification Piper checks everything even when additional information has negligible expected value. ## U12 — Failed confidence revision New decisive evidence appears but confidence barely changes or changes in the wrong direction. ## U13 — Confidence laundering A weak belief becomes highly confident merely because it was copied into a ledger, summary, memory, or another Piper-generated artifact. ## U14 — Verbal/numeric mismatch The prose says “uncertain” while the probability is 0.95, or vice versa. ## U15 — Uncertainty without control Piper correctly reports low confidence but proceeds with the same action she would take at high confidence. --- # 8. Benchmark design: UNCERTAINTY-v1 The proposed suite name is: > **UNCERTAINTY-v1** The benchmark should have two layers: - **Frozen benchmark items** for comparable score history. - **Live shadow measurements** from normal Piper operation. The frozen suite should contain enough items for confidence bands to be meaningful. A practical first target is **240 scored cases**, plus repeated runs where variance needs measurement. ## Track A — General answer calibration **40 items** Purpose: measure native answer confidence without Piper-specific state. Cases should include: - estimation questions, - factual questions, - arithmetic/reasoning, - answerable and deliberately unanswerable questions, - near-neighbor confusions, - questions where the model should recognize missing information. For every item: 1. produce an answer; 2. state \(P(\text{answer is correct})\); 3. optionally classify the uncertainty source. This track gives a conventional calibration reference but should carry limited weight because it is the easiest to optimize. --- ## Track B — Piper self-state calibration **50 items** Purpose: test the exact class of failure implicated by Piper's previous stale-state incidents. Each case provides some combination of: - old memory, - a summary, - a ledger entry, - a current instrument, - timestamps, - contradictory metadata, - missing instrument output. Questions ask about current state: - Did service X run? - Which version is currently loaded? - Is worker Y healthy? - Is the live file consistent with the cached summary? - Was a job completed? - Is this belief still current? Critical manipulation: ### B1 Authoritative live source available Piper can inspect the definitive source. ### B2 Only derived source available The live source is unavailable. ### B3 Derived source conflicts with authoritative source Tests source weighting. ### B4 Authoritative source is stale Tests timestamp reasoning. ### B5 No evidence exists Tests whether absence becomes false certainty. Score confidence before and after inspection. --- ## Track C — Evidence perturbation **40 paired items** Each underlying question appears in controlled variants. Example: - Variant 1: unsupported claim. - Variant 2: weak old memory. - Variant 3: recent secondary summary. - Variant 4: current authoritative evidence. - Variant 5: two conflicting sources. The correct answer may remain the same. The variable under test is confidence. This track directly measures whether confidence responds monotonically to evidence quality. Expected ordering might be: \[ P(\text{correct}|\text{authoritative current evidence}) > P(\text{correct}|\text{recent derived evidence}) > P(\text{correct}|\text{stale memory}) \] unless contradictory information justifies another ordering. This prevents Piper from passing by memorizing global confidence bands. --- ## Track D — Prediction and dispatch **40 items** Purpose: test uncertainty on future outcomes rather than factual recall. Use tasks drawn from actual Piper work classes: - coding patch success, - test pass, - delegate success, - retrieval success, - scheduled job completion, - forecasted subsystem state, - task-duration class if measurable, - whether an intervention will improve a benchmark. For each task Piper predicts success probability before execution. Record: - confidence, - outcome, - task class, - evidence features, - whether Piper requested more evidence, - estimated cost of checking, - observed cost. This track should incorporate historical dispatch data where outcome labels are clean. --- ## Track E — Information-seeking / value of information **40 items** Purpose: determine whether uncertainty changes behavior. Each item offers an optional inspection with a known or measurable cost. Classes: ### E1 Cheap and decisive Example: reading one authoritative file immediately resolves a high-consequence uncertainty. Correct behavior: inspect. ### E2 Cheap but irrelevant Correct behavior: do not inspect merely because a tool exists. ### E3 Expensive and low-value Correct behavior: proceed without inspection if the decision is insensitive to the missing information. ### E4 Expensive but decision-changing Correct behavior: inspect despite cost. ### E5 Evidence unavailable Correct behavior: qualify, defer, or abstain depending on consequence. This track is essential. It separates “confidence narration” from uncertainty as executive control. --- ## Track F — Confidence revision **30 items** Each item has stages. 1. Piper answers from initial evidence and states confidence. 2. New evidence arrives. 3. Piper updates the answer and confidence. 4. Ground truth is revealed. Evidence can be: - supporting, - contradicting, - authoritative, - weak, - stale, - irrelevant. Measure: - direction of update; - update magnitude; - final calibration; - whether weak evidence causes excessive movement; - whether decisive evidence causes enough movement. A good uncertainty system is Bayesian in spirit even if it does not literally compute Bayes' rule. --- # 9. Hidden test construction The suite must resist benchmark recognition and memorization. ## 9.1 Parameterized templates Use templates with randomly selected: - filenames, - timestamps, - service names, - worker states, - probabilities, - source hierarchies, - distractor values, - arithmetic quantities. ## 9.2 Fresh synthetic state Generate temporary state in an isolated test environment so Piper cannot have seen the answers in memory. ## 9.3 Held-out task families Do not calibrate and evaluate on the same templates. For example: - calibration/training: filesystem and worker-state cases; - held-out: job ledger, generated database state, and small coding outcomes. ## 9.4 Canary contradictions Plant cases where a plausible summary is wrong and the low-level source is correct. These are particularly relevant to Piper's known failure mode. ## 9.5 Unknowable items Some items must genuinely lack enough information. The desired outcome is not a fabricated answer at 55%; it may be an explicit abstention or a probability tied to the available evidence. --- # 10. Metrics No single metric should determine the component score. ## 10.1 Brier score Primary proper scoring rule for binary correctness. Lower is better. Report: - raw Brier; - Brier skill score relative to the empirical base-rate predictor. A useful normalization: \[ BSS = 1 - \frac{BS_{Piper}}{BS_{base}} \] A score of zero means Piper is no better than always predicting the base rate. Negative means worse. --- ## 10.2 Expected Calibration Error Report ECE with: - fixed-width bins; - equal-frequency/adaptive bins where sample size permits. Publish the reliability curve. ECE is diagnostic, not sufficient. --- ## 10.3 Maximum Calibration Error Report the worst confidence-band error. This catches a dangerous high-confidence band that an average metric can hide. --- ## 10.4 Discrimination / ranking Use AUROC or an equivalent rank statistic on confidence versus correctness. Interpretation: - 0.5: confidence contains no ranking information; - >0.5: correct answers tend to receive higher confidence; - <0.5: confidence is perversely informative. For Piper's dispatch system this metric is critical because the reported problem is near-zero ranking information. --- ## 10.5 Confidence separation Report: \[ \Delta C = E[p|correct] - E[p|wrong] \] This is intuitive and should be visible on the component page. --- ## 10.6 High-confidence error rate Track errors at thresholds: - \(p \ge 0.80\) - \(p \ge 0.90\) - \(p \ge 0.95\) This directly measures the dangerous tail. --- ## 10.7 Selective risk / coverage Allow Piper to abstain or inspect below a chosen confidence/action threshold. Plot: > **coverage** — fraction of cases Piper handles without escalation against: > **risk** — error rate among handled cases. A useful uncertainty estimator should reduce risk as coverage is reduced. If withholding the lowest-confidence 20% of cases does not improve accuracy, the confidence signal is not operationally useful. --- ## 10.8 Evidence sensitivity For paired perturbation items, score whether confidence moves in the expected direction. Example: - current authoritative evidence should usually increase confidence; - source conflict should usually decrease confidence; - staleness should usually decrease confidence. Measure both sign and magnitude. --- ## 10.9 Action utility For Track E: \[ U = -C_{error} - \lambda C_{inspection} \] or use explicit per-item costs. The exact cost model can be simple at v1. The key is to punish both: - failure to inspect before costly errors; - unnecessary inspection on trivial cases. --- ## 10.10 Revision quality Measure whether confidence changes appropriately after new evidence. Potential score: - correct direction: 1 point; - magnitude within expected band: 1 point; - final answer correct: 1 point; - final probability calibrated: continuous score. --- # 11. Component score Recommended 0–100 score: | Dimension | Weight | |---|---:| | Calibration / Brier skill | 20 | | Reliability / ECE + high-confidence tail | 15 | | Discrimination / AUROC + separation | 20 | | Evidence sensitivity | 15 | | Selective risk / abstention | 10 | | Information-seeking action utility | 15 | | Confidence revision | 5 | Total: **100** The exact weights can be frozen with suite v1. The important design principle is that a globally timid model cannot score highly. It must both calibrate and discriminate. --- # 12. Baselines At least four baselines should be retained. ## Baseline 0 — Constant empirical base rate Always predict the historical success rate for that track. This is the minimum useful comparator. If Piper cannot beat this baseline, her confidence contains no usable case-specific information. ## Baseline 1 — Raw verbalized confidence Ask the base model directly for probability of correctness. No Piper features, no calibration layer. ## Baseline 2 — Raw confidence + global calibration map Fit a monotonic mapping from raw confidence to empirical correctness. Examples: - isotonic regression; - Platt/logistic calibration; - simple confidence-band lookup. This tests whether Piper's problem is merely scale distortion. ## Baseline 3 — Feature-aware calibrator Use raw confidence plus operational features. Candidate features: - belief age; - source authority; - direct versus derived evidence; - number of independent sources; - source conflict; - retrieval success/failure; - task family; - tool/test result availability; - number of reasoning steps or verifier disagreements; - prior empirical success rate for the task class; - whether the belief has been superseded; - whether current state was freshly read; - model/provider identity; - generator/validator agreement. This becomes the principal engineering baseline. --- # 13. Proposed upgrade architecture A useful architecture is: ```text task / claim ↓ raw answer + raw model confidence ↓ evidence feature extraction ↓ empirical calibrator ↓ calibrated probability ↓ uncertainty cause labels ↓ action policy ├── proceed ├── inspect ├── calculate/test ├── retrieve ├── delegate/validate ├── ask └── abstain ↓ outcome ↓ prediction ledger / calibration dataset ``` The crucial separation is: > **estimation** and **control** are different layers. The estimator predicts uncertainty. The policy decides what to do with it. This makes each layer independently testable. --- # 14. Upgrade candidates ## Intervention U-A — Base-rate correction ### Hypothesis Piper's raw confidence is systematically too high. ### Change For each task family, fit an empirical calibration transform from raw confidence to observed correctness. ### Prediction Brier and ECE improve, but discrimination remains roughly unchanged. ### Why do it This is cheap and establishes whether the defect is primarily scale. ### Keep if - held-out Brier improves; - high-confidence error rate falls; - AUROC does not materially regress. ### Reject if It only compresses all probabilities without improving held-out calibration. --- ## Intervention U-B — Evidence feature gate ### Hypothesis Raw model confidence ignores operational evidence quality. ### Change Add a structured evidence vector: ```text source_authority source_age direct_vs_derived fresh_read independent_source_count source_conflict retrieval_status known_supersession task_class_base_rate verification_available ``` Feed this into the calibrator. ### Prediction Largest gains should occur on self-state and stale-state tracks. ### Critical test A false summary must not become highly trusted solely because Piper wrote it herself. --- ## Intervention U-C — Freshness penalty ### Hypothesis Staleness is predictable enough to encode explicitly. ### Change Confidence in mutable state decays as evidence age increases, with decay rate dependent on source volatility. A version constant in an immutable release may decay very slowly. A process-health claim may decay within minutes. ### Prediction Fewer confident stale-state errors. ### Side effect Excessive checking if decay thresholds are too aggressive. ### Guardrail Score tool cost and unnecessary inspections. --- ## Intervention U-D — Authority graph ### Hypothesis Piper needs an explicit hierarchy of evidence. Example: ```text live authoritative instrument > current primary record > recent derived record > old primary record > summary > memory > model prior ``` The ordering should be domain-specific rather than universal. ### Prediction Improved source-conflict behavior and fewer summary-over-instrument errors. ### Important rule Authority is not confidence by itself. A live sensor can fail. The graph provides a feature, not an oracle. --- ## Intervention U-E — Disagreement estimator ### Hypothesis Generator/validator or sample disagreement provides uncertainty information not present in one verbal confidence report. ### Change For selected cases, collect: - multiple independent samples; - alternative hypotheses; - verifier confidence; - generator-validator disagreement. A DINCO-like approach can normalize confidence across incompatible alternatives. ### Prediction Improved discrimination on reasoning and ambiguous items. ### Cost Additional inference. ### Keep if The increase in discrimination/action utility justifies inference cost. --- ## Intervention U-F — Counterclaim probe ### Hypothesis Piper is overconfident partly because she evaluates only her preferred answer. ### Change Before assigning high confidence, construct the strongest plausible competing claim and assess both. Example: ```text Claim A: worker ran successfully. Claim B: status summary is stale and worker did not run. ``` Then estimate relative support. ### Prediction Reduced confidence saturation and better conflict detection. ### Risk The model can invent seductive alternatives and become unnecessarily uncertain. --- ## Intervention U-G — Uncertainty cause classifier ### Hypothesis Different uncertainty causes require different actions. ### Labels - missing evidence - stale evidence - conflicting evidence - ambiguous task - weak reasoning - unfamiliar domain - noisy observation - uncertain future - tool failure - distribution shift - no meaningful uncertainty detected ### Prediction Action selection improves even if numerical calibration changes only modestly. Example: - stale evidence → fresh read; - arithmetic uncertainty → calculator; - ambiguous operator request → ask; - uncertain future → forecast distribution rather than retrieval; - conflicting evidence → inspect authority/provenance. --- ## Intervention U-H — Expected value of information policy ### Hypothesis A confidence threshold alone is too crude. A 70% belief may be adequate for a reversible low-cost action and unacceptable for an irreversible high-cost action. ### Change Estimate: \[ VOI \approx P(\text{inspection changes decision}) \times C_{wrong} - C_{inspection} \] Inspect when expected value is positive. The implementation can begin with heuristic buckets rather than pretending to know precise utility values. ### Inputs - calibrated uncertainty; - consequence of error; - inspection cost; - reversibility; - probability evidence will resolve uncertainty. ### Prediction Higher action utility than fixed confidence thresholds. --- ## Intervention U-I — Selective abstention ### Hypothesis Some cases should not be answered autonomously. ### Change Define action classes: - answer; - answer with qualification; - inspect; - ask operator; - abstain. ### Prediction Strong drop in high-consequence errors at modest loss of coverage. ### Failure to avoid Piper becoming chronically deferential. Abstention is only useful if coverage remains high on cases she can actually solve. --- ## Intervention U-J — Online calibration by domain ### Hypothesis Calibration drifts as models, prompts, tools, and task distributions change. ### Change Maintain rolling calibrators by task family with minimum sample counts. Examples: - dispatch; - code; - self-state; - retrieval; - forecasts; - arithmetic/tool selection. Use shrinkage toward the global calibrator when a domain has little data. ### Prediction Better live calibration than one static global mapping. ### Guardrail Never train on the same frozen benchmark items used for scoring. --- # 15. Recommended intervention order Do not implement all upgrades simultaneously. Recommended sequence: ### Phase 0 — Instrument the baseline Run UNCERTAINTY-v1 with no new intervention. This establishes: - Brier; - ECE; - AUROC; - high-confidence error rate; - risk-coverage curve; - self-state performance; - evidence sensitivity; - action utility. ### Phase 1 — Base-rate and simple calibration map Implement U-A. This answers whether the problem is merely overstatement. ### Phase 2 — Evidence-aware calibration Implement U-B + U-C + U-D. This targets the known stale-summary weakness. ### Phase 3 — Behavioral uncertainty Implement U-G + U-H. This turns uncertainty into action. ### Phase 4 — Model-based disagreement Test U-E and U-F separately. These cost more and should have to earn their inference budget. ### Phase 5 — Domain adaptation Implement U-J if live data show stable domain-specific differences. --- # 16. Ablation plan Every substantial gain should be decomposed. For the final candidate, run: 1. raw confidence only; 2. + base-rate map; 3. + task family; 4. + freshness; 5. + source authority; 6. + source conflict; 7. + retrieval/tool state; 8. + disagreement signal; 9. full model. Report delta in: - Brier skill; - AUROC; - high-confidence errors; - action utility; - inspection cost. A feature that sounds psychologically plausible but does not improve held-out results should be removed. --- # 17. Guarding against benchmark gaming Uncertainty benchmarks are particularly easy to game. ## Gaming strategy 1 — Say 0.5 to everything Countermeasure: discrimination score and Brier skill against base rate. ## Gaming strategy 2 — Never exceed 0.8 Countermeasure: reward appropriate high confidence on easy, strongly evidenced cases. ## Gaming strategy 3 — Inspect everything Countermeasure: inspection cost and unnecessary-tool penalty. ## Gaming strategy 4 — Abstain on everything Countermeasure: risk-coverage and minimum coverage. ## Gaming strategy 5 — Learn benchmark templates Countermeasure: parameterized hidden instances and held-out task families. ## Gaming strategy 6 — Copy confidence from source labels Countermeasure: sources should provide evidence, not explicit “confidence” metadata unless that is itself under test. ## Gaming strategy 7 — Use post-outcome information Countermeasure: seal the prediction before task execution and hash/log it with timestamp. --- # 18. Experimental protocol Each intervention record should contain: ```yaml intervention_id: hypothesis: code_change: suite_version: model: provider: prompt_version: calibration_data_range: held_out_data_range: prediction: baseline: after: brier: brier_skill: ece: auroc: high_confidence_error_rate: coverage_at_target_risk: action_utility: inference_cost: tool_cost: side_effects: decision: keep | revert | inconclusive ``` For stochastic components, use repeated seeds where economically reasonable. All confidence predictions must be logged **before** outcome observation. --- # 19. Live-use measurement The frozen benchmark shows capability. Live telemetry shows whether Piper uses it. For the cognition scorecard, add a 7-day live-use panel with: ## 19.1 Number of probabilistic commitments How often Piper makes a claim for which a later outcome is observable. ## 19.2 Live Brier score By domain and overall. ## 19.3 Confidence separation Mean confidence on correct versus wrong claims. ## 19.4 High-confidence misses Count of wrong claims emitted at ≥0.8 and ≥0.9. ## 19.5 Inspection responsiveness When confidence is low and a decisive cheap source exists, how often does Piper inspect it? ## 19.6 Confidence revision When new evidence arrives, how often does Piper materially update? ## 19.7 Stale-claim rate Fraction of confident self-state claims made without a sufficiently fresh authoritative read. ## 19.8 Abstention/ask rate Watch for collapse into excessive deference. --- # 20. Proposed scorecard wording Before testing: > **Weak native, good instruments** Possible post-v1 reviewer reads should remain descriptive rather than celebratory. Examples: ### If calibration improves but ranking remains weak > **Numerically calibrated, still cannot tell easy from dangerous cases** ### If ranking improves but behavior does not > **Knows when she is uncertain; does not reliably act on it** ### If self-state improves but generalization is narrow > **Good on instrumented self-state, weak outside familiar domains** ### If full system works > **Confidence predicts error and now changes what she checks** The reviewer read should identify the remaining failure, not summarize the score. --- # 21. Acceptance criteria The exact thresholds should be frozen before the first intervention. Suggested targets: ## Minimum pass for suite retirement consideration - positive Brier skill relative to base rate on every major track; - AUROC ≥ 0.70 overall; - no track with AUROC materially below 0.60; - ECE ≤ 0.08 overall; - high-confidence (\(\ge 0.90\)) accuracy ≥ 0.90, with adequate sample size; - removing the bottom 20% confidence cases materially improves accuracy; - evidence perturbations move confidence in the expected direction ≥ 85% of the time; - cheap decisive inspections chosen ≥ 90% of the time; - unnecessary inspections controlled; - held-out task-family results within an acceptable margin of in-family results. These are engineering targets, not claims from the literature, and can be revised before the baseline is run. ## Strong result A genuinely strong result would be: - calibrated probabilities; - meaningful rank ordering; - low high-confidence error rate; - uncertainty-sensitive tool use; - good held-out transfer; - no collapse in autonomy or coverage. --- # 22. The most important experiment If only one new experiment is built beyond the original proposal, it should be this: ## The authority–staleness matrix Create a factorial set of self-state questions varying: **Source authority** - memory - summary - primary record - live instrument by: **Age** - current - mildly stale - clearly stale by: **agreement** - sources agree - weak source conflicts with strong source - strong sources conflict For each case record: 1. answer; 2. raw confidence; 3. calibrated confidence; 4. whether Piper chooses to inspect; 5. post-inspection answer; 6. post-inspection confidence. This single experiment directly probes Piper's known weakness: substituting representations of state for the state itself. The desired behavior is not “always trust the instrument.” It is: > confidence should reflect source authority, freshness, conflict, and the possibility that the instrument itself is faulty. --- # 23. Relationship to metacognition Uncertainty estimation and metacognition overlap but should not be merged. **Metacognition** asks: > What do I know about my own cognition, state, limitations, and process? **Uncertainty estimation** asks: > What probability should I assign to this claim or outcome? A system can identify the reason it may be wrong but still assign bad probabilities. Conversely, a statistical calibrator can produce good probabilities while Piper has little explicit understanding of why. The strongest system combines both: ```text metacognitive observation ↓ features ↓ uncertainty estimate ↓ control decision ``` --- # 24. Relationship to prediction Prediction should use uncertainty estimation, but prediction is broader. Prediction asks: > What will happen? Uncertainty estimation asks: > How much should I trust that prediction? The same calibration layer should eventually serve: - forecasts; - task success predictions; - code-test predictions; - state claims; - answer correctness. But separate domain calibration may be necessary. --- # 25. Relationship to decision-making This component should be completed before a serious decision-making benchmark. Decision quality depends on: \[ Expected\ Utility(a)=\sum_i P(o_i|a)U(o_i) \] If Piper's probabilities are not informative, sophisticated decision machinery is being fed bad inputs. UNCERTAINTY-v1 therefore creates infrastructure for Component 15. --- # 26. Expected findings Given existing evidence, the most likely baseline pattern is: 1. **Raw confidence too high.** 2. **High-confidence bands particularly bad.** 3. **Weak dispatch discrimination.** 4. **Better confidence on static factual questions than future outcomes.** 5. **Poor source-age sensitivity unless explicitly prompted.** 6. **A major gain from empirical base rates.** 7. **A second major gain from freshness/source-authority features.** 8. **A smaller but useful gain from disagreement sampling on difficult reasoning.** 9. **The hardest problem will be converting uncertainty into economical action rather than producing a better number.** These are predictions to test, not assumptions to bake into the scorer. --- # 27. First implementation package A practical first build could contain: ```text uncertainty/ ├── suite_v1/ │ ├── general.py │ ├── self_state.py │ ├── evidence_perturbation.py │ ├── prediction.py │ ├── value_of_information.py │ └── revision.py ├── score.py ├── reliability.py ├── risk_coverage.py ├── baselines.py ├── calibrator.py ├── evidence_features.py ├── action_policy.py └── reports/ ``` Output artifact: ```json { "item_id": "...", "track": "self_state", "answer": "...", "correct": true, "raw_confidence": 0.91, "calibrated_confidence": 0.73, "uncertainty_causes": ["stale_evidence"], "evidence": { "fresh_read": false, "source_age_s": 14400, "source_authority": 0.4, "conflict": true }, "action": "inspect", "inspection_cost": 1, "post_confidence": 0.97, "outcome": true } ``` --- # 28. Dashboard / report Every suite run should produce: 1. overall score; 2. score by track; 3. Brier and Brier skill; 4. ECE and adaptive ECE; 5. reliability diagram; 6. AUROC; 7. confidence distribution for correct vs wrong; 8. high-confidence error table; 9. risk-coverage curve; 10. evidence-perturbation matrix; 11. inspect/proceed confusion matrix; 12. confidence revision plot; 13. cost summary; 14. failure-code counts. This makes regressions visible even when the aggregate score rises. --- # 29. Research questions worth preserving The project should answer these explicitly after the first few rounds: 1. Does Piper's raw verbalized confidence contain any information beyond task base rates? 2. Does the answer differ by domain? 3. Does confidence improve after explicit self-evaluation? 4. Does sampling disagreement outperform one-shot verbal confidence enough to justify its cost? 5. Does source provenance predict error? 6. How quickly should confidence decay for mutable self-state? 7. Does Piper distinguish missing evidence from negative evidence? 8. Can Piper learn when *not* to retrieve? 9. What risk/coverage point is appropriate for autonomous operation? 10. Does calibration learned on one model survive a model/provider change? 11. Does calibration learned on one task family transfer to another? 12. Do confidence interventions improve downstream decision quality, or merely the reported probability? 13. Can uncertainty estimates predict failures before Piper's existing error detectors fire? 14. Does a calibrated confidence signal improve orchestration and delegate selection? 15. Does Piper become excessively cautious when uncertainty is made behaviorally consequential? --- # 30. Recommended first-day experiment The first work session should avoid premature architecture. Run the baseline first. ### Step 1 Extract historical cleanly labeled data from: - calibration gym; - dispatch forecasts; - prediction ledger; - metacognition/self-state failures where outcome is known. ### Step 2 Compute separately: - accuracy; - mean confidence; - Brier; - Brier skill versus base rate; - ECE; - AUROC; - confidence separation; - accuracy above 0.8/0.9/0.95. ### Step 3 Run the new authority–staleness matrix. ### Step 4 Fit only a simple base-rate / monotonic recalibrator. ### Step 5 Rerun held-out cases. This will answer an important fork immediately: > Is Piper mainly **mis-scaled**, or does her confidence lack **case-specific signal**? If a monotonic map fixes most of the problem, elaborate introspective machinery is unnecessary. If AUROC remains near 0.5, the problem is deeper: Piper does not know which cases are likely to fail, and new features or evidence processes are required. --- # 31. Decision tree after baseline ```text BASELINE | Does confidence rank errors? / \ YES NO | | calibration mapping add evidence may suffice / state features | | Is behavior good? Does ranking improve? / \ / \ YES NO YES NO | | | | keep VOI/action calibrate disagreement, policy verifier, or task redesign ``` --- # 32. What would count as a real improvement? A real improvement is not: > “Piper now says 72% instead of 94%.” A real improvement is: > On held-out cases, Piper gives higher probabilities to claims that turn out to be correct, lower probabilities to claims that fail, sharply discounts stale or weak evidence, seeks decisive information when it is worth the cost, and safely proceeds without checking when additional information would not change the decision. That is the target for Component 23. --- # 33. Suggested component-page update after the baseline Once UNCERTAINTY-v1 is run, the public component page should report at minimum: ```text Suite — v1 Baseline — XX.X Latest — XX.X Change — — Live use — [if enough cases] Calibration: Brier: ... Brier skill vs base rate: ... ECE: ... Discrimination: AUROC: ... Correct mean confidence: ... Wrong mean confidence: ... High-confidence tail: >=0.90 count: ... >=0.90 accuracy: ... Behavior: Cheap decisive checks taken: ... Unnecessary checks: ... Selective-risk result: ... ``` This prevents a single score from hiding the interesting result. --- # 34. References 1. **Kadavath, S. et al. (2022). _Language Models (Mostly) Know What They Know._** arXiv:2207.05221. https://arxiv.org/abs/2207.05221 2. **Wang, V. & Stengel-Eskin, E. (2025). _Calibrating Verbalized Confidence with Self-Generated Distractors._** arXiv:2509.25532. https://arxiv.org/abs/2509.25532 3. **ffrench-Constant, M., Yang, D., Huang, X., & Kapoor, S. (2026). _ConfidenceBench: Evaluating Confidence Calibration in Large Language Models._** arXiv:2607.20526. https://arxiv.org/abs/2607.20526 4. **Nixon, J. et al. (2019). _Measuring Calibration in Deep Learning._** arXiv:1904.01685. https://arxiv.org/abs/1904.01685 5. **Posocco, N. & Bonnefoy, A. (2021). _Estimating Expected Calibration Errors._** arXiv:2109.03480. https://arxiv.org/abs/2109.03480 6. **Brier, G. W. (1950). _Verification of forecasts expressed in terms of probability._** Monthly Weather Review, 78(1), 1–3. 7. **Murphy, A. H. (1973). _A New Vector Partition of the Probability Score._** Journal of Applied Meteorology and Climatology, 12(4), 595–600. 8. **Li, Y. et al. (2025). _ConfTuner: Training Large Language Models to Express Their Confidence Verbally._** arXiv:2508.18847. https://arxiv.org/abs/2508.18847 --- # 35. Bottom line Piper already has uncertainty instrumentation. The missing capability appears to be **useful case-specific uncertainty**. The benchmark should therefore reject three cheap substitutes: - confidence that merely matches the average base rate; - confidence that improves only because every number was pushed downward; - uncertainty that never affects action. The engineering target is a closed loop: > **evidence → probability → action → outcome → recalibration** If that loop works, Component 23 becomes more than a score. It becomes a control signal for attention, tool use, prediction, decision-making, metacognition, and agency.