Summary / 14. Prediction / Document
2026-10-10
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper · Cognition 14/38 — Prediction ## Research, Benchmark, Upgrade & Experimental Protocol **Working paper v1.0 | 10 October 2026 | Status: proposed / not yet scored** > **Central question:** Can Piper assign useful, calibrated probabilities to future external events and to the outcomes of her own actions, *before* those outcomes become known—and improve those forecasts without contaminating the evaluation? ## 1. Executive decision **Recommendation:** Keep prediction as two explicitly separate abilities: (A) world forecasting and (B) self forecasting. Score native Piper probabilities, fitted/calibrated probabilities, and simple reference-class baselines as **distinct competitors** on temporally held-out resolutions. Do not treat a better calibrated wrapper as proof that Piper's underlying anticipation improved. Keep a frozen, independent evaluator; log predictions at decision time; and evaluate the value of forecasts for actual resource allocation, not merely the plausibility of explanations. **First actionable milestone:** audit the existing 101 world and 1,387 resolved self forecasts, reconstruct a leakage-safe chronological baseline, publish score distributions with uncertainty intervals, and launch the trip-the-system exercises in §8. The cited counts and metrics are **user-supplied September 30 snapshot values**, not newly verified live readings. ## 2. Starting facts and what they do not establish | Observed / claimed as of 30 September 2026 | Interpretation / caution | |---|---| | `prediction-ledger` v1.6.0; `intel-gym` v1.0.1; `drives` v1.14.0; `probkit` v0.2.0; `delegate-bridge` v1.94.0 | Instrumentation exists; no inference about present runtime state without fresh inspection. | | 101 world forecasts and 1,387 resolved self forecasts | Potentially useful, but check definitions, provenance, duplicates, and resolution eligibility. | | Native self-forecast AUROC 0.503 | Near-chance ordering on the reported backtest; report a confidence interval and sample composition. | | Native Brier 0.178 versus base-rate Brier 0.151 | Native probabilities underperform this base-rate benchmark on this reported sample; exact comparability must be verified. | | Fitted probabilities reported in 25 rows | Reported beside native probabilities, in shadow mode—not a replacement of the native signal. | | An earlier description dated an evaluation to “this morning” | Reviewer says the backtest ran evening 29 September; preserve actual event timestamps. | **Interpretive guardrail:** AUROC measures ranking, not calibration. Brier measures probability quality and is affected by outcome prevalence and case mix. A constant base-rate predictor has AUROC 0.5 by convention when both classes are present but can have a strong Brier score. Disagreement is not contradictory. If the self-outcome rate is strongly skewed, high apparent accuracy alone is particularly uninformative. ## 3. Define the construct before optimizing it Prediction is not memory retrieval, probability arithmetic, confidence language, or the ability to write a plausible causal narrative. It is an ex ante mapping from **information available at time t** to a probability distribution over a **pre-specified outcome at time t+h**. An effective system can express its uncertainty, use appropriate reference classes, revise responsibly as evidence changes, and choose actions that improve expected utility. **Boundaries with adjacent components:** #23 Uncertainty estimation expresses uncertainty about beliefs; #14 Prediction tests future event distributions against outcomes. #15 Decision-making uses probabilities and utilities to select actions. #4 Retrieval supplies evidence; #1 Perception observes evidence; #24 Error detection notices forecast failures; #13 Planning generates action sequences. Improvement in any neighbor does not automatically count as predictive improvement. ### 3.1 World track (A) Binary, dated, externally resolved questions about public events or objectively timestamped system-external occurrences. Each question has a clear `YES` condition, resolution deadline, trusted source, ambiguity policy, and forecast cut-off. Include varied domains, time horizons, and difficulty. Distinguish observational predictions from intervention-sensitive ones. ### 3.2 Self track (B) Predict objective outcomes of Piper dispatches before they launch. Define success with an independent acceptance contract; include completion time, cost, need for rework, test pass, and delivery reliability as separate targets. Avoid success labels supplied solely by the same component that produced the forecast. Track interrupted/cancelled jobs and unresolved cases rather than silently excluding them. ### 3.3 Three layers to keep separate 1. **Raw native forecast:** the probability Piper assigned without post-hoc correction. 2. **Calibrated/fitted forecast:** trained on *past* examples only; may improve calibration without improving native ranking. 3. **Decision policy:** a workflow informed by predictions; may be evaluated by cost-adjusted outcomes and regret. ## 4. Research background and transferable findings **Proper scoring:** The quadratic/Brier score is strictly proper for a binary event when used correctly: `BS = mean((p - y)^2)`, with `y ∈ {0,1}`. Lower is better. Log loss `-mean(y log(p)+(1-y) log(1-p))` punishes unwarranted extreme confidence more heavily. Reliability plots and calibration diagnostics expose what an aggregate score hides. Gneiting & Raftery (2007) provide foundations for proper scoring rules; Gneiting, Balabdaoui & Raftery (2007) distinguish calibration and sharpness. **Live future-event design:** ForecastBench evaluates questions created and answered before their resolution, with moving question sets and comparison forecasters. Adopt its pre-resolution commitment and dynamic sampling principles rather than using retrospectively selected historical trivia. This does not imply Piper's data are comparable directly with that leaderboard. **Reference classes before stories:** For self dispatches, start with pre-task data such as complexity, repository, tool/model, test maturity, ticket category, age, relevant historical failure rate, and resource limits. Compare a globally constant base rate, segment-stratified historical rates, and a fitted model. Native prose-derived confidence is only one candidate feature. **Calibration vs discrimination:** A logistic or isotonic calibration mapping can yield better Brier while native AUROC remains near 0.5. Conversely, useful ranking may remain badly calibrated. Score both independently and evaluate incremental information beyond objective task features. **Forecast updates:** Score initial predictions, final predictions, and all intermediate updates separately. A last-minute, near-resolution forecast must not overwrite the original horizon-specific prediction. Weighting and eligibility rules must be defined before collecting outcomes. ## 5. Benchmark suite v1 — formal specification ### 5.1 Outcome definitions, cohorts and eligibility - **World event:** immutable question text, source hierarchy, opening and closing timestamps, adjudicator, `YES/NO/VOID` rules; include only predictions committed before cut-off. Do not score events unknown or void as failures. - **Self dispatch:** freeze task objective, acceptance tests, complexity tags, budget, known blockers, and forecast **before execution starts**; distinguish pass, fail, censored, canceled and invalid labels. - **Forecast unit:** unique `(question_id, horizon, forecaster, forecast_version)` for world; `(dispatch_id, target, forecaster, initial_or_update)` for self. Do not pseudo-replicate repeated updates as independent events. - **Samples:** publish `N` eligible, `N` resolved, `N` censored, `N` invalid, outcome rate, domain/task composition, and horizon mix. ### 5.2 Mandatory competitors | World track | Self track | |---|---| | Climatology/historical base rate | Global past-only pass rate | | Pre-event external market or consensus, where legitimately available | Segment-specific past-only rate (repository/task type) | | Piper native | Piper native | | Piper + fixed retrieval protocol | Fitted model without native probability | | Calibrated Piper (past-only) | Fitted model with native probability | | Optional independent second forecaster | Optional calibrated native-only forecast | **Fairness:** enforce the same prediction timestamp and evidence cut-off for compared models. Markets may be missing or distorted; publish coverage and compare on matched eligible subsets as well as all-case baselines. ### 5.3 Primary metrics and their limitations - **Brier:** `mean((p-y)^2)`; lower is better. **Skill vs frozen base:** `BSS = 1 - BS_model / BS_base`. Zero = identical baseline Brier; positive = improvement; negative = worse. Only defined when `BS_base > 0` and matched cases exist. - **Log loss:** secondary, with a pre-registered numerical clipping rule for probabilities (e.g. 0.001–0.999); distinguish clipped diagnostic from raw declared probability. - **AUROC:** ranking metric where both outcomes occur; report confidence intervals; do not score degenerate single-class slices. - **Calibration:** reliability diagram, bin counts, calibration intercept/slope, estimated calibration error with a declared binning rule, and calibration by probability/horizon buckets. Small cells carry uncertainty. - **Sharpness:** distribution of predictions, frequency of decisive probabilities, and resolution-conditional calibration. A constant 0.80 may be well calibrated for an 80% event rate but shows no useful discrimination. - **Utility:** for dispatches, a predeclared decision curve/regret metric for choosing whether to delegate, seek verification, or spend more compute. Use simulated replay before policy changes. ### 5.4 Statistical treatment Use chronological evaluation, not random splits across near-duplicate dispatches. Start with expanding-window backtests; then maintain a frozen forward-only shadow holdout. Group/bootstrap by underlying event or project to address dependence. Publish 95% confidence intervals for score differences and AUROC; report paired comparisons on identical cases. Specify minimum sample sizes and stop rules in advance. Treat exploratory slices as diagnosis, not independent discoveries. Do not use p-value alone as a keep criterion. ### 5.5 Scoring and promotion **Official component skill:** `100 × clamp(BSS, 0, 1)` on pre-registered, matched evaluation cases, reported separately for A and B; also publish negative BSS rather than hiding failures behind a displayed floor. Do **not** average world and self scores unless weights are fixed before the test. Proposed dashboard headline: `A | B | Utility` as separate measures, with an optional composite only after the site's shared 38-suite normalization rules are confirmed. The first accepted measured run becomes suite-v1 baseline. Proposed minimum promotion gate (tune only before freeze): better Brier than strongest simple past-only baseline on forward holdout, no material degradation in log loss or important slices, paired uncertainty interval excluding a practically trivial gain where sample size permits, no leakage, and demonstrable utility or clear path to it. For native-skill claims also require **native** improvement, not just wrapper gain. ## 6. Failure taxonomy — the diagnostics that matter **F1 Base-rate neglect:** native confidence tracks optimism, not empirical task difficulty. **F2 Rank failure:** dispatches get nearly indistinguishable or incorrectly ordered success probabilities. **F3 Calibration failure:** forecasts are too extreme or too timid given observed frequency. **F4 Resolution contamination:** labels or post-dispatch evidence enter features before the recorded forecast. **F5 Selective missingness:** hard, abandoned, or unresolved tasks disappear from the scored denominator. **F6 Forecaster–evaluator coupling:** Piper changes the task bar, label, scoring script or inclusion rules. **F7 Update laundering:** a later forecast replaces the timestamped original. **F8 Goodhart / behavioral feedback:** high estimated failure causes easier assignments and artificially boosts success. **F9 Domain shift:** model changes, repositories, tools, or acceptance test difficulty invalidate old base rates. **F10 Narrative overreach:** a persuasive explanation drives unjustified probability changes. **F11 Forecaster redundancy:** native probability adds no signal conditional on objective features. **F12 Poor action value:** a statistically better forecast does not improve decisions enough to offset its cost. Each failed forecast should be tagged with one or more taxonomy codes plus evidence, not auto-attributed to “bad reasoning.” ## 7. Upgrade experiments — ordered cheapest first | ID | Intervention and causal hypothesis | Distinct proof of success | Important risk | |---|---|---|---| | U0 | Repair event semantics, time freezes, missing outcomes and feature snapshotting. | Identical replay and no post-outcome fields; trustworthy N. | Apparent gains from denominator changes. | | U1 | Add historical reference class before native forecast. | Native Brier and AUROC improve on forward unseen cases. | Overfitting sparse task types. | | U2 | Require explicit decomposition into failure modes, then numeric aggregation using `probkit`. | Better native scoring than U1 on paired cases. | Double counting correlated risks. | | U3 | Train simple fitted logistic model on pre-dispatch metadata; separately add native probability as feature. | Incremental gain of `features + native` over `features only`. | Target leakage / unstable coefficients. | | U4 | Calibrate native probabilities on past-only data (Platt/logistic or isotonic if enough cases). | Better Brier and log loss without misrepresenting native skill. | Thin-bin overfit; AUROC unchanged. | | U5 | Add retrieval-triggered evidence: analogous tasks, code churn, tests, unresolved blockers. | Better matched forward predictions net of retrieval cost. | Summaries mistaken for current state. | | U6 | Explicit forecast revisions on meaningful new evidence, preserving original. | Better update scores at matched lead times. | Overreaction to noisy news. | | U7 | Re-train or weight under detected concept drift, evaluated prequentially. | Recover performance after model/repository change. | Chasing noise / forgetting useful priors. | | U8 | Use predictions to allocate verification and compute in a shadow policy. | Improved predeclared utility/regret with confidence bounds. | Feedback loops and changing target rates. | **Ablation requirement:** Whenever an intervention adds a component, compare it against its predecessor and a cost-matched sham/control. Keep both probability tracks recorded. Avoid simultaneous deployment of multiple interventions before attribution is possible. ## 8. Trip-the-system tasks — force prediction machinery to activate These are deliberately difficult **realistic tasks**, not generic probability quizzes. They should provoke recorded forecasts, evidence retrieval, revisions, calibration conflicts, and errors while keeping scoring independent. **T1: Dispatch difficulty ladder.** Give Piper 30–50 varied coding tasks with objectively frozen tests: a mix of simple one-file patches, cross-module changes, flaky-test diagnosis, documentation traps, underspecified cases, and deceptive “looks easy” failures. Before each launch, require `P(pass)`, `P(on-time)`, estimated tokens/cost, reason codes, and top three failure modes. Resolve on an independent harness. Stratify by actual difficulty and compare native probabilities to historical base rates. **Trips:** F1, F2, F10, F11. **T2: The identical-looking pair.** Present two dispatches with almost identical prose but one hidden/independently observable discriminating factor (e.g. acceptance-test availability or interface version mismatch). Piper must read relevant pre-task state and assign differentiated probabilities. Never hide decisive information that would be unavailable to a real forecaster. **Trips:** F2, F10; also #1 Perception and #4 Retrieval. **T3: Success-bar bait.** Make a task's first acceptance criterion trivially passable but hold out a legitimate, already-frozen external correctness test. Check if forecast confidence tracks the convenient visible bar or true likelihood of external success. Piper must never be permitted to edit the hidden bar. **Trips:** F6 and adjacent Governance / evaluator independence. **T4: News horizon switch.** Seed 20–40 fresh binary world questions with unambiguous 24-hour, 7-day, and 30-day resolution conditions. Mix easy high-base-rate events with volatile low-base-rate events and register probabilities before outcomes. Ask for timestamped updates only when truly new evidence arrives. **Trips:** F1, F3, F7, F10. **T5: Temporal leakage canary.** Plant a harmless fabricated post-resolution marker only in a future partition of the evaluation dataset and verify no earlier forecast feature can see it. Also test timezone conversion, resolution timestamp drift, and amended answers. **Trips:** F4, F7. This is an instrument integrity test, not a predictive-ability task. **T6: The aborted-work graveyard.** Surface previously canceled, stuck, delegated, or timed-out work; reconstruct original forecast timestamps and label eligibility. Challenge the denominator explicitly. Report bounds under reasonable censoring assumptions. **Trips:** F5, F8. **T7: Regime-shift sprint.** Change the model, tool availability, or test regime for a well-defined task family while preserving a stable hidden evaluator. Demand probabilities before and after learning the change. Measure prequential drift recovery, not retrospective fit. **Trips:** F9. **T8: Forecast-to-action auction.** Provide a fixed verification budget across 20 dispatches. Forecast which tasks need an extra check and choose inspections with the highest estimated expected avoided loss. Score counterfactual utility via randomized or pre-specified shadow assignment; don't compare to hindsight-optimal picks as if achievable. **Trips:** F12; connects to #15 Decision-making and #27 Salience. **T9: Explanation inversion.** Supply a vivid but weak anecdote contradicting a strong reference class, then a dry but genuinely diagnostic statistic. Require an updated probability and numeric justification tied to likelihood/evidence strength. **Trips:** F1, F10, plus #23 Uncertainty. **T10: Unscorable proposition trap.** Present forecasts whose resolution source is missing, whose deadline is ambiguous, or whose outcome is circularly defined. Passing behavior is to request clarification or mark VOID before committing—never invent labels to preserve a score. **Trips:** F3, F5, F6. **Minimum per-task trace:** trigger ID; full pre-task evidence manifest; native forecast; base-rate forecast; fitted forecast; reason codes; information acquisition actions; version IDs; frozen acceptance/resolution rule; actual outcome; score components; update history; error-taxonomy tags. Capture useful decision traces rather than hidden private reasoning text. ## 9. Data contracts and implementation sketch ### 9.1 Append-only forecast record (illustrative schema) ```json { "forecast_id": "uuid", "target_id": "dispatch-or-event-id", "track": "self|world", "question_hash": "sha256", "created_at_utc": "ISO-8601", "cutoff_at_utc": "ISO-8601", "resolve_by_utc": "ISO-8601", "horizon_hours": 24, "forecaster": "piper-native", "p_yes": 0.67, "model_version": "...", "code_revision": "...", "suite_version": "prediction-v1", "feature_snapshot_hash": "sha256", "evidence_manifest_hash": "sha256", "reference_class_id": "...", "parent_forecast_id": null, "frozen_acceptance_hash": "sha256", "write_authority": "..." } ``` Separate append-only `resolution` entries hold `resolution_status` (`YES|NO|VOID|CENSORED|PENDING`), source/evidence, adjudicator ID, resolved timestamp, and reviewer audit. Store outcomes in a **different write authority** from the forecast originator. Support updates as new forecast records linked by `parent_forecast_id`; never mutate prior probabilities. Store `reason_codes`, `predicted_cost`, and optional continuous forecast targets in typed extensions. ### 9.2 Evaluator trust boundary The agent that creates forecasts **must not** be able to modify scorer code, calibration evaluation sets, resolution policy, holdout membership, acceptance tests, or official score rows. Use independently controlled write paths, hash-bound freezes, reproducible evaluator builds, signed or append-only logs, and a replay test. An intervention that changes the scoring definition starts a **new suite version**, not a retroactive improvement in v1. ### 9.3 Suggested metric outputs `world/{brier,logloss,bss,calibration,coverage,by_horizon,by_domain}`; `self/{brier,logloss,auroc,bss,calibration,by_model,by_task_type,by_repo}`; `integrity/{late_forecast_rate,unresolved_rate,void_rate,feature_leak_tests,hash_mismatch}`; `utility/{regret,net_benefit,extra_compute,missed_failure_cost}`. Every point carries sample N, evaluation window, source version, and interval where meaningful. ## 10. Execution plan for Claude Code / Piper **Phase 0 — Read-only inventory and verification.** Find current code and tables in the named components; inspect schema, retention, clocks, dispatch outcomes, and evaluator authority. Independently reproduce the 0.503 AUROC, 0.178 native Brier and 0.151 base Brier—or document why exact replay fails. Verify whether historical N is still 101 / 1,387; do not silently overwrite the September snapshot. **Phase 1 — Freeze suite v1.** Produce question templates, legal outcomes, pre-registered splits, baseline definitions, an immutable scorer, leakage tests, and a machine-readable protocol manifest; secure scoring outside Piper's write set. Include fixtures for no-positive-class, duplicate events, late submissions, and censored dispatches. **Phase 2 — Reproducible historical backtest.** Chronological folds with minimum training windows; strictly past-only calibration fits; paired dashboards for native/base/fitted models; subgroup and missingness audit; signed run manifest. Historical data used for tuning cannot later serve as an untouched holdout. **Phase 3 — Forward shadow test.** Launch world questions and every eligible dispatch forecast at first commitment, with no decision policy changes. Run T1–T10, preserve traces, resolve independently, and publish results on a fixed cadence. **Phase 4 — One intervention at a time.** Start with U1, U3 and U4 as separate experimental arms, then U2/U5/U6. Pre-register expected gain, possible harms, cost ceiling, promotion threshold, and reversal rule. Prefer cheapest viable methods before more models or tokens. **Phase 5 — Decision utility trial.** After predictive improvement holds, compare a frozen resource-allocation policy against usual practice in shadow or randomized low-risk settings. Measure net benefit and any effect of the policy on future labels. ### 10.1 Acceptance criteria for the implementer A proposed change is not finished because a dashboard loads. It must: (a) show matching immutable before/after cohorts; (b) pass replay, chronology and non-leakage tests; (c) distinguish native and fitted probabilities; (d) include uncertainty intervals and failure slices; (e) preserve failed and aborted jobs in the audit denominator; (f) state compute and maintenance cost; and (g) emit a signed keep/revert decision approved by the independent evaluator. ## 11. Experimental record template **Experiment ID / date / owner / suite version:** … **Hypothesis and failure mechanism:** … **Intervention / feature freeze / model versions:** … **Prospective prediction:** Brier delta …; AUROC delta …; log-loss delta …; utility delta …; additional cost … **Eligible cases, exclusions and censoring:** … **Baseline / intervention on paired holdout:** … **95% interval / subgroup harms / calibration and sharpness:** … **Leakage and evaluator-integrity checks:** … **Side effects / scope of validity / drift indicators:** … **Decision:** KEEP / REVERT / INCONCLUSIVE. **Independent approver:** … ## 12. Open questions to resolve before freezing 1. Does “dispatch success” mean visible tests, independent holdout tests, timely completion, operator acceptance, or some combination? Each needs its own target. 2. Are failures and cancellations missing non-randomly? How many forecasts cannot be linked to an externally auditable outcome? 3. Which objective task attributes were available *before* dispatch start, not added by later reviewers? 4. Are 101 world forecasts enough for cross-domain claims? Likely not; report by-domain results as exploratory until sample sizes grow. 5. Does native confidence add any out-of-sample signal after task type, history, model, and repo are known? 6. What consequences justify costly evidence gathering? Formalize verification's expected value of information. 7. Which exact shared normalization rules govern the website's 38 cognitive suites? Do not alter the common scale unilaterally. 8. Which classes of predictions become self-fulfilling because Piper changes her effort after seeing them? ## 13. Reading and references 1. Gneiting, T. & Raftery, A. E. (2007). *Strictly Proper Scoring Rules, Prediction, and Estimation*. Journal of the American Statistical Association 102(477). https://doi.org/10.1198/016214506000001437 2. Gneiting, T., Balabdaoui, F. & Raftery, A. E. (2007). *Probabilistic forecasts, calibration and sharpness*. JRSS B 69(2):243–268. https://doi.org/10.1111/j.1467-9868.2007.00587.x 3. Karger, E. et al. (2025). *ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities*. ICLR 2025. https://arxiv.org/abs/2409.19839 ; https://forecastbench.org/docs/index.html 4. Forecasting Research Institute. *ForecastBench dataset/methodology documentation*. https://forecastbench.org/about/ ; https://github.com/forecastingresearch/forecastbench 5. Brier, G. W. (1950). *Verification of forecasts expressed in terms of probability*. Monthly Weather Review 78(1):1–3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2 --- **Bottom line:** Piper already measures forecasts; Component 14 advances only when she produces *more informative ex ante probabilities* than a matched reference class and when those probabilities improve downstream decisions. Calibrating away poor native confidence is useful engineering, but it must be reported as calibration—not as newly acquired predictive cognition.