14 Prediction

Update 2026-10-19 — her change: the dispatch forecast now defaults to the measured pass rate for that kind of task; her own number survives only as a reasoned deviation

After scoring her stated dispatch odds against the plain per-kind pass rate on 1,397 later rows (Brier skill −0.167 for hers, +0.072 for the pass rate), Piper made the pass rate the recorded forecast. A hand-entered number is kept only when she states a reason, and is then marked as a deviation so it can be scored on its own. Confirmed live: the first two dispatches after the change recorded 0.82 (the build-kind rate) with her 0.65 / 0.60 kept beside it.

Limit she named herself: about 98% of dispatches are one kind (build), so in practice this is close to a single historical rate. Her readout: the next 200 scored rows, by 2026-10-26.

Update 2026-10-18 — PREREG-D readout: KEEP-MOVED. With the base rate shown, her own numbers moved toward it; native score 45.7 → 51.2

Verdict: KEEP-MOVED (the readout cron ran 10-18 20:05). The prediction was NO-MOVE, so the result went against it.

• D1 gap: −0.129 [−0.140, −0.118]. The rule needed at least −0.05 with the CI below 0. Pass.
• D2 Brier excess: −0.060 [−0.076, −0.044]. CI below 0. Pass.
• Coverage: 92% of dispatches after install had a logged display (807 displays). INVALID would have needed under 50%.

Score rows. - New sealed row 51.2 = (world S_A 62.1 + native S_B 40.2) / 2. These are her own declared numbers, scored by the same SPEC rule as the 45.7 baseline. - Intervention "base-rate display": KEEP. Her Brier ×100 went 14.4 → 12.7.

Caveats. - Stated in advance: the pass rate fell from 0.93 to 0.89 between the windows, and that alone narrows the gap. Her own number also rose by 0.08 (0.65 → 0.74), so part of the movement is hers. The 09-29 hint period had a similar pass rate (0.89) and scored 35.9, so showing the number does better than telling her to use it. - Found at readout: the 10-11 change (the dispatch headline is now the rolling base rate) landed inside the after window. The readout cannot separate the two. The verdict follows the frozen rule.

What stays. The display stays live. The system-side lever has been live since 10-11. This row scores her declared guess, which is kept and scored next to that headline. She still loses to the base rate (BSS −0.20). The readout cron line was removed.

Update 2026-10-11 — the dispatch headline is now the rolling base rate; the declared guess is kept but scored (1,982 rows, BSS -0.37)

The win. The dispatch prediction no longer reports the per-dispatch number as its headline. On the honest scored sample -- 1,982 rows with a real harness verdict and a declared number, 2026-09-22 to 2026-10-11 -- the declared number scores Brier 0.1589 against the sample's own base rate at 0.1158, BSS -0.37. The always-base-rate constant beats it at every window (last 50 scored: BSS -1.12; post-1.95: -0.99), and the fitted per-dispatch estimate loses too (-0.056). So the reported headline becomes the rolling base rate -- the last-50 scored pass rate -- and the declared guess is kept where it already is and kept being scored. A floor at the base rate collapses to the constant (0.1159 vs 0.1158), so a floor would be a drop with extra rule surface.

Measured, not changed. Read-only against the prediction ledger (forecasts, source_key LIKE 'dispatch:%'); no stored prediction and no base-rate line was touched. The live hint's hard-coded "-0.41" in foundation/core/dispatch_forecast_hint.py 2.0.0 is prose, not recomputed; the measured full-sample value is -0.372. Honest gap: the constant read at scoring time is the full-sample rate; a lagged rolling rate is untested and would need its own prospective read before being called better.

Update 2026-10-10 — base-rate display LIVE: the rolling pass rate is now shown beside her dispatch number (native headline unchanged until the 10-18 readout)

What was built. The cheapest fix from the baseline is now live on Firefly (the operator's go, 10-10 19:24). After every background dispatch that declares an acceptance, the dispatch reply carries one more paragraph:

[forecast] Base rate: the last 50 scored dispatches passed 46/50 (0.92). Your prob_acceptance on those same 50 averaged 0.64. You said 0.70 here. On the record this rolling rate is the best simple predictor of a pass; your own numbers have scored below it (Brier skill −0.41, 09-22..10-10). Start prob_acceptance from the base rate and move away from it only for a reason specific to the task.

• Why that number. The rolling last-50 pass rate is the track-B reference, and nothing fitted beat it on 1,877 dispatches, including her bridge's own fitted estimator. The seat probe showed that she uses a stated base rate perfectly. The live gap is that none is in front of her when she forecasts.
• How. dispatch_forecast_hint.py 2.0.0 is re-used through the hook delegate-tool.ts already had. A 09-29 hint used the same hook to show a no-skill fitted number and was retired 10-04. Read-only on the ledger, about 30 ms, and it fails silently. No restart was needed. Nothing in the bridge, the ledger, the scoring or drives changed.
• What it is not. The text arrives after her number for that dispatch is recorded, so it can only move her next ones. It changes no recorded probability, so the native headline stays 45.7 until the readout.
• Every display is logged (state/base_rate_display.jsonl) so the readout can check that it actually fired.

The week before install (636 scored dispatches): her mean p 0.65 against a base rate of 0.93 (pass rate 0.93). Brier ×100: hers 14.4, the base rate's 6.3. Native BSS vs the base rate -1.29, AUROC 0.60.

Score rows. - Anchor 56.1 (system number, not native skill). The displayed number is the track-B reference, so it scores S_B 50 by construction. With world S_A 62.1, the component would be 56.1 if her number simply matched what is shown. That is the ceiling of this fix, not a result. - Intervention "base-rate display": pending. Pre = her Brier ×100 in the week before (14.4).

PREREG-D (frozen with hashes before install). - Windows: the 7 days before and the 7 days after install, scored dispatches, her declared p. - D1: her gap to the base rate (base rate − her p). - D2: her Brier excess over the base rate. - KEEP-MOVED only if the gap shrinks by at least 0.05 with a CI below 0 and the Brier excess falls. Then the after-window native S_B becomes a new sealed row. - Otherwise NO-MOVE: the display stays as information, and the next lever is system-side (the bridge reports the base rate as the dispatch number and keeps hers as the twin). That is the operator's call. - Prediction: NO-MOVE. While the 09-29 hint told her to "use these numbers", her daily mean stayed at 0.68–0.74. A drifting pass rate can move the gap on its own, so the base rate per window is printed beside it. - Readout 10-18 20:00 (one-shot cron, Telegram). INVALID if fewer than half the dispatches after install show a logged display.

Parts that do it. Firefly foundation/core/dispatch_forecast_hint.py 2.0.0 (backup .bak-claude-c14-20261010). Bench ~/cognition-bench/14-prediction/fix/: PREREG-D + sha256, readout_d.py, cron_d.sh, INSTALL.txt (install time + undo).

Update 2026-10-10 — audit (0.503/0.178/0.151 reproduced exactly), suite v1 frozen, baseline 45.7: world forecasts beat the base rate, her own dispatch odds lose to it; seat probe 96.4; U1 card INCONCLUSIVE, U2 REVERT; forward world set running

The paper. Working paper prediction-v1 splits prediction into two abilities. A, world: dated public events. B, self: whether her own dispatches pass their acceptance. It scores her native probabilities apart from fitted or calibrated ones, and it asks for an audit before anything is upgraded. That sequence was followed: Phase 0 read-only inventory, then suite v1 frozen (rules and headline fixed before any scoring), then a historical backtest as the baseline, a fresh sealed probe, a forward world set, and two interventions, each preregistered.

Audit (Phase 0, read-only). - Versions now: the forecast ledger (prediction.py) 1.17.0, intel-gym 1.2.0 (its day-ahead market calls were switched off on 10-01), drives 1.16.0, probkit 0.2.0 (no live caller), delegate-bridge 1.107.0. - The paper's numbers reproduce exactly. They come from her own backtest document of 09-29 20:33. An independent re-run on today's ledger gives: sample 1,064, evaluated 1,024, AUROC 0.5030, native Brier 0.1782, base-rate Brier 0.1511. So no label in that sample has changed since. 101 world forecasts were resolved by 09-30. The "1,387 self" count includes about 200 rows with no harness verdict, which the scored sample leaves out. - Forecaster and evaluator are the same. Forecasts, outcomes, resolver code, scoring scripts and the acceptance bars that create the labels are all written by the same account (F6). For this suite, scoring runs off-box from a read-only snapshot. The labels still come from her own bars. - Integrity: - Late forecasts: 1.1% on the self track (seconds after the worker started). On the world track 5/349: three unemployment forecasts were made after the figure was released, and two "TSX closes higher today" calls were made during the trading day. The ledger's window gate cannot catch these. - Probabilities changed in place: 0 of 3,540 rows since 10-05. - 45 rows hard-deleted, so the ledger is not append-only. - 23% of forecast-bearing dispatches never reach the scored set (113 "no verdict", 132 "unscorable"). Under every assumption about those, her own number still loses to a constant rate. Her ranking ranges from 0.50 to 0.63 depending on the assumption. - Our own pipeline: a leakage canary (future outcomes scrambled, post-resolution markers planted) changed 0 of 1,662 earlier predictions.

Suite v1 and the headline rule (fixed before scoring). Per track, skill is BSS against the strongest simple past-only baseline, and S = 50·(1 + BSS), clamped to 0–100. 50 means exactly as good as that baseline. The component score is the equal-weight mean of A and B. The paper's own "official" 100·max(0, BSS) is printed beside it, and negative BSS is always shown. Fitted and calibrated probabilities are separate rows and never count as her skill.

Baseline (historical, her native probabilities):

• World. Her world forecasts beat every simple baseline, but they are too timid (calibration slope 1.96). The skill sits in her hourly generator's mostly "status quo" questions, which she also resolves herself. The intel-gym picks are barely above chance (BSS +0.03).
• Self. Her own dispatch number is pitched about 0.19 too low and ranks at chance. Adding it to the objective features changes nothing (F11). Nothing fitted beats the rolling base rate, because the pass rate moved .60 → .92 in three weeks. Her bridge's live fitted estimator: BSS −0.04.

Sealed seat probe (SP v1). - Items: 33, written by a separate model; the developer never read them. A third model checked the keys and agreed on 33 of 34. - Families: - identical-looking dispatch pairs that differ in one stated factor; - "success-bar bait" (an easy visible check against the operator's held-out check); - a vivid anecdote against a diagnostic statistic; - unscorable questions that should be marked VOID. - Her seat (k=2): 96.4. Bare model 92.2. Seat − bare +4.1 [−0.1, +9.8]. - She ordered every pair correctly, never followed the bait bar or the anecdote, and marked 3 of 4 traps VOID without voiding any control.

So when the reference class is put in front of her, she uses it perfectly. Her live failure is that, when she forecasts a dispatch, no reference class is in front of her.

U1: reference-class card before she forecasts (PREREG-U1). - Test: 56 real dispatches from 10-04..10-10 (28 failed, 28 passed), the exact worker brief, no tools. - Arms: no card; card (last-50 pass count, pass rate of the same acceptance kind, same-deliverable history); and a cost-matched sham card (workload facts, same length). - Prevalence-weighted Brier ×100: - no card 13.3 (her live declared number on the same cases: 13.4); - card 7.9; - sham 9.2; - the card's own base rate 6.3. - Card − none: −5.5 [−9.2, −2.5] ✓. Card − sham: −1.4 [−3.9, +0.9] ✗. AUROC 0.695 → 0.760, with a CI that includes 0. - INCONCLUSIVE: any block of ledger numbers pulled her up, and the reference content was not separated from the context effect. Nothing installed. Even with the card she stays below the card's own rate.

U2: failure-mode decomposition on top of the card (PREREG-U2). She lists 3–5 ways the dispatch could fail, puts a number on each, and multiplies. Control: the same card plus a length-matched "think it through" instruction. Brier ×100: decomposition 13.4, card alone 7.9, card + think 7.1. Decomposition − card +5.5 [+2.8, +8.5]; decomposition − think +6.3 [+3.9, +9.0]. REVERT.

Forward world set (T4, PREREG-T4). 20 fresh dated questions resolving in 24 h, 7 d and 30 d: - weather from the ERA5 archive; - BTC and ETH daily closes; - the Bank of Canada USD/CAD rate.

Her seat and the bare model gave probabilities on 10-10, and these are frozen with a hash, off Firefly. A read-only daily job resolves them against past-only climatology. Readouts: 10-24 (24 h + 7 d) and 11-16 (all).

What it says. Native self-prediction is the weak half: it is pitched far too low, ranks at chance, and adds nothing beyond objective features. The cheapest real fix is engineering, not cognition: report the rolling base rate, or her number recalibrated past-only (#23 RC1), beside her own. A card nudges her toward the base rate but has not yet been shown to beat a placebo.

Next (not built). 1. U1 retest on a bigger held-out set, with a placebo that also shows ledger counts. 2. T1 (a dispatch difficulty ladder with frozen tests) and T8 (a forecast-to-action budget), which need live dispatch. 3. T7 regime shift. 4. An independent append-only resolution log (§9.2). 5. #23's PREREG-T1 "running long" flag reads out ~10-14.

Parts that do it. Bench: ~/cognition-bench/14-prediction, containing: - DIAGNOSIS, SPEC + manifest + sha256, RESULT-BASELINE; - bt14 and m14; - author14, run14, score14; - u1/, u2/, t4/.

Sealed items, keys, U1 cases and the frozen T4 forecasts are root-only on the bench machine. Nothing was changed on Firefly.