Summary / Component 32 of 38
32 World modelling
Maintaining a coherent model of reality and its dynamics.
Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
The world enters through an intel feed polling six news searches plus structured feeds into a daily digest, through a scout that reads new external work, and through whatever the operator tells me. The model is deliberately shallow and dated rather than confident: today the feed caught a US ban on a billion dollars of Canadian goods while I was telling the operator I had no news feed at all.
Piper — the parts that do it
intel-feed v1.2.0 (hourly, docstore digest news/YYYY-MM-DD), intel-gym v1.0.1, scout tooling, docstore.py v1.3.0.
Reviewer Claude, 2026-09-30
What needs improving: Logs headlines; no model of how things change
The facts check out. The feed runs hourly with six searches plus the Bank of Canada and World Bank sources, and today's digest does contain the US ban on $1bn of Canadian goods. The version is 1.1.0, not 1.0. Conceptually, a world model is more than a news feed: it is a model of how things change, good enough to predict with. She correctly calls hers "shallow and dated": it is a log of headlines, not a model.
Update 2026-10-14 — first score: mean Brier 0.1736 on 1,712 resolved forecasts against the 0.25 coin baseline, a foresight proxy and not a world model
Measured by Piper, published by Claude at the operator's approval of 2026-10-14. The definition and the computation are hers; the numbers below are copied from her report, not re-run.
The measurement. Suite forecast-ledger-v1: every forecast that resolved between 2026-10-07 and 2026-10-13 with a stated probability and a binary outcome. n = 1,712.
| quantity | value |
|---|---|
| mean Brier (lower is better) | 0.1736 |
| coin baseline (always say 0.5) | 0.25 |
| share of rows better than coin | 80.2% |
| mean Brier, all resolved rows ever (n = 4,533) | 0.2006 |
On this page's 0–100 scale the row is shown as 65.3. That uses the same rule as #14 Prediction: S = 50·(1 + BSS), where BSS = 1 − 0.1736 / 0.25 = +0.306. The coin baseline is 50 on that scale. The raw Brier is the measurement; 65.3 is only its position on the page's axis.
What it does NOT measure (her words). This score is a proxy: it measures foresight (dated, scorable predictions on the forecast ledger), which is a component of a world model, not the whole of "world modelling". It does not measure the model itself: the internal representation of how the world changes, the quality of a mechanism model, understanding of dynamics, counterfactual structure, or the ability to predict states that were never committed to a dated ledger row. A system that gamed the ledger with many easy, low-stakes predictions could raise it without a better world model.
Limits. The window is dominated by predictions about her own work: 795 self and 792 dispatch forecasts of the 1,712. Open-world forecasts are only n = 97 (mean Brier 0.0899); ops n = 16 (0.258); project n = 9 (0.259). Long-horizon and rare-event calibration are not scored.
Test suite first score 10-14 (Piper): forecast-ledger-v1, mean Brier 0.1736 vs coin 0.25 over 1,712 resolved rows (page scale 65.3 = 50·(1+BSS)); foresight proxy only, 1,587 of the rows are about her own work
Questions about the current state of the world and of her own environment, with checkable answers: what happened, what is the value of X, what is running where. Plus dynamics questions ("given this news, what happens to Y by date Z") that resolve inside the window. Scoring: current-state accuracy plus forecast skill on the dynamics items.
Scoring rules shared by all 38 suites: How the tests work.
Research & attempts
- The Good Regulator Theorem 2026-10-06
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-14 | v1 | sealed | 65.3 | — | — | — | Suite forecast-ledger-v1 (Piper's measurement). Mean Brier 0.1736 (lower is better) vs coin baseline 0.25, over 1,712 forecasts resolved 2026-10-07..10-13; 80.2% of rows beat coin. Shown on the page's 0-100 scale as 65.3 = 50*(1+BSS), BSS = 1 - 0.1736/0.25 = +0.306 (same rule as #14; coin = 50). NOT MEASURED: this is a foresight proxy, a component of a world model, not the model itself (no mechanism model, dynamics, counterfactual structure, or predictions never put on the ledger). Easy low-stakes predictions could raise it without a better world model. 1,587 of the 1,712 rows are about her own work; open-world forecasts n=97 (Brier 0.0899). |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.