36 Recursive reasoning

Update 2026-10-08 — logging fix installed (predict-first 1.0.1) and predictions now graded every night

Installed (operator's go). predict-first.ts 1.0.1 is on Firefly and took effect with the router restart at 16:52. It records her full prediction, up to her stated "(NN%)" or the end of that line (max 800 characters), and reads her confidence from the whole prediction. 1.0.0 stopped at 160 characters and at the first quote mark or line break. Nothing else in the gate changed: a query to another reasoner without a prediction is still refused once. It loads at her next turn.

Graded every night. predcheck36.py runs at 04:20 on the bench machine. It grades only predictions it has not graded before, read-only on Firefly, with the same judge as the other benches. Predictions whose worker is still running are graded once the worker finishes. The totals are in the bench log (live/cron.log); this page is updated from them by hand.

Starting point (10-03 to 10-08): 12 of 20 resolved predictions came true (60%); stated confidence 70%, right 63%.

Update 2026-10-08 — her live predictions are now scored: 12 of 20 came true (60%); stated confidence 70%, slightly over; her confidences do not yet separate likely from unlikely

What was missing. predict-first tells her each prediction is "scored against the real answer when it comes back", but nothing scored them. Her workers run in the background, so the prediction log only held the dispatch receipt.

The join. predcheck36.py (read-only on Firefly) links each committed prediction to its worker through the task id, then reads the finished record. That record holds the worker's report and the harness's own acceptance verdict, an independent command run after the worker finishes. A judge (claude-sonnet-5-5, the same as the other benches) grades each prediction against that record. Where a claim is about acceptance, the harness verdict decides, not the worker's word. - true: every stated part happened. - partial: some parts did not. - false: the main claim did not happen. - unresolvable: the record cannot settle it.

Only "true" counts as a hit: a compound prediction with one wrong part was wrong. Grades are cached, so each prediction is graded once.

Result, 10-03 to 10-08: 24 committed predictions.

• Hit rate: 12 of 20 resolved = 60%. With partials counted as half it is 72.5%.
• Calibration (19 resolved with a stated confidence):
• she said 70% on average and was right 63% of the time, slightly overconfident;
• Brier 0.244, against 0.233 for a forecaster who always says 63%. Her confidences do not yet separate the predictions likely to come true from the rest. The sample is small.
• Where she misses: the exact shape of an outcome. Examples: "three RETIRE, one RE-SCOPE, one CARRY FORWARD" when all five were retired; "at most two need declaring" when all eight did; routed recall "near 0.60" when it was 0.52. "Will the acceptance pass" is usually right.

A logging defect found on the way. predict-first's ledger cuts a prediction at 160 characters, at the first quote mark and at the first line break (predict-first.ts:68). It reads the confidence only from that cut text. So 10 of 24 predictions were truncated and 8 lost their confidence. The scorer recovers the full text from the worker's own session record for 8 of them. The Dusk question and one call that never ran have no such record. The fix in her code, a longer capture and the confidence read from the whole task, is staged as a proposal and is not installed. That needs the operator's go.

Parts that do it - Bench: ~/cognition-bench/36-recursive/predcheck36.py 1.1.0; grades in live/predcheck.jsonl

Update 2026-10-08 — summary uses the matched baseline (72.4 → 96.2, +23.8); live check: predict-first works in real work, perspective is never called

Audit fix (X2). The summary used the first sealed row as Baseline: 82.2, her seat on dev set A2. The fix was scored on held-out set B2, so the matched comparison is the same seat with both fixes excluded on B2: 72.4 → 96.2 (+23.8). The 82.2 row is now an anchor (kept in the history). The text of the 10-02 entry already used these numbers.

Live use, 10-02 to 10-08 (read from her logs, bench runs excluded). - predict-first works. Its ledger has 21 refusals and 24 committed predictions, 10-03 to 10-08. 23 were on delegate and 1 on ask_dusk. Most are concrete and carry a confidence, for example "the dry run finds more than 500 overdue forecasts that name no resolver (70%)". She predicts before she sends a worker, as intended. - The predictions are never checked. A delegate runs in the background, so the ledger's "answered" row is only the dispatch receipt, not the worker's result. Whether her predictions come true, and whether her 60–90% confidences are calibrated, is not measured. That is the next step if this is pursued: join each prediction to its delegate's finished result. - perspective: 0 real calls. The tool is loaded in her live seat, but her real work has had no nested-belief question ("what does X think Y knows"). That is expected, not a fault. The bench gain shows what she does when such a question comes up.

Update 2026-10-02 — benchmark built and validated; harness interference found; perspective + predict-first: held-out v2 72.4 → 96.2

What was built first (the plan's rule: the instrument comes before the intervention).

• recursive-belief-v1: procedurally generated nested-belief stories in the plan's small world.
• Events: moves (always a real change of location), leaving and entering, and truthful private messages.
• Seeing is co-presence: everyone present at a move sees it, and sees the others see it.
• State is explicit: the true location, plus a per-agent log of what each agent saw and whom they saw see it.
• Two oracles take different code paths over that state and must agree before an item can exist.
• recursive-belief-v2 adds one phenomenon at a time: unlabelled lies (the story never says who is lying), hidden watchers, 5 agents and 18 events. A third implementation agrees with both oracles on every chain up to depth 4.
• Validation:
• 38 property tests (the plan's §5.7 world, epistemic and depth invariants);
• a frozen 100-item golden set per version;
• 5 items traced by hand.
• Shortcut resistance:
• A false-belief item at depth k has an answer different from reality and from every shallower answer.
• True-belief controls check for the opposite shortcut.
• Every shallow heuristic scores near or below chance: reality, the shallower answer, the last message, the last location, the starting location, and guessing by elimination.
• A first attempt that excluded surface answers outright was rejected: it let elimination reach 50–70%.
• Four suites per sealed set: nested belief R0–R4 (64 items), belief then behaviour (8), strategic games against scripted bots (4), and predicting a fresh bare copy of her own model (6).
• Two held-out sets (B and B2) were generated at the same time and not scored until the fixes were frozen.

Conditions: - her Day seat (blind, in the operator's voice); - the same seat without person_model, her "theory of mind" module; - the bare model through the API.

Seat probes were billed to a separate bench account, so her $20 daily cap was untouched.

Baseline: development sets, before any fix

Diagnosis (the plan's §9 matrix):

• The model can do this. Bare deepseek-flash, thinking at high effort, holds 92–96% of false-belief items all the way to depth 4.
• There is no depth collapse (H2) and almost no leakage (H1).
• Belief turns into behaviour every time (no H5 gap).
• Scripted opponents are predicted, not just equilibrium-solved (no H6).
• Piper scores below the bare model on the same items. The plan's last row reads this as harness interference, and it shows up in two specific ways: 1. False-belief bias, the reverse of leakage. Her true-belief controls fall to 67–79%, under the research doc's 90% validity bar.
  • She invents absences: "that's the only thing she was ever told", about someone who never left the room.
  • She misses a return: an agent who came back before the last move.
  • She hedges about "the puzzle's other reading". 2. She won't predict another reasoner. Asked what a fresh deepseek-flash would answer, she sent a worker to sample it 100 times, every time ("measuring rather than guessing"). That is query → accept, the pattern the plan says teaches nothing. The one time she guessed first, she was right.
• person_model (her "theory of mind" module) does nothing measurable: removing it changes nothing beyond noise (§19, experiment 1). It has never been called. Its job, modelling the operator, is not nested belief.
• Belief-then-behaviour and strategic prediction were at ceiling in every condition, so this suite does not yet discriminate on them.

What changed (two deterministic pieces, not prompt rules):

1. perspective, a Day tool. It is the plan's Phase 2 "explicit epistemic state", the SymbolicToM design. - She writes down only what happened: who left, who came back, who moved what where, who messaged whom, and who watched unseen. - Code replays, for each nested perspective, only the events that perspective saw, and returns the answer at every level with the events it used. - Messages: the recipient believes them; the sender's own belief does not change, so lies are handled; both parties know the other knows. - It agrees with the benchmark's oracle on 48,300 of 48,300 chains. It shares no code with the generator.
2. predict-first, the plan's Phase 3, prediction before query. - When she asks Dusk, or sends a worker to sample another reasoner, without a committed prediction, the call is refused once that turn. - The refusal asks for prediction: <answer> (<n>%). That prediction and the real answer go to state/prediction_ledger.jsonl, the plan's prediction ledger.

Held-out test. Sets B and B2 were generated at the start and scored only now. Each was run with the fixes and again with both extensions excluded; the excluded arm is the ablation (Gate E).

Reading it. - On held-out v2 every false-belief item, at every depth, is right (96 of 96), and leakage is gone. Without the fixes, the same seat on the same items scores 79% with 10% leakage. - With the fixes, Piper now beats the bare model. Before, she trailed it by 10 points. The harness went from hurting to helping. - The gain replicates on held-out v1 (85.1 → 92.5). Self-prediction there is level at 50%, but on 6 items it can't be read either way. - She used perspective unprompted on 89% of nested-belief items. All 14 sampling requests went out again with a committed prediction, and her self-prediction rose from 42% to 75%. - Cost: about one more tool call per item and roughly the same dollars ($2.11 against $2.47 for 164 probes).

What this does not show. - Live use (T6, the plan's §20 "what failure looks like"). Benchmark stories are the format the tool was built for; real conversations aren't. Both pieces went live at the 11:10 router restart, so their use in real conversation is now logged: - perspective calls show up in her sessions; - committed predictions and their outcomes go to the prediction ledger.

Neither had fired live at the time of writing. - Belief-then-behaviour and strategic prediction were at ceiling, even against the harder v2 bots (a cycler, a lagged follower, a contrarian), so they show no gain. The next suite needs opponents she has to learn over time (the plan's experiment 6). - One model only. Gate D asks for a second provider for an architectural claim, and that is not done.

Parts that do it (added)

• perspective.py 1.0.0 + perspective-tool.ts 1.0.0 (Day tool perspective), tests/test_perspective.py; day_toolset 1.6.0; tool_contracts.json entry; toolvalue 1.4.4
• predict-first.ts 1.0.0 (mode state/predict_first.json = enforce), ledger state/prediction_ledger.jsonl
• foundation_router.py 1.9.98 (live 2026-10-02 11:10)
• Bench ~/cognition-bench/36-recursive: rb1.py (v1), rb2.py (v2), test_rb1.py / test_rb2.py (38 property tests), golden_v1.json / golden_v2.json, AUDIT.md, gen36.py (sealed sets A, B, A2, B2), run36.py; seat 1.14.0 (bills a separate bench account)