9 Causal reasoning

Update 2026-10-09 — correction to the 10-04 rows: one item per arm zeroed by a failed judge call

Why this correction. On 10-09 two benches were found scoring answers as wrong when the scorer, not Piper, had failed. The answer judge (claude -p) went down without raising an error. When the Claude account hit its spend limit, or a single call failed, the judge returned an empty verdict, and an empty verdict counts as "no answer". Every bench run since 09-25 was swept (bin/sweep_judge_voids.py). Items scored only on tool use need no judge and were excluded. Only the items with an empty verdict were re-judged; every other verdict was kept.

Nothing changes in the decisions. The routing ship rule needed a CI above 0, so it still fails, and routing stays off. These are all reference rows now. The summary is the 10-09 matched pair (57.5 → 55.0).

Update 2026-10-09 — summary fixed: the live configuration, tested on held-out — no end-to-end effect (55.0 vs 57.5, −2.5 [−20, +15])

Why the summary was wrong. It read 70.0 → 67.5, and neither number is her live setup. - 67.5 was the routing-on arm. Routing is off live. - 70.0 was a 10-04 control, run on an older memory graph.

Live she runs the memory lookup fix on and routing off. That combination had only been run on a dev set.

The test (PREREG-GH2, written before the run). Held-out set GH, 40 questions about her own recorded links. Two arms ran side by side on today's graph: - control: both switches off, her behaviour before 10-04; - live: lookup v2 on, routing off.

Paired −2.5, 95% CI [−20.0, +15.0], 5 wins and 6 losses. No end-to-end effect detected. Under the preregistered rule the switch stays as the operator set it. It is still justified on the lookup controls: 400/400 correct lookups, against 118/400 before.

A void judging pass first. The 09:23 runs were scored while the judge (claude -p) was returning the account's spend-limit message. It silently produced empty verdicts on 36 of 40 items per arm, which made both arms look near 0. The stored replies were re-judged at 16:1x; the seat was not re-run. A failed judge call now aborts instead of scoring as "no answer" (rejudge9.py).

Why both arms are lower than on 10-04 (70.0). Her memory graph has grown since GH was sealed, so the old control is no longer comparable. That is why a fresh control ran beside the live arm.

Summary now: the matched pair, 57.5 → 55.0. The 10-04 held-out rows, the routing arm and the dev-set rows are kept as anchors.

Parts that do it - ~/cognition-bench/09-causal/PREREG-GH2.txt, rejudge9.py, paired9.py, ev9gh2.py

Update 2026-10-04 — lookup fix turned on by itself (operator's call); routing stays off

The operator approved turning on the lookup repair (MEMSTORE_CAUSAL_V2) by itself. memory/memstore.py 1.15.4 now defaults the switch to on, and it is live with no restart. Routing (TRIGGER_CAUSAL_V2) stays off.

Why it ships alone. The preregistered rule covered lookup and routing together, and that pair showed no end-to-end gain. The lookup repair is a plain bug fix with exact checks (118/400 → 400/400 on the controls; "does A cause B?" 2/80 → 80/80; no new false links), and the package allows shipping that kind of fix on those checks alone. Answers should not change much: she calls causal on about 12.5% of causal questions. When she does, she now gets the right link.

Live check. For "does an oom kill cause swap filled to 1.8g/2g?", the old path returned two unrelated chains. The new path matched both named things and returned the recorded link (oom kill -causes(+)-> swap filled to 1.8g/2g, memory #475). It also said plainly where no link is recorded in the other direction.

Revert: set MEMSTORE_CAUSAL_V2=0, or change the default back to 0. Neither needs a restart.

Update 2026-10-04 — benchmark built (the base model is at ceiling on formal causal reasoning); lookup and counterfactual repaired; held-out end-to-end test: no gain, switches stay off

Built from the operator's two documents: the "Causal Reasoning Engineering Package" and its source draft. Their central rule shapes everything below. Fix the measurement first; never call the component improved because the graph grew, the tool got called, or the prose sounds more scientific. Three questions are kept apart:

• Does she understand causation?
• Does the graph or tool contribute?
• Do her real answers improve?

Step 0: the facts. A read-only audit of the live system came first (DIAGNOSIS.md).

Track 1: does she understand causation? The test is generated worlds with an exact oracle. Each world is a small structural causal model: yes/no events, each set by a rule over its causes with random glitches and triggers. The oracle enumerates every random state in exact fractions and returns three things:

• seeing: P(Y | X);
• doing: replace X's rule and keep everything else;
• imagining: the counterfactual for one past episode. It first works out what the random parts must have been, given what was observed, then changes X and keeps them.

It reproduces the package's worked example exactly (0.30 / 0.50 / 1.00), and it agrees with an independent implementation and with Monte Carlo checks.

Result: the base model is not the bottleneck. On dev:

It handles the front door, M-bias and five-variable equivalence classes without help. Piper's 3-point loss on v1 is two "can't tell" answers on claims that were determined and one over-refusal. That is a small harness cost, not a missing capability. A causal-calculation tool would be the package's "optional tool unnecessary", so none was built.

Track 2: does the graph contribute? This track runs matched lookup controls on her live graph (package §5). Cases are sampled from her real causal_edges:

• Both ends named ("does A cause B?").
• Inverse wording.
• One end only.
• A "prevents" edge.
• A name variant.
• Two unrelated things.

The dev cohort and a held-out cohort are drawn with different seeds. Every case keeps the database truth, the raw tool result, the shown links and the verdict.

Track 3: do her answers improve? Operator-style questions about links she has recorded, asked in her seat:

• "From what you've got recorded: does X lead to Y, or is it the other way round?"
• "What led to Y? Was it X, or is that backwards?"

The answer is FORWARD, REVERSE, or no recorded link. Only edges whose source memory states the link are used, so the answer is defensible from her own notes. Polarity questions were left out, because the extractor's double negatives would reward copying an error.

On dev, the lookup fix alone changed nothing (+2.5, 95% CI [−15, +20]). She almost never calls the tool: causal on 12.5% of questions. Otherwise she spends about 11 search and read calls hunting through notes. With the routing also fixed, dev showed +25.0 [+5, +45]. But the dev control ran beside two other arms on a loaded Firefly (192 s per question; at times her memory store was refusing writes), so that comparison was not clean.

The held-out test was preregistered before it ran (PREREG-GH.txt). Set GH has 40 new questions from a different seed. Both arms ran side by side under the same load, because the dev control had run beside two other arms.

Paired −2.5, 95% CI [−20, +15]: the ship rule failed, so the switches stay off. Under matched load the control scores 70, not 55, which means the dev gain was mostly the load. The injection fired on all 40 held-out questions, so she had the right links. What she did with them is the finding:

• She treats the graph as one witness, not the answer. She cites the injected line ("no recorded link either way in the causal graph"), then reasons about the mechanism and commits to a direction anyway. Five of the no-link misses are that.
• She disputes edges, sometimes rightly. One example: "the graph read narration order as causation… worth knowing it does that". With 61% of edges being loose readings of their source, distrust is a defensible policy.
• The questions blur two things. "From what you've got recorded" mixes what is recorded with what is true. A no-link item marks a reasoned direction as wrong even when the reasoning is sound. That is a limit of this track, not only of her.
• The fix made her cheaper and quieter. 28% fewer calls, 13% faster, 25% cheaper and fewer non-answers, but no more correct.

The fixes.

• Lookup v2 (memory/memstore.py 1.15.2, switch MEMSTORE_CAUSAL_V2):
• Both directions. When a question names two things, the links between them are checked in both directions and reported with their recorded direction, a sign (prevents = −) and the memory they came from. "NO RECORDED LINK" is said explicitly.
• Starting points. These come only from graph nodes: exact normalised match, then a guarded substring match, then content-word overlap. Two-letter names no longer flood the starting set; one exact name used to match a median of 14 nodes.
• Endpoint split. Endpoints are split from the question deterministically, so the language model's entity split can't break "security flag block" into three generic words.

• Never blocks. The access-count write never waits on another writer. That had stalled the walk for 30 s under load. Both switches are installed but off (MEMSTORE_CAUSAL_V2, TRIGGER_CAUSAL_V2). The live system behaves exactly as before.

• Routing (mailtool/trigger_gate.py 1.4.0, switch TRIGGER_CAUSAL_V2):

• Case-insensitive. The automatic injection now matches regardless of case.
• Named-pair questions. It also fires on "does X cause / lead to / bring about Y" questions.
• Links first. It shows the named-pair links before the nearby chains.
• It is a machine trigger, not a prompt rule.
• Counterfactual (memory/counterfactual.py 1.1.0, live; it had no switch to begin with):
• But-for semantics. "What if X hadn't happened" is now a but-for analysis with redundant causes. An effect is lost only if every recorded supporter is X or another lost effect, and a "prevents" edge is not support.
• Honest output. Kept effects name their other supporters. The output says it is qualitative: recorded support only, no probabilities.
• It is still hidden from the Day.

Open decision for the operator. The lookup repair is an interface fix proven on exact controls: 400/400 lookup cases against 118/400 before, and "does A cause B?" 80/80 against 2/80, dev and held out. The package allows shipping such a fix on the controls alone. But the preregistered end-to-end test covered lookup and routing together and showed no gain, so it stays off until the operator says otherwise.

What this does not show.

• Real incident debugging. Workers debug with bash and file reads, and nothing here measures whether that reasoning is causal. The package's incident simulator and repair-regret track are not built.
• Whether a cleaner graph would change her trust. 61% of edges are loose readings of their source. The extractor that writes them is unchanged; a typed relation (cause / enables / chronology / dependency) is the package's P5 and not done.
• One model, k=1. Every question was asked once, of one model (deepseek-flash).

Parts that do it (added)

• memory/memstore.py 1.15.2 (causal_v2, _match_causal, _causal_endpoints, _touch_soft)
• mailtool/trigger_gate.py 1.4.0
• memory/counterfactual.py 1.1.0
• Bench ~/cognition-bench/09-causal:
• generators and oracle: scm9.py, gen9.py, gen9v2.py;
• scoring and runs: score9.py 1.2.0, run9.py 1.1.0;
• lookup and end-to-end tracks: q9_probe.py, g9_sample.py, fidelity9.py;
• tests: test_gen9.py, enhance/test_counterfactual.py;
• records: DIAGNOSIS.md, SPEC.md, PREREG-GH.txt;
• bench seat 1.18.0.

Sealed sets A/B, A2/B2 and G/GH are root-only on the bench machine.