Summary / 35. Counterfactual reasoning / Document
2026-10-02
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 35 — Counterfactual Reasoning ## Test and Upgrade Plan **Capability:** “What would happen if…?” **Current rating:** **NARROW** **Baseline:** _not yet scored_ **Live use:** `intel-gym` threshold replay only, pending feedback check **Target:** Operational counterfactual reasoning used automatically in decisions, postmortems, and credit assignment --- ## 1. Working definition Piper should receive credit for **counterfactual reasoning** only when she answers: > Given what actually happened in this specific case, what would the outcome have been if one relevant cause or action had been different while the rest of that case’s background state remained the same? This is different from: - **Observation:** “What happened?” - **Prediction:** “What will probably happen?” - **Intervention/planning:** “What happens if I do X?” - **Simulation:** “Run an alternate trajectory.” - **Ablation/impact:** “What depends on X if I remove its influence?” - **Sensitivity analysis:** “How does the score move if I vary X?” - **LLM hypothetical text:** “Imagine X were different.” The operational signature of true counterfactual reasoning is **case anchoring**: factual evidence about the realised episode changes the answer because Piper preserves the same episode-specific background state across the factual and alternate worlds. --- ## 2. Current diagnosis ### 2.1 `counterfactual v1.0` **Current classification:** knockout / downstream impact analysis, not a general Pearl-style counterfactual engine. If the reviewer’s code reading is correct, it removes or nullifies **outgoing influence** from a node. That can be useful for impact analysis and may coincide numerically with `do(X=baseline)` for simple forward-only descendant queries, but it is not the general intervention operation Piper currently claims. **Action:** rename/document this operation as `knockout()` or `impact()` unless source inspection proves otherwise. ### 2.2 `causal-graph v1.1` **Current classification:** causal structure store / supporting substrate. A graph of edges alone is insufficient for general counterfactual reasoning. Piper needs mechanisms: structural equations, executable functions, conditional tables, or a replayable software pipeline that maps inputs/background state to outcomes. ### 2.3 `abduction v1.0` **Status:** verify semantics. The critical question is whether this module performs **Pearlian abduction** — recovering or conditioning on case-specific latent/background state — or merely generates plausible causal explanations in natural language. These are useful but different capabilities and should have different names if both exist. ### 2.4 `intel-gym` hindsight threshold replay **Current classification:** narrow, live counterfactual via deterministic replay. This qualifies when: 1. the original forecasts/inputs and realised outcomes are recorded; 2. replacing the threshold leaves those recorded outcomes invariant; 3. the alternate threshold can be re-evaluated on the exact same cases. If the threshold changed which cases were observed, acted upon, reviewed, or resolved, the replay is feedback-contaminated and should not be treated as exact. ### Current capability statement > Piper has one live counterfactual mechanism: deterministic hindsight replay of forecast thresholds in `intel-gym`. A separate graph knockout/nullification tool exists but is not a general do-operator and is not in routine use. Piper does not yet have a general Abduction → Action → Prediction pipeline. --- ## 3. What Piper must have before this capability is “Operational” All six conditions should be present. ### C1 — Mechanisms, not just edges For the relevant domain Piper must possess something executable that maps causes and background state to effects. For Piper’s own software, the preferred mechanism is often the **actual replayable pipeline**, not a learned abstract model. ### C2 — Correct intervention A general `do(X=x)` operation must replace the mechanism that normally determines X, fix X to x, and preserve X’s downstream influence. `knockout(X)` must remain a separate operation. ### C3 — Case-specific abduction The alternate world must reuse the background state of the factual case. Preferred order: 1. recorded inputs/seeds/snapshots → exact lookup; 2. inferred residual/noise state → posterior over latent variables; 3. explicit abstention if neither is available. ### C4 — Identifiability gate Every answer must carry one of: - `identified` - `bounded` - `not_identified` When not identified, Piper should return the cheapest experiment/replay that would reduce the uncertainty rather than inventing a point answer. ### C5 — Live invocation Counterfactual reasoning must be automatically considered at defined trigger points rather than only when the operator asks for it. ### C6 — Resolution and learning Counterfactual claims must be logged, later compared with replay or real outcomes when available, scored, and used to update Piper’s self-model and calibration. --- # 4. Upgrade architecture ## 4.1 Route by computability, not by wording Any counterfactual question enters one router: ```text counterfactual query | +-- 1. Exact replay possible? ------------> replay | +-- 2. Matching A/B result exists? ------> logged paired result | +-- 3. SCM/mechanism available? ----------> identifiability check | | | +-- identified -> compute | +-- bounded ----> bounds | +-- no ---------> next route | +-- 4. Build explicit assumed mini-SCM? ---> compute, mark assumption-laden | +-- 5. Otherwise --------------------------> abstain + propose experiment ``` The LLM should primarily **formulate the query, identify variables, and propose mechanisms**. Deterministic code should compute the answer whenever possible. --- ## 4.2 Level 1 — Build now ### A. Preserve `knockout()` Do not throw away the current nullification machinery if it is useful. Rename it and give it honest semantics: ```text knockout(X) = remove/suppress X's downstream influence for impact analysis ``` ### B. Add a true `do()` operation Minimum behaviour: ```text do(X=x): replace X's normal mechanism with X := x remove X's incoming causal dependence preserve outgoing dependence from X forward-evaluate descendants ``` Unit-test it independently from the LLM. ### C. Generalise replay Turn the `intel-gym` pattern into a reusable counterfactual executor for: - thresholds; - scorers; - evaluator swaps; - voting vs selection; - retrieval `top-k`; - cached retrieval-policy changes; - configuration flags; - model/provider swaps on stored prompts; - algorithm versions on frozen datasets. For stochastic components, replay **k** times and return a distribution rather than pretending there is one exact alternate outcome. ### D. Add `CounterfactualClaim` Recommended schema: ```yaml id: cf-... trigger: pre-decision | postmortem | surprise | credit | ad-hoc factual_episode: ... query: ... rung: 2 | 3 route: replay | logged_ab | scm | llm_structured | abstain intervention: ... evidence_used: ... identifiability: identified | bounded | not_identified assumptions: [] estimate: direction: + | - | 0 | unknown point: null interval80: null causal_paths: [] decision_influenced: false resolution: method: replay | later_ab | observed | none observed: null resolved_at: null score: null ``` ### E. Add automatic triggers Invoke the router on: - failed or reverted changes; - outcomes outside Piper’s forecast interval; - A/B tests; - claimed self-improvements; - costly or hard-to-reverse decisions; - competing plans with similar predicted utility; - high uncertainty when a cheap replay is available. --- # 5. Test plan ## T0 — Source-code audit Before changing behaviour, answer these from code and logs: 1. Does `counterfactual.py` mutate incoming or outgoing adjacency? 2. Does it assign an intervention value to X? 3. Does it condition on factual evidence? 4. Does it propagate through actual mechanisms, weights, or only reachability? 5. Is `abduction v1.0` Pearlian latent-state inference or natural-language hypothesis generation? 6. Are any of these modules chained together today? 7. Which session/toolsets expose each tool? 8. How often has each been called outside its build session? **Output:** one-page factual audit. Do not infer from component names. --- ## T1 — 12-item Abduction Discriminator — PRIMARY FALSIFIER Build **6 evidence-flip pairs**. Within each pair, keep identical: - graph; - mechanisms; - intervention; - query. Change only the **factual evidence** so the correct counterfactual answer changes. Design each pair so that: 1. the ordinary interventional answer is identical for both members; 2. that interventional answer is wrong for the counterfactual query; 3. at least one commonsense/plausible answer is wrong; 4. variable names are meaningless tokens to prevent world-knowledge shortcuts. ### Pass - at least **10/12** answers within ±0.05 of the oracle; - both members correct in at least **5/6 pairs**; - no more than **2/12** answers collapse to the interventional value. ### Fail interpretation - same answer for evidence-flip twins → abduction is being skipped; - interventional value repeatedly returned → Rung 2 masquerading as Rung 3; - fluent qualitative answer but wrong number → narrative simulation, not counterfactual computation. This is the smallest serious test of whether Piper **can** perform counterfactual reasoning. --- ## T2 — Twin-world SCM regression suite Generate small 3–7 node SCMs and test three linked questions on each world: - **L1 observation** - **L2 intervention** - **L3 counterfactual** Minimum templates: - chain; - fork/confounding; - collider; - mediation; - multiple/overdetermined causes; - deterministic; - probabilistic; - temporal; - non-identifiable; - name-scrambled twins. Recommended initial size: **300 items — 100 per rung**. Oracle answers should be generated by executable enumeration, not an LLM judge. Run every item twice: 1. **LLM-only** 2. **tool-enabled** The difference is **tool lift**. If the counterfactual machinery produces no lift, it is decorative. --- ## T3 — Identifiability / abstention Make roughly 25% of L3 items intentionally underdetermined. Piper must distinguish: ```text identified bounded not_identified ``` Scoring principle: - correct point estimate when identified: reward; - valid bounds when only bounded: reward; - abstention when not identifiable: reward; - fabricated point estimate when not identifiable: strongest penalty. **Operational target:** fabrication rate on non-identifiable items ≤ **10%**. --- ## T4 — Historical replay benchmark Use Piper’s own experiments. ### Gold Both arms run on the **same frozen inputs and seeds**. The alternate arm is a true executable twin world. Examples: - voting vs selection on the same candidate outputs; - scorer A vs scorer B on the same artifacts; - threshold A vs B on the same predictions; - retrieval policy A vs B against a frozen corpus; - model A vs B on the identical stored prompt/context. ### Silver Same inputs, different stochastic seed. ### Bronze Similar experiment at a different time/state. Useful evidence, but not a unit-level oracle. Report tiers separately. Do not mix Bronze items into Gold accuracy. --- ## T5 — Prospective A/B pre-registration This should become Piper’s strongest real-world test. Whenever Piper schedules an A/B experiment: ```text 1. Run A. 2. Freeze A's factual state and outcome. 3. BEFORE running B, log Piper's prediction for B. 4. Run B on the same frozen inputs/seeds when possible. 5. Resolve and score the claim. ``` Every self-improvement experiment now produces a leak-proof counterfactual training/evaluation item almost for free. --- ## T6 — Liveness audit Capability possession and capability use are separate. Over a **14-day normal-operation window**, count: - qualifying trigger events; - counterfactual invocations; - resolved claims; - claims that changed a decision. ### Pass - trigger recall ≥ **0.80**; - at least one resolved claim per active day; - at least one documented case where a counterfactual result changed an actual decision during the window. If Piper passes T1 but fails T6, she **can** reason counterfactually but does not yet possess it as a **live cognitive primitive**. --- # 6. Scorecard Record these separately. Do not compress them into one score until enough data exist. | Metric | Target for “Operational” | |---|---:| | L3 accuracy on identified T2 items | ≥ 90% | | L2 − L3 accuracy gap | ≤ 5 percentage points | | T1 Abduction Discriminator | ≥ 10/12 | | Evidence-flip pairs both correct | ≥ 5/6 | | Non-identifiable fabrication rate | ≤ 10% | | Nonsense-name accuracy drop | ≤ 5 points | | Trigger recall | ≥ 80% | | Tool lift vs LLM-only | > 0 and statistically/operationally meaningful | | 80% interval empirical coverage | 70–90% | For stochastic real-world predictions, also score Brier/CRPS or log score rather than only correct/incorrect. --- # 7. Rating ladder ### ABSENT Fails the abduction discriminator and has no live invocation. ### NARROW — current best estimate At least one genuine domain-specific mechanism exists and is used — currently `intel-gym` replay — but no general pipeline exists. ### OPERATIONAL Piper passes T1/T2/T3 with tools, has measurable tool lift, and passes the live-use audit. ### STRONG Operational criteria plus: - broad historical/prospective calibration; - at least ~30 resolved real counterfactual claims; - robust nonsense-name/prior-resistance results; - mechanism-rich SCMs for domains where replay is impossible; - principled bounding/identifiability; - learning updates based on resolved claims. --- # 8. Implementation order ## Phase 0 — Establish truth 1. Audit existing code and call logs. 2. Correct Piper’s self-description. 3. Separate `knockout()` from `do()` semantically and in code. ## Phase 1 — Build measurement before intelligence 4. Implement the 12-item Abduction Discriminator. 5. Implement the twin-world oracle/generator. 6. Record LLM-only baseline. 7. Record current Piper/tool baseline. Do not optimise the engine before the baseline is frozen. ## Phase 2 — Exploit Piper’s strongest advantage: replay 8. Generalise `intel-gym` replay. 9. Add replay reproducibility fields: input snapshot, code/model version, seed, provider/model identifier, config and metric. 10. Add feedback-contamination detection. 11. Add the `CounterfactualClaim` ledger. ## Phase 3 — Make it live 12. Expose the router in the normal Day toolset. 13. Add postmortem and surprise triggers first. 14. Add prospective A/B pre-registration. 15. Start the 14-day liveness audit. ## Phase 4 — Operational causal self-model 16. Build a typed SCM of Piper’s own pipeline. 17. Record edge provenance and confidence. 18. Start logging routing propensities for later off-policy evaluation. 19. Add identifiability/bounds machinery. 20. Update causal beliefs from resolved claims. ## Phase 5 — Research-grade machinery only when necessary Use full probabilistic SCMs, learned noise models, ID*/IDC*, or neural causal models only in domains where exact replay is impossible and the decision value justifies the complexity. --- # 9. Design rules 1. **Replay beats inference.** If Piper can execute the alternate world on the same state, do that before building a model of it. 2. **Do not let the LLM be the oracle.** Use execution, enumeration, logged paired arms, or mathematical solvers for scoring. 3. **Separate planning from counterfactual reasoning.** “What will happen if I do B?” and “What would have happened in episode 8812 if I had done B?” are different operations. 4. **Separate Peircean and Pearlian abduction.** Hypothesis generation and latent-state inference deserve different interfaces. 5. **Abstention is a capability.** “Not identifiable” is often the correct answer. 6. **Preserve the current knockout tool.** It is useful; it is simply misnamed if described as a general counterfactual engine. 7. **Make every prediction resolvable where possible.** Unresolved what-if narratives teach Piper nothing. 8. **Freeze the measurement suite.** Piper must not be able to rewrite the evaluator that determines whether she improved. --- # 10. First experiment to run Do this before a larger rebuild: ### Experiment CF-001 — Abduction discrimination - Create 6 tiny deterministic/probabilistic SCMs. - Produce two evidence variants for each. - Same intervention and query within each pair. - Ensure the correct answer changes with evidence while the interventional answer does not. - Scramble variable names. - Run current Piper LLM-only. - Run current Piper with all currently available causal/counterfactual tools. - Freeze results. - Only then modify the architecture. **Success criterion:** ≥10/12 and ≥5/6 complete evidence-flip pairs. This experiment tells us immediately whether the present system is doing **abduction**, which is the cleanest boundary between ordinary what-if computation and true case-specific counterfactual reasoning. --- # 11. Upgrade objective The goal is not to make Piper better at saying: > “If I had done B, perhaps Y would have happened.” The target is: > “In episode E, evidence e implies background state U/posterior P(U|e). Under intervention do(B), preserving that background state, the executable model predicts Y with distribution D. The query is identified/bounded/not-identifiable. Here is the result, here are the assumptions, and here is how we will later resolve the claim.” When Piper can do that reliably **and invokes it herself at the right moments**, component 35 should be marked **Operational**.