source doc · Counterfactual reasoning

Summary / 35. Counterfactual reasoning / Document

2026-10-02

Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.

Rendering…

# Counterfactual Reasoning for an Autonomous Agent: Literature Review, Diagnosis of Piper's Implementation, and a Falsifiable Path to a Live Capability *Cognitive component #35 — "What would happen if…?"* *Prepared October 2026* --- ## 1. Executive summary **Short answer.** Neither Piper's self-description nor the reviewer's rating is quite right. - Piper's self-description is **inaccurate on the mechanism**. It claims a do-operator that "severs a node's incoming edges." If the reviewer's reading of `counterfactual.py` is correct, the code severs **outgoing** edges and calls itself "nullification." That is not Pearl's `do()`. It is also not a counterfactual in the technical sense, because no step infers the unobserved (exogenous) state of the specific case being reasoned about — the *abduction* step that defines rung-3 reasoning (Pearl 2009, §7.1; Vashishtha et al. 2025). - But the reviewer's framing overstates the error in one direction and understates the capability in another: - **Outgoing-edge deletion is a recognised graph operation** — it is the mutilated graph GX̲ used by the back-door criterion and rule 2 of the do-calculus (Pearl 1995; Pearl 2009 §3.4), and "arrow deletion" is the basis of Janzing et al.'s (2013) causal-strength measure. And for queries about **descendants of X with no conditioning on evidence**, "cut X's outgoing edges and feed children a baseline value" gives *numerically the same answer* as `do(X = baseline)`. The two diverge exactly when evidence is conditioned on, when the intervention value is not the null/baseline, or when the query concerns X or its ancestors (§5). - **The `intel-gym` hindsight-threshold replay is a genuine, if degenerate, unit-level counterfactual**: every exogenous quantity (the forecast probabilities and the resolved outcomes) is recorded, so abduction is exact and trivial, and replacing the threshold is a clean intervention on the decision rule — *provided* the threshold did not causally influence which outcomes were later observed (§13). That is a real, live, narrow counterfactual capability. - **Recommended re-rating:** *"Narrow — live in one domain (threshold replay); general counterfactual engine dormant, mis-documented, and below rung 3."* "Absent in practice" is correct for *general* counterfactual reasoning, but not for the capability as a whole. **What the literature says a real counterfactual engine needs** (§2–4): a model of *mechanisms* (structural equations or a re-executable simulator), not just edges; an intervention operator that replaces the intervened variable's mechanism; an abduction step that conditions the case-specific exogenous state on what was actually observed; and an *identifiability* gate that can say "this counterfactual is not determined by what I know" (Shpitser & Pearl 2008). LLMs alone are unreliable at the third step: recent benchmarks show large accuracy drops between interventional and counterfactual questions even for frontier models (Vashishtha et al. 2025; Chen et al. 2026). **The most important practical insight for Piper specifically:** Piper's world is mostly *software*. In software, a large class of counterfactuals is **computable exactly by replay** — re-running a deterministic pipeline on the recorded inputs with one setting changed. This is the cheapest, most reliable counterfactual engine available to any agent, and Piper already has one instance of it (`intel-gym`). The recommended Level-1 architecture generalises that pattern, adds a counterfactual-claim ledger that is resolved against reality, and turns every A/B test in the planned "presence" self-improvement loop into a free, leak-proof *prospective* counterfactual test. **Central question (answered in full in §19):** Piper can be said to have counterfactual reasoning as a *live* capability when (1) she holds mechanisms, not just edges, for the domain in question; (2) she performs abduction on the specific case; (3) she gates answers on identifiability; (4) she invokes this unprompted at defined trigger points; and (5) her logged counterfactual claims are resolved against reality and beat a no-machinery baseline. The smallest falsification test is a **12-item abduction discriminator** (§19.2): items where the interventional answer is fixed but the correct counterfactual answer flips with the factual evidence. An agent that is really doing rung-1 or rung-2 reasoning, or telling plausible stories, cannot pass it. --- ## 2. Definition of counterfactual reasoning ### 2.1 Ten things that get called "what-if" reasoning | # | Activity | Canonical question | Formal object | Pearl rung | |---|---|---|---|---| | 1 | Observational inference | What happened / what is associated? | P(Y \| X = x) | 1 (association) | | 2 | Prediction | What will probably happen? | P(Yt+1 \| history) | 1 (unless actions are involved) | | 3 | Intervention | What happens if I *force* X = x? | P(Y \| do(X = x)) | 2 | | 4 | Counterfactual | Given that I saw X = x′ and Y = y′ *in this case*, what would Y have been had X been x? | P(Yx \| X = x′, Y = y′) | 3 | | 5 | Simulation / rollouts | Run the model forward under a policy | Samples from a (learned or given) transition model | 2 if actions are set exogenously; 3 only if the rollout is anchored to the inferred noise of a specific past episode | | 6 | LLM hypothetical reasoning | "Imagine that…" | Text conditioned on a premise | None by default — it can imitate any rung, and is graded by whether it matches the right formal answer | | 7 | Ablation | Remove component C; measure the change | Usually an intervention do(C = off/mean/zero) | 2 | | 8 | Sensitivity analysis | How much does output move when input moves? | ∂Y/∂X or a sweep over inputs | 2 if the sweep is an intervention in a causal model; 1 if it is a regression slope | | 9 | Retrospective threshold analysis | What would last month's score have been at threshold τ′? | Re-evaluate a decision rule on recorded data | 3 *when* the recorded data are invariant to the threshold (no feedback); otherwise only bounds | | 10 | Alternative-plan evaluation | Is plan A better than plan B? | E[U \| do(plan)] compared across plans | 2 (prospective); 3 if asked retrospectively about an executed plan | ### 2.2 The ladder of causation Pearl's three rungs (Pearl & Mackenzie 2018; Pearl 2019) are formalised as the *Pearl Causal Hierarchy* (PCH). Shpitser and Pearl (2008) define the three layers as associational distributions over observables, distributions under external interventions, and counterfactual distributions that span multiple "parallel worlds" combining possibly conflicting observations and interventions. Bareinboim et al. (2022) prove the **Causal Hierarchy Theorem**: for almost all SCMs, the layers do not collapse — data or knowledge at a lower rung generically underdetermine the rung above. ### 2.3 Why P(Y \| X = x), P(Y \| do(X = x)) and P(Yx \| X = x′, Y = y′) are not interchangeable - **P(Y \| X = x)** averages over the sub-population in which X happened to equal x. Anything that makes X = x more likely (a confounder) leaks into the answer. - **P(Y \| do(X = x))** averages over the *whole* population after X has been set to x by a mechanism external to the system. Confounders keep their natural distribution. - **P(Yx \| X = x′, Y = y′)** is about the *same units* we observed with X = x′ and Y = y′. The evidence updates our beliefs about those units' unobserved background factors (U); then we change X and propagate. Two populations with identical interventional distributions can have different counterfactual answers — which is why rung 3 is strictly stronger. A worked numeric demonstration is in §5. The one-line version: in that model, **P(S \| do(W = 0)) = 0.15**, but for a case where we saw rain, wet ground and a slip, **P(SW=0 = 1 \| R = 1, W = 1, S = 1) = 0.526**, and for a case where we saw rain, wet ground and *no* slip, the same counterfactual is **0**. Same intervention, three different correct answers depending on the factual evidence. That evidence-dependence is the signature of counterfactual reasoning. ### 2.4 Senses of "counterfactual" that are *not* Pearl's The word is overloaded, and Piper's documentation should say which sense is meant: - **Counterfactual explanations** in ML (Wachter et al. 2017): the nearest input change that would flip a classifier's output. This is a search over inputs to a fixed function, usually with no causal model of the inputs. - **Counterfactual evaluation / off-policy evaluation** (Bottou et al. 2013; Dudík et al. 2011): estimating what a different policy *would have* earned on logged data, usually by importance reweighting. This is rung 2 (a population-level policy value), despite the name. - **Counterfactual regret** (CFR, game theory): an accounting device, not causal inference. - **Psychological counterfactual thought** (Byrne 2005; Epstude & Roese 2008): the human activity of imagining alternatives, often functional (preparing for next time) rather than accurate. --- ## 3. Pearl / SCM foundations ### 3.1 Structural causal model An SCM M = ⟨U, V, F, P(U)⟩ (Pearl 2009, def. 7.1.1; Peters et al. 2017): - **U**: exogenous (background) variables, with distribution P(U). They carry everything about the specific case that the model does not explain. - **V**: endogenous variables. - **F**: one structural equation per endogenous variable, Vi := fi(PAi, Ui). - The **causal graph** is derived from F: an edge PAi → Vi exists when fi depends on it. The graph alone is enough for rung 1 (with data) and often for rung 2 (via do-calculus identification). **Rung 3 generally needs F and P(U), or strong substitutes** — this is the formal content of "the machinery is real, the graph underneath is thin." ### 3.2 Intervention do(X = x) produces the submodel Mx: **replace** fX with the constant x, leaving every other equation untouched (Pearl 2009, def. 7.1.2). Graphically, this deletes the edges *into* X, because X no longer listens to its parents. The edges *out of* X stay, because the children still listen to X — that is how the intervention has any effect. ### 3.3 Counterfactual Yx(u) := the value of Y in Mx under background state u (Pearl 2009, def. 7.1.4). A counterfactual query conditions on evidence e from the actual world and asks about Yx. Balke and Pearl's (1994) **twin network** computes it by building two copies of the endogenous graph (actual and hypothetical) that *share the same U nodes*: evidence enters the actual copy, the intervention is applied to the hypothetical copy, and belief propagates through the shared U. ### 3.4 Identifiability A query is *identifiable* from the available information (the graph plus whichever distributions are known) if every SCM consistent with that information gives the same answer. - For rung 2, Shpitser and Pearl's ID algorithm and the do-calculus are complete. - For rung 3, the ID\* and IDC\* algorithms (Shpitser & Pearl 2007, 2008) decide whether a counterfactual is computable from experimental (rung-2) data, and Shpitser and Pearl (2008) also characterise graphically which queries can be computed by *no* method from a lower layer. Correa, Lee and Bareinboim (2021) extend this to nested counterfactuals. - Many important unit-level quantities — such as the **probability of necessity** ("would this outcome not have happened without the cause?") — are generally *not* point-identified even from perfect experimental data. They are only bounded (Tian & Pearl 2000). - Recent work studies what changes when some counterfactual data can actually be collected (Raghavan & Bareinboim 2025; the 2026 completeness and bounding follow-up, arXiv:2602.23541). This is directly relevant to §9, where Piper *does* sometimes observe both branches. **Implication for Piper:** a correct counterfactual reasoner must be able to answer "not identifiable — here are the bounds" rather than inventing a point estimate. --- ## 4. Abduction → Action → Prediction Pearl's three-step procedure (Pearl 2009, thm. 7.1.7), for evidence e, intervention X = x, and query Y: 1. **Abduction.** Compute P(U \| e): update beliefs about the background state of *this case* using everything observed. Computationally this is posterior inference — exact enumeration for small discrete models, MCMC/variational inference for continuous ones, or noise *inversion* when equations are invertible (e.g. additive noise: ui = vi − fi(pai)). 2. **Action.** Form Mx by replacing fX with X := x. 3. **Prediction.** Push P(U \| e) through Mx to get P(Yx \| e). Buesing et al. (2019) give exactly this as their counterfactual-inference subroutine inside an RL algorithm: sample noise from its posterior given the observed episode, replace the noise prior with that sample, apply the intervention, and simulate the modified model. Vashishtha et al. (2025) argue that most existing LLM counterfactual evaluations quietly skip abduction and so collapse into interventional reasoning, which overestimates model ability; when abduction is genuinely required, they report accuracy drops of roughly 25–40% from interventional to counterfactual questions for models such as o4-mini and Claude 4 Sonnet. **Two practical complications:** - *Non-uniqueness of P(U \| e) under categorical outcomes.* Different SCMs with the same interventional behaviour give different counterfactuals. Oberst and Sontag (2019) address this for discrete choices with the **Gumbel-Max SCM**, which satisfies a "counterfactual stability" condition. This is the natural choice if Piper ever needs counterfactuals over categorical events such as "which model would have been chosen" or "which tool would have been called." - *Recorded exogenous state.* When the background state is literally logged — inputs, random seeds, upstream data snapshots — abduction becomes a **lookup**, not an inference. That is the situation in deterministic software replay, and it is why replay is the gold standard for an agent like Piper. ### 4.1 Could Piper's components form this pipeline? | Stage | Required | Piper component | Plausible today? | What would need to be checked | |---|---|---|---|---| | Abduction | Posterior over the case's exogenous state given evidence | `abduction v1.0` | **Unlikely as-is.** "Abduction" in agent/LLM work usually means Peircean *inference to the best explanation* — generating hypotheses — not computing P(U \| e). Those are related, but they are not the same operation. | Does it output a *distribution over latent background variables tied to graph nodes*, or ranked natural-language hypotheses? Can its output be passed as numeric input to the graph? | | Action | Replace fX with a constant | `counterfactual v1.0` | **No** (per reviewer): it cuts outgoing edges; there is no evidence of mechanism replacement. | The function body: which adjacency list it mutates; whether X's value is set; whether evidence is conditioned before or after the cut. | | Prediction | Propagate through mechanisms | `causal-graph v1.1` | **Only if edges carry functional mechanisms.** A graph of qualitative or weighted edges supports reachability and perhaps linear propagation, but not true structural prediction. | Edge schema: is there an fi, a CPT, a sign, or a weight? Is there a propagation routine, and is it forward-only? | | Identifiability gate | Decide whether the query is answerable | none | No | — | | Live invocation | Called in normal operation | none (not in Day toolset) | No | Tool registry, call logs | **Verdict:** the three components *name* the three stages, but there is no evidence they are wired into one pipeline, and at least two stages (abduction semantics and the action operator) look mismatched to Pearl's definitions. --- ## 5. The intervention / edge-severing issue ### 5.1 What do(X = x) does, precisely - **Replaced:** the structural equation of X, fX → constant x. - **Edges modified:** *incoming* edges to X are removed, because X no longer depends on its parents. **Outgoing edges are kept.** - **Unchanged:** all other equations, and the prior P(U). - **Consequences:** descendants of X change; non-descendants (including ancestors) keep their pre-intervention distribution; evidence about X no longer carries information about X's ancestors. ### 5.2 What outgoing-edge removal is Removing X's outgoing edges is a recognised operation, but it plays a different role from intervention: 1. **A test graph, not an intervention.** GX̲ (Pearl's underbar notation: X's outgoing edges removed) is the graph in which the back-door criterion is checked. Rule 2 of the do-calculus checks a conditional independence in such a graph. The purpose is to ask, "If X had no causal effect, what association with Y would remain?" — that is, to *isolate the non-causal (confounding) paths* (Pearl 1995; Pearl 2009, §3.3–3.4). 2. **Arrow deletion as a strength measure.** Janzing et al. (2013) define the causal strength of an edge set by "corrupting" those edges — feeding the child an independent copy of the parent drawn from its marginal — and measuring the KL divergence between the original and the new joint distribution. 3. **Edge / path-specific interventions.** Blocking particular edges out of X while leaving others intact is the machinery of natural direct effects and path-specific effects (Pearl 2001; Avin, Shpitser & Pearl 2005; Shpitser & Tchetgen Tchetgen 2016). 4. **Knockout / ablation.** In biology and mechanistic interpretability, "remove X's influence on everything downstream" is *knockout* (zero-ablation) or *mean-ablation*. Formally, this is do(X = baseline) as experienced by X's children. 5. **Change-impact analysis.** On a purely qualitative graph, "cut X's outgoing edges and see what changes" reduces to computing X's descendant set. That is software change-impact analysis (Bohner & Arnold 1996) — useful, but rung 1.5 at most: structure, not quantity. ### 5.3 Worked example: Rain → WetGround → Slippery (+ a direct Rain → Slippery path) A chain alone cannot separate seeing from doing for downstream nodes, so we add a back-door path: rain can make the steps slick directly (for example, through wet leaves), independently of whether the ground under consideration is wet. **SCM** (all variables binary; U's independent): ``` U_R ~ Bern(0.3) R := U_R (rain) U_W ~ Bern(0.1) W := R ∨ U_W (wet ground; U_W = sprinkler etc.) U_S ~ Bern(0.9) U_RS ~ Bern(0.5) S := (W ∧ U_S) ∨ (R ∧ U_RS) (slippery) ``` All numbers below were computed by exact enumeration over the 16 exogenous states. | Operation | Formal | Result | Interpretation | |---|---|---|---| | Baseline | P(W), P(S) | 0.370, 0.348 | — | | **Observe** W = 1 | P(S \| W = 1) | **0.941** | Seeing wet ground also tells us it probably rained (P(R \| W=1) = 0.81), which adds the direct rain path | | **Observe** W = 0 | P(S \| W = 0); P(R \| W = 0) | **0.000**; 0.000 | Dry ground implies no rain in this model, so nothing can make it slippery | | **do(W = 1)** | P(S \| do(W = 1)) | **0.915** | Rain keeps its prior of 0.3; the gap 0.941 − 0.915 is confounding | | **do(W = 0)** | P(S \| do(W = 0)); P(R \| do(W = 0)) | **0.150**; 0.300 | Rain still happens 30% of the time and still makes things slick half of those times. Drying the ground does not stop the rain | | **Delete incoming edges only** (R → W cut, no value assigned) | W := U_W | P(W) = 0.10 | This is **not** do(W = 0). It is a *mechanism change* — a world where rain no longer wets this ground (for example, a roof was built). It is a soft/structural intervention, not an atomic one | | **Delete outgoing edges** (W → S cut; S's missing input treated as 0) | S := R ∧ U_RS | Unconditionally: P(S) = **0.150** = P(S \| do(W = 0)). Conditioning on W = 1 in the cut graph: P(S \| W = 1) = **0.405** | Forward-only, it **coincides with do(W = 0) for descendants**. Conditioned on evidence, it returns the *purely non-causal* part of the W–S association (0.405 = 0.81 × 0.5, all via rain). That is the back-door diagnostic, not an intervention | | **Nullify W** (remove node + all edges; default 0) | — | S behaves as under do(W = 0); W itself and any query about W become undefined | Knockout. Equivalent to do(W = null) for downstream variables only; loses the ability to ask questions about W or to set W to any *other* value | | **Counterfactual**, factual (R=1, W=1, S=1) | P(SW=0 \| e) | **0.526** | Abduction: since rain occurred and it was slippery, U_RS is now more likely to be 1 (0.5 → 0.526). Without wet ground, slipperiness would remain only via rain | | **Counterfactual**, factual (R=1, W=1, S=0) | P(SW=0 \| e) | **0.000** | No slip despite rain implies U_RS = 0, so the counterfactual is certain | | **Counterfactual**, factual (R=0, W=1, S=1) | P(SW=0 \| e) | **0.000** | No rain; the slip was entirely due to wet ground | | Conditional intervention | P(S \| do(W = 0), R = 1) | 0.500 | This differs from the 0.526 counterfactual: knowing the outcome adds information | **Lessons:** 1. **The incoming/outgoing distinction matters most when evidence flows backwards.** A forward-only propagator that never conditions on observations will give the same downstream numbers whether it cuts X's incoming edges and fixes X, or cuts X's outgoing edges and feeds the children a baseline. If Piper's engine is forward-only and the intervention value is always "absent," the reviewer's objection is mostly a **correctness-of-semantics and generality issue, not a wrong-number issue** — for now. 2. **It becomes a wrong-number issue when** (a) observations are conditioned on before or after the cut, (b) the intervention value is anything other than null/baseline, (c) the query is about X or its ancestors, or (d) the graph contains cycles or feedback. 3. **None of the edge operations produces the counterfactual row.** The values 0.526 / 0 / 0 require abduction over U. No amount of edge surgery on the graph alone yields them. ### 5.4 What Piper's "nullification" plausibly provides If it is implemented as described: **downstream knockout / impact analysis** — "which nodes depend on X, and (if edges carry weights) by roughly how much would they move if X's influence were removed?" That is a legitimate and useful operation, best named `knockout()` or `impact()`. It is a restricted rung-2 query, do(X = null) evaluated on descendants. It should not be called a counterfactual, and the self-description should stop claiming incoming-edge severing. --- ## 6. Counterfactual reasoning in modern AI agents The generic loop *state → alternate action → predicted trajectory → compare → choose* appears under several names, and the label matters for credit: | Literature label | What is actually done | Rung | Example | |---|---|---|---| | **Planning / MPC / tree search** | Prospective rollouts of candidate actions from the current state through a model | 2 (interventions on future actions) | MuZero (Schrittwieser et al. 2020); Dreamer (Hafner et al. 2020); RAP — LLM as world model + MCTS (Hao et al. 2023) | | **Imagination** | Learned-model rollouts used as extra input or training data | 2 | Imagination-Augmented Agents (Weber et al. 2017) | | **Counterfactual policy evaluation/search** | Infer the noise of a *real logged episode*, replay it under a different policy | **3** | CF-GPS (Buesing et al. 2019); Gumbel-Max SCM OPE (Oberst & Sontag 2019) | | **Counterfactual credit assignment** | Ask what reward would have been without an agent's action, or given hindsight information | 2–3 depending on construction | COMA (Foerster et al. 2018); Hindsight Credit Assignment (Harutyunyan et al. 2019); counterfactual credit assignment (Mesnard et al. 2021) | | **Hindsight relabelling** | Reinterpret a trajectory as if a different goal had been intended | Data augmentation — not causal inference about the world | HER (Andrychowicz et al. 2017) | | **Counterfactual data augmentation** | Recombine locally independent sub-trajectories into new, plausible ones | 2–3, under local factorisation assumptions | CoDA (Pitis et al. 2020) | | **Causal world models for LLM agents** | Learned causal representations queried by an LLM as a simulator | 2 (planning), sometimes 3 | Gkountouras et al. (2025, ICLR); Counterfactual Planning / "What-If-Not" reward (Fu et al. 2026, AAAI); Causal Cartographer (Gendron et al. 2025) | | **Verbal reflection** | LLM writes a post-hoc critique of its own failure | Not formally any rung | Reflexion (Shinn et al. 2023) | **The key distinction for crediting Piper.** Planning asks "what *will* happen if I do A?" — interventional. Counterfactual RL asks "what *would have* happened in *that* episode if I had done B?" and is distinguished by **anchoring to the inferred latent state of a specific realised episode**. Buesing et al. (2019) make the point explicitly: instead of synthesising experience from scratch, CF-GPS starts from logged real experience and models alternative outcomes under actions that were not taken. An agent that generates alternative text, or runs fresh rollouts that ignore the realised episode, is doing imagination or planning. Both are valuable, but neither earns credit for rung 3. Two further relevant threads: - **Model-based diagnosis** (Reiter 1987; de Kleer & Williams 1987): from observed misbehaviour and a component model, abduce which components failed, then test candidate diagnoses by predicting their consequences. This is the classical ancestor of the loop in §12. - **Human counterfactual simulation** (Gerstenberg et al. 2021): people judge causation by simulating what would have happened in the specific observed scene. This is cognitive-science evidence that rung-3 reasoning is anchored to the particular case, not run from a generic prior. --- ## 7. LLM counterfactual research ### 7.1 Findings - **Formal causal inference in natural language is hard for LLMs.** CLadder (Jin et al. 2023, NeurIPS) generates about 10K questions across all three rungs from known graphs, with ground truth from an oracle causal-inference engine, and finds the task highly challenging. Its CausalCoT prompting — explicitly extracting the graph and query type before computing — helps. - **Counterfactual questions are the hardest.** CounterBench (Chen et al. 2026, AAAI; 1.2K formal counterfactual questions with deliberately nonsensical variable names) finds most models performing near chance. The CoIn paradigm, which adds iterative reasoning and backtracking, raises accuracy substantially. - **Abduction is where LLMs fail.** Executable Counterfactuals (Vashishtha et al. 2025, ICLR 2026) shows that benchmarks which skip abduction overestimate ability. Making abduction mandatory (through code and math tasks) produces the 25–40% drop described in §4. RL training generalised better than supervised fine-tuning. - **Correlation ≠ causation is not robustly learned.** Corr2Cause (Jin et al. 2024, ICLR) tests inferring causal structure from correlational statements; models perform near random and fine-tuning gains do not survive paraphrase or renaming. - **Counterfactual task variants expose recitation.** Wu et al. (2024, NAACL) alter familiar tasks (e.g. base-9 arithmetic, modified chess rules) and find consistent degradation relative to the default versions. This suggests reliance on memorised procedures over abstract reasoning — the "against pretrained priors" failure mode. - **"Causal parrots."** Zečević et al. (2023) argue that LLMs may reproduce causal facts present in their training text without performing causal inference. Kıcıman et al. (2023) show strong performance on *commonsense / knowledge-based* causal tasks — consistent with the parrot view, because those tasks reward recall. - **Alternate-history and story counterfactuals.** TimeTravel (Qin et al. 2019) — minimally rewrite a story's ending after a premise change; CRASS (Frohberg & Binder 2022) — counterfactual-conditional multiple choice; IfQA (Yu et al. 2023) — open-domain QA under counterfactual presuppositions. These measure maintaining an altered-world premise, but they are not grounded in an SCM oracle. ### 7.2 Failure modes to test Piper against 1. **Prior override** — reverting to the factual or pretrained answer instead of the premised world. 2. **Abduction skipping** — returning P(Y \| do(x)) when P(Yx \| e) was asked (the 0.15-versus-0.526 error from §5). 3. **Backtracking** — changing the *causes* of the intervened variable ("if the ground hadn't been wet, it must not have rained") — i.e., treating do as see. 4. **Consistency violations** — breaking the composition axiom (if X = x was actually observed, then Yx must equal the observed Y) or the effectiveness axiom (under do(X = x), X must equal x). 5. **Name sensitivity** — accuracy changes when variable names are replaced with nonsense tokens. CounterBench and CLadder both include anti-commonsense or nonsense variants for this reason. 6. **Narrative plausibility over causal fidelity** — fluent alternate stories that do not preserve the downstream consequences of the explicit intervention. **Answer to the guiding question.** An LLM alone mostly produces plausible alternative stories, with unreliable preservation of intervention consequences once abduction is required. When it is *scaffolded* — given an explicit graph and executed computation (CausalCoT, CoIn, code execution) — accuracy improves markedly. **The design lesson for Piper: the LLM should formulate the query and the model; deterministic code should compute the answer.** --- ## 8. Existing benchmarks Ranked by usefulness to Piper. "Oracle" means the correct answer is computed from a known model rather than by human labelling. | Benchmark | Paper | Measures | Size | Deterministic answers? | Oracle? | LLM judge? | Cost | Implementation | Relevance | |---|---|---|---|---|---|---|---|---|---| | **Executable Counterfactuals** | Vashishtha et al. 2025 (arXiv:2510.01539; ICLR 2026) | Abduction + intervention + prediction over code and math; paired interventional vs counterfactual splits | Generator-based; scalable | Yes | Yes (execution) | No | Low (local generation; one LLM call per item) | Low | **Very high** — Piper is a coding agent, and the paired splits directly measure the rung gap | | **CLadder** | Jin et al. 2023 (arXiv:2312.04350; NeurIPS) | All three rungs over graphs; commonsense / anti-commonsense / nonsense stories | ~10K | Yes (yes/no) | Yes (causal-inference engine) | No | Low | Low (Hugging Face dataset) | **Very high** — the only widely used set that scores rungs separately | | **CounterBench** | Chen et al. 2025/2026 (arXiv:2502.11008; AAAI 2026) | Formal counterfactuals: basic, joint, conditional, nested; nonsense names | ~1.2K | Yes | Yes | No | Low | Low | **High** — pure rung 3 with prior-resistant naming | | **Corr2Cause** | Jin et al. 2024 (arXiv:2306.05836; ICLR) | Inferring causal relations from correlational statements (rung 1 → structure) | ~200K | Yes | Yes (graph enumeration) | No | Medium (sample a subset) | Low | Medium — tests whether Piper's causal-graph building is principled | | **Wu et al. counterfactual tasks** | Wu et al. 2024 (arXiv:2307.02477; NAACL) | Default vs altered-rule versions of familiar tasks | 11 task families | Mostly yes | Yes (programmatic) | No | Low | Medium | Medium — tests maintaining altered-world premises | | **CRASS** | Frohberg & Binder 2022 (arXiv:2112.11941; LREC) | Counterfactual conditionals, multiple choice | A few hundred items | Yes (MC) | No (human-labelled) | No | Very low | Very low | Low–medium — commonsense, not SCM-grounded | | **IfQA** | Yu et al. 2023 (arXiv:2305.14010; EMNLP) | Open-domain QA under counterfactual presuppositions | ~3.8K | Short answers (EM/F1) | No | No | Medium (retrieval) | Medium | Low–medium | | **TimeTravel** | Qin et al. 2019 (arXiv:1909.04076; EMNLP) | Counterfactual story rewriting | ~29K | No | No | Reference overlap or judge | Medium | Medium | Low — measures narrative fidelity, not causal correctness | | **Twin-world SCM generator (§10)** | This report | All rungs + identifiability + abstention | Unlimited | Yes | Yes | No | Negligible | Low (~300 lines) | **Highest** — tunable to Piper's failure modes | **Recommendation:** run CLadder (rung separation), CounterBench (rung 3 with nonsense names), and Executable Counterfactuals (abduction in code). Treat the in-house generator as the primary regression suite. Skip judge-based story benchmarks. --- ## 9. Historical-ground-truth benchmark for Piper ### 9.1 Is "both branches were eventually executed" a valid counterfactual benchmark? **Partly.** Executing branch B *later* does not reveal what B would have done *in A's world*. It reveals what B did in a *different* world — different time, different inputs, different random draws, a different state of Piper's memory. Formally, the observed B outcome is a sample relevant to P(Y \| do(B)) under drifted conditions, not the unit counterfactual YB(uA). There are two exceptions, which form the benchmark's gold tier: 1. **Paired replays on frozen inputs.** If both arms ran on the *identical* recorded inputs with controlled seeds — a paired harness run, an evaluator-selection experiment on a fixed item set — then the exogenous state really was shared. The B result **is** the unit counterfactual. This is literally a twin network executed in software. 2. **Deterministic re-evaluation.** Threshold changes, scorer changes, voting-versus-selection *aggregation* over already-collected candidate outputs. Here B can be recomputed exactly from A's logged data. ### 9.2 Threats and mitigations | Threat | Description | Mitigation | |---|---|---| | **Information leakage** | Piper's memory (the hive, consolidation summaries, belief ledger) may already contain B's result, or a gist of it | Evaluate against a **time-sliced memory snapshot** taken at tA⁻; run in a sandbox with no hive write access; scan the context for B's identifiers before the prediction | | **Temporal leakage** | Later commits, changelog lines, or docs mention the outcome | Restrict retrieval to artefacts with timestamp < tA; strip the git log after tA | | **Non-identical initial conditions** | B ran on a different day, dataset, or model version | Tier items: Gold = shared inputs and seeds; Silver = same inputs, different seeds; Bronze = different inputs. Report each tier separately | | **Stochasticity** | One B run is a single draw from a distribution | Require probabilistic predictions; where possible, re-run B k times to get an outcome distribution; score with CRPS / log score | | **Confounding** | B was chosen *because* A looked bad, or for a reason correlated with the outcome | Record the decision rationale; flag self-selected switches; prefer pre-planned A/B tests | | **Intervention fidelity** | "Branch B" as executed may differ from B as hypothesised (bug fixes bundled in, a different config) | Diff the actual configs of A and B; describe the intervention as the diff, not the label | | **Hindsight bias** | Knowing A's outcome distorts beliefs about B (Fischhoff 1975) | Use a two-condition protocol: predict B *with* and *without* A's outcome. A real counterfactual reasoner should use A's outcome through abduction — i.e., differently for different cases — not as a generic anchor | | **Small N** | Few historical pairs | Supplement with the prospective protocol below, which grows N with every A/B test | ### 9.3 Rigorous protocol **Retrospective tier (bootstrap set):** 1. Mine logs for pairs. For each: config diff, inputs, seeds, metric, outcome of A, outcome of B, tier. 2. Freeze Piper's memory snapshot at tA⁻. Provide the pre-decision context and A's outcome (condition 1) or not (condition 2). 3. Ask for a structured prediction: `{direction, point estimate, 80% interval, causal path(s) invoked, identifiability label ∈ {identified, bounded, not-identified}, abstain?}`. 4. Score (§16). Report per tier. **Prospective tier (the leak-proof version — strongly recommended):** every time the planned "presence" loop schedules an A/B test, *before* running arm B, Piper logs a prediction of B's outcome given A's realised result. This is pre-registration. It removes memory leakage and hindsight bias by construction, and every self-improvement experiment yields a benchmark item at zero marginal cost. Run B on the *same frozen item set and seeds* as A wherever possible, so that items qualify as Gold. --- ## 10. Deterministic SCM benchmark ("twin-world suite") ### 10.1 Design A generator samples small SCMs (3–7 nodes) from templates. Each item yields **three linked questions** on the same model and evidence: - L1 (observational): P(Y \| X = x, Z = z). - L2 (interventional): P(Y \| do(X = x)). - L3 (counterfactual): P(Yx \| X = x′, Y = y′, …). The answers are computed by exact enumeration over U. Each item also carries an **identifiability label**: the full SCM is hidden from Piper, and she receives only what the item specifies (sometimes only the graph plus observational data; sometimes the structural equations). The oracle runs ID / ID\* / IDC\* — for example, via the `cfid` R package (Tikka 2023) or DoWhy (Sharma & Kıcıman 2020) — or relies on identifiability-by-construction to label each query *identified*, *bounded* (with Tian–Pearl bounds), or *not identified*. ### 10.2 Templates (minimum set) | Template | Structure | What it isolates | Characteristic trap | |---|---|---|---| | Chain | A → B → C | Transmission | Over-counting via correlations | | Fork / confounding | X ← Z → Y, X → Y | L1 ≠ L2 | Answering P(Y \| X) for P(Y \| do(X)) | | Collider | X → C ← Y | Explaining away | Conditioning on C induces a spurious X–Y association | | Mediation | X → M → Y, X → Y | Direct vs indirect | Path-specific counterfactuals (natural direct effect) | | Multiple causes / over-determination | Two sufficient causes of Y | Actual causation, but-for failure | "Y would still have happened" (Halpern & Pearl 2005) | | Deterministic | Boolean equations, U observed or fully pinned down by evidence | Exact abduction | Backtracking | | Probabilistic | Bernoulli or additive-noise U | Posterior abduction | Skipping abduction (the 0.15 vs 0.526 trap) | | Temporal | Xt → Xt+1, At → Xt+1 | Rollout counterfactuals | Changing the past when intervening on the present | | Non-identifiable | Graph only, latent confounder, probability-of-necessity query | Abstention | Inventing a point estimate | | Name-scrambled twins | Same SCM with commonsense / anti-commonsense / nonsense names | Prior override | Accuracy depends on names | ### 10.3 Format Each item is a JSON object: `{graph, equations? , data_summary?, evidence, intervention, query, rung, ident_label, answer, bounds?}`. Answers are probabilities rounded to 0.01, or Booleans. **There is no LLM judge.** The generator script also emits an "evidence-flip twin" for every L3 item: identical model and intervention, different factual evidence, different correct answer (§19.2). ### 10.4 Oracle sketch ```python import itertools def enumerate_worlds(pU): # pU: {name: p(U=1)} for vals in itertools.product([0,1], repeat=len(pU)): u = dict(zip(pU, vals)); p = 1.0 for k, v in u.items(): p *= pU[k] if v else 1 - pU[k] yield u, p def solve(F, u, do={}): # F: ordered {var: lambda v,u: ...} v = {} for var, f in F.items(): v[var] = do[var] if var in do else f(v, u) return v def counterfactual(F, pU, evidence, do, query): # abduction→action→prediction num = den = 0.0 for u, p in enumerate_worlds(pU): if all(solve(F, u)[k] == x for k, x in evidence.items()): # abduction den += p num += p * query(solve(F, u, do)) # action+prediction return num / den if den else None ``` --- ## 11. Online counterfactual reasoning architecture ### 11.1 Principle: route by computability, not by phrasing Questions such as "what if I had…" reach a **counterfactual router**, which tries methods in decreasing order of reliability: 1. **Replay** — can the alternative be *re-executed* on recorded exogenous state (the same inputs, seeds, and snapshots)? Then run it. The answer is exact, and abduction is a lookup. This covers threshold changes, config changes, scorer or evaluator swaps, retrieval-policy swaps against cached corpora, and model swaps on stored prompts (stochastic — use k samples). 2. **Logged experiment** — is there a paired A/B result with matching conditions? Return it, with its tier. 3. **Model-based (SCM)** — is there a causal model with mechanisms covering the variables? Run the identifiability check. If identified, compute via abduction–action–prediction. If only bounded, return the bounds. 4. **Structured LLM reasoning** — otherwise, the LLM writes an *explicit* mini-SCM (variables, assumed mechanisms, evidence) that code then evaluates. The output is tagged `assumption-laden`, at low confidence, and must list the assumptions that would change the answer. 5. **Abstain** — "not identifiable from what I know; the cheapest experiment that would settle it is…" ### 11.2 The question types from the brief, mapped | Question | Rung | Best route | |---|---|---| | "If I make this code change, what downstream breaks?" | 2 (prospective) | Static impact analysis (descendant set — this is where `knockout()` belongs) + run tests on a branch = a real intervention | | "Would yesterday's failure still have occurred with fresh evidence retrieved?" | 3 | Replay yesterday's episode with retrieval re-run (cached fetch or live re-fetch) and the same seeds/prompt; compare | | "Would this task have succeeded with a different model?" | 3 (stochastic) | Replay the stored prompt context through model B, k samples; report a success rate with an interval | | "If I lower this threshold, which failures disappear and which appear?" | 3 (deterministic) | `intel-gym`-style replay; the output is a *set difference* of cases, not just a score | | "Was action A responsible for the improvement, or would it have happened anyway?" | 3 (probability of necessity) | Usually **not point-identified**. Use difference-in-differences against untreated metrics, a reverted run, or bounds. Abstain on a point estimate | | "What if I did nothing?" | 2 | The no-op is an intervention like any other; it needs a baseline trend model | | "Which intervention most likely changes the outcome I care about?" | 2 | Rank by expected effect under the causal graph, weighted by identifiability and cost; propose an experiment | ### 11.3 Invocation triggers Counterfactual reasoning should be a primitive invoked automatically at decision points, not a dormant tool. Triggers: - **Pre-decision** — any action tagged costly, irreversible, external-facing, or self-modifying, and any choice among more than one plan whose predicted values are within the noise of each other. - **Surprise** — an outcome outside the predicted 80% interval (from the belief ledger / forecast systems) → postmortem. - **Postmortem** — every failed task or reverted change → "which single change would most plausibly have prevented this?", answered by replay first. - **Credit** — every claimed improvement in the self-improvement loop → a necessity check ("would it have improved anyway?"). - **Uncertainty** — high predictive entropy combined with a cheap replay → run the replay rather than guess. - **Budget** — the Foundation daemon caps counterfactual compute per day; replays above a cost threshold require Bink's approval, consistent with the presence-loop policy. ### 11.4 Output record — the `CounterfactualClaim` ```yaml id: cf-2026-10-02-0017 trigger: postmortem | pre-decision | surprise | credit | ad-hoc query: "P(task_success_{model=B} | episode 8812 observed with model=A, failed)" rung: 3 route: replay | logged-AB | scm | llm-structured | abstain identifiability: identified | bounded | not-identified assumptions: [...] estimate: {direction: +, point: 0.62, interval80: [0.41, 0.80]} decision_influenced: true | false resolution: {method: replay|later-AB|none, observed: ..., resolved_at: ...} score: {...} # filled on resolution ``` Resolved claims feed (a) a calibration ledger, which Piper's metacognition can read, and (b) causal-graph updates: edges whose predicted effects keep failing are down-weighted, and edges that consistently predict well gain confidence. --- ## 12. Integration with Piper's other cognitive systems The loop proposed in the brief — **observe unexpected result → abduct causes → construct interventions → simulate → compare to evidence → update causal beliefs** — is well established under several names: - **Model-based diagnosis** (Reiter 1987; de Kleer & Williams 1987): generate candidate faults consistent with observations, then discriminate between them by predicting their consequences and choosing probing measurements. - **The hypothetico-deductive / active-learning cycle**: choose the intervention that maximally discriminates between hypotheses (expected information gain). - **Dyna** (Sutton 1990): learn a model from real experience, plan with it, correct it from new experience. - **Causal reflection** (arXiv:2508.04495, 2025): a recent agent framework that detects mismatches between predicted and observed outcomes and generates falsifiable causal hypotheses to revise the agent's model. - **CF-GPS** (Buesing et al. 2019): the RL instance, with real episodes plus a model yielding counterfactual trajectories that de-bias the model. **Connections:** | Component | Role in the counterfactual loop | |---|---| | Episodic memory | Supplies the factual episode and **recorded exogenous state** (inputs, seeds, snapshots) — the raw material for replay and abduction. *Design requirement: episodes must be logged with enough state to be replayable.* | | Semantic memory / causal graph | Supplies mechanisms (L3) or structure (L2) | | Abduction | Generates candidate causes (Peircean) **and** computes P(U \| e) (Pearlian). These should be two named functions | | Prediction / forecasting | Produces the factual predictions whose misses trigger the loop; shares the scoring machinery (Brier/CRPS already exist in Piper's forecast systems) | | Planning | Consumes rung-2 queries prospectively; the router serves both | | Imagination | An LLM-generated alternative is a *hypothesis generator* for the router, never its answer | | Uncertainty / metacognition | Identifiability labels and interval widths are first-class metacognitive signals. This connects directly to Piper's own finding that her metacognition fails at *evidence acquisition*: the router's step "cheapest experiment that would settle it" is that missing evidence-acquisition move | | Learning from failure / credit assignment | Postmortem and credit triggers | | Goal formation | "Which intervention most changes the outcome I care about?" ranks candidate improvement goals | --- ## 13. Assessment of Piper's existing implementation The ratings below rest on the self-description plus the reviewer's findings. Items marked **[verify]** cannot be settled without the evidence listed. ### 13.1 `counterfactual v1.0` - **Primary classification:** *ablation / knockout over a causal graph* — downstream impact analysis. Rung 2 at most, restricted to a single null intervention value. - **Not:** a Pearl intervention as documented (that would be incoming edges with a value set), and not a counterfactual (there is no abduction). - **Live status:** dormant — not in the Day toolset, and only executed during its build session (per reviewer). As a *capability*, therefore: **absent in practice**. - **[verify]** (1) the exact edge mutation (incoming or outgoing; which adjacency structure); (2) whether X's value is set, and to what; (3) whether any evidence conditioning occurs, and whether before or after the cut; (4) the propagation semantics (reachability only, signed or weighted linear, or functional); (5) call logs since the build session; (6) the tool registry for each session type (Dawn/Day/Dusk). ### 13.2 `causal-graph v1.1` - **Primary classification:** a *causal knowledge representation* (structure, possibly with signs or weights). It is a necessary substrate for rung 2, insufficient for rung 3. - **The self-diagnosis "the graph underneath is thin" is accurate and is the binding constraint.** Even a perfect operator yields little on a sparse graph without mechanisms. - **[verify]** node and edge counts; the provenance of edges (LLM-asserted, learned from data, or confirmed by experiment); whether edges carry mechanisms, CPTs, or effect sizes; whether any edge has ever been tested against an outcome. ### 13.3 `abduction v1.0` - **Likely classification:** *hypothesis generation* (inference to the best explanation). That is valuable for diagnosis, but it is **not** the Pearlian abduction step unless it outputs a posterior over latent background variables in a form the graph can consume. - **[verify]** its output schema; whether it has ever been chained with `causal-graph` or `counterfactual`; whether its hypotheses are scored against what was later found out. ### 13.4 `intel-gym` hindsight threshold system - **Primary classification:** *retrospective (hindsight) counterfactual via deterministic replay* — also describable as threshold sensitivity analysis. **This is a genuine unit-level counterfactual** under one condition, discussed below. - **Why it qualifies:** the exogenous state of each case (forecast probability, realised outcome) is recorded. Abduction is exact; the intervention (replace τ by τ′) acts on a mechanism, namely the decision rule; prediction is deterministic re-scoring. - **The condition — no feedback:** if τ influenced *which* outcomes were later observed or recorded, then the recorded outcomes are not invariant to τ, and the replay is biased. Examples: a trade that was not placed never resolves into P&L; a flagged item gets human review that alters its label; selection effects on which forecasts get resolved. In that case only bounds or off-policy estimates are valid. - **Strengthen it by** reporting the *set difference* of cases (which outcomes flip) rather than only the aggregate score, and by adding a feedback-check flag to each replay. - **Rung and status:** rung 3 (degenerate, fully observed), **live**. ### 13.5 Overall rating | Component | Classification | Rung reached | Live? | |---|---|---|---| | counterfactual v1.0 | Knockout / impact ablation | ≤ 2 (restricted) | No | | causal-graph v1.1 | Causal structure store | substrate | Unknown | | abduction v1.0 | Hypothesis generation (probably) | — | Unknown | | intel-gym hindsight | Deterministic replay counterfactual | 3 (narrow) | **Yes** | **Recommended rating:** *Narrow* (rather than *Absent in practice*). **Recommended action on the self-description:** replace it with an accurate one, for example: > "One live counterfactual mechanism: deterministic hindsight replay of forecast thresholds in intel-gym. A graph knockout tool (`counterfactual.py`, nullification of outgoing edges — impact analysis, not a do-operator) exists but is not in routine use. No general abduction–action–prediction pipeline yet." --- ## 14. Recommended architecture: three maturity levels ### Level 1 — Minimal (days of work) **Data structures:** `CounterfactualClaim` ledger (SQLite table, §11.4); `ReplaySpec {episode_id, inputs_ref, seed, config, metric}`. **Algorithms:** - (a) Generalise the `intel-gym` replay to any *deterministic, parameterised* pipeline: thresholds, scorers, aggregation rules, retrieval top-k against cached corpora. Add the feedback-check flag. - (b) Rename the current tool to `knockout()` and document it as impact analysis. - (c) Add a ~30-line correct `do()` (replace the node's mechanism or value; remove incoming edges; forward-propagate) next to it. - (d) Prospective A/B pre-registration hook in the presence loop. **Invocation:** postmortem and surprise triggers; every A/B test; the tool is exposed in the Day toolset. **Outputs:** claims with route, estimate, interval, and resolution. **Uncertainty:** an interval from k replays when stochastic; otherwise exact. **Failure modes:** replays that look deterministic but are not (unpinned model versions, time-dependent fetches); feedback-contaminated thresholds; ledger bloat. **Cost:** tiny for deterministic replays; k × task cost for stochastic ones (budgeted). **Tests:** T1–T3 and T7 in §15. ### Level 2 — Operational (weeks) **Data structures:** a **typed self-model SCM** over Piper's own pipeline. Nodes are controllable configs (model, provider, retrieval policy, thresholds, evaluator, voting/selection), contextual variables (task type, length, time of day, provider latency), and outcomes (success, cost, latency, score). Edges and effect sizes are learned from the experiment log plus replays, and each edge has a provenance field. **Algorithms:** - (a) Paired-evaluation harness with frozen item sets and seeds (makes future items Gold). - (b) Off-policy evaluation (IPS / doubly robust; Dudík et al. 2011) for routing decisions where logging propensities are recorded — *start recording propensities now*. - (c) Counterfactual router (§11.1) with steps 1–3 and 5. - (d) Graph updates from resolved claims. **Invocation:** all triggers in §11.3, including pre-decision for costly or irreversible actions. **Outputs:** ranked interventions plus "cheapest discriminating experiment." **Uncertainty:** bootstrap intervals; identifiability via back-door/front-door checks (DoWhy). **Failure modes:** non-stationarity (provider models change underneath); small-N overfitting of the self-model; Goodhart pressure on metrics. **Cost:** moderate — mostly the paired harness. **Tests:** T1–T5, T7, T8. ### Level 3 — Strong counterfactual reasoner (months; research-grade) **Data structures:** SCMs with explicit mechanisms and noise models — additive-noise or invertible models for continuous variables (DoWhy-GCM, Blöbaum et al. 2024, supports counterfactual samples for invertible models); Gumbel-Max mechanisms for categorical choices (Oberst & Sontag 2019); temporal unrolling. **Algorithms:** - Full abduction–action–prediction. - ID\*/IDC\* identifiability gate with Tian–Pearl bounds for probabilities of causation. - Causal discovery from interventional logs (Piper can run her own experiments — active causal learning). - LLM-to-SCM compilation for domains outside her own pipeline (the Causal-Cartographer pattern), with every LLM-asserted edge marked unverified. - Neural causal models (Xia et al. 2021, 2023) only if tabular methods prove too restrictive. **Invocation:** as in Level 2, plus planning queries routed through the same SCM. **Outputs:** posterior counterfactual distributions with identifiability certificates. **Failure modes:** model misspecification masquerading as precision; mechanism assumptions that cannot be tested; compute. **Tests:** all of T1–T9. **Recommendation:** Level 1 immediately, Level 2 next. Build Level 3 only for domains where replay is impossible and decisions are high-stakes — for example, market-facing forecasts, where actions feed back into outcomes and no replay is possible. --- ## 15. Proposed test suite | ID | Test | Rung | Oracle | Size | Purpose | |---|---|---|---|---|---| | T1 | **Ladder separation** — CLadder subset (balanced across rungs and the three story types) | 1–3 | Dataset | 300 | Detect collapse between rungs | | T2 | **Twin-world SCM suite** (§10) with all templates | 1–3 | Enumeration | 300 (100 per rung) | Main regression suite | | T3 | **Abduction discriminator** — evidence-flip twins (§19.2) | 3 | Enumeration | 12 → 60 | The falsifier | | T4 | **Identifiability / abstention** — 25% of T2 items deliberately not identifiable | 3 | ID\*/IDC\* or by construction | 80 | Calibrated abstention | | T5 | **Executable counterfactuals** — paired interventional/counterfactual code items | 2 vs 3 | Execution | 200 | Rung gap in Piper's home domain | | T6 | **Prior resistance** — CounterBench nonsense-name items + T2 name-scrambled twins | 3 | Dataset / enumeration | 200 | Recitation vs reasoning | | T7 | **Historical pairs** (§9), Gold/Silver/Bronze tiers | 3 | Logged B outcome | All available | Real-world grounding | | T8 | **Prospective A/B pre-registration** | 3 | Later-run B arm | Grows continuously | Leak-proof online validity | | T9 | **Live-use audit** — over a 14-day window: invocations per trigger event, claims resolved, decisions changed | — | Logs | 14 days | Is the capability *live*? | Each item is run in two conditions: **tool-enabled** (the router and code are available) and **LLM-only** (no tools). The difference between them measures what the machinery contributes. If the machinery contributes nothing, it is decoration. --- ## 16. Metrics and pass criteria ### 16.1 Metrics - **Rung-separated accuracy** (A1, A2, A3) and the **rung gap** A2 − A3. - **Direction accuracy** — sign of the effect. - **Magnitude** — absolute error on point estimates; for intervals, **CRPS** or the interval score, which rewards narrow intervals that contain the truth. - **Calibration** — Brier score and log score on probabilistic answers; ECE on claimed confidence; 80% interval coverage (target 0.75–0.85). - **Causal-path identification** — Jaccard similarity between the paths Piper cites and the oracle's active paths. - **Intervention correctness** — on T2 L2 items, no backtracking: ancestors of X must keep their prior distribution. - **Counterfactual consistency** — a battery of axiom checks: - *Composition:* if X = x is observed, then Yx = Y. - *Effectiveness:* under do(X = x), X = x. - *Evidence-sensitivity:* the answer changes across evidence-flip twins when it should, and stays the same when it should not. - **Abstention quality** — a payoff matrix with the oracle's identifiability label: | | Truth: identified | Truth: bounded | Truth: not identified | |---|---|---|---| | Piper answers a point estimate | Scored by CRPS / accuracy | −0.5 (overclaim) | **−1** (fabrication) | | Piper gives bounds | −0.2 if needlessly loose | +1 if valid; tightness bonus | +0.5 if valid bounds | | Piper abstains | −0.3 | 0 | **+1** | Report the **risk–coverage curve** (selective prediction): accuracy as a function of the fraction of questions answered. - **Tool lift** — tool-enabled score minus LLM-only score. - **Liveness** — trigger recall (invocations ÷ qualifying trigger events), resolution rate, and decision influence (the fraction of claims where `decision_influenced = true`, plus the outcome difference for those decisions). ### 16.2 Pass criteria for awarding "counterfactual reasoning: present" | Criterion | Threshold | |---|---| | A3 on T2 (tool-enabled) | ≥ 0.90 on identified items | | Rung gap A2 − A3 | ≤ 0.05 | | T3 abduction discriminator | ≥ 10/12, *and* both members of a pair correct in ≥ 5/6 pairs | | T4 abstention | Net payoff > 0; fabrication rate on non-identified items ≤ 10% | | T6 prior resistance | Accuracy drop from commonsense to nonsense names ≤ 5 points | | T7/T8 calibration | 80% intervals cover 70–90%; CRPS beats both baselines: "B = A" (no effect) and the LLM-only prediction | | T9 liveness | Trigger recall ≥ 0.8; ≥ 1 resolved claim per active day; ≥ 1 documented decision changed by a counterfactual claim during the window | | Self-description | Matches the code (reviewed) | Suggested rating ladder: **Absent** (fails T3 and T9) → **Narrow** (one live domain, e.g. replay) → **Operational** (T2/T3/T4 pass with tools; T9 passes) → **Strong** (all pass, including T7/T8 calibration over at least 30 resolved claims). --- ## 17. Recommended implementation sequence 1. **Fix the documentation** (hour): rename `counterfactual.py` → `knockout.py` (impact analysis); correct the self-description (§13.5). 2. **Correct `do()` operator** (hours) next to `knockout()`, with unit tests reproducing the §5.3 table exactly. 3. **Twin-world oracle + T2/T3/T4 generator** (a day): the §10.4 sketch plus templates. Run tool-enabled vs LLM-only to get a baseline. 4. **`CounterfactualClaim` ledger** and the replay router, step 1 (one to two days); generalise `intel-gym`'s replay; add the feedback-check flag. 5. **Expose in the Day toolset with triggers** (postmortem, surprise) (a day). Start T9 logging immediately — liveness is measured from this point on. 6. **Prospective A/B pre-registration** in the presence loop (a day) — T8 begins accumulating. 7. **Mine historical pairs** for T7 with time-sliced memory snapshots (days). 8. **Run external benchmarks** T1, T5, T6 (a day each; low API cost). 9. **Level 2:** paired harness with frozen item sets; propensity logging; self-model SCM; DoWhy identifiability checks (weeks). 10. **Level 3**, only where replay is impossible and stakes are high (months). The ordering is deliberate: steps 1–5 move the rating from "Absent" to "Narrow/Operational" cheaply, and create the measurement apparatus *before* any heavier engineering. --- ## 18. Open research questions 1. **Counterfactuals over LLM calls.** What is the right exogenous-noise model for an LLM's own outputs (sampling noise vs. model-version drift vs. provider-side changes)? Gumbel-Max gives a principled answer for token sampling at fixed weights, but provider models change silently. 2. **Self-model identifiability.** When Piper changes her own code based on counterfactual conclusions, the system generating the data changes. How should the self-model handle a moving target? (Related to performative prediction.) 3. **Counterfactual realizability in agent logs.** When an agent *can* run both branches on shared inputs, more rung-3 quantities become identifiable. What is the optimal experiment-design policy for an agent that can cheaply replay? (Raghavan & Bareinboim 2025; arXiv:2602.23541.) 4. **Abduction transfer.** Does RL-trained abduction in code (Vashishtha et al. 2025) transfer to operational postmortems? 5. **LLM-asserted edges.** How should edges proposed by an LLM be weighted before any experiment confirms them, and how quickly should they decay if never tested? 6. **Actual causation for postmortems.** "What caused this failure?" is an actual-causation question (Halpern & Pearl 2005; Halpern 2016), not just a counterfactual dependence question — over-determination is common in software failures. Which definition should Piper's postmortems use? 7. **Liveness metrics.** There is little literature on measuring whether an agent capability is *used*, as opposed to *possessed*. T9 is a proposal, not an established standard. --- ## 19. Central question ### 19.1 What machinery would Piper need? Piper can legitimately be said to possess counterfactual reasoning **as a live cognitive capability** when all six of the following hold. Each item is necessary; together they are sufficient for a credible claim. 1. **Mechanisms, not just edges.** For the domain in question there is something that maps parent values plus background state to child values. That can be structural equations, or a re-executable pipeline. In Piper's software world, *the replayable pipeline itself is the structural model* — which is why replay is the right foundation. 2. **A correct intervention operator.** Replace the target's mechanism with the set value (cutting incoming dependence), keep its outgoing influence, and leave ancestors untouched. Distinct from `knockout()`. 3. **Abduction on the specific case.** Either recorded exogenous state (inputs, seeds, snapshots) used by replay, or posterior inference P(U \| e) when the state was not recorded. Without this step the system is doing rung 2, however the output is phrased. 4. **An identifiability gate.** The ability to return *identified / bounded / not identified*, and to propose the cheapest experiment that would settle the question. 5. **Live invocation.** Automatic triggers in the operating loop (postmortem, surprise, pre-decision for costly actions, credit checks), exposed in the Day toolset, under a compute budget. 6. **Resolution and learning.** Every claim is logged, later resolved against reality where possible, scored for calibration, and fed back into the causal model and into metacognition. Today Piper has (1)–(3) **only inside `intel-gym`'s threshold replay**. She has (6) partially, through the forecast scorers, and none of (4)–(5) for general counterfactuals. Hence: *Narrow*. ### 19.2 The smallest empirical test that could falsify the claim **The 12-item Abduction Discriminator.** - Build **6 evidence-flip pairs** from small SCMs (the §5.3 model is pair 1). Within a pair, the graph, the mechanisms (given to Piper explicitly), the intervention, and the query are identical. Only the *factual evidence* differs, and the correct counterfactual answers differ — for example, P(SW=0) = 0.526 vs 0.000. Choose every pair so that: - (a) the *interventional* answer P(Y \| do(x)) is the same for both members and differs from both counterfactual answers (it is 0.15 in pair 1); - (b) the commonsense-plausible answer is wrong for at least one member; - (c) variable names are nonsense tokens. - Ask Piper each of the 12 questions in the tool-enabled condition, in normal operation (no special prompting beyond the question). - **The claim is falsified if any one of these holds:** - (i) fewer than 10/12 answers are within ±0.05 of the oracle; - (ii) in more than 1 of the 6 pairs, Piper gives the same answer to both members — insensitivity to evidence, the signature of rung 2 or storytelling; - (iii) more than 2 answers equal the interventional value. - Under random guessing on a binarised version, 10/12 correct has p ≈ 0.02. An agent doing only rung-2 reasoning scores near 0 by construction, because criterion (a) makes the interventional answer wrong for every item. **Plus the minimal liveness check (so that "live" is also falsifiable):** over the next 14 days of normal operation, count qualifying trigger events (failed tasks, reverted changes, A/B tests, out-of-interval surprises). If Piper invokes counterfactual reasoning on fewer than 80% of them **unprompted**, the claim of a *live* capability is falsified, even if she passes the 12 items. Passing the first test shows Piper *can* reason counterfactually. Passing the second shows she *does*. The capability claim requires both. --- ## References **Foundations** - Avin, C., Shpitser, I., & Pearl, J. (2005). Identifiability of path-specific effects. *IJCAI 2005*. - Balke, A., & Pearl, J. (1994). Counterfactual probabilities: Computational methods, bounds and applications. *UAI 1994*. arXiv:1302.6784. - Bareinboim, E., Correa, J. D., Ibeling, D., & Icard, T. (2022). On Pearl's hierarchy and the foundations of causal inference. In *Probabilistic and Causal Inference: The Works of Judea Pearl*. ACM. doi:10.1145/3501714.3501743. - Correa, J. D., Lee, S., & Bareinboim, E. (2021). Nested counterfactual identification from arbitrary surrogate experiments. *NeurIPS 2021*. arXiv:2107.03190. - Halpern, J. Y., & Pearl, J. (2005). Causes and explanations: A structural-model approach. Part I: Causes. *British Journal for the Philosophy of Science*, 56(4), 843–887. - Halpern, J. Y. (2016). *Actual Causality*. MIT Press. - Janzing, D., Balduzzi, D., Grosse-Wentrup, M., & Schölkopf, B. (2013). Quantifying causal influences. *Annals of Statistics*, 41(5), 2324–2358. doi:10.1214/13-AOS1145. arXiv:1203.6502. - Pearl, J. (1995). Causal diagrams for empirical research. *Biometrika*, 82(4), 669–688. - Pearl, J. (2001). Direct and indirect effects. *UAI 2001*. arXiv:1301.2300. - Pearl, J. (2009). *Causality: Models, Reasoning, and Inference* (2nd ed.). Cambridge University Press. - Pearl, J. (2019). The seven tools of causal inference, with reflections on machine learning. *Communications of the ACM*, 62(3), 54–60. doi:10.1145/3241036. - Pearl, J., & Mackenzie, D. (2018). *The Book of Why*. Basic Books. - Peters, J., Janzing, D., & Schölkopf, B. (2017). *Elements of Causal Inference*. MIT Press (open access). - Raghavan, A., & Bareinboim, E. (2025). Counterfactual realizability. *ICLR 2025*. See also: Causal identification from counterfactual data: completeness and bounding results, arXiv:2602.23541 (2026). - Shpitser, I., & Pearl, J. (2007). What counterfactuals can be tested. *UAI 2007*. arXiv:1206.5294. - Shpitser, I., & Pearl, J. (2008). Complete identification methods for the causal hierarchy. *JMLR*, 9, 1941–1979. https://jmlr.org/papers/v9/shpitser08a.html - Shpitser, I., & Tchetgen Tchetgen, E. J. (2016). Causal inference with a graphical hierarchy of interventions. *Annals of Statistics*, 44(6), 2433–2466. - Tian, J., & Pearl, J. (2000). Probabilities of causation: Bounds and identification. *Annals of Mathematics and Artificial Intelligence*, 28, 287–313. **Agents, RL, and world models** - Andrychowicz, M., et al. (2017). Hindsight experience replay. *NeurIPS 2017*. arXiv:1707.01495. - Bottou, L., et al. (2013). Counterfactual reasoning and learning systems: The example of computational advertising. *JMLR*, 14, 3207–3260. arXiv:1209.2355. - Buesing, L., Weber, T., Zwols, Y., et al. (2019). Woulda, coulda, shoulda: Counterfactually-guided policy search. *ICLR 2019*. arXiv:1811.06272. - Dudík, M., Langford, J., & Li, L. (2011). Doubly robust policy evaluation and learning. *ICML 2011*. arXiv:1103.4601. - Foerster, J., et al. (2018). Counterfactual multi-agent policy gradients. *AAAI 2018*. arXiv:1705.08926. - Fu, J., Ding, L., Wei, Q., & Guo, Y. (2026). Counterfactual planning for generalizable agents' actions. *AAAI 2026*. - Gendron, G., Rožanec, J. M., Witbrock, M., & Dobbie, G. (2025). Causal Cartographer: From mapping to reasoning over counterfactual worlds. arXiv:2505.14396. - Gkountouras, J., Lindemann, M., Lippe, P., Gavves, E., & Titov, I. (2025). Language agents meet causality — Bridging LLMs and causal world models. *ICLR 2025*. arXiv:2410.19923. - Hafner, D., et al. (2020). Dream to control: Learning behaviors by latent imagination. *ICLR 2020*. arXiv:1912.01603. - Hao, S., et al. (2023). Reasoning with language model is planning with world model. *EMNLP 2023*. arXiv:2305.14992. - Harutyunyan, A., et al. (2019). Hindsight credit assignment. *NeurIPS 2019*. arXiv:1912.02503. - Mesnard, T., et al. (2021). Counterfactual credit assignment in model-free reinforcement learning. *ICML 2021*. arXiv:2011.09464. - Oberst, M., & Sontag, D. (2019). Counterfactual off-policy evaluation with Gumbel-Max structural causal models. *ICML 2019*, PMLR 97:4881–4890. arXiv:1905.05824. - Pitis, S., Creager, E., & Garg, A. (2020). Counterfactual data augmentation using locally factored dynamics. *NeurIPS 2020*. arXiv:2007.02863. - Schrittwieser, J., et al. (2020). Mastering Atari, Go, chess and shogi by planning with a learned model. *Nature*, 588, 604–609. arXiv:1911.08265. - Shinn, N., et al. (2023). Reflexion: Language agents with verbal reinforcement learning. *NeurIPS 2023*. arXiv:2303.11366. - Sutton, R. S. (1990). Integrated architectures for learning, planning, and reacting based on approximating dynamic programming (Dyna). *ICML 1990*. - Weber, T., et al. (2017). Imagination-augmented agents for deep reinforcement learning. *NeurIPS 2017*. arXiv:1707.06203. - Xia, K., Lee, K.-Z., Bengio, Y., & Bareinboim, E. (2021). The causal-neural connection. *NeurIPS 2021*. arXiv:2107.00793. - Xia, K., Pan, Y., & Bareinboim, E. (2023). Neural causal models for counterfactual identification and estimation. *ICLR 2023*. arXiv:2210.00035. - Causal reflection framework (2025). arXiv:2508.04495 (NeurIPS 2025 Workshop on Efficient Reasoning). **LLM causal and counterfactual reasoning** - Chen, Y., et al. (2025/2026). CounterBench: Evaluating and improving counterfactual reasoning in large language models. *AAAI 2026*. arXiv:2502.11008. Data: https://huggingface.co/datasets/CounterBench/CounterBench - Frohberg, J., & Binder, F. (2022). CRASS: A novel data set and benchmark to test counterfactual reasoning of large language models. *LREC 2022*. arXiv:2112.11941. - Jin, Z., Chen, Y., Leeb, F., et al. (2023). CLadder: Assessing causal reasoning in language models. *NeurIPS 2023*. arXiv:2312.04350. Data: https://huggingface.co/datasets/causalNLP/cladder - Jin, Z., et al. (2024). Can large language models infer causation from correlation? *ICLR 2024*. arXiv:2306.05836. - Kıcıman, E., Ness, R., Sharma, A., & Tan, C. (2023). Causal reasoning and large language models: Opening a new frontier for causality. arXiv:2305.00050. - Qin, L., et al. (2019). Counterfactual story reasoning and generation (TimeTravel). *EMNLP 2019*. arXiv:1909.04076. - Vashishtha, A., et al. (2025). Executable counterfactuals: Improving LLMs' causal reasoning through code. arXiv:2510.01539 (ICLR 2026). - Wu, Z., et al. (2024). Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. *NAACL 2024*. arXiv:2307.02477. - Yu, W., et al. (2023). IfQA: A dataset for open-domain question answering under counterfactual presuppositions. *EMNLP 2023*. arXiv:2305.14010. - Zečević, M., Willig, M., Dhami, D. S., & Kersting, K. (2023). Causal parrots: Large language models may talk causality but are not causal. *TMLR*. arXiv:2308.13067. **Diagnosis, cognition, tooling** - Blöbaum, P., et al. (2024). DoWhy-GCM: An extension of DoWhy for causal inference in graphical causal models. *JMLR*, 25. arXiv:2206.06821. - Bohner, S., & Arnold, R. (1996). *Software Change Impact Analysis*. IEEE Computer Society Press. - Byrne, R. M. J. (2005). *The Rational Imagination: How People Create Alternatives to Reality*. MIT Press. - de Kleer, J., & Williams, B. C. (1987). Diagnosing multiple faults. *Artificial Intelligence*, 32(1), 97–130. - Epstude, K., & Roese, N. J. (2008). The functional theory of counterfactual thinking. *Personality and Social Psychology Review*, 12(2), 168–192. - Fischhoff, B. (1975). Hindsight ≠ foresight: The effect of outcome knowledge on judgment under uncertainty. *JEP: Human Perception and Performance*, 1(3), 288–299. - Gerstenberg, T., Goodman, N. D., Lagnado, D. A., & Tenenbaum, J. B. (2021). A counterfactual simulation model of causal judgments for physical events. *Psychological Review*, 128(5), 936–975. - Reiter, R. (1987). A theory of diagnosis from first principles. *Artificial Intelligence*, 32(1), 57–95. - Sharma, A., & Kıcıman, E. (2020). DoWhy: An end-to-end library for causal inference. arXiv:2011.04216. - Tikka, S. (2023). Identifying counterfactual queries with the R package cfid. *The R Journal*. CRAN: https://cran.r-project.org/package=cfid - Wachter, S., Mittelstadt, B., & Russell, C. (2017). Counterfactual explanations without opening the black box. *Harvard Journal of Law & Technology*, 31(2). arXiv:1711.00399. *Verification note: arXiv IDs for CLadder, CounterBench, Executable Counterfactuals, CF-GPS, Oberst & Sontag, Janzing et al., Shpitser & Pearl (2008), Causal Cartographer and Gkountouras et al. were checked against primary listings while preparing this report; other identifiers are given from the literature and should be spot-checked before external citation.*