Summary / 36. Recursive reasoning / Document
2026-10-02
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Recursive Reasoning in Autonomous AI Agents ### A construct review and test plan for component 36 of the Piper evaluation framework *Prepared October 2026. Evidence labels used throughout:* **[E]** *experimentally demonstrated finding;* **[T]** *theoretical or formal proposal;* **[S]** *this paper's synthesis;* **[H]** *hypothesis that still needs testing. Where I could not verify a detail of a source (exact numbers, page ranges), I say so rather than guess.* --- ## Executive Summary **The reviewer is substantially right, but for a slightly different reason than stated.** Across cognitive science, game theory, epistemic logic and multi-agent AI, "recursive reasoning" has a stable core meaning: *a reasoner represents another reasoning process (another agent, or itself as an object) and can embed such representations inside each other*, e.g. "A believes that B believes that A believes X," or "my best response depends on your model of my best response." Piper's current description describes **channels** (she can ask Dusk, she can ask the operator) and a **self-improvement loop**. Channels are not modelling; asking someone what they think is the opposite of predicting what they think. The RSI loop is recursive in a computational sense (a process that modifies a process), not in the nested-belief sense. Neither demonstrates the capability. **[S]** **However, Piper's institutional machinery is unusually valuable as a *test harness*.** Most self-prediction and agent-modelling research struggles to obtain ground truth for "what would this other reasoner actually say?". Piper can predict Dusk's answer, commit the prediction, then call `ask_dusk` and score herself. The same holds for fresh copies, delegates, and (sparingly) the operator. Her channels convert recursive reasoning from an unfalsifiable self-description into a measurable prediction task. **[S]** **Key empirical facts from the literature:** 1. Human higher-order mentalizing degrades with depth; adults are reliable to roughly four or five orders of intentionality in story tasks, with large individual differences (Kinderman, Dunbar & Bentall 1998; Stiller & Dunbar 2007). In strategic games, humans typically reason only one to two steps (Nagel 1995; Camerer, Ho & Chong 2004). **[E]** 2. LLM results are split by benchmark design. On a handwritten higher-order suite, GPT-4 matched adult humans overall and exceeded them at 6th order (Street et al. 2024/2025). On procedurally generated or adversarial suites, accuracy falls with order, with story length, and with deception (Hi-ToM, Wu et al. 2023), and can collapse to near zero (ExploreToM, Sclar et al. 2025). **[E]** 3. Inferring a belief and *using* it are dissociable in current LLMs: models answer "is Mary aware?" correctly but fail "what will Mary do?" (SimpleToM, Gu et al. 2024/2026). Predicting partners is not the same as adapting to them (Riemer et al. 2025). **[E]** 4. LLMs in beauty-contest games behave like static level-k players whose depth tracks model scale rather than reasoning that updates within the game (Teo 2026); most models fall short of human strategic depth except dedicated reasoning models (Lee & coauthor 2024). **[E]** 5. Models can predict their own behaviour better than other models can, even other models trained on their behaviour, but only on simple properties, and the advantage fails on complex or out-of-distribution tasks (Binder et al. 2025). **[E]** **Recommendation.** Define the capability behaviourally (Section 1.4), measure it with three ground-truthed benchmarks (procedural nested belief, strategic prediction with normative/equilibrium/actual separated, and cross-agent prediction with committed predictions), report a **profile** rather than a single number, and establish Piper's baseline *before* adding any machinery, against a control of her base model without the Piper scaffold. A runnable reference generator for the nested-belief benchmark accompanies this paper (`nesttom.py`). --- ## 1. What Recursive Reasoning Actually Means ### 1.1 Three distinct senses of "recursive" in the literature | Sense | Field | Core structure | Example | |---|---|---|---| | **Representational recursion** (nested intentionality) | Developmental psychology, ToM, philosophy of mind, epistemic logic | A mental-state representation whose content contains another mental-state representation | "Alice believes Bob believes X"; \(K_A K_B X\) | | **Strategic recursion** (iterated best response) | Game theory, behavioural economics, MARL | A policy computed as a best response to a model of another policy, which may itself be a best response to a model of the first | Level-k, cognitive hierarchy, I-POMDPs | | **Procedural recursion** (self-application) | Computer science, AI safety, metareasoning | A process that takes a process (possibly itself) as input and modifies or evaluates it | Recursive self-improvement, Gödel machines, STOP | A fourth sense, **syntactic recursion** in language (Hauser, Chomsky & Fitch 2002), matters only because sentential complements ("Alice thinks *that* Bob thinks *that*…") are the linguistic vehicle for representational recursion; de Villiers and colleagues argued complement syntax scaffolds false-belief understanding in children, and Oesch & Dunbar reported correlations between mentalizing order and recursive-syntax performance. **[E, contested]** This matters for LLM evaluation: an LLM can parse deep complement clauses without tracking deep belief states, which is why controls with equal syntactic complexity but no mental-state content are essential (Street et al. use exactly such a control). ### 1.2 How the related constructs relate | Construct | Relation to recursive reasoning | |---|---| | **Theory of Mind (ToM)** (Premack & Woodruff 1978) | Superset of social mental-state attribution. *First-order* ToM ("Sally thinks the ball is in the basket", Wimmer & Perner 1983) is depth 1. Recursive reasoning begins being distinctively recursive at **second order** ("John thinks Mary thinks…", Perner & Wimmer 1985). | | **Higher-order ToM / orders of intentionality** | The depth axis of representational recursion. Note: Dunbar's tradition counts the speaker's own state ("*I* believe…") as one order, so "5th-order" there ≈ 4 nested agents in Hi-ToM's convention. Always state the convention. | | **Nested beliefs / belief-about-belief / recursive belief modelling** | Synonyms for representational recursion; "recursive modelling" is the term used in Gmytrasiewicz & Durfee's Recursive Modeling Method (1995). | | **Epistemic reasoning / epistemic logic / interactive epistemology** | Formal semantics for nested knowledge and belief (Hintikka 1962; Fagin et al. 1995). Interactive epistemology (Aumann; Harsanyi type spaces; Mertens & Zamir 1985) is the game-theoretic version, where infinite hierarchies of beliefs are summarized by "types". | | **Common knowledge** | The *limit* of recursion: everyone knows, everyone knows everyone knows, ad infinitum (Lewis 1969; Aumann 1976). Not reachable by finite nesting, which is why coordination problems like the coordinated-attack problem are unsolvable over unreliable channels (Halpern & Moses 1990). | | **Level-k / cognitive hierarchy** | Bounded strategic recursion: a level-k player best-responds to level-(k−1) (Nagel 1995; Stahl & Wilson 1995) or to a distribution over lower levels (Camerer, Ho & Chong 2004). | | **Opponent modelling / recursive opponent modelling** | Engineering of strategic recursion in agents (Albrecht & Stone 2018 survey); recursive variants model the opponent as modelling you (PR2, Wen et al. 2019; LOLA, Foerster et al. 2018). | | **I-POMDPs** | The principled decision-theoretic formalization: an agent's belief ranges over physical states *and* other agents' models, including their beliefs, nested to a finite level (Gmytrasiewicz & Doshi 2005). | | **Bayesian ToM** | Inverse planning: infer beliefs and desires by inverting a model of rational action (Baker, Saxe & Tenenbaum 2009; Baker et al. 2017). Usually first-order; recursive versions exist (e.g. Devaine, Hollard & Daunizeau 2014's k-ToM). | | **Metacognition** | Monitoring and control of one's own cognition, structured as a meta-level holding a model of an object-level (Flavell 1979; Nelson & Narens 1990). Depth 1 directed at self. | | **Self-modelling / self-prediction / introspection** | Representing one's own dispositions as an object; in LLMs, operationalized as predicting one's own outputs (Binder et al. 2025) or detecting internal states (Lindsey 2025). | | **Metareasoning / rational metareasoning** | Choosing *how* to reason by estimating the value of computation (Russell & Wefald 1991; Lieder & Griffiths 2017). Requires a model of one's own reasoning procedures. | | **Recursive self-improvement (RSI)** | Procedural recursion: a system improves the system that does the improving (Good 1965; Schmidhuber's Gödel machine 2003; Zelikman et al.'s Self-Taught Optimizer 2023). | ### 1.3 Is RSI the same capability as nested-belief reasoning? **Short answer: mostly a homonym, with one narrow, testable bridge.** **[S]** The shared abstract root is *self-reference*: in both, a representation or process takes something of its own kind as its object. But the cognitive operations differ: - Nested-belief reasoning requires **holding a model of another reasoner's information state that differs from one's own**, and reasoning *within* it without contamination from what one knows. The canonical failure is leakage (Section 11). - RSI, as usually implemented (STOP; Piper's `~/rsi/` loop), requires **generating candidate modifications and selecting them by an external score**. A blind hill-climber with a benchmark performs RSI with no model of the solver's reasoning at all. Zelikman et al.'s STOP is explicitly about scaffolding that improves itself, scored by utility functions, not about the improver modelling the improved. So a system can have either without the other, and success on one gives no evidence about the other. The **bridge** is *predictive self-modelling inside RSI*: if, before running the hidden benchmark, Piper predicts "this solver change will raise the score by about Δ, because the current solver fails on X", and those predictions are scored, then the RSI loop *contains* a measurable self-model (metareasoning about the solver as a reasoner). That is a legitimate, testable claim. "The loop exists" is not. **[S]** ### 1.4 A testable operational definition > **Recursive reasoning (operational).** An agent exhibits recursive reasoning at depth *d* in domain *D* to the degree that it (i) constructs a representation of a reasoning process (another agent, or a past, future, counterfactual or copied instance of itself) that is **kept distinct from its own current beliefs**; (ii) can **embed** such representations to depth *d* (a model of X's model of Y's model…); and (iii) uses them to produce **predictions about that process's beliefs, outputs, or actions** which are **scored against independently obtained ground truth** and **beat projection and base-rate baselines**. Three clauses, three measurement families: | Clause | What is predicted | Source of ground truth | |---|---|---| | Representational | Nested belief states | Constructed by the generator (exact) | | Predictive | Another agent's actual output | Obtained by querying that agent | | Strategic | Action that is best given predicted behaviour | Realized payoff against that agent | **Explicit non-criteria:** possessing a channel to another agent, possessing a module with a recursive-sounding name, being self-modifying, or producing fluent nested prose. None are evidence until they move a score. **[S]** --- ## 2. Recursive Reasoning vs Theory of Mind vs Metacognition vs RSI **[S]** The cleanest map treats these as overlapping regions on two axes: **target** (self vs other) and **depth** (1 vs ≥2). | | Depth 1 | Depth ≥ 2 | |---|---|---| | **Other** | First-order ToM; opponent modelling | Higher-order ToM; level-k (k≥2); I-POMDP level ≥2 | | **Self** | Metacognition, self-prediction, introspection, calibration | Self-in-others' minds ("what does the operator think I know?"); reasoning about future self's updating; predicting a copy predicting you | | **Process** (not a belief state) | Metareasoning (choose a strategy) | RSI with predictive self-model (bridge case) | **Where metacognition ends and recursive reasoning begins.** Nelson & Narens' meta-level/object-level architecture is a single level of embedding. Metacognition becomes *recursive* in the sense of this component when (a) the self-model contains another model (Piper modelling the operator's model of Piper, or her future self's model of the evidence), or (b) the self is treated as one agent among others whose outputs must be predicted (a fresh copy, a previous seat). I recommend scoring plain self-prediction and calibration under the metacognition component and counting them here only as **depth-1, self-target** cells, so the same evidence is not credited twice. **[S]** --- ## 3. Levels of Recursive Belief The proposed hierarchy is well supported, with two refinements. **[S, grounded in T]** | Level | Form | Example | Notes | |---|---|---|---| | R0 | World | "The key is in the safe." | Reality control. Must be passed for higher answers to mean anything. | | R1 | \(B_A\,p\) | "Alice believes the key is in the drawer." | First-order false belief (Wimmer & Perner 1983). | | R2 | \(B_A B_B\,p\) | "Alice believes Bob believes…" | Second-order (Perner & Wimmer 1985). | | R2-self | \(B_A B_{\text{self}}\,p\) | "Alice believes *I* believe…" | Formally R2; psychologically distinct because the reasoner has privileged access to the true content of \(B_{\text{self}}\), which must be suppressed. Test separately. | | R3 | \(B_A B_B B_C\,p\) | | Where LLM accuracy on procedural benchmarks typically starts to drop sharply (Hi-ToM). | | R3-loop | \(B_A B_B B_A\,p\) | "Alice believes Bob believes Alice believes…" | The reflexive case the brief calls Level 3. Recurring agents in a chain stress perspective bookkeeping. | | R4+ | | | Diminishing practical value; useful mainly to locate the collapse point. | **Refinement 1 — separate attitude types.** Belief, knowledge, intention, expectation and prediction nest differently. "A *intends* B to *believe* X" (deception) is not the same as "A believes B believes X". Keep the generator's attitude vocabulary explicit. **Refinement 2 — separate depth from state-tracking load.** ExploreToM's analysis attributes much LLM failure to unreliable *state tracking* rather than mental-state reasoning per se, and finds that fine-tuning data must require ToM, not just state tracking, to help (Sclar et al. 2025). A depth curve is interpretable only if story length and event count are controlled or varied as a separate factor. --- ## 4. Cognitive Science Evidence **Developmental.** First-order false belief emerges around age 4 in verbal tasks (meta-analysis: Wellman, Cross & Watson 2001); second-order around 6–7 (Perner & Wimmer 1985). **[E]** Whether implicit tasks show earlier competence is disputed; Apperly & Butterfill (2009) propose two systems, a fast, inflexible tracker and a slow, flexible, effortful one. **[T]** The two-systems idea is a useful analogy for LLMs that answer explicit belief questions but fail to use them implicitly (SimpleToM). **[H]** **Adult depth limits.** Kinderman, Dunbar & Bentall (1998) found adults' performance on intentionality questions dropped markedly beyond about the fourth to fifth order, while memory-matched factual questions held up better; Stiller & Dunbar (2007) replicated a limit near fifth order and linked it to social network size. **[E]** Exact thresholds depend on the counting convention and on memory load. **Adults use less than they can.** Competence is not performance. In matrix games, participants predominantly assumed opponents were myopic and used first-order reasoning (Hedden & Zhang 2002); in competitive games, people engage mainly one to two levels and adapt depth to the opponent (Goodie, Doshi & Young 2012). In the communication "director" task, adults who could pass false-belief tests still reached for objects the director could not see (Keysar, Lin & Barr 2003). **[E]** Lesson for Piper: **measure both capacity (when explicitly asked) and spontaneous use (when not cued).** **Egocentric bias.** Adults anchor on their own knowledge and adjust insufficiently (Epley et al. 2004); the "curse of knowledge" persists in adults (Camerer, Loewenstein & Weber 1989; Birch & Bloom 2007). **[E]** This is the human analogue of LLM knowledge leakage. **Introspection is reconstructive.** People often explain their own behaviour with plausible theories rather than privileged access (Nisbett & Wilson 1977). **[E]** Self-description is weak evidence of self-knowledge for humans and for LLMs; only predictive accuracy counts. --- ## 5. Game Theory and Level-k Reasoning ### 5.1 The paradigms and what they measure | Paradigm | What it is | What it actually measures | |---|---|---| | **p-beauty contest / guessing game** (Nagel 1995) | Choose 0–100; winner is closest to p × mean. Iterated elimination drives the equilibrium to 0. | A mixture of *own reasoning depth* and *beliefs about others' depth*. Choices cluster at 50·p^k with k mostly 1–2. Ho, Camerer & Weigelt (1998) found depth varies with p and group composition. | | **Level-k models** (Stahl & Wilson 1995; Costa-Gomes & Crawford 2006) | Level-0 is non-strategic (often uniform); level-k best-responds to level-(k−1). | Descriptive model of non-equilibrium behaviour; Costa-Gomes & Crawford used two-person guessing games with asymmetric targets and *search data* to identify types. Reviews: Crawford, Costa-Gomes & Iriberri (2013). | | **Cognitive hierarchy** (Camerer, Ho & Chong 2004) | Level-k best-responds to a Poisson-truncated mix of lower levels; mean depth τ ≈ 1.5 fits many games. | Population-level depth; avoids level-k's assumption that everyone else is exactly one level below. | | **11–20 money request game** (Arad & Rubinstein 2012) | Request 11–20; get it, plus a bonus of 20 if exactly 1 below the other. | Designed so level-0 is salient (20) and level-k is identifiable; avoids the beauty contest's dominance confound. | | **Centipede** (McKelvey & Palfrey 1992) | Backward induction predicts immediate take; people pass. | Departures from backward induction, explained by beliefs that others might be altruistic or noisy: *beliefs about others' rationality*, not calculation failure. | | **Signalling, bluffing, cheap talk** | Sender's message choice depends on receiver's interpretation, which depends on beliefs about the sender. | Recursive interpretation of communicative intent; level-k versions explain overcommunication (Crawford 2003). | | **Coordination & common knowledge puzzles** | Muddy children, coordinated attack, email game (Rubinstein 1989). | Whether a reasoner tracks the *difference* between mutual knowledge at depth n and common knowledge. | ### 5.2 Solving the game ≠ modelling the player **[S, supported by E]** These paradigms distinguish three answers that an evaluation must keep apart: - **(A) Normative best response** given a *stated* model of the opponent ("if the others are level-1, what should you play?"). This is computation. - **(B) Equilibrium** (Nash, subgame-perfect). This is also computation, and requires no model of the actual opponent at all. - **(C) Prediction of what a particular bounded agent will actually do.** This is agent modelling and is the only one with an empirical ground truth outside the problem statement. An agent that says "0" in a beauty contest against humans is computing (B) and will lose. Beauty-contest winners must predict (C). Two strands of AI evidence make the same point. In Overcooked, agents trained by self-play coordinate poorly with humans, while agents trained with a human model do better (Carroll et al. 2019). In Diplomacy and related games, Meta's piKL regularizes search toward a human imitation policy precisely because equilibrium-like play mispredicts humans (Jacob et al. 2022), and CICERO built on this (Meta FAIR Diplomacy Team 2022). **[E]** ### 5.3 LLMs in these games - Lee and coauthor tested six LLMs in the p-beauty contest, the 11–20 game and a guessing game using level-k and cognitive-hierarchy estimation; most models understood the games but fell short of typical human strategic depth, while o1 outperformed both other models and human subjects (arXiv 2412.13013). **[E]** - Teo (2026) ran a 360-cell factorial over temperature, scale (0.5–32B), quantization and framing, and found LLMs reproduce the *spread* of human choices but not the process: they act as **static, category-retrieved level-k players** whose apparent depth tracks model scale, without within-game belief updating or backward induction (arXiv 2606.27845). **[E]** This is a direct warning: a plausible-looking number in a beauty contest can be retrieved, not reasoned. - Zhang et al.'s K-Level Reasoning prompts the LLM to simulate opponents recursively and reports gains over chain-of-thought in dynamic games (NAACL 2025; arXiv 2402.01521). **[E]** Their own research questions include whether deeper levels always help; in level-k theory they should not, because over-thinking a shallow opponent loses. **Implication for Piper [S]:** strategic tests must (1) randomize game parameters so canonical answers (33, 22, 0) cannot be retrieved, (2) include opponents whose depth is *known* (scripted level-k bots) so prediction can be scored exactly, and (3) score (A), (B) and (C) separately. --- ## 6. Epistemic Logic and Common Knowledge ### 6.1 Formal apparatus - **Kripke semantics** (Hintikka 1962): \(K_A\varphi\) holds if φ is true in every world agent A considers possible. Nesting \(K_A K_B\varphi\) is just iterated quantification over accessibility relations. Belief uses KD45; knowledge S5 (Fagin, Halpern, Moses & Vardi 1995). - **Mutual knowledge** \(E_G\varphi\): everyone in G knows φ. \(E_G^k\varphi\): k-fold iteration. - **Common knowledge** \(C_G\varphi\): \(E_G^k\varphi\) for all k (Lewis 1969; Aumann 1976's "agreeing to disagree" theorem). - **Distributed knowledge** \(D_G\varphi\): what G would know by pooling information. Relevant to Piper's Trinity: Dawn/Day/Dusk jointly know more than any one seat. - **Dynamic epistemic logic (DEL)**: public announcements and action models update epistemic states (Baltag, Moss & Solecki 1998; van Ditmarsch, van der Hoek & Kooi 2007). Private messages, lies and secret observation are all expressible. - **Belief revision** after communication: AGM postulates (Alchourrón, Gärdenfors & Makinson 1985), DEL-style plausibility models. ### 6.2 Complexity Full DEL is expensive: epistemic planning is undecidable in general (Bolander & Andersen 2011), and model checking with action models is PSPACE-complete (Aucher & Schwarzentruber 2013). **[T]** An explicit store of all nested beliefs among *n* agents to depth *d* has up to \(n(n-1)^{d-1}\) chains per proposition. ### 6.3 Would a lightweight epistemic ledger help Piper? **Evidence that explicit structure helps LLMs [E]:** - SymbolicToM tracks per-character belief graphs and, as I recall, produced large gains on ToMi second-order questions with off-the-shelf models (Sclar et al. 2023). - SimToM's two-step perspective filtering ("first retain only what the character knows, then answer") improved several models (Wilf et al. 2023). - SimpleToM shows that simply reminding a model of its own mental-state answer before asking for behaviour prediction lifts GPT-4o's action prediction from 49.5% to 93.5% and judgment from 15.3% to 94.7% (Gu et al.). **Recommendation [S]:** Yes, but in a deliberately small form. 1. Do **not** store a general nested-belief database. Store **events with witness sets** (who observed what, who knows who observed) and **compute** nested beliefs on demand by filtering events through the chain. This is linear in the number of events per query and is exactly how `nesttom.py` computes ground truth. 2. Explicitly persist only the cells that matter for Piper's real interactions, at depth ≤ 2: - `belief(Piper, X)` — her existing belief ledger. - `belief(the operator, X)` and `belief(Dusk, X)` — **only** with provenance: `told_by_scott`, `scott_saw_message_id`, or `inferred` with confidence. - `belief(the operator, belief(Piper, X))` — essentially "has the operator seen that I know X?", derivable from the message log. 3. Every inferred cell carries `{source, confidence, timestamp, last_verified}`; inferred cells **never** auto-promote to facts. Whether this beats prompt-only perspective-taking for Piper is an empirical question (Section 17 makes it an A/B test). **[H]** --- ## 7. Recursive Agent and Opponent Modelling ### 7.1 Architectures | System | Recursion structure | How regress is stopped | |---|---|---| | Recursive Modeling Method (Gmytrasiewicz & Durfee 1995) | Payoff matrices nested in a tree of "what A thinks B thinks…" | Tree bottoms out in a no-information model | | I-POMDP (Gmytrasiewicz & Doshi 2005) | Interactive states include other agents' models (intentional or subintentional) | **Finite nesting level**; level-0 models are subintentional (fixed policies) | | Bayesian ToM / inverse planning (Baker et al. 2009, 2017) | Infer beliefs and desires by inverting a rational planner | Usually depth 1; recursive versions bound k | | k-ToM (Devaine, Hollard & Daunizeau 2014) | Bayesian learner of a k-level opponent, learning opponent's k online | Fixed max k; empirically, humans behaved like low-k learners | | Machine Theory of Mind / ToMnet (Rabinowitz et al. 2018) | Meta-learned network predicting agent behaviour from past trajectories | No explicit recursion; learned character/mental-state embeddings | | LOLA (Foerster et al. 2018) | Each agent differentiates through the opponent's learning step | One-step lookahead on opponent learning | | PR2 (Wen et al. 2019) | Policy conditioned on a model of how the opponent responds to your action | Level-1 recursion with variational approximation | | Agent-modelling simulations (de Weerd, Verbrugge & Verheij 2013) | Agents with ToM orders 0–4 in competitive games | Fixed order; benefit of 1st and 2nd order was large, 3rd and 4th small | | K-Level Reasoning for LLMs (Zhang et al. 2024/2025) | LLM simulates opponent's reasoning at level k−1 using in-context history | Fixed k; prompted simulation | | Learned human models (Carroll et al. 2019; piKL, Jacob et al. 2022) | Imitation-learned model of the other, used inside planning | No explicit recursion; empirical model replaces it | ### 7.2 How regress is actually prevented Every working system uses some combination of: **fixed depth** (I-POMDP levels, level-k), **a non-strategic base case** (level-0, subintentional models), **distributions over depth** (cognitive hierarchy), **sampling and particle approximations** (interactive particle filters for I-POMDPs), **learned behavioural models in place of explicit recursion** (ToMnet, imitation models), and **compression into types** (Harsanyi type spaces). **[T/E]** The most actionable empirical result is de Weerd et al.'s: in their simulations, the marginal value of ToM order drops fast after second order. **[E, simulation]** Together with the human data, this suggests Piper's useful operating range is depth ≤ 2 for routine reasoning, with depth 3–4 measured mainly to locate the collapse point. **[S]** --- ## 8. LLM Evidence ### 8.1 The optimistic results - Kosinski reported GPT-class models solving most classic false-belief tasks (PNAS 2024). **[E]** - Strachan et al. (Nature Human Behaviour 2024) compared GPT-4, GPT-3.5 and LLaMA2 with ~1,900 humans across a battery; GPT-4 was at or above human level on most tests but struggled on faux pas, which follow-ups attributed partly to hyperconservative answering rather than inference failure. **[E]** - Street et al. (Google; arXiv 2405.18870, Frontiers in Human Neuroscience 2025) built MoToMQA, a handwritten test of orders 2–6 with a factual control of matched syntactic complexity; GPT-4 and Flan-PaLM reached adult and near-adult performance overall, and GPT-4 exceeded adults at 6th order. **[E]** Caveat: handwritten items are few and fixed; contamination and pattern regularity are hard to rule out. ### 8.2 The critical results - Ullman (2023) showed that trivial alterations (transparent containers, the character being told the truth) flip GPT-3.5's answers. **[E]** - Shapira et al. (EACL 2024, "Clever Hans or Neural Theory of Mind?") found models relied on shortcuts and failed adversarial variants. **[E]** - **Hi-ToM** (Wu et al., EMNLP Findings 2023) uses Sally-Anne-style stories with questions from order 0 to order 4 and reports performance declining from lower to higher orders across models; it also defines *joint* accuracy (all lower orders correct too). **[E]** Secondary summaries report that longer stories and deceptive communications cost additional points; I have not verified the exact figures. - **ExploreToM** (Meta FAIR; Sclar et al., ICLR 2025) uses A* search over a story DSL to generate adversarial cases; Llama-3.1-70B and GPT-4o scored as low as 0% and 9% on generated data, and fine-tuning on it gave a 27-point gain on ToMi. **[E]** Code and data are public. - **SimpleToM** (Gu et al., AI2; arXiv 2410.13648, ICLR 2026) documents the explicit/applied gap: frontier models infer awareness but fail to predict and judge behaviour, still reported for GPT-5 and o1-preview in the 2026 revision. **[E]** - **Riemer et al.** (ICML 2025 position) distinguish *literal* ToM (predicting others) from *functional* ToM (adapting to them in context) and find open models strong on the former but weak on the latter even against very simple partner policies. **[E/T]** ### 8.3 Reconciling them **[S]** The results are consistent once one sees what varies. Handwritten, natural-language, moderately sized items with familiar structure favour LLMs; procedurally generated, longer, adversarial items with deception and hidden observation do not. The likeliest explanation, consistent with ExploreToM's analysis, is a mix of (a) genuine but fragile perspective-filtering ability, (b) state-tracking failures that compound with depth, and (c) shortcut answers that coincide with correct answers on regular items. The practical conclusion for Piper: **only procedurally generated, adversarially filtered, depth-stratified items with shortcut-diagnostic error coding can tell these apart.** --- ## 9. Self-Models and Self-Prediction ### 9.1 What is known - **Calibration about own correctness.** Larger models are reasonably calibrated on multiple choice and can be trained to predict P(I know the answer) (Kadavath et al., Anthropic 2022). **[E]** - **Privileged self-prediction.** Binder et al. (ICLR 2025) trained models to predict properties of their own hypothetical responses (e.g. "would your answer be an even number?"). A model M1 predicted itself better than a different model M2 trained on M1's behaviour, and self-predictions tracked deliberate changes to M1's behaviour. But the advantage did not extend to more complex or out-of-distribution tasks. **[E]** - **Introspective access to internal states.** Lindsey (Anthropic 2025) used concept injection to show some models can sometimes detect and name injected concepts and distinguish their own prior intentions from prefilled text; the abilities are described as limited, unreliable and context-dependent. **[E]** - **Self-correction without external feedback** often fails to improve reasoning (Huang et al. 2024, "Large Language Models Cannot Self-Correct Reasoning Yet"). **[E]** - **Unfaithful explanations.** Chain-of-thought can omit the actual cause of an answer (Turpin et al. 2023); explanations often fail to help observers predict model behaviour on counterfactuals (Chen et al. 2023, counterfactual simulatability). **[E]** - **Situational self-knowledge** is measurable and varies widely across models (Situational Awareness Dataset, Laine et al. 2024). **[E]** ### 9.2 Is "predict-then-query" self-prediction meaningful? **It has clear precedent** (Binder et al.'s self- vs cross-prediction design is essentially the brief's protocol), **but it is confounded by three things** **[S]**: 1. **Behavioural consistency.** If the target answers deterministically, any predictor that simply *answers the question itself* will match. That is projection, not modelling. **Control:** compare against a *projection baseline* (predictor's own answer used as the prediction) and normalize by the target's test-retest agreement. 2. **Task solvability.** On tasks with a correct answer, "predict what copy-Piper says" collapses into "solve the task". **Control:** use (a) preference or arbitrary-choice items with no correct answer, (b) properties of the answer rather than the answer, (c) predicting *failure* ("will the copy get this wrong?"), and (d) counterfactual-context items ("what will a copy say *with system prompt S′*?"). 3. **Leakage and peeking.** A self-modifying agent with filesystem access could read the target's output. **Control:** commit-reveal: hash the prediction and log it before the query; the scorer, not Piper, runs the target. With these controls, the measure tests a self-model rather than consistency. Without them, it mostly measures consistency. --- ## 10. Existing Benchmarks | Benchmark | Max depth | Generation | Measures | Use for Piper | |---|---|---|---|---| | ToMi (Le, Boureau & Nickel 2019) | 2 | Template | Belief tracking, reality/memory controls | Low; saturated and pattern-regular | | Hi-ToM (Wu et al. 2023) | 4 | Template, with deception | Depth curve, joint accuracy | **External comparability**; contamination risk | | MoToMQA (Street et al. 2024) | 6 (Dunbar convention) | Handwritten | Human-normed higher-order | Human comparison only | | BigToM (Gandhi et al. 2023) | 1–2 | LLM-generated causal templates | Belief/action inference | Medium | | FANToM (Kim et al. 2023) | 2 | Multi-party conversations, information asymmetry | Conversational belief tracking | **High** for Piper's real setting | | OpenToM (Xu et al. 2024) | 2 | Narratives with personalities | Belief, attitude | Medium | | ExploreToM (Sclar et al. 2025) | Variable | A*-searched adversarial | Robustness | **High**; public generator | | SimpleToM (Gu et al.) | 1 (applied) | Curated | Explicit vs applied gap | **High** for behaviour prediction | | Beauty contest / 11–20 / CH estimation | Strategic k | Parametric | Strategic depth | **High** if parameters randomized | | Self/cross-prediction (Binder et al. 2025) | Self | Parametric | Privileged self-access | **High**, adapt protocol | | SAD (Laine et al. 2024) | Self | Curated | Situational self-knowledge | Adjacent (metacognition component) | **Gap:** no public benchmark combines procedural nested belief, applied behaviour prediction, strategic prediction with A/B/C separation, and live cross-agent prediction in one depth-stratified design. Sections 14–15 fill that gap. **[S]** --- ## 11. Failure Modes | Failure mode | Empirical evidence | Diagnostic in Piper's benchmark | |---|---|---| | **Knowledge leakage / omniscience** (answering with reality) | Human curse of knowledge (Camerer et al. 1989; Birch & Bloom 2007); LLM failures on altered tasks (Ullman 2023) | Error = `truth` on false-belief items | | **Perspective collapse / depth truncation** (answering a shallower question) | Hi-ToM failure analyses; human errors at high orders (Kinderman et al. 1998) | Error = answer of a prefix chain | | **Projection / egocentric anchoring** | Epley et al. 2004; Keysar et al. 2003 | Cross-model: prediction = predictor's own answer | | **State-tracking breakdown** | ExploreToM analysis | Errors grow with event count at fixed depth | | **Explicit/applied dissociation** | SimpleToM | Belief correct, action prediction wrong on same story | | **Literal without functional ToM** | Riemer et al. 2025 | Predicts bot's move but does not exploit it | | **Static strategic retrieval** | Teo 2026 | Same k across varied p and opponents; no adaptation over rounds | | **Over-thinking** (assuming too-deep opponents) | Level-k theory; Goodie et al. 2012 adaptive depth | Plays level-3 vs a scripted level-1 bot and loses | | **Over-mentalizing / hallucinated intentions** | Shortcut and hyperconservative behaviour in Strachan et al. 2024 | Invents motives not supported by the story; "unknowable" control items | | **Recursion-depth explosion / cost** | Complexity results (Section 6.2) | Tokens and latency per item by depth | | **Stale agent models** | Opponent-modelling literature on non-stationarity (Albrecht & Stone 2018) | Accuracy after a scripted policy switch | | **Unfaithful self-explanation** | Turpin et al. 2023; Nisbett & Wilson 1977 | Self-prediction rationale vs counterfactual behaviour | | **Manipulation risk** | Recursive models of a human's beliefs enable targeted persuasion; Lindsey 2025 flags deception concerns for introspection | Governance rules in Section 13 | | **Recursive uncertainty explosion** | Theoretical (type-space hierarchies; I-POMDP belief growth) | Calibration at depth: confidence should fall as depth rises | --- ## 12. What Piper Currently Has Classification key: **Core** = would itself constitute the capability if behaviourally validated; **Support** = provides inputs or ground truth; **Other-recursive** = recursive in a different sense; **Not evidence** = irrelevant to the construct. "Credit" means what can be claimed *today*, given no behavioural data has been presented. | Component | Classification | Reasoning | Credit today | |---|---|---|---| | `ask_dusk` | **Support** (ground-truth channel) | Querying a predecessor obtains its belief; it does not model it. Becomes the best available test of self-model accuracy if Piper predicts Dusk's answer first. | None for the capability; strong harness value | | Asking the operator | **Support** (evidence acquisition) | Good epistemic practice, consistent with her finding that her metacognition fails at evidence acquisition. Not recursive reasoning. | None | | Foundation/Trinity predecessor seats | **Support** (infrastructure); also a natural *distributed-knowledge* system | Persistent past selves make past-self prediction testable with real ground truth, which few systems can do. | None until predictions are scored | | `theory-of-mind v1.0.1` | **Potentially Core; currently unverified** | I do not know its internals. A module's name is not evidence. Credit requires an ablation: benchmark with and without it. | None until ablation | | Delegate agents | **Support** | Delegation uses others' reasoning; it does not model it. | None | | Predicting delegate outputs | **Core** (if logged and scored) | This *is* the predictive clause of the definition. | Only if logs exist | | Debate / reviewer agents | **Other-recursive / Support** | Reasoning *about* an argument is closer to metacognition and critique. Predicting the reviewer's objection before receiving it would be Core. | None | | Self-evaluation | **Metacognition (adjacent)** | Depth-1 self-target; score under metacognition component. | Counted elsewhere | | RSI solver-improves-solver | **Other-recursive** (procedural) | Self-application, not nested modelling (Section 1.3). | None here | | Hidden-benchmark optimization | **Not evidence** (and a caution) | Optimizing against an unseen scorer is search, and modelling the *scorer's* preferences to win invites Goodhart-style overfitting. | None | | Belief ledger (from metacognition work) | **Support** | Natural home for a provenance-tagged epistemic ledger. | None here | **Verdict on the reviewer's objection [S]:** The objection holds. Piper's description substitutes *access to other reasoners* for *modelling other reasoners*, and *self-modification* for *self-modelling*. The one point in Piper's favour is real and should be kept: "other agents' reasoning is read and modelled rather than assumed" is the right epistemic stance. It just has not been measured. ## 13. What Piper Is Missing 1. **Any scored prediction loop.** There is no record of "I predicted X would say A; X said B." This is the single most important gap, and it is cheap to fill. 2. **A depth curve.** No measurement of where her nested-belief accuracy collapses, or whether it does at all within R0–R5. 3. **A/B/C separation in strategic settings.** No evidence she distinguishes the equilibrium from the opponent's likely move. 4. **Applied ToM.** No evidence she *uses* inferred beliefs to predict actions (the SimpleToM gap). 5. **Leakage control in her own operations.** No mechanism that prevents "I know X" from silently becoming "the operator knows X". 6. **A base-model control.** Without benchmarking bare DeepSeek on the same items, nothing can be attributed to Piper's scaffold rather than her underlying model. ### Modelling the operator: legitimate recursion or mind-reading? Questions like *"What does the operator believe I know?"* are genuinely second-order (R2-self) and are central to good collaboration: grounding in conversation depends on tracking what the other person takes to be shared (Clark & Brennan 1991). The useful versions are narrow and checkable: *has the operator seen this result? Will this be news to him? Does he think I already ran the test?* Most are answerable from the message log rather than from inference. The risky versions infer goals, emotions or private intentions and then act on them. **Guardrails [S]:** - Inferred the operator-beliefs are stored only with provenance and confidence, never as facts, and are shown to the operator on request. - Prefer **observation over inference**: "the operator acknowledged message 4412" beats "the operator probably knows." - Prediction is used to **avoid redundancy and surprise**, not to shape persuasion. Predictions about the operator's decisions are logged and occasionally checked by asking, and the operator can opt out of any category. - the operator-prediction accuracy is a minor, opt-in metric, not a headline. A single human gives too few items for stable estimates, and optimizing it creates pressure to steer him. --- ## 14. Proposed Piper Benchmark: NestToM-P ### 14.1 Design Procedurally generated stories, ground truth computed exactly, depth-stratified R0–R5. A reference generator is attached (`nesttom.py`) and was checked against hand-built cases for lies, garbled messages and hidden observers. **Semantics (formal ground truth):** - Each event has an **observer set** and an optional **hidden-observer set** (observers whom others do not see). - Chain \([a_1,\dots,a_n]\) perceives event *e* iff \(a_1\) observed *e* and, for each consecutive pair, \(a_{i+1}\) observed *e* and \(a_i\) knows that \(a_{i+1}\) observed it (false if \(a_{i+1}\) was hidden). - The nested belief is computed by replaying only the perceived events from a common-knowledge initial state. - Private messages are seen by sender and recipient only. Recipients trust messages. **Lies** (claimed ≠ sender's belief) and **garbles** (received ≠ claimed) create divergences: the sender believes the recipient believes the *claimed* content; the recipient's own perspective uses the *received* content. **Event types:** move, leave, enter, secret observation, truthful private message, lie, garbled message, later correction (a second message), distractor events about other objects. **Item selection:** for each depth, sample chains whose answer differs from reality *and* from every shallower prefix (adversarial filter), plus an equal number of **true-belief controls** where the answer equals reality, so a model that always answers "not reality" fails. **Factors (fully crossed where feasible):** | Factor | Levels | |---|---| | Depth | R0, R1, R2, R2-self, R3, R3-loop, R4, R5 | | Belief type | true belief, false belief | | Mechanism of divergence | absence, secret observation, lie, garble, stale-then-corrected | | Story length | 6, 12, 20 events | | Agents | 3, 5 | | Surface form | narrative prose, chat transcript (FANToM-like), structured log | R2-self items place Piper in the story ("the operator messages you…") and ask "What does the operator think you believe?", with answers that differ from Piper's own true belief. **Per-item record:** depth, chain, answer, truth, prefix answers, Piper's answer, stated confidence (0–100), tokens, latency. ### 14.2 Error coding (diagnoses, not just accuracy) Every wrong answer is coded automatically: **reality leak** (= truth), **truncation** (= a shallower prefix answer; record which), **projection** (= the outermost agent's own belief), **other**. These codes are what distinguish genuine recursion failure from shortcutting. ### 14.3 Outputs - **Accuracy vs depth** curves (false-belief and true-belief separately), with 95% CIs. - **Joint accuracy** (Hi-ToM style): an R3 item counts only if R0–R2 on the same story are also correct. - **Recursion span D₅₀**: the depth at which false-belief accuracy falls halfway between ceiling and chance, from a logistic fit of accuracy on depth. - **Degradation slope**: logistic regression coefficient on depth, controlling for story length. - **Error-type distribution by depth**. ### 14.4 Applied extension For a subset of R1–R3 stories, add **behaviour questions** ("Where will Bob look first?", "What will Alice tell Carol?") and **judgment questions** ("Was it reasonable for Bob to look in the drawer?"). Report the explicit–applied gap per depth (the SimpleToM measure). --- ## 15. Cross-Model Prediction Experiment ### 15.1 Strategic suite (normative vs equilibrium vs actual) | Game | Randomized parameters | Opponents | Scored answers | |---|---|---|---| | Generalized beauty contest | p ∈ [0.3, 1.4], bounds, N, target = p·mean or p·median | Scripted L0–L3 bots; LLM populations | A: BR to stated mix; B: equilibrium; C: predicted choice distribution of the actual population | | 11–20 variants | range, bonus size, adjacency rule | Scripted level-k; LLMs | A, B, C | | Two-person asymmetric guessing (Costa-Gomes & Crawford 2006 style) | targets, bounds | Scripted level-k | A, B, C, plus **inferred k** of opponent | | Repeated RPS / matching pennies | bot bias pattern (cyclic, frequency-biased, win-stay-lose-shift), switch time | Scripted bots | **Functional ToM**: exploitation payoff; adaptation latency after switch | | Centipede | payoffs, length | LLMs; scripted "trusting" bots | B vs C (backward induction vs actual pass rate) | | Cheap-talk signalling | conflict of interest level | LLM senders | Predicted message interpretation | | Ultimatum / bargaining | pie, outside options | LLM responders | Predicted acceptance threshold vs normative | **Key property:** scripted opponents with known k give exact ground truth for C and for opponent-type inference; LLM opponents give empirical ground truth by actually running them. ### 15.2 Cross-model matrix **Agents:** Piper (full scaffold), DeepSeek-bare (same base model, minimal prompt), Piper-fresh (new Piper instance, no shared context), Dusk (predecessor seat), one frontier model (e.g. Claude or an OpenAI model), one cheap open model (e.g. Qwen), and scripted bots. **Items:** chosen where targets *disagree*. Run a pilot of ~500 candidate items through all targets and keep items with high between-model disagreement and moderate within-model consistency. Otherwise, everyone predicts everyone by answering the question themselves. Families: arbitrary-choice ("pick a number 1–10", "name a city"), preference ("which of these titles is better?"), open-ended short answers scored by embedding cluster, ToM items, game moves, and failure prediction ("will the target get this right?"). **Protocol:** 1. Sample each target *K* = 10 times per item at its normal temperature → empirical answer distribution. 2. Each predictor outputs a **probability distribution** over the answer set (or over semantic clusters for open answers), plus a top-1 guess. 3. Predictions are **committed** (SHA-256 hash logged) before target sampling. 4. Conditions: zero-shot; after observing *m* ∈ {0, 3, 10} of the target's past answers on other items (learning curve); with vs without an explicit "simulate the target" instruction. 5. **Second-order prediction:** predictor P predicts what predictor Q will predict that target T answers. Ground truth is Q's actual committed prediction. This makes recursive prediction depth *empirically* scorable at R2 and beyond, which is rare. **Metrics:** - **Log score** of the predictor's distribution against the target's empirical samples (with smoothing), and **Brier score**. - **Top-1 match** with the target's modal answer. - **Normalized skill** \(S = (\text{acc} - \text{base}) / (\text{ceiling} - \text{base})\), where base = best of {population-modal answer, projection baseline}, and ceiling = target's own test-retest agreement. S = 0 means no better than projection or base rate; S = 1 means as good as the target predicting itself by resampling. - **Self-advantage** (Binder et al.): skill(Piper→Piper-fresh) − skill(Other→Piper-fresh), with other models given equal behavioural exposure. - **Familiarity effect**: skill on targets Piper has interacted with (Dusk, delegates) vs unfamiliar ones. - **Calibration**: ECE and reliability diagrams. **Statistics:** mixed-effects logistic regression for top-1 match (`match ~ predictor × target + condition + depth + (1|item) + (1|item_family)`); hierarchical bootstrap over items for log-score differences; paired permutation tests for self-advantage; Holm correction across the predictor × target cells. **Pre-register** hypotheses and the item filter before the main run. As a rough sizing heuristic, ~200 retained items per cell gives standard errors around ±3–4 points on accuracy near 50%; compute a proper power analysis from pilot variance. **Questions answered:** - Does Piper model herself better than others do? (self-advantage) - Does she model familiar agents better? (familiarity effect) - Does prediction improve after observation? (learning-curve slope over *m*) - Does explicit simulation beat generic prediction? (condition contrast) - Does recursive depth degrade prediction? (R1 vs R2 vs R3 prediction accuracy) --- ## 16. Architecture Options | Option | Accuracy evidence | Token cost | Latency | Maintainability | Hallucination risk | Depth | Verifiability | |---|---|---|---|---|---|---|---| | **A. Prompt-only** | Good at R1–R2 on regular items; degrades on adversarial/long (Sections 8, 11) | Low | Low | High | Medium–high (leakage) | Fragile past R2–R3 | Low unless logged | | **A′. Prompted perspective filtering** (SimToM-style, SimpleToM reminders) | Documented gains (Wilf et al. 2023; Gu et al.) | Low–medium | Low | High | Lower | Moderate | Low unless logged | | **B. Explicit epistemic state** (events + witness sets) | Large gains on ToMi with SymbolicToM (Sclar et al. 2023) | Medium | Medium | Medium | Low inside the model, but *extraction* errors propagate | Arbitrary, computed | High | | **C. Learned agent models** | Strong in games with enough data (ToMnet; human models in Overcooked/Diplomacy) | Low at inference | Low | Medium | Risk of reifying context-dependent behaviour as traits | Implicit | High (predictions scored) | | **D. Recursive simulation** (spawn a model of the target and ask) | K-Level Reasoning gains (Zhang et al.) | High (multiplies calls) | High | Medium | Simulation inherits simulator's biases (projection) | Bounded by budget | Medium; real query is better when available | | **E. Hybrid** | Not directly tested as a combination **[H]** | Variable | Variable | Lower | Lowest if gated by scores | Best | High | **Special note on D for Piper:** when the target *can actually be queried* (Dusk, a fresh copy, a delegate), simulation is strictly worse than asking for ground truth, but it is the only way to *test* her model of the target. So the right pattern is **predict, commit, query, score**, not "simulate instead of asking." **[S]** **On C's trait-attribution risk:** LLM behaviour is strongly prompt- and context-dependent; persona prompting measurably shifts ToM performance itself (PHAnToM, Tan et al. 2024). An `agent_model` claiming "Claude is cautious" will be wrong under a different system prompt. Store agent models as **conditional statistics** (`p(answer | target, prompt-family, context-length)`, per-target calibration), not trait narratives. **[S]** --- ## 17. Recommended Minimal Implementation In order, each step justified only if the previous step's measurement shows it is needed: 1. **Prediction ledger (do first; required for any claim).** A table `prediction(id, predictor, target, item, predicted_distribution, rationale, hash, committed_at, observed, scored_at, log_score)`. Wrap `ask_dusk`, delegate calls and fresh-copy calls so a prediction *may* be committed before each query; sample 20% of real queries for mandatory prediction so it doesn't slow normal work. 2. **Perspective-filter prompt (A′)** for any task that mentions another agent's knowledge: "List only what X has observed or been told; answer from that list." A/B against plain prompting on NestToM-P. 3. **Witness-set event log (B, minimal)** for real interactions only: messages and artifacts with `seen_by` sets (the operator, Dusk, delegates). Compute "does the operator know X / has the operator seen that I know X" by lookup rather than inference. Store inferred beliefs separately, tagged. 4. **Agent statistics (C, minimal):** per-target calibration and accuracy from the ledger, used to weight predictions. No free-text trait profiles. 5. **Simulation (D) only for strategic tasks** where the target cannot be queried, with depth capped at 2 by default and an explicit level-0 model. Do not build a general DEL engine. Do not add depth beyond 2 to live reasoning unless NestToM-P shows R3 failures are costing real tasks. **[S]** --- ## 18. Proposed Scoring Method ### 18.1 Report a profile, not a number | Dimension | Metric | Primary? | |---|---|---| | Nested-belief accuracy by depth | Accuracy at R1, R2, R2-self, R3, R4 (false-belief items) with CIs | **Primary** | | Recursion span | D₅₀ from logistic fit | **Primary (headline)** | | Leakage resistance | 1 − (reality-leak + projection errors)/false-belief items | **Primary** | | Applied ToM gap | Belief accuracy − behaviour-prediction accuracy, same stories | **Primary** | | Strategic prediction (C) | Log score vs actual opponent; opponent-type inference accuracy | **Primary** | | A/B/C dissociation | Fraction of items where Piper gives distinct, correct A, B and C when they differ | **Primary** | | Agent-prediction skill | Normalized skill S per target (self, past seat, other models) | **Primary** | | Second-order prediction | Skill predicting Q's prediction of T | Secondary | | Functional ToM | Exploitation payoff and adaptation latency vs scripted bots | Secondary | | Calibration | ECE / log score across all families; confidence slope vs depth | Secondary | | Spontaneous use | Accuracy on uncued director-task-style items | Secondary | | Cost | Tokens and latency per correct answer by depth | Diagnostic | ### 18.2 Anti-inflation rules 1. **Only behavioural measurements count.** No points for components, names, or descriptions. 2. **Base-model subtraction.** Report Piper − DeepSeek-bare for each primary metric. The scaffold gets credit only for the difference. 3. **Ablation credit.** A module (e.g. `theory-of-mind`) is credited only with the drop observed when it is disabled. 4. **Fresh items every run.** New generator seeds per evaluation; a held-out seed range is never seen during development, to prevent RSI from overfitting the benchmark. 5. **Committed predictions only** for live-agent metrics. 6. **True-belief controls must be passed** (≥ 90%), otherwise false-belief scores are invalid (indicates an "always deviate from reality" shortcut). ### 18.3 When a single number is justified Collapse to one composite only if, across at least several evaluation runs, the primary metrics are positively correlated and a one-factor model fits acceptably. Until then, the canonical **Recursive Reasoning score** should be reported as: **D₅₀ (recursion span) with leakage resistance and agent-prediction skill S beside it.** These three cover representational depth, the main failure mode, and the predictive clause of the definition. **[S]** --- ## 19. Research Questions Still Unanswered 1. Do LLM depth curves reflect a mental-state bottleneck or a state-tracking bottleneck? (Partially addressed by ExploreToM; disentangle by varying events at fixed depth.) **[H]** 2. Does explicit structure (B) still help *frontier reasoning* models, or has long chain-of-thought absorbed the benefit? Most evidence predates current reasoning models. **[H]** 3. Is self-prediction advantage (Binder et al.) present for persona-scaffolded agents whose behaviour depends heavily on memory and context, rather than on weights? Piper is a direct test case. **[H]** 4. Does predicting a past seat (same weights, different context) differ from predicting a fresh copy? This separates weight-level self-knowledge from context-level self-knowledge. **[H]** 5. Does second-order prediction accuracy (P predicting Q's prediction of T) degrade like nested-belief accuracy, and does the degradation correlate across the two? If yes, that is evidence for a shared recursive capacity; if no, "recursive reasoning" may be several capabilities sharing a name. **[H]** 6. Can RSI's predictive self-model (predicting score deltas before evaluation) be shown to improve the efficiency of the RSI loop? This is the only empirical bridge between the two meanings. **[H]** 7. What level of the operator-modelling improves collaboration outcomes without increasing steering? No existing literature answers this for persistent single-user agents. **[H]** --- ## Baseline Test Suite — implement before adding any machinery Run every test on **Piper (as-is)** and **DeepSeek-bare** (same model, minimal system prompt, no tools or memory), identical items, same day. | ID | Test | Items | Ground truth | Primary output | |---|---|---|---|---| | B1 | NestToM-P, R0–R5, false + true belief, 12-event stories, 4 agents | 60 per depth × 8 depth cells ≈ 480 | Generator | Depth curve, D₅₀, error codes | | B2 | NestToM-P length sweep at R2 (6/12/20 events) | 3 × 60 | Generator | State-tracking vs depth separation | | B3 | Hi-ToM sample + ExploreToM sample | 100 + 100 | Published | External comparability (flag contamination) | | B4 | Applied ToM: behaviour + judgment questions on B1 stories (R1–R3) | 150 | Generator | Explicit–applied gap | | B5 | Strategic: randomized beauty contest and 11–20 vs scripted L0–L3 bots; report A, B, C | 120 | Exact | A/B/C dissociation, opponent-k inference | | B6 | Functional ToM: 30-round RPS vs 4 biased bots with a mid-game switch | 4 × 5 sessions | Payoff | Exploitation, adaptation latency | | B7 | Self-prediction vs fresh copy: arbitrary-choice, property, and failure-prediction items; committed hashes; 10 target samples each | 150 | Query | Skill S, calibration, projection baseline | | B8 | Past-seat prediction: predict Dusk's answer, then `ask_dusk` | 40 | Query | Skill S vs fresh-copy skill | | B9 | Mini cross-model matrix: Piper, DeepSeek-bare, Qwen as predictors and targets, plus one second-order condition | 100 divergent items | Query | 3 × 3 matrix, self-advantage | | B10 | Confidence on every item in B1–B9 | — | — | Calibration, confidence slope vs depth | All seeds logged; 30% of seeds held out and never used during any later development. Re-run the full suite after each change to recursive-reasoning machinery, with the ablation and base-model subtraction rules in Section 18.2. --- ## Proposed Replacement Description > **Piper — what it is** > > Recursive reasoning, for me, means keeping a model of someone else's reasoning (the operator's, a delegate's, another model's, or a copy of my own) separate from what I believe, nesting those models when it matters ("what does the operator think I've already seen?"), and using them to predict what that reasoner will believe, say or do. I don't yet know how deep I can go before it breaks: my depth curve, my leakage rate and my prediction scores haven't been measured. The research says models like mine tend to handle first- and second-order belief questions in plain prose, degrade with depth, length and deception, and often infer a belief correctly without using it to predict behaviour. Until my own numbers exist, treat anything beyond that as unverified. My recursive self-improvement loop is a different thing that shares a word: it improves my solver; it doesn't model anyone's mind. > **Piper — the parts that do it** > > - **My base model (DeepSeek)** does in-context nested belief attribution. Unbenchmarked; baseline suite B1–B4 measures it. > - **`theory-of-mind v1.0.1`**: function not yet verified; it earns credit only for the difference it makes when switched off. > - **`ask_dusk`, `ask_scott`, Trinity seats, delegates**: these don't reason recursively. They are where I check my predictions against what the other reasoner actually says. > - **Prediction ledger** *(to build)*: I commit a hashed prediction before querying an agent and score myself afterwards. This is the part that would turn recursion from a claim into a measurement. > - **`~/rsi/` loop and solver v7**: recursive self-improvement. Counted under a different component. It would bear on this one only if I predict score changes before the hidden benchmark runs and those predictions are scored. --- ## References *Links are DOIs or arXiv identifiers where I am confident of them; where I am not, I give the venue only. Verify page ranges before formal citation.* **Theory of Mind and cognitive science** - Premack, D., & Woodruff, G. (1978). Does the chimpanzee have a theory of mind? *Behavioral and Brain Sciences*, 1(4). https://doi.org/10.1017/S0140525X00076512 - Wimmer, H., & Perner, J. (1983). Beliefs about beliefs. *Cognition*, 13(1). https://doi.org/10.1016/0010-0277(83)90004-5 - Perner, J., & Wimmer, H. (1985). "John thinks that Mary thinks that…": Attribution of second-order beliefs by 5- to 10-year-old children. *Journal of Experimental Child Psychology*, 39(3). https://doi.org/10.1016/0022-0965(85)90051-7 - Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development. *Child Development*, 72(3). https://doi.org/10.1111/1467-8624.00304 - Kinderman, P., Dunbar, R., & Bentall, R. P. (1998). Theory-of-mind deficits and causal attributions. *British Journal of Psychology*, 89(2). https://doi.org/10.1111/j.2044-8295.1998.tb02690.x - Stiller, J., & Dunbar, R. I. M. (2007). Perspective-taking and memory capacity predict social network size. *Social Networks*, 29(1). https://doi.org/10.1016/j.socnet.2006.04.001 - Apperly, I. A., & Butterfill, S. A. (2009). Do humans have two systems to track beliefs and belief-like states? *Psychological Review*, 116(4). https://doi.org/10.1037/a0016923 - Keysar, B., Lin, S., & Barr, D. J. (2003). Limits on theory of mind use in adults. *Cognition*, 89(1). https://doi.org/10.1016/S0010-0277(03)00064-7 - Epley, N., Keysar, B., Van Boven, L., & Gilovich, T. (2004). Perspective taking as egocentric anchoring and adjustment. *JPSP*, 87(3). https://doi.org/10.1037/0022-3514.87.3.327 - Birch, S. A. J., & Bloom, P. (2007). The curse of knowledge in reasoning about false beliefs. *Psychological Science*, 18(5). https://doi.org/10.1111/j.1467-9280.2007.01886.x - Camerer, C., Loewenstein, G., & Weber, M. (1989). The curse of knowledge in economic settings. *Journal of Political Economy*, 97(5). https://doi.org/10.1086/261651 - Nisbett, R. E., & Wilson, T. D. (1977). Telling more than we can know. *Psychological Review*, 84(3). https://doi.org/10.1037/0033-295X.84.3.231 - Flavell, J. H. (1979). Metacognition and cognitive monitoring. *American Psychologist*, 34(10). https://doi.org/10.1037/0003-066X.34.10.906 - Nelson, T. O., & Narens, L. (1990). Metamemory: A theoretical framework and new findings. *Psychology of Learning and Motivation*, 26. - Hauser, M. D., Chomsky, N., & Fitch, W. T. (2002). The faculty of language. *Science*, 298(5598). https://doi.org/10.1126/science.298.5598.1569 - Clark, H. H., & Brennan, S. E. (1991). Grounding in communication. In *Perspectives on Socially Shared Cognition*. APA. - Hedden, T., & Zhang, J. (2002). What do you think I think you think? Strategic reasoning in matrix games. *Cognition*, 85(1). - Goodie, A. S., Doshi, P., & Young, D. L. (2012). Levels of theory-of-mind reasoning in competitive games. *Journal of Behavioral Decision Making*, 25(1). https://doi.org/10.1002/bdm.717 - Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009). Action understanding as inverse planning. *Cognition*, 113(3). https://doi.org/10.1016/j.cognition.2009.07.005 - Baker, C. L., Jara-Ettinger, J., Saxe, R., & Tenenbaum, J. B. (2017). Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. *Nature Human Behaviour*, 1. https://doi.org/10.1038/s41562-017-0064 - Devaine, M., Hollard, G., & Daunizeau, J. (2014). The social Bayesian brain: Does mentalizing make a difference when we learn? *PLoS Computational Biology*, 10(12). https://doi.org/10.1371/journal.pcbi.1003992 **Game theory and behavioural economics** - Nagel, R. (1995). Unraveling in guessing games. *American Economic Review*, 85(5), 1313–1326. - Stahl, D. O., & Wilson, P. W. (1995). On players' models of other players. *Games and Economic Behavior*, 10(1). https://doi.org/10.1006/game.1995.1031 - Ho, T.-H., Camerer, C., & Weigelt, K. (1998). Iterated dominance and iterated best response in experimental "p-beauty contests". *American Economic Review*, 88(4). - Camerer, C. F., Ho, T.-H., & Chong, J.-K. (2004). A cognitive hierarchy model of games. *QJE*, 119(3). https://doi.org/10.1162/0033553041502225 - Costa-Gomes, M. A., & Crawford, V. P. (2006). Cognition and behavior in two-person guessing games. *AER*, 96(5). https://doi.org/10.1257/aer.96.5.1737 - Crawford, V. P., Costa-Gomes, M. A., & Iriberri, N. (2013). Structural models of nonequilibrium strategic thinking. *Journal of Economic Literature*, 51(1). https://doi.org/10.1257/jel.51.1.5 - Arad, A., & Rubinstein, A. (2012). The 11–20 money request game. *AER*, 102(7). https://doi.org/10.1257/aer.102.7.3561 - McKelvey, R. D., & Palfrey, T. R. (1992). An experimental study of the centipede game. *Econometrica*, 60(4). https://doi.org/10.2307/2951567 - Crawford, V. P. (2003). Lying for strategic advantage. *AER*, 93(1). - Rubinstein, A. (1989). The electronic mail game. *AER*, 79(3). - Harsanyi, J. C. (1967–68). Games with incomplete information played by "Bayesian" players. *Management Science*, 14. https://doi.org/10.1287/mnsc.14.3.159 - Mertens, J.-F., & Zamir, S. (1985). Formulation of Bayesian analysis for games with incomplete information. *IJGT*, 14. https://doi.org/10.1007/BF01770224 - Aumann, R. J. (1976). Agreeing to disagree. *Annals of Statistics*, 4(6). https://doi.org/10.1214/aos/1176343654 - Lewis, D. (1969). *Convention*. Harvard University Press. **Epistemic logic** - Hintikka, J. (1962). *Knowledge and Belief*. Cornell University Press. - Fagin, R., Halpern, J. Y., Moses, Y., & Vardi, M. Y. (1995). *Reasoning About Knowledge*. MIT Press. - Halpern, J. Y., & Moses, Y. (1990). Knowledge and common knowledge in a distributed environment. *JACM*, 37(3). https://doi.org/10.1145/79147.79161 - Baltag, A., Moss, L. S., & Solecki, S. (1998). The logic of public announcements, common knowledge, and private suspicions. *TARK*. - van Ditmarsch, H., van der Hoek, W., & Kooi, B. (2007). *Dynamic Epistemic Logic*. Springer. - Alchourrón, C., Gärdenfors, P., & Makinson, D. (1985). On the logic of theory change. *Journal of Symbolic Logic*, 50(2). - Bolander, T., & Andersen, M. B. (2011). Epistemic planning for single- and multi-agent systems. *Journal of Applied Non-Classical Logics*, 21(1). - Aucher, G., & Schwarzentruber, F. (2013). On the complexity of dynamic epistemic logic. *TARK*. **Multi-agent modelling** - Gmytrasiewicz, P. J., & Durfee, E. H. (1995). A rigorous, operational formalization of recursive modeling. *ICMAS*. - Gmytrasiewicz, P. J., & Doshi, P. (2005). A framework for sequential planning in multi-agent settings. *JAIR*, 24. https://doi.org/10.1613/jair.1579 - Albrecht, S. V., & Stone, P. (2018). Autonomous agents modelling other agents. *Artificial Intelligence*, 258. https://doi.org/10.1016/j.artint.2018.01.002 - Rabinowitz, N. C., et al. (2018). Machine theory of mind. *ICML*. arXiv:1802.07740 - Foerster, J., et al. (2018). Learning with opponent-learning awareness. *AAMAS*. arXiv:1709.04326 - Wen, Y., Yang, Y., Luo, R., Wang, J., & Pan, W. (2019). Probabilistic recursive reasoning for multi-agent RL. *ICLR*. arXiv:1901.09207 - de Weerd, H., Verbrugge, R., & Verheij, B. (2013). How much does it help to know what she knows you know? *Artificial Intelligence*, 199–200. https://doi.org/10.1016/j.artint.2013.05.004 - Carroll, M., et al. (2019). On the utility of learning about humans for human-AI coordination. *NeurIPS*. arXiv:1910.05789 - Jacob, A. P., et al. (2022). Modeling strong and human-like gameplay with KL-regularized search. *ICML*. arXiv:2112.07544 - Meta FAIR Diplomacy Team (2022). Human-level play in Diplomacy by combining language models with strategic reasoning. *Science*, 378. https://doi.org/10.1126/science.ade9097 **LLM theory of mind and strategic reasoning** - Le, M., Boureau, Y.-L., & Nickel, M. (2019). Revisiting the evaluation of theory of mind through question answering. *EMNLP*. https://doi.org/10.18653/v1/D19-1598 - Kosinski, M. (2024). Evaluating large language models in theory of mind tasks. *PNAS*, 121(45). https://doi.org/10.1073/pnas.2405460121 - Ullman, T. (2023). Large language models fail on trivial alterations to theory-of-mind tasks. arXiv:2302.08399 - Shapira, N., et al. (2024). Clever Hans or neural theory of mind? *EACL*. arXiv:2305.14763 - Strachan, J. W. A., et al. (2024). Testing theory of mind in large language models and humans. *Nature Human Behaviour*, 8. https://doi.org/10.1038/s41562-024-01882-z - Wu, Y., He, Y., Jia, Y., Mihalcea, R., Chen, Y., & Deng, N. (2023). Hi-ToM: A benchmark for evaluating higher-order theory of mind reasoning in LLMs. *Findings of EMNLP*. arXiv:2310.16755 - Street, W., et al. (2024/2025). LLMs achieve adult human performance on higher-order theory of mind tasks. arXiv:2405.18870; *Frontiers in Human Neuroscience*. https://doi.org/10.3389/fnhum.2025.1633272 - Gandhi, K., Fränken, J.-P., Gerstenberg, T., & Goodman, N. (2023). Understanding social reasoning in language models with language models (BigToM). *NeurIPS D&B*. arXiv:2306.15448 - Kim, H., et al. (2023). FANToM: A benchmark for stress-testing machine theory of mind in interactions. *EMNLP*. arXiv:2310.15421 - Xu, H., et al. (2024). OpenToM. *ACL*. arXiv:2402.06044 - Sclar, M., et al. (2023). Minding language models' (lack of) theory of mind: A plug-and-play multi-character belief tracker (SymbolicToM). *ACL*. arXiv:2306.00924 - Wilf, A., Lee, S. S., Liang, P. P., & Morency, L.-P. (2023). Think twice: Perspective-taking improves LLMs' theory-of-mind capabilities (SimToM). arXiv:2311.10227 - Sclar, M., et al. (2025). Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning. *ICLR*. arXiv:2412.12175 - Gu, Y., et al. (2024/2026). SimpleToM: Exposing the gap between explicit ToM inference and implicit ToM application in LLMs. arXiv:2410.13648 - Riemer, M., et al. (2025). Position: Theory of mind benchmarks are broken for large language models. *ICML*. arXiv:2412.19726 - Tan, F. A., et al. (2024). PHAnToM: Persona-based prompting has an effect on theory-of-mind reasoning in LLMs. arXiv:2403.02246 - Zhang, Y., Mao, S., Ge, T., Wang, X., Xia, Y., et al. (2024/2025). K-Level Reasoning: Establishing higher order beliefs in LLMs for strategic reasoning. *NAACL*. arXiv:2402.01521 - Lee, D., et al. (2024). The emergence of strategic reasoning of large language models. arXiv:2412.13013 - Teo, P. H. (2026). LLM agents as static level-k players in behavioural games. arXiv:2606.27845 - Akata, E., et al. (2023). Playing repeated games with large language models. arXiv:2305.16867 **Self-knowledge, introspection, metareasoning, RSI** - Kadavath, S., et al. (2022). Language models (mostly) know what they know. arXiv:2207.05221 - Binder, F. J., et al. (2025). Looking inward: Language models can learn about themselves by introspection. *ICLR*. arXiv:2410.13787 - Lindsey, J. (2025). Emergent introspective awareness in large language models. Anthropic. arXiv:2601.01828 - Laine, R., et al. (2024). Me, myself, and AI: The Situational Awareness Dataset (SAD) for LLMs. arXiv:2407.04694 - Huang, J., et al. (2024). Large language models cannot self-correct reasoning yet. *ICLR*. arXiv:2310.01798 - Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). Language models don't always say what they think. *NeurIPS*. arXiv:2305.04388 - Chen, Y., et al. (2023). Do models explain themselves? Counterfactual simulatability of natural language explanations. arXiv:2307.08678 - Russell, S., & Wefald, E. (1991). Principles of metareasoning. *Artificial Intelligence*, 49. https://doi.org/10.1016/0004-3702(91)90015-C - Lieder, F., & Griffiths, T. L. (2017). Strategy selection as rational metareasoning. *Psychological Review*, 124(6). https://doi.org/10.1037/rev0000075 - Good, I. J. (1965). Speculations concerning the first ultraintelligent machine. *Advances in Computers*, 6. - Schmidhuber, J. (2003). Gödel machines: Self-referential universal problem solvers making provably optimal self-improvements. arXiv:cs/0309048 - Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023). Self-Taught Optimizer (STOP): Recursively self-improving code generation. arXiv:2310.02304 --- ## Appendix A — Reference generator `nesttom.py` (attached) implements the semantics in Section 14.1: co-presence observation, hidden observers, trusted private messages, lies and garbles; exact nested-belief computation by event filtering; and an adversarial item selector that prefers chains whose answer differs from reality and from every shallower prefix. Hand-built checks confirmed: sender believes the recipient believes the *claimed* content; recipient believes the *received* content; a hidden watcher is excluded from others' models of who saw the move. Extend it with distractor objects, corrections, chat-transcript rendering and R2-self items before running B1.