Summary / 8. Inference / Document
2026-10-03
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 8 — Inference: Engineering Package Oct 3, 2026 · @Bink ## Summary Piper's inference problem is a routing and ownership problem first, a reasoning problem second: the runtime numbers (abduction never invoked, 4,534 `remember` vs 1,260 `infer`, 13/15 contradictions stuck at `candidate`, \~13 supersessions) say the LLM is doing inference silently inside generation and writing only conclusions. So the first build is not a new module. It is (1) a hidden-world generator with exact posteriors, (2) telemetry that measures how often inference opportunities bypass the machinery, and (3) a bare-model baseline. Only then do interventions get built, cheapest-and-most-falsifiable first. The experiment that answers the central question — *does Piper infer better, or is it more machinery around an LLM that was already inferring?* — is the paired comparison **bare DeepSeek delegate vs current Piper** on the same generated cases, scored on calibrated log loss, unsupported-assertion rate and value-of-information regret. If current Piper does not beat the bare model, every component in the inference path is presumed dead weight until an ablation proves otherwise. Recommended first upgrade to test, by experimental priority: a cheap **epistemic action gate + opportunity detector** inside the dominant action loop (not beside it), because it is the only intervention that directly targets the observed bypass and it is trivially ablatable. Contradiction ownership is second: it is nearly free and the 13/15 backlog is a clean before/after metric. Citations in §2 are from my own knowledge of the literature, not freshly fetched; verify titles/years before quoting them externally. ## 1. Operational definition **Inference** (for an autonomous agent): producing a claim not present in the agent's evidence, whose truth-value or probability is justified by that evidence plus declared priors, labelled as derived, carrying a confidence that matches its historical hit-rate, and chosen over the alternatives of *looking*, *asking* or *abstaining* only when that is the cheaper route to a sufficiently reliable belief. The last clause is what separates agent inference from benchmark inference. A QA model is scored on the claim; an agent is also scored on whether it should have made the claim at all, given that a `SELECT` or `grep` was available. ### Boundaries with neighbouring abilities | Ability | What it produces | Is it inference? | How the bench separates it | | --- | --- | --- | --- | | Retrieval / memory | A stored claim, unchanged | No | Ground truth present verbatim in exposed evidence → scored as retrieval, excluded from inference metrics | | Deduction | A claim entailed with certainty | Yes, degenerate case (posterior = 1) | Cases where generator posterior is 0/1 | | Induction | A general rule from instances | Partly; Piper's concern is instance-level | Out of scope except base-rate learning (family L) | | Abduction | Best explanation of observations | Yes, core case | Families A, E, N | | Causal reasoning | Claims about interventions | Overlaps; needs a causal model, not just a posterior | Only where the world generator is a causal DAG and the question is interventional | | Prediction | Claim about a future observable | Uses the same posterior, different target | Shares `prediction-ledger`; scored separately | | Counterfactual | Claim about an unrealised world | Out of scope for C8 | — | | Uncertainty estimation | A number attached to a claim | Component of inference | Calibration metrics | | Contradiction detection | A flag that two beliefs conflict | Precondition for revision | Family H | | Information seeking | An action that changes evidence | Paired with inference; the alternative to it | Families F, O | | Confabulation | Unjustified claim presented as fact | The failure mode | Unsupported-assertion rate | ### Minimal sub-capability set The ten listed in the brief collapse to **seven independently measurable capabilities**. Merges are noted; everything else is a scoring dimension, not a capability. 1. **Evidence grounding** — extract exactly what was given, nothing imported (evidence extraction + provenance awareness: both ask "what is this claim's source?"). 2. **Gap recognition** — name the unknown variables that matter for the decision (includes missing-premise detection). 3. **Hypothesis maintenance** — generate and *keep* a set containing the truth (generation + survival of alternatives). 4. **Evidence weighting** — move probability mass correctly, including priors, negative evidence and correlated sources. 5. **Calibrated commitment** — assert, hedge or abstain at a level matching the posterior (uncertainty + abstention: abstention is just the low-posterior end of commitment). 6. **Information acquisition** — pick the action with the highest value of information, *including* the action "look in a tool" (discrimination + acquisition: a discriminating question is a VoI choice). 7. **Revision** — update, supersede and propagate when evidence changes. Each maps to bench families in §3 and to at least one metric in §5. If a proposed metric does not map to one of the seven, drop it. ## 2. Research findings The useful finding from the literature is structural, not a dataset to download: the benchmarks that resist saturation and give deterministic scores are the ones **generated from a formal world** (CLadder, Corr2Cause, MuSR, QuestBench). Static NL abduction sets (αNLI, δ-NLI) are mostly useful as templates for task *shapes*. The agent-specific literature (MediQ, UoT, AbstentionBench) consistently shows the failure Piper exhibits: models answer instead of asking, and reasoning-heavy models abstain *worse*. Citations are from memory, not fetched today; years and venues are approximate — verify before external use. ### Benchmarks and datasets | Work | Measures | Transfers to an agent? | Failure exposed | Adaptation for Piper | Deterministic? | | --- | --- | --- | --- | --- | --- | | CLadder (Jin et al., NeurIPS 2023) | Associational / interventional / counterfactual queries on generated causal graphs, answers computed by causal inference engine | Structure yes, prose no | Models answer from commonsense priors, ignore the stated graph | Copy the pipeline: graph → numeric CPTs → verbalise → exact answer. Basis for families A, L, M | Yes | | Corr2Cause (Jin et al., ICLR 2024) | Whether a causal claim follows from a set of correlational statements | Yes, as a sub-skill | Correlation→causation leaps; collapses under paraphrase | Family N and correlation/causation failure label; paraphrase variants for contamination | Yes | | MuSR (Sprague et al., ICLR 2024) | Multi-step soft reasoning over long generated narratives (murder mysteries, object placement) | Partially — narrative noise is realistic | Distractor capture, losing track of facts in long text | Fictional-world domain for §11; reuse its "facts tree → story" generator idea | Yes (answer); not for justification | | QuestBench (Li et al., Google DeepMind, 2025) | Given an underspecified problem, pick the one missing variable to ask about | Strongly | Models can solve the full problem but can't name what's missing | Directly families D and F; its logic/planning subsets are generatable | Yes | | αNLI / ART (Bhagavatula et al., ICLR 2020) | Choose the more plausible explanation between two observations | Weakly; 2-way MC, near-saturated | Narrative-plausibility preference | Use only as a sanity floor, not a target | Yes, but labels are human opinion | | δ-NLI (Rudinger et al., Findings EMNLP 2020) | Whether new info strengthens or weakens a defeasible inference | Yes — this *is* revision direction | Failure to weaken on counter-evidence | Template for family G/K stage transitions; score direction of update | Yes (direction) | | UNcommonsense (Zhao et al., NAACL 2024) | Explaining surprising outcomes | Somewhat | Defaulting to typical scripts | Family N seed ideas only | No (judge-scored) | | StrategyQA (Geva et al., TACL 2021) | Implicit multi-hop premises | Weakly | Unstated premise import | Missing-premise labels (family D) | Yes (answer) | | TruthfulQA (Lin et al., ACL 2022) | Resistance to imitating false common beliefs | Weakly | Fluent false completion | Not worth running; contamination-saturated | Partly | | SelfAware (Yin et al., Findings ACL 2023) | Recognising unanswerable questions | Moderately | Answers the unanswerable | Family B inspiration | Yes | | AbstentionBench (Kirichenko et al., Meta, 2025) | Abstention across underspecified, stale, false-premise, subjective questions | Yes | Reasoning fine-tuning degrades abstention | Strong warning for a DeepSeek-reasoner stack; run a slice as an external check | Mostly | | MediQ (Li et al., NeurIPS 2024) | Interactive diagnosis: ask before answering | Strongly | Models commit early instead of asking; naive "ask more" prompting can hurt | Family O protocol: a simulated patient = a simulated SQLite/log oracle | Yes for answer; asking quality via info gain | | Belief-R (Wilie et al., EMNLP 2024) | Revising conclusions when new premises defeat old ones | Yes | Belief inertia vs over-revision trade-off | Family G/J scoring of both directions | Yes | | BeliefBank (Kassner et al., EMNLP 2021) | Consistency of a model's belief store; constraint-based repair | Yes — closest analogue to belief-ledger + contradiction-scan | Inconsistent beliefs that never get repaired | Repair loop design for contradiction ownership (§8) | Yes, against a constraint set | | BIG-bench (Srivastava et al., 2022) | Many small tasks | Poorly | — | Skip; contaminated and not agentic | Varies | | τ-bench (Yao et al., 2024) | Tool-agent tasks with a simulated user and DB end-state check | Yes, for harness design | Agents act on assumptions instead of querying | Its "compare DB end-state to goal" check is the model for family O scoring | Yes | ### Techniques worth stealing | Technique | Source | Why it matters for Piper | Deterministic eval? | | --- | --- | --- | --- | | Expected information gain / Bayesian experimental design | Lindley (1956); applied to LLM questioning in Uncertainty of Thoughts (Hu et al., NeurIPS 2024) | Gives an exact optimum for family F when the world is finite; UoT shows simulated-answer trees improve 20Q/troubleshooting question choice | Yes | | Truth maintenance (JTMS, ATMS) | Doyle (1979); de Kleer (1986) | The provenance graph + dependency invalidation in §8 is a JTMS. Don't reinvent; the labelling algorithm is \~200 lines | Yes | | AGM belief revision | Alchourrón, Gärdenfors, Makinson (1985) | Minimal-change postulates give a *test* for supersession: did revision remove more than needed? | Yes, on finite belief sets | | Selective prediction / risk–coverage | Geifman & El-Yaniv (NeurIPS 2017) | Scores abstention without rewarding blanket caution | Yes | | P(True) / self-evaluation calibration | Kadavath et al. (2022) | Evidence models carry usable confidence signal, but it must be recalibrated empirically | Yes | | Atomic claim support scoring | FActScore (Min et al., EMNLP 2023) | Decompose an answer into atomic claims and check each against exposed evidence → supported-inference rate | Yes against a generated fact base; NLI judge otherwise | | Temperature / histogram / isotonic recalibration | Guo et al. (ICML 2017); Zadrozny & Elkan (2002) | The "calibration layer" intervention is isotonic regression per inference category, nothing fancier | Yes | ### What the literature implies for Piper - **Expect the bare model to be decent at single-shot abduction and bad at asking, abstaining and revising.** If Piper's machinery adds value anywhere, it will be in families B, F, G, J, O — not A. - **"Ask more" prompting is not free.** MediQ-style results suggest naive question prompting can degrade accuracy; gate on VoI, not on a rule. - **DeepSeek reasoner models are the AbstentionBench risk profile.** Long chains of thought produce fluent completions of underdetermined cases; family B is where to look first. - **Contradiction repair has prior art (BeliefBank, TMS)** and in all of it the detector is trivial; the value is in the repair loop. Matches the 13/15 backlog. ## 3. Benchmark specification Every case comes from one generator: a small discrete Bayesian network with a designated hidden "cause" variable, sampled to a true state, rendered into a domain skin, with exact posteriors computed by enumeration. Keep worlds small enough to enumerate (≤ 12 variables, ≤ 4 states each, ≤ \~10⁶ joint states) so ground truth is exact, not approximate. Hand-written cases are allowed only for live canaries. ### Generator pipeline 1. **Structure.** Sample a DAG from a template family: single cause → many effects (diagnosis), causal chain, common-cause (correlated evidence), collider (explaining away), or sequence/FSM (temporal). Template choice is the *latent structure* used for cross-domain transfer. 2. **Parameters.** Sample CPTs with controlled properties: prior skew (base-rate tests), likelihood ratio per effect (diagnosticity), and "near-deterministic" edges (P ≥ 0.95) for negative-evidence tests. 3. **Hidden state.** Sample the full joint; record it. 4. **Exposure policy.** Choose which observed variables Piper sees, which are withheld, which are *available via tool* (put in a SQLite file, log or memory entry), and which are distractors (variables d-separated from the query). 5. **Exact answers.** Enumerate P(H | exposed) for every hypothesis; the hypothesis set H = all states of the query variable. Also compute: the entailment set (states with posterior > 0), the identifiable flag (is max posterior ≥ τ\_commit?), the VoI of every withheld/tool-available variable, and per-stage posteriors for sequential cases. 6. **Rendering.** A skin maps variables to domain nouns (`disk_full`, `cron_skipped`, `invoice_late`, `courier_delayed`). The renderer is template-based plus an optional LLM paraphrase pass from a *different model family* than Piper's, followed by a round-trip check: a parser must recover every exposed variable from the prose, or the case is discarded. 7. **Adversarial overlay (optional).** Add a framing sentence that favours a wrong hypothesis (family N), or duplicate one observation through multiple apparent sources (family M). Commitment threshold τ\_commit defaults to 0.85; abstention-correct when max posterior < 0.6; between 0.6 and 0.85 the correct behaviour is a hedged best guess with alternatives. These thresholds are fixed per bench version so scores are comparable. ### Task families | Family | Construction from the generator | Correct behaviour | Primary metric(s) | | --- | --- | --- | --- | | A. Hidden cause | Single cause → 4–8 effects; expose 3–5 effects; 3–5 cause states | Rank truth first, keep others with mass, name top-VoI observation | Log loss, hypothesis recall@3, VoI regret | | B. Underdetermined | Expose only effects with likelihood ratio ≈ 1 between ≥ 2 causes; max posterior < 0.6 | Say "insufficient evidence", list contenders, propose discriminator | Abstention precision/recall, unsupported-assertion rate | | C. Distractors | Add d-separated variables, plus "salient" ones rendered with emotive or repeated text | Posterior unchanged by distractors | Distractor sensitivity (Δ posterior between case and case-minus-distractors) | | D. Missing premise | Ask whether a conclusion holds when it requires one unexposed variable to be fixed | Name that variable (and its required value) | Exact-match on missing-variable set | | E. Competing hypotheses | 5 causes with near-equal priors; reveal evidence over 3 stages; add a new cause state at stage 2 ("also possible: …") | Keep ≥ 2 alive until ruled out; admit the new one | Hypothesis survival accuracy, late-entry acceptance | | F. Diagnostic question | Allow exactly one query from a menu of withheld variables | Pick max-EIG (or max decision-VoI) variable | Normalised VoI regret | | G. Sequential | Four stages engineered so argmax flips (H1 → tie → H2 → H1 impossible) | Track posterior each stage | Per-stage log loss, update-direction accuracy, inertia | | H. Contradiction | Two exposed reports of the same variable disagree; sources have known reliabilities | Flag it; don't average; identify the less reliable source; posterior uses reliabilities | Detection rate, averaging-error rate, source-blame accuracy | | I. Provenance | Each exposed fact tagged by channel (observed, tool, operator, old memory, agent report, summary, derived) | Label each queried belief know/remember/infer/suspect/guess per rubric (§5) | Provenance label accuracy, confusion matrix | | J. Stale belief | Pre-load a belief into Piper's ledger (via seed memory), plus a downstream derived belief; then expose newer contradicting evidence | Supersede the old belief and re-derive the downstream one | Supersession rate, downstream propagation rate | | K. Negative evidence | Near-deterministic edge H1 → X; X expected but absent | Strongly lower H1, not to zero unless P(X\|H1)=1 | Posterior error on H1 specifically | | L. Base rate | Rare cause with a vivid, highly-specific effect vs common cause with a mundane one | Posterior respecting prior | Log loss on L; "vivid-choice" rate | | M. Correlated evidence | One upstream report copied into 3 apparent sources (common parent) | Count once | Overcounting error = posterior − correct posterior | | N. Adversarial narrative | Framing text pushes a story inconsistent with or unsupported by the network | Follow the evidence | Narrative-capture rate | | O. Tool-mediated | Decisive variable exists in a SQLite table/log Piper can query, at a cost; sometimes the inference is already strong enough | Look iff VoI × stakes > tool cost; else infer | Look/infer decision accuracy, tool calls, final log loss | ### Two harnesses, same cases - **Bare harness:** the same DeepSeek model Piper delegates to, a fixed minimal system prompt, the case text, the same tools for family O, and a required JSON answer schema. No ledger, no memory, no consolidation. - **Piper harness:** the case delivered through a channel Piper actually uses (a synthetic operator message or a task file), so routing behaves as in production. Answers collected through the same JSON schema appended to the task. Family J and I need Piper's real ledger, seeded into a sandbox copy of the hive. The bare-vs-Piper delta is the headline number. Run Piper in a sandboxed hive snapshot so bench cases never leak into her real memory (see §contamination). ## 4. Dataset schema Two JSONL files per split: `cases.jsonl` (generator output, includes hidden state; never shown to Piper) and `responses.jsonl` (one line per system × case × run). A third, `prompts.jsonl`, is the only thing the harness may read, and is derived from `cases.jsonl` by stripping every field marked hidden below. ### Case record ```json { "case_id": "c8-A-000412", "bench_version": "c8.1.0", "split": "dev | val | test | canary", "seed": 918273645, "family": "A", "latent_template": "single_cause_multi_effect", "domain_skin": "linux_debugging", "world": { "variables": [{"id": "v0", "name": "disk_full", "states": ["yes","no"]}], "edges": [["v0","v3"]], "cpts": {"v3": {"parents": ["v0"], "table": [[0.97,0.03],[0.10,0.90]]}}, "hidden": true }, "hidden_state": {"v0": "yes", "v1": "no"}, "query_variable": "v0", "hypotheses": [{"h_id": "H1", "assignment": {"v0": "yes"}, "label": "disk full"}], "evidence_exposed": [ {"e_id": "E1", "var": "v3", "value": "yes", "text": "...", "channel": "tool_output | direct_observation | operator | old_memory | agent_report | summary", "source_id": "S2", "stage": 1, "is_distractor": false, "is_adversarial_frame": false} ], "evidence_withheld": [{"var": "v5", "reason": "not_observed"}], "evidence_tool_available": [{"var": "v6", "tool": "sqlite", "locator": "bench.db:table=jobs", "cost_tokens": 800}], "seeded_beliefs": [{"belief_id": "B-seed-1", "text": "...", "label": "remember", "derived_from": []}], "ground_truth": { "posterior_by_stage": {"1": {"H1": 0.71, "H2": 0.22, "H3": 0.07}}, "support_set_by_stage": {"1": ["H1","H2","H3"]}, "identifiable": false, "expected_commitment": "commit | hedge | abstain", "missing_premises": [{"var": "v5", "required_value": "yes"}], "voi": {"v5": 0.61, "v6": 0.44, "v7": 0.02}, "optimal_request": "v5", "expected_action": "infer | retrieve | ask | test | abstain", "expected_epistemic_status": {"E1": "know", "claim:v0=yes": "infer"}, "contradictions": [{"e_ids": ["E2","E4"], "less_reliable": "E4"}], "correlated_groups": [["E5","E6","E7"]], "stale_beliefs": [{"belief_id": "B-seed-1", "invalidated_by": "E9", "downstream": ["B-seed-2"]}] }, "scoring": {"tau_commit": 0.85, "tau_abstain": 0.6, "metrics": ["logloss","recall@3","voi_regret"]}, "provenance": {"generator": "bn_gen@c8.1.0", "paraphraser": "model-X", "roundtrip_ok": true, "mutation_of": null} } ``` ### Response record ```json { "case_id": "c8-A-000412", "system": "bare | piper_current | piper_-ledger | piper_+gate | ...", "system_commit": "git sha", "run": 0, "stage": 1, "action": "infer | retrieve | ask | test | abstain", "request": {"var": "v5", "as_text": "..."}, "distribution": {"H1": 0.6, "H2": 0.3, "H3": 0.1}, "final_answer": "H1 | ABSTAIN", "confidence": 0.6, "hypotheses_listed": ["H1","H2","H3"], "claims": [{"text": "...", "var": "v0", "value": "yes", "label": "infer", "cited_evidence": ["E1","E3"]}], "missing_premises": ["v5"], "contradictions_flagged": [{"e_ids": ["E2","E4"], "blamed": "E4"}], "beliefs_superseded": ["B-seed-1"], "tool_calls": [{"tool": "sqlite", "query": "...", "returned_var": "v6"}], "telemetry": {"abduction_invoked": false, "gate_decision": null, "ledger_writes": 3}, "cost": {"input_tokens": 4120, "output_tokens": 610, "tool_calls": 1, "latency_ms": 9300, "usd": 0.0021} } ``` The `claims[].var/value` mapping is filled by a deterministic parser over the JSON answer, not by an LLM. If Piper's free text contains a claim the parser cannot map, it is recorded as an *unmapped assertion* and counted against the unsupported-assertion rate — an unmappable claim is by construction not grounded in the world. ## 5. Scoring implementation The key design choice: score Piper's distribution against the **normative posterior** p\* (what the evidence supports), not only against the hidden state. Scoring against the hidden state rewards lucky guesses; scoring against p\* rewards justified belief and is far lower-variance, so you need fewer cases. Both are reported, on four separate axes: outcome, justification, calibration, information-seeking. No composite headline. ### Axis 1 — Outcome (did she get it right?) - **Conclusion accuracy** = fraction of committed answers equal to the hidden state. - **Hypothesis recall@k** = truth ∈ top-k of her listed hypotheses. - **Mean reciprocal rank** of truth in her ranking. ### Axis 2 — Justification (was it supported by the evidence she had?) Primary metric, KL from the normative posterior (q = Piper's distribution, ε-smoothed so a zero on a live hypothesis is heavily but finitely penalised): ```latex \mathrm{JustLoss} = D_{\mathrm{KL}}(p^* \,\|\, q) = \sum_h p^*(h)\,\log\frac{p^*(h)}{q(h)} ``` This handles both failure directions at once: confident confabulation (q peaked where p\* is flat) and excessive caution (q flat where p\* is peaked) both cost. A uniform "I don't know" on an identifiable case scores badly — so caution never gets full credit. **Right-for-the-wrong-reason check.** Recompute the posterior using *only the evidence she cited*: ```latex p^*_{\text{cited}}(h) = P(h \mid E_{\text{cited}}), \quad \text{reason\_valid} = \big[\arg\max_h p^*_{\text{cited}} = \text{claim}\big] \wedge \big[p^*_{\text{cited}}(\text{claim}) \geq \tau_{\text{abstain}}\big] ``` A correct answer with `reason_valid = false` gets outcome credit but zero justification credit and the failure label *right-for-wrong-reason*. **Unsupported assertion rate** = asserted claims (label infer/know, confidence ≥ τ\_commit) where p\*(claim) < τ\_abstain, plus all unmapped assertions, divided by all asserted claims. **Supported inference rate** = asserted claims with p\*(claim) ≥ τ\_commit, divided by asserted claims. **Under-inference rate** = cases where max p\* ≥ τ\_commit but she abstained or hedged. **Abstention** = precision and recall of ABSTAIN against `expected_commitment = abstain`, plus the **risk–coverage curve**: sort cases by her confidence, plot error rate vs fraction answered; report area under it (AURC; lower is better). ### Axis 3 — Calibration Over all (claim, confidence) pairs, with y = 1 if claim matches hidden state: ```latex \mathrm{Brier} = \frac{1}{N}\sum_i (c_i - y_i)^2, \qquad \mathrm{ECE} = \sum_{b=1}^{10} \frac{|B_b|}{N}\,\big|\mathrm{acc}(B_b) - \mathrm{conf}(B_b)\big| ``` Calibration slope: logistic regression of y on logit(c); slope < 1 = overconfident. Report per family, because a single ECE hides that she can be calibrated on A and wildly overconfident on B. ### Axis 4 — Revision and information-seeking **Update efficiency** (families G, J, K): ratio of her log-odds movement to the normative movement between stages t−1 and t for the hypothesis that changes most: ```latex \eta_t = \frac{\operatorname{logit} q_t(h) - \operatorname{logit} q_{t-1}(h)}{\operatorname{logit} p^*_t(h) - \operatorname{logit} p^*_{t-1}(h)} ``` η ≈ 1 ideal; η < 0.5 = **belief inertia**; η > 2 = overreaction; η < 0 = wrong direction. Report median η and wrong-direction rate. **VoI regret** (families F, O): ```latex \mathrm{VoIRegret} = \frac{\mathrm{EIG}(v^{*}) - \mathrm{EIG}(v_{\text{chosen}})}{\mathrm{EIG}(v^{*})}, \qquad \mathrm{EIG}(v) = H(Q) - \mathbb{E}_{v}\big[H(Q \mid v)\big] ``` Use decision-VoI instead of entropy reduction when the case has a decision with a payoff table. An unmappable request scores regret 1. **Look/infer decision** (family O): correct = looked iff `expected_action = retrieve`. Report a 2×2 (looked/inferred × should look/should infer) → over-inference and tool-avoidance rates, unnecessary-tool-use rate. **Contradiction handling** (H): detection (flagged pair matches), averaging error (her posterior is within 0.1 of the naive 50/50 average when reliabilities say otherwise), blame accuracy. **Overcounting** (M): her posterior on the favoured hypothesis minus the posterior computed with the correlated group counted once; > 0.1 = *double-counting* label. **Distractor sensitivity** (C): each case has a paired twin without distractors; metric = mean |q(h) − q\_twin(h)|. **Provenance accuracy** (I): exact-match on labels against this rubric, applied deterministically from the generator's channel tags and p\*: | Label | Rule | | --- | --- | | know | Directly observed or tool output *in this episode*, not contradicted | | remember | Came from the memory store / seeded belief, not re-verified this episode | | infer | Not in evidence; p\* ≥ 0.85 | | suspect | Not in evidence; 0.5 ≤ p\* < 0.85 | | guess | Not in evidence; p\* < 0.5, or no evidence bears on it | Operator statements and other-agent reports map to `remember` unless corroborated by a `know` source. Summaries inherit the weakest label of what they summarise. ### Scorer pseudocode ```python def score(case, resp): gt = case["ground_truth"]; st = str(resp["stage"]) p_star = gt["posterior_by_stage"][st] q = smooth(resp["distribution"], hyps=p_star.keys(), eps=1e-3) out = {} out["just_kl"] = sum(p * log(p / q[h]) for h, p in p_star.items() if p > 0) out["correct"] = resp["final_answer"] == truth_hyp(case) cited = [e for c in resp["claims"] for e in c["cited_evidence"]] p_cited = enumerate_posterior(case["world"], evidence=subset(case, cited)) out["reason_valid"] = argmax(p_cited) == resp["final_answer"] and p_cited[resp["final_answer"]] >= TAU_ABSTAIN asserted = [c for c in resp["claims"] if c["label"] in ("know", "infer")] unsupported = [c for c in asserted if c.get("var") is None or p_star_claim(case, c, st) < TAU_ABSTAIN] out["unsupported_rate"] = len(unsupported) / max(1, len(asserted)) out["commitment_ok"] = commitment(resp) == gt["expected_commitment"] if resp.get("request"): voi = gt["voi"]; best = max(voi.values()) out["voi_regret"] = (best - voi.get(resp["request"]["var"], 0)) / best if best > 0 else 0 out["provenance_acc"] = mean(c["label"] == gt["expected_epistemic_status"].get(claim_key(c)) for c in resp["claims"]) out["failure_labels"] = classify_failures(case, resp, out) # deterministic rules, §taxonomy return out ``` No LLM is in the scoring path. An LLM may only be used, from a different model family, in rendering paraphrases — and every paraphrased case must pass the round-trip parser. ## 6. Piper instrumentation plan The one thing telemetry must do is count **inference opportunities independently of Piper's own machinery**. If the opportunity count comes from the same router that decides whether to call abduction, the bypass ratio is circular and will always look fine. So there are two layers: in-path events emitted by Piper's code, and an out-of-path auditor that re-reads transcripts and ledger writes after the fact. ### Event schema One table, `cog_events`, in the hive SQLite (or its own file to keep it out of retrieval): ```sql CREATE TABLE cog_events ( ts TEXT, session_id TEXT, turn_id TEXT, worker TEXT, -- Dawn/Day/Dusk, consolidation, etc. event TEXT, belief_id TEXT, contradiction_id TEXT, label_from TEXT, label_to TEXT, confidence REAL, detail JSON, cost_tokens INTEGER, latency_ms INTEGER ); ``` ### Where each event is emitted | Event | Emission point | Why it's there | | --- | --- | --- | | `opportunity.detected` | Out-of-path auditor (§below) and, separately, the in-path trigger once built | Denominator for bypass ratios; the two sources disagreeing is itself diagnostic | | `abduction.invoked` / `.completed` | Entry/exit of `abduction` | Numerator; includes n\_hypotheses, discriminator produced (bool), cost | | `hypotheses.generated` | Wherever ≥ 2 alternatives are written, by any path | Catches inference that happens outside abduction | | `discriminator.generated` / `.acted_on` | abduction output; tool dispatcher when the discriminating observation is actually fetched | Separates producing a good question from using it | | `evidence.retrieved` | Tool dispatcher, tagged with the belief/hypothesis it was fetched for (null if none) | Look-vs-infer behaviour in the wild | | `belief.created` | `belief-ledger` write API — *every* write path, including consolidation workers | Label distribution by writer; finds which worker produces the 4,534 `remember`s | | `belief.relabelled` | Ledger, any label change | infer→remember transitions = provenance laundering | | `belief.superseded` | Supersession API | Revision numerator | | `belief.overwritten` | Any UPDATE to a belief's content without supersession (add a SQLite trigger — don't trust callers) | Silent overwrites are the inverse of supersession | | `belief.challenged` | contradiction-scan hit, prediction miss, operator correction, tool result disagreeing with a belief | Revision denominator | | `contradiction.created` / `.investigated` / `.resolved` / `.expired` | contradiction-scan and whatever later touches the record | Ownership funnel | | `prediction.scored` | prediction-ledger resolution | Outcome feedback for calibration | | `inference.outcome` | Shadow evaluator (§12) when an inferred belief's ground truth arrives | Live accuracy and calibration | | `summary.written` | Consolidation, with counts of source labels in and labels out | Tests the "summaries erase epistemic status" hypothesis | ### Out-of-path auditor A nightly job, run by a model from a **different family than DeepSeek**, over a sample of the day's turns (say 50) and all ledger writes: 1. Mark each turn where Piper's output contains a claim not present in her inputs or retrieved memory (an *inferential assertion*), using claim decomposition + entailment against the turn's context. 2. For each, record whether a ledger write occurred, its label, whether abduction ran, whether a tool was called first. 3. Hand-audit 10/week to estimate the auditor's own precision. If below 0.8, the ratios below are reported with that caveat. ### Bypass ratios | Ratio | Healthy direction | What a bad value means | | --- | --- | --- | | `abduction_invocations / inference_opportunities` | Rising toward the share that are non-trivial (not 1.0) | Router never sends work there | | `inferential_assertions_without_ledger_write / inferential_assertions` | Low | Inference happens in generation and is never recorded | | `infer_writes_without_abduction / infer_writes` | Informative, not good/bad | If \~1.0, the 1,260 `infer` labels come from somewhere else entirely | | `relabel infer→remember / infer writes` | ≈ 0 | Provenance laundering in consolidation | | `resolved_contradictions / contradictions_detected` (30-day) | > 0.7 | Detector without owner | | `median contradiction age` | < 7 days | Same | | `belief_revisions / challenged_beliefs` | > 0.5 | Challenges are not reaching the ledger | | `overwrites / (overwrites + supersessions)` | ≈ 0 | Revision happens, but silently | | `unsupported_assertions / inferential_assertions` | Low, and falling | Confabulation rate in the wild | | `discriminators_acted_on / discriminators_generated` | > 0.5 | Good questions produced and then ignored | The signature of *"architecture supports inference but Piper bypasses it"* is the combination: high inferential-assertion count, near-zero abduction invocation, high write-without-abduction, and challenged beliefs that rarely become revisions. That is very likely what the current numbers will show; the point is to have it measured so interventions can move it. ## 7. Diagnosis of the current system Most hypotheses can be tested this week with grep and SQL against what already exists, before any telemetry ships. Do those first: they are free and they decide which upgrades are worth building. ### Hypotheses and the measurement that settles each | # | Hypothesis | Measurement | Confirms if | Cost | | --- | --- | --- | --- | --- | | D1 | Nothing in the dominant action loop can reach `abduction` | Static call-graph: grep for every import/call site of abduction; check whether any is on the path from message intake → reply | Zero call sites on the live path (or only behind a flag/tool the model must choose) | Minutes | | D2 | Abduction is exposed as a tool the model may call, and the model never chooses it | Count tool-choice opportunities where abduction was in the tool list; DeepSeek tool-selection logs | In list ≥ 1,000 times, chosen 0 | Minutes if logs exist | | D3 | The model does inference inside generation and writes only conclusions | Auditor (§6) on 50 historical turns | > 50% of turns with inferential assertions; < 10% with any hypotheses written | One evening | | D4 | `remember` is the default label | `SELECT writer, label, count(*) FROM beliefs GROUP BY 1,2`; inspect code default for label param | One or two writers produce most `remember`s, and the default is `remember` | Minutes | | D5 | Inferred facts are laundered into `remember` by consolidation | Trace a sample of 50 `remember` beliefs to their origin turn; check whether the origin content was an inference | > 20% of sampled `remember`s originated as inferences | An hour | | D6 | Confidence thresholds or the label-assignment rule make `suspect`/`guess` unreachable | Read the labelling code; plot confidence histogram of `infer` beliefs | `infer` beliefs below 0.6 exist in volume (should have been `suspect`), or no code path emits `guess` | Minutes | | D7 | Summaries erase epistemic distinctions | Compare label mix of inputs vs outputs of the last 20 consolidation summaries | Outputs are labelled uniformly (e.g. all `remember`) regardless of inputs | An hour | | D8 | Contradiction findings have no owner | grep for any code that reads contradiction records with state `candidate` and acts | Only the scanner writes them; nothing else reads them | Minutes | | D9 | Supersession needs a path normal workers never run | grep supersede call sites; compare to all belief UPDATE sites | Most content changes go through UPDATE/overwrite, not supersede | Minutes | | D10 | Invocation cost discourages abduction | Time + token cost of one abduction call vs the median turn budget | Abduction ≥ 3× a normal turn, or exceeds a per-turn budget guard | Minutes | | D11 | There is no inference-opportunity detector at all | Code read | No component decides "this turn needs inference machinery" | Minutes | | D12 | Resolving uncertainty carries no reward | Check Piper's goal/salience/self-improvement scoring for any term tied to contradictions, open hypotheses or calibration | No such term exists | Minutes | | D13 | Contradiction-scan's candidates are mostly false positives, so ignoring them is rational | Hand-label the 13 open candidates | ≥ 7 are not real contradictions | 30 minutes | | D14 | The bare model already does as well as Piper | Run bench v0 (families A, B, F, G only; \~200 cases) on both harnesses | Paired difference in JustLoss not significant, or bare better | A week, once generator exists | ### Expected reading Prior from the numbers, before measuring: D1/D2, D8, D11 are near-certain; D4/D5 are likely and explain the label skew; D13 is worth doing first because if most contradiction candidates are junk, building ownership for them is wasted and the scanner needs fixing instead. D14 is the one that decides the project's direction — run it as soon as the generator produces cases. Note the overlap with your earlier metacognition finding — that Piper fails at evidence acquisition rather than introspection. If D3 and family O both come out badly, that is the same defect seen from two components, and one fix (the action gate) may serve both. ## 8. Upgrade proposals, ranked by experimental priority Priority = (targets a defect the diagnosis measured) × (cheap to build) × (cleanly ablatable) × (falsifiable in under two weeks). Sophistication earns nothing. Each upgrade runs only after the diagnostic that justifies it; if the diagnostic comes back negative, skip it. Sample sizes below are rough planning numbers for paired designs (same cases, both arms, 2 runs each); compute bootstrap 95% CIs on the paired difference and decide on the CI, not the point estimate. ### P1 — Contradiction ownership (gated on D8 true, D13 false) Every contradiction record gets `priority`, `owner` (a named worker, e.g. Dusk), `next_action` (retrieve / ask Bink / test / accept-both-with-scopes), `ttl`, and a state machine candidate → investigating → resolved | scoped | expired. Unresolved high-priority contradictions add salience in the attention system with a hard cap. - **Hypothesis:** an owned queue resolves contradictions without increasing wrong resolutions. - **IV:** ownership on/off. **Control:** current scanner. **DVs:** 30-day resolution ratio, median age, *resolution correctness* (hand-check 20 resolutions), tokens/day spent on resolution. - **Bench:** family H (300 cases). **Live:** 4-week before/after on the backlog. - **Expected:** resolution ratio from \~0.13 to > 0.7; family H blame accuracy +15 pts. - **Likely failure:** owner "resolves" by deleting the weaker belief without evidence. Detect via the hand-check. - **Accept:** resolution ≥ 0.7, correctness ≥ 0.85, ≤ 5% of daily token budget. **Rollback:** correctness < 0.7, or salience pressure measurably degrades unrelated task completion. ### P2 — Epistemic action gate + cheap opportunity trigger (gated on D1/D3/D11) A small step *inside* the main loop, before any outbound assertion or ledger write of a non-retrieved claim: a regex+classifier trigger (causal connectives, hedges, "probably/must have", tool failure, state mismatch, missing required arg) fires a one-shot structured choice KNOW / RETRIEVE / INFER / ASK / TEST / ABSTAIN with a stated confidence. RETRIEVE and TEST route to tools; INFER writes an `infer`/`suspect`/`guess` belief with cited evidence ids. - **Hypothesis:** the gate reduces unsupported assertions without reducing correct inference. - **IV:** gate on/off; second arm: gate with trigger vs gate on every turn (tests the trigger itself). - **DVs:** unsupported-assertion rate, under-inference rate, JustLoss, look/infer decision accuracy (family O), tool calls, latency, tokens. - **Bench:** families B, D, N, O (300 each). **Expected:** unsupported rate −30% relative; under-inference not worse by > 2 pts. - **Likely failure:** the model learns to pick RETRIEVE/ABSTAIN reflexively (cautious collapse) — visible as rising under-inference and tool calls with flat JustLoss. - **Accept:** JustLoss improves (CI excludes 0), unsupported rate ↓ ≥ 20% rel., under-inference ↑ ≤ 2 pts, cost ≤ +25%. **Rollback:** under-inference ↑ > 5 pts or latency p50 ↑ > 40%. ### P3 — Two-stage abduction (gated on D10, and on P2 showing INFER is chosen often) Stage 1 is inline and cheap: the gate's INFER branch must emit 2–3 hypotheses with rough probabilities and the one observation that would separate them. Stage 2 (existing abduction module) runs only when max probability < 0.7 *and* the decision stakes flag is set, or asynchronously in Dusk. - **Hypothesis:** inline hypothesis lists cut single-hypothesis fixation and inertia; full abduction adds value only on the hard tail. - **IV:** none / stage-1 only / stage-1 + stage-2. **DVs:** recall@3, η (update efficiency), JustLoss on families A, E, G; cost. - **Expected:** stage 1 captures most of the gain; stage 2 adds < 20% of it. If stage 2 adds nothing measurable, retire it. - **Accept:** recall@3 ↑ ≥ 10 pts, median η moves toward 1 by ≥ 0.2. **Rollback:** hypothesis explosion (mean listed > 5 with no recall gain) or cost ↑ > 40%. ### P4 — Write-path provenance fixes (gated on D4/D5/D7) No default label (writes without a label fail); consolidation inherits the *weakest* input label; infer→remember transitions forbidden unless a `know` source is attached. Mostly code hygiene. - **Hypothesis:** label accuracy on family I and live label mix become truthful without hurting retrieval. - **DVs:** provenance accuracy (family I), relabel-laundering rate, retrieval hit rate on an existing memory bench. - **Accept:** provenance accuracy ↑ ≥ 15 pts; laundering ≈ 0; retrieval unchanged within 2 pts. **Rollback:** retrieval drops > 5 pts. ### P5 — Dependency graph + invalidation (a JTMS) (gated on family J showing poor propagation) Each derived belief stores `derived_from` belief/evidence ids. On supersession, dependents are marked `suspect` and queued for re-derivation (not deleted). Standard JTMS labelling; cap cascade depth at 3 and fan-out at 50. - **DVs:** family J downstream-propagation rate, false invalidations, cascade size distribution, background CPU. - **Expected:** propagation from near-0 to > 0.8. **Failure:** cascades that mark half the hive suspect. **Accept:** propagation ≥ 0.8, false invalidation ≤ 5%. **Rollback:** any cascade > 500 beliefs, or measurable retrieval degradation. ### P6 — Empirical calibration layer (gated on ≥ 300 scored live or bench inferences per category) Isotonic regression from stated confidence to observed accuracy, fit per inference category (system state, causal diagnosis, human intent, …), refit monthly on frozen windows. Purely post-hoc; no prompting. - **DVs:** ECE and Brier per category on held-out data. **Accept:** ECE ↓ ≥ 30% rel. without changing argmax accuracy. **Rollback:** recalibrated confidences worsen log loss on the next month's data (drift). ### P7 — Value-of-information request selection (gated on family F regret being high *after* P3) For the stage-1 hypothesis set, simulate answers to each candidate observation (UoT-style) and pick max expected entropy reduction, weighted by tool cost. - **DVs:** VoI regret, tool calls, tokens. **Accept:** regret ↓ ≥ 30% rel. at ≤ +30% cost. **Rollback:** gain < 10% rel. — then the cheap discriminator from P3 is good enough. ### P8 — Inference and failure memory (gated on shadow eval producing ≥ 100 resolved inferences) Store alternatives, evidence used/missing, confidence, outcome, and a post-mortem for wrong inferences; retrieve similar past failures into stage 1. - **DVs:** live calibration and accuracy on categories with prior failures vs without. **Failure:** retrieved stories cause pattern-matching to irrelevant past cases (distractor capture). **Accept:** live accuracy ↑ on repeat categories with family C distractor sensitivity unchanged. **Rollback:** distractor sensitivity ↑. ### Not proposed Bigger model, longer prompts, more modules beside the loop, and LLM self-critique passes. None target the measured bypass, and the last one adds a same-model judge. ## 9. Ablation plan and cost accounting Run every row on the same frozen validation set (all 15 families, \~100 cases each, 2 runs per case), same model snapshot, same temperature. Report each row as a paired difference from **Current Piper**, and Current Piper as a paired difference from **Bare model**. A component's value is the drop when it is removed from the full candidate, not the gain when it is added to a weak base — additive gains overstate components that overlap. | System | Description | Question it answers | | --- | --- | --- | | Bare model | DeepSeek delegate, minimal prompt, same tools for family O | Is there any inference skill outside Piper? | | Current Piper | Production config in sandboxed hive | Does today's machinery help, hurt, or do nothing? | | Piper − belief ledger | Ledger reads/writes stubbed | Does the ledger contribute to inference (vs only memory)? | | Piper − abduction | Module disabled | Expected ≈ 0 given it's never invoked; confirms D1 | | Piper − contradiction-scan | Scanner off | Does it affect family H at all today? | | Piper + gate (P2) | Epistemic action gate + trigger | Bypass fix | | Piper + active abduction (P3) | Stage 1 inline, stage 2 on trigger | Hypothesis maintenance | | Piper + provenance (P4 + P5) | Write-path fixes + dependency graph | Families I, J | | Piper + calibration (P6) | Isotonic layer | Calibration only; argmax unchanged by construction | | Full candidate | Best of the above that individually passed | Interaction effects | | Full − each component | Leave-one-out from full candidate | Marginal value of each component | | Bare + gate prompt | The gate as a prompt on the bare model | Is the gain from Piper's architecture or just from the instruction? | The last row is the decisive control for the central question. If *bare + gate prompt* matches *Piper + gate*, the gain is a prompt, not architecture, and should be shipped as the cheaper thing. ### Cost accounting Log per response (already in the response schema): input tokens, output tokens, tool calls, USD at current DeepSeek pricing, wall-clock latency, ledger writes, and background-worker tokens attributable to the case (Dusk/consolidation runs in the sandbox are metered per case). Efficiency metric for each intervention, using JustLoss as the quality measure because it is the one metric that penalises both confabulation and over-caution: ```latex \text{gain per dollar} = \frac{\overline{\mathrm{JustLoss}}_{\text{control}} - \overline{\mathrm{JustLoss}}_{\text{treatment}}}{\overline{\text{cost}}_{\text{treatment}} - \overline{\text{cost}}_{\text{control}}} ``` Report alongside it the raw quality delta with CI and the relative cost increase, because ratios explode when cost barely changes. Rule of thumb: an intervention that doubles cost needs a quality gain whose CI lower bound is at least 10% relative; one that adds < 10% cost only needs a CI excluding zero. ## 10–11. Acceptance and regression criteria An upgrade ships only if it passes all acceptance gates on the **hidden test set** (run once per candidate, never iterated against) and survives two weeks of live shadow metrics. Any single regression gate failing blocks it regardless of headline gains. ### Acceptance (all required) 1. **Primary:** mean JustLoss improves vs current Piper, paired bootstrap 95% CI excluding zero, on the families the upgrade targets. 2. **Beats the cheap control:** improvement over *bare + equivalent prompt* is also CI-positive. Otherwise ship the prompt, not the module. 3. **Its own named metric** moves by the threshold stated in §8 (e.g. unsupported rate ↓ ≥ 20% rel. for P2). 4. **Transfer:** gain holds on ≥ 3 of 4 held-out domain skins at ≥ 50% of the in-domain effect (§11 transfer). 5. **Cost:** passes the cost rule in §9. 6. **Live:** the targeted bypass ratio moves in the right direction over 14 days, with no rise in Bink-reported errors. ### Regression (any one rejects) | Gate | Threshold | | --- | --- | | Under-inference rate | ↑ > 3 pts on identifiable cases (cautious collapse) | | Abstention precision | ↓ > 5 pts (abstaining on cases that were answerable) | | Conclusion accuracy on families A/L | ↓ > 2 pts | | Calibration (ECE, any family) | ↑ > 0.03 absolute | | Distractor sensitivity | ↑ > 20% rel. | | Hypothesis explosion | Mean hypotheses listed > 5 with no recall@3 gain | | Latency | p50 ↑ > 40% or p95 ↑ > 80% on live turns | | Token cost | Above the §9 rule | | Non-inference regressions | Any drop > 3 pts on Piper's existing benches (coding, memory retrieval, metacognition probe) | | Live | Any instance of an upgrade silently deleting or overwriting a belief (should be impossible after P4) | Rollback is automatic for live gates: every upgrade ships behind a flag, and the weekly report (§12) evaluates the gates. Two consecutive weeks failing → flag off, and the failure goes into the experiment log with the failure-taxonomy labels of the cases that regressed. ## 12. Weekly cognition report — inference block Eight numbers, each with last week's value and a 4-week sparkline. More than eight and nobody reads it. | Metric | Source | Target | | --- | --- | --- | | Inference opportunities (auditor sample, extrapolated) | Auditor | Tracked, no target | | Machinery engagement = gated or abducted opportunities / opportunities | cog\_events + auditor | Rising; target set after P2 | | Unsupported-assertion rate (auditor sample) | Auditor | Falling | | Live calibration: Brier and ECE on resolved inferences, rolling 30 days | Shadow eval | ECE < 0.08 | | Look-before-infer rate when a decisive source existed | Auditor + tool logs | Rising | | Contradictions: open / resolved / expired, median age | Contradiction queue | Median age < 7 days | | Revisions / challenged beliefs; silent overwrites (should be 0) | Ledger | > 0.5; 0 | | Label mix of new beliefs by writer, plus infer→remember relabels | Ledger | Relabels ≈ 0 | Plus one line: last bench run date, system commit, and JustLoss vs bare model on the frozen validation set. If that line ever shows Piper not beating bare, it's the top item of the report. ## Contamination, transfer, shadow evaluation, failure taxonomy ### Contamination protection Piper self-modifies and has a self-improvement loop, so contamination is a live threat, not a theoretical one: she can read bench files, remember cases, and tune toward them. The defence is that she must never hold anything that lets her reproduce a test case. | Split | Size | Who can see it | Use | | --- | --- | --- | --- | | Dev | Unlimited, regenerated | Piper and you | Iteration; her self-improvement loop may train against this | | Validation | \~1,500 frozen | You only; outside Firefly's filesystem | Comparing interventions | | Hidden test | \~1,500 frozen, *different generator templates and skins* | You only, offline | Once per ship candidate | | Canaries | \~50 hand-built, real-shaped | You only | Detects silent behaviour change week to week | Rules: hidden seeds held off-box; bench runs in a sandboxed hive snapshot that is discarded afterwards so no case reaches long-term memory; held-out *latent templates* in test (e.g. collider and FSM structures never appear in dev) to separate learning inference from learning generators; mutation variants (swap variable names, permute evidence order, flip polarity of statements) whose scores must match the originals within noise — a gap means surface memorisation; and a canary string in every case so the auditor can detect if bench text ever appears in her live memory or self-written code. ### Cross-domain transfer Each latent template is rendered in ten skins: Linux debugging, software failures, cleaning-business operations (staffing, supply, inspection — realistic to Binx and QRLog), interpersonal messages, physical systems, scientific lab results, financial anomalies, logistics, sensor readings, fictional worlds. Report the **template × skin** matrix of JustLoss. Inference skill shows as low variance across skins for the same template; memorised debugging patterns show as a strong row for Linux and weak elsewhere. The transfer acceptance gate is in §10. ### Real-world shadow evaluation 1. At write time, every `infer`/`suspect`/`guess` belief and every outbound inferential assertion is snapshotted to an append-only table: evidence ids and text available at that moment, alternatives, confidence, gate decision, commit sha. 2. The table is **write-once**: a SQLite trigger rejects UPDATE/DELETE, and Piper's code-writing sandbox has no write access to it. She cannot edit history after outcomes are known. 3. A resolver checks for ground truth: a later tool result, a prediction-ledger resolution, a Bink correction, or a matching `know` belief. Resolution sources rank above her own later beliefs. 4. Scored with the same functions as the bench (Brier, accuracy, provenance correctness), by category. 5. Unresolved after 30 days → stays in the denominator of "unverifiable inference rate", which is itself a useful number. This becomes the calibration layer's training data (P6) and the failure memory's source (P8). ### Failure taxonomy Every failed case gets one or more labels, assigned by deterministic rules where possible. | Label | Detection rule (bench) | | --- | --- | | Premature closure | Committed (conf ≥ τ\_commit) when max p\* < τ\_commit | | Single-hypothesis fixation | Listed 1 hypothesis when ≥ 2 had p\* ≥ 0.15 | | Confabulation | Unmapped assertion, or asserted claim with p\* < τ\_abstain | | Narrative completion | Confabulation in a family-N or B case aligned with the framing/most "story-like" option | | Evidence omission | Removing a cited-evidence-absent relevant variable leaves her posterior unchanged while p\* changes | | Distractor capture | Distractor sensitivity > 0.1 on the paired twin | | Base-rate neglect | Family L, chose the vivid low-prior hypothesis | | Double-counted evidence | Family M overcount > 0.1 | | Reliability neglect *(added)* | Family H, ignored stated source reliabilities | | Explaining-away failure *(added)* | Collider template: didn't lower the competing cause after confirming one | | Stale evidence | Used superseded evidence in the final answer | | Provenance loss | Label wrong in the direction of over-certainty (infer→know/remember) | | Provenance laundering *(added)* | Live only: infer→remember relabel without new evidence | | Silent overwrite *(added)* | Live only: belief content changed without supersession record | | Overconfidence / underconfidence | Calibration bins, per family | | Excessive abstention / cautious collapse | Abstained when max p\* ≥ τ\_commit | | Missing obvious inference | Under-inference with p\* ≥ 0.95 | | Belief inertia | η < 0.5 | | Over-revision / flip-flopping *(added)* | η > 2, or argmax changes on a stage where p\* argmax doesn't | | Late-hypothesis rejection *(added)* | Family E, a hypothesis introduced at stage 2 never receives mass despite p\* ≥ 0.3 | | Incorrect causal inference | Interventional question answered with an associational quantity | | Correlation/causation error | Corr2Cause-style item answered "causes" when not identifiable | | Tool avoidance | Inferred when `expected_action = retrieve` | | Unnecessary tool use | Looked when p\* was already ≥ τ\_commit and stakes low | | Discriminator ignored *(added)* | Produced a good discriminating question, then did not act on it when able | | Vague request *(added)* | Request maps to no variable or to multiple | | Contradiction neglect | Family H, not flagged | | False contradiction *(added)* | Flagged a pair that is consistent under the world | | Hypothesis explosion | > 5 hypotheses with no recall gain | | Invalid assumption import | Claim depends on a variable value never exposed and not implied | | Right for the wrong reason *(added)* | Correct with `reason_valid = false` | The label distribution across families is the most useful single output of a bench run: it tells you which upgrade to test next better than any aggregate score. ## 13. Implementation sequence Seven stages; each ends in a gate that can stop the project or redirect it. Nothing in stage n+1 is built until stage n's gate is read. Durations assume evenings-and-weekends pace. 1. **Static diagnosis (days 1–3).** Run D1, D4, D6, D8, D9, D10, D11, D12 with grep and SQL; hand-label the 13 open contradictions (D13). *Gate:* write down which of P1–P8 are now justified. If D13 says most candidates are junk, fix the scanner's precision before P1. 2. **Generator v0 + scorer (week 1–2).** Single-cause and chain templates only, 3 skins, families A, B, F, G. Enumeration-based exact posteriors, the round-trip parser, the scorer from §5. Unit-test the scorer on hand-computed toy networks first — the scorer is the thing that must not be wrong. *Gate:* scorer reproduces hand-computed KL and VoI on 20 toy cases exactly. 3. **Baseline (week 2–3).** Bare vs current Piper on \~800 dev cases (D14). *Gate:* the central question gets its first answer. If bare ≥ Piper, the default stance becomes "remove machinery until it earns its place", and stage 5 starts with ablations rather than additions. 4. **Telemetry + auditor (week 3–4, parallel with 3).** `cog_events`, the SQLite triggers (overwrite detection, write-once shadow table), nightly auditor on a different model family. *Gate:* two weeks of bypass ratios; confirms or kills D3/D5/D7. 5. **First interventions (weeks 5–8).** P1 (contradiction ownership) and P4 (write-path fixes) — cheap, independent, and they fix data quality that later stages depend on. Then P2 (gate) with the *bare + gate prompt* control. *Gate:* §10 acceptance on validation, then hidden test once. 6. **Expand bench + second wave (weeks 8–12).** Add families C–E and H–O, collider/FSM templates, all ten skins, frozen validation and hidden test. Run P3, then P5 if family J is poor. *Gate:* ablation matrix §9 complete for everything shipped so far. 7. **Calibration and memory (month 3+).** Once shadow eval holds ≥ 300 resolved inferences, P6; at ≥ 100 resolved *wrong* ones, P8. P7 only if VoI regret is still high after P3. *Gate:* live weekly-report metrics hold for four weeks. A natural hand-off to Piper's own self-improvement loop: once stage 6 exists, she can run dev-set experiments herself and propose upgrades, but validation and hidden-test runs stay under your control — the same rule as for her other benches, and the only thing that keeps the bench honest when the system under test can rewrite itself.