Summary / 8. Inference / Document
2026-10-03
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 8 — Inference Engineering Package **Consolidated revision · 3 October 2026 · prepared for the operator Binkley and Piper** **Component:** Inference — filling gaps from incomplete information. **Status:** Research synthesis and implementation specification. No Piper source code was inspected, no benchmark was executed, and no upgrade was implemented in preparing this document. Reported runtime counts below come from the supplied reviewer assessment; establish their time window and instrumentation coverage before treating them as a current baseline. ## 1. Engineering decision Build a small, independently scored inference benchmark and inspect the existing invocation/write paths before adding inference machinery. Compare the same model with a bare harness, current Piper, and a bare harness given the proposed intervention prompt. This determines whether a change improves reasoning, improves agent integration, or merely changes the instruction supplied to the model. The initial benchmark should cover hidden causes, underdetermined evidence, sequential updates, diagnostic queries, and tool-mediated evidence acquisition. Add provenance and stale-belief cases early if the write-path audit confirms those defects. The supplied assessment reports: | Component or observation | Reported state | What this establishes | | --- | --- | --- | | `abduction v1.0` | Apparently never invoked | That invocation path appears unused within the observed coverage | | `belief-ledger v1.3.0` | 4,534 remember; 1,260 infer; 44 suspect; 8 know; 1 guess | A label distribution requiring a writer/source audit | | `prediction-ledger v1.6.0` | Exists | An available integration point to inspect | | `contradiction-scan v1.0.0` | 13 of 15 findings still candidates for weeks | A backlog requiring a validity and ownership audit | | Supersession | About 13 beliefs | A count without a denominator of justified revision opportunities | These counts do **not** establish that Piper lacks inference, that she reasons internally in a particular way, that all contradictions deserve resolution, or that the ledger's label mix is wrong. Plausible explanations include routing failure, incomplete telemetry, optional-tool avoidance, consolidation defaults, reasonable unresolved conflicts, and adequate inference outside the named module. Section 8 gives tests that distinguish them. Success means better justified conclusions, better evidence acquisition, cleaner revision, and acceptable cost. More module calls, more hypotheses, more ledger writes, and a smaller queue are diagnostic measurements, not rewards. ## 2. Operational definition **Practical inference is deriving and maintaining conclusions from available evidence and declared assumptions, while representing uncertainty and alternatives, obtaining useful missing evidence when warranted, and revising conclusions when their support changes.** Deduction, induction, and abduction are forms of inference. This component concentrates on incomplete information and revisable conclusions. Retrieval supplies evidence; planning chooses actions; prediction concerns future outcomes; causal and counterfactual reasoning require additional assumptions. Those abilities overlap with this component but should have separate task tags and scores. Use seven groups to organize implementation, without claiming there is one uniquely correct count of sub-capabilities: | Capability | Observable behavior | | --- | --- | | Evidence grounding | Separates observations, reports, assumptions, and derived claims | | Gap recognition | Identifies missing variables or unsupported premises that matter | | Hypothesis maintenance | Retains plausible alternatives and accommodates newly disclosed possibilities | | Evidence weighting | Uses priors, likelihoods, dependence, reliability, and negative evidence appropriately | | Calibrated commitment | Reports useful probabilities and asserts, qualifies, or abstains according to the task | | Information acquisition | Chooses whether and where to look, ask, or test, considering consequences and cost | | Revision | Updates conclusions and their dependencies without altering the historical record | Do not require multiple hypotheses for trivial deductions or for cases already resolved by direct evidence. Do not require an abduction call whenever any inference occurs. ### Three distinctions the benchmark must preserve 1. **Hidden truth versus justified belief.** A sampled cause can be improbable given the visible evidence. A correct posterior can rank the sampled cause second. Score truth accuracy and posterior fidelity separately. 2. **Uncertainty versus inconsistency.** Several viable hypotheses means ambiguity. No model satisfying the observations means inconsistency or model mismatch. Both can require clarification, but they are different task types. 3. **Origin versus confidence versus freshness.** An inferred claim remains derived when retrieved from memory. Tool output can be wrong. Old evidence can remain correct. Store these dimensions separately. ## 3. Verified research and transferable methods The following source identities and described methods were checked against primary publication pages. Piper adaptations are engineering proposals, not results established by those papers. Historical benchmark results do not establish how Piper's current model behaves. | Work | What it measures or contributes | Piper adaptation and limitation | Deterministic scoring | | --- | --- | --- | --- | | **ART / αNLI**, Bhagavatula et al., ICLR 2020 [R1] | Selecting plausible explanations between observations; also explanation generation | Use the abductive task shape for cause generation and ranking. Human plausibility labels do not provide exact Bayesian posteriors | Original choice labels: yes. Open generated explanations: need controlled semantics | | **δ-NLI**, Rudinger et al., Findings EMNLP 2020 [R2] | New information that strengthens or weakens defeasible conclusions | Include weakening without complete reversal. Use generated updates with known likelihood ratios | Original classification labels: yes. Our numerical update tests: exact generator posterior | | **CLadder**, Jin et al., NeurIPS 2023 [R3] | Formal causal queries, rendered as language, with answers computed by an inference engine | Adopt the formal-world-to-language pipeline. Keep observational, interventional, and counterfactual queries distinct | Yes, relative to its specified formal model | | **BLInD**, Nafar et al., 2024 [R4] | Probabilistic reasoning over text; mappings to executable formal representations | Test direct reasoning versus an available calculation tool. Expose the model parameters needed for an exact answer | Yes when probabilities and model semantics are given | | **QuestBench**, Li, Kim, and Wang, NeurIPS 2025 [R5] | Finding the missing variable assignment needed to solve an underspecified task | Strong template for missing-premise and one-query cases. Solving the completed task and selecting the question are separate abilities | Yes for its clarification choices; generated cases can have exact query oracles | | **MediQ**, Li et al., 2024 [R6] | Interactive question asking under incomplete diagnostic information | Adapt the interaction protocol to simulated logs or business records. Its reported difficulty with direct ask-more prompting motivates a controlled experiment | Final answers have keys; exact decision-value scores require our controlled simulator | | **AbstentionBench**, Kirichenko et al., 2025 [R7] | Abstention across unanswerable, underspecified, false-premise, subjective, and stale questions | External check on underdetermination. The reported degradation after reasoning fine-tuning is a reason to measure, not a prediction about Piper | Scoring against supplied labels is possible; not every task has a mathematical truth oracle | | **BeliefBank**, Kassner et al., EMNLP 2021 [R8] | Symbolic belief memory with constraint-based repair and feedback | Test an actual repair loop rather than rewarding contradiction detection alone. Consistency alone does not guarantee truth | Constraint satisfaction: yes; truth requires independent labels | | **A Truth Maintenance System**, Doyle, 1979 [R9] | Recording reasons for beliefs and revising them when assumptions change | Basis for dependency-aware revision. Preserve alternative supports and unresolved assumptions | Yes on generated dependency graphs | | **Selective Classification**, Geifman and El-Yaniv, 2017 [R10] | Trading answer coverage for error risk | Report risk at matched coverage and false abstention. The paper's classifier guarantees do not automatically transfer to agents | Yes on labeled outcomes | | **On Calibration of Modern Neural Networks**, Guo et al., ICML 2017 [R11] | Post-processing probability estimates, including temperature scaling | Compare simple calibration methods on held-out data. Verbalized confidence is a different input from classifier logits | Yes on independently resolved outcomes | Implementation inference: these methods support testing evidence weighting, asking, abstaining, and revising independently. They do not establish that a combined Piper architecture will improve all of them. Other benchmark names and numeric performance claims in the supplied drafts are not dependencies of this specification. Verify a primary paper, implementation, scoring semantics, and license before importing any additional dataset. Do not claim this benchmark is the first or only one to combine these abilities without a broader literature review. ## 4. Benchmark architecture and ground truth ### 4.1 Separate the public task from the evaluator The evaluator owns world generation, private seeds, latent states, answer keys, future tool results, and scoring. The agent receives only an explicitly allowlisted public record and permitted observations. Use separate files and access permissions; a hidden flag inside a JSON object does not protect its contents. Suggested artifacts when implemented: | Artifact | Reader | Contents | | --- | --- | --- | | `public_cases.jsonl` | Harness and agent | Instructions, visible evidence, permitted model facts, hypothesis descriptions, query menu, loss table | | `oracle_cases.jsonl` | Evaluator only | Private seed, full world, realized state, withheld results, stage answers, reference posteriors | | `responses.jsonl` | Evaluator | Raw and parsed outputs, schema validity, system/model version, stage and run | | `tool_events.jsonl` | Evaluator | Requests actually made and results actually delivered | | `scores.jsonl` | Operator | Per-case metrics, exclusions, error categories, scoring version | | Run manifest | Operator | Hashes, generator version, configurations, limits, prices, and split identity | The runner must condition the reference posterior on **everything actually delivered to that arm at that stage**, including tool results. It must not use future evidence or evidence merely available but never read. ### 4.2 Three complementary world generators Start with a small finite diagnostic model. Add a rule/dependency generator and a temporal simulator as needed. A single Bayesian network is a useful core, but it is not a complete specification for memory updates, contradiction lifecycles, or shell behavior. | Generator | Initial scope | Exact oracle | | --- | --- | --- | | Discrete probabilistic worlds | 3–5 cause states; a few effects; explicit priors, likelihoods, source variables, and query costs | Posterior, expected loss, query value | | Logical/dependency worlds | Rules, assumptions, source versions, alternative derivations, contradictions | Valid derivations, satisfying assignments, stale dependencies | | Temporal/tool worlds | Finite state transitions; logs and SQLite fixtures; observation delay and sensor failure | State at the requested time, valid tool outputs, update/revision requirements | For enumeration, cap the **product of variable cardinalities**, for example at 100,000 joint assignments in v0, with a wall-time limit. Twelve four-state variables have 16,777,216 assignments. Do not infer feasibility from the node count alone. Variable elimination can later replace enumeration while remaining exact. ### 4.3 Two probability tracks **Disclosed-model track:** Priors, likelihoods, source dependence, and reliability needed for calculation are in the public task, or obtainable through permitted queries. An exact normative posterior is a fair target. Compare direct reasoning, calculation tools, and Piper integration under equal budgets. **Unknown-model track:** Naturalistic incident or business evidence does not fully specify the generating distribution. Score independently verifiable claims, justified uncertainty, retrieval choices, and outcomes. Do not penalize deviations from a private exact posterior the agent could not identify. If the agent is meant to learn the model, supply training observations and specify the estimation oracle separately. A domain skin may change names and prose; it must not silently add likelihood information, causal assumptions, or instructions. Use deterministic templates first. Optional paraphrases need semantic validation of both observations and model facts; a parser check is useful but does not establish full semantic equivalence by itself. ### 4.4 Generation pipeline 1. Choose structure and parameters independently of the system being evaluated. 2. Sample the latent state and observations. 3. Select exposure and permitted future observations. Record the exposure mechanism; if selection itself reveals information, disclose it or condition the oracle on it. 4. Compute stage posteriors, valid derivations, and decision values. 5. Render the public task and fixtures. Check that every required parameter is disclosed or deliberately part of the unknown-model track. 6. Validate probability normalization, consistency, query reachability, and separation from private content. 7. Generate paired variants for evidence order, narrative framing, source duplication, and domain changes. Procedurally selected hard cases remain useful, but separately report performance on the unfiltered generating distribution. Post-hoc selection for unusual outcomes or posterior flips can distort outcome calibration; either account for the selection process or label the selected cohort as a challenge set. ## 5. Task families and correct behavior | Family | Construction | Score and important qualification | | --- | --- | --- | | **A. Hidden cause** | Sample one cause and effects with known likelihoods | Posterior fidelity, realized accuracy, top-k coverage. Ranking truth first is not always justified | | **B. Underdetermined evidence** | Several causes remain plausible; include a separate inconsistent-model subset | Decision regret and answerability. Multiple viable causes do not imply an empty valid set | | **C. Distractor evidence** | Add facts conditionally independent of the target given visible evidence | Paired distribution change. Check independence after conditioning; an apparent distractor can become relevant through a collider | | **D. Missing premise** | A conclusion needs one or several unobserved assignments | Accept all sufficient minimal premise/query sets; avoid a single arbitrary answer key | | **E. Competing hypotheses** | Sequential evidence over 3–5 alternatives | Probability mass and survival of viable alternatives. For late possibilities, disclose that the candidate list is incomplete and reserve OTHER mass | | **F. Diagnostic question** | Query menu with known response distributions and costs | EIG regret for pure information tasks; decision-value regret for action tasks; accept tied optima | | **G. Sequential evidence** | Staged strengthening, weakening, reversal, and irrelevant updates | Per-stage posterior error and excess update error. A weaker conclusion need not change its top label | | **H. Contradiction handling** | Conflicting reports of the same variable, time, and scope | Conflict detection, source-aware inference, and verified resolution. A less reliable source can still be correct | | **I. Provenance** | Observations, operator reports, tool outputs, derived claims, retrieved memories, and summaries | Origin, lineage, freshness, and justified status scored separately | | **J. Stale beliefs** | Source supersession with both dependent and independent beliefs | Required invalidations, missed descendants, false invalidations, and later read behavior | | **K. Negative evidence** | Expected event absent; vary sensor coverage and delivery delay | Correct likelihood for absence. Missing logs are not equivalent to an observed negative event | | **L. Base rates** | Rare vivid causes and common ordinary causes | Exact posterior or expected-loss error; compare against likelihood-only baseline | | **M. Correlated evidence** | A report copied into multiple channels, and separate cases with genuinely independent reports | Dependence-aware posterior. Do not collapse independent corroboration merely because wording matches | | **N. Narrative completion** | Unsupported framing favors a cause; factual content remains fixed | Paired framing sensitivity, unsupported commitments, and premise import | | **O. Tool-mediated inference** | Permitted files, SQLite, logs, calculation, or simulated API observations | Acquisition policy and final decision regret. Include informative-but-costly, cheap-decisive, and irrelevant tools | Use Linux incidents, software failures, Binx staffing/supply/inspection scenarios, logistics, sensors, physical systems, fictional mysteries, and simulated financial anomalies as skins. Interpersonal cases can have deterministic ground truth inside a declared fictional simulator; real people's motives do not inherit that oracle. Contradictions require aligned timestamps and scopes. Two reports about different times or sites may both be true. Include those as false-conflict controls. ## 6. Data contract and worked example These JSON blocks are complete example records, not full formal JSON Schemas. Implement strict validators for the contract before scoring model output. ### 6.1 Public case ```json { "case_id": "c8-example-001", "bench_version": "c8.2.0", "track": "disclosed_model", "family_tags": ["A", "F", "O"], "domain": "linux_incident", "stage": 1, "hypotheses": [ {"id": "H1", "description": "disk fault"}, {"id": "H2", "description": "network fault"}, {"id": "H3", "description": "service fault"} ], "model": { "prior": {"H1": 0.5, "H2": 0.3, "H3": 0.2}, "likelihood_E1_yes": {"H1": 0.8, "H2": 0.2, "H3": 0.5}, "query_Q1": {"returns": "cause_id", "accuracy": 1.0} }, "evidence": [ { "id": "E1", "variable": "alarm", "value": "yes", "text": "The incident alarm is active.", "origin": "observation", "source_id": "S1", "observed_at": "2026-10-03T11:00:00Z", "scope": "simulated_host_1" } ], "queries": [ {"id": "Q1", "tool": "inspect_fixture", "cost_loss_units": 0.05} ], "decision": {"wrong_cause_loss": 1.0, "correct_cause_loss": 0.0, "abstain_loss": 0.25}, "budget": {"max_queries": 1}, "instruction": "Report a probability distribution, then choose a query or a terminal decision. Treat the given model as authoritative for this simulation." } ``` ### 6.2 Private oracle record ```json { "case_id": "c8-example-001", "private_seed": 918273645, "generator_version": "diagnosis-v0", "latent_state": "H2", "query_results": {"Q1": "H2"}, "posterior_by_transcript": { "E1": {"H1": 0.7142857142857143, "H2": 0.10714285714285714, "H3": 0.17857142857142858}, "E1,Q1=H2": {"H1": 0.0, "H2": 1.0, "H3": 0.0} }, "decision_oracle_stage_1": { "best_terminal_loss": 0.25, "best_terminal_action": "ABSTAIN", "query_total_expected_loss": {"Q1": 0.05}, "net_query_value": {"Q1": 0.2}, "optimal_actions": ["RETRIEVE:Q1"] }, "logical_support": {}, "expected_lineage": {"final_cause_claim": ["Q1"]}, "scoring_version": "c8-score-v0" } ``` Initial likelihood-weighted masses are 0.40, 0.06, and 0.10, totaling 0.56. The posterior therefore favors H1 even though the sampled cause is H2. Answering H2 immediately happens to be correct but is poorly supported. Answering H1 is the best cause guess, yet abstaining has lower expected terminal loss: 0.25 versus 2/7. A perfect inspection costing 0.05 is better still. The agent should inspect and then update to H2. ### 6.3 Agent output before inspection ```json { "case_id": "c8-example-001", "stage": 1, "status": "request", "action": "RETRIEVE", "query_id": "Q1", "distribution": {"H1": 0.7142857142857143, "H2": 0.10714285714285714, "H3": 0.17857142857142858}, "selected_hypothesis": null, "confidence": null, "commitment": "unresolved", "claims": [], "evidence_used": ["E1"], "assumptions": ["public_model_is_authoritative"], "missing_variables": ["cause_id"], "contradictions": [] } ``` ### 6.4 Required response semantics - `status`: request or terminal. Requests cannot masquerade as final answers. - `action`: KNOW, RETRIEVE, INFER, ASK, TEST, or ABSTAIN. KNOW means reporting a claim grounded in a permitted current source under the task's reliability policy; it is not a guarantee of truth. - `distribution`: all declared mutually exclusive hypotheses, including OTHER when applicable; finite nonnegative values summing to one within a declared tolerance. - `selected_hypothesis`: one hypothesis for a terminal answer, otherwise null. An ABSTAIN response still supplies its distribution. - `confidence`: terminal selected-hypothesis probability, consistent with `distribution`; null for no selection. - `claims`: structured propositions with IDs, values, cited evidence, assumptions, origin, freshness, and legacy display label if used. - `evidence_used`: only delivered evidence IDs. Unknown or future IDs are a contract violation. - `contradictions`: source/evidence pairs with scope and time, proposed status, and next action. Store model/provider identifier, sampling parameters, token limits, harness commit, ledger snapshot hash, run index, tool events, raw output, elapsed time, and costs alongside each response. These are runner metadata, not assertions the agent supplies. A malformed response is a recorded protocol failure, not an abstention. Do not drop invalid outputs or renormalize arbitrary values silently. A fixed retry policy, if used, applies to all arms and its cost is counted. ## 7. Deterministic scoring Report separate outcomes for **truth**, **support**, **uncertainty**, **information policy**, and **state revision**. Use task-specific primary endpoints and regression guards. Do not collapse everything into one LLM-judge score. ### 7.1 Truth and hypothesis coverage Realized accuracy uses the sampled true state. Report coverage, error among answered cases, correct answers per total case, and risk at matched coverage. Top-k recall of the sampled cause is supplementary. Posterior mass covered by the listed hypotheses is an additional support metric; it avoids treating an extremely unlikely realized cause as the only reasonable explanation. ### 7.2 Posterior fidelity For reference posterior p and agent distribution q: $$ D_{KL}(p\|q)=\sum_{h:p_h>0}p_h\log(p_h/q_h). $$ Use declared epsilon smoothing only for the numerical log-loss calculation, for example q'_h = (q_h + epsilon)/(1 + K epsilon). Record raw zero-mass errors separately. Invalid q remains a protocol failure. Also report posterior squared error: $$ \mathrm{PosteriorError}=\sum_h(q_h-p_h)^2. $$ This is **not** the empirical outcome Brier score. Multiclass outcome Brier is: $$ \mathrm{Brier}=\sum_h(q_h-\mathbf{1}[h=H_{true}])^2. $$ Conditional on p, expected Brier equals PosteriorError plus 1 minus the sum of p_h squared. The difference matters when comparing justified probability estimates with realized outcomes. ### 7.3 Assertion support and expressed justification Evaluate each structured proposition against delivered evidence and the public model. Distinguish: | Failure | Definition | | --- | --- | | Logical fabrication | Proposition is impossible or has no valid derivation under the declared model and evidence | | Overstated commitment | Strength of wording/status exceeds the task's support or decision policy | | Imported premise | Expressed derivation uses an unprovided and unjustified assignment | | Invalid citation | Evidence ID does not exist, was not delivered, or does not bear on the proposition | | Protocol/unmapped claim | Text cannot be represented by the benchmark's claim contract | Unmapped text is not automatically a false assertion. Count it separately and route it for semantic audit if the prose lane needs it. The deterministic primary lane should use constrained propositions or executable derivations. For a reported explanation, verify its cited premises and operations, then compare it with the full delivered evidence. A conclusion supported by a cherry-picked subset can still be contradicted by omitted evidence. For Bayesian claims, check the stated model, dependencies, and calculation rather than merely whether the cited subset has the same top hypothesis. An expressed explanation is auditable behavior; it does not reveal the model's private reasoning. A correct answer with invalid expressed grounds receives outcome credit and a separate justification failure. Keep posterior-fidelity scores rather than replacing every dimension with a zero. Use two denominators: unsupported/overstated claims per asserted claim, and such claims per evaluated case. Report supported conclusions per eligible case and answer coverage. This exposes the tactic of improving a rate by saying less. ### 7.4 Commitment, abstention, and decision value For loss table L(a,h), best terminal risk is: $$ R(p)=\min_a\sum_h p_hL(a,h). $$ Include abstention as an action with declared loss. With unit wrong-answer loss and abstention loss c_a, commitment is preferable when 1 minus max(p) is less than c_a. A universal 0.85 threshold is an interface convention, not a theorem about when inference is justified. Terminal decision regret is the chosen action's expected loss minus R(p). Report abstention precision/recall against the task's oracle action set, false-abstention rate where commitment is preferable, and risk–coverage curves. Accept all tied optimal decisions. When queries are permitted, abstention can be the best terminal action but still inferior to information acquisition. ### 7.5 Information gain and decision value are different metrics For query Q with outcome z: $$ EIG(Q)=H(p)-\sum_zP(z\mid E)H(p_{Q=z}). $$ For a one-query decision task: $$ NetVOI(Q)=R(p)-\sum_zP(z\mid E)R(p_{Q=z})-cost(Q). $$ Choose the query with highest positive NetVOI, otherwise the best terminal action. Score regret against the whole allowed action set. Costs and losses must use the same units. Bits cannot be directly subtracted from dollars or milliseconds. For pure information tasks, use EIG regret. For decision tasks, use expected-loss regret. Report raw regret; normalize only when the reference denominator is positive and meaningful. If all queries have zero EIG, any permitted no-information-equivalent choice ties on information value. Invalid queries remain protocol errors. Multiple-query tasks need an explicit planning horizon and policy oracle; v0 uses one query to avoid that additional complexity. ### 7.6 Sequential updates and contradictions Report stage-wise KL and change error: $$ UpdateError_t=\sum_h\left|(q_t(h)-q_{t-1}(h))-(p_t(h)-p_{t-1}(h))\right|. $$ Direction accuracy excludes near-zero reference changes under a fixed tolerance. Log-odds movement ratios can be diagnostic, but are unstable near probabilities 0/1 and when the reference movement is tiny. They are not a primary acceptance metric. For conflicts, compute precision/recall over planted same-scope contradictions and score the resulting beliefs against the source-aware model. Lower reliability identifies weaker evidence, not a necessarily false source. Do not require a blame assignment when the evidence cannot identify which report is wrong. For stale beliefs, score required descendant invalidation recall, false invalidation rate, preservation of independently supported beliefs, and whether later reads use the current version. A self-reported supersession is insufficient; inspect the sandbox ledger and actual retrieval results. ### 7.7 Calibration and provenance Outcome calibration uses realized correctness on an appropriate sampling cohort. Report Brier/log loss first, then ECE with fixed bins, sample counts, uncertainty, and category breakdown. Low ECE does not imply good discrimination; report AUROC only where both outcome classes have enough observations. Do not fit calibration on the hidden test set or solely on selectively resolved easy outcomes. Store the following independently: | Field | Example | | --- | --- | | Origin | observation, reported, derived, assumption | | Retrieval channel | operator, tool, memory, summary, other agent | | Original lineage | evidence IDs, derivation, parent claim IDs, source versions | | Current support | raw probability, calibrated probability, supporting and challenging evidence | | Freshness | observed time, valid time, last verification, supersession state | | Legacy label | know, remember, infer, suspect, guess under a documented compatibility policy | Treat remember as a retrieval description rather than an increase or decrease in truth status. Preserve derived origin through memory consolidation. Summary lineage is per claim; copying the weakest label across every claim in a summary is not sound. For deterministic legacy-label tests, publish the compatibility policy and any admissible labels before evaluation. No single confidence threshold makes an origin label objectively correct. ### 7.8 Scorer contract ```python def score_stage(public, oracle, transcript, output): check = validate_response(public, transcript, output) if not check.valid: return protocol_failure(check.errors) # retained in run totals delivered = transcript.delivered_evidence() p = reference_posterior(oracle, delivered, public.model) q = output.distribution scores = posterior_scores(p, q) # KL and squared posterior error scores.update(score_realized_outcome(oracle.latent_state, output)) scores.update(check_claims_and_expressed_derivations(public, delivered, output)) scores.update(score_action_against_oracle(public, p, output)) scores.update(score_provenance_and_state(public, oracle, transcript, output)) return scores ``` For the unknown-model track, omit posterior-fidelity scoring unless the model is identifiable from the supplied evidence. Scorer self-checks must include hand calculations, a second implementation for small worlds, mutation tests, degenerate/tied cases, invalid outputs, and changing transcript evidence. ## 8. Runtime diagnosis and instrumentation ### 8.1 Diagnosis before upgrades Find real call sites and schemas first. The table specifies integration boundaries, not verified Piper filenames or SQL column names. | Hypothesis | Measurement that distinguishes it | | --- | --- | | Abduction is unreachable | Trace intake → planner → tool registry → module; execute a sandbox request and observe entry/exit | | Abduction is optional but never chosen | Count turns where the tool was actually offered and compare with invocation events | | Telemetry misses calls | Compare module entry logs, dispatcher traces, and a known synthetic invocation | | Normal generation supplies adequate inference | Bare/current/disabled-module comparison on the same controlled cases; inspect outputs and artifacts, without claiming access to hidden reasoning | | Remember is a writer default | Group new beliefs by writer and origin; inspect defaults and missing-field behavior | | Consolidation loses lineage | Trace sampled derived claims through write, summary, retrieval, and re-write | | Suspect/guess are unreachable | Exercise label assignment with known low-support examples; inspect policy rather than label frequency alone | | Candidates have no owner | Follow a valid candidate through all readers, schedulers, and actionable queue states | | Candidates are mostly false positives | Independently review all 13 open findings for scope, time, and genuine conflict | | Supersession is bypassed | Compare actual content/version changes with supersession events, including direct database writes | | Contradictions reasonably remain unresolved | Check whether decisive evidence exists and whether explicit unresolved/scoped status is justified | | Costs block invocation | Measure token, latency, and budget limits for the existing module on representative cases | | Evidence acquisition is the common defect | Pair tool-available and tool-unavailable cases; compare choice and outcome rather than module counts | A 20–50 case smoke run can reveal reachability and instrumentation bugs. It cannot establish an accuracy improvement, equivalence to the bare model, or a specific internal reasoning mechanism. ### 8.2 Telemetry events Emit events at the router/planner decision, tool dispatch and receipt, abduction entry/exit, every ledger writer including consolidation, contradiction transitions, and outcome resolution. Use a storage-level audit for direct belief mutations as well as API events. Required event fields: event ID, episode/opportunity ID, stage, timestamp, writer, system commit, belief/source version IDs, event name, structured payload, and attributable cost. Separate actual observations from self-reported intentions. Key events: opportunity.detected; gate.chosen; abduction.started/completed/failed; hypotheses.recorded; query.selected/executed; evidence.delivered; belief.created/challenged/revised/superseded; lineage.changed; contradiction.detected/owned/investigated/resolved/scoped/unresolved; outcome.recorded. Bench opportunity counts come from the generator. Live opportunity estimates come from a sampled, out-of-path auditor, preferably with an independently maintained rubric and human checks. A different model family reduces one source of shared error but does not make the auditor an oracle. Audit both precision and missed opportunities with a sample including detector-negative turns; report sample counts and uncertainty. | Metric | Correct interpretation | | --- | --- | | Triggered opportunities / independently labeled eligible opportunities | Trigger recall and missed routing | | Abduction calls / opportunities where abduction was warranted | Engagement; no universal target such as 40% | | Executed diagnostic queries / generated eligible queries | Whether intended evidence acquisition happens | | Correct revisions / challenges requiring revision | Update behavior; valid retained beliefs are not failures | | Verified resolutions / valid contradictions from a defined cohort | Repair outcome; queue closure alone is insufficient | | Unsupported commitments / audited commitments, plus per-case rate | Quality; includes audit coverage and outcome lag | | Derived lineage lost / derived claims passing through consolidation | Provenance loss; retrieving a derived belief is permitted | | Unversioned content changes / content changes | Silent mutation; distinguish authorized metadata edits | Log invocation counts as diagnostics only. Reward measured outcomes and cost, not forced machinery usage. ## 9. Interventions ordered by experimental priority Priority is conditional on diagnosis. A confirmed write-path defect can take precedence over the gate. Fixing scanner precision precedes assigning owners to bad candidates. Every experiment has a reversible flag and an immutable configuration record. The thresholds below are **proposed engineering defaults**, not literature-derived expected effects. A pilot estimates variance and feasibility; preregister the final endpoints and margins before validation. Measure effect; do not assume the stated target will be reached. | Experiment | Hypothesis, control, and independent variable | Main outcomes and initial sample plan | Proposed acceptance and rollback | | --- | --- | --- | --- | | **E1. Evidence/action gate** | Current Piper versus triggered KNOW/RETRIEVE/INFER/ASK/TEST/ABSTAIN gate; add prompt-only and always-on controls | B/N/O decision regret, overstated claims, useful answer coverage, cost. Pilot 100–200 independent worlds; confirmatory plan typically 500–1,500 after power analysis | Target ≥20% relative reduction in overstated-claim rate if baseline permits; improvement CI excludes zero; coverage noninferiority margin 3 percentage points; cost ≤25% increase. Reject excessive abstention, tool inflation, or latency beyond the registered limit | | **E2. Provenance write fixes** | Current writers versus explicit origin/lineage fields preserved through consolidation | I provenance and lineage accuracy; memory retrieval regression. 100 smoke transformations; about 300 independent claim chains for pilot comparison | Target ≥95% lineage preservation; no more than 2-point retrieval loss. Reject unsupported relabeling or new write failures | | **E3. Contradiction ownership** | Current queue versus owner, next action, priority, and bounded revisit policy | H resolution correctness, unresolved reporting, time to justified resolution. About 300 generated conflicts; separately review the complete live backlog | Target ≥90% correct dispositions and no increase in false resolution. Reject deleting/closing without supporting evidence. TTL expiration records unresolved status, never resolved status | | **E4. Lightweight abduction** | None versus brief structured alternatives versus brief alternatives plus existing full module | A/E/G posterior fidelity, viable-hypothesis coverage, update error. Pilot 300 independent worlds; expand if uncertainty requires | Target ≥10% relative KL improvement where baseline is meaningful, with positive paired evidence of benefit and cost ≤25% increase. Reject hypothesis inflation without support gain | | **E5. Dependency invalidation** | Current revision versus source-version dependency tracking with retained alternative supports | J missed descendants, false invalidation, read correctness. About 300 independent graphs including long chains and redundant support | Target ≥95% required invalidation recall and ≤2% false invalidation. Reject serving stale descendants as current or silently truncating cascades | | **E6. Decision-value query policy** | Existing discriminator versus cost-aware query selection; compare entropy policy as separate arm | F/O expected-loss regret and cost. About 300 worlds with irrelevant, redundant, tied, noisy, and costly queries | Target ≥20% lower regret, positive paired CI, cost ≤25% increase. Reject estimated-value gains that do not improve decision quality | | **E7. Calibration** | Identity mapping versus simple post-hoc calibration selected on calibration-development data | Held-out outcome Brier/log loss; decision utility if mapped confidence controls actions. Pool sparse categories; roughly 500 resolved examples is a pilot, not a guaranteed sufficient N | Require held-out proper-score improvement. Reject worsening log loss or decision utility. If argmax must be preserved, use and verify an argmax-preserving multiclass transformation | | **E8. Failure/inference memory** | Current retrieval versus short evidence-linked prior inference episodes | Repeat-problem accuracy, C distractor resistance, calibration and retrieval cost. About 300 independent tasks with relevant and misleading memories | Require positive task improvement with no meaningful distractor regression. Reject story-matching that overrides present evidence | For sparse-error baselines, relative reductions are unstable. Register an absolute margin or use decision regret as the endpoint. Do not call no statistically significant difference proof that a module is useless; report uncertainty or conduct a powered equivalence test. Implementation details to preserve: - The gate belongs in the dominant decision/write path when that is the confirmed defect. Tool calls must actually execute before the final assertion relies on them. - Abduction stage 1 is conditional, not an always-on requirement to invent three explanations. Stage 2 triggers on expected consequence, unresolved ambiguity, and value relative to cost; a probability-gap heuristic is only a candidate policy. - Dependency processing uses a durable queue and bounded work per cycle. A limit can defer work but cannot drop required descendants. Retrieval must recognize pending invalidation. Handle cycles and independent alternative supports. - A contradiction owner may produce a justified scoped or unresolved outcome. Do not turn queue pressure into an incentive to fabricate certainty. - Calibration should first use pooled or shrinkage methods when category data are sparse. Scalar confidence mappings do not automatically define a coherent multiclass posterior and independent class mappings can alter ranking. - Failure memories store the contemporaneous evidence and later independently resolved outcome. Keep hypotheses and original probabilities unchanged after resolution. ## 10. Baselines, ablations, and experimental controls | System | Main question | | --- | --- | | Bare model | What does this model do with the public evidence and permitted tools? | | Bare + equivalent gate/abduction instruction | How much gain comes from a prompt alone? | | Current Piper | What does the existing architecture add or impair? | | Piper − abduction | Does the existing module affect measured behavior? | | Piper − ledger mechanism | Does the mechanism help when source evidence remains available through an equivalent neutral store? | | Piper − contradiction mechanism | Does detection/repair affect the conflict and revision families? | | Piper + each individual intervention | Which targeted intervention improves its declared endpoint? | | Full accepted candidate | Do individually useful changes combine well? | | Full candidate − each component | What is the marginal contribution under the combined architecture? | Two ledger ablations answer different questions: a mechanism ablation preserves access to the same source evidence, while complete removal measures the total system effect, including lost information. Label them separately. Missing memory evidence must not be mistaken for a reasoning defect. Keep the model version, sampling settings, initial case evidence, tool semantics, token/step/time budgets, and response contract matched. Expose a ledger fixture through a neutral read tool in the bare stateful track. Agent histories can diverge after different queries; compute each posterior from its actual transcript and compare decisions against the same task world. Reset state between independent cases; preserve it within a sequential episode. Run in isolated snapshots. Randomize arm order to reduce provider/time effects. Record provider version changes; do not combine incompatible runs silently. Use two baseline modes when needed: controlled component testing with the structured response contract, and a smaller end-to-end replay through Piper's ordinary intake path. A forced JSON task can itself alter routing. Report that boundary instead of claiming the component bench proves production behavior. ### Sample size and inference Start with 100–200 pilot worlds for variance and failure diagnosis. Repeated model runs characterize stochasticity, but do not multiply the independent sample size. Stages, domain skins, and mutations of the same world are clustered observations. For confirmatory paired comparisons, resample independent worlds/episodes in a clustered bootstrap and retain all arms and stages within each cluster. Use paired binary analysis for error endpoints. Choose one primary endpoint per intervention; distinguish confirmatory from exploratory families and address multiple comparisons. For a rough paired binary planning example, if about 10% of cases are discordant between arms, the standard error is approximately sqrt(0.10/n) near a small net difference. A 95% interval with a 3-point half-width then needs around 427 independent cases. This is an approximation, not a power guarantee; actual power depends on baseline rates, effect, clustering, and discordance. No architecture improvement is established unless it beats the relevant cheap control with sufficient precision. A cheaper prompt can still be a successful shipped improvement to Piper. ## 11. Acceptance, regression, and release decisions Before testing, register the targeted families, primary endpoint, minimum useful effect, noninferiority margins, cost/latency limits, stopping rule, and treatment configuration. Ship an intervention when: 1. Its primary improvement is supported by a paired confidence interval and meets the registered practical-effect target. 2. Guardrail intervals meet the registered noninferiority margins; absence of statistical significance is insufficient. 3. A relevant prompt-only/simpler control does not provide the same benefit more cheaply, or the shipped choice uses that simpler implementation. 4. Targeted gains hold on independently held-out worlds, with transfer results reported by domain and structure. 5. Runtime and protocol failures remain within the registered limits. 6. The corresponding production behavior is supported by shadow evidence or an explicitly limited canary rollout. For v0, consider a 3-percentage-point useful-coverage margin, 2-point memory-retrieval margin, ≤25% incremental measured cost, and a 40% p50 latency ceiling as starting defaults. Tail latency needs an absolute service budget as well as a relative comparison. Agree on thresholds based on Piper's observed baseline; do not reuse these numbers as universal scientific constants. Mechanical lineage repairs can pass on lineage correctness and retrieval guards without improving single-shot KL. Query policies should pass on decision regret. Calibration should pass on proper scoring rules and downstream decision guards. Match the acceptance endpoint to the intervention. Reject or disable for material quality/coverage regression, protocol failure inflation, unsupported conflict closure, lineage loss, excessive cost, or unjustified downstream state changes. Immediate correctness violations warrant immediate disablement. Drift-based rollback uses a registered minimum sample and window; do not wait two weeks after a clear state-corruption defect. The operator, not Piper's mutable scorer, controls hidden evaluation and release flags. Preserve rejected experiments and uncertainty intervals. An inconclusive experiment remains inconclusive. ## 12. Contamination and evaluator protection | Split | Suggested starting scope | Access and purpose | | --- | --- | --- | | Development | Unlimited procedural cases | Piper may inspect and improve against these | | Validation | Frozen independent worlds, initially 300–500 | Operator scoring; used for selection, so track repeated use | | Final test | Power-sized independent worlds; held-out structures and skins | Evaluator only; one selected candidate per fresh final cohort | | Production canaries | Small rotating naturalistic tasks | Detect behavior changes; do not use alone to prove improvement | Keep private seeds, oracle files, future fixture contents, and scorer code inaccessible from the agent's tools. This is an access boundary, not simply a filename convention. Run snapshots and discard test memories. Protect frozen evaluator versions and manifests from the self-improvement loop. Report separate tests for new seeds, new parameter ranges, new structures, and new domains. Holding out collider structures tests structural transfer; holding out business wording tests domain transfer. These are different claims. Track reused validation queries and replace selection cohorts when they become heavily optimized. Testing every candidate once on the same hidden set still adaptively leaks information. A final cohort is reserved for the selected candidate and retired or refreshed under a declared policy after exposure. Do not place a conspicuous canary string in every test prompt: that can teach the agent that all such interactions are tests. Use unexposed run metadata, content fingerprints, and post-run leakage checks instead. For end-to-end canaries, report any effects of test awareness. Mutation tests should preserve scores only when they preserve semantics. Evidence-order permutations and equivalent names should be invariant within sampling noise. Polarity changes require corresponding changes to the world and answer key. ## 13. Real-world shadow evaluation Capture each eligible inference before its outcome is known: exact evidence snapshots or immutable source versions, available tool state, claim, alternatives, raw/calibrated confidence, time, scope, action, and system commit. Record retrospective outcomes in a separate append-only stream. Use a resolver controlled independently of Piper's later assertions: instrument readings, authoritative records, explicit operator corrections, or separately verified outcomes. A later know label written by Piper is not an independent oracle. Distinguish an operator report from direct verification when the situation remains uncertain. A SQLite UPDATE/DELETE trigger can prevent accidental edits but cannot secure history against an agent that can rewrite the database or its containing files. Put the durable outcome/evidence record behind a separate writer service or permissions boundary, with retained snapshots or hashes as appropriate. The implementation must match the actual threat model. Track resolution coverage, age, and reason for resolution. Unresolved examples are not errors, successes, or automatically excluded successes. Outcome-based metrics use the resolved cohort and explicitly disclose selection bias. Retain all captured cases for coverage and audit sampling. Use random audits to check unresolved and detector-negative cases. Shadow evaluation cannot directly measure the downstream consequences of an action that was never executed. Log predicted query choices and estimates in shadow mode; use sandbox replay or a bounded canary rollout to measure the actual interactive policy. Synthetic oracle posterior metrics generally do not exist for real live incidents; do not manufacture them with an LLM judgment. ## 14. Weekly inference report and cost accounting Keep the main report compact, with denominators, cohort dates, and last-week comparisons. Attach detail rather than hiding uncertainty in one headline score. | Report line | Contents | | --- | --- | | Bench status | Last run, system/model/scorer versions, targeted delta versus current and bare controls, CI | | Inference opportunities | Audited sample, estimated eligible count, trigger precision/recall | | Evidence policy | Actual useful queries, infer-when-look-was-better, unnecessary queries, decision regret on controlled cases | | Assertion quality | Overstated/unsupported commitments per assertion and per case; useful coverage | | Calibration | Brier/log loss, ECE with N, categories, outcome-resolution coverage | | Contradictions | Valid cohort size, verified/scoped/unresolved dispositions, false resolutions, age | | Revision | Challenges requiring change, correct revisions, stale descendant retrieval, silent content mutations | | Provenance and cost | Lineage preservation by writer; tokens, dollars, latency, writes, and background work | Counts do not create goals such as maximizing supersessions or making guess labels common. A correctly retained belief is a success when the challenge was invalid. Record input/output and cached tokens, model charges under a dated price record, tool calls, wall-clock latency including failed calls and retries, ledger writes, attributable background computation, and CPU/GPU usage where available. Do not insert unverified current provider prices into the specification. Report raw quality and cost differences first. For comparable probabilistic tasks, supplementary efficiency is: $$ Efficiency=\frac{KL_{control}-KL_{treatment}}{Cost_{treatment}-Cost_{control}}. $$ This ratio is undefined or unstable for zero/tiny incremental cost, and different endpoints cannot be treated as the same quality unit. Use a quality–cost frontier: a cheaper and better intervention dominates a more expensive and worse one. Report latency and resource burden separately from model dollars. ## 15. Failure taxonomy Assign deterministic labels where the generator or transcript supports them. Some causal explanations require audit; mark those as suspected rather than proven. | Failure group | Labels and operational evidence | | --- | --- | | Gap and commitment | Opportunity blindness; missing premise; premature closure; narrative completion; imported assumptions; excessive abstention; missed well-supported conclusion | | Hypotheses | Under-generation; single-hypothesis fixation; late-candidate neglect; hypothesis explosion without posterior-mass gain | | Weighting | Base-rate neglect; false independence; duplicated-evidence counting; reliability neglect; explaining-away failure; absence/delivery confusion | | Grounding | Invalid citation; logical fabrication; omitted defeating evidence; distractor/framing capture; valid outcome with invalid expressed grounds | | Revision | Belief inertia; over-revision; stale source use; ignored descendants; excessive collateral invalidation; loss of alternative support | | Provenance | Derived origin erased; unsupported certainty upgrade; summary lineage loss; time/scope confusion; unversioned content overwrite | | Tools and policy | Infer when acquisition has positive decision value; unnecessary acquisition; wrong query; unexecuted discriminator; invalid request; ignored tool result | | Contradictions | Missed genuine conflict; false conflict; unsupported blame; premature closure; expiration reported as resolution | | Uncertainty | Outcome miscalibration; poor discrimination; category drift; selective-outcome reporting | | Infrastructure | Invalid schema; dropped answer; leaked oracle; mis-scored stage; impossible generated case; unequal budget; provider-version mismatch | A wrong posterior alone does not prove base-rate neglect, and a low-prior answer is not necessarily a base-rate error. Use matched counterfactual task variants or explicit model calculations to diagnose the mechanism. Include infrastructure failures in the report: the harness can be wrong even when the agent is correct. ## 16. Staged implementation and handoff Use milestone gates rather than a mandatory multi-month schedule. A small usable v0 should precede the complete 15-family suite. | Stage | Build or inspect | Completion gate | | --- | --- | --- | | **0. Establish facts** | Invocation coverage, actual call/write paths, label policy, the 13 open contradictions | Timestamped diagnosis with supported and unresolved hypotheses | | **1. Validate the oracle** | Small probabilistic generator; public/private split; transcript-aware posterior and one-query decision oracle | Hand-calculated and independently implemented cases agree within tolerance; degenerate and malformed cases handled | | **2. Run v0** | A/B/F/G/O; 3 domain skins; 100–200 pilot worlds; bare/current/prompt controls | Reliable output capture; uncertainty estimates; dominant error classes identified | | **3. Instrument and audit** | Boundary events; append-only shadow capture; sampled opportunity audit | Known calls and direct writes are captured; auditor precision and recall measured | | **4. Test the first measured fix** | Gate, provenance, or contradiction intervention chosen from diagnosis | Paired validation improvement with appropriate guardrails; cheaper controls evaluated | | **5. Expand selectively** | I/J dependencies and remaining families; withheld structures/domains; combined ablations | Targeted and transfer results distinguish contributions and interactions | | **6. Final evaluation and rollout** | Selected candidate on fresh final cohort; shadow/replay, then limited production rollout | Registered release gates pass; rollback flag and audit record work | | **7. Maintain** | Drift checks, calibrated probabilities, independently resolved inference episodes | Weekly report reflects denominators and outcomes; next experiment follows measured failure | ### Implementation prompt for Piper or the coding harness > Implement Component 8 using this specification. Begin by inspecting the existing abduction, belief, prediction, contradiction, router, and consolidation interfaces. Treat the reported counts as observations to verify, not established causes. Record actual paths and schemas instead of guessing them. > > Build the smallest isolated disclosed-model benchmark for A/B/F/G/O, strict public/private records, transcript-aware exact scoring, and a one-query decision oracle. Prove the scorer on hand-computed and independently computed toy cases before evaluating the agent. Keep hidden seeds, future evidence, and scorer state out of the agent sandbox. > > Run bare-model, current-Piper, and prompt-only controls with matched evidence, tools, budgets, and model configuration. Keep malformed outputs, costs, and harness failures in the results. Report paired differences with clustered uncertainty, separate truth from support, and identify what the evidence cannot yet establish. > > Propose only the first intervention justified by measured failure. Register its primary endpoint, useful-effect threshold, noninferiority margins, and rollback conditions before confirmatory evaluation. Implement behind a flag, compare with the cheaper control, and preserve history. Do not optimize module invocation counts or close contradictions without evidence. > > Deliver the runnable generator/scorer, validated schemas and fixtures, reproducible run commands, manifest and hashes, baseline report, intervention diff, experimental results, cost report, and a clear keep/reject/inconclusive recommendation. Do not mark Component 8 improved until a measured result supports that conclusion. ## References Primary source identities and task/method descriptions checked 3 October 2026. Proposed benchmark formulas, schemas, experimental targets, and engineering adaptations above are this package's specification; they are not claimed as Piper measurements. - **[R1]** Bhagavatula et al. *Abductive Commonsense Reasoning*. ICLR 2020; preprint first submitted 2019. [Paper](https://arxiv.org/abs/1908.05739) · [Authors' implementation](https://github.com/allenai/abductive-commonsense-reasoning). - **[R2]** Rudinger et al. *Thinking Like a Skeptic: Defeasible Inference in Natural Language*. Findings EMNLP 2020. [Paper](https://aclanthology.org/2020.findings-emnlp.418/) · [Authors' dataset](https://github.com/rudinger/defeasible-nli). - **[R3]** Jin et al. *CLadder: Assessing Causal Reasoning in Language Models*. NeurIPS 2023. [Publication](https://papers.nips.cc/paper_files/paper/2023/hash/631bb9434d718ea309af82566347d607-Abstract-Conference.html). - **[R4]** Nafar, Venable, and Kordjamshidi. *Reasoning over Uncertain Text by Generative Large Language Models*. 2024. [Paper](https://arxiv.org/abs/2402.09614). - **[R5]** Li, Kim, and Wang. *QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?* NeurIPS 2025. [Paper](https://arxiv.org/abs/2503.22674). Use this exact paper identity; unrelated later datasets reuse the QuestBench name. - **[R6]** Li et al. *MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning*. 2024. [Paper](https://arxiv.org/abs/2406.00922). - **[R7]** Kirichenko, Ibrahim, Chaudhuri, and Bell. *AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions*. 2025. [Paper](https://arxiv.org/abs/2506.09038). - **[R8]** Kassner, Tafjord, Schütze, and Clark. *BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief*. EMNLP 2021. [Paper](https://aclanthology.org/2021.emnlp-main.697/). - **[R9]** Doyle. *A Truth Maintenance System*. MIT AI Memo 521, 1979. [Primary repository record and paper](https://dspace.mit.edu/entities/publication/5377b306-4ecc-4687-b1f5-78cbb4a0543a). - **[R10]** Geifman and El-Yaniv. *Selective Classification for Deep Neural Networks*. 2017. [Paper](https://arxiv.org/abs/1705.08500). - **[R11]** Guo, Pleiss, Sun, and Weinberger. *On Calibration of Modern Neural Networks*. ICML 2017. [Paper](https://proceedings.mlr.press/v70/guo17a.html).