Summary / 7. Reasoning / Document
2026-10-08
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 7 — Reasoning ## Evidence-to-Conclusion Integrity: Benchmark, Instrumentation, Enhancement, and Experimental Plan **Project:** Piper **Cognitive component:** 7 of 38 — Reasoning **Working definition:** Deriving conclusions from evidence. **Status at start:** Not scored **Current mechanisms:** `reason-worker v1.2.1`, `abduction v1.0`, `belief-ledger v1.3.0`, `delegate-bridge v1.94.0` **Prepared:** 2026-10-08 --- ## Abstract Piper has a functioning reason worker, an epistemic belief ledger, provenance labels, and a rule that inferred claims must not be presented as observations. Those are useful foundations. They do not yet establish that Piper reasons correctly. The central question for Component 7 is narrower and more demanding: > **When Piper is given a body of evidence, do her derived conclusions actually follow from that evidence through a valid sequence of intermediate steps?** This paper defines reasoning as **evidence transformation**: selecting relevant supplied facts, composing them through one or more justified steps, producing intermediate conclusions when needed, and arriving at a final conclusion whose strength does not exceed the support available. Under this definition, provenance is necessary but not sufficient. A perfectly labelled false inference is still false. The benchmark therefore does not score how persuasive Piper's explanation sounds. It scores whether a conclusion is correct, whether each necessary intermediate step is supported, whether irrelevant evidence is ignored, whether missing evidence causes appropriate abstention, and whether observed, stated, inferred, and speculative claims remain correctly distinguished. The primary suite uses frozen evidence packets requiring two to five reasoning steps. Some packets contain distractors, contradictions, tempting shortcuts, or deliberate information gaps. A second production audit blindly samples real rows created by `reason-worker` and checks whether each derived row follows from its cited sources. Research on ProofWriter, EntailmentBank, MuSiQue, verifier-guided proof search, process verification, and chain-of-thought faithfulness supports several design choices here: reason in small auditable steps; distinguish evidence retrieval from evidence composition; include unanswerable cases; verify intermediate steps independently where possible; and never assume that a fluent natural-language rationale faithfully represents the process that produced the answer. The proposed engineering target is a structured **Derivation Record**: not a free-form chain of thought, but an auditable graph of evidence references, intermediate claims, transformations, and verifier results. This gives Piper a way to prove that her derived beliefs are supported without relying on the persuasiveness of her prose. The intended end state is simple: > **A derived Piper belief should be reproducible from its recorded evidence, or it should not be allowed to masquerade as a supported conclusion.** --- # 1. Current State Piper currently describes deliberate reasoning in three parts: 1. reasoning in the active seat; 2. heavier derived work delegated to `reason-worker`; 3. delegated workers required to show evidence. Her epistemic rule is strong: > A conclusion should preserve whether its basis was observed, stated, inferred, or speculative, and an inference should never be represented as direct observation. The reviewer correctly identified the missing measurement. At the 2026-09-30 review: - `reason-worker v1.2.1` was real and active; - it ran approximately every 30 minutes; - it had produced roughly **1,166 derived rows**; - production volume was roughly **10–30 derived rows per day**; - `abduction v1.0` had never run; - the actual database labels were `observed`, `stated`, `inferred`, and `speculative`; - Piper's prose described a different label set; - no mechanism measured whether the derived rows were logically or evidentially justified. This means Component 7 already has unusually valuable raw material: a running stream of real derived beliefs. The first priority is not to add more "reasoning." It is to determine whether the existing reasoning is correct. --- # 2. Scope: What Component 7 Owns The cognition map contains several adjacent components that can easily collapse into one vague reasoning score. They should not. The clean operational boundary is: | Component | Primary question | |---|---| | **6 Logic** | Does the conclusion follow under explicit formal rules or constraints? | | **7 Reasoning** | Does a multi-step conclusion actually follow from the supplied evidence? | | **8 Inference** | What should be believed when important information is missing and conclusions must remain uncertain or revisable? | | **9 Causal reasoning** | Does the evidence support a cause-and-effect claim rather than mere association or sequence? | | **23 Uncertainty** | Does Piper correctly represent how sure she should be? | | **24 Error detection** | Can Piper notice that a reasoning process or conclusion is wrong? | | **25 Self-correction** | Can Piper repair the error once detected? | The boundaries matter because the same case can activate several components, but a benchmark must know which capability it is scoring. ## 2.1 Component 6 — Logic Logic owns formal structure: - if/then rules; - satisfiability; - entailment under explicit axioms; - contradiction; - constraint solving; - quantified rules; - scheduling constraints; - mutually exclusive propositions; - formal proofs. A case that can be translated cleanly into Z3 or another formal solver belongs primarily to Component 6. Component 7 may use formal logic as one reasoning instrument, but it should not duplicate the Logic suite. ## 2.2 Component 7 — Reasoning Reasoning owns **composition over evidence**. Typical cases look like: > Log A establishes event X. > Log B establishes that Y happened after X. > Configuration C means Y would disable Z. > Z is absent. > Therefore the evidence supports conclusion Q. The difficult part is not necessarily formal deduction. It is selecting the right evidence, joining facts across records, preserving temporal or relational dependencies, constructing justified intermediate claims, and not adding assumptions that were never supplied. This component should therefore contain a broad mixture of: - textual evidence; - logs; - timelines; - database rows; - short incident reports; - configuration snippets; - messages; - experiment results; - benchmark outputs; - and structured records. The evidence is supplied. Piper's job is to transform it correctly. ## 2.3 Component 8 — Inference Component 8 has already been defined in the Piper project as **filling gaps from incomplete information**. That distinction is extremely useful. Reasoning asks: > Given these premises, what follows? Inference asks: > Given incomplete evidence, what is the best supported belief, what alternatives remain, and what missing evidence would discriminate between them? Reasoning should therefore be tested mostly on **closed evidence packets** where the required premises are present, plus explicit insufficient-evidence cases where the correct answer is to stop. Inference can then own: - competing hypotheses; - Bayesian updating; - abduction; - evidence acquisition; - confidence revision; - discriminating questions; - belief supersession. This also suggests an architectural cleanup: `abduction v1.0` is conceptually a better fit for Component 8 than for Component 7. If it remains listed under Reasoning, its ownership should be documented explicitly rather than allowing two components to claim the same subsystem. ## 2.4 Provenance is not reasoning Provenance answers: > Where did this claim come from? Reasoning answers: > Does this claim follow? These are independent properties. A claim can be: - correctly marked `inferred` and still be wrong; - correctly reasoned but incorrectly marked `observed`; - unsupported but confidently cited; - supported by the right source but connected through an invalid step. The suite must therefore score **reasoning validity** and **epistemic/provenance hygiene** separately. --- # 3. Research Basis ## 3.1 ProofWriter: build deeper conclusions from auditable local steps Tafjord, Dalvi, and Clark's **ProofWriter** is directly relevant. Rather than producing one large opaque proof, it iteratively generates one-hop implications, adds those implications back to the context, and composes them into deeper proofs. That suggests a useful engineering principle for Piper: > **Long conclusions should be decomposable into locally checkable steps.** For example, if a conclusion requires four joins across logs, Piper should not simply attach four sources to one prose sentence. The derivation should make the intermediate dependencies visible. ProofWriter is primarily a formal/natural-language deduction system, so it should not define Piper's entire reasoning architecture. But its stepwise structure is useful even for less formal evidence. Reference: Tafjord, O., Dalvi, B., & Clark, P. (2021). *ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language.* Findings of ACL-IJCNLP 2021. https://aclanthology.org/2021.findings-acl.317/ ## 3.2 EntailmentBank: explanation as a tree, not merely evidence citation Dalvi et al.'s **EntailmentBank** distinguishes showing relevant evidence from showing **how the evidence produces the answer**. It represents explanations as multi-step entailment trees containing intermediate conclusions. That distinction maps almost exactly onto Piper's current weakness. Today Piper can often point to evidence. Component 7 needs to test the missing relationship: > Does source A plus source B actually justify intermediate C, and does C plus source D justify the final conclusion? EntailmentBank therefore supports evaluating **derivation structure**, not only citation quality. Reference: Dalvi, B., Jansen, P., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. (2021). *Explaining Answers with Entailment Trees.* EMNLP 2021. https://aclanthology.org/2021.emnlp-main.585/ ## 3.3 MuSiQue: connected multi-hop reasoning and unanswerable contrasts Many supposedly multi-hop QA benchmarks can be solved using shortcuts. **MuSiQue** was designed so that one reasoning step depends on another, making disconnected shortcut answering more difficult. It also includes unanswerable contrast questions. Two lessons matter for Piper: 1. benchmark items should be constructed so that multiple pieces of evidence are **actually necessary**; 2. some cases should be deliberately unanswerable even though they look structurally similar to answerable cases. Without these protections, Piper could score well by guessing from one salient clue. Reference: Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). *MuSiQue: Multihop Questions via Single-hop Question Composition.* TACL. Preprint: arXiv:2108.00573. https://arxiv.org/abs/2108.00573 ## 3.4 Verifier-guided reasoning Yang, Deng, and Chen's **NLProofS** generates proof steps and uses an independent verifier to check whether proposed steps are valid. Their central problem is familiar: stepwise generators can hallucinate intermediate steps that sound relevant but do not follow. This suggests that Piper's `reason-worker` should not be the only judge of its own derivations. A lightweight verifier can independently evaluate each transition: > Given supporting claims A and B, is derived claim C supported? For deterministic domains, the verifier can be code or a solver. For naturalistic evidence, it may be another model with a deliberately narrow adjudication prompt. Reference: Yang, K., Deng, J., & Chen, D. (2022). *Generating Natural Language Proofs with Verifier-Guided Search.* EMNLP 2022. https://aclanthology.org/2022.emnlp-main.7/ ## 3.5 ProcessBench: locate the first bad step **ProcessBench** evaluates whether a system can find the earliest erroneous step in a reasoning trace. Although the published dataset focuses on mathematics, the evaluation idea is useful for Piper. A reasoning system should not only know that a final conclusion is wrong. It should be possible to identify the first unsupported transformation. This yields a valuable diagnostic: ```text evidence -> step 1 ✓ -> step 2 ✓ -> step 3 ✗ -> step 4 unsupported ``` A benchmark that records only final-answer accuracy cannot distinguish one small late error from a derivation that was nonsense from the first step. Reference: Zheng, C., Zhang, Z., Zhang, B., et al. (2024). *ProcessBench: Identifying Process Errors in Mathematical Reasoning.* arXiv:2412.06559. https://arxiv.org/abs/2412.06559 ## 3.6 Chain-of-thought is not automatically a faithful audit trail Research by Turpin et al. and Lanham et al. shows why Piper should not simply log free-form chain-of-thought and call it reasoning instrumentation. Models can produce explanations that rationalize a conclusion without faithfully exposing the process that produced it. This paper therefore does **not** propose storing unrestricted internal monologue. Instead, Piper should store a constrained audit artifact: - evidence IDs; - explicit derived claims; - parent relationships; - transformation type; - verifier result. This can be inspected without pretending that eloquent prose is a faithful window into model cognition. References: Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). *Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.* arXiv:2305.04388. https://arxiv.org/abs/2305.04388 Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). *Measuring Faithfulness in Chain-of-Thought Reasoning.* arXiv:2307.13702. https://arxiv.org/abs/2307.13702 ## 3.7 Faithful and constrained reasoning Lyu et al. separate natural-language translation from deterministic problem solving in **Faithful Chain-of-Thought Reasoning**. Poesia et al. similarly use external logical guides to constrain reasoning to valid statements in formal domains. The engineering lesson is not "turn all Piper reasoning into formal logic." That would be impossible and would duplicate Component 6. The useful lesson is: > **Whenever a derivation can be mechanically verified, prefer mechanical verification over model confidence.** A timestamp comparison, set relationship, numeric calculation, dependency graph, SQL condition, formal constraint, or test result should not be left to prose intuition if a deterministic checker can validate it cheaply. References: Lyu, Q., Havaldar, S., Stein, A., et al. (2023). *Faithful Chain-of-Thought Reasoning.* arXiv:2301.13379. https://arxiv.org/abs/2301.13379 Poesia, G., Gandhi, K., Zelikman, E., & Goodman, N. D. (2023). *Certified Deductive Reasoning with Language Models.* arXiv:2306.04031. https://arxiv.org/abs/2306.04031 ## 3.8 Self-consistency is an intervention, not proof Wang et al. showed that sampling multiple reasoning paths and aggregating their answers can improve accuracy on several reasoning tasks. That makes self-consistency worth testing for Piper, but it must not be mistaken for verification. Ten reasoning paths can agree because they share the same hidden assumption. Therefore: > **agreement is evidence of stability, not evidence of correctness.** Self-consistency belongs in the experimental ladder and should be evaluated against cost and correlated-error rate. Reference: Wang, X., Wei, J., Schuurmans, D., et al. (2022). *Self-Consistency Improves Chain of Thought Reasoning in Language Models.* arXiv:2203.11171. https://arxiv.org/abs/2203.11171 --- # 4. Proposed Representation — The Derivation Record Piper needs a reasoning artifact that is more structured than prose and less invasive than attempting to record internal chain-of-thought. Call it a **Derivation Record**. A derivation is a directed acyclic graph whose leaf nodes are evidence and whose internal nodes are derived claims. Example: ```text E1: worker log says job J started at 02:00 [observed] E2: API log records HTTP 429 at 02:03 [observed] E3: retry policy permits only 2 retries [stated/config] E4: no success event exists after 02:03 [observed] D1: J encountered provider rate limiting [inferred from E1,E2] D2: J exhausted its configured retry path [inferred from D1,E3,E4] C1: J did not complete successfully [inferred from D2] ``` This is not intended as a transcript of hidden cognition. It is an **audit proof** for a stored conclusion. ## 4.1 Suggested schema ```text derivation_id created_at producer # seat / reason-worker / delegate producer_version model_provider model_version question_or_trigger final_claim final_epistemic_label final_confidence evidence_nodes[] evidence_id source_ref source_type excerpt_hash timestamp epistemic_label derived_nodes[] node_id claim parent_node_ids[] transform_type assumptions[] verifier_type verifier_result verifier_score confidence missing_evidence[] contradictions[] alternative_conclusions[] status # supported / insufficient / contradicted / verifier-fail written_to_belief_ledger belief_id ``` ## 4.2 Transformation types Do not require philosophical perfection. Use a small practical vocabulary: - conjunction; - comparison; - temporal ordering; - identity/entity join; - category/subsumption; - arithmetic; - rule application; - elimination; - aggregation; - consistency check; - direct entailment; - domain-specific transform; - unsupported/other. The category mainly helps later error analysis. ## 4.3 Assumptions must be explicit A frequent failure mode in reasoning is inserting a plausible missing premise without noticing. Every derived node should therefore distinguish: - evidence; - declared rule; - known standing assumption; - newly introduced assumption. If a new assumption is necessary to complete the derivation, the system should normally move from **supported conclusion** to either: - conditional conclusion; or - insufficient evidence. This is where Component 7 hands off to Component 8. --- # 5. The Central Rule For every derived belief: > **No conclusion may be stronger than the strongest derivation that supports it.** That means: - evidence supports `X` → store `X`; - evidence makes `X` likely → do not store `X` as certain; - evidence supports `X or Y` → do not choose `X` without more evidence; - evidence supports correlation → do not store causation; - evidence supports occurrence after time T → do not claim occurrence because of T; - evidence supports no observed success → do not claim proven failure unless the monitoring coverage makes that valid. This rule should be enforced at the boundary between reasoning and the belief ledger. --- # 6. Failure Taxonomy Every benchmark error and production-audit failure should receive a category. ## R-1 — Evidence omission A necessary supplied fact was ignored. ## R-2 — Distractor capture An irrelevant but salient fact influenced the conclusion. ## R-3 — Unsupported bridge An intermediate claim does not follow from its parents. ## R-4 — Hidden assumption The derivation requires an unstated premise. ## R-5 — Overreach The evidence supports a weaker conclusion than the one asserted. ## R-6 — Invalid join Two records were combined despite referring to different entities, jobs, versions, users, runs, or time windows. ## R-7 — Temporal error Events were ordered, scoped, or related incorrectly. ## R-8 — Quantifier error Examples: - "some" becomes "all"; - "usually" becomes "always"; - absence in one sample becomes universal absence. ## R-9 — Negation error A negative condition is dropped, inverted, or interpreted as a positive observation. ## R-10 — Contradiction blindness Conflicting evidence exists but Piper produces a clean conclusion without acknowledging it. ## R-11 — Premature closure Piper stops at the first plausible explanation instead of completing the required evidence chain. ## R-12 — Insufficient-evidence failure The correct answer is "cannot determine," but Piper chooses a conclusion. ## R-13 — Excessive abstention The necessary evidence is present, but Piper refuses to conclude. ## R-14 — Provenance error The conclusion may be correct, but observed/stated/inferred/speculative status is wrong. ## R-15 — Source laundering A derived claim is later treated as if it were an independent source, producing circular support. ## R-16 — Circular derivation A conclusion directly or indirectly supports itself. ## R-17 — Stale-evidence composition The reasoning is valid over old evidence but invalid against the current state. ## R-18 — Correct answer / invalid path The final conclusion happens to be correct, but one or more necessary intermediate steps are unsupported. This category is important. Final-answer accuracy alone would mistakenly reward it. --- # 7. Benchmark v1 — Frozen Evidence Packets ## 7.1 Objective Measure whether Piper can derive conclusions correctly from supplied evidence while resisting shortcuts, distractors, unsupported assumptions, and overclaiming. The suite should avoid retrieval as the primary bottleneck. All necessary evidence should be present in the packet unless the case is deliberately classified as insufficient. This isolates reasoning from Component 4 Retrieval. ## 7.2 Case families Build **80 frozen cases**. | Family | Cases | Primary stress | |---|---:|---| | A. Clean multi-step derivations | 15 | Basic 2–5 hop composition | | B. Logs and timelines | 15 | Temporal joins, identity, state transitions | | C. Distractor-rich packets | 10 | Relevance selection | | D. Insufficient evidence | 15 | Correct abstention | | E. Contradictory evidence | 10 | Conflict recognition | | F. Shortcut/adversarial cases | 10 | Hidden assumptions, intuitive but unsupported answers | | G. Alternate-path cases | 5 | Robustness to different valid derivations | | **Total** | **80** | | Use a mixture of synthetic and Piper-shaped cases. ## 7.3 Case construction Every case should contain: ```text case_id question evidence[] hidden_answer acceptable_answer_variants required_support_set optional_support_set forbidden_support_set gold_intermediate_nodes[] minimum_steps maximum_reasonable_steps answerability failure_tags ``` For naturalistic cases where there are multiple valid reasoning paths, the gold should store a **proof graph** rather than a single exact prose explanation. ## 7.4 Depth bands Tag cases by minimum required depth: - D1 — one meaningful transformation; - D2 — two dependent steps; - D3 — three dependent steps; - D4 — four dependent steps; - D5 — five dependent steps. The suite should report accuracy by depth. If scores collapse from D2 to D4, the failure is more informative than one aggregate percentage. ## 7.5 Evidence density Also tag packets by distractor ratio: - low noise: almost every item relevant; - medium noise: 30–50% distractors; - high noise: most evidence irrelevant. This tests whether reason quality depends on a pre-cleaned packet. --- # 8. Benchmark v1 — Production Derivation Audit The frozen suite measures controlled competence. The production audit measures whether the real `reason-worker` uses that competence in Piper's actual system. ## 8.1 Sampling Each evaluation window should randomly sample **100 derived rows** from the reason worker, stratified where possible by: - source type; - age; - confidence; - project; - derivation depth; - producer/model; - consequence. Do not let Piper select which rows are audited. ## 8.2 Blind review packet For each sampled row, the reviewer receives: - the derived claim; - its cited source evidence; - timestamps; - relevant source context; - no information about whether Piper believed the conclusion strongly; - no indication of the production outcome if that would bias evaluation. The reviewer labels: - **supported**; - **partially supported / overstated**; - **unsupported**; - **contradicted**; - **cannot evaluate from recorded evidence**. Then score provenance separately. ## 8.3 Consequential sampling Random samples can hide rare but important failures. Add a second sample of the most consequential derived rows: - rows used to change a project; - rows used to dispatch work; - rows causing alerts; - rows that modified a belief; - rows that influenced money or compute use; - rows later cited in operator-facing statements. The consequential sample should be reported separately, not mixed into the random rate. --- # 9. Scoring Component 7 score: **100 points**. Use both frozen and live evidence. ## 9.1 Frozen benchmark — 75 points ### Final conclusion correctness — 25 Does Piper reach the hidden supported answer? For insufficient packets, the correct answer is an explicit insufficiency result. ### Step validity — 20 Are the required intermediate transformations supported? Score the earliest invalid step and downstream contamination. A correct final answer does not receive full credit if the derivation is invalid. ### Evidence sufficiency and relevance — 10 Does Piper use the evidence actually needed? Penalize: - missing required evidence; - reliance on forbidden distractors; - source mismatch; - unsupported assumptions. ### Abstention discrimination — 10 Across answerable and unanswerable cases, measure whether Piper knows when the packet is insufficient. Use both: - false conclusion rate; - unnecessary abstention rate. ### Provenance hygiene — 5 Are claims correctly labelled as: - observed; - stated; - inferred; - speculative? ### Robustness — 5 Test harmless perturbations: - evidence order; - paraphrase; - irrelevant extra record; - changed entity names; - different but equivalent reasoning route. The answer should remain stable when the underlying evidence relations are unchanged. ## 9.2 Production audit — 25 points ### Supported derived-row rate — 15 Blind audit of random real `reason-worker` rows. ### Consequential derivation integrity — 5 Audit high-impact rows. ### Reproducibility / provenance completeness — 5 Can the evaluator reconstruct why the row exists from stored sources and derivation metadata? If a conclusion cannot be audited because its evidence is missing, that is a Component 7 failure even if the conclusion later turns out to be true. --- # 10. Diagnostic Metrics Track these without necessarily including all of them in the headline score. ## 10.1 Accuracy by depth ```text D1 D2 D3 D4 D5 ``` ## 10.2 Earliest-error position Where does the first unsupported step occur? ## 10.3 Bridge validity Percentage of intermediate nodes independently judged valid. ## 10.4 Evidence precision Of the evidence cited, how much is actually relevant? ## 10.5 Evidence recall Of the evidence required for the conclusion, how much was used? ## 10.6 Abstention precision When Piper says "insufficient," how often is the packet genuinely insufficient? ## 10.7 Abstention recall Of genuinely insufficient packets, how many does Piper correctly refuse? ## 10.8 Assumption injection rate How often does a derivation depend on a premise not supplied or declared? This is likely one of the most useful live metrics. ## 10.9 Overclaim rate How often is the final claim stronger than its support? ## 10.10 Provenance accuracy Do `observed`, `stated`, `inferred`, and `speculative` match the actual derivation? ## 10.11 Circular-support rate Target: zero. ## 10.12 Unsupported production-belief rate Of beliefs written from reasoning, what proportion fail blind support review? ## 10.13 Derivation coverage What percentage of real inferred beliefs have enough source linkage to be independently audited? --- # 11. The Benchmark Must Contain "Can't Tell" The most important adversarial family is deliberate insufficiency. A model can appear intelligent by always constructing a story. Piper should instead have a first-class result: ```text status: insufficient_evidence supported: - A - B not_supported: - C missing: - X ``` Example: ```text Evidence: - Worker started at 03:00. - No completion row appears in the work ledger. - Ledger ingestion was unavailable from 03:05 to 04:00. Question: Did the worker fail? ``` Correct answer: > Cannot determine from this evidence. Absence of a completion row is non-diagnostic while ledger ingestion was unavailable. Worker-side completion/failure evidence is required. A model that confidently says "the worker failed" has not reasoned better because its story is cleaner. --- # 12. Adversarial Benchmark Patterns ## 12.1 Salient distractor A dramatic log line appears near the actual failure but belongs to another job. Test: entity join discipline. ## 12.2 Timestamp trap Records are listed in narrative order rather than chronological order. Test: whether Piper reasons from timestamps rather than presentation sequence. ## 12.3 Version trap A configuration rule changed between run A and run B. Test: stale rule application. ## 12.4 Missing negative A rule says: > Alert if X **and not Y**. The packet contains X and Y. Test: negation preservation. ## 12.5 Correlation lure Two events repeatedly co-occur. Question wording tempts a causal conclusion. Correct behavior: association can be concluded; causation cannot. This case touches Component 9, but Component 7 is scored only on whether Piper overreaches beyond the evidence. ## 12.6 Closed-world trap A record is absent from one table. Nothing states that the table is exhaustive. Correct behavior: absence is not automatically evidence of nonexistence. ## 12.7 Duplicate-source trap Three summaries all derive from the same original event. Test: Piper should not treat them as three independent confirmations. ## 12.8 Circular belief trap A belief generated by the reason worker is later retrieved as evidence for itself through a summary. Test: source laundering and circularity. ## 12.9 Operator-language lure The operator says, "looks like X is broken," while the evidence is ambiguous. Correct behavior: the statement is evidence that the operator suspects X, not direct evidence that X is broken. ## 12.10 Correct-answer trap Construct a case where a common shortcut yields the correct answer but through invalid reasoning. Score the conclusion as correct but the derivation as failed. This prevents the suite from teaching Piper that lucky guesses are sound reasoning. --- # 13. Instrumentation Phase Do this before improving the reasoner. ## 13.1 Canonicalize provenance vocabulary The system description and database currently disagree. Choose one canonical set. The live database already has: - `observed` - `stated` - `inferred` - `speculative` That is a reasonable taxonomy. Update documentation and interfaces so Piper does not describe labels that do not exist. ## 13.2 Add source lineage Every `inferred` belief should have machine-readable parent references. Not merely: ```text source = "logs" ``` but: ```text parents = [log:abc123, config:def456, worker:run789] ``` ## 13.3 Add derivation status Suggested: ```text supported insufficient contradicted verification_failed superseded ``` ## 13.4 Log failed derivations Do not record only conclusions that survived. Log: - candidate conclusion; - failed bridge; - missing evidence; - rejection reason. Otherwise the system cannot learn where reasoning attempts fail. ## 13.5 Record write consequences If a derived row later influences: - an operator answer; - a project; - an alert; - a dispatch; - an experiment; - another derivation; record that dependency. This makes consequential reasoning auditable. --- # 14. Enhancement Ladder Each intervention should be measured independently. Do not replace the entire reasoning stack at once. --- ## Intervention 0 — Measurement only ### Change Add: - canonical provenance; - source lineage; - derivation IDs; - blind production sampling; - outcome labels. Do not change `reason-worker` behavior. ### Hypothesis The current system's main weakness is unknown because there is no ground-truth audit. ### Prediction Capability score may not improve, but unsupported derivation rate becomes measurable. ### Keep if At least 95% of sampled derived rows can be traced to their cited sources. --- ## Intervention 1 — Structured Derivation Record ### Change Require production-derived beliefs to contain: - evidence parents; - intermediate claims where needed; - final claim; - epistemic label; - explicit assumptions. ### Hypothesis Forcing explicit support structure will expose unsupported bridges and hidden assumptions. ### Prediction Provenance completeness and bridge validity increase; latency rises modestly. ### Failure risk The model may generate attractive post-hoc structures that merely rationalize its answer. Therefore this intervention alone is insufficient. --- ## Intervention 2 — Independent step verifier ### Change For each derived node: > Given only the parent claims and the proposed child claim, judge whether the child is supported. Use deterministic verification when available. Otherwise use a narrow independent verifier model/process. ### Hypothesis The reason worker is poorly positioned to detect its own plausible unsupported steps. ### Prediction Unsupported-bridge rate falls. ### Key metric First-invalid-step rate on the frozen suite. ### Important rule The verifier should not be given the desired final answer when judging an intermediate step if that can bias it toward approving the path. --- ## Intervention 3 — Evidence sufficiency gate ### Change Before writing the conclusion, ask: ```text Can the requested conclusion be established from the available evidence without introducing a new premise? ``` If no: ```text status = insufficient ``` ### Hypothesis Piper currently has a structural incentive to produce a conclusion whenever the reason worker fires. ### Prediction Insufficient-evidence accuracy rises substantially while answerable-case abstention stays low. ### Danger Overcorrection into excessive refusal. Measure both kinds of error. --- ## Intervention 4 — Minimal support challenge ### Change After deriving a conclusion: 1. identify the minimal support set; 2. remove each key premise in turn; 3. ask whether the conclusion still follows. ### Hypothesis This detects evidence that is cited decoratively rather than causally necessary to the derivation. ### Prediction Evidence precision improves and source laundering falls. --- ## Intervention 5 — Counter-derivation ### Change For consequential conclusions only, ask an independent process: > What is the strongest alternative conclusion compatible with the same evidence? This is not full abduction. The goal is to detect whether the evidence uniquely supports the proposed conclusion. ### Hypothesis Piper prematurely closes on one plausible interpretation. ### Prediction Overclaim rate falls. ### Component boundary If genuine competing hypotheses must be generated and updated under missing evidence, hand the case to Component 8. --- ## Intervention 6 — Verify-then-repair ### Change If the step verifier rejects a derivation: 1. locate earliest invalid step; 2. discard all descendants; 3. rebuild from the last valid state; 4. reverify. ### Hypothesis Local repair is cheaper and more reliable than asking the model to reconsider the entire answer vaguely. ### Prediction More failed derivations are repaired without increasing hallucinated support. --- ## Intervention 7 — Multi-path reasoning ### Change For hard or high-consequence packets, sample multiple derivations. Compare: - conclusions; - support sets; - intermediate nodes. ### Hypothesis Alternative reasoning paths reduce brittle single-path errors. ### Prediction Final accuracy improves most on depth D4–D5. ### Important caveat Do not accept a conclusion merely because paths vote for it. Shared-model errors are correlated. Agreement should trigger confidence adjustment, not bypass verification. --- ## Intervention 8 — Consequence-sensitive verification ### Change Increase verification effort when a derived belief can cause a consequential action. Example tiers: ```text Tier 0: ephemeral thought Tier 1: stored low-impact belief Tier 2: operator-facing claim Tier 3: project/action change Tier 4: money, security, destructive action, governance ``` ### Hypothesis Uniform reasoning cost wastes compute on trivial conclusions and under-verifies consequential ones. ### Prediction Consequential derivation integrity rises with tolerable compute growth. --- # 15. Ablation Plan After the baseline, test mechanisms rather than only whole-system versions. | Condition | Structured derivation | Verifier | Sufficiency gate | Multi-path | |---|---:|---:|---:|---:| | A | off | off | off | off | | B | on | off | off | off | | C | on | on | off | off | | D | on | on | on | off | | E | on | on | on | on | Compare: - final accuracy; - step validity; - abstention; - latency; - token cost; - production unsupported-row rate. Interpretation: - A→B tests whether structure alone helps. - B→C tests independent verification. - C→D tests insufficiency handling. - D→E tests whether additional paths are worth their cost. --- # 16. Worker Architecture Experiment Piper currently reasons both in the seat and through `reason-worker`. That creates a useful natural experiment. Run the same frozen cases through: 1. seat only; 2. reason-worker only; 3. delegate worker only; 4. reason-worker + verifier; 5. seat + verifier. Do not assume the dedicated reason worker is automatically better. Measure: - accuracy; - derivation validity; - abstention; - latency; - cost. The result should determine routing policy. A dedicated worker that is more verbose but not more correct should not own heavy reasoning merely because its name is `reason-worker`. --- # 17. Real-World "Trip-the-System" Probe Component 7 needs a live task that reliably creates multi-step reasoning traces. Use an **Incident Reconstruction Drill**. ## 17.1 Goal Give Piper a realistic operational incident whose answer cannot be obtained from one record. The case should require joining evidence from several sources and should contain at least one tempting but unsupported narrative. ## 17.2 Example shape Create a fictional or sandboxed incident involving: - a worker launched at 02:00; - a service API error at 02:07; - a watchdog message at 02:10; - a stale dashboard snapshot; - a configuration change from the previous day; - one unrelated warning from another process; - a ledger gap; - and a later successful retry from a different run. Ask Piper: > Reconstruct what can be established about the incident. What failed, what did not fail, and what remains unknown? Cite the evidence supporting each derived conclusion. Do not tell her which records matter. ## 17.3 Deliberate traps Include: - a dramatic irrelevant warning; - records displayed out of chronological order; - one summary derived from another summary; - one missing record whose absence is not diagnostic; - two similar run IDs; - an old configuration that no longer applies. ## 17.4 What should fire A good run should create evidence of: - relevance selection; - temporal reasoning; - entity matching; - multi-step composition; - insufficiency detection; - provenance labeling; - derived-belief creation. It should also create opportunities for Components 24 and 25 if the first path contains an error. ## 17.5 Evaluation Blindly compare Piper's incident reconstruction with a hidden ground-truth event graph. Score: - valid conclusions; - invalid conclusions; - missed conclusions; - first bad reasoning step; - correct unknowns; - provenance; - source lineage. This is preferable to another trivia-style reasoning benchmark because it looks like the work Piper actually performs. --- # 18. Additional Live Probe — Derived-Row Challenge The `reason-worker` already creates 10–30 rows per day. Use that stream directly. Once per day: 1. randomly select three newly derived rows; 2. hide Piper's original explanation; 3. give another evaluator only the cited evidence; 4. ask whether the row follows; 5. record the verdict; 6. if unsupported, identify the failure category. Once per week, select one row that materially affected subsequent work and perform a deeper audit. This should become permanent infrastructure even after the Component 7 suite is retired. --- # 19. Contamination and Shortcut Controls Reasoning benchmarks are especially vulnerable to models recognizing known questions or exploiting dataset artifacts. The Piper suite should therefore be mostly local and generated. ## 19.1 Use fictional entities Avoid famous people, common benchmark names, or known puzzle phrasing. ## 19.2 Randomize surface forms Generate equivalent variants: - names; - IDs; - timestamps; - record order; - harmless wording. The underlying relation graph remains constant. ## 19.3 Counterbalance answers Do not allow: - most insufficient cases to use one phrase; - contradiction cases to have longer packets; - answerable cases to contain more logs; - particular labels to correlate with correct answers. ## 19.4 Keep hidden test cases frozen Development set and test set must be separate. Interventions tune on development cases only. ## 19.5 Regenerate after retirement When the suite is retired, produce a new hidden set from the same generators. A memorized benchmark is not evidence of improved reasoning. --- # 20. Benchmark Generator Design Rather than hand-authoring all 80 cases forever, build case generators. Each case begins as a hidden evidence graph. Example: ```text A -> B B + C -> D D + E -> conclusion ``` Then render the graph into: - logs; - prose; - messages; - config; - records. Distractors are generated separately. For insufficient variants, remove one necessary node. For contradiction variants, add a conflicting observation. For shortcut traps, add a clue correlated with the answer but not logically sufficient. This gives exact hidden ground truth while keeping the surface language fresh. --- # 21. Scoring a Derivation Graph A useful automated comparison can score nodes and edges. ## Node score Does the derived claim correspond to an allowed gold intermediate or a logically equivalent one? ## Edge score Do the cited parents actually support the child? ## Support recall Did Piper include the required gold evidence? ## Support precision Did Piper rely on irrelevant or forbidden evidence? ## Terminal score Does the final claim match the supported answer? ## Abstention score If the graph cannot reach the target, did Piper stop instead of inventing an edge? This is more informative than string-matching an explanation. --- # 22. Handling Naturalistic Cases Not every real Piper conclusion can be represented with perfect formal semantics. Use a hierarchy of graders. ## Tier 1 — Deterministic Use for: - arithmetic; - timestamps; - set membership; - exact config rules; - database equality; - version comparisons; - graph reachability; - formal constraints. ## Tier 2 — Rule-based semantic Use for controlled synthetic cases with known templates. ## Tier 3 — Independent model verifier Use for natural-language entailment where deterministic checking is impractical. Give the verifier only: - parent claims; - proposed child; - narrow instruction. ## Tier 4 — Human/operator adjudication Use for: - ambiguous high-consequence cases; - benchmark disputes; - sampled production audits. The suite should record grader type because a 95% score from deterministic checks is not equivalent to 95% agreement with one LLM judge. --- # 23. Interaction with the Belief Ledger The belief ledger should become the enforcement point. Suggested write rules: ## Observed May be written only from a source classified as direct system evidence. ## Stated May be written when a person/source asserted the claim, without promoting the assertion itself to fact. ## Inferred Requires: - parent evidence references; - successful derivation status; - no unresolved failed verifier step; - confidence/strength appropriate to support. ## Speculative May exist without a validated derivation, but must not silently upgrade itself into `inferred`. A later fresh read should be able to distinguish: > "the operator said X" from: > "X was observed" from: > "Piper inferred X" from: > "Piper considered X possible." That distinction is one of the strongest parts of the current architecture and should be enforced rather than merely described. --- # 24. Source Laundering Defense This deserves explicit engineering because Piper has layered memory, summaries, workers, and derived beliefs. Suppose: ```text raw evidence E -> reason-worker derives D -> summary S mentions D -> later reasoning retrieves S -> Piper cites S as independent support for D ``` The system can accidentally turn one inference into multiple "sources." Every derived and summarized claim should therefore preserve an **ultimate-source lineage**. When two pieces of evidence descend from the same original source, the reasoner should know they are not independent confirmations. Suggested field: ```text root_evidence_ids[] ``` This will also help Component 23 calibration and Component 8 inference later. --- # 25. Staleness and Reasoning A derivation can be valid when produced and false later. Example: ```text 09:00 evidence -> "service is down" 09:05 service recovers 10:00 summary still says "service is down" ``` This is not necessarily a reasoning error at 09:00. It becomes a state-management error if the conclusion is reused at 10:00 without revalidation. Therefore store: - derivation time; - evidence timestamps; - validity horizon where known; - dependencies that can invalidate the conclusion. Component 7 should score whether the original derivation followed. Component 1 Perception / Component 4 Retrieval / state freshness machinery should own whether new evidence was noticed. But production audits should tag **stale-evidence composition** separately because it affects real behavior. --- # 26. Acceptance Criteria for Suite v1 Do not retire Component 7 on aggregate score alone. Suggested minimums: - overall score >= **88/100**; - frozen final-answer correctness >= **92%**; - step validity >= **95%**; - insufficient-evidence true-positive rate >= **90%**; - unnecessary-abstention rate <= **8%**; - provenance accuracy >= **98%**; - unsupported-bridge rate <= **3%**; - circular-support rate = **0%**; - source-lineage completeness >= **98%**; - blind random production audit supported-or-properly-qualified >= **95%**; - consequential production audit supported-or-properly-qualified >= **98%**; - no depth band D1–D5 below **80%**; - no regression greater than 3 points when evidence order and surface wording are perturbed. The production requirements matter. A model that aces frozen packets while `reason-worker` continues producing unsupported beliefs has not solved Component 7. --- # 27. Baseline Procedure Before changing the reasoning architecture: 1. Freeze current `reason-worker`, seat prompts, and delegate behavior. 2. Canonicalize only enough instrumentation to capture the run. 3. Build 20 development cases. 4. Build 20 hidden baseline cases. 5. Include at least five insufficient-evidence cases. 6. Run three seeds per case where stochasticity exists. 7. Sample 100 historical or recent production-derived rows. 8. Blind-audit them. 9. Record the first score as the official v1 baseline. 10. Only then begin interventions. Do not fix obvious errors before establishing the baseline. The baseline's purpose is to preserve evidence that the intervention worked. --- # 28. Experimental Record Template ```text Component: 7 Reasoning Intervention: Date: Reason-worker version: Belief-ledger version: Verifier version: Model/provider: Benchmark version: Hypothesis: Predicted primary effect: Predicted side effects: Before: - overall: - conclusion accuracy: - step validity: - abstention: - provenance: - production supported-row rate: - latency: - cost: After: - overall: - conclusion accuracy: - step validity: - abstention: - provenance: - production supported-row rate: - latency: - cost: Depth breakdown: D1: D2: D3: D4: D5: Failure taxonomy changes: Production audit result: Decision: KEEP / MODIFY / REVERT Reason: ``` --- # 29. Research Questions for Later Versions ## RQ1 — Does structured derivation improve actual correctness or only auditability? This must be tested. A structured explanation can still be a rationalization. ## RQ2 — What verifier architecture generalizes best? Compare: - same model; - different model; - small cheap verifier; - deterministic checker; - ensemble; - consequence-tiered verifier. ## RQ3 — What derivation depth requires multi-path reasoning? Maybe D1–D2 is best single-pass, while D4–D5 benefits from search. Measure the crossover rather than applying expensive search universally. ## RQ4 — Can Piper predict when her reasoning will fail? Before verification, ask for probability that the derivation contains an unsupported step. Compare that prediction to verifier and human outcomes. This links Component 7 with Component 23. ## RQ5 — Does reasoning degrade when evidence mixes fresh and summarized sources? Create controlled cases where the same fact appears through: - raw log; - summary; - belief; - worker report. Measure source laundering and overcounting. ## RQ6 — Can Piper compress a valid derivation without losing support? For long-lived beliefs, full derivation graphs may be expensive. Test whether a compact support certificate preserves enough information for future audit and invalidation. ## RQ7 — Can reason-worker failures predict downstream mistakes? Measure whether unsupported derived rows later correlate with: - wrong answers; - wasted work; - bad project choices; - unnecessary retrieval; - self-correction events. That establishes the real utility of improving Component 7. --- # 30. Suggested Dashboard Metrics For live use, display: ```text Reasoning — last 7 days derived rows: N auditable derivations: % verified derivations: % verifier reject rate: % repaired after reject: % insufficient-evidence exits: N unsupported bridge rate: % assumption injection rate: % provenance accuracy: % source-lineage completeness: % circular derivations: N blind audit supported: % consequential audit supported: % median derivation depth: X median verification cost: $ ``` Do not treat more derived rows as better. The target is **more trustworthy conclusions**, not more reasoning output. --- # 31. What Should Happen to `abduction v1.0` The current page lists `abduction v1.0` under Reasoning even though it has never run. Architecturally, abduction is the process of finding plausible explanations for incomplete observations. That aligns more closely with Component 8's established definition: filling gaps from incomplete information using competing hypotheses and discriminating evidence. Recommended options: 1. **Move primary ownership to Component 8**, while allowing Component 7 to consume an abductive hypothesis as input; or 2. explicitly define a narrow Reasoning use where abduction proposes candidate bridges but cannot write a supported conclusion without Component 8 uncertainty machinery. Do not turn on abduction merely to make Component 7 look more active. The important current evidence is the 1,166 real derived rows from `reason-worker`. Measure them first. --- # 32. Recommended Initial Implementation Order The first engineering cycle should be: ### Phase 1 — Measurement - provenance vocabulary alignment; - source lineage; - derivation IDs; - production sampler; - blind auditor. ### Phase 2 — Benchmark - 20 development packets; - 20 hidden packets; - depth tags; - insufficient cases; - automated score report. ### Phase 3 — Baseline - frozen suite; - production sample; - seat/worker comparison. ### Phase 4 — Smallest intervention Choose based on actual failure distribution. Examples: - mostly hidden assumptions → sufficiency gate; - mostly invalid bridges → step verifier; - mostly source mismatch → lineage/join discipline; - mostly D4–D5 collapse → multi-path or decomposition; - mostly correct reasoning but wrong labels → provenance enforcement. ### Phase 5 — Retest Keep or revert based on predicted metrics. This avoids building a large "reasoning engine" before identifying what is broken. --- # 33. Example Claude Code Requests This paper should be usable as a working engineering specification. ## Instrumentation > Review the Component 7 Reasoning paper. Implement Phase 1 only. Add canonical provenance labels, machine-readable source lineage for inferred beliefs, derivation IDs, producer/version metadata, and a random production-derived-row sampler. Do not alter reasoning prompts or selection behavior. Include migrations, tests, metrics, and rollback instructions. ## Benchmark > Review the Component 7 Reasoning paper. Build Reasoning Benchmark v1 infrastructure for frozen evidence packets. Support 2–5 step gold derivation graphs, answerable and insufficient cases, required/optional/forbidden evidence sets, evidence-order perturbations, final-answer scoring, step scoring, and failure taxonomy output. Do not tune the reason worker. ## Verifier > Review the Component 7 Reasoning paper. Implement Intervention 2 as an isolated experiment. Given parent evidence nodes and one proposed derived node, verify whether the child is supported. Use deterministic checks when the transform type permits them and an independent narrow model judge otherwise. Record verifier type and result. Do not automatically repair rejected derivations yet. ## Production audit > Review the Component 7 Reasoning paper. Build a weekly blind audit job that samples 100 reason-worker derived rows plus a separate consequential sample. Package each row with its source evidence but hide Piper's original rationale. Store supported / overstated / unsupported / contradicted / unevaluable labels and generate the Component 7 live dashboard. ## Ablation > Review the Component 7 Reasoning paper. Run the frozen suite under baseline, structured-derivation-only, structured+verifier, structured+verifier+sufficiency-gate, and full multi-path conditions. Report accuracy, step validity, abstention, depth curves, cost, and latency. Make no production changes. --- # 34. Recommended First Research Experiment The highest-value first experiment is not a new prompt. It is: > **Blindly audit 100 of the 1,166 existing derived rows.** That sample can answer several questions immediately: - Is the reviewer concern mostly theoretical, or are derived rows actually wrong? - What percentage are unsupported? - Are failures caused by evidence omission, invalid bridges, hidden assumptions, stale evidence, or provenance? - Are the wrong rows low consequence? - Does confidence correlate with correctness? - Are multi-source derivations worse or better than single-source ones? - Does one producer/model account for most errors? The answer determines what should be built next. If 98 of 100 rows follow cleanly, the problem is mostly measurement. If 30 of 100 contain hidden assumptions, build the sufficiency gate. If errors cluster only in depth 4+, add decomposition/search. If source laundering is common, fix lineage before touching the reasoner. This is the cheapest way to turn Component 7 from speculation into engineering. --- # 35. Interpretation Piper's current epistemic discipline is a real strength. The rule: > an inference must not pretend to be an observation solves one important class of failure. But it does not solve: > **Was the inference valid?** Component 7 exists to answer that second question. The mature version of the system should make a derived belief behave less like a sentence Piper happened to generate and more like the output of a small scientific notebook: ```text Here is the claim. Here is the evidence. Here are the transformations. Here is what was assumed. Here is what remains unknown. Here is whether the steps were independently checked. Here is what would invalidate the conclusion. ``` The benchmark should reward neither eloquence nor sheer reasoning volume. It should reward **support-preserving transformation**. The most important behavioral transition is from: > "I found several things that point toward X." to: > "E1 and E2 establish A. A together with E3 establishes B. B is sufficient for X under rule R. No unstated premise is required. Therefore X is supported." And just as importantly: > "The chain stops at B. X would require evidence E4, which I do not have. Therefore I cannot conclude X." That is the capability Component 7 should eventually certify. --- # 36. One-Sentence Definition for the Piper Summary Page **Reasoning is the ability to transform supplied evidence through valid, auditable intermediate steps into conclusions that are no stronger than their support — including stopping when the evidence does not complete the chain.** --- # 37. Short Reviewer Update After Implementation A future reviewer should be able to replace: > **What needs improving: Nothing checks whether her derived conclusions are right** with something like: > **Reasoning is now measured both synthetically and in production. Every inferred belief can carry a source-linked derivation record; intermediate steps are independently checked where practical; insufficient evidence is a first-class result; and random live reason-worker rows are blind-audited. The remaining weakness is concentrated in depth-4+ mixed-source derivations rather than reasoning being wholly unmeasured.** That would be evidence of an actual capability improvement rather than a better description of the same system. --- # References 1. Tafjord, O., Dalvi, B., & Clark, P. (2021). **ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language.** Findings of ACL-IJCNLP 2021, 3621–3634. https://aclanthology.org/2021.findings-acl.317/ 2. Dalvi, B., Jansen, P., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. (2021). **Explaining Answers with Entailment Trees.** EMNLP 2021, 7358–7370. https://aclanthology.org/2021.emnlp-main.585/ 3. Trivedi, H., Balasubramanian, N., Khot, T., & Sabharwal, A. (2022). **MuSiQue: Multihop Questions via Single-hop Question Composition.** Transactions of the Association for Computational Linguistics. Preprint arXiv:2108.00573. https://arxiv.org/abs/2108.00573 4. Yang, K., Deng, J., & Chen, D. (2022). **Generating Natural Language Proofs with Verifier-Guided Search.** EMNLP 2022. https://aclanthology.org/2022.emnlp-main.7/ 5. Han, S., Schoelkopf, H., Zhao, Y., et al. (2022). **FOLIO: Natural Language Reasoning with First-Order Logic.** arXiv:2209.00840. https://arxiv.org/abs/2209.00840 6. Lyu, Q., Havaldar, S., Stein, A., et al. (2023). **Faithful Chain-of-Thought Reasoning.** arXiv:2301.13379. https://arxiv.org/abs/2301.13379 7. Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). **Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting.** arXiv:2305.04388. https://arxiv.org/abs/2305.04388 8. Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). **Measuring Faithfulness in Chain-of-Thought Reasoning.** arXiv:2307.13702. https://arxiv.org/abs/2307.13702 9. Poesia, G., Gandhi, K., Zelikman, E., & Goodman, N. D. (2023). **Certified Deductive Reasoning with Language Models.** arXiv:2306.04031. https://arxiv.org/abs/2306.04031 10. Dhuliawala, S., Komeili, M., Xu, J., et al. (2023). **Chain-of-Verification Reduces Hallucination in Large Language Models.** arXiv:2309.11495. https://arxiv.org/abs/2309.11495 11. Wang, X., Wei, J., Schuurmans, D., et al. (2022). **Self-Consistency Improves Chain of Thought Reasoning in Language Models.** arXiv:2203.11171. https://arxiv.org/abs/2203.11171 12. Zheng, C., Zhang, Z., Zhang, B., et al. (2024). **ProcessBench: Identifying Process Errors in Mathematical Reasoning.** arXiv:2412.06559. https://arxiv.org/abs/2412.06559 13. Sprague, Z., Bostrom, K., Chaudhuri, S., & Durrett, G. (2022). **Natural Language Deduction with Incomplete Information.** EMNLP 2022. https://aclanthology.org/2022.emnlp-main.564/