Summary / 29. Tool use / Document
2026-10-01
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Tool-Use Improvement Plan ## Distilled Research and Implementation Direction **Date:** 2026-10-01 **Focus:** Improving specialized tool use in an agentic AI system. --- # Executive conclusion The research responses converge on one core point: > **Tool use is not a single capability. It is a sequence of distinct cognitive decisions that can fail independently.** For Piper, the most useful decomposition is: 1. **Necessity** — Should I use a tool at all? 2. **Selection** — Which tool should I use? 3. **Parameterization** — What arguments should I pass? 4. **Execution / observation handling** — Did the tool actually work, and what did it return? 5. **Verification / recovery** — Should I trust the result, retry, switch, or abstain? 6. **Chaining / termination** — What tool comes next, and when am I done? This framing is stronger than treating tool use as "can the model call APIs?" It also maps directly onto Piper's observed failure modes: - tool underuse; - wrong-tool selection; - overuse of a generic tool; - incorrect arguments; - ignoring tool failure; - trusting stale outputs; - failing to verify results; - failing to chain tools appropriately; - using the LLM when a deterministic tool exists; - continuing after the problem is already solved. The implementation lesson is equally clear: > **Put deterministic guarantees in the harness, train or learn judgement where judgement is actually required, and evaluate each failure mode separately.** Prompting alone is not enough. --- # 1. Tool use as an epistemic decision A useful unifying idea from the literature is that **internal reasoning and external tools are both ways of reducing uncertainty**. Internal reasoning: - recombines information already available; - costs tokens and latency; - can make inferential mistakes; - cannot create unavailable facts. External tools: - acquire new information; - perform reliable computation; - access current state; - execute actions; - introduce their own costs and failure modes. The key question becomes: > **Can the remaining uncertainty be resolved internally, or does the agent need external evidence or deterministic computation?** This provides a principled basis for deciding when tools are necessary. ### Tool underuse The model reasons internally when it actually needs: - new information; - current information; - deterministic computation; - execution; - independent verification. ### Tool overuse The model calls an external tool even though the answer is already reliably available internally. ### Wrong tool The agent correctly determines that external help is needed, but chooses an instrument that does not resolve the relevant uncertainty. ### Bad arguments The correct instrument is chosen, but the request sent to it is malformed or incomplete. ### Ignoring tool failure The agent confuses "a call occurred" with "valid evidence was returned." ### Stale output The agent treats old evidence as if it still described current state. ### Failure to verify The agent treats a plausible result as authoritative without checking whether it satisfies the task. ### Bad chaining The agent does not recognize that several sequential uncertainty-reduction steps are required. ### Continuing after success The uncertainty has already been resolved, but the agent keeps acting. --- # 2. The six-stage tool-use pipeline ## Stage 1 — Necessity Question: > **Do I need a tool?** Possible actions: ```text ANSWER_DIRECTLY USE_TOOL ASK_FOR_INFORMATION ABSTAIN ESCALATE ``` This stage addresses: - tool underuse; - unnecessary tool use; - use of LLM reasoning where a deterministic tool should be preferred. The important research result is that tool necessity should be explicitly trained and evaluated in both directions. A dataset containing only cases where tools are needed teaches: > always call something. A dataset containing only cheap direct-answer examples teaches: > skip tools. So necessity training must contain: - tool-required cases; - direct-answer cases; - unavailable-tool cases; - ambiguous cases requiring clarification; - impossible cases requiring abstention. --- # 3. Necessity should not be solved by prompting alone Several supplied research summaries make the same point: > Models may internally contain useful signals about whether a tool is needed while failing to convert that signal into the correct action. That mirrors Piper's metacognition problem. The model may "know" that a calculation, fresh state query, or retrieval is warranted, yet still answer directly. Therefore the practical architecture should expose necessity as an explicit decision. Example: ```text NEED_EXTERNAL_INFORMATION? NEED_DETERMINISTIC_COMPUTATION? NEED_CURRENT_STATE? NEED_EXECUTION? NEED_INDEPENDENT_VERIFICATION? ``` If none apply, direct reasoning may be appropriate. If one applies, the candidate tool set should be constructed accordingly. --- # 4. Deterministic-tool precedence One of Piper's highest-value rules should be: > **When a deterministic, appropriate, healthy tool exists, it should normally take precedence over probabilistic language-model calculation.** Examples: - arithmetic → calculator / Python; - constraint satisfaction → solver; - current process state → system probe; - current database count → query; - code dependency → code graph; - exact syntax / compile state → compiler/linter; - current web state → retrieval/search; - deterministic test → test harness. This should not necessarily be learned from scratch. It can begin as a hard routing rule. The model still decides **what problem needs solving**. The harness decides that some solution methods dominate others for reliability. --- # 5. Stage 2 — Tool selection Question: > **Which tool best resolves the need?** Tool selection becomes harder as the catalog grows. The research responses agree on several practical lessons. ## Keep the active tool set small Do not place hundreds of schemas into every context. Instead: ```text task ↓ tool-family retrieval ↓ small candidate set ↓ discrimination / selection ``` Candidate filtering can use: - semantic retrieval; - categories; - preconditions/effects; - capability registry; - health; - current goal; - previous outputs; - causal relevance. This reduces: - distractor tools; - wrong-tool selection; - hallucinated tool names; - generic-tool dominance. ## Tool descriptions should include negative boundaries Tool descriptions should state: - what the tool does; - when to use it; - what inputs it requires; - what it returns; - **when not to use it**; - what similar tool should be used instead. This turns the catalog into a discriminative decision surface rather than a collection of positive advertisements. --- # 6. Separate selection from parameterization A recurring mistake in agent designs is treating: > "I selected the right tool" as equivalent to: > "I produced a valid tool call." They are different abilities. For every call, log separately: ```text tool_selected_correctly arguments_schema_valid arguments_semantically_correct execution_successful result_used_correctly ``` This makes the failure visible. Without this separation, a correct tool with bad parameters can be misdiagnosed as a selection problem. --- # 7. Stage 3 — Parameterization Question: > **What exact request should be sent to the tool?** Common parameter failures from the supplied material include: - missing required arguments; - hallucinated parameter names; - redundant information; - wrong types; - mismatched specifications; - task deviation; - pulling a parameter from the wrong previous tool result. ## Hard schema enforcement This is an area where deterministic controls are superior to model training. Before execution: ```text parse → validate schema → validate types → validate required fields → validate enums/ranges → only then execute ``` If the call cannot pass validation: ```text DO NOT EXECUTE ``` Return the validation error to the controller. ## Semantic validation Schema-valid can still be wrong. Example: ```text tool: weather(location, date) ``` The model supplies: ```text location = Toronto date = user's birthday ``` The types are valid; the semantics are wrong. For high-value tools, parameter validation should include task-grounded checks. --- # 8. Tool contracts Each tool should expose a contract containing: ```text name purpose positive triggers negative triggers preconditions input schema output schema failure states side effects freshness semantics cost latency determinism verification method fallbacks ``` That contract should be machine-readable. The LLM should not have to infer these facts repeatedly from prose descriptions. This contract becomes useful to: - orchestration; - selection; - parameter validation; - verification; - fallback; - benchmark generation. --- # 9. Stage 4 — Execution and observation handling Question: > **Did the tool actually work, and what kind of evidence did it produce?** Tool execution should return more than raw content. A useful normalized envelope: ```json { "status": "success", "tool": "example", "call_id": "...", "executed_at": "...", "source_timestamp": "...", "expires_at": "...", "version": "...", "result": {}, "warnings": [], "confidence": null, "error": null } ``` For failures: ```json { "status": "timeout", "retryable": true, "fallback_class": "search_provider" } ``` This prevents the LLM from having to infer whether a textual response means success. --- # 10. Tool success is not task success A central design rule: > **HTTP 200 / process exit 0 / valid JSON does not mean the user's goal has been achieved.** There are three separate levels: ```text CALL SUCCESS ↓ RESULT VALID ↓ TASK SATISFIED ``` All three should be tracked. Example: A database query succeeds. But it queried the wrong date range. The tool succeeded. The result may even be valid. The task is still wrong. This distinction belongs in Piper's benchmark and logs. --- # 11. Stage 5 — Verification Question: > **Should I trust and use this result?** The strongest common lesson from the supplied research is: > **Verification needs an external signal.** Having the same model reread its own answer is weak verification. Useful external signals include: - deterministic tests; - a second query; - checksums; - schema checks; - independent source; - execution; - compiler; - solver; - comparison against current state; - different model family; - hidden expected result. A result requiring strong verification should not be certified by the same mechanism that produced it. --- # 12. Silent corruption is more dangerous than explicit failure An error such as: ```text HTTP 500 ``` is easy. A tool returning: ```text plausible but wrong result ``` is much harder. Therefore tool robustness testing must include both: ## Explicit faults - timeout; - exception; - unavailable; - permission denied; - malformed output. ## Silent faults - stale value; - wrong value; - partial value; - swapped field; - plausible fabrication; - outdated world state. Silent corruption should be a first-class benchmark condition. --- # 13. Freshness and temporal validity Tool results should carry: ```text observed_at source_time expires_at / TTL ``` A cached tool output should never silently masquerade as present state. A useful rule: ```text if current_time > expires_at: result cannot support a present-state claim ``` For volatile facts, the system should re-query. For stable facts, reuse is appropriate. This connects directly to Piper's metacognition work on provenance and staleness. --- # 14. Result-Ignore detection One particularly relevant failure for Piper is: > She calls the right tool, receives the correct result, then answers from remembered prior knowledge anyway. This deserves its own diagnostic. For each call: ```text tool observation → extracted load-bearing facts → final response ``` Check whether the final response: - used the tool result; - contradicted it; - ignored it; - fabricated something else. This is different from tool selection. The tool may have been perfect. The integration failed. --- # 15. Recovery policy: retry, switch, abstain When a tool fails, Piper should not improvise randomly. Normalize recovery into three choices. ## Retry Appropriate when: - error is transient; - call was malformed but repairable; - timeout is likely temporary; - retry cost is reasonable. ## Switch Appropriate when: - an equivalent source/tool exists; - the first provider is down; - another instrument can satisfy the same contract. ## Abstain / escalate Appropriate when: - all viable paths are blocked; - the requested action cannot be verified safely; - required information does not exist; - the tool capability is unavailable. A fallback map can explicitly encode tool equivalence classes. Example: ```text weather_provider_A ↔ weather_provider_B local_memory ↔ authoritative_file ? external_search ``` The fallback itself must respect source authority. --- # 16. Recovery and verification are different A subtle but important distinction: ### Recovery asks: > Can I obtain a result another way? ### Verification asks: > Is the result I have actually trustworthy? Switching providers fixes outages. It does not necessarily fix silent corruption. Therefore: ```text fallback ≠ verification ``` Piper needs both. --- # 17. Stage 6 — Tool chaining Question: > **What tool call should happen next?** Single-call tool performance substantially overstates true agentic tool ability. Long chains introduce: - parameter propagation; - dependencies; - stale intermediate state; - order constraints; - branching; - retries; - cost compounding; - failure compounding. The key implementation principle is: > **Represent dependencies explicitly.** Instead of: ```text LLM remembers that output A should become input B ``` prefer: ```text A.output.customer_id ↓ B.input.customer_id ``` as structured state. --- # 18. Tool chains as executable graphs A useful representation: ```text NODE: tool preconditions inputs expected outputs success predicate failure routes EDGE: data dependency control dependency ``` This turns a tool chain into a DAG or state machine rather than a sequence embedded only in conversation. Benefits: - easier validation; - easier replay; - easier fault injection; - easier stopping; - easier verification; - reduced context dependence. For deterministic workflows, programmatic orchestration may be preferable to repeatedly asking the LLM what to do next. --- # 19. Search instead of greedy chaining When several plausible tool paths exist, a single greedy trajectory can lock onto a poor path. Useful alternatives include: - beam search; - depth-first search with backtracking; - MCTS; - multiple candidate plans; - reverse planning from required final evidence. However, search should only be used where branching justifies its cost. For common sequences, package the sequence into a reusable macro or option. --- # 20. Stopping Question: > **Has the tool use already accomplished the goal?** This is part of tool discipline. A good stopping system should recognize: - deterministic verifier passed; - required state achieved; - all requested fields obtained; - uncertainty reduced sufficiently; - further calls have negative expected value; - repeated calls are no longer producing new evidence. Useful controls: - hard token/time/cost cap; - success predicates; - maximum consecutive no-progress steps; - marginal-value threshold; - explicit final action. The harness should own hard limits. A prompt saying "do not overuse tools" is not a limit. --- # 21. Training progression The research suggests a clear progression rather than jumping directly to RL. ## Phase 0 — instrumentation Log every call: ```text was_tool_needed? selected_tool available_tools arguments schema_valid execution_status result_freshness result_used result_verified fallback_taken next_action stopping_state cost latency final_outcome ``` Without these fields, tool learning collapses into end-task accuracy. ## Phase 1 — hard runtime controls Implement: - schema validation; - tool health; - TTL/freshness; - deterministic-tool precedence; - explicit error states; - structured result envelopes; - termination predicates; - cost/time limits. ## Phase 2 — improve tool descriptions and retrieval - positive use cases; - negative use cases; - preconditions; - effects; - active candidate filtering; - small visible tool menu. ## Phase 3 — build diagnostic benchmarks Measure each failure class independently. ## Phase 4 — replay and synthetic training data Generate examples of: - correct calls; - unnecessary calls; - missed calls; - wrong tools; - wrong parameters; - silent corruption; - unavailable tools; - changing state; - correct abstention; - correct stopping. ## Phase 5 — supervised training / preference learning Train on good and bad trajectories. Important: > Include **negative tool-use examples**, not only successful tool calls. ## Phase 6 — reward-driven optimization Only after the evaluation stack is reliable. Possible reward decomposition: ```text R = task_success + necessity_correct + tool_selection_correct + parameter_correct + result_integration_correct + verification_correct + recovery_correct + termination_correct - tool_cost - latency_cost - redundant_calls ``` Avoid optimizing only final answer success. --- # 22. Why coarse RL rewards are dangerous Suppose reward is: ```text +1 if final answer is correct ``` Then several bad policies may survive: ### Always call tools Expensive but occasionally robust. ### Never call tools Cheap and often good enough on easy cases. ### Delegate everything Pushes difficult decisions into another model. ### Use ten tools One of them eventually produces the answer. The reward must distinguish **how** success was achieved. Tool discipline requires process-level rewards. --- # 23. Diagnostic failure taxonomy for Piper A compact Piper-specific taxonomy: ## N1 — Tool-Skip A tool was required, but none was executed. ## N2 — Unnecessary Tool Use A tool was called even though reliable direct reasoning was sufficient. ## S1 — Wrong Tool A real but inappropriate tool was chosen. ## S2 — Generic-Tool Collapse A general-purpose tool was selected despite a clearly superior specialist. ## S3 — Tool Hallucination A nonexistent tool or unsupported capability was invoked. ## P1 — Invalid Parameters Schema/type/value failure. ## P2 — Semantically Wrong Parameters Call is valid but does not represent the task. ## E1 — Explicit Failure Ignored Tool reports failure and Piper proceeds as if it succeeded. ## E2 — Silent Corruption Trusted Plausible but wrong output is accepted. ## E3 — Stale Observation Used Expired state is treated as current. ## I1 — Result Ignore Correct tool result is obtained but final reasoning uses another value. ## I2 — Output Fabrication Piper claims a tool returned something it did not. ## V1 — Verification Missing A high-risk or uncertain result is accepted without required validation. ## C1 — Chain Dependency Failure Wrong ordering or wrong propagation between calls. ## C2 — Premature Stop Tool chain stops before the goal is satisfied. ## C3 — Post-Success Action Piper continues acting after success is established. This taxonomy should appear directly in tool-use logs and benchmarks. --- # 24. Benchmark design A strong tool-use benchmark should contain paired positive and negative scenarios. ## Suite A — necessity Pairs: ```text same surface task A: requires external tool B: answerable internally ``` Score whether Piper distinguishes them. ## Suite B — deterministic precedence Give problems where: - LLM can attempt the answer; - deterministic tool exists; - tool produces exact truth. Measure routing. ## Suite C — tool discrimination Several similar tools are available. Only one satisfies the true preconditions. Include: - tempting generic tool; - distractor tool; - unavailable tool. ## Suite D — parameterization Generate exact-schema tasks with known arguments. Score: - name; - required fields; - values; - types; - semantic correctness. ## Suite E — explicit failure Inject: - timeout; - permission error; - unavailable; - malformed response. Score retry/switch/abstain. ## Suite F — silent corruption Return plausible but incorrect data. A second independent verifier is available. Measure whether Piper detects the fault. ## Suite G — stale output Provide: - cached result; - timestamp; - changed current state. Measure freshness handling. ## Suite H — result integration Tool returns a value contradicting Piper's prior belief. Score whether the final answer follows authoritative evidence. ## Suite I — chaining Tasks require 2–5 dependent calls. Score: - ordering; - data propagation; - preconditions; - intermediate validation. ## Suite J — stopping Insert a clear success predicate at step N. Score all post-success actions as unnecessary. --- # 25. Metrics Do not use only task pass rate. Track: ### Necessity precision Of tool calls made, how many were actually necessary? ### Necessity recall Of cases needing a tool, how often was one used? ### Tool-selection accuracy Was the best available tool selected? ### Generic-tool dominance Fraction of calls going to the dominant general tool when specialists were available. ### Parameter validity Schema-valid calls / total calls. ### Parameter semantic accuracy Arguments that correctly encode the task. ### Execution recognition Fraction of explicit tool failures correctly recognized. ### Silent-corruption recovery Fraction of corrupted results detected/recovered. ### Result-adherence Fraction of final answers consistent with authoritative tool output. ### Freshness compliance Fraction of volatile claims supported by unexpired evidence. ### Verification rate when required Not universal verification — required verification. ### Tool productivity ```text successful tasks / tool calls ``` ### Cost per successful task ### Recovery success rate ### Chain completion rate ### Post-success action count ### Premature-stop rate ### Tool hallucination rate ### Abstention correctness --- # 26. Tool use and orchestration should remain distinct There is overlap, but keeping the two components separate is useful. ## Orchestration asks: > **What cognitive process should run next?** It may choose: ```text REASON RETRIEVE USE_TOOL DELEGATE VERIFY STOP ``` ## Tool Use asks: > **Given that a tool should be used, can Piper use it correctly?** That includes: ```text necessity boundary at the tool level selection arguments execution observation handling verification recovery chaining termination ``` The orchestrator chooses the cognitive family. The tool-use layer executes the tool policy. The results then feed back into orchestration. --- # 27. Relationship to metacognition Metacognition supplies: - uncertainty; - evidence freshness; - source authority; - confidence; - knowledge-gap detection. Tool use consumes those signals. Example: ```text Metacognition: "I do not have current evidence for X." Orchestration: "Acquire external evidence." Tool Use: "Choose tool Y, call it with Z, validate the result." Metacognition: "The returned result conflicts with my prior belief." Tool Use: "Verify." Orchestration: "Stop once verified." ``` This is the beginning of a coherent cognitive stack rather than independent features. --- # 28. Suggested first implementation for Piper The first intervention should **not** be fine-tuning or RL. Build a deterministic **Tool Discipline Layer**. For every proposed tool call: ```text 1. Is a tool necessary? 2. Is a deterministic specialist available? 3. Is the chosen tool healthy? 4. Are its arguments valid? 5. Did execution succeed? 6. Is the result fresh? 7. Does the result satisfy the requested evidence need? 8. Is verification required? 9. Is another call necessary? 10. Has the success condition already been met? ``` Record all ten answers. Initially, Piper may still propose the call. The harness evaluates and constrains it. That creates the training data for later learning. --- # 29. A clean first experiment Compare three conditions. ## A — Current Piper Normal tool behavior. ## B — Tool Discipline Layer Same Piper, with: - necessity gate; - deterministic-tool precedence; - schema validation; - freshness checks; - explicit error envelopes; - required verification predicates; - stop predicates. ## C — Tool Discipline + candidate filtering Same as B, but Piper only sees a small task-relevant tool subset. Measure: - final task score; - tool calls; - cost; - latency; - Tool-Skip; - Unnecessary-Tool-Use; - wrong-tool rate; - invalid parameter rate; - result-ignore; - failure recovery; - post-success actions. If B/C strongly outperform A, the limitation is largely system design rather than model capability. --- # 30. Second experiment: negative examples Once the diagnostic suite exists, construct matched pairs: ```text USE TOOL DO NOT USE TOOL ``` and: ```text TRUST RESULT VERIFY RESULT ``` and: ```text RETRY SWITCH ABSTAIN ``` This provides the data needed to train **tool restraint**, not merely tool competence. That distinction is important. Many agents know how to use tools. Far fewer know when **not** to use them. --- # 31. Longer-term learning path Once instrumentation exists: ```text hard guards ↓ empirical failure profiles ↓ synthetic / replay training data ↓ SFT / DPO on tool decisions ↓ fine-grained reward optimization ↓ adaptive tool policy ``` A future learned policy could estimate: ```text P(tool needed) P(tool appropriate) P(call valid) P(result trustworthy) P(next call useful) P(task complete) ``` Those probabilities could become first-class inputs to Piper's orchestration controller. --- # 32. Research reading list The following sources recur across the supplied research and are the most directly relevant. ## Foundations ### ReAct — Synergizing Reasoning and Acting in Language Models Yao et al. Interleaves reasoning with actions and observations; foundational agent loop. https://arxiv.org/abs/2210.03629 ### Toolformer — Language Models Can Teach Themselves to Use Tools Schick et al. Self-supervised learning of which API to call, when, with what arguments, and how to use returned results. https://arxiv.org/abs/2302.04761 ### Gorilla — Large Language Model Connected with Massive APIs Patil et al. Important for retrieval-aware API use, tool documentation grounding, and reducing hallucinated calls. https://arxiv.org/abs/2305.15334 ### ToolLLM / ToolBench Qin et al. Large-scale real API training and evaluation; includes multi-tool planning and search over trajectories. https://arxiv.org/abs/2307.16789 --- ## Deterministic computation and necessity ### PAL — Program-aided Language Models Gao et al. Strong example of shifting deterministic computation from language-model reasoning to an execution runtime. https://arxiv.org/abs/2211.10435 ### Position: Agent Should Invoke External Tools ONLY When Epistemically Necessary Frames tool use as an epistemic decision: invoke tools when external evidence is necessary rather than automatically. https://arxiv.org/abs/2506.00886 ### SMART Focuses on reducing unnecessary external tool use while retaining or improving performance. https://arxiv.org/abs/2502.11435 ### When2Tool Benchmark/research direction centered on deciding whether a tool is actually necessary. https://arxiv.org/abs/2605.09252 ### When2Call Evaluates the decision among tool use, clarification, inability/abstention, and direct response. https://github.com/NVIDIA/When2Call --- ## Reward-driven tool discipline ### ToolRL — Reward is All Tool Learning Needs Fine-grained rewards for tool name, parameters, parameter values, and outcome. https://arxiv.org/abs/2504.13958 ### Acting Less is Reasoning More / OTC-PO Relevant to penalizing unnecessary tool calls while preserving correct answers. https://arxiv.org/abs/2504.14870 --- ## Verification and recovery ### CRITIC — Large Language Models Can Self-Correct with Tool-Interactive Critiquing Gou et al. External tool feedback enables much stronger verification than introspection alone. https://arxiv.org/abs/2305.11738 ### Reflexion Shinn et al. Stores feedback from failed attempts and uses it in subsequent trials. https://arxiv.org/abs/2303.11366 ### Large Language Models Cannot Self-Correct Reasoning Yet Huang et al. Useful counterpoint: unsupported introspective correction is unreliable. https://arxiv.org/abs/2310.01798 --- ## Tool-use benchmarks ### Berkeley Function Calling Leaderboard / BFCL Function selection, argument generation, relevance/irrelevance, and multi-turn function-calling evaluation. https://gorilla.cs.berkeley.edu/leaderboard.html ### ToolSandbox Apple Stateful, conversational tool-use evaluation with state dependencies and distractor tools. https://arxiv.org/abs/2408.04682 ### τ-bench Evaluates tool-agent-user interaction under explicit policies and multi-turn state. https://arxiv.org/abs/2406.12045 ### MAST — Why Do Multi-Agent LLM Systems Fail? Large failure taxonomy including verification, repetition, coordination, and termination failures. https://arxiv.org/abs/2503.13657 ### ToolFailBench Diagnostic taxonomy including tool skipping, result ignoring, output fabrication, and unnecessary tool use. https://arxiv.org/abs/2607.04686 ### ToolMaze Focuses on tool-side faults and recovery, including difficult implicit/silent failures. https://arxiv.org/abs/2606.05806 ### STALE Relevant to detecting and handling outdated assumptions or observations. https://arxiv.org/abs/2605.06527 ### STT-Arena Relevant to state-tracking and adapting to changed tool/world observations. https://arxiv.org/abs/2605.18548 ### ContractBench Tests whether agents preserve and obey requirements attached to tool outputs across later actions. https://arxiv.org/abs/2605.17281 --- # 33. What is strongest versus still provisional The supplied research includes a number of very recent 2025–2026 papers and benchmarks. The most mature ideas are: - separate tool-use stages; - deterministic schema validation; - retrieval/tool filtering; - negative examples for when not to call; - external verification; - explicit failure handling; - provenance/freshness; - stateful chaining; - termination outside the model. The more experimental ideas include: - highly specialized RL reward schemes; - hidden-state necessity probes; - some recent benchmark-specific numerical claims; - new self-healing / recovery architectures. Those newer results are useful as design leads, but the architecture should not depend on any single reported number being correct. --- # 34. Bottom line Piper should not simply become **better at calling tools**. She should become better at the entire policy: ```text SHOULD I USE A TOOL? ↓ WHICH TOOL? ↓ WITH WHAT ARGUMENTS? ↓ DID IT WORK? ↓ DO I TRUST THE RESULT? ↓ DO I NEED ANOTHER TOOL? ↓ AM I DONE? ``` The highest-value immediate work is not RL. It is: 1. **instrument every stage;** 2. **add deterministic runtime guards;** 3. **reduce the active tool menu;** 4. **make tool contracts discriminative;** 5. **test negative cases and silent failures;** 6. **build a diagnostic benchmark;** 7. **only then learn the policy from the resulting data.** That would turn Piper's tool use from an opaque language-model habit into a **measured, fault-aware, trainable cognitive capability**.