Summary / 02. Tool selection and tool calling / Document
Paper 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# When to Act: A Tool-Calling Policy for a Weak-Model Coding Harness *Component 02 of 27 — Coding-Harness Program. This document specifies the tool-calling policy that sits above the Action-Calling Interface (component 01) and the primitives it defines. It is written to drive a reversible build; the executable specification follows in a companion document.* --- ## Abstract Tool calling in a coding harness is usually treated as a formatting problem: how to render a function call so that a language model can emit one. This paper argues the opposite. Tool calling is a sequential decision problem under uncertainty, in which each invocation is a choice about whether to act, which instrument to wield, and how to bind arguments to it — made in the face of imperfect information, measurable cost, and a world state that changes after every observation. The policy decomposes into three decisions: trigger (whether and when to invoke a tool), selection (which tool, with which arguments), and allocation (which computation lives outside the model) — together with a fourth condition, trust, governing whether the agent should believe a tool's result. For a weak model on CPU-only hardware under an agentless-first bias, the argument is that the trigger, retrieval, and trust-gate decisions should be deterministic and read off cheap signals — model confidence, retrieval scores, invariant checks — while only selection and argument binding require learned judgment. The visible toolset should be small and retrieved rather than enumerated, explicit no-op affordances (ask, stop) must be first-class, and dependent calls must be serialized. The resulting policy acts rarely, acts correctly, and verifies what it is told. ## 1. Introduction Component 01 of this program specified an Action-Calling Interface (ACI): the set of actions the harness may perform, their schemas, and their side-effect contracts. It said what the machine *can* do. This paper specifies the layer above it: the policy that decides what the machine *should* do, and when. Every ACI command is, from the policy's point of view, simply a tool — an instrument with a name, a schema, a cost, and a consequence. The policy is therefore not a catalog of capabilities but a controller over a catalog that already exists. The central claim is that tool calling is a sequential decision problem, not a function-call formatting problem. Three observations motivate this reframing. First, every decision is made under uncertainty: the agent does not know, at decision time, whether invoking a tool will help, whether the right tool is visible, or whether the result it receives can be trusted. Second, every decision has information value: a tool call is not merely an action but a query, an attempt to resolve some part of the uncertainty, and the value of that information must be weighed against its cost. Third, every decision changes state: an observation after a call is new evidence that alters what the agent should do next. A policy that treats each call as an independent, stateless formatting event forfeits exactly the structure — the closed loop of act and observe — that makes tool use work at all. These considerations are sharpened by the deployment constraints, which are deliberately harsh. The model is weak: a small language model with limited capacity for multi-step planning and long-horizon reasoning. The hardware is CPU-only, which caps both the throughput and the size of any model that can be served, and which makes token expenditure a real cost rather than an accounting detail. The operating bias is *agentless-first*: the harness should prefer not to act, should prefer the minimal action when it does act, and should avoid autonomous loops unless the marginal information justifies them. Under these constraints, the design question is not "how do we make the model call more tools," but "how do we make the model call the *right* tools, as rarely as possible, and verify what it is told." Two consequences of these constraints deserve emphasis, because they invert much of the received wisdom. First, a weak model's planning is a scarce resource: every token spent reasoning about *whether* to act is a token not spent on *what* to do, and the model has little capacity to spare. The policy should therefore make the cheap decisions cheaply — by deterministic signal, not by deliberation — and reserve the model's judgment for the one decision that genuinely needs it, namely choosing an action and binding its arguments. Second, the cost asymmetry between acting and abstaining is severe: a wrong tool call on real state can corrupt the working tree, and the remedy is expensive, whereas a missed call is cheap to recover from. An agentless-first bias is therefore not a stylistic preference but the rational response to an environment where the downside of over-acting dominates the downside of under-acting. A policy built for a strong model can afford to treat tool use as a default; a policy built for ours must treat it as a decision that has to be earned. The paper proceeds as follows. Section 2 reviews related work across five axes — trigger, selection, allocation, tools-as-tokens, and trust — with a philosophical anchor. Section 3 derives five design principles. Section 4 specifies a concrete policy flow with deterministic and learned nodes separated. Section 5 states the recursive self-improvement protocol. Section 6 gives the measurement plan. Section 7 concludes. ## 2. Related Work The literature divides cleanly into questions that map onto the policy's decisions. We review each group with its evidence, then state the implication for a weak-model, CPU-only, agentless-first harness. ### 2.1 The trigger: whether and when to act **Toolformer** [1] frames the trigger as a self-supervised filtering problem. Candidate API-call insertions are sampled, executed, and kept only if they reduce the perplexity of the following tokens beyond a threshold (L⁻_i − L⁺_i ≥ τ_f), with a "do-nothing" baseline as the comparison point. This formulation suppresses two symmetric failure modes at once: never calling, and always calling. Tool use is not hard-coded but emerges from the training signal, and the behavior appears around 775M parameters; at inference, greedy decoding with k=1 is reported to be well-calibrated. The lesson is that the *whether* decision can be trained and evaluated as a filter, and that the correct baseline is inaction, not a uniform tendency to call. For this program the relevance is structural rather than literal: we will not self-supervise the harness, but we adopt the same two-sided objective — penalize both the omitted necessary call and the spurious unnecessary one — and the same comparison point, namely doing nothing. **ReAct** [2] makes the trigger structural. Reasoning and acting are interleaved as Thought → Action → Observation, and the interleaving itself does the work: pure chain-of-thought reasoning hallucinates (a false-positive rate of 14% versus ReAct's 6%, with hallucination the single largest CoT failure mode at 56% versus 0% for ReAct), while pure acting flails (25.7 on HotpotQA). The interleaved policy reaches 71% on ALFWorld against 45% for a pure-acting baseline — a +34-point improvement over the imitation/RL baseline — and 40% on WebShop, a +10-point gain. The implication is not "use ReAct verbatim" but "reason first, then let a gate decide whether to act": thought and action are separated so that the decision to act is itself an explicit, inspectable step. For a weak model the warning is sharper still: ReAct's evidence is that *reasoning alone* is the hallucination source, so we should not ask our weaker model to make a larger share of its decisions by deliberation — we should shrink that share, and move the trigger out of the deliberative path entirely. **Self-RAG** [3] operationalizes the gate as learned reflection tokens — [Retrieve], [IsRel], [IsSup], [IsUse] — emitted by the model itself. A decode-time adaptive threshold δ tunes when retrieval is triggered, making "when to act" a continuous, tunable knob rather than a fixed policy. This is the direct ancestor of our τ in Section 4: the trigger is a threshold over a model-internal signal, exposed as a dial that can be tuned by measurement rather than fixed by hand. **FLARE** [4] reaches the same idea from a different direction: proactive retrieval is triggered on low token confidence during generation, so the signal for "I need help" is read off the model's own uncertainty rather than reasoned about abstractly. **ReTool** [5] moves the trigger into reinforcement learning, optimizing *when* and *how* to invoke tools against outcome rewards; the reported result is emergent, earlier, and more efficient tool use alongside code self-correction — evidence that the trigger is learnable from terminal reward alone. **When2Tool** [6] (recent preprint) is the strongest statement of the cheap-signal hypothesis: tool necessity is linearly decodable from pre-generation hidden states, and its Probe&Prefill method cuts calls by 48% at only 1.7% accuracy loss. Crucially, the same work notes that prompt-only control — telling the model "use tools sparingly" — suppresses necessary calls too, which is precisely the failure mode an agentless-first policy must avoid. ### 2.2 Selection: which tool, which arguments **ToolLLM** [7] confronts scale directly: a catalog of 16,464 APIs is unusable if enumerated in-context, so it retrieves a candidate subset with a neural retriever (Sentence-BERT, NDCG@5 of 84.9 against 18.5 for BM25) and only then selects. The retrieve-then-select structure is the durable contribution; the DFSDT tree search used for training is too expensive for a weak CPU model and is noted here only to be excluded. **ToolSandbox** [8] (venue unverified) supplies the cautionary evidence. It identifies three hard categories — state dependency, canonicalization, and insufficient information — in which the correct action is often *not to act at all*, i.e., to ask the user or decline. Distraction tools are punishing: adding irrelevant tools to the visible set (from zero to thirty-four) costs Claude-3-Sonnet roughly ten points. Dependent tools must be serialized because parallel calls race on shared state. And, contra the intuition that scaling fixes everything, stronger models hallucinate calls *more*, not less. **How Many Tools Should an LLM Agent See** [9] (recent preprint) confirms the direction: function-calling accuracy degrades as the catalog grows and as semantically similar tools are added, measured on a chance-corrected basis. **Gorilla** [10] offers the complementary remedy: retriever-aware fine-tuning that teaches the model to use retrieved API documentation kills API hallucination and gave rise to the BFCL benchmark. The combined message is unambiguous for our setting: never enumerate a large catalog; retrieve a short list and select within it. Even though we cannot fine-tune a weak model as Gorilla does, the principle transfers: whatever selection capacity the model has should be spent *within* a shortlist, not squandered on discriminating among dozens of near-synonyms, because that discrimination is exactly where the accuracy collapses [9]. ### 2.3 Allocation: what computation lives outside the model **ToRA** [11] answers the *allocation* question — which computation should the model do, and which should it hand off. Natural language is reserved for planning and interpretation; arithmetic and symbolic manipulation are emitted as code and handed to an interpreter. The result is a 13–19% absolute gain across scales, with ToRA-7B reaching 44.6% on MATH against 22.7% for WizardMath-70B — a small model with good allocation outperforming a far larger one with poor allocation. The lineage (PAL, PoT) reinforces the point: the question is not only *when* to call a tool but *what should be a tool in the first place*. A weak model's scarce capacity should be spent on judgment, not on arithmetic it will get wrong. This is the single most important efficiency lever available to us: on CPU-only hardware, every operation the model performs by generation is orders of magnitude more expensive than the same operation performed by a deterministic primitive, so allocation is not merely a correctness question but a throughput question. The ACI (component 01) is the menu of things that *can* live outside the model; this policy is the rule for deciding what *should*. ### 2.4 Tools-as-tokens **ToolkenGPT** [12] represents each tool as a learned embedding — a "toolken" — in the output vocabulary, so that tool selection becomes next-token prediction over a fixed, frozen backbone. This matters for our setting because it decouples tool selection from prompt-space enumeration: adding tools becomes a matter of adding embeddings, and selection inherits the model's existing, well-trained decoding machinery rather than requiring new reasoning overhead. (The correct identifier is 2305.11554; an earlier citation to 2309.05753 was in error.) For a weak, frozen model the full toolken scheme may be out of reach, but its insight is not: selection should lean on the decoder's existing token-prediction competence rather than on new deliberative machinery, which is precisely why we route selection through ordinary next-token generation over a shortlist. ### 2.5 Trust and deference **When the Tool Decides** [13] (preprint) reports that agents defer to tool output blindly — accepting the result without exercising independent judgment — and, disturbingly, that deference *increases* with stronger backbones. The implication for a coding harness is direct: a linter, test, or type-checker result must not be trusted uncritically, because the failure mode (blind acceptance) grows as the model improves rather than shrinks. This is the empirical warrant for an explicit trust condition in the policy. ### 2.6 The philosophical anchor **The Extended Mind** [14] supplies the governing principle. The parity principle holds that a well-integrated external process is, functionally, part of the cognitive system; the coupling conditions specify what "well-integrated" means — reliable access, automatic endorsement, easy availability. A tool the agent can call reliably and automatically is an extension of its mind; a flaky tool is not. The policy's trust condition is, in this vocabulary, the enforcement of the coupling conditions: a tool that fails them should not be treated as cognition but as unverified input. ## 3. Design Principles Five principles follow from the review, each tied to evidence above. **P1 — Read the signal.** The trigger decision should be read off the model's own confidence signal — log-probabilities, token confidence, or hidden-state projections — not produced as a reasoned judgment. This is cheap, deterministic, and calibrated in the literature: Toolformer's perplexity filter [1], FLARE's low-token-confidence trigger [4], and When2Tool's linearly decodable necessity signal [6] all locate "I need help" in a measurable quantity rather than an opinion. A reasoned trigger costs tokens and invites the very hallucination ReAct shows reasoning produces under uncertainty [2]. **P2 — Small and retrieved.** The visible toolset should be a top-5 shortlist produced by a cheap retriever over tool descriptions (BM25 or a small embedding model), never an enumerated catalog. ToolLLM shows retrieval dominates enumeration at scale [7]; ToolSandbox shows distraction tools cost points [8]; How Many Tools shows accuracy falls as the catalog grows [9]. Five is the ceiling, not the target. The retrieval ranking itself is deterministic and cheap, which keeps the most failure-prone step — discrimination among many candidates — off the model entirely and hands it to a comparator whose errors are at least predictable. **P3 — Explicit no-op.** *Ask the user* and *stop* must be first-class affordances with the same standing as any action, because the correct answer is sometimes "no action yet." ToolSandbox's insufficient-information category is the evidence: the right move is frequently to decline [8]. An agentless-first bias requires that the model be able to say "I don't have enough to act" without it being a failure mode. **P4 — Serialize and guard.** No parallel dependent calls; a call that depends on a prior call's output must wait for that output, because parallel calls race on shared state [8]. Preconditions must be checked rejectively, minefield-style — the harness rejects an invalid call and tells the model *why*, rather than silently executing a plausible-but-wrong action. **P5 — Gate trust.** Tool output is evidence, not verdict. The policy must validate a tool's result against cheap invariants (exit codes, schemas, round-trips, sanitizer and linter passes) before it is incorporated into the agent's state, because agents defer blindly to tool output and deference grows with model strength [13]. The Extended Mind's coupling conditions [14] make this precise: a result the policy cannot verify is not cognition, it is unverified input. Note the subtlety the literature raises: the danger is not that tools are unreliable, but that *the model stops checking*. A deterministic gate is the only remedy that holds as the model improves, because a learned check would itself be subject to the same deference it is meant to prevent. ## 4. The Architecture The policy is a loop over a state that accumulates evidence. Each node is labeled deterministic (D) or learned (L). The design target is that the trigger, retrieval, and trust-gate nodes are D, and only selection and argument binding are L. ``` current state │ ├─(D) NEED EXTERNAL HELP? ── no ──► REASON (L) ──► updated state │ trigger: model logprob / token confidence │ vs threshold τ (Toolformer/FLARE/When2Tool signal) │ ├─ yes │ ├─(D) TOOL RETRIEVAL ──► candidate set ≤ 5 │ BM25/embedding over tool descriptions (ToolLLM) │ ├─(L) SELECTION ──► choose tool, bind arguments │ (P4: serialize — no parallel dependent calls) │ ├─(D) PRECONDITION CHECK ── reject ──► tell model why ──► back to SELECTION │ ├─(D) EXECUTE ──► OBSERVATION │ ├─(D) TRUST GATE ──► validate (exit code, schema, round-trip, lint/sanitize) │ fail ──► treat observation as unverified ──► flag, optionally retry │ └─► updated state ──► loop ``` The **trigger** (D) reads the model's confidence signal against a threshold τ. When the model's confidence in its own next-token continuation is low — or, in the hidden-state formulation, when necessity is decodable from pre-generation activations [6] — the loop branches to tool use; otherwise it reasons in-language. This is the single most important cheap decision, and it must not be reasoned about, because a reasoned trigger is a hallucination surface [2]. The threshold τ is itself a tunable parameter, exactly the knob Self-RAG exposes with δ [3]. The **retrieval** node (D) maps the current state to a candidate set of at most five tools, ranked by a cheap retriever over tool descriptions. Selection (L) then operates on five candidates, not a catalog: the model's job is to pick and bind arguments, the only genuinely learned step in the loop. **Precondition check** (D) is the minefield guard of P4: an invalid call is rejected with an explanation, and the model is given another selection attempt. After execution, the **trust gate** (D) validates the observation against deterministic invariants — exit code, expected schema, round-trip reversibility, and, where applicable, linter, sanitizer, or test results. A failing gate does not discard the observation outright; it marks it as unverified and routes it back into the loop as evidence-with-a-caveat, so the model can reason about a suspicious result rather than silently absorbing it [13]. The gate keeps the coupling honest: a tool whose output cannot be verified is not an extension of the mind [14]. The asymmetry is deliberate and cost-aware. The three deterministic nodes are the ones that prevent the expensive mistakes — acting when it should not, choosing from a bloated catalog, trusting a wrong answer. Selection and argument binding remain learned because that is where a language model's judgment is genuinely the best instrument available. ## 5. Recursive Self-Improvement Protocol This is component 02 of a 27-component program, and it follows the same discipline as component 01. The specification is written against the *complete* policy — the 100% target. The first implementation round is expected to land at roughly 80% of that target, with the gap concentrated in the least-critical affordances, and later rounds climb toward full coverage. Every change is A/B-gated: a candidate revision of the trigger threshold, the retrieval ranking, or the trust invariants is deployed behind a gate and accepted only if it improves the metrics of Section 6 on a fixed evaluation sample, never by fiat. The policy is reversible by construction — each node is a swappable module — so a regression is a rollback, not a rewrite. ## 6. Measurement & Evaluation A trigger policy and a selection policy are tested separately, because they fail differently. **Trigger.** Compare call precision and recall against two baselines: a "no tool" policy (never act) and an "always tool" policy (act at every step). The target is high precision with bounded recall — the agentless-first bias tolerates a missed call more than a wasted or harmful one, but the When2Tool warning [6] means prompt-only suppression is not a substitute for a measured trigger. Report the token and cost delta against both baselines. **Selection.** Measure tool-selection accuracy on a fixed task sample, with the catalog size varied (top-1 through top-5, plus full enumeration as a stress test). The How Many Tools and ToolSandbox findings [8][9] predict accuracy should hold at small k and collapse under distraction tools and semantically similar decoys; the evaluation must confirm this on our own weak model rather than assume it transfers. **Trust.** Run minefield probes in the ToolSandbox style: inject incorrect tool output, a plausible-but-wrong linter pass, and a race between two dependent calls, and measure how often the harness's trust gate correctly refuses to absorb the bad result. The relevant metric is not whether the model is fooled, but whether the deterministic gate catches what the model would have accepted. A necessary honesty constraint: every number above was measured on models far stronger than ours. A weak model on CPU hardware will not inherit strong-model accuracy, and the cheap-signal hypothesis — that a weak model's confidence is *more* informative than a strong one's, not less — is exactly what the evaluation must verify rather than assume. The numbers from the literature are directions, not guarantees. The evaluation therefore carries a meta-hypothesis to be tested alongside the policy: that the relative ordering of the design choices (retrieve over enumerate, serialize over parallelize, gate over defer) survives the transfer to a weak model even though the absolute numbers do not. If the ordering holds, the policy is sound and merely needs threshold tuning; if it inverts — if, say, a weak model's selection is so fragile that even a five-candidate shortlist overwhelms it — the policy must shrink the shortlist further or move more decisions into the deterministic nodes. The architecture of Section 4 is deliberately built so that this kind of correction is a parameter change, not a redesign. ## 7. Conclusion Tool calling is a decision, not a format. The policy that governs it must decide, under uncertainty, whether to act, which instrument to use, and what computation to hand off — and it must decide whether to believe what it is handed back. For a weak model on CPU-only hardware under an agentless-first bias, the right policy is economical and asymmetric: read the *whether* off the model's own confidence signal, retrieve a short list rather than enumerate a catalog, make "ask" and "stop" first-class, serialize dependent calls, and gate trust in tool output with deterministic invariants. Only selection and argument binding are left to learned judgment, because those are the only places judgment earns its keep. This is a policy that acts rarely, acts correctly, and verifies what it is told — a controller for a machine whose scarce resource is not capability but attention. --- ## References 1. Schick, T., et al. *Toolformer: Language Models Can Teach Themselves to Use Tools.* arXiv:2302.04761. NeurIPS 2023. 2. Yao, S., et al. *ReAct: Synergizing Reasoning and Acting in Language Models.* arXiv:2210.03629. ICLR 2023. 3. Asai, A., et al. *Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection.* arXiv:2310.11511. ICLR 2024. 4. Jiang, Z., et al. *Active Retrieval Augmented Generation (FLARE).* arXiv:2305.06983. EMNLP 2023. 5. *ReTool: Reinforcement Learning for Strategic Tool Use in LLMs.* arXiv:2504.11536. April 2025. (Authors not verified.) 6. Chung-En Sun, Linbo Liu, Ge Yan, Zimo Wang, and Tsui-Wei Weng. *LLM Agents Already Know When to Call Tools — Even Without Reasoning.* arXiv:2605.09252. May 2026. **Recent preprint** — treat findings as provisional. 7. Qin, Y., et al. *ToolLLM: Facilitating Large Language Models to Master 16,000+ Real-World APIs.* arXiv:2307.16789. ICLR 2024. 8. *ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use.* arXiv:2408.04682. Venue unverified. 9. *How Many Tools Should an LLM Agent See.* arXiv:2605.24660. May 2026. **Recent preprint** — treat findings as provisional. 10. Patil, S., et al. *Gorilla: Large Language Model Connected with Massive APIs.* arXiv:2305.15334. 2023. 11. Gou, Z., et al. *ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving.* arXiv:2309.17452. ICLR 2024. 12. Hao, S., et al. *ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddings.* arXiv:2305.11554. NeurIPS 2023. *(Corrected identifier; an earlier citation to 2309.05753 was erroneous.)* 13. *When the Tool Decides: LLM Deference to Tool Output.* arXiv:2606.14476. June 2026. **Preprint** — treat findings as provisional. 14. Clark, A., & Chalmers, D. *The Extended Mind.* Analysis 58(1):10–23, 1998.