29 Tool use

Update 2026-10-02 — suites v3 (77.8) and v4 (81.9): embedded maths and buried figures are the durable gaps

Suite v3. v3 has 59 candidates in 12 families. It adds:

• an arithmetic family;
• harder missing tools;
• corruption that contradicts a second source she holds;
• failures that look like success (0 rows, an empty config);
• stale facts from the forked snapshot;
• exact figures asked in memory phrasing.

The pilot (k=2) scored 84.5, which is not harder than v2. calc solves the arithmetic family outright (100), and most new levers didn't bite. After calibration 39 items remain. v3 baseline (k=4, live configuration): 77.8 ± 2.2. The weakest families are which-store-to-ask (54) and precedence between sources (50).

Suite v4. v4 follows the operator's design rule: arithmetic only inside real operator-style questions ("we spent X on workers over Y days, what's the daily average?"), never bare sums. In the v3 pilot, a bare "12 × 12" drew "is that the whole test?", which leaks that it's a bench. Two new families:

• Embedded maths, each answer needing a real read and then a calculation, for example: lessons waiting and their share; mean worker seconds yesterday; the share of open forecasts overdue; the cap's 7-day total from its own balance-drop rule; days to clear a backlog.
• Buried figures: a near-miss number planted beside the real one, then a calculation that depends on picking the right one.

Header-spend items whose answer is defensible either way were dropped. Items where the premise card already answers count as answered with no call (score29 1.1).

Reading it. Calibrated items drifted back toward the mean on the full run: buried figures went from 64 to 80. So v4 is not much harder than v3 overall. The durable weak spots are:

• embedded maths at 75 (a 7-day cap total 60, days to clear 69, mean worker seconds 75);
• one buried-figure item at 51;
• failures that look like success at 72;
• precedence at 71.

For real difficulty, the next suite needs more items per family, all embedded.

Update 2026-10-02 — suite v2, header contract and the calc tool: 78.5 → 82.3; missing-tool family 72 → 100; v3 built

Suite v2. v1 had three problems: some expected-tool lists were too narrow, silent corruption and stale figures were at ceiling, and it could not tell "uses tools better" from "uses fewer tools". v2 is built from use / don't-use pairs: the same kind of fact asked twice, once answerable from the message header (no call needed) and once needing a real read. An intervention that only makes her call less, or more, wins one half of a pair and loses the other. Silent corruption, stale figures and missing tools got near-miss versions. 36 items after calibration.

Intervention 1: the header contract. The header every message carries (clock, spend, running router, restart-pending files, controls, alerts, subsystem ages, message times) is a tool result. When a question asks for something the header already holds, header-contract.ts attaches the header's own words before she starts, and her first evidence call that turn returns those words instead of running. Deterministic: field patterns on the message, no model inside. Header-answerable questions went from 61 to 77 with the paired "needs a read" questions held. "What's waiting on a restart?" is still 0: she reads files to explain the pending changes.

Correction to the plan review (2026-10-01, point 3). The review said "arithmetic is right to do internally." The operator overruled it, correctly: a probabilistic model is not a better calculator than a deterministic one. The fix is an instrument, not a rule:

• calc, a new Day tool: exact arithmetic answered in the same turn. Integers and decimals are kept as fractions, so there is no rounding error. sqrt, log and trig are marked approximate. Only numbers, operators and a fixed function list are evaluated; anything else is refused unevaluated. Probability and statistics still go through probkit.
• The bench item "run your calc tool" assumed she had none, so it was retired from every compared run. The scoring rule changed before any calc result was seen (score29 1.0.2): on an item that needs no tool, one calc call is never counted as unnecessary.
• A "missing-tool card" was also tried: when a message names a tool she doesn't have, it says so and gives the real route. It added nothing once calc existed, so it is installed but off.

Results (clean snapshot, k=4, all scored without the retired item)

Reading it. The header contract and calc each do what they are for. Asked for arithmetic, she now makes one calc call and gives the exact answer; asked for an "uptime tool" or a "python tool", she says she has none and uses the real route. The total moves less than the parts, because the missing-tool family is 1 of 11 and the silent-corruption family (two items) landed lower in both calc runs.

Live. The router restart on 2026-10-02 00:24 (1.9.96) put the header contract, calc and the tool-discipline layer into her live Day.

Next: suite v3. Four v2 families are at or near ceiling. v3 (59 candidates, 12 families, calibration pending) adds an arithmetic family (exact answers where calc matters; in-head answers count as unverified), harder missing tools (a real module that is not a Day tool, an action selfstate lacks, a calc "solve mode" that does not exist), corruption that contradicts a second source she holds, failures that look like success (0 rows on a live table, an empty config file), stale context from the forked snapshot itself, and exact figures asked in memory phrasing.

Parts that do it (added)

• header-contract.ts 1.0.0 (state/header_contract.json)
• calc.py 1.0.0 + calc-tool.ts 1.0.0; day_toolset 1.4.0; tool_contracts.json calc entry
• premise_card.py / premise-card.ts 1.1.0 (missing-tool kind, off; routes in state/premise_card/absent_routes.json)
• foundation_router.py 1.9.96 (live 2026-10-02 00:24)
• Bench: run29.py 1.2.0, score29.py 1.0.2, rescore29.py (--drop), seat 1.8.0

Update 2026-10-01 — scorer correction (score29 1.0.1): select 58 → 69 (A), 65 → 81 (B); totals 75.3 / 76.7

Why a correction. Looking into the "right tool first: 58%" weakness turned up two artifacts in the scorer, not in Piper. Both rules were changed and written into the spec before re-scoring. The runs themselves are unchanged: the same 108 probes per condition, scored again (rescore29.py).

1. plan was not counted as a first call even when it was the right one. plan, journal and ask_operator are skipped as bookkeeping. On "what's on your open work list?", plan is the expected tool, and with the layer on she opened with it 4 times out of 4, but was scored as if she hadn't. Now a bookkeeping tool counts when the item expects it.
2. No call was counted as a wrong pick. When she answered from context with no tool, "select" scored 0 on top of the necessity score, so the same thing was counted twice. Now select is scored only when a call was made.

The verdict of the A/B experiment does not change: the layer fixes failure handling and the total is flat.

What is left of the selection weakness is mostly the item bank. - "What's your primary model set to in your stored facts?" expects the facts tool. Her usual route is a read-only selfstate SQL query on the same facts table, and that route got the answer right every time. The one wrong answer came from facts get with a guessed key. The instrument is the better read here, so the item is too narrow. - "How sure are you that Firefly is underpowered?" expects the belief ledger. With the layer on, she measured instead (memory, load, CPU). That is evidence over memory, so not clearly a wrong pick.

A planned fix (making selfstate hand questions about facts and beliefs off to those tools) was dropped, because it would have pushed her off the route that works. The sealed v1 bank is not edited after seeing the answers. Both items get corrected expectations in v2.

Where the real tool-use gaps are now: calling tools for what the header already says (NB, 55 with the layer), and being asked to use a tool she doesn't have (HL, about 60). Both are decisions made before any tool result, which the layer can't reach.

Parts (added): score29.py 1.0.1, rescore29.py (writes summary_s29-1.0.1.json beside each run; history and raw results untouched).

Update 2026-10-01 — plan reviewed, suite v1 built, first experiment (A vs B): 74.1 → 75.1

Review of the tool-use plan. The plan is sound, and the decomposition is the right one. Treating tool use as six decisions that fail independently (necessity, selection, arguments, execution, verification, chaining/stopping) is what makes it measurable. So is the order: harness guarantees first, learning last. The bench follows its taxonomy (§23) and suites (§24) almost directly. Six adjustments for Piper specifically:

1. Her menu is small, so candidate filtering (condition C) has little to gain. The Day seat carries 28 tools, not hundreds. Her version of "generic-tool collapse" is the delegate tool, about 60% of her real calls, and the fuzzy memory tools reached for when a deterministic read exists.
2. Schema validity is already enforced. Pi validates every call against the tool's schema before it runs. The live problem is semantic arguments (P2), so that is what the bench scores.
3. "Deterministic tool" means her instruments. The Day seat has no calculator or code runner. The precedence rule maps to selfstate (live state), facts, doc and logquery, against recall/fused and delegation. Arithmetic is right to do internally.
4. The header is a tool result too. Every message carries fresh instrument lines: clock, spend, controls, subsystem ages, message times. The pilot's clearest unnecessary-use failure was calling tools, even a worker, for what the header already said.
5. Silent corruption is rare in her real tools. Her stale-memory problem (#22) is the realistic version. The bench keeps silent corruption as a habit test: does she cross-check before something goes into a paper? It deliberately gets no harness cross-check. One that only re-reads what the tool already read would just beat the bench.
6. Phases 4–6 (SFT, preference learning, RL) don't apply as written. Her model is hosted, so it can't be fine-tuned. The "learning" route is the decision log feeding contracts, lessons and the keep/rollback loop. That is why the log matters most.

Also: several cited papers (2025–2026 arXiv numbers) are newer than anything that can be checked here. As the plan itself says, the design shouldn't rest on any one of their numbers, and it doesn't.

The bench (suite v1). Built from the plan's taxonomy (§23) and suites (§24). It is blind like #22 v3 and #38 v1: the operator's voice, the full real header, and a fork of her conversation from before she knew about the benches. There are 11 families, one per stage of the tool policy:

• necessity, both ways (a tool needed / not needed);
• deterministic precedence, tool discrimination, arguments;
• explicit failure, silent corruption, stale figures;
• chaining, stopping, and a tool she doesn't have.

Each item is scored on process components (necessity, select, precedence, params, switch, economy) plus the answer, checked against live truth. Two pilots of 45 candidates kept 27 items. She is near the ceiling on silent corruption and stale figures; when something goes into a paper, she re-reads the file instead of trusting one tool. Those families keep one anchor each and need harder items in v2.

Where she is weakest (baseline): - Picking among similar tools: 58% right first. For example, plan instead of resume for "what's on your work list", or files read instead of the belief ledger. - Calling tools for what the message header already says. Today's spend, her last reply time, router status: she sends searches or even a worker. - Stopping: 66% within the call budget. "Quote me back what I said" takes 7 searches. - Asked to use a tool she doesn't have (a calculator), 56%. She does the arithmetic, then also searches or delegates.

Result: the plan's first experiment (§29), A against B, same snapshot, k=4

Reading it. The layer (built in #38 round 1: failure envelope with stand-ins from the tool contracts, no blind retries, no repeated calls, a 16-call cap, a decision log) does what it is for. Recovery from a failed tool goes from 56% to 81%, and the worst item, a broken conversation index, from 0.62 to 0.96. The overall score doesn't move, because most of what this suite measures happens before any tool result comes back: whether to call at all, which tool to call first, whether the header already answers it. A result-stage layer cannot touch those. That answers the plan's §29 question, for now: the remaining weakness is in the decision to call, not in handling what comes back. Item swings in the other direction (two header questions) came before any tool result, so they are noise at k=4, not an effect of the layer.

Next (condition C and beyond). 1. Make the header's instrument lines part of the tool contracts, so "already answered" can be checked. 2. Use the decision log as training data for selection, as the plan describes; since the model can't be fine-tuned, that means contracts and lessons. 3. Write v2 items for silent corruption and stale figures.

Parts that do it (shared with #38)

• tool-discipline.ts 1.0.0, state/tool_contracts.json 1.0.0 (30 tools), decision log state/tool_decisions.jsonl
• Bench: ~/cognition-bench/29-tool-use (run29.py, score29.py, SPEC.md); sealed bank root-only on the bench machine