38 Orchestration

Update 2026-10-08 — read-before-delegating guard: the operator leans keep; observe mode is on live to count real cases

The guard (tool-discipline rule 8) is now in observe mode on Firefly, since 10-08 20:15 UTC. When she asks a worker to research or verify before reading anything herself in that turn, the guard logs a "would refuse" but blocks nothing. This is the live count the 10-02 intervention called for.

Readout on 10-13 (live38.py): if real delegate-first turns are frequent, switch the guard to enforce and mark the intervention kept. If they are rare, it has little to fix. The summary stays on her live configuration (75.4) until then.

Update 2026-10-08 — summary corrected: Latest is her live configuration, 75.4 (round 1), not 77.9 (guard, off live)

Audit fix (X1). The summary showed the last sealed row as "Latest". For #38 that was the read-before-delegating guard (77.9), which is off live. Firefly today has state/tool_discipline.json = {"mode": "enforce", "delegate_first": "off"}, and the guard has 0 log entries. The 77.9 row is now an anchor (kept in the history, not her current score), so the summary reads 70.1 → 75.4, the round 1 configuration she actually runs.

Still open: the operator decides keep or revert. The guard intervention has been "pending" since 10-02, and the observe-only live count it called for never ran. Live, delegate is 41% of her real tool calls (1,091 of 2,641 since 10-02, bench runs excluded). That share alone does not show how many are delegate-first (the case the guard targets).

Update 2026-10-02 — scorer fixed (score38 1.1): round 1 was 75.4, not 67.5; read-before-delegating guard: 77.9

Round 1 re-measured. Round 1 scored 67.5 against a 70.1 baseline, and the review traced most of the drop to the scorer: "used the specialist tool" needed a tool call, so a correct answer taken straight from #22's premise card with no call scored 0. The rule was fixed before re-running (score38 1.1): when the card or a header-contract line was attached in that probe and she did not delegate, the specialist component counts as met. The bench seat now reports, per probe, which of those were attached. The old session files were gone, so round 1's exact setup was re-run on the same clean snapshot rather than rescored.

Round 2: read before delegating. In round 1, five answers were a single research-mode delegate as her very first call, sending a worker to read files the Day seat reads itself (instruments "read, codesearch") to reconcile the card with a cron record. A new rule in the tool-discipline layer (1.1.0) refuses, once, a research or verify delegate issued before any Day read that turn, naming the Day tools that read it now (selfstate, read, logquery, doc). A second delegate call runs.

Reading it. With the scorer fixed, round 1 is a gain over baseline (70.1 → 75.4), not a loss. The guard's +2.5 is within noise: the delegate-first habit had mostly faded on its own between runs (8 → 3), so the guard fired once. It is installed and off in her live seat; the next step is to switch it to observe-only and count how often it would fire on real conversations before enforcing it.

Still weakest: dead components (50–58), where a stalled part is used without a health check, and injected failure (56–59).

Parts that do it (added)

• tool-discipline.ts 1.1.0, rule 8 (switch delegate_first in state/tool_discipline.json: off / observe / enforce)
• Bench: score38.py 1.1.0, run.py 1.2.0 (--env), seat 1.9.0 (attached)

Update 2026-10-01 — baseline 70.1, and round 1 (tool discipline): failures handled, net flat

Baseline. On the 14 sealed items, forked from a snapshot of her conversation taken before she knew she was being benchmarked: 70.1 ± 7.6. The weakest families were injected failure (45, a broken tool kept being retried) and dead components (48, stalled parts used without a health check).

What changed in round 1 (the orchestration plan's steps 1–2, deterministic, no prompt rule): the tool-discipline layer, a pi extension on every tool call in her seat.

• Decision log. Every call is logged: position in the turn, tool, an argument hash, how often that tool had already failed, the outcome, size and time.
• Failure envelope. A failing result gets one line added: how many times the tool has failed this turn, and the tools whose contract answers the same kind of question, with what each answers.
• No blind retries. An identical call that already failed is refused before it runs, and so is a tool that has failed twice this turn. The refusal carries the same list of stand-ins.
• Tool contracts. A machine-readable file (state/tool_contracts.json) gives, for each of 30 tools, the question it settles, whether it is deterministic and fresh, what not to use it for, and its stand-ins.
• Also in the layer, aimed at #29: an identical call already made this turn is not re-run; there is a cap of 16 evidence calls per turn; and when the ask is for an exact figure about a named component and the first call is a fuzzy memory tool, its result notes that selfstate reads the value now.

Result (same clean snapshot, k=4 each)

The tool-discipline layer works on its target. With everything else equal, it lifts the score by 8 points, and switching after a failure goes from 44% to 88%. In the worst baseline item, she used to call a locked document store seven times. Now she is told after the first failure which tools can stand in, and she moves.

Net, the round is flat (70.1 → 67.5, within noise), because of an interaction with #22's premise card. The card answers many "is X running?" questions outright, for example "24 runs in 24 hours, last at 15:02, all clean". In about half those probes she answered correctly from the card with no tool call, and this suite's scorer counts no call as "didn't use the instrument". That is a scoring artifact: the card is an instrument read. The other part is real. Having answered from the card, she sometimes sent a worker anyway to dig deeper ("running isn't the same as producing"), and needless delegation rose.

Next. 1. Score the card as an instrument read in the next suite version. The rule is fixed in advance, not tuned to these runs. 2. Target the follow-up delegation: a delegate call whose task is only a read that a direct tool can answer.

The dead-component family (48 → 57) is still the weakest.

Parts that do it (added)

• tool-discipline.ts 1.0.0 (mode in state/tool_discipline.json; decision log in state/tool_decisions.jsonl; health in state/tool_health.json)
• state/tool_contracts.json 1.0.0 (30 tools)
• foundation_router.py 1.9.94: loads it in her live Day at the next router restart (the bench seat loads it now)

Bench note: injected failures now arrive as the tool's error result, as a real failure would, instead of as a pre-execution block. The text she sees is unchanged.

Update 2026-09-30 — suite v1 built — calibration pilot results

This is a calibration pilot, not the baseline. 25 candidate scenarios were each run twice, blind, inside her live conversation. The baseline runs after a day or two of normal conversation, on the 14 sealed items.

All 25 candidates: 70.6 ± 2.5. That was too high to leave room to grow, so the sealed v1 keeps the 14 items that discriminate. On the same answers it scores 51.3, the expected baseline.

Delegation was 9.5% of tool calls in the pilot, far below her real-life ~60%, because these are mostly quick questions, not work requests. When she does delegate here, it's often on something one call could answer.

Weakest family: dead components. That matches the first gates in the orchestration plan: readiness checks before use, and "running" meaning a live lease, not a remembered dispatch. It's the natural first target after the baseline.

Pilot cost $0.65.