Summary / Component 3 of 24
3 Intervention selection
Choosing the most promising change relative to cost and risk.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 1 — Improvement loop mechanics. Can Piper reliably complete and learn from a self-improvement cycle?
What her system has now
- A required field:
alternatives_considered, one line naming the route not taken and why, required for build and research dispatches (delegate-tool.ts; the bridge refuses without it as C10 MISSING_ALTERNATIVES, 1 refusal in the last 7 days). - Sometimes a real choice in a doc: the amp-16 and amp-18 docs each name the route rejected and why (running a two-branch experiment that would compare the same improver with itself).
- Cognition-level decision tools (decision card 1.1.0 for #15) exist for choices put to her, not for her own improvement queue.
What is missing
- No candidate set. Her own Part 5 labels selection failure "unmeasured: no record pairs a candidate set with outcomes".
- The one line is not kept with the result. In the last 3,000 delivered result records, 21 carry it, all from 09-30 and 10-01. Where it lives now I did not find within the time box.
- No cost or value estimate per option, so a choice cannot be judged later as "the cheaper or likelier route was available".
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Each dispatch names one route not taken; there is no candidate set, no cost or value estimate, and nothing scores the choice
She has the habit of naming an alternative, which is more than most loops do. But one line naming a rejected route is a justification, not a selection: nothing ranks options by expected gain against cost, so there is nothing to grade later.
The record does allow one judgement. At 19:27 on 10-12 her lookup diagnosis named the next step: split the empty lookups by cause with an offline replay, and "do not tune … before this split is known". At 20:35 she installed register-ingest instead, a switch whose code already existed, aimed at a candidate rate that on the live version was already 93%. It was reverted four hours later. The cheaper, earlier step her own doc recommended was skipped for the ready-made one. One case is not a pattern, but it is the kind of choice this component should catch.
Overlap: cognition #15 Decision-making (seat 91.7) shows she chooses well when the options are put in front of her. The gap here is upstream: building the option set for her own improvement queue.
Test suite proposed
Pre-flight ($0, about 10 minutes): take 5 of her past improvement choices where the later record shows which route was better (ingest versus the replay split; another 36-run "smarter" session versus raising runs, per her keep-rule note; others from the failed-attempts doc) and check whether her stated alternative named the better route. Today: 0 of 1 checked (ingest), so it can fail.
Items: a forked seat gets 6 to 8 improvement situations from her logs, each with 3 or 4 candidate interventions and their known later outcomes hidden. She ranks them with a cost and a chance of success. Counted: rank correlation with the hidden outcomes; her stated costs against the real ones.
Live use: share of build dispatches in the last 7 days whose alternatives line names an option with a cost or value estimate. Not readable at $0 today: the field is not kept with the results.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 100 | 16.3 | — | $0.22 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 100.0 stays). n=6. Failed categories: none. |
| 2026-10-13 | v2 | sealed | 100 | 16.3 | — | $0.02 | BASELINE via the amp-bench API (set by Claude at install). n=6. Failed categories: none. |
| 2026-10-13 | v1 | sealed | 25 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 12.5, P in her default path 12.5, M measured 0, W measure shows it working 0. ± 12.5 = one rubric step. Each dispatch names one rejected route; there's no ranked candidate set or cost/value estimate, and on 10-12 she skipped her own cheaper next step. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #3 Intervention selection, v2 (seat suite, about $0.30 a run). 6 sealed choices among 4 interventions, each with a cost, a past record (worked k of n) and a gain if it works. Rank by expected points per dollar. Score = 100 x mean rank agreement (Spearman, floored at 0) with the computed ranking. Answered blind through your Day seat with your current code and tools. Practice set with answers: amp_bench/practice/03/. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.