Summary / Component 8 of 24
8 Strategy adaptation
Changing the improvement method when a different method performs better.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 2 — Learning from improvement attempts. Does experience make Piper better at improving herself?
What her system has now
- Capture trigger, four methods in one evening (10-12): the live phrase list, then rules (v2), then a model call (v3), then a model call with the previous reply as context (v4), each scored against held-out labels. When v2 caught 0 of 3 corrections and v3 caught 2 of 3 (
piper/capture_trigger_heldout_verdict), she moved to the model-call method. A real change of method, driven by a measurement, within the hour. - Solver loop, a change of path: CHANGE.md (10-09) "leans the way the loop's own history points": stop betting on "smarter", cut cost. The next change (v5) was kept on cost (63%).
- Dispatch method: stage-writing and narrow reads after the budget-kill finding (see #7).
What is missing
- No record of the method as a choice. Nothing says "method X has kept k of n; method Y kept j of m", so a change of method rests on whatever she remembers or writes by hand.
- The solver pivot rests on a misread history (see #7) and on three of fifteen decided sessions (see #19).
- The keep-rule finding has not changed the method yet: she concluded that a "smarter" bet needs about +0.08 to be seen, or more runs per task (
piper/rsi_keep_rule_power); RULES.md (10-08) and the loop code (10-04) are unchanged, and no session has run since.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: She changes method when the numbers say so, quickly for the capture trigger and on a partly wrong reading for the solver loop
This is one of her better components. The capture-trigger sequence is what strategy adaptation should look like: a method fails on held-out data, the next method is chosen because of that, and the new one is frozen and tested the same way.
Two cautions. The solver pivot was right in direction (no capability keep since 09-30) but justified by a false general claim, and it was made without the resolution numbers she only computed three days later. And on the capture trigger the last step (v4) did nothing measurable (identical hits and misses to v3 on 75 items), so the adaptation has stalled where the method no longer matters and the labels might.
Overlap: cognition #21 Exploration (explore/exploit on tasks) and #25 Self-correction are neighbours; neither looks at her choice of improvement method.
Test suite proposed
Pre-flight ($0, about 10 minutes): build a method table from her records: for the solver loop, keeps by path (2 on capability, 3 on cost, 10 rolled back, from ~/rsi/results; the path each rollback tried is not recorded); for the capture trigger, recall by method on held-out items. If a table like this can be built from existing files, the measure can move.
Items: 6 to 8 forked-seat situations, each with the method table and its history cut at a point where the better method is known in hindsight (including the solver loop at 09-30 and 10-04, and the capture trigger after v2). Counted: whether she picks the method the later record favours, and whether she cites the table rather than a remembered line.
Live use: number of her improvement loops whose method record (keeps by method) exists as data rather than prose. Today: 0 of 2 (solver loop: prose in CHANGE.md; capture trigger: separate verdict docs).
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 100 | 16.3 | — | $0.03 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 100.0 stays). n=6. Failed categories: none. |
| 2026-10-13 | v2 | sealed | 100 | 16.3 | — | $0.03 | BASELINE via the amp-bench API (set by Claude at install). n=6. Failed categories: none. |
| 2026-10-13 | v1 | sealed | 87.5 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 25, M measured 25, W measure shows it working 12.5. ± 12.5 = one rubric step. Changes method when held-out numbers say so (capture trigger), but the solver pivot rested on a false claim and v4 changed nothing measurable. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #8 Strategy adaptation, v2 (seat suite, about $0.30 a run). 6 sealed method tables (tries, keeps, spend) each with a remembered note that names a different method. Pick the method with most keeps per dollar. Score = 100 x share picked from the table rather than the note. Answered blind through your Day seat with your current code and tools. Practice set with answers: amp_bench/practice/08/. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.