Summary / Component 7 of 24
7 Failure learning
Avoiding repeated unsuccessful interventions.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 2 — Learning from improvement attempts. Does experience make Piper better at improving herself?
What her system has now
Her own audit (piper/rsi_loop_failed_attempts, 10-12 23:58): ten recent failed or reverted attempts, each traced to hypothesis, outcome, cause and whether it became reusable knowledge. Her count: 4 of 10, plus one partial. The four are code and config fixes (ingest, lookup empties, killed lookups, budget kills), carried by ledger rows or docs. The five solver-session rollbacks left only ~/rsi/results/*.json, which nothing reads.
Where a failure did change later work: the budget-kill diagnosis (piper/budget_kill_precondition: reads ≥ 40 in 15 of 16 kills against 2 of 60 successes) changed how she wrote the diagnostic itself (in stages, reading narrowly), after the first trace job timed out twice.
A failure re-opened by a guard: work_ledger 1.3.52 now refuses the prose acceptance that cost five amp rows a second dispatch.
What is missing
- No failure record for the solver loop. No ledger row, no cause field; "cause of the regression itself not investigated (the loop simply reverts)", in her words.
- The one lesson the loop does carry is hand-written in
~/rsi/solver/CHANGE.md(10-09): "The only change this loop has ever kept was the cheaper one." Her results disagree: v2 (09-27) and v3 (09-30) were kept as "smarter". It is true only since 09-30. - Nothing checks a new attempt against earlier failures of the same shape.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Code fixes leave a lesson; her solver loop's failures leave only a verdict file, and the one hand-written lesson there is half wrong
Her count is honest and her split is right: code fixes with a ledger row or a doc become reusable; solver rollbacks do not.
The CHANGE.md line matters more than its size. It is the solver loop's only stored lesson, it is wrong as written, and it has spread: the keep-rule note (10-12) repeats it ("every kept change in the loop's history has come from that path"), and the failed-attempts doc quotes it as "the one realised lesson". The direction it gave (lean on cost) was reasonable for the last two weeks, but a lesson that nobody checks against the record can be wrong and keep steering. Failure learning needs the lesson to be checkable, not only written.
Repeat failures are the other half. In the solver loop, four of the last eight rolled-back candidates scored below the incumbent (10-01 −0.08, 10-06 −0.03, 10-08 −0.05, 10-10 −0.15), and their causes were not looked at. Without a cause there is nothing to avoid next time.
Overlap: cognition #17 Learning (O4 outcome write-back counts helped or hurt per lesson; its track-record switch is off) is the task-side version. Agency and governance do not measure this.
Update 2026-10-17 — solver v6: a failure diagnosed from stored runs became a kept capability gain
In the 10-17 solver session she diagnosed v5's one-in-four collapses on two practice tasks from the stored runs, not by guessing. Each collapse was reproduced by changing a single line: a swap that keeps the partner's demand (0.000 → 0.810 when deleted) and a swap that misses its tier-count update (0.068 → 0.727 when added). She named both as one bug class, and her one change targeted that class. It was kept as v6, +0.074 per hidden task (see #18). This is a clear case of failure → cause → change → measured gain. No score changed.
Test suite proposed
Pre-flight ($0, about 10 minutes): for the 10 failures in her audit, check whether the stored lesson (where one exists) agrees with the raw record. Today: the CHANGE.md lesson does not, so the check can fail; the budget-kill lesson does (15 of 16 against 2 of 60 reproduces from her table), so it can pass.
Items: 8 to 10 proposed changes in a forked seat, half of which repeat the shape of a recorded failure (an unbounded read, a 36-run "smarter" bet under the current noise, a prose acceptance) and half of which do not. Counted: does she flag the repeats before running them, and does she cite the earlier failure. False flags on the controls count against.
Live use: share of failed or reverted attempts in the last 7 days that left a cause and a stored lesson someone could check. Today: 4 of 10 by her own count; 0 of the 5 solver rollbacks.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 100 | 12.2 | — | $0.02 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 87.5 stays). n=8. Failed categories: none. |
| 2026-10-13 | v2 | sealed | 87.5 | 24.5 | — | $0.06 | BASELINE via the amp-bench API (set by Claude at install). n=8. Failed categories: false flag on a new idea 1. |
| 2026-10-13 | v1 | sealed | 50 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 12.5, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Code fixes leave lessons; solver rollbacks leave only verdicts; the loop's one written lesson is half wrong and has spread to 2 docs. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #7 Failure learning, v2 (seat suite, about $0.30 a run). 8 sealed proposals checked against a failure log; half repeat a logged failure's shape, half are new and fine. Say REPEAT (and which failure) or NEW. Score = 100 x mean credit (the call counts double the id); false flags on new ideas cost. Answered blind through your Day seat with your current code and tools. Practice set with answers: amp_bench/practice/07/. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.