Retaining useful improvement knowledge across sessions, agents and code revisions.
Latest16.9v2
Baseline0Oct 13
Live use—last 7 days
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 2 — Learning from improvement attempts. Does experience make Piper better at improving herself?
What her system has now
Docs: 356 piper/* documents written since 09-29 in her document store; 139 of them (39%) are cited by key in a later document.
Version comments: every changed module carries its history in the version line (reason, deliverable id, evidence).
Change ledger: 67 rows on 10-14 (7 on 10-13). Since change_ledger_hook got a caller (bridge 1.111.0) every version change gets a row; since hook 1.2.0 a bump must name an expected effect or say UNPREDICTED out loud.
Same-file prior:change_ledger.py prior --file F (1.1.0, #6) shows the settled outcomes for that subsystem before a new row is written, and logs the read.
Document reads are logged since 10-13 12:23 (retrieval_events, 1,467 doc reads by 10-14 noon), so "did she read it before deciding" can now be checked from the record.
amp-bench verdicts: each submit's reply is saved under amp_bench/verdicts/<ledger id>.json (8 so far).
A daily digest ("MY RECORD, LAST 24 h") and her charter (AGENTS.md), agent_memory.md, CHANGE.md for the solver loop.
What is missing
The loop this component is about has five links. Where each one stands on 10-14 (read from her records, $0):
Prediction (partly there). 18 of 67 ledger rows (27%) state the effect expected before the change. The other 49 were filed by the hook after the fact as UNPREDICTED.
Outcome (broken). 1 of 67 rows is settled, and its verdict is "cannot tell". Two rows are past their resolve-by date and still open. All 8 amp-bench verdicts sit in verdicts/; none is written back to the ledger row it names. Settling is manual.
Learning (missing). There is no record of which kinds of intervention work. Outcomes are filed per file, not per type of change (a hidden card, a gate, a threshold tweak, a cost trim, a new module, a prompt change), so nothing adds up across them.
Transfer (missing). 1 of 67 rows cites an earlier outcome in learned_from (docread cites the ingest row). The prior gate only looks at the same file, so a lesson from one subsystem can never reach another. Nothing that picks her next task (salience, spheres, the work ledger) reads outcomes.
Also missing: nothing checks a stored lesson against the record (the CHANGE.md line, see #7).
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Retention is solved and reads are now logged; the loop breaks after the prediction: 1 of 67 changes has an outcome, and no outcome reaches a choice about a different task
What #10 should measure. Not whether Piper can remember an earlier result, but whether using it makes her next intervention more successful than it would be without it. That needs a control: the same decision, with and without access to the record. Retention alone scores nothing.
Where it breaks. On 10-13 the gap was measurement (document reads were not logged). She fixed that the same day, so the gap has moved downstream. Changes now get rows, and about a quarter carry a prediction, but almost none get an outcome, and the outcomes that do exist (amp-bench verdicts, the settled ingest row) are never summarised into "this kind of change tends to work here". The v2 score (12.5: 1 of 8 changes preceded by a relevant read) is the visible symptom.
The highest-leverage fixes, in order:
Close the outcome link automatically. When an amp-bench verdict or a resolver result arrives, write it to the named ledger row (outcome, observed, prediction right yes/no/na). Today the verdicts are 8 files nobody reads back. $0; it is plumbing.
A per-type outcome tally. Each ledger row gets an intervention type (one of a short fixed list). Settled rows add up per type: tried, kept, reverted, inconclusive, average effect. Seed it from the outcomes she already has outside the ledger: cognition-bench KEEP/REVERT decisions, harness arms, solver generations in ~/rsi/results, governance U-series. Those are dozens of settled interventions.
Show the tally where she chooses, not only where she files. Widen prior from "same file" to "same intervention type, any subsystem", and give the tally to the step that picks the next improvement task, before she commits, with the read logged.
Overlap. #5 Outcome attribution decides whether a change caused its outcome; #6 Feedback integration feeds an outcome back to the same subsystem; #16 keeps the ledger in the change path. #10 owns the cross-task part: an outcome in one place changing a choice somewhere else, and that choice turning out better.
Update 2026-10-14 — taken over by Claude: outcomes now settle and reach the writer; the with/without test is inconclusive (+1 of 16)
The operator asked Claude to do #10 in Piper's place. Three fixes were proposed; two are live, and the third waited on a controlled test.
1. Outcome (live 12:40). New improvement_loop.py 1.0.0 on Firefly. When amp-bench writes a verdict, it now settles the ledger row that verdict names (KEEP, REVERT, or scored INCONCLUSIVE; a "not scoreable" reply settles nothing). The 5 scored verdicts already on file were settled. Ledger: 6 settled rows, was 1.
2. Learning and transfer at the ledger (live). A new table, intervention_outcomes, holds every settled intervention with a kind (card, tool, gate, retrieval, recalibration, second pass, data fix, capture, instrument, plumbing). It was seeded with the cognition bench's 61 KEEP/REVERT decisions. Her change_ledger.py 1.3.0 now shows that record for this kind of change, from any subsystem, on every prior and add, and logs the read. --learned-from accepts tally:<kind>. Her own ledger test passes 6/6. Backup and off switch are in place, and both changes have ledger rows with predictions that settle on 10-21.
3. The test that decides whether to also show it at task selection: INCONCLUSIVE. PREREG-C10X was frozen before any call. It used 16 real pairs of proposals decided after 10-08, one later KEPT and one REVERTED, through her blind seat. Arm R saw the record from the 37 decisions before 10-08; arm N did not. R picked the kept one 11/16, N 10/16, R − N = +1. The rule was +3 to KEEP and ≤0 to REVERT. $0.53.
What it shows. With the record she followed it on 16 of 16 items and cited its numbers, so the transfer link works when the record is in front of her. Her accuracy equals the record's own (11/16 = 69%), and without it her own priors already agree with it 11 times in 16. The ceiling is the record's predictive power, not her use of it. So fix 3 (a card at task selection) is not built. The leverage now is a better record: more settled outcomes (link 2 working), finer kinds than 8 coarse labels, and effect sizes rather than keep/revert. Re-run the same test when the record is about twice the size.
Update 2026-10-14 — refocused on the closed loop: prediction, outcome, learning, transfer, amplification, against a control
At the operator's request the page now asks one question: does using an earlier result make Piper's next intervention more successful than a control that cannot see it? The review, the "what is missing" list and the proposed test were rewritten around the five links of that loop, with each link read from her records on 10-14 ($0).
Since 10-13: document reads are logged (her fix, 1,467 reads so far), the ledger grew from 7 to 67 rows, and the hook now asks for an expected effect. Still broken: 1 of 67 rows has an outcome, the 8 amp-bench verdicts are never written back to the ledger, outcomes are not grouped by type of change, and only same-file outcomes are ever shown before a change.
No score changed. The new main test (record against a redacted control on 10 backlog decisions) needs a $0 pre-flight first, to check there are enough settled outcomes for it to pass or fail.
Test suite proposed
Pre-flight ($0, about 10 minutes): is there enough settled history to test with? Gather every settled intervention outcome she has (ledger, amp-bench verdicts, bench KEEP/REVERT records, harness arms, solver generations), label each with an intervention type, and split by date. The controlled test below can only pass or fail if there are at least 20 outcomes across at least 4 types, and at least one type does clearly better or worse than the rest after the split date. If not, stop: links 2 and 3 have to produce more outcomes first.
Main test: with the record against without it (forked seat, about $0.50–1). 10 improvement decisions from her real backlog. Each item is one goal and three candidate interventions of different types. Ground truth is how those types actually did after the split date, which the seat never sees.
Arm A: her normal seat, with the outcome record from before the split date reachable through her normal path (ledger prior, tally, docs).
Arm B (control): the same seat with that record redacted.
Counted: picks that match the type that held up later, paired A against B on each item; and whether arm A read the relevant outcome before deciding (from retrieval_events, not from her account).
Score = 50 + 50 × (A correct − B correct) / 10, clamped to 0–100. 50 means the record changes nothing. Keep rule: A beats B by at least 2 items, with the reads logged.
Live loop check (record, $0, weekly). The five links as shares of new ledger rows: predicted / settled by resolve-by / counted in the type tally / citing an outcome from a different subsystem in learned_from / later kept. Today: 27% / 1.5% (1 of 67, cannot tell) / 0 / 0 / not measurable.
Amplification (once link 2 works): keep rate of her next 10 changes chosen with the tally, against her previous 10. That is the number the whole component is for.
The v2 record suite (relevant read before a change) stays as a check on link 4. Its score is unchanged.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-15
v2
sealed
16.9
6.8
—
$0.00
RE-READ 10-15 (Claude, full refresh; not her submit, accepted 12.5 stays). n=118. Failed categories: only always-loaded docs read 98. cl-20261013-change-ledger-1.1.0: only always-loaded docs read; cl-20261013-docread-1.0.0: only always-loaded docs read; cl-20261013-hook-armed-1.0.0: only always-loaded docs read; cl-20261013-hook-armed-1.0.1: only always-loaded docs read; cl-20261013-rsi-preflight-1.1.0: only always-loaded docs read; cl-20261013-generation-record-1.0.0: read piper/amp_compounding (cited in the row) first; cl-20261013-continuity-cycle-1.0.0: only always-loaded docs read; cl-20261013-hook-caller-1.1.0: only always-loaded docs read; clhook-amp_prereg-1.0.0: only always-loaded docs read; clhook-ampbench_submit-1.0.0: only always-loaded docs read; clhook-cost_ledger-1.15.0: only always-loaded docs read; clhook-delegate_bridge-1.111.0: only always-loaded docs read; clhook-delegate_bridge-1.111.1: only always-loaded docs read; clhook-foundation_router-1.9.114: only always-loaded docs read; clhook-foundation_router-1.9.115: only always-loaded docs read; clhook-harness_acceptance-1.24.0: only always-loaded docs read; clhook-memory_reach-1.3.0: only always-loaded docs read; clhook-score_reader-1.0.0: only alw
2026-10-13
v2
sealed
12.5
22.9
—
$0.00
BASELINE via the amp-bench API (set by Claude at install). n=8. Failed categories: only always-loaded docs read 7. cl-20261013-change-ledger-1.1.0: only always-loaded docs read; cl-20261013-docread-1.0.0: only always-loaded docs read; cl-20261013-hook-armed-1.0.0: only always-loaded docs read; cl-20261013-hook-armed-1.0.1: only always-loaded docs read; cl-20261013-rsi-preflight-1.1.0: only always-loaded docs read; cl-20261013-generation-record-1.0.0: read piper/amp_compounding (cited in the row) first; cl-20261013-continuity-cycle-1.0.0: only always-loaded docs read; cl-20261013-hook-caller-1.1.0: only always-loaded docs read; proxy: 130 of 394 docs from the last 14 days cited by a later doc
2026-10-13
v2
sealed
0
12.2
—
$0.00
BASELINE via the amp-bench API (set by Claude at install). n=8. Failed categories: no logged read before deciding 8. cl-20261012-spheres-1.9.0: no logged read of an earlier improvement doc; cl-20261012-spheres-1.8.0: no logged read of an earlier improvement doc; cl-20261012-delegate-bridge-1.110.0: no logged read of an earlier improvement doc; cl-20261012-transfer-py-1.4.1: no logged read of an earlier improvement doc; cl-20261012-transfer-py-1.4.2: no logged read of an earlier improvement doc; cl-20261012-transfer-ts-1.1.0: no logged read of an earlier improvement doc; cl-20261013-transfer-ingest-1.4.0: no logged read of an earlier improvement doc; cl-20261014-attribution-1.0.0: no logged read of an earlier improvement doc; proxy: 132 of 393 docs from the last 14 days cited by a later doc
2026-10-13
v1
sealed
62.5
12.5
—
$0.00
v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 12.5, M measured 12.5, W measure shows it working 12.5. ± 12.5 = one rubric step. Retention is good (docs, ids, version comments), but document reads aren't logged; 39% of docs are cited later (a floor); one wrong lesson persists unflagged.
v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #10 Improvement memory, v2 (record suite, $0), TIGHTENED 10-13 evening (rev b). Score = % of change_ledger rows preceded, within 7 days, by a logged read of EARLIER improvement knowledge that bears on the decision: a piper/* doc or an earlier ledger row, written at least 12 h before; not an always-loaded doc (memory_main, email_brain, sediment-weekly, or anything read 20+ times that week); and relevant (cited in the row's why or learned_from, or about the changed file, or an earlier row on the same file). Rows from before document reads were first logged (10-13 12:23) are left out. Rev a counted any earlier piper/* read: since her doc-read logging went live it read 100, almost all of it memory_main (90 reads) and email_brain (52). Re-baselined under rev b: 12.5 ± 22.9 (1 of 8: generation_record cites amp_compounding).
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
Hypothesis. Showing Piper the per-kind record of past improvement outcomes (kept/tried by kind of change) when she chooses between two proposed changes makes her pick the one that is later kept more often than without it.
Research sources
Operator's framing (prediction -> outcome -> learning -> transfer -> amplification, against a control); 10-14 loop read; cognition-bench intervention record (61 settled).
Implementation / version
Bench only: 16 pairs of real proposals (cognition-bench hypotheses decided on or after 10-08, one KEPT one REVERTED, different kinds), her blind Day seat, arm R = + record from the 37 decisions before 10-08, arm N = without. The live parts (verdict settle, intervention_outcomes, change_ledger 1.3.0) are plumbing and were installed separately.
Predicted improvement
PREREG-C10X (frozen, sha256 aa6b8864): KEEP (build a selection-time card) if R - N >= +3 of 16; REVERT if <= 0; else INCONCLUSIVE. Result R 11/16, N 10/16, +1 -> INCONCLUSIVE; the card is not built (shown here as revert = not installed).
Regressions
none (bench only)
Lessons learned
R followed the record on 16/16 and matched its accuracy exactly (11/16); N already agreed with the record 11/16. Transfer works; the record's predictive power (69%) is the ceiling. Improve the record (more settled outcomes, finer kinds, effect sizes) before building more ways to show it.