Summary / Component 11 of 24
11 Improvement throughput
Verified beneficial changes produced per unit of time.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 3 — Improvement efficiency. Does Piper become faster or cheaper at improving herself?
What her system has now
- Work volume: 838 deliverables created since 10-06, 703 verified (
deliverables). Verified means the acceptance passed, not that something improved. - Change ledger: 7 changes recorded, 1 settled, 0 settled as right.
- Solver loop (
~/rsi/results): 5 kept versions in 17 days (09-26 to 10-12): 2 for capability (v2, v3, by 09-30), 3 for cost (v1, v4, v5). - A gain that is visible but unsettled:
transfer.py1.4.1 cut empty lookups from 73% to 4% on live rows; its ledger row waits until 10-19.
What is missing
- No count of "verified beneficial changes" anywhere: the work ledger counts finished tasks, the change ledger is mostly unsettled, and the solver loop's keeps are not joined to either.
- No time axis: nothing reports keeps per week.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Verified gains per week fell 2, 1, 0 while changes tried rose to 127; seven verdicts settled 10-15 put this week at 2 measured gains
Throughput of activity is very high: dozens of verified deliverables a day. Throughput of verified improvement is close to zero on the instruments she has: the change ledger has settled nothing as right, and the solver loop has kept no capability change for two weeks. The gap between the two is the program's main finding so far (see #16 and #19): she can change things much faster than anything settles whether the change helped.
The 1.4.1 result shows the instruments are not hopeless: a clear gain, measured on her own log, available today. It only needs reading.
Overlap: governance #1's nightly hollow-pass check reads whether "verified" deliverables could have passed without the work. That is the right filter for this count.
Update 2026-10-15 — a weekly count of verified gains now exists: it fell to zero for ten days while attempts went from 7 to 127 a week
What Claude built (read-only, $0, Piper not told). A weekly series of verified gains, credited to the week the change was made, from three records: change-ledger rows settled right, solver-loop keeps, and amp-bench KEEP verdicts. Each gain is tiered: measured gain (a quantity moved: solver "smarter" keeps, amp-bench KEEPs, ledger rows whose prediction named a quantity), cost gain (solver "cheaper" keeps), and mechanism verified (the change works as designed, but nothing measured improved).
| week of | changes tried | measured | cost | mechanism |
|---|---|---|---|---|
| 09-21 | 2 | 1 (solver v2) | 1 (v1) | 0 |
| 09-28 | 7 | 1 (solver v3) | 1 (v4) | 0 |
| 10-05 | 5 | 0 | 1 (v5) | 0 |
| 10-12 | 127 | 0 → 2* | 0 | 0 → 5* |
* After 19:25 on 10-15, when seven verdicts from Claude's settle-from-logs pass on #13 are written to her ledger: the lookup fix transfer 1.4.1 (empty lookups 79% → 4%) and the hook caller (unrowed version changes 5 of 8 → 1 of 14) as measured gains, five more as mechanism checks. Before that, the last measured gain was solver v3 on 09-30.
Read: verified gains per week fell from 2 to 1 to 0 while changes tried went from 7 to 127. Part of the jump in "tried" is counting: from 10-13 the hook files a row for every version change, where before only hand-written rows existed. Even so, the gap between changing and verifying is the program's main finding (see #16 and #19), now as a number per week. The settle pass shows most of the gap is bookkeeping: of the 23 predicted rows, every one that could be decided was right; nobody had written the verdicts down. Lag from change to verdict is 1 to 3 days when someone looks.
Bench, rev b (amp-bench commit 8214900): rev a gave 25 points to anything settled right in the last 7 days, so clearing a backlog of "works as designed" checks would have read 100 with one measured gain behind it. Rev b weights measured 25, cost 12.5, mechanism 5 (cap 100). Re-baselined 25.0 → 12.5 (the only gain verified in the last 7 days is the v5 cost keep). After tonight's write it reads 87.5, then falls back as the week rolls on unless new gains are verified. It can move both ways.
Test suite proposed
Pre-flight ($0, about 10 minutes): settle the 1.4.1 row by hand from transfer.jsonl (empty share by version, live rows) and the 10-12 to 10-19 solver sessions, and check that a weekly "verified gains" count can be formed from existing files. Today it reads 1 (v5, a cost keep) for the last 7 days, and 1.4.1 would make it 2. The count can move.
Items: none beyond the count itself: keeps per week, by kind (capability, cost, reliability), only for changes with a settled prediction and, where possible, a governance #1 hollow-pass check.
Live use: verified beneficial changes in the last 7 days. Today: 1 (solver v5, cost), plus 1.4.1 once settled.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 12.5 | 25 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 12.5 stays). n=1. Failed categories: no measured gain this week 1. solver v5 kept 2026-10-09 (cost) |
| 2026-10-15 | v2 | sealed | 12.5 | 25 | — | $0.00 | RE-READ by Claude under rev b (tiered: measured 25 / cost 12.5 / mechanism 5; was rev a, 25 flat). Accepted re-baselined 25.0 -> 12.5. Credited: solver v5 kept 2026-10-09 (cost). Failed category: no measured gain this week. Projected 87.5 after the 19:25 write of 7 settle-from-logs verdicts (2 measured, 5 mechanism). Weekly series on the update card. |
| 2026-10-13 | v2 | sealed | 25 | 25 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=1. Failed categories: none. solver v5 kept 2026-10-09 |
| 2026-10-13 | v1 | sealed | 37.5 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 0, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Very high activity, near-zero verified gain: no ledger row settled right, and no solver capability keep since 09-30. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #11 Improvement throughput, v2 (record suite, $0). Score = min(100, 25 x verified beneficial changes in the last 7 days). Counted: change_ledger rows settled with prediction_right = yes, solver keeps (capability or cost), and amp-bench KEEP verdicts. A change you only believe helped does not count. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.