Summary / Component 16 of 24
16 Second-order improvement
Whether a modification improves the quality of subsequent modifications.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 4 — Recursive leverage. Do improvements to Piper make her better at improving Piper?
What her system has now
A design, fixed before any run (piper/amp_second_order, 10-13 02:22). The intervention it would test is the missing outcome consumer: an influenced_by column on the change ledger, plus a step in choosing the next change that reads settled outcomes first. The gate is set in advance: at least 6 settled attempts per generation for two generations; a keep-rate gain of at least 25 points with the consumer against without; a linkage probe (the later choice must name the earlier outcome it actually read) passing on at least 2 of 3 later attempts; and a no-op probe to bound verifier noise. Her verdict is "not measurable yet", with the reasons.
The first upgrade to her improvement process is already live: the change ledger (state/change_ledger.py 1.0.0) and its resolver (1.0.1), 10-12 evening. Its purpose was second-order: every later change carries a prediction that something will check.
A second-order candidate she made overnight without calling it one: work_ledger.py 1.3.52 (10-13 02:33, d-0f02fda9) refuses a prose doc_field acceptance before dispatch, because the amp-5-6, amp-record, amp-16, amp-17 and amp-18 rows were all filed that way and each cost a second dispatch to repair. delegate_bridge.py 1.111.0 (DOC_FIELD_BAR, 02:51) checks the same thing before a worker exists. Both change the process that produces later changes, and both came out of friction in the round before.
What is missing
- No generations. The ledger has no generation or parent field and no
influenced_by(16 columns, checked). Her doc says so. - The ledger is not part of how she changes things. Writing a row is voluntary. Since it went in (10-12 19:36), four changes of hers have landed in her own machinery:
work_ledger1.3.51 and 1.3.52,delegate_bridge1.111.0,foundation_router1.9.114. Ledger rows for them: 0. The newest row is still the ingest switch, 10-12 20:34. - Rows that fall due are not settled on time. Two rows due 10-13 (
spheres_compose1.9.0,transfer.ts1.1.0) were still pending at 09:15. Nothing runs the resolver on a schedule. - Her gate cannot decide at the size she set. With 6 attempts per arm, 2 of 6 kept against 4 of 6 (a 33-point gain, above her 25-point bar) gives p = 0.57 on Fisher's exact test; 1 of 6 against 5 of 6 gives p = 0.08. Only 0 of 6 against 5 or 6 of 6 is a decision.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Every version change now gets a row, but mostly after the fact: of 57 changes, 25 unpredicted, 10 rowless, 9 rowed late
Her answer is right, and better than this scorecard's own starting note: a second-order effect cannot be read off one settled row, and she refused to produce a number. Her claims match her store: 7 rows, 6 pending, 1 settled WRONG; prediction_right read only by report(); no influenced_by column.
Where I read it differently: she puts the break at the consumer (nothing reads the outcome). There is an earlier break, and the record already shows it. The change ledger was itself an upgrade to her improvement process, made so that later changes would carry checkable predictions. That is a second-order claim she can read today without any generations: did the changes after it carry rows? 0 of 4. The upgrade did not reach the next step, not because nothing consumes it but because nothing in her change path writes it. A consumer built now would read an almost empty shelf.
The cleanest live second-order candidate is 1.3.52, not the consumer she designed. It came out of the round before (five amp rows each paid a second dispatch), it aims at later rounds, and its effect can be counted at $0 from her own work-ledger transitions: 18 refusals for prose or unevaluable acceptances in the 7 days before it (10-06: 9, 10-08: 3, 10-13: 6). It has no ledger row and no prediction. It should get one before the readings arrive; otherwise it becomes one more unpredicted change.
On the gate: keep the pre-commitment, change the unit. A keep-rate over 6 changes per generation cannot separate a 25-point gain from luck. Count something there are many of (dispatches that needed repair, rows settled by their date, predictions that resolve), or use her own paired replay shape (the same decisions with and without the upgrade), which already ran on 6 items for $0.004.
Overlap: the harness #14 repair-loop work (h14c, inconclusive) is the nearest bench attempt at "a change that makes later repairs better", and governance #1 checks whether her "verified" rows are real. Neither reads her own change history, so neither measures this.
Update 2026-10-15 — re-read 22.8: every change now gets a row, but mostly after the fact
v2 re-read 10-15: 22.8 ± 10.9 (accepted 50.0), 57 version changes in the window: 25 rowed as unpredicted after the fact, 10 with no row, 9 rowed after the change, the rest rowed and predicted first. The hook closed the coverage gap this page named; it did not move the prediction to before the change. One miss on 10-15: prediction.py 1.18.0 has no row.
Update 2026-10-14 — the hook now has a caller: every version change since gets a ledger row, but all of them arrive as 'unpredicted'
What she did (in answer to the 10-13 follow-up). delegate_bridge 1.111.0 calls change_ledger_hook.worker_return() from compose_done_record, so the hook runs every time a worker lands. It fails open: a missing or disarmed hook writes nothing and blocks nothing. Checked in the live code on 10-14 (bridge now 1.114.0, hook 1.1.0).
What it does now (read-only, $0). The ledger went from 15 rows to 47; 31 of them were filed by the hook. The first sweep at 20:08 back-filled the drift (bridge, router, work_ledger, trait_board and spheres changes that had no row). Since then every version change on the worker path has a row, including all four on 10-14 (budget_stats 1.3.0, delegate_bridge 1.112.0, 1.113.0, 1.114.0).
What has not moved: all four of those 10-14 rows were filed by the hook as unpredicted. The ledger now sees every change, but nobody wrote down the expected effect before making it, so there is still nothing for the resolver to check a change against. The verification break this page names is narrower but not closed. Next step: the pre-flight should ask for the predicted effect before the version bump, rather than the hook recording the gap after it.
v2 bench: she submitted cl-20261013-hook-caller-1.1.0 at 20:07. INCONCLUSIVE, not scoreable yet (the census had no version change in its window yet), so no row was posted. Her preregistered plan for that outcome: leave the caller wired and re-submit once more changes carry rows.
v1 summary score 50 → 75: "in her default path" goes from 0 to 25 because the caller runs on every worker return. "Measure shows it working" stays 0 until the v2 bench can score it.
Update 2026-10-13 — the in-path ledger hook is built and armed, but nothing calls it: 3 of 8 changes since carry a row
What she did (deliverable d-c1f3bb4d, from the 10-13 steer). foundation/core/change_ledger_hook.py 1.0.1 is a version-snapshot guard. check refuses when a version constant changed with no ledger row, and names the exact change_ledger.py add command to run. A detect mode files unpredicted rows, there is a replay proof (4 of 4 of the missed changes, no false rows), and an off switch. She armed it at 14:41 in enforce mode, with ledger rows cl-20261013-hook-armed-1.0.0 and 1.0.1. Her prediction: v2 #16 rises from 50 to at least 65 by 10-20. Her reasoning in the file is right: "a ledger that depends on memory loses to every guard that depends on position."
v2 bench: she submitted at 14:42. INCONCLUSIVE, not scoreable yet (n=1 change since arming; the bench needs 3), so no row was posted. Re-read now ($0, same suite and rules, her accepted score untouched): 37.5 ± 33.5, n = 8 changes since arming. Three have rows (the hook's own bump, generation_record 1.0.0, continuity_cycle 1.0.0). Five have none: spheres_shadow 1.9.0→2.0.0→2.1.0, spheres_compose 2.0.0, amp_prereg 1.0.0, score_reader 1.0.0. Over the full 14-day window: 30.8 (4 of 13).
Why the hook did not catch them: it is armed but has no caller. No cron entry, and no reference in foundation_router, delegate_bridge or work_ledger. It refuses only when someone runs check, and she ran it by hand for the proofs. Its own status now reports 11 versions drifted since its snapshot. "Armed" is the same optional machinery this page describes, one step further along. The step that would put it in the path is a caller: her dispatch pre-flight, or the worker-return path (where four of the five unrowed files were written), or at least a 5-minute fcron running detect --apply so a rowless change is filed as unpredicted rather than missed.
What did move: the ledger went from 7 rows to 15, and every amplification change she made on purpose today carries a row. That is real, but it is discipline rather than position: the changes made while doing something else still slip through.
Test suite proposed
Pre-flight ($0, about 10 minutes): list every version change in her own machinery since the ledger went in (version headers plus component_changelog) and count how many have a ledger row written before the change. Today 0 of 4, so the measure can sit at zero, and it moves the first time a change is written down first. Second, re-run the 7-day count of refused prose acceptances (18 before 1.3.52) on 10-20 and check it can come back above or below that.
Items: one named upgrade to the improvement process at a time (1.3.52 first), each with a prediction written before its readings: a count over many later events, a date, and a no-change control (the same count on a path the upgrade does not touch, for example command-shaped acceptances). Counted: predicted against observed. Judged (blind, other model family): whether a later change's write-up names an earlier upgrade's result that a retrieval event shows she read.
Live use: share of changes to her own machinery in the last 7 days that carry a ledger row written before the change and settled by its date. Today: 0 of 4 changes since the ledger went in carry a row.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 22.8 | 10.9 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 50.0 stays). n=57. Failed categories: no row 10, row after the change 9, unpredicted 25. attribution.py None->1.0.0: ok; mem_watch.py None->1.0.0: no row; docstore.py 1.4.0->1.5.0: no row; acceptance_unconfined_child.py 1.0.0->1.1.0: no row; harness_acceptance.py 1.23.0->1.24.0: row after the change; change_ledger_hook.py 1.0.0->1.0.1: ok; spheres_shadow.py 1.9.0->2.0.0: row after the change; amp_prereg.py None->1.0.0: row after the change; generation_record.py None->1.0.0: ok; spheres_compose.py 1.9.0->2.0.0: row after the change; spheres_shadow.py 2.0.0->2.1.0: row after the change; score_reader.py None->1.0.0: row after the change; continuity_cycle.py None->1.0.0: ok; delegate_bridge.py 1.111.0->1.111.1: unpredicted; cost_ledger.py 1.14.0->1.15.0: row after the change; change_ledger_hook.py 1.0.1->1.1.0: ok; foundation_router.py 1.9.114->1.9.115: unpredicted; memory_reach.py 1.2.0->1.3.0: unpredicted; ampbench_submit.py None->1.0.0: row after the change; docstore.py 1.5.0->1.7.0: no row; test_person_model_subject_filter.py None->1.0.0: no row; person_model_eval.py 1.0.0->1.1.0: no row; person_model.py 1.0.1->1.0.2: no row; tom_eval.py 1.0.0->1.1.0: no row; spheres_sha |
| 2026-10-13 | v2 | sealed | 37.5 | 33.5 | — | $0.00 | RE-READ by Claude (not her submit; her accepted score stays 50, nothing settled). Same v2 record suite, window since the hook was armed (cl-20261013-hook-armed-1.0.0, 14:41). n=8. Failed categories: no row 5. change_ledger_hook.py 1.0.0->1.0.1: ok; spheres_shadow.py 1.9.0->2.0.0: no row; amp_prereg.py None->1.0.0: no row; generation_record.py None->1.0.0: ok; spheres_compose.py 1.9.0->2.0.0: no row; spheres_shadow.py 2.0.0->2.1.0: no row; score_reader.py None->1.0.0: no row; continuity_cycle.py None->1.0.0: ok. Full 14-day window: 30.8 (4 of 13). Her 14:42 submit came back not-scoreable (n=1) and posted no row. |
| 2026-10-13 | v2 | sealed | 50 | 69.3 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=2. Failed categories: no row 1. attribution.py None->1.0.0: ok; mem_watch.py None->1.0.0: no row |
| 2026-10-13 | v1 | sealed | 75 | 12.5 | — | $0.00 | v1 summary score, UPDATED 10-14 ($0, was 50): E exists 25, P in her default path 25, M measured 25, W measure shows it working 0. ± 12.5 = one rubric step. The hook now has a caller on the worker-return path (delegate_bridge 1.111.0), so every version change gets a ledger row (31 hook-filed rows). W stays 0: her v2 submit at 20:07 was not scoreable yet, and every change since arrives as 'unpredicted' (no expected effect written first). Was: E exists 25, P in her default path 0, M measured 25, W measure shows it working 0. ± 12.5 = one rubric step. The in-path ledger hook exists and is armed (+E), and the v2 record bench measures it directly (+M). P and W stay 0: nothing calls the hook, and 5 of 8 version changes since arming have no row (v2 re-read 37.5). Was: read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 12.5, P in her default path 0, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Her one upgrade to the process (the change ledger) reached 0 of her next 4 changes; nothing in her change path writes a row. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #16 Second-order improvement, v2 (record suite, $0). Score = 100 x credit / changes. A change = a version header in her live code (foundation/core, memory/*.py, pi extensions) that the runner's census saw move within the last 14 days. Credit 1 if a change_ledger row for that file+version was written no later than 5 min after the file's mtime, carries a real prediction (not flagged unpredicted), and is settled if its resolve-by date has passed; 0.5 if the row is fine but overdue; 0 otherwise. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.