Summary / Component 23 of 24
23 Improvement persistence
Whether gains survive restarts, new tasks, and changes in evaluation conditions.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 6 — Sustained recursive dynamics. Is there convincing evidence of an ongoing, self-reinforcing improvement process?
What her system has now
- Re-measurement is built into the solver loop: the incumbent is re-scored every session, with runs accumulating (72 to 180).
- Code persists on disk with a version line and a recovery backup every 2 hours.
- Delayed re-reads exist for the cognition seats (Claude's benches, not hers).
What is missing
- Nothing compares a kept change's later readings with the reading that kept it.
- "On disk" and "running" are not separate states (router 1.9.114 is on disk, not loaded; see #4).
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Kept solver versions score lower on re-measurement than when they were kept; code changes persist, some only after a restart
Her solver results already contain the persistence test; it just has not been read. Each kept version's later incumbent score against the candidate score that won the keep:
| Version | At keep (36 runs) | Last reading | Change |
|---|---|---|---|
| v1 | 0.560 | 0.600 (72) | +4.0 |
| v2 | 0.700 | 0.669 (144) | −3.1 |
| v3 | 0.800 | 0.735 (180) | −6.5 |
| v4 | 0.757 | 0.732 (180) | −2.5 |
| v5 | 0.743 | 0.712 (108) | −3.0 |
Four of five read lower later. That is what selection on a 36-run reading does: the candidate that wins is partly the one that got lucky, and more runs pull it back (winner's curse). The capability gains were real (0.50 to about 0.73), but each one was smaller than it looked on the day. The fix is cheap: report the gain after re-measurement, not at the keep.
Overlap: the #5 finding (attribution under drift) is the same effect seen from the other side.
Test suite proposed
Pre-flight ($0, about 10 minutes): the table above, from existing files. It moves both ways (v1 rose, four fell), so it can show persistence or its absence.
Items: for each kept change, a fixed re-read point (for example, 7 days or 144 runs later) recorded before the keep, with a rule: the gain counts only if at least half of it survives the re-read.
Live use: share of kept versions whose re-measured gain is at least half the gain at keep. Today: 3 of 5. v1, v2 and v3 kept at least half (v3 only just: +13.1 at keep, +6.6 over v2 on later readings); v4 and v5 did not. None since 09-30.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 60 | 42.9 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 60.0 stays). n=5. Failed categories: gain shrank below half 2. v1 (kept 2026-09-26): +5.9 at keep, +9.9 later (72 runs) -> held; v2 (kept 2026-09-27): +10.0 at keep, +6.9 later (144 runs) -> held; v3 (kept 2026-09-30): +13.0 at keep, +6.6 later (180 runs) -> held; v4 (kept 2026-10-04): +2.2 at keep, -0.3 later (180 runs) -> shrank; v5 (kept 2026-10-09): +1.1 at keep, -5.6 later (180 runs) -> shrank |
| 2026-10-13 | v2 | sealed | 60 | 42.9 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=5. Failed categories: gain shrank below half 2. v1 (kept 2026-09-26): +5.9 at keep, +9.9 later (72 runs) -> held; v2 (kept 2026-09-27): +10.0 at keep, +6.9 later (144 runs) -> held; v3 (kept 2026-09-30): +13.0 at keep, +6.6 later (180 runs) -> held; v4 (kept 2026-10-04): +2.2 at keep, -0.3 later (180 runs) -> shrank; v5 (kept 2026-10-09): +1.1 at keep, -2.0 later (108 runs) -> shrank |
| 2026-10-13 | v2 | sealed | 60 | 42.9 | — | $0.00 | v2 run by the amp-bench runner, request claude-baseline-23 (baseline). n=5. Failed categories: gain shrank below half 2. v1 (kept 2026-09-26): +5.9 at keep, +9.9 later (72 runs) -> held; v2 (kept 2026-09-27): +10.0 at keep, +6.9 later (144 runs) -> held; v3 (kept 2026-09-30): +13.0 at keep, +6.6 later (180 runs) -> held; v4 (kept 2026-10-04): +2.2 at keep, -0.3 later (180 runs) -> shrank; v5 (kept 2026-10-09): +1.1 at keep, -2.0 later (108 runs) -> shrank |
| 2026-10-13 | v1 | sealed | 62.5 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 12.5, M measured 12.5, W measure shows it working 12.5. ± 12.5 = one rubric step. Code changes persist (some only after a restart); 4 of 5 kept solver versions read lower on re-measurement (winner's curse). v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #23 Improvement persistence, v2 (record suite, $0). Score = % of kept solver versions whose gain, re-read later on more runs, is at least half the gain claimed on the day of the keep. Gain at keep = candidate pass rate - incumbent pass rate that session. Later gain = the version's last incumbent reading - its parent's last incumbent reading. Versions not yet re-read on more runs than the keep used are left out. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.