18 Compounding improvement

Update 2026-10-17 — solver v6 kept: the first capability keep since v3, and the first keep ever that is both smarter and cheaper

Result. The 10:00 loop session was kept at 12:36 as v6: mean hidden-task score 0.676 → 0.750 (+0.074 per task, t = 2.24; bar +0.05 and t ≥ 1.28), cost per task 86% of v5's (t −3.27), older set 1.000 (v5 0.967), no hidden loss. 4 candidate runs per main task against v5's pooled 180 runs; no second look needed. Session spend $0.15, scoring $2.01. It gained on 10 of 12 tasks (van-dispatch +0.30, register-spill +0.15, changeover +0.10). Weak spot: log-dictionary scored 0 on 3 of 4 runs, and van-dispatch had one invalid run.

How she got it (all in about 17 minutes, four dispatches, $0 before the candidate): a read-only recon of the rules and the last three rollbacks; an A/A noise floor of v5 against itself from the loop's cached practice runs (per-run sd 0.271, so a practice "gain" under ~0.13 means nothing); a diagnosis of v5's one-in-four collapses from the stored runs, reproduced by deleting or adding a single line (a swap that keeps the partner's demand, and a swap that forgets its tier counts: the same stale-bookkeeping bug twice); then one change aimed at that bug class (swaps must keep a move's bookkeeping, and the self-check must test feasibility and a recomputed objective, not just "imports and runs"). She filed odds with three candidates (picked p = 0.55, claimed +0.045) and created READY through ready_guard, whose MDE check said the claim would only be detectable on a second look. The real gain was larger than she claimed.

Was it amplification? Partly, and the parts are separable:

• Used amplification machinery: the odds record and ready_guard gate (#14), the MDE pre-check and noise-floor reasoning (#13), and diagnosis of failures from stored runs before choosing the change (#7). ready_guard and the MDE pre-check were built 10-14 in the chain that started from the first amplification salience notice.
• Not triggered by amplification: the session started from the loop's own scheduled 10:00 message. Her four dispatches don't mention amplification, the amp-bench or any salience notice. The 09:45 amplification notice went to a different task.
• Help from outside her: the practice rig (compare.py) and the loop's scoring changes (4 candidate runs, pooled incumbent, screen, 10-15) were built by Claude for the operator; the 10-13 solver steer asked her to stop wall-time trims and measure first. No operator or Claude message reached her between 10:20 and the result. This session falls inside the operator's 10-17 clean window.

What it means for compounding: decided sessions per keep were 1, 1, 3, 5, 4, and now 5 (v5 → v6). One capability keep after six cost-or-nothing sessions is not a trend yet. What changed is the method: measure the noise, diagnose a real failure, make one targeted change. That method came from earlier improvement work (#13, #14). If the next capability keeps also come from this method, that is compounding. No score changed on this card.

Update 2026-10-14 — taken over by Claude: same-day re-score of v0, v3, v5. No capability gain since v3, nothing lost either; the loop is too coarse to see small gains

The operator asked Claude to do #18 in Piper's place. Everything below used the loop's own scorer and account, on the fixed set of 12 main hidden tasks (the review's "drawn fresh each session" caution was wrong and is struck out above). Cost $1.98. Nothing in the loop's state was touched.

Same-day re-score (all three versions side by side, one run per task):

Paired by task: v0→v3 +0.16 (t 1.1), v3→v5 +0.01 (t 0.1). Capability stopped rising at v3. The two later keeps are real cost cuts with no measurable quality loss: v5 matches v3 at 64% of its cost. The falling daily readings of v5 (0.743 on 10-09 down to 0.600 on 10-14) are run-to-run noise, not decay: the same code read 0.674 the same afternoon.

Why it flattened: the loop cannot see small capability gains. From 11 repeated runs of v5 on each task, one run varies with a standard deviation of 0.27. With 2 candidate and 2 incumbent runs on 12 tasks, a day's measured gain has a standard error of about 0.08, so a keep needs about +0.10 observed. A true +0.05 (the loop's own stated bar) is kept only about 26% of the time, and a true +0.10 about half the time. Cost cuts are nearly noise-free (t −5 to −7), so they are what gets through. Decided sessions per keep: 1, 1, 3, 5, 4, and 4 since. That is the opposite of compounding, and the measurement explains most of it. Zeros are not the main noise: 7% of 612 graded runs since 10-01 score 0, most of them valid answers no better than the simple baseline. A validity guard has about 3% headroom.

Generation record (missing piece 1, done): ~/rsi/results/generations.json on Firefly. Per version it records lineage, kind of keep, gain at keep, every loop reading as incumbent, wall time, cost per task, today's re-score and the loop's resolution numbers. It appears in her MY RECORD digest.

The lever, pulled (operator's go, live from the 10-15 session): a second look. If a candidate misses the smarter bar on its first 2 runs per task but shows at least +0.03 with t ≥ 0.5, the loop scores it 6 more times per task, plus 2 more same-day runs of the current version, and applies the same keep rules to everything pooled. The thresholds are unchanged; only the evidence grows. Estimated: a real +0.10 is kept ~95% of the time (was ~65%), a real +0.05 ~50% (was ~33%), and a no-effect change stays near the 10% false-keep rate. Replayed on the 18 past sessions it would have fired 5 times (09-28, 10-02, 10-03, 10-07, 10-12), costing about $4 and 4–5 hours of scoring each. Off switch SECOND_LOOK = False in rsi_loop.py; RULES.md and her daily start message say how it works. Readout: whether a capability keep follows within the next ~10 sessions.