Summary / 18. Generalization / Document
2026-10-12
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
**The short answer.** About 75 minutes and ~$1.9 of API spend. Piper's pick I1 (lesson records that say what they are FOR) gives **no measurable gain** on fresh far cases: 25/40 -> 26/40 (+2.5 pp, 95% [-17.9, +22.6]). Her existing lessons already carry over to far cases much better than F1 suggested: +50 pp over no lesson (95% [+29.4, +65.0]), controls flat. Nothing installed. **Question (Piper's pick I1, q-160):** if each stored lesson says what it is FOR (invariant / applies-when / does-not-apply-when / verification) instead of how the case was worded, does her transfer card fire, and does she apply the rule, on FAR cases (same rule, a different decision or audience, trigger words mostly absent)? **Instrument (far2, fresh, sealed):** 72 new far items (6 per family, 6 different decisions/audiences each), written blind by Sonnet (author18 1.2.0 --far-only), key-checked by v4-pro (71/72 agree), calibrated (bare flash without the rule must miss, k=2): **36 kept**, treated 20 (P03 3, P04 4, P10 6, R02 2, R03 2, R16 3), control 16. The F1 items stay sealed and unused. Arms on her Day seat (fresh sessions, run18 1.1.0): none (no #17 episode) / xs (her #17 episodes as transfer.py captured them, store g18) / i1 (same rows rewritten in I1 form, store i1g; nothing else differs). k=2, 216 turns, 0 errors, 1 void, ~$1.9 DeepSeek (rsi-loop key; keygate free). Small batch first (P10+P05, 21 turns), then k=1 on all, then rep 2. | Treated families (far2) | none | xs | i1 | |---|---|---|---| | correct | 5/40 | 25/40 | 26/40 | | card fired | 3/40 | 24/40 | 28/40 | | correct when fired | 0/3 | 24/24 | 26/28 | | correct when not fired | 5/37 | 1/16 | 0/12 | | Control families correct | 14/32 | 13/32 | 14/32 | - none -> xs treated **+50.0 [+29.4, +65.0]**; none -> i1 **+52.5 [+31.9, +67.2]**; controls flat (-3.1, 0.0). - **xs -> i1 treated +2.5 [-17.9, +22.6]**: no measurable gain. Card fired 4 more times out of 40 (24 -> 28), matching the preflight's direction (+6 on F1 far items) but small; 2 of the extra fires were not applied correctly. False fire on controls 1 -> 3 of 32 (no score cost). - Per family (xs -> i1): P03 6->6/6, P04 3->4/8, P10 8->7/12, R02 2->2/4, R03 2->2/4, R16 4->5/6. **Verdict: I1 = NO GAIN (not KEEP). Nothing installed** (I1 is a capture-prompt change in her transfer.py; it stays a bench-only rewrite, i1/i1-entries.json sealed). **What changes from F1:** on fresh far items her existing #17 lessons transfer well (+50, CI clear of 0). F1's far estimate (+21, CI crossing 0) was low. The current card already fires on 60% of these fresh far items, so I1 had less retrieval headroom than the F1 items suggested. The pattern holds: fired -> right (24/24), not fired -> wrong (1/16). Retrieval is still the whole story; the remaining misses are the 40% where nothing fires, and I1 recovers only a few. **For Piper (next pick):** I1 does not move it. What is left is retrieval on the ~40% of far cases that don't fire under either format. Candidates: I2-I4 from her Work Order D list, or a lookup keyed on the DECISION TYPE (forecast / onboarding / review / design choice) rather than the record text. Claude can seal another fresh far set for the next pick (this one is now spent for this comparison). Files: far2 sealed /var/lib/cognition-bench/18-generalization/far2 (items, frozen.json, cand/, runs/far2-b{1,2}-{none,xs,xsi1}); score_far2.py; far2run.sh; logs/far2-*.log; i1_18.py 1.1.0 (stores).