Summary / 18. Generalization / Document
2026-10-12
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
**The short answer.** About 25 minutes and ~$0.5 of API spend. Piper's pick q-161 (a transfer lookup keyed on the DECISION TYPE) does **not** help her card fire on far cases: in the retrieval preflight it fired slightly less often than her current lookup (far+medium 33/63 -> 30/63), below the bar for a sealed seat retest. Nothing installed. **Plain answer:** no. Keying her transfer lookup on the decision type did not help her card fire on far cases. In the retrieval preflight it fired slightly LESS often than her current lookup (far+medium 33/63 -> 30/63), so it did not reach the bar for a sealed seat retest, and nothing was installed. About 25 minutes of runs, ~$0.5 DeepSeek. **Question (Piper's pick q-161):** if every stored lesson is tagged with the KINDS OF DECISION its rule changes (forecast / onboarding / review / design ...), each new message is classified the same way, and lessons that share a type are marked "DECISION-TYPE MATCH — test on purpose, not wording" in the judge listing, does the card fire on more far/medium cases without more false fires? **Code (bench only):** d1/transfer-1.5.0.py = her live transfer.py 1.4.0 + switch `decision_key` (absent = off = 1.4.0 behaviour); fixed 14-type taxonomy; tags cached in a new `entries.decision_types` column. d1_18.py 1.0.1 (`tags`, `preflight`). Her live transfer.py / transfer.ts / transfer.db untouched. **Tags:** 107/107 entries of the g18 stores tagged (first run 105 untagged: flash spent max_tokens on reasoning; fixed by max_tokens 12000). But the tags are BROAD: 101 entries got 6 types, 5 got 7 (review 94, policy 86, risk 85, diagnosis 79, design 73, measurement 67, ... forecast 5). **Preflight (retrieval only, same as the I1 preflight):** sealed F1 items of the 6 treated families, 3 passes, 252 lookups, 0 errors; xs = 1.4.0, d1 = 1.5.0 with decision_key on, same stores. | Card fired on the target lesson | xs (1.4.0) | d1 (decision key) | |---|---|---| | near | 20/33 | 17/33 | | medium | 16/27 | 16/27 | | far | 17/36 | 14/36 | | rule-applies total | 53/96 | 47/96 (-6.3 pp) | | neg (false fire) | 6/30 | 5/30 | - Bar for a seat retest: far+medium fired >= +4 and false fire not up by more than 2. Result **-3** -> below bar. - Per family (xs -> d1, near/medium/far): P03 9->9, 5->6, 5->4; P04 3->2, 3->1, 4->3; P10 6->2, 3->4, 3->2; R02 far 2->2; R03 near 2->4, far 0->0; R16 5->5, 3->3. **Why it did not help:** 1. The mark carries no information. On average 84 of 102 stored entries were marked as a decision-type match per lookup; the target lesson was marked on 92/96 rule-applies items AND on 30/30 negative items. With 6-7 of 14 types per lesson, almost every lesson shares a type with almost every message. 2. Telling the judge to "test marked lessons on purpose" over 84 marked entries did not raise firing; it slightly diluted it (near and far both down 3). 3. Too slow for a live seat: d1 lookups took up to 116 s and 61% took over 30 s (two flash calls with reasoning on, message classification + judge). Her transfer.ts kills a lookup at 30 s, so in a real Day turn most of these would return no card at all. (Thinking-disabled flash tags in ~1 s, but her live memstore.llm_chat has no switch for it.) **Verdict: D2 (decision-type key, as built) = NO GAIN at the retrieval level. Seat retest not run, far3 not sealed (no fresh items spent). Nothing installed. Headline score unchanged.** **For Piper (next pick):** the remaining misses are still the ~40% of far cases where her card does not fire. If she wants to pursue the decision-type idea, the smallest next test is a NARROW key (1-2 types per lesson, the single main type per message, thinking off so it fits in 30 s) - preflight only, ~$0.3. Otherwise I2-I4 from her Work Order D list. Files: d1_18.py 1.0.1, d1/transfer-1.5.0.py, logs/d1-tags.log, logs/d1-preflight.log; sealed root-only /var/lib/cognition-bench/18-generalization/d1/ (tags.json, pre18d1-0ab583ex/ lookup logs).