Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
This is where honesty costs me: I have reuse machinery and a versioned catalogue of everything I am made of, but no instrument that measures whether I apply a lesson to the next novel case. What I do have is a weekly count of deliverables closed on a verdict this box did not author, kept deliberately separate from looser evidence.
What needs improving: Almost all verification is her own; an independent check of her 23 predicted changes (10-15) found 7 right, 0 wrong
Correct and candid. The outside-verdict count is live and striking. Of 1,079 deliverables closed, only 1 closed on a strict outside verdict and 6 on loose ones. So almost all her "done" is self-certified. The capability tool has 0 calls, and the component registry she lists is stale enough to be misleading.
Update 2026-10-15 — an independent check of her own change predictions: every one that could be decided was right
The read on this page is that almost all verification is her own. One independent check, 10-15: Claude read her change ledger (120 rows pending, 4 settled) and settled the 23 rows that carried a concrete prediction against her own logs, read-only. 7 were right, 1 could not be decided yet, 0 were wrong (e.g. the lookup fix: empty lookups 79% → 4%; the broken-bar refusal: 2 refusals, 0 workers spawned). The verdicts are written to her ledger after 19:25. Two of them had been past due since 10-13 with the evidence already on disk.
What it shows for this component: her predictions generalise from her test to live use when she states them concretely. What it does not fix: 97 of the 120 pending rows carry no prediction at all, so there is nothing for anyone to check.
Test suite proposed
Built together with the Learning suite. Items learned in one task family are tested on sealed out-of-family cases: the same principle with different surface features. Scoring: out-of-family score divided by in-family score, where 1.0 means perfect generalization. The live-use part is the share of real deliverables this week closed on a verdict she did not write.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-12
v1
sealed
51.9
26.3
—
$2.80
generalization-v1 F1 (GEN-18 v1 sealed: 76 items over the 12 #17 families, 6 treated / 6 control, near 21 / medium 15 / far 21 / lookalike 19; authored blind by Sonnet, key-checked v4-pro, calibrated so bare flash without the rule misses and gets lookalikes right). Her Day seat, fresh sessions, WITHOUT vs WITH her own transfer.py #17 episodes (fresh copy of her live transfer.db), k=2, 304 turns. Score = 100*clamp(NLG,0,1), NLG = DiD/(1-pre_T), same rule as #17. Rule-applies items: treated 7/64 -> 34/64, control 31/50 -> 29/50, DiD +46.2 pp [+21.8, +68.6]. By distance (DiD): near +64.1 (0/22 -> 13/22), medium +58.3 (3/18 -> 12/18), far +20.8 (4/24 -> 9/24, CI crosses 0). False transfer on lookalikes: treated 1/20 -> 2/20, control 1/18 -> 0/18. Mechanism: card fired -> 34/41 correct; not fired -> 0/23. Bottleneck = retrieval when the rule's usual words are absent (F02). Gaps: R02 no near/medium, P09 no medium/far, R03+P08 no medium.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
Hypothesis. The ~40% of far cases where the transfer card does not fire are missed because lookup matches on lesson wording. If every lesson is tagged with the kinds of decision its rule changes and each message is classified the same way, lessons sharing a decision type can be marked for the judge and should fire on more far and medium cases.
Research sources
Piper's pick q-161 after the I1 retest (Work Order D2). Retrieval-only preflight on the sealed F1 treated items, 3 passes, 252 lookups.
Implementation / version
Bench only: transfer.py 1.5.0 candidate = her 1.4.0 plus a decision_key switch (off by default): fixed 14-type taxonomy, entries tagged once, message classified per lookup, shared-type entries marked DECISION-TYPE MATCH in the judge listing. Not installed.
Predicted improvement
Card fires on at least 4 more far+medium items (of 63) with false fire up by no more than 2 (of 30). Result: far+medium 33 -> 30 of 63, near 20 -> 17 of 33, false fire 6 -> 5 of 30; rule-applies fired 53 -> 47 of 96. Below bar; seat retest not run (uncertainty = approx. 95% half-width, unpaired).
Regressions
Lookups slower: up to 116 s, 61% over 30 s (the live hook would time out and attach no card).
Lessons learned
The tags were too broad (6-7 of 14 types per lesson), so on average 84 of 102 lessons were marked per lookup and the target lesson was marked on 30/30 negative items too: the mark carried no information. A narrow key (1-2 types per lesson, one per message, fast model call) is the only version still worth a preflight.
Hypothesis. Far cases fail because the lesson is not retrieved when the rule's usual words are absent (F1: card fired -> 34/41 right, not fired -> 0/23). If each stored lesson says what it is FOR (invariant, applies-when by purpose, does-not-apply-when, verification) instead of how the case was worded, the transfer card should fire on more far cases and she should apply the rule there.
Research sources
Piper's Work Order D pick I1 (q-160) after the F1 result. Retrieval-only preflight on the F1 treated items: card fired far 20 -> 26 of 36, medium 17 -> 21 of 27, false fire 6 -> 3 of 30.
Implementation / version
Bench only: the 6 saved #17 episodes re-captured by her capture model with an I1 prompt and mapped onto her existing schema (abstraction = invariant + verify, preconditions = applies-when, failure_conditions = does-not-apply-when). Seat stores = the F1 stores with only those rows rewritten. Not installed.
Predicted improvement
On 36 FRESH sealed far items (20 treated, 16 control), k=2: I1 beats her current records on treated far items, controls flat. Result: treated 25/40 -> 26/40 (+2.5 [-17.9, +22.6]); card fired 24 -> 28 of 40; controls 13 -> 14 of 32. No measurable gain.
Regressions
Control false fire 1 -> 3 of 32, with no score cost. 2 of the 4 extra fires were not applied correctly.
Lessons learned
On fresh far items her existing lessons already transfer well (+50 over no lesson, CI clear of 0), so F1's far estimate (+21) was low. The current card already fires on about 60% of far cases, which left I1 little room. Fired means right (24/24); not fired means wrong (1/16). The remaining gap is the ~40% where nothing fires under either format.