Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Update 2026-10-03 — length gate removed (transfer.py 1.3.0): cross-session A2 52.5 → 75.0, held-out C2 100
What changed.
- transfer.py 1.3.0 has no length gate. It skips only an empty message or a bare acknowledgement ("ok", "thanks", "go ahead"), and every skip is logged, so a silent miss can't happen again.
- The header's board-truncation line ("… N more line(s) not shown") is dropped along with the header.
- The bench now cleans the copied header to its "[" lines, so no run depends on what her board showed at that moment.
- Stored episodes were recaptured under 1.3.0.
- The change went live at about 07:55 with no restart (the engine is spawned per call).
- A smoke run of the three A2 B1 questions went from 0/6 to 3/3, each with a lookup that applied the right episode.
Results on 1.3.0 (k=2, same calibrated items, arms and seat as before). Each cell is the score on target items, with TG against placebo in brackets.
| Arm |
A2 on 1.2.0 |
A2 on 1.3.0 |
C2 on 1.2.0 (header artifact) |
C2 on 1.3.0 |
| control |
15.0 |
10.0 |
26.9 |
23.1 |
| placebo |
12.5 |
15.0 |
26.9 |
19.2 |
| transfer (same conversation) |
92.5 |
95.0 |
92.3 |
100 |
| oracle |
97.5 |
95.0 |
100 |
100 |
| xs-control |
12.5 |
12.5 |
23.1 |
19.2 |
| xs-placebo |
7.5 |
10.0 |
19.2 |
23.1 |
| xs-transfer (next session, engine only) |
52.5 |
75.0 (TG +62.5) |
88.5 |
100 (TG +83.3) |
| foils, xs-transfer |
58.3 |
75.0 |
83.3 |
100 |
Reading it.
- The short questions now carry. On A2, xs-transfer is 100 on B1 and right on all seven short targets that scored 0 before. Cross-session transfer reaches B5 on both sets.
- On the held-out families, the next session now matches having the episode in the conversation: 100 against 100. The unrelated-episode control is unchanged (23.1), so the gain is the matching.
- What A2 still misses is two families (R1 and R2) at B2–B5. Most are split 1/2 across the two repeats, so the lookup works there but the far mapping is not reliable yet. One paired item went the wrong way (negative-transfer rate 2.5%), against 0 before.
- Cost: about 4 s per message for the lookup (the store listing is prompt-cached), not the 10–13 s first estimated. The two runs cost $2.17 and $0.73.
Live, first two hours on 1.3.0: 18 lookups on operator messages, 4 with a match. There were 8 logged skips, all header-only system turns; no acknowledgement has been skipped yet. Still 2 provisional episodes and none promoted, so live capture is now the open question, not lookup.
Parts that do it (changed)
core/transfer.py 1.3.0 (backup transfer.py.bak-claude-transfer-<stamp>)- Bench
run19.py 1.2.0 (header cleaned to "[" lines; --keep for tracing)
Update 2026-10-03 — benchmark built; transfer engine: cross-session transfer 12.5 → 52.5, held-out families 23.1 → 88.5
Built from the operator's two documents. These are the plan, "Transfer as an Explicit Cognitive Subsystem in Piper", and the research report, "Transfer in LLM-Based Autonomous Agents". The plan's hard rule shapes the whole bench: transfer is a counterfactual. TG = the score on target B after experience A, minus the score without it, with the model, seat, budget and target held fixed. Without a no-exposure comparison there is no transfer claim.
The bench.
- Lesson families. Each has a source episode A (a task, a correct solve, and the operator's verified-outcome feedback), and targets B1–B5 at increasing distance from it:
- B1, very near: the same system;
- B2, near: another language or tool;
- B3, medium: shell, SQL or config;
- B4, far: non-code technical;
- B5, very far: non-technical.
Each family also has a foil: it looks like A, but applying the lesson gives the wrong answer.
- Arms:
- control (nothing before the target);
- placebo (another family's episode, same length and format);
- transfer (this family's episode, earlier in the same conversation, never named as a lesson);
- oracle (the lesson stated outright, the upper bound);
- three cross-session arms. The episode is not in the conversation; it is only in a scratch copy of her transfer store, so only machinery can carry it: xs-transfer, plus xs-placebo and xs-control as its controls.
- Scoring is objective:
- numbers within a tolerance;
- exact words from a closed option list;
- affected sets by F1;
- "attack the check" answers are executed against the check.
A free model extracts the answer and never sees the truth; it was checked 30/30 against Sonnet.
- Seat mode. Her Day seat forks the clean 09-30 snapshot. It has the real header, tools and extensions, and calls are billed to a separate bench account.
- Held out by whole family, not by item (report §11.1). Set C2's three families were sealed before any engine result existed.
v1 was retired at once: the base model already knows those lessons. The first suite used familiar engineering lessons: guard an optional value, bisect the change set, two-point calibration, blast radius, attack the check, trace copies to the source.
- The bare model scored 100% on every target with no exposure at all, even on unflagged, buried versions.
- Its only signal was negative transfer on a foil. With the copies lesson in context she counted copies as independent sources (placebo 2/3, transfer 0/3).
v2 measures what transfer is actually for: regularities of her environment that no model can know in advance. Examples: how one system's log timestamps are offset, or a vendor's invoice quirk. There are six families (R1–R6, sets A2/B2), and held-out families R7–R9 form set C2. The bare-model calibration pilot kept items where control has headroom (≤ 0.75) and the lesson can matter (≥ 0.25).
Baseline (A2, before the engine). In the same conversation she transfers well:
|
Score |
| control |
2.5 |
| placebo |
7.5 |
| transfer |
80.0 |
| oracle |
97.5 |
That is TG +72.7, positive out to B5, with no negative transfer. Across sessions she had nothing. A lesson learned yesterday reached today only if it happened to be in a memory or lessons row. The cross-session arms equal control by construction.
The fix: a transfer engine (the plan's online and offline loops).
- transfer.py + transfer.ts. On every message the engine always retrieves; the plan's rule is that the gate decides whether to use a memory, never whether to look.
- One model call restates the problem's structure in domain-free terms.
- It then judges every stored experience against that structure, MAC/FAC style, not nearest-neighbour on surface words.
- For each candidate it checks preconditions and "not if" conditions, maps source roles to target roles, and gives a verdict: apply, consider or reject.
- She gets a hidden card with the matches and at most one near-miss that must not be carried over (the negative-transfer guard). The card holds facts only, no instruction.
- Capture. When the operator reports how her last answer turned out ("that was it", "still broken"), the problem, answer and outcome become an episode. Each episode stores:
- a domain-free abstraction;
- preconditions and failure conditions;
- the raw specifics (named systems, vendors, values). Specifics carry near transfer; the abstraction carries far transfer.
- Consolidation (nightly). Episodes are promoted to a lesson only with ≥ 2 that agree and ≥ 1 verified outcome. The store was seeded with the 72 checked rows of her lessons register.
- A bug the bench caught. 1.1.0 skipped short questions as "simple lookups". That is exactly the noticing failure the plan warns about, and it killed the first run. 1.2.0 removed that gate, and a learned specific naming the same system now counts as a match.
Results on 1.2.0, k=2. Each row is the score on target items, with TG against placebo in brackets.
| Arm |
A2 (dev families) |
C2 (held-out families) |
| control |
15.0 |
26.9 |
| placebo |
12.5 |
26.9 |
| transfer (same conversation) |
92.5 (TG +81.5) |
92.3 (TG +65.0) |
| oracle |
97.5 |
100 |
| xs-control |
12.5 |
23.1 |
| xs-placebo |
7.5 |
19.2 |
| xs-transfer (next session, engine only) |
52.5 (TG +38.5) |
88.5 (TG +60.0) |
| foils, xs-transfer |
58.3 |
83.3 |
Reading it.
- Cross-session transfer exists now, and it holds on families never used in development. On C2 the engine takes a fresh session from 23 to 88.5, nearly as high as having the episode in the conversation (92.3).
- An unrelated stored episode does nothing. xs-placebo stays at or below control on both sets, so the gain comes from the matching itself, not from having a store at all.
- No negative transfer on targets in any treatment arm. Foil accuracy did not fall either: xs-transfer is level with xs-control on both sets (58.3 = 58.3 on A2, 83.3 = 83.3 on C2).
- The weak spot is A2's nearest band, and the trace found the cause: a length gate left in the engine. transfer.py 1.2.0 still skips any message under 120 characters, after header lines are dropped, without a lookup or a log line. Seven of the 20 calibrated A2 targets are that short (xs-transfer scored 0 on all seven), including all three B1 questions ("Sefple weighed another pallet: 333.8 kg. what does it actually weigh?" is 69 characters). A rerun with the scratch stores kept showed no lookup and no ledger row for any of them. She never saw a card, so she searched for the raw record instead. The stored episodes themselves were fine: they hold the right specifics.
- C2 dodged the gate by accident. The bench copies her real message header from her latest live message. At the C2 start (06:44), that header ended with a stray board line ("… 350 more line(s) not shown") that does not start with "[". It was counted as part of the question and pushed nearly every C2 item over 120 characters, so lookups ran. Two of those short C2 items, re-run with a clean header, got no lookup and failed. C2's 88.5 is therefore what the engine does when it looks; A2's 52.5 is closer to the engine as it stood live, where short operator messages get nothing.
- Fixed in 1.3.0, and both sets re-run: see the next update.
- C2 is small: 13 target items and 3 foils at k=2. Band numbers there move by one item. The overall gap (23.1 → 88.5) is far outside that.
What this does not show.
- Live use. The engine went live at the 00:30 router restart. By 07:20 it had made 33 live lookups (5 with matches) and captured 5 outcomes, giving 2 provisional episodes. None has been promoted, and no live answer has yet been checked against a stored rule.
- Capture quality in the wild. In the bench, episodes enter through the same capture code but from clean planted feedback. Real feedback is vaguer.
- One model only (deepseek-flash).
Parts that do it (added)
core/transfer.py 1.2.0 + core/transfer.ts 1.0.0 (Day extension), store state/transfer/transfer.db (72 seeded lessons), log state/transfer/transfer.jsonl, mode state/transfer.json (on/on), nightly transfer.py consolidate (fcron)foundation_router.py 1.9.99 (live 2026-10-03 00:30); bench seat 1.15.0- Bench
~/cognition-bench/19-transfer: gen19.py (v1, retired), gen19v2.py (v2), score19.py, run19.py 1.1.1, calibrate19.py, SPEC.md; sealed sets A/B/C and A2/B2/C2 root-only on the bench machine