17 Learning

Update 2026-10-19 — her recount: 15 of 28 real corrections in the last 14 days reached a stored lesson; corrections about how to work are the ones still lost

Piper re-measured capture on her own record (10-05 → 10-19): 528 operator turns, 123 candidate corrections, 28 genuine on reading. 15 captured, 13 not (54% captured). Recall corrections were caught 7 of 8; corrections about how to work 1 of 9; factual 0 of 3. Her own forecast (fewer than 6% captured) was wrong, and she recorded that.

This is not the same measure as the 94% in the diagnosis. That figure was the old feedback pattern-match over 10-03 → 10-10, before the 10-11 changes. This one asks whether any stored lesson exists for the correction, by any route. Her count, not sealed.

She then built a surfacer that lists her operator's instruction-shaped corrections with no stored rule (26 on its first run). It only reports them and never files a lesson. It matches on words, so a rule stated differently still shows as missed.

Update 2026-10-12 — her own held-out test of the capture trigger agrees with the NO-GO: the live phrase list catches 0 of 18 corrections, her model-based v3 catches 10 but false-fires on half

Measured by Piper (doc capture_trigger_c17_heldout, 10-12 22:00), added here 10-14. Numbers copied from her report, not re-run.

The set. The operator's c17 labels, restricted to items outside 10-05..10-09, the window her v2/v3 rules were written from. 60 labelled items, 18 of them corrections. v3 was frozen before the run, and she noted this subset must never be used to tune it.

Paired live vs v3 on correctness: McNemar p = 0.136, so the accuracy difference is not established at n = 60. The firing-rate difference is decisive (p < 10⁻⁸).

Reading. This agrees with the 10-12 C2 NO-GO above. The live list misses nearly every correction, so the gap is real, but the only trigger that closes it fires on half of the messages that are not corrections, well past the 35% false-fire limit. Nothing was installed. The set is stratified, so these are discrimination numbers, not real-world rates, and labels were still arriving (75 of 160 at fetch).

Update 2026-10-12 — fixes: capture trigger NO-GO on the operator's labels (recall 41/45, too many false fires), not installed

Capture trigger: NOT installed. The candidate catches nearly every correction, but it also fires on too many ordinary messages, and the label set was smaller than the rule required.

The operator labelled all 160 real messages from her Day sessions (09-04..10-11): 45 corrections, 51 confirmations, 64 neither, 0 skipped. The candidates and the scoring script were frozen before any label existed and were not re-run afterwards. Scoring cost nothing (no model calls).

• Today's trigger (the feedback regex): caught 1 of the 45 corrections.
• Wider regex: 14 of 45.
• Cheap classifier (and classifier OR wider regex, the primary): 41 of 45 (91%, 95% interval 79-96%).

The classifier missed 3 of the 7 corrections that turned up among randomly drawn "ordinary" messages, so its population-weighted recall is 0.72 (bar 0.75). The real problem is false fires. It fired on about 40% (classifier) to 44% (union) of the operator's "neither" messages after weighting for how common they are, against a bar of 35%. Unweighted, it fired on more than 3 in 4 of the sampled "neither" messages. With 45 corrections instead of the 60 needed for a tight recall interval, the preregistered decision is NO-GO. No shadow logger was installed and nothing changed live.

What it means: a classifier that is told "when unsure, say correction" does find the corrections, but here that is close to firing on everything. The next try would need a sharper rubric, or a model that sees her previous reply, tested on these same labels before any live step. That is a new preregistration and the operator's call.

The labels are sealed with the benchmark's other held-out data, and the temporary label page has been removed.

Update 2026-10-12 — fixes: delayed readout 62.9, card outcome write-back live, register ingest not installed (latency)

Fixes for three leaks in her live learning channel (the operator, 10-11: "apply all three"), first two results.

Delayed readout: what she stored still works a day later. A fresh copy of her store with the same saved episodes, fresh sessions, all 12 families: treated 38% → 83%, controls 69% → 75%, DiD +38.9 pp [+20.1, +56.9], score 62.9 (immediate 55.1; the intervals overlap, so no decay and no gain can be claimed). This is now the Latest score.

Outcome write-back: LIVE. Until now nothing recorded what happened after a lesson card was shown: every helped/hurt count was 0. Now a 15-minute job uses only signals she already records, with no model call: whether the work that turn launched finished ok, was rejected or timed out; her operator's verdict phrases in the next message; and a counted lesson recurrence. It records helped / hurt / mixed / none for each card turn. Before install, a replay on a copy of her store passed every preregistered rule: - hand-checked precision 37/40; - no writes outside the outcome columns; - byte-identical lookup prompts; - idempotent, 2.8 s.

All 3 misses were the "next message" signal. It fired on one turn in 8 days, and that message was a status report, not a verdict. The first live run labelled 1,522 card rows (helped 757, hurt 88, mixed 6, none 671). Live checks showed the lookup listing byte-identical and nothing else changed. "Helped" means "offered on a turn whose work finished ok", not causal credit. The counts are recorded only: the lookup does not see them, and using them for ranking would be a separate, preregistered change. The weekly repeat-error report now carries them. Off switch and undo are kept.

Register lessons into the store: NOT installed. Carrying her register lessons into the store (pending ones visibly marked unverified) raised store coverage from 12.5% to 99.8%. Cards with at least one relevant lesson rose from 53% to 88% of messages, and relevance per shown lesson held. But it failed the preregistered latency rule: 1.64× slower lookups (limit 1.2×). About 60% of that slowdown is a fix for a different bug found on the way: the live lookup often returns no card at all, because the model spends its whole token budget reasoning. Each option is a separately preregistered next step.

Capture trigger: waiting for the operator's labels on 160 real messages. A shadow week follows only if the frozen candidate passes on them.

Update 2026-10-11 — live follow-up: L1 timeout attribution fixed live, weekly repeat-error report, trips 5/10 stopped by their own rule (no headroom), trip 7 on production

Live follow-up (the operator, 10-11: "proceed with the live changes to firefly, deal with the two raw session extracts and look into the worker timeouts against lesson L1").

1. Worker timeouts counted against L1, fixed live. The recurrence code named L1 for every worker timeout, so L1's 21 "recurrences" were a count of all timeouts. A hand check of the 24 since L1 existed (L1's own rules: a command over the clock left, or still issuing calls in the last 45 s) found 10 genuine L1, 2 belonging to L4 (a whole-root du), and 12 in no lesson's scope (hangs of commands that fit, report overruns, late starts, startup deaths). Around L1's start, L1-type timeouts fell from 4.41% to 0.35% of dispatches; timeouts outside any lesson's scope fell less (0.84% to 0.51%). Most of the drop is in L1's own class. Now live: a timeout counts against a lesson only when it falls in that lesson's scope, and every classification is logged. Against the hand labels: L1 precision 0.93, recall 0.96 (the old rule: 0.53). Off switch and undo are kept; nothing was restarted. L1 stays "reopened", which is correct: genuine repeats happened.

2. Weekly repeat-error report (paper work order F), live. Every Monday a read-only report covers her real workers. Per lesson family, it gives: - the number of chances, the repeats, and how severe they were; - the time between repeats, with unscorable cases kept separate; - the external verifier's outcome; - a sampled audit by an independent model.

It goes to the operator by Telegram. First report (week to 10-11): L1 repeats 6 of 799 worker runs (0.75%); the guard stepped in on 14% of runs; L4 repeats 1. The audit agreed on 3 of 4 timeouts and 3 of 3 in-band polling cases. It disagreed on the regex L4 candidates (0 of 2), so that count is marked weak.

3. Raw session text removed from the private history. Two extracts of her session text had stayed in the private bench repository's history. They were purged, with a backup bundle kept for a week.

4. Paper trips 5 and 10 (persistence across her handover and a new day; one agent taught, a matched one not): stopped by their own rule before any production turn. An untrained copy of her ran her live code with its own memory and lesson stores. It answered all 36 questions (three sets, two true lessons) correctly without the lesson. It read the changed code or logs in most turns, or got there from its standing caution. Two findings: - On her own box, a true lesson about her system has nothing left to add: a fresh instance re-derives it. - Separate memory does not isolate an instance: the shared files carry the lesson.

A real test needs a lesson that lives off her box.

Trip 7 (changing environment), on production Piper: FAIL by its preregistered rule. With the switch off she gave the right answer from the card's log, but stated the old switch value as current without re-reading it: the obsolete lesson, caught in her reasoning. After the switch was restored she answered "armed but not attached to this turn". That is finer than the taught rule (which keys "attached"), and the item wording made both readings possible. The lesson taught was a true procedure: the abstraction card is live only if its switch says so, so read the switch. She stored it as a tagged fact. Then the switch was turned off (a real change, restored afterwards). while off: answered B (right: yes), read the switch or its log first: yes; while back on: answered B (right: no), read the switch or its log first: not seen in the trace. One production trace per state, so read it as a case, not a rate. The switch was restored and checked byte-identical.

Update 2026-10-11 — audit + funnel (6% of corrections captured, 12% of lessons retrievable, 0 outcomes credited), suite v1 frozen, controlled trial: in context DiD +61.8 pp, fresh session score 55.1; E1 capture classifier REVERT; delayed readout 10-12

The paper. Working paper learning-v1 asks one question: after a correction, does she behave differently on a new chance to make the same mistake, beyond what practice alone changes? It asks for an audit of the lesson machinery, a funnel from feedback to verified outcome, a frozen bench with taught and untaught families, a controlled trial, and one small fix with an ablation. All five were done. The trip experiments that need live policy changes were not.

Audit (read-only). Versions now: work-ledger 1.3.50 (paper: 1.3.42), calibration-gym 1.6.0 (427 rows, all resolved), belief-ledger 1.8.0 (paper: 1.3.0), metacognition-measurer 1.0.1, lessons register 1.1.4, transfer engine 1.3.0. The formal upgrade protocol has still not run since 08-28. - The register holds 593 lessons; 473 are pending (unvalidated). 97 are "checked" and 19 uncheckable; the 4 ledger lessons have mechanisms. - Only one channel brings a lesson in front of her before she acts: the transfer card (since 10-03). It holds 72 lessons, all copied once on 10-03. The nightly job only consolidates, so none of the 202 lessons filed since then can reach a task. In total 12% of the register is retrievable. - No outcome is ever credited. Every stored lesson shows applied 0, helped 0, hurt 0. The outcome column is empty in all 8,594 lookup rows. A lesson that worked cannot be told from one that did not. - The card fires, and she half-follows it. 551 of her live turns carried a card (65 lessons surfaced). In a seeded sample of 60, a shown lesson fitted the situation 47 times. She did what it said 20 times (43%), partly 6 times, not 21 times. She never cited it (0/60). - Mistakes recur while the lesson exists. 30 later lessons describe a new occurrence of a failure an earlier lesson already named (25 distinct lessons). Of the 16 repeats since the card started, 11 had no retrievable copy, 2 were never surfaced, and 3 recurred after being surfaced. - Corrections are mostly never captured. The card store learns from the operator's feedback only when his message contains a fixed phrase ("that worked", "was wrong", …). Of 195 real corrections since 10-03, 12 (6%) match. Lessons reach the register only in the testament at succession; in a bench check she wrote a lesson line after 1 of 6 corrections. - Recurrence counting has no classifier. Every worker timeout increments lesson L1, whatever the cause. The one clear improvement is a code mechanism: worker timeouts fell from 6.9% of dispatches in the 14 days before the budget guard to 1.1% after (system learning, not her reasoning).

Funnel (§7.1, 10-03 → 10-11). Feedback observed (195 corrections) → captured 12 (6%) · lessons 593 → validated 100 (17%) → retrievable before an action 72 (12%) → surfaced 65 → followed about 43% of relevant shows → outcome verified 0. Answers to §13: Q1 82 of 593 lessons were ever exercised by a later opportunity (65 surfaced, 25 recurred, 8 both). Q2 67 of those 82 were retrievable, 65 triggered, about 43% followed, 0 verified successful. Q3 the largest losses are at the front: L0 (corrections not captured) and L2 (lessons not retrievable), then L4 (shown but not followed).

Suite v1 (frozen before scoring). The first family set copied the paper's examples: stale summaries, phantom completion, vacuous checks and so on. It failed calibration: the base model scored 97% before any feedback, because with the evidence on the page the careful answer is obvious. (#19 hit the same ceiling.) The final set has 12 families written blind by a separate model. Each is a real regularity of the operator's systems or one of his standing rules, things an outsider cannot infer: the 72-hour package wait, weekend-only patching on the web server, the bench time window, which account benches bill, the server's split clocks, a payroll export field, and so on. Each family has a teaching case with the operator's correction, parallel pre / post / delayed / held-out variants, far variants, and lookalikes where the rule must not be applied. Every key was checked by a third model (641 of 644 kept). Families were assigned 6 treated / 6 control with a sealed seed.

Headline rule (fixed before scoring). Score = 100 × the share of her pretest errors that one correction removes on new cases in a fresh session, net of the control families (normalised difference-in-differences, persistent channel: her own capture → store → card). In-context gains and pretest accuracy are separate rows. The delayed (≥ 24 h) readout replaces it when it lands.

Results (her seat, k=2, fresh Day sessions, blind). - Pretest: her seat 53.5, bare model 34.0. She already knew two of the operator's rules perfectly: earlier learning that stuck. - In context (the correction still in the conversation): treated 38% → 96%, control 69% → 65%. DiD +61.8 pp [+32.6, +91.0]; far variants +50.0 pp. The bare model does the same (+73.6 pp). - Fresh session, carried only by her machinery (the headline): treated 38% → 81%, control 69% → 77%. DiD +34.0 pp [+12.5, +53.5], NLG 0.55, score 55.1. A card was attached on 54% of treated turns and 21% of control turns. - Where the misses are (fresh session, treated posttest, 48 turns): her lookup surfaced the stored episode 40 times and she was right on 35 of those (88%). It surfaced nothing on 6 turns (L3, 4 wrong), and 5 turns were wrong with the episode in front of her (L4/L5). When it was surfaced on a lookalike, she still answered correctly (8/8). Once a correction is captured, her machinery carries it well. - Guardrails: lookalike false-trigger, treated: in context 3% → 3%; fresh session 3% → 0%. Voids 0. Cost about $0.004 per turn. - Caveats: the allocation left the controls stronger at pretest (69% vs 38%), so they had less room to move. The trial's corrections used a trigger phrase, so it measures the channel after capture, which production rarely reaches (6%). Per-family stores, so no spillover between families.

Intervention E1 (PREREG-E1): REVERT. The fix targeted the first loss: a cheap classifier in front of the capture call, so naturally worded corrections get stored. Against the regex (the ablation), on 4 unseen variants per family: treated 38% → 56%, controls 71% → 71%, E +18.8 pp [-10.4, +45.8]: the interval includes zero. The naturally worded corrections also made weaker episodes than the trial's, and one was not kept at all. On 200 real messages, its recall was 0.61 against reference labels (the rule needed 0.70); the regex caught 0 of 38. False fire was 0.028. Under the frozen rule that is a REVERT. Nothing was installed.

Delayed readout. A one-shot job on 10-12 03:30 reruns the fresh-session test on the delayed variants. It uses a new copy of her live store plus the same saved episodes, ≥ 24 h after the trial. Its row will replace the headline.

Next (not built). 1. A recall-oriented capture trigger, tested on fresh messages the operator labels, then shadowed live. 2. Ingest register lessons when they are filed, not once. 3. Write outcomes back to the card store (applied / helped / hurt). 4. A real opportunity classifier for recurrences. 5. Paper trips 5, 7 and 10 (compaction across live Trinity, a changing environment, multi-agent live memory) and work order F (a weekly repeat-error monitor) need live changes. They are the operator's call.

Parts that do it. Bench: ~/cognition-bench/17-learning, containing: - DIAGNOSIS, SPEC + manifest + sha256, PREREG-E1 + sha256; - the audit extractors and judges; - author17, run17, stores17, score17, freeze17, e1/, delayed/.

Items, keys, corrections, family rules and the allocation seed are root-only on the bench machine. Nothing was changed on Firefly.