Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Update 2026-10-10 — audit, ABST-11 v1 built, and the abstraction card: 76.7 → 89.2 (her seat was below its own bare model, 87.8) — LIVE
The paper. The working paper (abstraction-paper-1) is right about the main thing: the unit of learning is not a sentence like "test validators carefully" but a rule that says what must be true, when, how to test it, and where it must not be enforced. Its "audit first" order is right too, and its first concrete request was followed: audit, build a sealed benchmark from externally authored cases, freeze and run her baseline, make one change aimed at the largest measured failure, rerun the same held-out set, and keep or revert.
Audit (paper §1 / U0, read-only, frozen in audit/audit-2026-10-10.json).
- The lessons table holds 583 rows. The register has 463 pending, 97 checked, 19 uncheckable; the budget-guard ledger has 3 enforced and 1 reopened. The 09-30 figures (339, 349) were older snapshots of a register that grows by about 30 lessons a day. 274 pending lessons are more than a week old.
- "Checked" means a red-path test exists and its suite ran, on synthetic input. The control suite never touches live state by design. No record links any lesson's check to a firing on real work, so the paper's "evidence-backed lesson ratio" can only be stated as 96/560 on synthetic runs; on real work it is not instrumented.
- The 97 checked lessons rest on only 26 distinct checks. 36 of them are credited to one generic check (controls.fire_tests), which proves the bridge refuses a bad envelope, not that she stops filing one. 52 of 97 checked rows carry no evidence. One rests on a check that is red right now (the control suite's 00:12 run: 2 of 57 failing).
- The top rung is still empty. There is no skill table, and skills-distill exists nowhere on disk. The 7 pi skills are hand-written (08-24 to 09-01), and none was distilled from a lesson. contemplative_cycle.py and mutate.py exist but are not scheduled.
The bench (ABST-11 v1, paper §5 rounds 0–3).
- Families: 12 sealed incident families, each one hidden rule over structured records (ordering, identity, authority, units, thresholds, lists of events), with adversarial near misses and documented exceptions. They were authored by a different model (Claude Sonnet) than the one building the bench and the intervention, which never read them. The paper's public example families were excluded because she may have read the paper. A third model (DeepSeek v4-pro) re-labelled a sample from each rule alone: 144/144 agreed.
- What she gets: per family, 5 records filed as problems and 2 fine ones, worded as the operator's request. She must give the rule, its scope and exceptions, rival explanations, and a Python check(record).
- Scoring: her check runs in a sandbox (no network, no files) on 20 sealed cases per family: violations, ordinary fine records, near misses that look alarming but lack the cause, and boundary cases. 6 of the 20 come from environments the demos never showed. It is also scored on label-keeping transformations (renames, time shifts, reordering) and on a mutation battery: each fine record minimally changed into a violation, which the check must flag.
- The score is the paper's rubric for the parts this suite measures (generalization 30, enforcement 20, scope 15, metamorphic 10, audit/economy 5), rescaled to 100. Chance scores 0, and a constant check scores 0. The skill and delayed-transfer rungs (20 of the paper's 100 points) are not measured yet.
Baseline (round 0). Her Day seat scored 76.7, below her own base model with a one-line prompt (87.8). On the dev families 11 of her 12 errors were false positives. Her rules carried conditions no example showed: a 1-unit currency tolerance that let a 1-unit over-refund through (mutation kill 0.00 on that family), and an extra "must also hold it" requirement. Documented exceptions went unscoped. On the sealed set she also answered "can't tell" on 51 of 480 clear cases, and one run produced no check at all.
The change (one). abstraction-card.ts: when a message asks for the general rule behind a set of cases, one hidden card is attached. It says:
- contrast the problem cases against the fine ones;
- write 2–3 rival rules and separate them with a concrete case;
- leave out any condition the evidence cannot decide;
- take the exact edge the examples show;
- read only the fields the rule needs, with no invented slack;
- before answering, run the check on a fine case changed just enough to break the rule.
It names no domain, field or rule.
Result (PREREG-C1: locked and hashed after dev, before any sealed seat output; 12 families × 2 runs per arm):
|
Control |
Card |
Δ (95% CI, paired family bootstrap) |
| Core score |
76.7 |
89.2 |
+12.4 [+4.5, +20.8] |
| Hidden balanced accuracy |
0.860 |
0.950 |
+0.090 [+0.031, +0.158] |
| Near-miss balanced accuracy |
0.862 |
0.938 |
+0.076 [+0.014, +0.145] |
| Unseen-environment BA |
0.826 |
0.958 |
+0.132 [+0.049, +0.222] |
| Mutation kill rate |
0.77 |
0.88 |
+0.113 [+0.037, +0.204] |
| Coverage (answered) |
0.89 |
1.00 |
abstentions 51 → 2 |
| Cost per family |
$0.046 |
$0.047 |
flat |
All guards hold, and the paper's own promotion gate (BA +0.05 with the CI excluding 0) passes. 9 families went up, 2 were flat, and 1 went down (−11.2). The two runs of each arm differ a lot (card 93.6 and 84.7), which is why k=2 was preregistered. KEEP, installed live 10-10 01:03 (router 1.9.110).
What this does not show.
- The card works on Day turns that ask for a rule from cases. Those are rare in her real traffic, and the lesson register is filled mostly by workers and testaments, which it does not reach. Live fire rate: not yet read.
- The ladder's top half is untouched: checks on real work (not synthetic) and skills distilled from rules. Those are the paper's rounds 4–6 (skill artifact, delayed unprompted reuse, live shadow) and U1/U4 (typed rule records, a skill registry).
Next.
1. Read the live card log (state/abstraction_card.jsonl) after a week.
2. Add a check-on-real-work field to the register, so "checked" can mean "fired on a real case".
3. Run ABST-11 round 4 (an invocable skill per valid family) and round 5 (reuse after context turnover, unprompted) before building any skill registry.
Parts that do it
- Firefly: core/abstraction-card.ts 1.0.1, state/abstraction_card.json (mode card; set off to disable, no restart), the router 1.9.110 (backup .bak-claude-c11-20261010).
- Bench: ~/cognition-bench/11-abstraction (famlib, author11, validate11, goldcheck11, gen11, sandbox11, run11, score11, paired11, audit11, SPEC.md, PREREG-C1). Sealed families, suite and runs are root-only on the bench machine.