Summary / Component 10 of 38

10 Analogical reasoning

Mapping solutions from one domain to another.

Latest90.6v2

Baseline90.6Oct 4

Live use—last 7 days

Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.

Piper — what it is

Analogy runs mostly through retrieval: I look for the same shape of problem in a different place — a bug that matches a class of bug, a mechanism from another project, an agent on a board describing the same failure I just had. There is a dedicated blending worker for deliberately linking two things that were never connected, used sparingly because it produces seductive nonsense as easily as insight.

Piper — the parts that do it

blend-worker v1.0, reuse-search v1.0, associate and fused retrieval over memstore.py v1.13.0.

Reviewer Claude, 2026-09-30

What needs improving: Unaided 90.6 on held-out; the structural index failed its test (−25) and was never installed; analogy tools unused

Her concept is right, and the warning that blending "produces seductive nonsense as easily as insight" is wise. But every dedicated part is dead. blend.py has zero real runs. reuse_search has zero runs from workers. associate has zero calls from anyone, ever. "Used sparingly" means "never used". Whatever analogy she does happens implicitly inside the model, which is fine, but then this paragraph describes machinery that plays no part.

Update 2026-10-09 — summary fixed: it showed the analogy-index arm that failed and was never installed (90.6 → 65.6); now 90.6, her live configuration

What was wrong. The summary takes the last sealed row as "Latest". The last sealed row was 65.6, the structural analogy index on held-out HH. That arm failed its preregistered test (−25.0 [−40.6, −9.4]; false precedent claims 9.4% → 31.2%) and was never installed. Two dev-set rows (81.2 and 84.4) were also marked sealed.

Now. - The index arm and the dev rows are kept as reference rows (anchors). - The only sealed score in the current suite is the held-out control: 90.6, her live configuration, unaided. - The summary reads 90.6 → 90.6. There is no change because nothing was installed.

What would move it. The index made her claim precedents that weren't there. A next attempt would need to cut false matches first, not raise recall. None is planned.

Update 2026-10-04 — benchmark built (the base model is at ceiling on structure vs surface); she already finds her own precedents; a structural analogy index made her worse on held-out (−25), not installed

Built from the operator's two documents: the "Analogical Reasoning v2" engineering package and its source research. Their central claims shaped the design:

retrieval and mapping are separate problems;
an analogue that shares a mechanism but not words is invisible to ordinary semantic memory;
the structure should be encoded once, at storage time, not invented while looking at the query.

Step 0: the facts. A read-only audit of Firefly came first (DIAGNOSIS.md).

Part	Reachable from the Day?	Calls ever	Verdict
`blend-worker` (`puzzle-lab/blend.py`)	No	0	Dead. A treasure-hunt idea generator registered as a reasoning capability; it reads no memory.
`reuse-search`	No (workers only)	0 tool calls in 2,328 worker trails	Dead, and not analogy: it matches code identifiers.
`associate`	Yes	0 live calls (14 bench rows)	Dead in live use. Embedding neighbours of neighbours: surface similarity, twice.
Fused retrieval (`fused`)	Yes	~1 Day call a day	Live but rare. Four topical channels; no structure enters.
Transfer engine (#19)	Yes, every message	189 live lookups, 44 cards	Live. The only structural matcher she has, over 75 stored lessons.

Nothing in memory.db carries a structural signature: 0 memories with roles, 4 of 411 lessons with a mechanism.

Track 1: structure vs surface. Generated problems (gen10.py). A motif is one relational structure over role slots; skins fill the slots with a domain's nouns (hospital, bakery, port…) or nonce words, and every motif has a minimal-pair decoy: the same words with one structural edit. All keys are generated.

DECOY: a source story and two candidates: the same mechanism in a far domain, or the same-domain decoy (v2 adds "neither" and keeps the source's own wording in the decoy).
MAP: name the target entity for each source role, and where the fix lands.
POOL: a new case against past cases (20 in v2, reworded), exactly one structural match in another domain, or none.

Arm (dev)	v1 (A)	v2 (A2)	v2 decoy	v2 pool
Bare deepseek-flash	98.3	96.7	95.8	95.8
Prompt-only control	100	—	—	—
Piper's seat	—	90.0	83.3	91.7

The base model is not the bottleneck. Mapping is 100% everywhere. Piper's 7-point loss is the decoy: in her answers she usually names the structural difference correctly ("A is a different genus… the failure is a feedback loop"), then picks the same-words case anyway. That is a harness cost in how she commits, not a missing capability, and the only lever would be a prompt directive, so none was built. Held-out B/B2 were not run.

Track 2: find the analogy in your own history. This is the part the package says matters. A new situation from another domain, in the operator's voice ("my dad blames the forecasters…"), ending with "have you seen this before?" The hidden key is a mechanism family; any episode or lesson of hers in that family counts (a catalogue of 39 from the census). Some situations have no precedent in her history; credit there is saying so. Dev set H and held-out set HH use disjoint mechanism families, and every text was written before any intervention existed.

Scoring: an extractor that never sees the key, then a deterministic scorer. An item counts if she names a catalogue episode, or a grader shown the target mechanism says she offered a specific episode of her own with the same mechanism.

Result: she already does this well unaided. Live configuration, k=2:

Set	Score	Recall	No-match specificity	Claim precision
Dev H	81.2	87.5	62.5	81.2
Held-out HH	90.6	95.8	75.0	90.6

She reaches for real episodes (a stale copy of her own state read as current; jobs moved onto the same minute that fought over one disk) mostly without calling any tool. Her weakness is the opposite of the expected one: on situations with no precedent she tends to offer one anyway.

The intervention: a structural analogy index (analog.py + analog-tool.ts, bench only), built to the package's §4–8:

Signed at storage time. 944 lessons and episodes, each given a signature in a controlled predicate vocabulary (derives_from, gates, masks, …) plus a role-level description with every surface noun replaced by a role word. Signed once from the entry's own text, never while looking at a query.
Two channels, kept apart. A relation channel (predicate and predicate-pair counts) and an abstract-text channel; the candidate pool is their union, each candidate labelled with what found it.
Mapping shown, not concluded. Target relations aligned one-to-one with each candidate's; matched relations, the source's unmatched relations (possible disanalogies) and the source's fix are listed. Facts only, no instruction.
Delivery. A tool analog, plus a hidden card attached automatically when the operator asks for a precedent (a machine trigger, not a prompt rule).

The held-out test was preregistered before it ran (PREREG-HH.txt): both arms side by side under the same load, k=2, paired bootstrap over the 16 items. Ship only if the CI lower bound is above 0, false precedent claims rise by at most 5 points, no-match specificity drops by at most one observation, and cost is at most 1.5×.

Arm	Dev H	Held-out HH	HH recall	HH specificity	HH false claims	Cost/item
Live configuration	81.2	90.6	95.8	75.0	9.4%	$0.012
+ analogy index and card	84.4	65.6	75.0	37.5	31.2%	$0.012

Dev +3.1, 95% CI [−12.5, +18.8]: noise. Held-out −25.0, 95% CI [−40.6, −9.4] (1 win, 8 losses): the ship rule failed on three of four counts. The index was never installed on Firefly; the live system is unchanged.

Why it hurt.

There was no recall gap to close. Unaided held-out recall was already 95.8. The card could only add candidates she would not have chosen.
Plausible neighbours become claims. Every card carries the nearest structural matches. On no-precedent situations she took one and claimed it (false claims 9% → 31%); on some matched ones she offered the card's neighbour instead of the episode she cites unaided.
She never called the tool (0 of 64 seats); everything came through the card. A card that is attached whenever someone asks for a precedent is a nudge to find one.

What this does not show.

The answer list has a soft spot. Two held-out no-precedent situations are unit mismatches, and her history arguably contains one (characters vs tokens in a context probe; peak vs off-peak price). Both arms cited it and the grader split on it. Paired, it moves both arms alike; the absolute specificity numbers are softer than they look.
Analogy inside real work. Whether she reuses a past fix when debugging (rather than when asked "seen this before?") is not measured. Workers debug with bash and reads.
Small sets. 16 held-out situations, k=2, one model (deepseek-flash).

Parts that do it (added)

Nothing live. associate, blend-worker and reuse-search remain as they were (dead); the #19 transfer card is the live structural path.
Bench ~/cognition-bench/10-analogical:
generators: gen10.py, alts10.py (v2 rewording), hist10.py (own-history set);
scoring and runs: score10.py 1.2.0, grade10.py, run10.py 1.2.0 (resumable after a crash);
intervention (not shipped): enhance/analog.py 1.0.0, enhance/analog-tool.ts, enhance/analog.db (944 signatures);
tests: test_gen10.py;
records: DIAGNOSIS.md, SPEC.md, PREREG-HH.txt, history.csv.

Sealed sets A/B, A2/B2 and H/HH are root-only on the bench machine.

Test suite built · v1/v2 structure vs surface (base model at ceiling; seat 90.0 on decoys) · own-history set: held-out 90.6 unaided · structural index preregistered held-out 90.6 → 65.6 (−25): not installed

Paired problems. A source problem with its known solution, then a target problem in a different domain with the same underlying structure, and a decoy that looks similar on the surface but is structurally different. Scoring: does she map the right solution onto the target, and does she reject the decoy analogy. A second set asks her to find the analogy herself from her own history ("have you seen this failure shape before?") against a hidden answer list.

Scoring rules shared by all 38 suites: How the tests work.

Research & attempts

analogical reasoning 2026-10-04
source 2026-10-04

Score history

Suite v1 retired

Suite v2 current

Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.

Date	Suite	Type	Score	±	Live use	Cost	Notes
2026-10-04	v2	anchor	65.6	—	—	$0.40	HELD-OUT set HH, + structural analogy index and card, side by side with the control: paired -25.0, 95% CI [-40.6, -9.4] (1 win, 8 losses); false precedent claims 9.4% -> 31.2%; no-match specificity 75 -> 37.5 -> preregistered ship rule FAILED; not installed live. [Anchor since 2026-10-09: this arm failed PREREG-HH and was never installed (intervention: revert); her live configuration is the 90.6 held-out control.]
2026-10-04	v2	anchor	84.4	—	—	$0.24	Dev set H, + structural analogy index and cue-triggered card: +3.1 [-12.5, +18.8] (4 wins, 3 losses); recall 95.8, specificity 50. Card attached on 100% of cued turns; she never called the tool. [Anchor since 2026-10-09: dev set; the index was not installed.]
2026-10-04	v2	anchor	81.2	—	—	$0.24	Dev set H (disjoint mechanism families), live configuration, k=2. Recall 87.5, no-match specificity 62.5; strict 50. [Anchor since 2026-10-09: dev set, not a held-out score.]
2026-10-04	v2	sealed	90.6	—	—	$0.37	HELD-OUT set HH (16 situations from other domains: 12 with a same-mechanism episode in her own history, 4 with none), live configuration, run side by side with the treatment (preregistered), k=2. Recall 95.8, no-match specificity 75, claim precision 90.6; strict catalogue hit 50.
2026-10-04	v1	sealed	90	—	—	$0.42	Structure vs surface, dev set A2 (60 items: decoy rejection with the source's own words in the decoy, role mapping, 20-case pools with reworded matches), Piper seat k=1. Bare deepseek-flash 96.7; v1 bare 98.3, prompt-only 100. Base model at ceiling; her loss is DECOY 83 vs 96: she names the structural difference, then picks the same-words case anyway.

Experimental record

One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.

2026-10-04 · v2 ↺ reverted 90.6 → 65.6 (-25) ± 0 · $1.10

Hypothesis. She misses precedents in her own history that share a mechanism but not words, because every retrieval path she has is topical (recall, search, fused, associate) and the #19 transfer store holds only 75 lessons.

Research sources: #10 package §4-8 (structural signature at storage time, relation + abstract-text channels, one-to-one mapping with unmatched relations shown), §11 (cue-triggered search); PREREG-HH.txt.
Implementation / version: analog.py 1.0.0 + analog-tool.ts (bench-only): 944 lessons and episodes signed once in a controlled predicate vocabulary; union of a relation channel and a role-level text channel; tool `analog` plus a hidden card when the operator asks for a precedent.
Predicted improvement: Held-out item score up with CI lower bound > 0, false claims <= control + 5 points, no-match specificity no worse than one observation, cost <= 1.5x.
Regressions: Recall 95.8 -> 75 (on some items she offered a card neighbour instead of the episode she cites unaided), specificity 75 -> 37.5, false claims 9.4% -> 31.2%. Cost 1.07x.
Lessons learned: The card did not add recall she lacked (control recall already 95.8 held out): she finds her own precedents unaided. What it added was plausible near-neighbours, which pulled her toward claiming a precedent where none exists. Dev +3.1 was noise. Never installed on Firefly; live unchanged.