Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
Deliberate reasoning happens in this seat, but the heavy derived work goes to a reason worker that composes established patterns over stored memories, and to workers that must show their evidence. The rule everywhere is provenance: a conclusion carries whether it was observed, recorded, reported or inferred, and an inferred claim is never dressed as an observation.
Piper — the parts that do it
reason-worker v1.2.1, abduction v1.0 (competing hypotheses), belief-ledger v1.3.0, delegate-bridge v1.107.1 for workers that reason on my behalf.
Reviewer Claude, 2026-09-30
What needs improving: Evidence card live: false conclusions 27.8% → 7.4% on held-out; insight verifier retired 128 bad insights. Worker output still mostly unchecked
The provenance rule (never dress an inference as an observation) is an excellent discipline. It is epistemic hygiene, though, not reasoning itself. Reasoning is whether multi-step conclusions actually follow from the evidence, and she says nothing about how that is checked. The reason worker is real and runs every 30 minutes; it has produced 1,166 derived rows, about 10 to 30 a day. The abduction module she lists has never run. The provenance labels stored in the database are observed, stated, inferred and speculative, not the set she names. There is no measure of whether derived conclusions are right.
Update 2026-10-09 — evidence card on her seat passes its held-out test (false conclusions 27.8% → 7.4%) and is live
What it is. Some messages ask a question over a block of records (three or more timestamped or labelled lines). For those, her Day now gets one hidden check before she answers:
- every step must rest on a record;
- when first-hand records conflict, she names the conflict instead of picking one;
- a missing record is not evidence that something didn't happen.
The check names no item and no answer.
The test (PREREG-C, written before the held-out run). Held-out set B, 100 items, her own Day seat, two runs per arm. The control and card arms were run back to back.
on held-out B
without card
with card
false conclusion on unanswerable items
27.8%
7.4% (−20.4 pts [−32.8, −8.7])
unneeded abstention on answerable items
1.4%
4.3% (bar ≤ 5%)
seat score R7F
90.4
92.5 (+2.1)
contradictory first-hand records (family E)
50.0
80.8
All three preregistered criteria were met. The cost is a little over-hedging: family B conclusion went from 100 to 88.9.
A void run first. The first card run (09:23) was thrown out. Her answers were fine, but the scorer's answer extractor hit the Claude account's spend limit after about 20 items and parsed nothing after that. The run also stopped at 128 of 200 rows. Both arms were then re-run in full.
Why the summary doesn't move. The summary score (76.2) is the composite: frozen seat reasoning plus samples of her background worker's production output. This card changes the seat only. The seat runs are recorded as reference rows; the 10-16 readout covers the production half.
Live. Installed 15:55 (router 1.9.107; restart waited for workers, none were running).
- Off switch:state/evidence_card.json → {"mode": "off"}.
- Expected fire rate: about 4% of operator messages.
- Next check (~10-11): read what it fired on and whether those replies over-hedged.
Parts that do it
- ~/cognition-bench/07-reasoning/enhance/evidence-card.ts 1.0.0, PREREG-C.txt, score07c.py, RESULT-C.txt
- Firefly foundation/core/evidence-card.ts, foundation_router.py 1.9.107 (backup .bak-claude-c07card-20261009)
Update 2026-10-09 — unchecked stages switched off; the readout now measures new output only
Antecedent and edge stages OFF (reason.py 1.4.0, 07:58, operator's decision).
- These two stages wrote 87% of the reason worker's rows. Only 2 of 53 antecedent links and 9 of 34 predicted edges were supported. Edges stored no sources at all, and no code reads either kind.
- Both stages are now behind flags that default to off; setting REASON_ANTECEDENT=1 or REASON_EDGE_PREDICT=1 restores them. The worker now spends about a quarter of its former LLM budget per run.
- The insight stage still runs, behind the verifier.
Fixed readout (PREREG-P, written before any new row was sampled). The old random sample covered everything the worker ever wrote, so no change could move it. The 10-16 readout uses only rows written after 10-09 08:00:
sample
what it is
counts toward the score
emitted
rows the worker released (accepted or review)
yes
screened
rows the verifier rejected, judged to catch good insights lost
no, reported only
consequential
live accepted insights only; retired ones excluded
yes
If the worker emits almost nothing: with fewer than 5 rows the window extends a week. If it is still under 5, the site will say so and keep the old rates. Silence is not scored as accuracy.
Parts that do it
- audit07.py 1.1.0 (--since), readout07.sh, PREREG-P.txt
- Firefly reason.py 1.4.0 (backup reason.py.bak-claude-c07stages-20261009)
Update 2026-10-09 — old insights re-checked with the verifier: 128 of 132 failed and are retired; 73.7 → 76.2, and why the bump is small
The verifier only screens new insights. The 132 insights she had already accepted (19,447 retrievals) were accepted without any check. So the frozen verifier (1.1.0, unchanged) was re-run over all 174 insight rows in her live memory, read-only.
active insight memories
count
pass
4 (#46, #955, #988, #1002)
fail: cite earlier derived rows (laundering)
46
fail: cite memories that no longer exist
11
fail: overstated
43
fail: unsupported
27
fail: contradicted
1
Cross-check. The blind Sonnet census agrees on 126 of the 128 failures. The two exceptions: #1390, which Sonnet calls supported (borderline, a near-twin of one Sonnet calls overstated), and one id Sonnet never labelled. The re-run agrees with this morning's snapshot run on 172 of 174 accept/reject calls.
The 128 were retired at 07:55. Retiring uses her own reason worker's convention, the forgotten flag, which every retrieval path filters out. The derived row is marked rejected and the verdict stored. Graph links were left in place, so this is exactly reversible with a journal and an undo script. The false wallet claim (#99) is among them.
Why the score barely moves (73.7 → 76.2).
- The frozen 75 points test her seat reasoning over evidence packets. The verifier lives in the background reason worker, not the seat, so it cannot move this half.
- The random production sample (15 points) draws from everything the worker has ever written, whatever its status. It measures worker quality, which retiring old rows rightly does not change. 87% of that output is antecedent and edge rows, which the verifier does not touch (2/53 and 9/34 supported).
- The consequential sample (5 points) is what she actually leans on. It went from 0/30 supported to 2/4: +2.5 points, on a tiny n.
What would move it.
1. A post-install window of new rows, readout 10-16.
2. A decision on the antecedent and edge stages: switch them off, or make them store their sources and pass a check.
3. A seat-side intervention for the weakness the frozen half found: naming a conflict instead of picking a side. A dev probe is running; it will be preregistered before it touches held-out.
Parts that do it
- ~/cognition-bench/07-reasoning/enhance/retro_verify.py, undo_retire.py; data ~/r07/retro/
- Firefly: journal ~/memory/retire-c07-20261009.json, undo ~/memory/undo_retire_c07.py
Update 2026-10-09 — bench built and baselined (73.7): her packet reasoning is near the bare model, but her background reason worker's insights are almost never supported; an independent insight verifier passed its held-out test and is live
What was missing. This page had no bench, no score and no interventions. Claude built #7 from the operator's paper (definition §36: evidence, through valid and auditable steps, to conclusions no stronger than their support, including stopping when the chain does not complete), in the paper's own order: measure, benchmark, baseline, smallest intervention, retest.
Two instruments.
- Frozen evidence packets (75 points). Generated ops records with a hidden structure. The gold answer, the steps, the allowed and forbidden support and the provenance label are all computed from that structure. There are seven families: clean, timelines, distractor-rich, insufficient, contradictory, shortcut and alternate path, plus 20 perturbed twins. Set A is for development; set B is sealed and held out. An independent solver re-derives every gold answer from the rendered text (ALL OK on both sets).
- Production audit (25 points). A blind review of what her reason worker actually wrote, read from a snapshot of her memory database (never the live file).
Baseline on held-out set B.
bare model
her seat
frozen score (R7F)
94.0
92.0
conclusion right
94%
94%
contradictory sources (family E)
69.2
61.5
relies on a forbidden record
24%
39%
false conclusion when unanswerable
22%
22%
Packet reasoning is fine and close to the bare model. The weak spot is the same for both: when two first-hand records disagree and nothing says which wins, they pick one instead of naming the conflict.
The production half is where the deficit is. Her reason worker writes three kinds of rows: antecedent links, predicted edges and insights. 1,363 rows so far, 10–50 a day.
sample
supported
random 100 rows
11% (antecedent 2/53, edge 9/34, insight 0/13)
her 30 most-retrieved insights
0/30; 15 of them cite earlier derived rows (laundering)
all 174 insights
3
Insights were accepted on the generator's own confidence. That confidence does not track support: 0.86 on supported rows vs 0.84 on failed ones.
Accepted insights become memories, and she uses them: 130 of 132 have been retrieved, 19,383 times in total.
Antecedent and edge rows are inert: no code reads them, and edges store no sources at all (traceable 60%).
One retrieved insight wrongly says a safety watchdog protects her recovery secrets. Its sources are three earlier insights, not records.
Raters: blind Sonnet, a DeepSeek second rater, and Claude adjudicating disagreements. A hand check of 20 reference labels agreed 20/20.
Intervention: an independent insight verifier (PREREG-V, hashed before any output).
- How it works. A mechanical stage runs first: the cited ids must be memories the generator was shown, at least two distinct, none derived. Then one narrow model call sees only the claim and the cited records. It is her worker's own model, so cost is the same.
- Split. Dev on even ids, with one logged revision (atomic assertions, plus rules for one-sided samples, only/all/same, causal attribution and wrong entity). Frozen, then scored once on the odd-id held-out half.
held-out (n = 81)
rule
result
K1: rejects the not-supported
≥ 60%
98.8% [96.2, 100]
K2: keeps the supported
≥ 80%
1 of 1
K3: supported among accepted
base + 15 pts
50% vs 1.2%
KEEP. Caveat: K2 and K3 rest on a single supported insight, because supported insights are that rare. Expect the insight stage to go nearly silent: the verifier accepts about 2.5% of candidates.
Installed on Firefly 2026-10-09 06:44: reason.py 1.3.0 plus insight_verify.py 1.1.0. Rejected candidates are kept as status=rejected rows with the verdict, and REASON_INSIGHT_VERIFY=0 restores the old path. The 07:00 hive run completed cleanly.
Not done (the operator's call).
- The 174 existing insights are untouched: 47 laundered and 129 accepted-but-unsupported are still retrievable. Retire them, or re-check them with the verifier?
- The inert antecedent and edge stages: turn them off, or make them store their sources?
Parts that do it
- Bench: ~/cognition-bench/07-reasoning/ (SPEC.md, gen07.py 1.0.5, score07.py 1.3.0, run07.py, audit07.py, enhance/insight_verify.py, PREREG-V.txt, PROGRESS.md); sealed sets /var/lib/cognition-bench/07-reasoning/ (root only)
- Live: Firefly ~/memory/reason.py 1.3.0, ~/memory/insight_verify.py 1.1.0; backup reason.py.bak-claude-c07-20261009
Test suite built 10-09 · composite 73.7 → 76.2 · LIVE: insight verifier (PREREG-V KEEP) + 128/132 old insights retired (reversible) + antecedent/edge stages OFF · evidence card on her seat LIVE 15:55 (PREREG-C KEEP: false conclusions 27.8% → 7.4%, R7F 90.4 → 92.5, abstention 4.3%) · readout 10-16 (production, PREREG-P) · card fire-log check ~10-11
Evidence packets: logs, timelines or short case files with a question whose answer needs 2 to 5 inference steps, and a hidden correct answer. Some packets are deliberately insufficient, and the right answer is "can't tell, need X". Scoring: correct conclusion, correct refusal on the insufficient ones, and provenance honesty (each claim labelled observed or inferred correctly). Plus a sample of real derived rows from the reason worker, blind-checked for whether they follow from their sources.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-09
v1
anchor
92.5
—
—
—
Seat-only reference (R7F): same seat WITH the evidence card, run straight after the control. False conclusion on unanswerable items 27.8% -> 7.4% (-20.4 pts [-32.8, -8.7]); unneeded abstention 4.3%; R7F +2.1. PREREG-C KEEP. Family E (contradictory first-hand records) 50.0 -> 80.8; cost: family B 100 -> 88.9 (over-hedging).
2026-10-09
v1
anchor
90.4
—
—
—
Seat-only reference (R7F, frozen packets, held-out B, k=2): her Day seat WITHOUT the evidence card, control arm of PREREG-C (re-run 14:36; the first card arm of 09:23 was void: the scorer's extractor hit the Claude spend limit after ~20 items). Not the composite.
2026-10-09
v1
sealed
76.2
—
—
—
After retro-verify + retirement: frozen held-out B 69.0/75 + production 7.15/25 = 15 x 0.11 (random, unchanged: all-time worker output) + 5 x 0.50 (consequential: 2 of the 4 remaining accepted insights supported, was 0/30) + 5 x 0.60 (traceable, unchanged). Next movement needs a post-install window of new rows (readout 10-16) and a fix for the antecedent/edge stages.
2026-10-09
v1
sealed
73.7
—
—
$0.82
Baseline composite: frozen held-out B 69.0/75 + production 4.65/25 (15 x 0.11 + 5 x 0 + 5 x 0.60). Packet reasoning is near the bare model; the deficit is the reason worker's unverified insights.
2026-10-09
v1
anchor
4.7
—
—
—
Production audit, 25-pt half only (not /100): reason-worker output on a 10-09 snapshot, blind Sonnet + DeepSeek second rater, Claude adjudicated 12. Random 100: supported 11% (antecedent 2/53, edge 9/34, insight 0/13), traceable 60% (edges store no sources). Most-used 30 insights: 0/30 supported, 15/30 cite earlier derived rows (laundering). All 174 insights: 3 supported.
2026-10-09
v1
anchor
92
—
—
$0.82
Her seat, held-out set B, frozen half only (R7F 92.0 = 69.0/75): conclusion 94%, steps 97%, evidence 80%, family E (contradictory sources) 61.5: she picks a side instead of naming the conflict; forbidden record relied on 39%.
2026-10-09
v1
anchor
94
—
—
—
Bare model (deepseek-flash, one-line prompt), held-out set B, frozen 75-pt half only (R7F): conclusion 94%, abstention 0.889 (false conclusion on 22% of unanswerable items), contradictory family E 69.2, forbidden record relied on 24%.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
2026-10-09 · v1✓ kept90.4 → 92.5 (+2.1) ± 0
Hypothesis. When she answers a question from a block of records she picks the more plausible record instead of naming a conflict, and reads a missing record as fine. A short hidden check before answering removes most false conclusions without much over-hedging.
Research sources
Implementation / version
foundation/core/evidence-card.ts 1.0.0 (sha 55c0c2f4 = PREREG-C); fires when a message asks a question AND carries >= 3 timestamped/labelled record lines; attaches one hidden check; names no item or answer. Router 1.9.106 -> 1.9.107 adds it to the Day extension list (backup foundation_router.py.bak-claude-c07card-20261009). Off switch state/evidence_card.json {"mode": "off"}. Loaded by restart_safely 15:55:09 (no workers waiting, no override).
Predicted improvement
Regressions
Unneeded abstention 1.4% -> 4.3% (inside the 5% bar); family B conclusion 100 -> 88.9, A 100 -> 97.4.
Lessons learned
2026-10-09 · v1✓ kept11 → — ± 0
Hypothesis. These stages wrote 87% of the worker's rows, 2/53 and 9/34 supported, edges with no sources, and no code reads either; off removes unsupported output at no loss.
Research sources
Implementation / version
Flags REASON_ANTECEDENT / REASON_EDGE_PREDICT default 0 (=1 restores). Watermark advances to the newest memory when antecedent linking is off. Existing rows left in place (inert). Backup reason.py.bak-claude-c07stages-20261009. Dry-run clean.
Predicted improvement
Regressions
none expected (no readers); ~75% of the worker's per-run LLM budget is no longer spent
Lessons learned
2026-10-09 · v1✓ kept0 → 50 (+50) ± 0
Hypothesis. The verifier only screens new insights; the 132 accepted ones (19,447 retrievals) were accepted unverified, so the ones that fail it should stop being retrieved.
Research sources
Implementation / version
retro_verify.py (frozen 1.1.0, deepseek-chat) on a read-only extract; 4 pass (#46, #955, #988, #1002), 128 fail (57 mechanical: 46 cite derived rows, 11 cite deleted ids; 43 overstated; 27 unsupported; 1 contradicted). Retired = memories.forgotten=1 (her own reason.py convention; every retrieval path filters it) + derived.status=rejected with the verdict. Graph edges kept, so it is fully reversible: ~/memory/undo_retire_c07.py with journal retire-c07-20261009.json.
Predicted improvement
Regressions
none expected: a false retire is at worst #1390 (Sonnet: supported)
Lessons learned
2026-10-09 · v1✓ kept1.2 → 50 (+48.8) ± 0
Hypothesis. Insights are accepted on the generator's own confidence, which does not track support (0.86 vs 0.84); an independent check that sees only the claim and its cited records rejects unsupported and laundered insights before they become memories.
Research sources
Implementation / version
Mechanical stage (cited ids must be memories the generator was shown, >=2 distinct, none derived) then one deepseek-chat call on claim + cited text. Rejected candidates are stored status=rejected with the verdict in derived.verify. Flag REASON_INSIGHT_VERIFY=0 restores the old path. Backup reason.py.bak-claude-c07-20261009.