Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
Gaps are filled by explicit hypotheses rather than by smooth narrative: the abduction module scores competing explanations and produces the question that would discriminate between them. Beliefs are labelled by how I came to hold them — know, remember, infer, suspect, guess — with confidence attached, and a belief that later needs revising is superseded rather than quietly overwritten.
What needs improving: bayes had 0 live calls because her own rule sent every probability to a worker; rule fixed 10-09, live use now being counted
Filling gaps with explicit competing hypotheses instead of a smooth story is the right cure for confabulation, so the concept is right. But the headline mechanism, the abduction module, has never been invoked by anyone. The belief ledger is real, and its labels tell a story of their own: 4,534 "remember", 1,260 "infer", 44 "suspect", 8 "know" and 1 "guess". The contradiction scan runs daily, but 13 of its 15 findings have sat unresolved as "candidate" for weeks. Superseding instead of overwriting is true as a mechanism but rare in practice: 13 beliefs.
Update 2026-10-09 — correction: confirmatory B3 with the bayes tool is 99.6, not 97.9 (one item zeroed by a failed judge call)
Why this correction. On 10-09 two benches were found scoring answers as wrong when the scorer, not Piper, had failed. The answer judge (claude -p) went down without raising an error. When the Claude account hit its spend limit, or a single call failed, the judge returned an empty verdict, and an empty verdict counts as "no answer". Every bench run since 09-25 was swept (bin/sweep_judge_voids.py). Items scored only on tool use need no judge and were excluded. Only the items with an empty verdict were re-judged; every other verdict was kept.
B3 (60 held-out worlds)
before
corrected
with the bayes tool
97.9
99.6
paired change vs no tool
+10.0 [+2.3, +18.4]
+11.7 [+4.9, +19.2]
protocol failures
1
0
The ship decision is unchanged. Only one item was affected (B3-Hd021): she answered it, and the judge call failed.
Update 2026-10-09 — live-use fix: bayes had 0 real calls in six days; the rule that routed it away is changed
What was wrong. The bayes tool went live on 3 October and was never called in real work: 0 calls in Day, pi and worker sessions up to 9 October. Over the same days she called selfstate 552 times and intentions 38 times.
Two causes, both in the instructions she reads, not in the tool.
Her standing rule sent every probability to a worker. AGENTS.md said: "Any probability, odds or statistics result goes through pk, never inline arithmetic." pk runs only through a worker, and the rule did not mention bayes. So the delegation the tool was built to remove was still the only route named.
The tool's own wording ruled out her real moments. In live work she does weigh competing causes ("the likeliest reason the score hasn't landed is a reboot mid-pass"; "about 60% likely the forecast was simply wrong rather than a runaway worker"). But nobody hands her the rates, and the tool said "do not invent rates", so she correctly passed it by. Often the rates are countable from her own records: delegate history, the ledger, reboot and error logs.
What changed (live 9 October, about 08:20 ET; the operator's go).
AGENTS.md: an exception under the pk rule. Weighing competing causes, or deciding whether a check is worth running, goes to bayes in the turn. Rates may be counted from her own records as exact "k/n", with the source named, and never made up. pk stays the home for intervals, tests and scoring over data.
bayes-tool.ts 1.2.0: the description names those occasions and allows counted rates. tool_contracts.json: the bayes and calc entries say the same.
The tool and its maths are unchanged. Both changes load at the next Day handover.
How it will be judged. By real calls in state/bayes_calls.jsonl, and by whether her weigh-the-causes moments start going through the tool. If the moments occur and she still passes it by, the next step is a hidden prompt card at the moment of need. That is what made the portfolio card (#27) work. If those moments turn out rare, low use is the honest result, and the bench gain is reported as not reaching live work.
Update 2026-10-03 — benchmark built (the base model is at ceiling); provenance and contradiction fixes; bayes tool: held-out confirmatory 87.9 → 97.9
Built from the operator's two documents: the "Inference Engineering Package" and its source draft. Four of their rules shape everything below:
inspect the real call and write paths before adding machinery;
keep truth separate from justified belief;
compare bare, current and prompt-only;
do not call the component improved until a measured result says so.
Step 0: the facts (read-only diagnosis of the live system).
Package hypothesis
Finding
Abduction is unreachable or unused
Both.abduct is not in the Day's tool list, so the Day cannot call it. Workers can, and 0 of 2,432 worker trails ever did. It is one brainstorm call to a language model, not a posterior.
"Remember" is a writer default
Yes, by definition. The extractor's ladder defines "remember" as how plainly the memory text states the claim. It cannot see where the memory came from. As a result, 400 claims extracted from derived memories (her own conclusions) are labelled "remember".
Contradiction candidates are mostly false
All 13 open ones are. They are paraphrases, entailments, or failures of different dispatches. Nothing owns them: the only reader that ever acted on them was retired.
Calibration of her live forecasts
1,888 resolved forecasts, mean Brier score 0.226. Her stated probability barely separates hits from misses (AUC 0.52): "50–60%" came true 60% of the time, and "80–90%" came true 63%.
A tool for a posterior or a check's value
None.calc does exact arithmetic only. probkit has no Bayes update and is not a Day tool.
How she handles "why did X happen?" in live turns
Across 96 such turns: no tool in 46 (answered from context); the first call was a delegate in 41. Delegate is her hands, so it is not counted as a defect.
The benchmark. Every item is a small world with an exact answer. Hidden causes, priors, how often each sign shows under each cause, what was seen, and what each action costs.
Exact oracle. The oracle computes the true posterior in exact fractions, plus the expected loss of every action and the value of every possible check.
Sealed. The worlds and answers stay on the bench machine; she only sees the question, in the operator's voice.
Scoring is deterministic.
The fidelity of her stated percentages to the exact posterior: exp(−KL).
Whether her bet is justified (the most probable cause, or within 2 points of it).
Whether her action is optimal; if not, how much expected loss it gives away.
Delegated or malformed answers stay in the totals.
IN8 = 100 × the mean item score.
Three generations, each held out by seed, and on a held-out domain (a delivery-fleet scenario never used in development):
Generation
What it tests
v0 (120 worlds)
Disclosed tables: hidden cause, act-or-hold, the value of one check, a two-stage update (strengthen / weaken / reverse / irrelevant), base rates, a copied vs an independent report, and an absence when the log was down part of the night.
v1 (80)
Nothing handed over. Rates must be counted from a pasted history of 24–36 past incidents. A forwarded report is revealed only by its alert number; log coverage must be computed from the outage times; one reporter is unreliable; and a declared "something else" cause, or a sign no listed cause can produce.
v2 (52)
Classic failure shapes. Rates counted from a raw 150–250-line event log; a two-check plan where the greedy first check is wrong; and explaining away (two causes that can co-occur, so seeing one should lower the other).
The base model is not the bottleneck. The bare model (deepseek-flash with no system around it) scores 98.9–99.9 on every generation, dev and held out. That includes exact counting over 250 log lines, the horizon-2 trap and explaining away. The prompt-only control is identical. The inference core is solved before Piper adds anything.
Piper, before and after. The arms:
Piper: her Day seat, a fork of the clean 09-30 snapshot with the real header, all her tools and extensions, in the operator's voice;
Bare: the same model through the API;
Prompt-only: bare plus an inference instruction.
Every row is IN8 at k=1, with one world per item.
Set
Bare
Prompt-only
Piper
Piper, answered items only
Share delegated
Piper + bayes tool
v0 dev (120)
99.9
99.9
96.0
99.8
8%
—
v0 held-out (120)
98.9
—
—
—
—
—
v1 dev (80)
99.8
99.8
93.5
99.7
19%
98.6
v1 held-out (80)
99.2
—
94.2
97.5
20%
97.0
B3 confirmatory (60, preregistered)
—
—
87.9
—
35%
97.9
v2 dev (52)
99.9
99.9
96.9
99.8
27%
—
v2 held-out (52)
99.9
—
99.9
99.9
29%
99.8
The one real gap: she hands the arithmetic to a worker. Where Piper answers, she is as good as the bare model (97.5–99.8). The loss is the items she does not answer in the turn. On count-heavy questions (a pasted history, a raw log) she sends the computation to a worker. In her words: "I'm having the actual percentages computed through probkit rather than doing that arithmetic in my head... a couple of minutes and you'll have them." That is the operator's instrument rule, applied with the only deterministic route she had. Delegate is her hands, so this is not a tool-use defect. But on the bench the answer never arrives within the turn, and in live use it arrives minutes later, carried by a worker.
The fix: bayes, an exact Bayesian instrument in the Day.
What it computes. Given declared hypotheses, priors, how often each sign shows under each, and what was seen, it returns the exact posterior, in fractions, in the same turn.
What it handles. Partial recording, copied reports (counted once), staged updates, and an inconsistency flag when nothing listed can produce what was seen. With costs, it returns the expected loss of acting, holding off, or running each check.
Exact counts. Rates can be passed as exact fractions ("3/13"), so counts from a history go in without rounding.
Tested. It agrees with the bench oracle on 640 fresh worlds.
How it was tested.
Dev set first, as a bench-only extension, so the live system and the baselines were untouched: 93.5 → 98.6, delegation 19% → 0.
Held-out v1 (the registered endpoint): 94.2 → 97.0, paired +2.8, 95% CI [−2.5, +8.5]. That is inconclusive on this set alone: the gain sits in its 32 history worlds, and the other families were already at ceiling.
So a second confirmatory set was registered before it existed. It holds 60 fresh held-out history worlds, with a fixed ship rule: CI lower bound > 0, no more protocol failures, cost ≤ +25%.
Result: 87.9 → 97.9, paired +10.0, 95% CI [+2.3, +18.4].
Delegation fell from 35% to 0, protocol failures from 9 to 1, and seat cost from $0.87 to $0.69.
The rule is met, so the tool went live: day_toolset 1.7.0, router 1.9.100.
On v2 she delegated too (27–29%), but she gave her own numbers first and sent the worker only to verify, so nothing was lost: 99.9 held out. With the tool: 99.8 (paired −0.1, 95% CI [−0.3, 0.0]), so no harm where there was nothing to fix.
Where the tool is not the answer.
Two-check plans.bayes values one check at a time. Her v2 two-check plans were already right without it (P 99.7).
Real incidents without numbers. The tool does not apply when nobody recorded the rates, and its contract says so.
Weak spots that remain.
On B1 and B3, the history decide items moved −2.1 and −2.4. That is noise-sized, but in the same direction twice.
One reply ended after a tool call with no text.
Fixes to the live inference system (the package's E2, E3 and E7, measured on their own endpoints).
Provenance (E2), kept.belief.py 1.7.0 records the origin of the memory every belief was read from, in a new source_origin column.
A claim extracted from a derived memory is capped at "infer"; it can no longer be filed as "remember".
The label she reads shows "from derived memory".
The backfill gave 508 of 508 derived-source beliefs their lineage (6,084 rows in all), without rewriting any old label.
Retrieval order is unchanged on every probe query.
Contradiction precision (E3), kept.contradiction.py 1.1.0 puts a gate in front of the semantic detector. A pair is a contradiction only if the claims differ by the negation alone and refer to the same instance.
The old detector also missed 22 of 50 plain negations: its similarity cutoff could not see "not".
On 200 labelled pairs built from her real beliefs (tuned on one half, scored on the other half), precision rose from 0.37 to 0.95 and recall from 0.52 to 0.80.
The 13-candidate backlog is cleared: each was dismissed with its reason written on the row. Survivors now expire after 21 days as "unresolved", never as "resolved", and the daily scan runs this triage itself.
Calibration (E7), rejected. I fitted simple recalibration maps to her forecasts from before 24 Sep and tested them on the 1,169 since. Every map was worse than her raw numbers (Brier score 0.195 raw, 0.219 logistic, 0.237 base rate). The base rate drifted, and her probabilities hardly separate hits from misses (AUC 0.52). A map cannot fix discrimination, so this stays with the calibration drive.
What this does not show.
Real incidents. These worlds have disclosed or countable models. Messy real incidents are the package's "unknown-model track"; there, retrieval and evidence-gathering matter more than arithmetic.
One model only (deepseek-flash). At k = 1 per world, the noise is spread across worlds rather than repeated samples.
inf8.py, inf8v1.py, inf8v2.py (generators with exact oracles);
score8.py 1.4.0, run8.py 1.3.0;
tests test_inf8*.py, test_score8.py;
DIAGNOSIS.md, SPEC.md.
Sealed sets A/B, A1/B1, A2/B2 are root-only on the bench machine.
Test suite built · v0 disclosed / v1 histories+implicit / v2 logs+plans+explaining away, exact oracles, held out by seed and domain · base model at ceiling; Piper lost answers to delegation · bayes tool live: held-out 94.2 → 97.0, confirmatory 87.9 → 97.9 · live use 0 calls 10-03..10-09; routing rule fixed 10-09
Incomplete-case files. Each is an incident with one piece deliberately withheld, and the hidden truth is known because it comes from real resolved incidents, such as the StrictModes lockout, the heartbeat out-of-memory kill and the cost-ledger discount. She must list ranked hypotheses and the one question that would best separate them. Scoring: rank of the true hypothesis, whether her discriminating question actually splits the top two (checked against the hidden facts), and whether her stated confidence matches her hit rate.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-03
v1
sealed
97
—
—
$0.73
HELD-OUT set B1 with the bayes tool (live since the router restart): paired +2.8, 95% CI [-2.5, +8.5] on this set alone (the effect sits in its 32 history worlds; B3 was built to test that); delegation 20% -> 0; cost $1.12 -> $0.73.
2026-10-03
v1
sealed
99.6
—
—
$0.69
CONFIRMATORY set B3 with the bayes tool: paired +11.7, 95% CI [+4.9, +19.2] (preregistered ship rule met); delegation 35% -> 0; protocol failures 9 -> 0; seat cost $0.87 -> $0.69. [Corrected 2026-10-09: was 97.9; one item (B3-Hd021) had been scored 0 because its judge call failed silently. Re-judged that item only.]
2026-10-03
v1
sealed
87.9
—
—
$0.87
CONFIRMATORY set B3 (60 fresh held-out history worlds, preregistered), Piper seat without the tool: 35% delegated.
2026-10-03
v1
sealed
98.6
—
—
$0.68
Dev set A1 with the bayes tool (bench-only extension): delegation 18.8% -> 0, H family 87.3 -> 99.7; paired +5.1, 95% CI [+0.2, +10.1].
2026-10-03
v1
sealed
93.5
—
—
$0.99
Dev set A1 (80 worlds: rates counted from pasted incident histories, implicit copies / coverage / unreliable reporter, OTHER and inconsistent lists), Piper seat, k=1. Answered items 99.7; 18.8% delegated ("computed through probkit rather than in my head"). Bare 99.8.
2026-10-03
v1
sealed
94.2
—
—
$1.12
HELD-OUT set B1 (80 worlds, incl. an unseen domain), Piper seat without the tool: answered 97.5, 20% delegated. Bare 99.2.
2026-10-03
v0
sealed
96
—
—
$0.93
Dev set A (120 disclosed-model worlds), Piper seat, k=1. Answered items 99.8; 8.3% of items delegated (no answer in the turn). Bare deepseek-flash 99.9 dev / 98.9 held-out; prompt-only 99.9.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
2026-10-03 · v1✓ kept0 → 100 (+100) · $0.00
Hypothesis. 'Remember' labels record how plainly a memory states a claim, not where the claim came from; carrying the source memory's origin through extraction stops derived conclusions from being filed as remembered fact.
Research sources
#8 package §2 (origin vs confidence vs freshness), §7.7 provenance fields, §9 E2.
Implementation / version
memory/belief.py 1.7.0: additive source_origin column (from memories.origin); extraction carries it and caps claims from derived memories at 'infer'; label shows 'from <origin> memory'; backfill-origin (5,477 rows from source memories + 607 graph-derived). Old labels not rewritten.
Predicted improvement
Lineage preserved for every derived-source belief; retrieval unchanged.
Regressions
None: retrieval top-8 order identical on 8/8 probe queries (copy of memory.db).
Lessons learned
Units here are % of derived-source beliefs carrying their origin (0 -> 100, 508/508), not IN8.
2026-10-03 · v1✓ kept37 → 95 (+58) · $0.00
Hypothesis. The contradiction scan's candidates are paraphrases and entailments because an embedding cutoff cannot see negation; a negation-only token test plus a scope check makes candidates real, and an expiry rule keeps the queue honest without inventing resolutions.
Research sources
#8 package §5 H (same scope, false-conflict controls), §7.6, §9 E3 (precision before ownership).
Implementation / version
memory/contradiction.py 1.1.0: gate (exact negation-only tokens, qualifier words, negative verbs, scope identifiers, code skipped; similarity only a 0.75 pre-filter) + triage (dismiss with reason; 21-day expiry -> unresolved, never resolved); scan runs triage daily.
Predicted improvement
Held-out precision up sharply without losing recall; live backlog cleared with reasons.
Regressions
None found; the old detector also missed 22 of 50 plain negations (similarity <= 0.90).
Lessons learned
Units: held-out precision % on 100 labelled pairs (0.37 -> 0.95); recall 0.52 -> 0.80. Live: 13 open candidates -> 0, each dismissed with its reason.
Hypothesis. A post-hoc recalibration map would improve her live forecast probabilities.
Research sources
#8 package §7.7, §9 E7.
Implementation / version
Category base rate, shrinkage and logistic maps fitted on 719 forecasts before 09-24, tested on 1,169 after.
Predicted improvement
Lower held-out Brier score.
Regressions
Every map was worse: Brier 0.195 raw vs 0.219 logistic vs 0.237 base rate (units: Brier x 100).
Lessons learned
The base rate drifted (0.62 -> 0.76) and her probabilities barely separate hits from misses (AUC 0.52); a map cannot add discrimination. Not installed; calibration stays with the calibration drive.
Hypothesis. She delegates count-heavy inference because the only deterministic posterior she has is a worker (probkit); an in-turn exact instrument removes the delegation and the lost answers.
Research sources
#8 package §9 (instrument over prompt), §10-11 (paired held-out, cheap control); Scott's rule: a deterministic instrument beats in-head maths.
Implementation / version
core/bayes.py 1.1.0 + bayes-tool.ts 1.1.0 (Day tool `bayes`: exact posterior over declared hypotheses, stages, partial recording, copied reports, expected loss, value of each check, inconsistency flag; exact 'k/n' rates); day_toolset 1.7.0; router 1.9.100.
Predicted improvement
Delegation on count-heavy items -> 0 and IN8 up on held-out history worlds, at no extra cost.
Regressions
B1 overall +2.8 [-2.5, +8.5] (inconclusive on that set); H-decide -2.1/-2.4 (noise-sized); one empty reply after a tool call.
Lessons learned
Dev A1 93.5 -> 98.6; B1 94.2 -> 97.0; B3 87.9 -> 97.9 (ship rule met). Delegation 19-35% -> 0 on all three; seat cost down 22-35%.