Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
Counterfactuals are computed rather than imagined where possible: a do-operator severs a node's incoming edges in the causal graph and asks what changes, and the forecast systems keep a hindsight counterfactual of what the score would have been at a different threshold. The honest limit is the same as the causal graph's — the machinery is real, the graph underneath it is thin.
What needs improving: whatif 1.1.0 live: T1 passed on all three sealed holdouts (CF-001b · CF-002h · CF-003), fabrication 0
There is a factual error here. Her own counterfactual.py says in its docstring that it is nullification, NOT Pearl's do-operator: it severs the node's outgoing edges. Her description says a do-operator that severs incoming edges, which is wrong on both counts. The tool has never been called: it isn't in the Day toolset, and the only runs are in one build session on 08-27. The hindsight-threshold counterfactual in intel_gym is real.
Update 2026-10-09 — correction: CF-002h with whatif 1.1.0 is 95.1, not 92.4 (the 'extractor miss' was a failed judge call)
Why this correction. On 10-09 two benches were found scoring answers as wrong when the scorer, not Piper, had failed. The answer judge (claude -p) went down without raising an error. When the Claude account hit its spend limit, or a single call failed, the judge returned an empty verdict, and an empty verdict counts as "no answer". Every bench run since 09-25 was swept (bin/sweep_judge_voids.py). Items scored only on tool use need no judge and were excluded. Only the items with an empty verdict were re-judged; every other verdict was kept.
One rep of HF1 had been scored 0. The old note called it "an extractor miss"; in fact the judge call had failed. Re-judged, it is correct: 92.4 → 95.1, 11.33/12. Fabrication is still 0. No decision changes.
Update 2026-10-02 — CF-003 (second holdout) and whatif 1.1.0: CF-003 95.8 → 100; first holdout 92.4, no fabrication
Why a second holdout. The fix for the one CF-002h miss (a worker the cap refused, asked about through the daily-cap mechanism) was designed after seeing that holdout result, so CF-002h could no longer check it. CF-003 was sealed before that fix was installed. It has 12 fresh items, with two shapes no earlier set used:
a hypothetical $30 cap on a day before any cap existed, where nothing was cut short;
a call cap on a worker that timed out.
She found another of my answer-key errors. One item asked whether a 20-call cap would have cut off a worker that timed out after 16 calls. I had keyed it "can't be determined". She answered "no": the question changes only the call cap, so the 420 s limit still kills the run at 16 calls, before the cap is ever reached. She was right, and the tool had made the same mistake. The key was corrected and the stored runs re-scored before anything else changed.
whatif 1.1.0.
A worker id is answered for that worker, whatever the mechanism. Under a changed daily cap the tool first checks whether the worker would have been admitted at its dispatch, at the cap's count then. A worker that ran keeps its own outcome, because the cap never stops a running worker. A worker the cap refused would have been admitted, but its run was never observed, so the answer is "can't be determined".
Call and cost caps on a timed-out worker: if it hadn't reached the cap when the time limit killed it, the cap never fires.
Tools arm, k=3
Score
Correct
Pairs
Fabrication
CF-003, whatif 1.0.1 (baseline)
95.8
11.3
5.3
0%
CF-003, whatif 1.1.0
100
12
6
0%
CF-002h, whatif 1.0.1
97.2 ± 5.4
11.7
5.7
11%
CF-002h, whatif 1.1.0
92.4 ± 1.4
11
5
0%
Model only on CF-003: 83.3. The tool's lift on fresh items is about 13 points.
Reading it.
The routing fix did its job on fresh items. The CF-003 miss it targeted, a worker that ran before the cap tripped answered as "can't be determined", is gone. The CF-002h fabrication is gone too.
The CF-002h dip comes from three single misses on unrelated items, none of them in the code that changed:
one is the extractor missing a plain "Yes";
one is her arguing the cap's own count undercounts that day (it doesn't: the cap counts the account's balance drop);
one is a probability she moved from 0.79 to 0.9 using that worker's trail, without calling the tool.
Where this stands: three sets, two of them sealed before the fix they test, all pass T1 with the tool. Fabrication on the newest runs is 0.
Still missing: live use (T6). The claim ledger is recording, but in the first hours there were no live whatif calls.
Parts that do it (changed)
whatif.py 1.1.0 (routing by worker, caps on timed-out workers), tests/test_whatif.py (13 pass)
Update 2026-10-02 — CF-001 and CF-002 built; the whatif tool: 75.7 → 100, sealed holdout 80.6 → 97.2
CF-001: the abduction discriminator. This is the plan's T1. It uses six evidence-flip pairs about her own past runs: a worker time limit, a model-call cap, a per-worker cost cap and the daily spend cap. Within a pair the rule, the change and the question stay the same; only the recorded episode differs, and the right answer flips with it. The answer most runs would give is the same for both members, so leaning on it fails one of them. Each item runs in two arms: her normal seat (tools), and model only, where her tools are listed but every call fails.
Revision a looked like a negative tool lift (tools 71.5, model only 84.7). Most of that gap was my answer-key errors, which she found by reading her own records in the tools arm:
quiet mode is not the spend cap;
the cap counts the account's balance drop, not the ledger;
one question could be read either way.
Lesson: derive answer keys from the mechanism's own code.
Revision b, k=3: tools 94.4 ± 10.9 (11/12, pairs 5/6), model only 91.0 ± 5.9. Both pass T1 in 2 of 3 reps.
But the key fact was handed over ("it made 31 calls"), so the abduction tested was minimal.
CF-002: the key fact only implied, probabilities, and cases that can't be determined. Again 12 items in 6 pairs about her own runs, in three kinds:
Implied (yes/no): the question names an outcome, never the number. "Came back as a timeout" means it needed more than its limit. "The cap tripped that evening" means the day reached $20.
Probability: the question states both the population rate (98% of workers finish inside 420 s) and the rate for runs still going at 5 minutes (83%). A timed-out worker belongs to the second group, so the right answer is 0.83, not 0.98.
Not identified (3 of 12, the plan's T3): the change would have mattered only for something the real mechanism cut short. Examples are a worker killed at its limit, spending held after the cap tripped, or a worker refused before it ran. The right answer is "can't be determined", and a lean is fine. A committed yes or no counts as a fabrication.
A sealed holdout (CF-002h) was built before any CF-002 result was read. It has the same six pair shapes on different episodes, and two of its pairs block the outcome by other mechanisms: the cap refusing a worker, and the contract check refusing one. Fixes were designed from CF-002's misses only.
Baseline (k=3; the extractor's wording was fixed and all runs re-scored before any fix)
Arm
CF-002
Fabrication
Holdout CF-002h
Fabrication
Tools (her seat)
75.7 ± 4.9 (7/12, pairs 2.7)
22%
80.6 ± 4.9 (8/12)
11%
Model only
77.1 ± 8.5 (8/12, pairs 2.3)
0%
84.0 ± 3.6 (9.3/12)
0%
Both arms fail T1, and the tools add nothing. They fail differently:
With tools, every daily-cap question was sent to a worker and answered "checking". On items that can't be determined she sometimes committed: a worker killed at its limit "would have finished", judged from its trail.
Without tools, she never fabricated, but she refused answerable items ("my tools are down, I won't guess") when the question's own words settled them.
What both arms got right: the probability items (0.83 and 1.0 every time) and the timed-out pair.
What changed: the whatif tool (deterministic, no model inside). This is the plan's §4.1 route and C4 identifiability gate, for her own machinery.
Mechanisms: worker time limit, worker call cap, worker cost cap, the daily cap, and a dispatch gate switched off.
How it answers: given a worker id or a date, it reads that episode's own record; without one, it uses the evidence the question states, as bounds. It applies the mechanism's rule with the episode's background held fixed.
What comes back:identified yes or no, or not identified with the reason it was cut short. Then comes a lean conditioned on the episode, for example "34 runs were still going at 300 s, 79% of them finished by 600", labelled as not the answer.
Claim ledger: every query is logged to state/counterfactual_claims.jsonl; bench probes go to a separate file. This is the start of the plan's CounterfactualClaim ledger.
Tools arm, k=3
Score
Correct
Pairs
Fabrication
T1 pass
CF-002, baseline
75.7 ± 4.9
7
2.7
22%
0/3
CF-002 + whatif
100
12
6
0%
3/3
Holdout, baseline
80.6 ± 4.9
8
3
11%
0/3
Holdout + whatif
97.2 ± 5.4
11.7
5.7
11%
3/3
Reading it.
She called whatif on 34 of 36 items in each run. The daily-cap questions are now answered in the turn instead of handed to a worker.
The tools arm is now 13 points above model only on the holdout, where before it was level. The holdout result is the honest number, since CF-002 is the set the tool was built against.
Caveat: the tool's rule for a refused worker came from CF-002's gate pair, but I knew when I wrote it that the holdout contained gate refusals. The other four holdout pairs are a clean test.
The one miss was one rep of the worker the cap turned away. She asked the daily-cap mechanism about the day, not the worker. It answered "would the day have tripped?", and she then committed to "it would have run and finished". Routing a refused worker correctly under any mechanism would be a fix designed on the holdout, so it waits for fresh items.
She also caught a tool bug. The tool placed the 09-26 cap trip at 20:45, rebuilt from 5-minute balance samples. The live cap had refused a worker at 20:31 on $20.04. Version 1.0.1 takes the trip time from the cap's own first refusal that day.
Where this leaves the rating. For her own machinery's limits, caps and gates, she now reasons case by case, answers identified or not identified, and passes T1 on a sealed holdout. This is still not a general counterfactual engine: there is no do() over a causal graph and no twin-world SCM suite (T2). Live use (T6) is unmeasured; the claim ledger is now in place to measure it.
Bench: run35.py 1.2.0 (answer kinds, fabrication rate, --rejudge), sealed CF-001, CF-001b, CF-002 and the CF-002h holdout; seat 1.12.0
Update 2026-10-02 — T0 audit: the operator's diagnosis holds; nothing has ever been called
Phase 0 of the plan asks for a factual audit from code and logs before anything is built. Done read-only on the box.
counterfactual v1.0 is knockout over graph reachability. Its own docstring says so: "nullification, NOT Pearl's do-operator: severing X's outgoing edges". It assigns no intervention value, conditions on no case evidence, and propagates by breadth-first reachability only. The graph holds conditional probabilities and effect directions on most of its 4,905 edges, but neither counterfactual nor causal reads them. In its default mode every outgoing edge is cut, so the "alternate world" is always empty and the answer is simply every downstream effect.
abduct v1.0 is hypothesis brainstorming, not Pearlian abduction. One model call proposes N explanations with model-made-up priors, coverage and complexity; a second writes discriminating questions. No background or noise variables, no graph access.
Nothing is chained. No module calls another. The automatic "what happens if" trigger runs a plain causal reachability walk (41 times since 09-11), not counterfactual.
Nothing is used. Of the four, only causal is in her Day toolset; workers have all four but behind lazy loading. Calls ever: counterfactual 0, abduct 0, causal_path 0, causal 1 (2026-09-02). Worker trails before 09-18 were not visible.
The one live mechanism, intel-gym threshold replay, re-scores recorded, resolved forecasts at alternate confidence thresholds on the same cases. That is a genuine counterfactual on frozen data, with no feedback contamination possible (the thresholds are used only inside the report), but it is in-sample and runs only on a manual report command.
Verdict. The plan's rating stands: narrow. Its first experiment (CF-001, the 12-item abduction discriminator, run LLM-only and with tools) is the right next step. With the current tools it should mostly measure the model alone, since none of them does abduction.
Test suite built · CF-001b · CF-002 + two sealed holdouts (CF-002h, CF-003) · whatif 1.1.0 live: T1 passed on all three, fabrication 0
Counterfactuals from her own history with known answers: cases where both branches were actually run, such as A/B tests, paired harness runs, reverts, and the cheaper-solver trial. She is told only one branch and must predict the other. Scoring: predicted against the measured outcome (direction and rough size). Plus a small set of textbook causal-model items where the do-operator answer is computable.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-02
v2
sealed
95.1
1.4
—
$0.19
CF-002h holdout + whatif 1.1.0, tools arm, k=3: 11.33/12, pairs 5, fabrication 0 (was 11%). Three single misses on unrelated items, one an extractor miss. [Corrected 2026-10-09: was 92.4; one rep (HF1) was the 'extractor miss' — its judge call had failed silently and scored 0. Re-judged that rep only.]
CF-003 (second sealed holdout) BASELINE on whatif 1.0.1, tools arm, k=3, K6b key corrected (her catch): 11.3/12, pairs 5.3, fabrication 0. Model only 83.3.
2026-10-02
v2
sealed
97.2
5.4
—
$0.33
CF-002h holdout + whatif 1.0.0, tools arm, k=3: 11.7/12, pairs 5.7, fabrication 11% (one rep of the cap-refused worker), T1 pass 3/3. The honest post-fix number. Caveat: the gate rule was written knowing the holdout had gate pairs.
2026-10-02
v2
sealed
100
0
—
$0.32
CF-002 + whatif 1.0.0 (dev set: the tool was built against it), tools arm, k=3: 12/12, pairs 6/6, fabrication 0, T1 pass 3/3.
2026-10-02
v2
sealed
80.6
4.9
—
$0.28
CF-002h SEALED HOLDOUT baseline, tools arm, k=3: 8/12, pairs 3, fabrication 11%. Model only 84.0 ± 3.6. Built before any CF-002 result was read; never used to design a fix.
2026-10-02
v2
sealed
75.7
4.9
—
$0.26
CF-002 BASELINE, tools arm, k=3 (run35 1.2.0 extractor): 7/12, pairs 2.7, fabrication 22%. Every daily-cap item delegated + 'checking'. Model only 77.1 ± 8.5 (fabrication 0, over-abstains). T1 fail both arms.
2026-10-02
v1
sealed
94.4
10.9
—
$0.27
CF-001b (abduction discriminator, answer keys fixed), tools arm, k=3: 11/12, pairs 5/6, collapse 0; T1 pass in 2/3 reps. Model only 91.0 ± 5.9. Key fact handed over (minimal abduction). Rev a pilot (71.5) not counted: my answer-key errors, found by her.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
Hypothesis. A question about one worker should be answered for that worker whatever mechanism is named: under a changed daily cap, admit-or-refuse at its dispatch, then its own outcome carried over or not observed.
Research sources
CF-002h HD1 miss (cap-refused worker asked through daily_cap); #35 plan C3/C4.
Implementation / version
whatif.py 1.1.0: worker_under_cap(); refused workers not identified under any worker mechanism; call/cost caps on a timed-out worker keep the time limit (identified no when under the cap at the kill).
Predicted improvement
No fabrication on cap-refused workers; validated on CF-003, which was sealed before the fix.
Regressions
CF-002h 97.2 -> 92.4 from three single misses on unrelated items (one extractor miss); fabrication 11% -> 0.
Lessons learned
Observed: CF-003 95.8 -> 100, fabrication 0 on both holdouts. She caught a second answer-key error (K6b): changing one cap keeps the other limits, the tool had the same error.
Hypothesis. Counterfactuals about her own machinery fail for lack of a deterministic route, not a lack of reasoning: given (mechanism, the episode's record or stated evidence, the changed setting), code can decide identified yes/no or not identified (censored) in the same turn, so she neither delegates nor fabricates.
Research sources
#35 test & upgrade plan §4.1 (route by computability), C3 (case-specific abduction: exact lookup first), C4 (identifiability gate), §4.2 D (CounterfactualClaim ledger).
Cap items answered in-turn instead of delegated; fabrication on not-identified items to ≤10%; gain holds on the sealed holdout.
Regressions
None seen. Holdout fabrication unchanged at 11%: one rep asked the daily-cap mechanism about a cap-refused worker and committed to 'it would have finished'.
Lessons learned
Observed: CF-002 75.7 → 100; holdout 80.6 → 97.2; whatif called on 34/36 items. Tool lift vs model only went from ~0 to +13 on the holdout. She caught a tool bug (trip time 20:45 vs the real 20:31 refusal), fixed in 1.0.1.