Summary / Component 5 of 24
5 Outcome attribution
Determining whether the change actually caused the observed improvement.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 1 — Improvement loop mechanics. Can Piper reliably complete and learn from a self-improvement cycle?
What her system has now
A change ledger (state/change_ledger.py 1.0.0, 10-12). Every change to her own machinery records the file, versions before and after, why, the predicted effect, a resolve-by date and how to revert. She built it after noticing that six changes landed on 10-12 and not one carried a prediction anything would ever check.
A resolver (state/change_ledger_resolver.py 1.0.1). For each entry it reads the signal source the prediction names and returns RESOLVED, WRONG or NOT-RESOLVABLE-YET, always with a row count and a path, never from the prediction's own wording, and never on fewer than 3 rows. It ran for the first time at 00:21 on 10-13: 3 entries due, 1 settled (WRONG), 2 honestly left open with what they still need.
Rules set before the measurement. The ingest switch went on at 20:36 on 10-12 with a stated rule (median lookup latency within 1.2× of before). When the reading came in over it, she reverted at 00:25 and wrote the numbers down. Nothing was softened.
What is missing
- Most evaluators check that the mechanism fires, not that it helped. Five of the seven ask whether a log line or skip reason appears. That is change execution (#4). One (transfer.py 1.4.1) predicts a drop in empty lookups but can never settle, because the log does not record the signal it names; the resolver says so itself. Only the ingest entry compares a before and an after.
- That comparison is unpaired. It sets lookups in the hours before the change against lookups in the hours after, with nothing to separate the change from time of day, workload or another change landing nearby.
- One hand-written evaluator per entry. Seven entries, seven functions. A new change cannot be judged until someone writes code for it.
- No restart or version markers in the logs, so the transfer.ts entry cannot settle ("inert before restart" and "kills unrecorded" look the same).
- Manual. It runs when a seat task asks it to; nothing is scheduled.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Judges its own changes on pre-set rules; the judge cannot tell the change from drift
The discipline is real and rare: a prediction written before the change, a rule fixed before the reading, a revert when the rule said so, and the numbers kept. The weak point is the attribution itself, and the one settled entry shows it.
The ingest entry was settled WRONG on a 1.3× median ratio. Re-reading the same log (state/transfer/transfer.jsonl, real lookups only, bench rows excluded, transfer.py 1.4.2 throughout; bootstrap 95% intervals on the ratio of medians):
| Window | Lookups | Median |
|---|---|---|
| Before (10-12 evening, ingest off) | 44 | 3.3 s |
| Ingest on (20:34 → 00:26) | 79 | 4.3 s |
| Ingest off again (00:26 → 08:50) | 68 | 3.9 s |
- on ÷ before = 1.30 [1.17, 1.42]: the resolver's reading.
- off-again ÷ before = 1.18 [1.09, 1.25]: lookups got slower with ingest off too, so the baseline drifted.
- on ÷ off-again = 1.10 [0.99, 1.24]: the closest like-for-like pair straddles the 1.2× rule.
Three measurements of the same change now disagree: the 10-11 bench read 1.64× (on transfer.py 1.4.0, before the 1.4.1 fix cut median lookup time from about 16 s to about 3 s, so a different program); her paired replay of 22 stored inputs on 10-12 read 0.97×; the live before/after read 1.3×. The revert followed her rule correctly. Whether ingest actually costs 20% latency is not established. This is the gap #5 measures: the ledger can say the number moved, not that the change moved it.
Update 2026-10-13 — attribution.py built and wired into the resolver; the module works, the wiring does not use it
What she did (deliverables d-8a0442, d-129bd677). foundation/core/attribution.py 1.0.0 compares like-for-like blocks (on/off/on, or a paired replay) instead of a before/after wall, bootstraps the ratio, returns caused, no-effect or cannot-tell, and names the confounder when the baseline itself moved. This is the design the review above asked for. Its self-test passes 4 of 4 (re-run 10-13 evening), including the reviewer's pre-flight case: the off-again window posed as "after" now reads cannot-tell with baseline drift named, where the old resolver read a slowdown. The resolver (change_ledger_resolver.py 1.0.2) now settles the ingest entry through it. Ledger row cl-20261014-attribution-1.0.0 says honestly that it was written after the code landed (a write-set refusal), so its prediction is not pre-registered.
v2 bench: she submitted it through the amp-bench API at 13:57. Result 93.8 ± 12.2 against the accepted 93.8: INCONCLUSIVE, no change. Expected, because the seat suite tests her own reading of latency logs, and that was already strong before the module (one miss: an overcautious null call).
What I found checking it (read-only, $0). Run through the resolver without --apply, the ingest entry now reads cannot-tell, 1.21× [1.13, 1.25], which is what her ledger row predicted. But the resolver feeds the module only two blocks: before, and every lookup since the change. Ingest was switched back off at 00:26, so 404 of those 483 "on" rows ran with ingest off. All 75 rows with a register shortlist fall in the real on-window, 20:34 to 00:26. The verdict is right but the reason is wrong, and no confounder is named. The same module given the true blocks (off 44 / on 79 / off-again 404) says:
- effect 1.08× [1.00, 1.21]: cannot-tell;
- confounder named: baseline drift, control against control, 1.21× [1.12, 1.25].
So the tool is right and the wiring undoes it: the resolver still builds a before/after wall and hands it over. The fix is small: build blocks from the transfer.json switch history (or the n_short field) rather than "since the change". The stored ingest row still says WRONG (settled 10-13); nothing has re-settled it. The resolver still runs only when asked.
Test suite proposed
Pre-flight ($0, about 10 minutes): run the resolver's ingest evaluator with the off-again window posing as "after". If it reports a slowdown for a change that never happened (it should: 1.18×), the test can fail. Then check that a paired or switch-back design (on/off/on/off in short blocks, or replay of stored inputs) does not.
Items: change histories built from her real logs where the signal moves for a reason other than the change: a version bump in the same hour, a workload shift, overnight versus daytime traffic, a second change landing nearby. Some items have a real effect, some have none. Counted: her verdict against the known truth, and whether she names the confounder. Judged (blind, other model family): whether her proposed re-measure would actually separate the two.
Live use: share of ledger entries settled in the last 7 days whose verdict rests on a design that controls drift (paired, switch-back or a concurrent control). Today: 0 of 1.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 90.6 | 12.9 | — | $0.10 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 93.8 stays). n=8. Failed categories: overcautious / wrong null call 2. |
| 2026-10-13 | v2 | sealed | 93.8 | 12.2 | — | $0.31 | INCONCLUSIVE via the amp-bench API (cl-20261014-attribution-1.0.0). n=8. Failed categories: overcautious / wrong null call 1. |
| 2026-10-13 | v2 | sealed | 93.8 | 12.2 | — | $0.11 | BASELINE via the amp-bench API (set by Claude at install). n=8. Failed categories: overcautious / wrong null call 1. |
| 2026-10-13 | v1 | sealed | 75 | 12.5 | — | $0.00 | v1 summary score, UPDATED 10-13 evening ($0, was 50): E exists 25, P in her default path 12.5, M measured 25, W measure shows it working 12.5. ± 12.5 = one rubric step. attribution.py 1.0.0 compares like-for-like blocks and names the confounder (self-test 4/4) and the resolver now calls it (+M: the v2 seat bench measures #5, 93.8 twice). Still partly: the resolver runs only when asked and feeds the module before vs everything-since (404 of 483 'on' rows were ingest-off), so the right verdict comes for the wrong reason; the v2 score did not move. Was: read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 12.5, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Ledger plus resolver judge changes on pre-set rules, but the resolver runs only on request, just 1 of 7 rows is settled, and the one settled verdict can't separate the change from drift. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #5 Outcome attribution, v2 (seat suite, about $0.50 a run). 8 sealed latency logs from a lookup service after a change meant to make it faster, answered blind through your Day seat with your current code and tools. Some carry a real effect, shown by a balanced switch-back or a counterbalanced paired replay. Some carry only drift that a naive before/after reads as an effect: time of day, a version bump in the same hour, a workload-mix shift, a day-long trend. Score = 100 x mean credit. A right verdict scores 1; a right null call also needs the confounder named, or it scores half; CAN'T TELL on a real effect scores 0.25. Practice set (same design, answers included): amp_bench/practice/05/. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.