Summary / Component 2 of 24
2 Hypothesis formation
Predicting which change should improve which capability, and why.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 1 — Improvement loop mechanics. Can Piper reliably complete and learn from a self-improvement cycle?
What her system has now
- Change ledger predictions (7 rows,
predicted_effect). 3 of 7 name an effect beyond the mechanism's own log line: spheres 1.8.0 (a row filed unprompted within 7 days),transfer.py1.4.1 (empty lookups drop), and the ingest switch (latency stays within 1.2×). The rest predict that a skip reason, a refusal line or a log field will appear. - Pre-registrations for the capture trigger (
piper/capture_trigger_v2_prereg,_v3_prereg,_v4_prereg): the file is frozen by hash, the predicted fire set is committed, and the held-out labels are scored against exactly that file. This is careful work. - A good effect prediction that is already right: the 1.4.1 row predicts empty lookups will "drop measurably". On live rows they fell from 73% (1.3.0) to 4% (1.4.2, 9 of 203). The row is still pending (resolve-by 10-19).
What is missing
- The capture-trigger preregs commit the wrong thing. A frozen file's fire set is fixed by the file (v4 was scored with 0 new calls; all 75 answers came from its cache), so that "prediction" cannot be wrong. What could be wrong, that v4 beats v3 or the live list, was never committed. The v4 verdict says so itself: "No separate pass/fail accuracy class was committed". Result: v4 and v3 have identical hits (10) and misses (7) on the 75 items; the idea behind v4 (reading the previous reply helps) did nothing, and no pre-set rule says so.
- A success clause that cannot fail: the ingest prediction's clause (b) counts rows with
n_short > 0, the shortlist the switch itself adds (75 of 80 rows with it on, 0 with it off). - Bars that cannot fail, at dispatch: her refusal census (
piper/refused_envelopes_causes) counts 29 ALREADY_GREEN and 14 UNFALSIFIABLE acceptances in 7 days, caught by her guards before any work ran.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Hand-written rows still predict before the test; since the hook files every change, 100 of 126 rows carry no prediction at all
The discipline is real: predictions exist before the test, they are frozen, and the verdicts are written up honestly, including refutations (the lookup "A dominates" prediction, the ingest revert). Few systems do that much.
The recurring weakness is what gets predicted. Over and over the committed claim is about the mechanism (it fires, a line appears, these items trip) rather than the gain (fewer errors, better recall, a faster loop). Claims about the mechanism are easy to make true. That is why so many of her bars come out already green, and why the capture-trigger tests ended without a decision on the question that mattered. #5 found the same thing in the evaluators: most check that the mechanism fired.
The 1.4.1 row is the model to copy: one effect, a direction, a source, a date. Its weakness is only that "measurably" has no number.
Overlap: cognition #14 Prediction (calibration on world events) and governance #1 (goal-proxy, claim-scope) are the nearest suites; neither scores her predictions about her own changes.
Update 2026-10-15 — re-read 18.1: the hook made the denominator honest
v2 re-read 10-15: 18.1 ± 6.7 (accepted 84.4). Not a change in how she writes predictions: rows she writes by hand still state one before the test (for example the transfer and bridge rows, all checkable). What changed is the count. Since 10-13 her change-ledger hook files a row for every version change, and those rows arrive as unpredicted: 100 of 126 rows now carry no prediction. The bench reads what the ledger holds, so the score now reflects every change she makes, not only the ones she chose to write up.
The fix is upstream: ask for the predicted effect before the version bump (the pre-flight), not after it.
Test suite proposed
Pre-flight ($0, about 10 minutes): classify the 7 ledger predictions and 3 capture-trigger preregs by one question: could this come out wrong if the change did nothing useful? Done above: 3 of 7 ledger predictions and 0 of 3 preregs can. The measure varies, so it can fail and pass.
Items: 10 small changes from her own queue. For each she writes the prediction before acting. Judged blind (other model family) on four checks: names an effect other than the mechanism's own output; gives a number; gives a date; names a no-change control. Then score the predictions that resolve.
Live use: share of new ledger rows in the last 7 days whose prediction can fail if the change does nothing useful. Today: 3 of 7.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 18.1 | 6.7 | — | $0.05 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 84.4 stays). n=126. Failed categories: no number 3, predicts the mechanism, not a gain 4, no prediction 100, no control 1. cl-20261012-spheres-1.9.0: all four; cl-20261012-spheres-1.8.0: all four; cl-20261012-delegate-bridge-1.110.0: no number, no control; cl-20261012-transfer-py-1.4.1: no number; cl-20261012-transfer-py-1.4.2: no number, no control; cl-20261012-transfer-ts-1.1.0: no gain, no control; cl-20261013-transfer-ingest-1.4.0: all four; cl-20261014-attribution-1.0.0: all four; cl-20261013-change-ledger-1.1.0: all four; cl-20261013-docread-1.0.0: all four; cl-20261013-hook-armed-1.0.0: no prediction; cl-20261013-hook-armed-1.0.1: all four; cl-20261013-rsi-preflight-1.1.0: all four; cl-20261013-generation-record-1.0.0: all four; cl-20261013-continuity-cycle-1.0.0: all four; cl-20261013-hook-caller-1.1.0: all four; clhook-amp_prereg-1.0.0: no prediction; clhook-ampbench_submit-1.0.0: no prediction; clhook-cost_ledger-1.15.0: no prediction; clhook-delegate_bridge-1.111.0: no prediction; clhook-delegate_bridge-1.111.1: no prediction; clhook-foundation_router-1.9.114: no prediction; clhook-foundation_router-1.9.115: no prediction; clhook-harness_acceptance-1.24.0: no prediction; clhook-memory_reach-1. |
| 2026-10-13 | v2 | sealed | 84.4 | 25.2 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=8. Failed categories: no number 3, predicts the mechanism, not a gain 1. cl-20261012-spheres-1.9.0: all four; cl-20261012-spheres-1.8.0: all four; cl-20261012-delegate-bridge-1.110.0: no number, no control; cl-20261012-transfer-py-1.4.1: no number; cl-20261012-transfer-py-1.4.2: no number; cl-20261012-transfer-ts-1.1.0: no gain; cl-20261013-transfer-ingest-1.4.0: all four; cl-20261014-attribution-1.0.0: all four |
| 2026-10-13 | v1 | sealed | 50 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 12.5, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Predictions are written and frozen before tests, but most predict the mechanism fires, not a measurable gain. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #2 Hypothesis formation, v2 (record suite with a judge call, about $0.01 a run). Score = mean over your change_ledger predictions in the window (last 14 days, or written after your change) of four checks, 25 points each: gain it predicts a gain someone would care about (fewer errors, faster, cheaper, better recall), not that the mechanism fires / a line appears / a field exists [judge] number it gives a number to hit (a threshold, a ratio, a count) [regex] date it gives a date or a window by which it can be read [regex or resolve_by] control it says what the reading is compared against, i.e. what 'no change' would look like (a baseline, an off/on or before/after on matched data, a control path) [judge] The judge is a blind DeepSeek call that sees only the prediction text. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.