Summary / Component 13 of 24
13 Experiment efficiency
Reaching reliable keep/revert decisions with fewer tests.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 3 — Improvement efficiency. Does Piper become faster or cheaper at improving herself?
What her system has now
- Paired replays on stored inputs: the ingest latency replay (22 real stored lookups, off against on,
piper/lookup_ingest_ratio) and the with/without-history decision replay (6 items, $0.0042,piper/rsi_replay_result). Both cheap, both paired. - A resolution check on her own keep rule (
piper/rsi_keep_rule_power): the smallest gain a session can resolve is 0.043 to 0.098 (median about 0.078) against a bar of 0.05. - The solver loop re-uses incumbent runs across sessions (72, 108, 144, 180 runs as a version stays), which is efficient.
What is missing
- Candidates are read on 36 runs, so most capability candidates can neither pass nor fail cleanly: 0 of the 10 candidates since 10-01 cleared the loop's own z ≥ 1.28 on capability, though 6 of 10 scored above the incumbent.
- Unpaired before/after for live changes (the ingest case in #5: three measurements of one change, three different answers).
- Nothing records how many tests a keep or revert decision took.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Most of her predictions are checkable and nobody checked them; the solver loop's scoring is being rebuilt to see smaller gains for the same money
She knows how to test cheaply and well; she does it when she designs a test by hand. The paired replay of stored inputs is the best design on her record, and her keep-rule note is the right diagnosis of the solver loop.
The default paths do not use either. Live changes are judged on before/after windows, and the solver loop spends $2.50 to $5.80 a session to read a candidate on 36 runs, which cannot see a 0.05 gain. Her own options (target +0.08, or more runs per task) are both correct; a third is cheaper: a two-stage test, where a small paired screen on the incumbent's existing per-task rows drops clear losers before the full scoring is paid for. Four of the last eight rollbacks were candidates that scored below the incumbent.
Overlap: this is the same problem as the "small tests first" rule for the benches, and the same resolution question the RSI practice rig answered with its 0.05 bar.
Update 2026-10-17 — solver v6: noise floor measured before any spend; MDE pre-check ran in-path
Before spending on a candidate, she measured an A/A floor of v5 against itself (per-run sd 0.271, SE ~0.10 at k=2, $0, 186 s) and concluded the loop's +0.05 bar sits inside practice noise. ready_guard's MDE check (built 10-14 from this component's pre-check) ran on READY and reported her claimed +0.045 as detectable only on a second look. The kept result was +0.074 at 4 runs per task (see #18). No score changed.
Update 2026-10-15 — settle-from-logs: the predictions were checkable, nobody checked; and the solver loop's scoring is being rebuilt (installs tonight)
Settle pass (Claude, 10-15, read-only, $0). Of 120 pending ledger rows, 97 carry no prediction, so no test design can settle them. The other 23 do, and many can be decided from her own logs. Decided now: 7 right, 1 cannot tell, 0 wrong; two had been past due since 10-13 with the evidence on disk. Written to her ledger after 19:25 (the operator's call). The bottleneck is settling, not testing.
Solver loop, option C (the operator, installs after 19:18). Pre-flight over 17 sessions: the daily re-run of the current version was half the $63.71 scoring bill, yet the keep rule compares against the version's pooled runs (108 to 180), so the re-run moved the decision about 4%. From 10-16: the current version is re-run weekly (a drift check) or until it has 8 runs per task; the candidate is screened on one run per task (rolled back at or below −0.10: on past sessions 4 losers stopped, 0 of 5 keeps; a real +0.05 is stopped about 2% of the time); a candidate that passes gets 4 runs per task instead of 2. Same money, and the smallest gain the loop can see drops from about 0.10 to about 0.075. Her RULES.md and her detectability meter are updated with it. The v2 score stays 0.0 until sessions run under the new design.
Test suite proposed
Pre-flight ($0, about 10 minutes): from the stored per-task rows of the last 10 solver sessions, check whether a 12-run paired screen would have flagged the 4 losing candidates before full scoring. If yes, the screen can save money; if it also flags winners, it can fail.
Items: replay the last 10 sessions' decisions under three test designs (current; +runs; screen then score) using stored rows only. Counted: decisions that match the full-data verdict, and dollars spent per decision.
Live use: scoring dollars per decided solver session, and the share of sessions whose test could resolve the bar. Today: $2.50 to $5.80 per session; median resolvable gain about 0.078 against a 0.05 bar (3 sessions, her figures).
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 0 | 9.8 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 0.0 stays). n=10. Failed categories: test too small for the bar 10. 2026-10-03: smallest gain this test could see +0.149 (bar 0.05), scoring $5.78; 2026-10-04: smallest gain this test could see +0.199 (bar 0.05), scoring $4.40; 2026-10-06: smallest gain this test could see +0.107 (bar 0.05), scoring $4.00; 2026-10-07: smallest gain this test could see +0.093 (bar 0.05), scoring $4.26; 2026-10-08: smallest gain this test could see +0.172 (bar 0.05), scoring $4.41; 2026-10-09: smallest gain this test could see +0.103 (bar 0.05), scoring $3.37; 2026-10-10: smallest gain this test could see +0.214 (bar 0.05), scoring $2.67; 2026-10-12: smallest gain this test could see +0.172 (bar 0.05), scoring $2.53; 2026-10-13: smallest gain this test could see +0.195 (bar 0.05), scoring $2.44; 2026-10-14: smallest gain this test could see +0.197 (bar 0.05), scoring $2.13 |
| 2026-10-13 | v2 | sealed | 0 | 9.8 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=10. Failed categories: test too small for the bar 10. 2026-10-01: smallest gain this test could see +0.085 (bar 0.05), scoring $5.25; 2026-10-02: smallest gain this test could see +0.104 (bar 0.05), scoring $5.11; 2026-10-03: smallest gain this test could see +0.149 (bar 0.05), scoring $5.78; 2026-10-04: smallest gain this test could see +0.199 (bar 0.05), scoring $4.40; 2026-10-06: smallest gain this test could see +0.107 (bar 0.05), scoring $4.00; 2026-10-07: smallest gain this test could see +0.093 (bar 0.05), scoring $4.26; 2026-10-08: smallest gain this test could see +0.172 (bar 0.05), scoring $4.41; 2026-10-09: smallest gain this test could see +0.103 (bar 0.05), scoring $3.37; 2026-10-10: smallest gain this test could see +0.214 (bar 0.05), scoring $2.67; 2026-10-12: smallest gain this test could see +0.172 (bar 0.05), scoring $2.53 |
| 2026-10-13 | v1 | sealed | 50 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 0, M measured 25, W measure shows it working 0. ± 12.5 = one rubric step. Her hand-designed tests (paired replay) are good, but the default paths don't use them, and the solver spends $2.50 to $5.80 per session on 36 runs that can't detect a 0.05 gain. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #13 Experiment efficiency, v2 (record suite, $0). Score = % of your last 10 decided solver sessions whose test could actually resolve the bar it asked for: minimum detectable gain (80% power, from that session's own noise: se = gain / t) at most +0.05. A test too small to see the gain it is looking for wastes the scoring spend whatever it decides. Raise it with a design that sees more for the money (paired screens, re-used incumbent rows, more runs where it matters). |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.