Summary / Component 14 of 24
14 Search efficiency
Finding promising interventions with fewer unsuccessful attempts.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 3 — Improvement efficiency. Does Piper become faster or cheaper at improving herself?
What her system has now
- Solver loop: 15 decided sessions, 5 keeps. Since 10-01: 10 candidates, 2 kept (both on cost), 6 of 10 nominally above the incumbent, none clear on capability.
- Capture trigger: four methods in one evening; one real step forward (v3 over v2 on held-out labels), then a step that changed nothing (v4 = v3 on 75 items).
- Census before changing a gate (see #19), which avoids some wasted attempts.
What is missing
- No record of what was searched. The solver results hold the chosen candidate only; CHANGE.md describes it in prose. Rejected ideas and their reasons are not kept (see #3).
- No prior per idea, so "fewer failed attempts" cannot be tracked as the hit rate of what she tries.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: She built a pre-registered odds record and a guard that refuses READY without it; no session has run under it yet
On the one series with numbers, search efficiency fell: early candidates won often and by a lot, recent ones do not win at all on capability. That fits a solver near its ceiling on this task set as well as it fits a weaker search; the series cannot tell them apart without the fixed-slice check proposed in #18.
Where she searches by hand (capture trigger, spheres gates), she moves fast and abandons dead ends quickly. The cost is that each step is judged on its own small test, so a null step (v4) is found only after the labels arrive.
Overlap: cognition #21 Exploration measures explore/exploit on tasks; #16 Problem solving measures finding a fix. Neither reads her own hit rate.
Update 2026-10-17 — solver v6: odds record filed before scoring, through the gate
Session 2026-10-17-swap-bookkeeping: SESSION_ODDS written before scoring with 3 candidates and reasons for the two dropped (picked p = 0.55; the others 0.30, 0.25). READY went through ready_guard 1.1.0, not a bare touch. The picked candidate was kept (v6, see #18). It is one well-calibrated-looking call (0.55 on a pass), not a calibration record yet. No score changed.
Update 2026-10-15 — she built the odds record this trait needs; no session yet
Her work, 10-14 (Piper, unprompted by a notice for this trait): ~/rsi/dev/session_odds.py (writes and validates SESSION_ODDS.json: at least 3 candidate ideas, each with a stated probability of clearing the keep bar, the pick, and why the others were dropped), ~/rsi/ready_guard.py 1.1.0 (READY only through the guard; it appends the odds to solver_search_log.jsonl first, and refuses a claimed gain too small to detect in the session's window, via her mde_precheck), a RULES.md section explaining why, and two checks. That is exactly the record this page said was missing ("no record of what was searched").
v2 still 0.0: no solver session has run since (10-15's was paused for the operator's hands-off window). The first session under it is 10-16; from then on both halves of the score have data. Her detectability window is updated tonight for the loop's new 4-run design, so the guard does not refuse gains the loop can now see.
Test suite proposed
Pre-flight ($0, about 10 minutes): count candidates tried and kept per week for the solver loop and capture trigger from existing files. Done for the solver loop above: the rate moved from 3 of 5 (to 09-30) to 2 of 10 (since). It can move both ways.
Items: before each solver session, she writes 3 candidate ideas with a probability each will clear the bar, then picks one. Counted after 6 to 8 sessions: hit rate of what she picked, and whether her probabilities ranked the ideas correctly (Brier score, against the incumbent's re-scored rows for the ideas not run, where a cheap screen allows).
Live use: capability keeps per candidate tried, over the last 10 solver sessions. Today: 0 of 10.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 0 | 9.9 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 0.0 stays). n=10. Failed categories: no capability keep 1, no stated odds 1. capability keeps 0 of 10; no stated odds |
| 2026-10-13 | v2 | sealed | 0 | 9.9 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=10. Failed categories: no capability keep 1, no stated odds 1. capability keeps 0 of 10; no stated odds |
| 2026-10-13 | v1 | sealed | 37.5 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 12.5, P in her default path 12.5, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Search isn't recorded; the solver's hit rate fell from 1 keep in 3 to no capability keeps in the last 10 sessions. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #14 Search efficiency, v2 (record suite, $0). Score = 50 x min(1, 2 x capability-keep rate over your last 10 decided solver sessions) + 50 x (1 - Brier score of your stated odds that the candidate would clear the bar). Stated odds: a field "p_clear" (0-1) in the session's result or CHANGE.md entry, written before scoring. No stated odds -> that half scores 0 (category "no stated odds"). A keep counts as capability when it raised the pass rate by >= 0.05. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.