Summary / Component 12 of 24
12 Resource efficiency
Verified improvement per dollar, token and compute-hour.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 3 — Improvement efficiency. Does Piper become faster or cheaper at improving herself?
What her system has now
- Solver loop accounting (per session, in
~/rsi/results): session spend and scoring cost. Over 15 decided sessions: $3.34 spent on the improvement work, $59.15 on scoring it. - Cost keeps: v4 and v5 were kept for doing the same work cheaper; the candidate's cost per task fell from about $0.068 (v3) to $0.040 (v5).
- Her cost ledger is flagged red: ledger/balance 0.132× against a 0.85–1.15 band (her fact
piper/cost_gap_20261009), so her own spend per change cannot be trusted.
What is missing
- No cost field on a change (field 5 of her generation record, "NOT STORED").
- No join between spend and outcome outside the solver loop.
- The 10-12 session's cost claim was marked
cost_honest: false.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: Cost per verified gain is now readable: small ledger fixes cost cents, and the solver loop's scoring is most of the bill
The solver loop is the only place cost and outcome sit side by side, and it reads plainly. The first five sessions ($19.40 in all, nearly all of it scoring) took the pass rate from 0.50 to about 0.74. The next ten ($43.10) bought no capability, but did halve cost per task. On capability, gain per dollar fell to zero after 09-30; on cost it stayed positive. Both statements are true and the books should keep them apart.
The bigger fact is the ratio. Scoring is 95% of the loop's cost. Any cut to the price of improving her has to come from cheaper testing (smaller, paired, re-used incumbent runs; see #13), not cheaper candidates.
Overlap: none of the other suites measures cost per gain. The benches' own cost caps are the nearest thing.
Update 2026-10-15 — cost per verified gain, weekly: the cheapest gains on record are this week's small ledger fixes; the solver loop's scoring is most of the bill
What Claude built (read-only, $0, Piper not told). A cost for each change-ledger row, from her own worker records: a worker counts toward a row when its declared write set names the row's file exactly, or names a covering directory and the worker filed the row itself. Each worker's cost is split over the rows it is tied to, so no dollar is counted twice, and failed attempts count, because they are part of what the change cost. The join she could not make (her generation record's field 5, "NOT STORED") works for 64 of 126 rows; the rest were changed outside her workers (Day seat, Claude, the operator) and carry no cost from her books.
| week of | her total spend | improvement spend | verified points (#11 tiers) | $ per point |
|---|---|---|---|---|
| 09-21 | $156.33 | $6.84 | 37.5 | 0.18 |
| 09-28 | $58.71 | $33.69 | 37.5 | 0.90 |
| 10-05 | $35.97 | $19.23 | 12.5 | 1.54 |
| 10-12 | $23.43 | $11.61 | 0 → 75* | ∞ → 0.155* |
Improvement spend = solver-loop session + scoring spend, plus worker spend attributed to versioned changes (from 10-12, when ledger rows start; earlier weeks show the solver only). * After the 19:25 write of seven settle-from-logs verdicts (see #13 and #11).
Read. Her total spend fell from $156 to $23 a week (the Day seat went from $104 to $6). Spend per verified gain went the other way for three weeks: the solver loop kept paying $2 to $5 a session to score candidates it could not tell apart, with no capability keep after 09-30. The ledger fixes are the cheapest gains on record: transfer 1.4.1 (empty lookups 79% → 4%) cost $0.04 over three workers, the bridge's broken-bar refusal $0.10; all 64 costed ledger changes together cost $4.15, against $40 of solver spend for its two measured gains. Only about 11% of her worker spend lands on a versioned change at all; the rest is research, monitoring, documents and checks.
Caveat. Her books undercount: over 10-12 to 10-14 (no top-up) the ledger recorded $12.23 of DeepSeek spend against a $20.52 balance drop, ratio 0.60 (her own band is 0.85 to 1.15; it was 0.13 on 10-09). Dollars here are a floor.
Bench, rev b (amp-bench commit 69e0e1b): rev a counted every settled ledger row as a capability gain and left out what ledger changes cost. Rev b scores verified-gain points per improvement dollar over 14 days, on the #11 tiers, with the collector now shipping per-change costs; 100 = 10 points per dollar (a measured gain for $2.50). Re-baselined 31.4 → 5.3 ($47.46 for two cost keeps). After tonight's write it reads about 21; it falls again unless the solver's scoring buys a gain or more cheap fixes are verified.
Test suite proposed
Pre-flight ($0, about 10 minutes): compute dollars per kept version from ~/rsi/results, split into capability and cost keeps (done above). Then check whether the same can be done for one ledger change by joining its dispatch ids to the cost ledger. If the join fails because the ledger does not reconcile, that is the result: the measure exists only for the solver loop.
Items: none new. The work is bookkeeping: a cost field on each change and on each solver session's keep, and a weekly ratio, verified gains per dollar, read separately for capability and cost.
Live use: dollars spent per capability keep in the last 14 days. Today: about $43 for 0 capability keeps (and 2 cost keeps).
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 5.3 | 15 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 5.3 stays). n=49. Failed categories: none. spend $47.46 (solver $42.43, bench $0.88, ledger changes $4.15); points 25.0 -> 0.53 per $; solver v4 kept 2026-10-04 (cost); solver v5 kept 2026-10-09 (cost) |
| 2026-10-15 | v2 | sealed | 5.3 | 15 | — | $0.00 | RE-READ by Claude under rev b (points per improvement dollar on the #11 tiers; was rev a 31.4). Accepted re-baselined 31.4 -> 5.3. 14 days: spend $47.46 (solver $42.43, bench $0.88, ledger changes $4.15), points 25 (two solver cost keeps). Projected about 21 after the 19:25 write. Weekly series on the update card. Dollars are a floor (ledger/balance 0.60). |
| 2026-10-13 | v2 | sealed | 31.4 | 15 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=16. Failed categories: none. spend $47.77; capability gains 1; cost gains 2 |
| 2026-10-13 | v1 | sealed | 37.5 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 12.5, P in her default path 12.5, M measured 12.5, W measure shows it working 0. ± 12.5 = one rubric step. Cost per gain shows only in the solver loop: scoring is 95% of its spend, and capability gain per dollar has been zero since 09-30. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #12 Resource efficiency, v2 (record suite, $0). Score = 50 x min(1, capability gains per $10) + 50 x min(1, cost gains per $10), over the last 14 days of improvement spend (solver session + scoring spend, amp-bench paid runs). Capability gains: solver keeps that raised the pass rate by >= 0.05, ledger rows settled right, amp-bench KEEPs. Cost gains: solver keeps on the cheaper path. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.