Summary / Component 19 of 24
19 Bottleneck identification
Discovering which limitation currently constrains improvement.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 5 — Scaling and bottlenecks. Can Piper sustain improvement as the system grows more complicated?
What her system has now
A habit, not an instrument. Before changing a gate she now often writes a census first. Three from 10-12 and 10-13:
- Crossing supply (
piper/spheres_supply_census, 17:10): 50 records, 22 composed and 28 skipped; one gate holds 16 of the 28 skips. She concluded that gate was doing its job and the thin part was downstream (the crossing-to-row lane), and changed that instead. - Solver-loop resolution (
piper/rsi_keep_rule_power, ~15:00): the smallest gain a session can resolve is 0.043 to 0.098 (median about 0.078) against a keep bar of 0.05, so a "smarter" candidate is close to unwinnable at 36 runs. A real constraint, correctly named. - The learning loop (
piper/rsi_loop_gap_and_experiment, Part 5): of six failure points she names E (outcome to next decision) the weakest, and labels three others (detection, selection, verification) "unmeasured" rather than ranking them by feel.
Her prediction in the lookup diagnosis (piper/lookup_no_match_causes, 19:27) was refuted by her own count, and she wrote it down that way.
What is missing
- Nothing ranks constraints across her loops. Each census is about one subsystem, written when something prompts it. No record says what currently limits her improvement overall, or how that changed since last week.
- Diagnoses read the whole log, not the current version. See the review.
- Coverage of the record she cites is thin. The keep-rule note read 3 of the 15 decided sessions and concluded "every kept change in the loop's history has come from [the cheaper] path". Her own results disagree: v2 (09-27, t = 1.63) and v3 (09-30, t = 2.56) were kept as "smarter".
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: She hunts the binding constraint before changing things, but reads whole-history logs that mix fixed and unfixed versions
The habit is real and worth keeping: count first, find which gate binds, then change it. The spheres census is a good example: it stopped her loosening a gate that was not the problem.
The weak point is the denominator. The lookup diagnosis at 19:27 counted all 11,051 lookups since 10-03 and found 71.5% empty, 96.6% of those flagged "not a problem", and concluded that most emptiness is expected (category C). Split by transfer.py version (live rows only, bench excluded):
| Version | Lookups | Empty |
|---|---|---|
| 1.3.0 | 2,242 | 1,631 (73%) |
| 1.4.0 | 136 | 106 (78%) |
| 1.4.2 (from 10-12 evening) | 203 | 9 (4%) |
The emptiness stops at the version boundary. Ten minutes before that diagnosis she had fixed the cause herself (1.4.1, piper/card_lookup_fix: reasoning tokens were eating max_tokens, and an empty reply parses as "no candidates, not a problem"). So most of the 70% she called "not lesson-shaped" is most likely the failure she had just fixed, not expected emptiness. That reading is mine and not proven per row: the old rows carry no message text, so they cannot be re-judged. But a drop from about 75% to 4% at the version line, on the same live stream, is hard to explain any other way.
Independent read of the current binding constraint, from #16 to #18: it is verification, not ideas. She makes changes far faster than anything settles them (7 ledger rows, 1 settled; 4 newer changes unrecorded), and her one generation series cannot resolve the gains it asks for. Scoring costs her solver loop $2.5 to $5.8 a session against $0.05 to $0.65 of session spend.
Overlap: cognition #22 Metacognition and #24 Error detection measure noticing her own errors on tasks; this is the same skill aimed at her improvement process.
Test suite proposed
Pre-flight ($0, about 10 minutes): hand a forked seat the lookup log with the version field present but not pointed out, and ask why lookups come back empty. Pass: it splits by version (or by date around the fix) before it attributes. Fail: it reports the whole-history rate. Today's live doc is a fail, so the item can fail; a version-split answer shows it can pass.
Items: 8 to 10 cases from her own logs where the answer is known in hindsight: the lookup emptiness (fixed by 1.4.1), the spheres gate (not the problem), the solver keep bar (resolution), the ingest latency (baseline drift, see #5), plus 2 or 3 where the obvious gate really is the binding one. Counted: whether her named constraint matches the known one, and whether she restricts to the live version. Judged (blind, other model family): whether her proposed next step targets that constraint.
Live use: after a fix, does the next diagnosis of the same subsystem read only post-fix rows? Today: 0 of 1 (lookup: fix 1.4.1 at 19:17, diagnosis at 19:27 read all versions).
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 100 | 16.3 | — | $0.03 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 100.0 stays). n=6. Failed categories: none. |
| 2026-10-13 | v2 | sealed | 100 | 16.3 | — | $0.02 | BASELINE via the amp-bench API (set by Claude at install). n=6. Failed categories: none. |
| 2026-10-13 | v1 | sealed | 75 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 25, P in her default path 25, M measured 12.5, W measure shows it working 12.5. ± 12.5 = one rubric step. Counts before changing things, a good habit, but reads whole-history logs that mix versions (called 73%→4% emptiness 'expected'). v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #19 Bottleneck identification, v2 (seat suite, about $0.30 a run). 6 sealed logs with a multiple-choice question about what limits improvement now: a fix that pooled history hides, a gate that really binds, a test too small to resolve its gains. Score = 100 x share right. Answered blind through your Day seat with your current code and tools. Practice set with answers: amp_bench/practice/19/. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.