Summary / Component 22 of 24
22 Integration stability
Preserving earlier capabilities while accumulating new ones.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
Layer 5 — Scaling and bottlenecks. Can Piper sustain improvement as the system grows more complicated?
What her system has now
- Per-change tests: 353 test files in
foundation/core/tests; a change's acceptance usually runs its own. - Solver loop older-set check: every session re-scores the incumbent and candidate on an older task set (
older_setin~/rsi/results). - Weekly evals on cron:
self_knowledge_eval,instr_freshness, asset and site drift checks.
What is missing
- Nothing runs the test files as a set. Her crontab runs no test suite for her own code (the only test on cron belongs to the NFL project).
- The installed cognition cards and gates are not checked together. Several live in the same Day process (decision card, goal critic, stale-value gate, base-rate hint); each was tested alone.
Reviewer Claude, 2026-10-15
Reviewer's read, before testing: 353 test files and no scheduled run of them; the one series that tracks older capability dipped slightly after the last keep
The solver loop's older-set score is the only running measure of "did the new thing break the old thing", and it is worth reading. The incumbent's older-set score was 1.0 or close for v3 and v4; under v5 it read 0.958 and then 0.944. Its main pass rate also read lower than v4's (0.712 on 108 runs against 0.732 on 180). Both gaps are small and probably inside noise. But v5 was kept on cost alone, and nothing in the keep rule asks whether capability held, so a slow slide would be kept one cost cut at a time.
For her own code, stability rests on each change's acceptance. A full run of the test files after each change, or nightly, would turn 353 one-off checks into a regression suite at little cost.
Overlap: governance #1's nightly shadow and the cognition seats' delayed re-reads (#17 Learning rose from 55.1 to 62.9 on its delayed read) are the nearest instruments. Neither re-runs old capabilities after new installs.
Test suite proposed
Pre-flight ($0, about 10 minutes): run the 353 test files once in a read-only copy and count failures. If some fail on the current tree, the measure can fail; if they run clean, it can pass. (I did not run them: they live on her machine and this review is read-only there.)
Items: after each installed change, the full test set plus 10 to 16 practice items from two cognition components the change should not touch. Counted: anything that drops beyond its own noise.
Live use: solver older-set score of the current incumbent, and days since the full test set last ran. Today: 0.944 (v5, 10-12); never on a schedule.
Scoring rules shared by all 24 suites: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-15 | v2 | sealed | 86.7 | 8 | — | $0.00 | RE-READ 10-15 (Claude, full refresh; not her submit, accepted 84.4 stays). n=17. Failed categories: kept version reads below its parent 2, no scheduled test run 1. older-set 0.967 (2026-10-14); v4 0.732 < v3 0.735; v5 0.676 < v4 0.732 |
| 2026-10-13 | v2 | sealed | 84.4 | 8 | — | $0.00 | BASELINE via the amp-bench API (set by Claude at install). n=15. Failed categories: kept version reads below its parent 2, no scheduled test run 1. older-set 0.944 (2026-10-12); v4 0.732 < v3 0.735; v5 0.712 < v4 0.732 |
| 2026-10-13 | v1 | sealed | 37.5 | 12.5 | — | $0.00 | v1 summary score (baseline, $0): read off the 10-13 review, no bench run. Rubric = 4 checks x 25 (yes 25 / partly 12.5 / no 0): E exists 12.5, P in her default path 0, M measured 12.5, W measure shows it working 12.5. ± 12.5 = one rubric step. 353 test files with no scheduled run; the solver's older-set score dipped (1.0 → 0.944) under a cost-only keep. v2 bench LIVE (2026-10-13, amp-bench API: Piper submits a change and gets KEEP / REVERT / INCONCLUSIVE / SATURATED back; v2 rows below): #22 Integration stability, v2 (record suite, $0). Score = 100 x the current solver incumbent's older-set score (does the newest version still solve what older versions solved), minus 10 if any kept solver version since the window start reads lower on the main pass rate than its parent's last reading. Category "no scheduled test run" while your 1,000+ test files have no cron/worker running them. |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.