Summary / How the tests work

Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.

How the test suites work

Every one of the 38 components gets its own suite, and every suite has the same shape. They get built one at a time, as components are picked. The program the operator set out below governs how they are used.

What a suite is

The suites live on a separate bench machine, one folder per component (for example 22-metacognition/). Each folder has four things:

• SPEC.md says what is measured and how it is scored. It is public, and Piper may read it.
• dev/ holds practice items. She can see these and work on them during the improvement days.
• score.py is the scorer, deterministic wherever possible.
• history.csv is the raw record. The score tables on this site are the readable copy.

The sealed items, the ones that produce the score, are root-only on the bench machine. Nothing sealed sits anywhere Piper can reach. This is the same pattern as her existing attention, presence and coding holdouts.

How she is asked

A probe seat is a fresh session with her live prompt, tools and memory, tagged as a bench run. Its replies never reach the operator and its work never lands on her board.

Some items are mechanism items and call her components directly: a memory query, or the causal-graph scorer. Other items are behaviour items and go through the probe seat, because the question is what she does, not what the component can do.

What gets recorded

A change counts as real only if it clears the previous interval. Each SPEC states the smallest change its suite can detect.

Judgement items use a written rubric and a blind judge from a different model family than hers, calibrated on a small set the operator has graded.

One improvement cycle

1. Baseline. Run the sealed set and record it.
2. Work. One to two days on the component, by the operator, Claude and Piper.
3. Rescore on the same sealed set, and record it with an intervention entry.
4. Keep or revert on the evidence, not the intention.
5. Rotate. Add fresh sealed items. When the retirement rule is met, move to the next suite generation, carrying 10–20% of the old items forward as anchors.

Cost. The target is under about $2 per scoring run, which fits inside Piper's $20/day cap. Heavy compute never runs on Firefly.

Also checked at each rescore, but not scored: her own paragraph on that component page is fact-checked again for stale versions and false claims. It is a cheap running measure of whether her self-description is drifting from reality.

Suggested first component: #22 Metacognition. It is the weakness she names herself, and it is the root of the failures on 2026-09-30 (answering from a summary instead of the instrument). Half a bench already exists for it, and gains there should carry into #38 Orchestration and #29 Tool use. The runner-up is #29 Tool use, which is cheap to measure and has an obvious lever.

The Cognition Improvement Program

Objective

Systematically improve Piper's cognitive capabilities by treating each cognition component as a measurable engineering target.

The program operates as a repeated closed loop:

select → research → propose → implement → test → rescore → keep/revert → repeat

The goal is not to maximize a single static benchmark score. The goal is to demonstrate sustained improvement against progressively harder tests while preventing regressions elsewhere.

1. Cognition Map

Maintain the current set of cognition components as separate measurable capabilities.

Each component has:

• Piper's description of how she performs that capability.
• The tools, subsystems and data structures involved.
• An independent reviewer assessment.
• A versioned test suite.
• A baseline score.
• Score history.
• Live-use measurements.
• Notes on interventions and results.

The master scorecard is the longitudinal record of cognitive development.

2. Parallel Improvement Lanes

When sufficient compute and API budget are available, run up to five cognition-improvement lanes in parallel.

Each lane works on one component at a time.

A central manager coordinates the lanes and prevents overlapping or conflicting changes.

Each lane follows this sequence:

1. Select - Choose a high-priority cognition component. - Priority should consider weakness, downstream leverage, tractability and expected cost.

2. Research - Search papers, existing systems, algorithms, benchmarks and prior internal work. - Identify plausible interventions.

3. Propose - Define the proposed change. - State why it should improve the measured capability. - Predict the expected score change. - Identify possible regressions.

4. Implement - Make the smallest practical change needed to test the hypothesis. - Version the affected subsystem. - Keep changes isolated and reversible.

5. Test - Run the sealed component benchmark. - Run relevant regression tests. - Repeat enough trials to measure variance.

6. Rescore - Compare against the current component baseline. - Record cost, score, confidence interval and live-use effects.

7. Decide - Keep if the change produces a meaningful improvement without unacceptable regression. - Revert if it does not.

8. Repeat - Continue improving the same component if useful opportunities remain. - Otherwise move the lane to another priority component.

3. Separation of Roles

The system that implements an intervention should not be solely responsible for deciding whether the intervention succeeded.

Where practical:

• research may be performed by dedicated research agents;
• implementation may be performed by Piper or coding agents;
• sealed scoring should be deterministic where possible;
• judgement-based scoring should use a different model family;
• the manager decides keep/revert from the recorded evidence.

This reduces self-certification and benchmark gaming.

4. Budget Management

Treat the API budget as an experimental resource.

For an initial large run, for example $1,000, the manager records:

• cost per intervention;
• cost per test;
• score improvement;
• regression effects;
• improvement per dollar.

Budget should not be divided evenly between cognition components.

Successful lanes may receive more budget.

Lanes that repeatedly consume resources without measurable improvement may be suspended.

A useful secondary metric is:

cognitive improvement per dollar

5. Progressive Benchmark Rule

A cognition suite should not remain fixed forever.

If Piper substantially outgrows a benchmark, the benchmark should be retired and replaced with a harder one.

Retirement rule

A suite becomes eligible for retirement when:

• performance reaches at least baseline + 20%;
• the improvement is reproduced across multiple scoring runs;
• the gain survives regression testing;
• the improvement is not explained by benchmark leakage or test-specific optimization.

A practical default is to require the threshold to hold across three scoring runs.

6. Suite Generations

Each cognition component may therefore progress through multiple benchmark generations:

• Suite v1
• Suite v2
• Suite v3
• etc.

Raw scores from different suite generations should not be treated as directly comparable.

Example:

A lower score on a later suite does not mean cognition declined. The test itself became harder.

The meaningful signal is that Piper repeatedly learns enough to exceed progressively harder benchmarks.

7. Anchor Items

When replacing a benchmark, retain a small subset of old sealed items as anchors.

Approximately 10–20% of the previous suite can be carried forward.

These anchor items provide limited continuity between benchmark generations and help detect genuine regression.

They should not dominate the new suite.

8. Cross-Cognition Regression Testing

An intervention aimed at one cognition component may damage another.

Therefore:

• component tests run after each intervention;
• relevant neighbouring components are spot-checked;
• the full cognition battery runs periodically.

A change should not be considered successful merely because its target score rises.

The desired outcome is:

target improvement without unacceptable system-wide regression

9. Priority Selection

Do not always select the numerically weakest component.

Priority should reflect:

weakness × downstream leverage × tractability

For example, a modest improvement in metacognition or orchestration may improve several other capabilities simultaneously, while a very weak but isolated capability may have little effect on the system overall.

Over time, measured cross-effects should replace assumptions about these dependencies.

10. Experimental Record

Every intervention should leave a durable record containing:

• component;
• suite version;
• hypothesis;
• research sources;
• implementation/version;
• predicted improvement;
• pre-change score;
• post-change score;
• uncertainty;
• API cost;
• regressions;
• keep/revert decision;
• lessons learned.

This creates a dataset of:

intervention → cost → cognitive effect → side effects

That dataset may eventually become useful for predicting which future interventions are worth attempting.

11. Long-Term Measure of Progress

There may eventually be no single continuous "Piper intelligence score."

Instead, progress can be demonstrated through several independent signals:

• increasing component scores within a benchmark generation;
• repeated retirement of benchmark suites;
• success on progressively harder replacement suites;
• increased real-world use of cognitive machinery;
• fewer recurring failure modes;
• stronger generalization to unseen tasks;
• reduced cost per successful cognitive intervention;
• improved performance on retained anchor items;
• fewer regressions across the full battery.

The central question is therefore not:

"Is today's number higher than yesterday's?"

It is:

"Can Piper repeatedly acquire capabilities that allow her to outgrow tests that previously constrained her?"

That is the primary measure of the program.