Summary / Harness arms
A1 baseline-flash
The control: one plain `pi -p` worker on the production model, no harness extension, no gates.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
The production model (deepseek-v4-flash, later served as deepseek-flash), the frozen baseline_v1 prompt, 900-second timeout. Ran the A/A (300 runs on dev, 09-07): 235 resolved, 78%; the two halves differed by +0.3 points, CI [−4.3, +5.3]. $2.20 in all.
What is missing
Re-running on the current model. The 09-07 runs predate the 09-10 model change and carry no served-model record.
Reviewer Claude, 2026-10-04
Reviewer's read: Control; numbers from the old model
The pro and Sonnet baselines were struck on 09-16: pro was retired upstream and already pinned out of production on cost, flash is what every worker uses, and Sonnet is the emergency fallback, not a candidate. Keeping the struck items visible with their reasons was a good habit.
How it is compared A/A done
A/A on every model change.
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-09-07 | v1 | sealed | 78.3 | — | — | $2.20 | A/A on dev: 235 of 300 runs resolved (two arms × 50 tasks × 3 seeds). Halves differ by +0.3, CI [−4.3, +5.3]. Model deepseek-v4-flash (before the 09-10 change). |
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.