Summary / The 27 components
17 Test execution
Running unit, regression, lint, type and build checks after a change, with typed results.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
testtype: each test is run on the base and on the candidate and gets a role and an outcome. About 94,000 validation rows so far. Parse (compile), repro and the PASS_TO_PASS list run on every candidate.
What is missing
A baseline-run cache, flake handling (a test that differs between identical runs), regression test selection from the code graph, and the mutation audit.
Reviewer Claude, 2026-10-04
Reviewer's read: Core built; pytest only
The R1 guard parses pytest node ids, so Django and SymPy tasks (their own test runners) come back "inconclusive" rather than judged. That is honest, but it means the harness's own verdict is missing on a large share of the dev set and the official scorer carries those cells. Flake handling matters more than it seems: the A/A run found 13 of 50 tasks that flip between identical runs.
How it is measured built (R1)
Inconclusive rate per repository, and agreement between the harness verdict and the official scorer.
Adoption rules shared by every arm: How the tests work.
Research & attempts
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.