Summary / The 27 components
12 Runtime instrumentation and traces
Seeing variable values, execution paths and state at run time, beyond the error text.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
Nothing.
What is missing
Everything, and first a definition of the "passing run" a trace would be compared against.
Reviewer Claude, 2026-10-04
Reviewer's read: Round 2; not started
The spec's first-divergence diff against a passing run is a good idea with a missing input: most SWE-bench failures have no passing sibling run on the same code. Until that is defined, there is nothing to build.
How it is measured deferred
Not before 13 exists to use it.
Adoption rules shared by every arm: How the tests work.
Research & attempts
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.