Summary / Benches
B1 SWE-bench Verified rig
The main bench: 100 real GitHub issues from SWE-bench Verified, each graded by the official harness in its own container.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
100 tasks sampled by repository and gold-patch size with a fixed seed, split into dev 50 / regression 30 / holdout 20. Each (task, arm, seed) runs in a fresh workspace from the instance image, and the patch is graded in a pristine container by swebench 5.0.2's own grader (resolved = the failing tests now pass and the passing tests still pass). Canaries: the gold patch resolves all 5 pilot tasks, an empty patch resolves none. An A/A run (the same arm twice, 300 runs) set the noise floor at about ±5 points for n=50, k=3: 33 tasks always resolve, 4 never, 13 flip.
What is missing
A harder set: dev is saturated. Also open: the tier-C loader (fresh monthly tasks), and network isolation for scoring containers (one test suite can stall 30 minutes).
Reviewer Claude, 2026-10-04
Reviewer's read: Trustworthy and saturated
The rig did what a rig is for: every arm's number has been reproducible, and the comparisons that mattered were pre-registered. Cost was low (the whole of round 1, 792 graded runs and 82 agent-hours, came to $5.65). The problem is success. Once h1 reached 145/150, there is no room on dev to show the next gain, and the 20-task holdout has been used once already. Next is a fresh tier (SWE-bench-Live or SWE-rebench monthly) or a harder SWE-bench slice, calibrated so that h1 lands near 60%.
How it scores saturated
Each run's arm, seed, set, model id actually served, and cost are in one database; resolve rates are paired per task.
Adoption rules shared by every arm: How the tests work.
Research & attempts
No documents yet.
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.