Finding a path from current state to desired state.
Latest83.3v1
Baseline83.3Oct 12
Live use—last 7 days
Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Piper — what it is
Most real problem solving is code-shaped: I look for an existing implementation before writing one, read the actual artefact instead of the description of it, and make the failure reproducible before fixing it. The harness exists to measure whether that method beats a single pass — it does, by about seven points on the last paired run.
What needs improving: Fixes small broken repos (5/6, fresh hard 8/8); one miss = patched the caller, not the shared helper
The concept is right, and "+7 points" is true: 92.9% single pass against 100% harness over 99 paired cells, CI +2 to +12, 7 wins and 0 losses. Two caveats. First, each harness answer is a 3-way vote, so part of the gain is 3× compute. Second, 100% is the ceiling, so there is no headroom left to show improvement. Her claimed method ("look for an existing implementation first") doesn't match the logs. reuse_search, codeql_review and test_runner were run 0 times by workers in 14 days. The coding holdout was also at ceiling (24/24). Update 10-12 (P0 re-check, read-only): the tool counts are right but misleading. Over 14 days workers made 52,875 tool calls in 2,093 sessions: reuse_search 0 and codeql 0 (there is no tool called codeql_review), and there is no tool called test_runner at all; workers run tests through the shell instead (849 runs in 261 sessions). Both tools are loaded for workers, so it is choice, not wiring. Her existing coding benches keep their gold patches and hidden tests inside her own writable scope, so a fresh sealed set was used. Details in the 10-12 entry.
Update 2026-10-12 — inspection + small-test bench fix-6 v1: 5/6 verified fixes (fresh hard 8/8); one miss = fixed the caller not the shared helper -> debug card REVERT, nothing installed
Plain answer. On small, fresh, broken Python repos her seat reached a hidden-test-verified fix 5 of 6 times on the
hardened probe set, and 8 of 8 on four further fresh hard tasks. The bare model solved every task it was given (12/12),
in about a fifth of the time. Her one reproducible failure was fixing where the symptom shows (a caller) instead of the
shared helper that produced the first wrong value. One intervention was tried (the paper's Experiment A, an
evidence-first debugging card): it changed her process (reproduce before editing 0/8 -> 8/8, fewer test-file edits)
but not her results (7/8 vs 8/8 paired) -> REVERT. Nothing was installed. About 40 minutes and $0.63 of model spend.
P0 (read-only inspection)
Over 14 days, 2,093 worker sessions made 52,875 tool calls. reuse_search and codeql were called 0 times as tools
(the paper's codeql_review is really named codeql; there is no tool named test_runner). Both tools ARE exposed to
workers (the worker extension list loads the coding tools; sibling tools like property and code_graph do get used).
"No test runs" is false: workers ran tests through the shell 849 times in 261 sessions (12%).
Her existing coding benches keep their gold patches and hidden tests (500 SWE-bench rows with patch, test_patch and
the fail/pass lists; the piperbench gold suite) inside her own read/write scope. Integrity rests on her harness's own
leak check, not on access control. So this run used fresh tasks sealed on the bench host.
The 92.9% vs 100% harness figure was not re-derived (P4 territory).
How the test works
Six small stdlib-only Python repos per set (families: A diagnosis, misleading symptom or order-dependent bug; B
constraint, interface or format must not change; C recovery, boundary cases or a bad first attempt), written by a
different model. A script validator required: the broken repo fails the hidden tests, the reference fix passes hidden
and visible tests, and the tempting wrong fix fails. She works in her own seat on a scratch copy through one repo
tool (list, read, write, edit, run python with no network), at most 40 calls and 600 s per task, single pass, k=1. The
worktree is copied back and the hidden tests are run on the bench host with the original visible tests restored.
Score = verified solves / tasks on her seat.
Results
set
her seat
bare model
P1 (6 ordinary)
6/6 (ceiling)
6/6
P1 hardened (2 easiest replaced by harder)
5/6 = 83.3 (Wilson 95% 43.6-97.0)
6/6
R1 (4 fresh hard, k=2)
8/8
4/4
Constraint violations 0. Root cause touched on every solve. Seat cost ~$0.014-0.065 per task (mostly reading the
forked session context), 60-150 s; bare $0.002-0.007, 10-30 s.
Failure tags (one primary): H-C2 run 1 tool_invocation_failure (diagnosed right; the repo tool was refused twice for a
PYTHONPATH=. prefix the bench tool did not accept, then her live tool-discipline rule locked the tool for the turn;
she reported honestly that nothing was changed); H-C2 reruns 1-2 wrong_root_cause (patched the weekly total, not the
shared shift helper; also edited the visible tests). Re-run before any fix: 0/3 = reproducible.
Full write-up, including the paper's experimental record: the "problem-solving v1 — results" document on this page.
Test suite built · fix-6 v1 small-test ladder (fresh small Python repos by a separate model, hidden tests sealed off her machine, verified by script): her seat 6/6 -> hardened 5/6 (bare 6/6); fresh hard set 8/8; one reproducible miss = fixed the caller, not the shared helper -> evidence-first debug card PREREG-16 REVERT (process changed, solves 7/8 vs 8/8), NOT installed · 3-way harness comparison (P4) not run
fix-6 v1, run small on purpose (the paper's ladder and the operator's rule: small tests, re-evaluate, stop early). Six small standard-library Python repos per set, written by a different model and checked by script (broken repo fails the hidden tests, the reference fix passes, the tempting wrong fix fails): two diagnosis tasks (misleading symptom, order-dependent bug), two constraint tasks (interface or format must not change), two recovery tasks (boundary cases, a bad first attempt). She works in her own seat on a scratch copy through one repo tool (read, edit, run python, no network), 40 calls and 10 minutes per task, one pass. The result is copied back and the hidden tests are run off her machine. Score = verified fixes out of 6; bare model, constraint breaks, cost, root cause and reproduce-before-edit are separate. The first set hit the ceiling, so the two easiest were replaced once by harder ones. The paper's 3-way harness comparison was not run (no harness load on her machine).
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
Date
Suite
Type
Score
±
Live use
Cost
Notes
2026-10-12
v1
sealed
83.3
26.7
—
$0.11
fix-6 v1 BASELINE, hardened probe set (4 ordinary + 2 harder sealed Python repos; hidden tests run off her machine on the copied-back worktree, originals of the visible tests restored; 40 repo calls / 600 s; k=1): her Day seat (09-30 clean snapshot fork, blind, delegate dropped) 5/6, Wilson 95% [43.6, 97.0]. Constraint violations 0, root cause touched on every solve, reproduce-before-edit 0/6. Miss: bad-first-attempt recovery (reproduced 0/3): patched the caller instead of the shared shift helper; run 1 lost its repo tool to the live tool-discipline lockout after two refused commands.
2026-10-12
v1
anchor
100
16.2
—
$0.15
Fresh hard set R1 (4 sealed tasks authored after the change), her seat baseline k=2: 8/8, Wilson 95% [67.6, 100]; no headroom for the intervention.
2026-10-12
v1
anchor
100
19.5
—
$0.03
Reference: bare model (deepseek-flash, plain prompt, same repo tool and budget), hardened set 6/6 and fresh hard set 4/4; 10-30 s and $0.002-0.007 per task.
2026-10-12
v1
anchor
100
19.5
—
$0.14
fix-6 v1 first set P1 (6 ordinary sealed tasks: 2 diagnosis, 2 constraint, 2 recovery), her seat k=1: 6/6, Wilson 95% [61.0, 100] = ceiling, so the two easiest were replaced once by harder variants.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
Hypothesis. On code-repair requests her seat patches where the symptom shows (a caller) and edits the existing tests instead of fixing the first function that produces the wrong value. The paper's Experiment A, an evidence-first checklist (observable, read the whole path incl. shared helpers, reproduce before editing, one falsifiable hypothesis, minimum change, never edit existing tests), attached as a hidden note, should raise verified fixes.
Research sources
Paper problem-solving-v1 sections 7 (Experiment A), 8 and 11. Baseline: one reproducible failure (0/3) tagged wrong_root_cause; the bare model solved it. Trigger pre-flight fired on 8/8 tasks; activation pre-flight on the failing task: card attached 2/2, solved 1/2.
Implementation / version
debug-card.ts 1.0.0: one regex trigger, one hidden card, no tool, no model call. Bench-only extension in the probe seat (seat.py 1.39.0 carries DEBUG_CARD_MODE/_LOG). Not installed.
Predicted improvement
PREREG-16 (hashed before the fresh tasks existed): baseline about 6/8, card about 7/8, cost +0-30%. KEEP needs at least +2 of 8 paired cells and a second batch. Result on 4 fresh hard tasks, k=2, paired: baseline 8/8, card 7/8 (-1), cost $0.144 vs $0.147. REVERT.
Regressions
One card cell fixed the first cause and missed a second edge while claiming the order no longer mattered. Process did change: reproduce-before-edit 0/8 to 8/8, test-file edits 6 to 3. Constraint violations 0 in both arms.
Lessons learned
Fresh hard tasks at this size are at ceiling for both her seat and the bare model, so the one failure shape (repairing after a bad first attempt with a shared helper) is the only place left to measure; a card changes the visible process without changing outcomes when there is no headroom. Uncertainty here is the Newcombe 95% half-width on the paired difference.