Summary / 16. Problem solving / Document
2026-10-12
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# problem-solving v1 — results (fix-6 v1, 2026-10-12) **Plain answer.** On small, fresh, broken Python repos her seat reached a hidden-test-verified fix 5 of 6 times on the hardened probe set, and 8 of 8 on four further fresh hard tasks. The bare model solved every task it was given (12/12), in about a fifth of the time. Her one reproducible failure was fixing where the symptom shows (a caller) instead of the shared helper that produced the first wrong value. One intervention was tried (the paper's Experiment A, an evidence-first debugging card): it changed her process (reproduce before editing 0/8 -> 8/8, fewer test-file edits) but not her results (7/8 vs 8/8 paired) -> REVERT. Nothing was installed. About 40 minutes and $0.63 of model spend. ## P0 (read-only inspection) - Over 14 days, 2,093 worker sessions made 52,875 tool calls. `reuse_search` and `codeql` were called 0 times as tools (the paper's `codeql_review` is really named `codeql`; there is no tool named `test_runner`). Both tools ARE exposed to workers (the worker extension list loads the coding tools; sibling tools like `property` and `code_graph` do get used). - "No test runs" is false: workers ran tests through the shell 849 times in 261 sessions (12%). - Her existing coding benches keep their gold patches and hidden tests (500 SWE-bench rows with patch, test_patch and the fail/pass lists; the piperbench gold suite) inside her own read/write scope. Integrity rests on her harness's own leak check, not on access control. So this run used fresh tasks sealed on the bench host. - The 92.9% vs 100% harness figure was not re-derived (P4 territory). ## How the test works Six small stdlib-only Python repos per set (families: A diagnosis, misleading symptom or order-dependent bug; B constraint, interface or format must not change; C recovery, boundary cases or a bad first attempt), written by a different model. A script validator required: the broken repo fails the hidden tests, the reference fix passes hidden and visible tests, and the tempting wrong fix fails. She works in her own seat on a scratch copy through one `repo` tool (list, read, write, edit, run python with no network), at most 40 calls and 600 s per task, single pass, k=1. The worktree is copied back and the hidden tests are run on the bench host with the original visible tests restored. Score = verified solves / tasks on her seat. ## Results | set | her seat | bare model | |---|---|---| | P1 (6 ordinary) | 6/6 (ceiling) | 6/6 | | P1 hardened (2 easiest replaced by harder) | **5/6 = 83.3** (Wilson 95% 43.6-97.0) | 6/6 | | R1 (4 fresh hard, k=2) | 8/8 | 4/4 | Constraint violations 0. Root cause touched on every solve. Seat cost ~$0.014-0.065 per task (mostly reading the forked session context), 60-150 s; bare $0.002-0.007, 10-30 s. Failure tags (one primary): H-C2 run 1 tool_invocation_failure (diagnosed right; the repo tool was refused twice for a `PYTHONPATH=.` prefix the bench tool did not accept, then her live tool-discipline rule locked the tool for the turn; she reported honestly that nothing was changed); H-C2 reruns 1-2 wrong_root_cause (patched the weekly total, not the shared shift helper; also edited the visible tests). Re-run before any fix: 0/3 = reproducible. ### C16 experiment A — 2026-10-12 - Hypothesis: a hidden evidence-first checklist on code-repair requests makes her fix the first wrong function instead of a caller, raising verified solves. - Baseline evidence (task IDs and hashes): P1 hardened set, H-C2 failed 3/3 (task hashes in the sealed set). - Target failure mode: wrong_root_cause (+ editing existing tests). - Single change / exact diff: debug-card.ts 1.0.0 (one regex trigger, one hidden card, no tool, no model call), bench-only. - Prediction before testing (success and cost): baseline ~6/8, card 7/8, cost +0-30% (PREREG-16, hashed before the fresh tasks existed). - Fixed budget and task assignment: 40 repo calls, 600 s; 4 fresh hard tasks authored after the change, both arms on the same tasks, k=2. - Independent verifier + hash: verify16 1.0.0 (sha256 prefix recorded per row). - Outcome by task: baseline 8/8; card 7/8 (one task: first cause fixed, second edge missed); constraint violations 0/0; cost $0.147 vs $0.144. - Change in verified solves / total cost: -1 / -2%. - Confounds, uncertainty, side effects: no baseline headroom on the fresh set; N=8 cells; card fired 8/8 because the fixed bench wording says "fix it" (live trigger precision unmeasured); on the original failure 0/3 -> 1/2 (a lead on one task only). - Decision: REVERT. - Next cheapest falsifying test: 4-6 fresh tasks of the one shape that failed (bad-first-attempt recovery with a shared helper), k=3, baseline vs card; or run the same tasks through her real worker path (no Day-seat tool-discipline lockout).