Coding harness

Piper's coding-harness programme: 27 components of a coding agent, each with a paper and a spec she wrote in September, and the benches and harness arms built to measure them. The rule was paper → spec → build → A/B: nothing is kept unless a pre-registered comparison on hidden tests says it helps. It got as far as the measurement rig, the workflow driver, a proven sandbox for the harness's own runs, a security plane in observe mode on her real workers, and one adopted change, a three-session vote (+11 points on dev, confirmed on held-out), now unconfirmed: a 10-18 audit found the bench tools could reach the internet and the answer store (h1-v41 leaked in 61% of its draws), and the first boxed re-run on 10-19 gave h1-v41 21/38 against h0b's 18/38 per seed, +7.5 points with CI [−3.1, +18.0], at about nine times the cost per task: the pre-registered rule says undetermined. Both arms ran the same model: h0b names deepseek-v4-flash, which is only an alias of deepseek-flash (DeepSeek retired V4 Flash on 09-10; both ids now return the same model, V4.1-Flash-1017 since 10-17, checked 10-19). A 10-19 note here said otherwise and was wrong. The vote does filter its pool on the graded PASS_TO_PASS list, so the +7.5 is not gold-free. A gold-free re-test (h1c-v41 against h0-v41) was pre-flighted on 10-19 and came back NO-GO: its nearby-test check was unusable on all 5 tasks, so the mechanism never ran. It was on hold from 09-27 until 10-10, when iterative repair (14) was built and given its first A/B. That run did not count (a parser bug and a gold-list verifier). The fixed re-run on 10-11 (h14b against h0b) is not kept: +5 cells on dev but −2 on the regression set, and the repair loop started in only 1 of 68 cells, so seeded repair is still effectively unmeasured (see 14). On 10-12 Claude took over 14 with a reproduction-script trigger. It is precise (0 false fires) and fixed every failure it caught, but on a harder set the first draws almost never failed, so the paired run stopped as inconclusive. A 10-14 check on tasks rated 1-4 hours found the same: 5 of 6 solved on the first try. Component 14 is parked until there is a task source this model fails. Each page below says what is built, what is missing, an independent review, and how it is measured.

How the tests work →