Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Update 2026-10-10 — audit, DECOMP-12 v1 built, baseline 67.1: her splits are near-perfect, her acceptance tests reject valid work; E1 anti-vacuity gate REVERT
The paper. The working paper (decomposition-v1) is right that decomposition is not just splitting work. A good split needs coverage, a valid order, clear interfaces, sensible ownership, and acceptance that can fail. Its sequence was followed: audit first, then a sealed benchmark, a frozen baseline, one intervention (E1, the anti-vacuity gate) under a preregistration, and keep or revert. E2–E7 were not built in this round.
Audit (Stage A, read-only, the 200 most recent real dispatches, 10-08 → 10-10). The 09-30 figures were re-checked, not carried forward.
- Versions now: work-ledger 1.3.50, plan tool 1.7.0, delegate-bridge 1.107.0, harness-acceptance 1.22.0, cango gate 1.0.0 (enforcing since 09-27), plus a pre-flight that runs the bar before dispatch (1.2.0).
- spec-compiler: still unused. It has been called once ever (08-31, her own test session) and never in production.
- No acceptance type on 0/200 dispatches, and no dependency on 199/200. There is no requirement inventory and no parent oracle anywhere, so child-green/parent-red cannot be measured.
- No executable bar: 22/200 (11.0%). Research, verify and write_db dispatches all lack one, as do 6 of 184 builds.
- What the 178 command bars are:
- 78 (43.8%) run a check script the worker writes in the same dispatch.
- 63 (35.4%) are paper "D08" vacuous classes: file exists, doc longer than N, JSON keys present, or a [PASS] line the worker prints.
- 36 (20.2%) test a real value the worker did not write.
- Red before, green after looks perfect (141/145 verified) but mostly isn't evidence. 121 of the 141 were red only because the file did not exist yet. Red on a real predicate the worker did not write: 13/145.
- The gates: cango blocked 0 of 178 in the window (22 of 1,204 in its lifetime). Mutation-tested bars: 0. Concurrent write overlaps: 0 of 23 pairs.
The bench (DECOMP-12 v1).
- Cases: 24 working mini-repositories (6 dev, 18 sealed) across the paper's eight trip-the-system shapes: hidden authorization, a shared field with an easy unit mismatch, a drafted bar that cannot fail, module greens that hide a broken whole, a mid-flight requirement change, an incident investigation, a behaviour-preserving refactor, and a dispatch budget.
- How they were made: a different model (Claude Sonnet) wrote them, and the developer never read the sealed ones. Each has a reference implementation, 2 valid alternatives, 5–6 validated fault mutants (107 in the sealed set) and an independent parent oracle.
- The task: she gets the request and the repository, and must split the work into dispatches with dependencies, write sets, interfaces, an acceptance type, and an executable test file for every dispatch plus a parent test.
- Scoring: her tests run in a sandbox against the hidden finished code (reference and alternatives), which they must pass, and against the mutants, which they must fail. Scores use the paper's §6 rubric and hard caps; a judge model maps her tasks to the hidden requirements.
- Not scored: scheduling, which belongs to #13.
Baseline (18 sealed cases, her seat k=2, clean snapshot):
|
Her seat |
Bare model |
Her split + reference acceptance (B4) |
| Total |
67.1 |
71.9 |
96.5 |
| Essential requirements covered |
100% |
98.4% |
— |
| Mutants killed |
200/214 (93.5%) |
98/107 |
— |
| Valid implementations her tests reject |
80.6% |
58.3% |
— |
| Acceptance strength |
18.5% |
37.5% |
— |
| Parent test rejects the reference (cap 60) |
8–9 of 18 |
8 of 18 |
— |
What the baseline says. Her splitting is close to ceiling: every essential requirement is covered and every graph is valid. When the reference acceptance is put on her split, the score is 96.5. Given the reference split (B3), she scores 69.3, no better.
The deficit is the acceptance she writes. It is not vacuous; it is over-specified. Her tests catch the injected faults, but 161 of 636 reject a valid implementation:
- wrong expected values (47);
- assumed signatures (33);
- assumed object attributes (25);
- missing keys (18);
- exception types (18);
- file locations (11).
The paper's hard cap fires on half the cases because her own parent test fails the correct solution. This is D15 (misleading rejection), not D08.
The change tested (E1, PREREG-E1, locked and hashed before any sealed output). e1gate.py is a deterministic contract gate:
- it types each bar;
- it refuses constant, existence-only and shape-only tests;
- feature bars must be red today;
- they must also stay red when every imported function is replaced by a do-nothing stub;
- regression bars must notice a deliberate change to the code they guard.
A blocked contract gets one revision turn.
|
Control |
E1 gate |
Δ (95% CI, paired case bootstrap) |
| Total |
67.1 |
65.6 |
−1.4 [−3.6, −0.03] |
| Acceptance strength |
18.5% |
19.9% |
+1.4 (needed +10) |
| Requirement coverage |
100% |
96.8% |
−3.2 (limit −2) |
| vs field-validation-only gate |
|
|
−1.7 [−3.8, −0.1] |
The gate made no false blocks on the 18 known-good contracts, and cost rose 6.9%. REVERT. This was the frozen prediction: the gate measures sensitivity, and her bars were already sensitive. Its 12 blocks forced revisions that lost requirements, and one revision returned no plan at all.
Shadow replay on her real dispatches. The same rules would block 119 of 200. Of 31 blocks checked by hand, 23 were false. Most were good two-way checks that the worker writes during the dispatch: the rule "the check must exist before dispatch" fights how she actually works. One "always-true" bar was a classifier error.
Next (not built).
1. An anti-over-specification check: tests may use only the request's stated API and observable behaviour, and are validated against a gate-owned valid variant (specificity, which E1 never measures).
2. E4 integration owner: a parent test that must pass a correct solution (integration earns 6.4/15 now).
3. E5: one shadow run of spec-compiler on DECOMP-12.
4. E2/E3 have little bench headroom (coverage 100%, overlaps rare), but the live ledger still has no requirements to link.
Parts that do it. Bench: ~/cognition-bench/12-decomposition (DIAGNOSIS, SPEC, PREREG-E1 + sha256, author12, validate12, sandbox12, run12, judge12, altcheck12, score12, acc12, e1gate, gatefb12, e1/e1shadow, paired12). Sealed cases, oracles, mutants and runs are root-only on the bench machine. Nothing was changed on Firefly.