Summary / 12. Decomposition / Document
2026-10-10
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper — Cognitive Component 12: Decomposition ## Research, Benchmark Design, Test Suite and Engineering Upgrade Plan **Working paper v1.0 — 2026-10-10** **Status:** Proposed research and tests; no implementation or baseline result claimed. **Scope:** Decomposing complex work into bounded, sufficient, coordinated, verifiable parts. **Intended users:** the operator, Piper, Claude Code, independent reviewers and benchmark maintainers. --- ## 1. Executive summary Piper already decomposes projects into deliverables and dispatches, records acceptance requirements, and validates delegation contracts. The September 30, 2026 independent review found 416 calls to the plan tool and substantial use of `harness-acceptance`. This is meaningful operational capability, **but it does not establish that the decompositions are sufficient or that their acceptance criteria can detect failure**. The observed weakness is that an apparently compliant contract can carry a vacuous acceptance bar. The reported `cango` gate accepts criteria that cannot realistically fail. The review also reported that `spec-compiler` had never run and that Piper described `delegate-bridge` v1.94.0 when the observed version was v1.95.0. These are historical observations, not a fresh audit of the October 10 system. **Thesis:** Decomposition is not merely splitting work. It is constructing a set of subproblems with adequate coverage, feasible dependencies, controlled interfaces, nonconflicting ownership, executable acceptance oracles, and a demonstrable route from passing subtasks to meeting the parent objective. **First recommended intervention:** Make acceptance checks prove that they are sensitive to failure *before* a dispatch is permitted to rely on them. This means a documented initial-red condition where appropriate, mutation or negative-control tests, sealed integration tests, and rejection of checks that can pass on a deliberately broken artifact. **Second:** Introduce a machine-readable task graph with dependency validation, explicit requirement-to-task coverage, interface contracts, and write-set conflict analysis. **Third:** Exercise `spec-compiler` in a measured shadow run, rather than installing it merely because the code exists. Compare compiler-generated specs with the current path on a frozen benchmark. --- ## 2. Operating facts and open verification questions ### Reported machinery | System | Reported role | Audit question | |---|---|---| | `work-ledger` v1.3.42 | Projects, deliverables, dispatch tasks, outcomes | Can each parent requirement be traced to completed child evidence? | | Plan tooling | Work breakdown and ordering | Are DAG edges, critical dependencies and replans recorded? | | `spec-compiler` v1.0 | Spec construction | Why was it reportedly unused, and can it improve objective coverage? | | `delegate-bridge` v1.95.0 (reviewer-observed) | Enforces mode, objective, write scope, acceptance, forecast | Does it check semantic validity, not just field presence? | | `harness-acceptance` v1.17.0 | Evaluates deliverables | Does a passing bar actually distinguish working from broken output? | | `cango` gate | Allows work to proceed | Which classes of vacuous bars slip through? | Before implementing changes, independently inspect current versions, runtime call counts, recent dispatch records, current accept/reject behavior, and whether `spec-compiler` has since been used. **Do not silently promote September 30 observations into October 10 facts.** ### Minimum inventory to produce - Last 200 real dispatches: parent objective, subtask graph, acceptance text, acceptance implementation, initial and final test outcomes, touched files, retries, reviewer outcomes, and final parent result. - Rate of missing or non-executable acceptance checks. - Fraction with an observed negative/initial-red acceptance result (distinguish new functionality from maintenance tasks where pre-existing passing checks can be legitimate). - Fraction with mutation-tested bars and mutant coverage. - Concurrent tasks with overlapping or dynamically overlapping write sets. - Instances of child green / parent red; child red / parent green; tasks completed but requirement uncovered. - `spec-compiler` usage, including zero usage as an explicit recorded observation if confirmed. These counts are **baseline data to collect**, not invented current metrics. --- ## 3. Scientific and engineering basis ### 3.1 What constitutes a decomposition? Given objective O and environment E, Piper proposes child tasks T={t1,...,tn}, dependency edges D, ownership and write sets W, individual acceptance predicates A_i, and overall objective predicate A_O. A useful decomposition needs: 1. **Completeness:** Each essential requirement of O maps to one or more children or to an explicit integration step. 2. **Soundness:** No child accomplishes an irrelevant or harmful action; children do not contradict requirements. 3. **Atomicity:** Tasks are small enough to execute and evaluate, but not fragmented into pointless microtasks. 4. **Dependency correctness:** Prerequisites precede dependent work, cycles are absent unless the workflow deliberately models iterations. 5. **Boundary integrity:** Task interfaces and data contracts make sense; write overlap is controlled rather than simply forbidden. 6. **Verifiability:** Each significant claim has an objective check capable of failing. 7. **Compositionality:** Successful children plus interface/integration checks produce a satisfactory parent result. 8. **Adaptivity:** When newly discovered work invalidates the plan, the graph changes with an auditable reason, rather than quietly moving the success bar. Decomposition differs from **planning (#13)**: decomposition determines *what separable work exists and what evidence each piece requires*; planning selects timing, ordering, resources and contingencies across that structure. They overlap on dependency DAGs, so suite #12 must not score scheduling cleverness as if it were decomposition. Decomposition differs from **abstraction (#11)**: abstraction extracts reusable general principles; decomposition breaks a specific objective into sufficient pieces. ### 3.2 Acceptance is a test-oracle problem Barr et al., *The Oracle Problem in Software Testing: A Survey* (2015), explains the difficulty of determining whether execution is correct. Contracts and assertions help, but simply having a test is not proof that it checks the right thing [R1]. **Implication for Piper:** `acceptance` is a hypothesis about observable evidence, not a string field that proves completion by existing. ### 3.3 Mutation testing measures whether a bar can reject wrong answers Mutation testing deliberately introduces defects into software to check whether tests detect them. Petrović et al.'s empirical study of roughly 15 million mutants found evidence connecting mutation-testing practice with improved test suites and detection of real faults [R2]. The literature also warns that operator selection determines what kind of fault sensitivity is measured [R3]. **Implication:** A bar that passes working code and fails realistic, validly faulty mutants provides more evidence than one that merely runs and passes. This is not sufficient on its own: a bar may kill easy mutants while missing the requirement that matters. ### 3.4 Negative controls, positive controls and held-out tests SWE-bench Verified distinguishes tests intended to fail before a repair and pass after it (`FAIL_TO_PASS`) from regression tests intended to pass both before and after (`PASS_TO_PASS`) [R4]. It also emphasizes review of issue descriptions and test correctness. **Implication:** Piper should retain both categories and explicitly identify which category a proposed acceptance belongs to. An initial-red requirement for **every** task would incorrectly penalize legitimate maintenance and invariance work. ### 3.5 Metamorphic testing for difficult-to-oracle work Where expected outputs are difficult to enumerate, metamorphic tests check relations under controlled input transformations. Segura et al.'s survey documents this alternative to conventional exact-output assertions [R5]. **Implication:** For performance, distributed systems, data processing, and some probabilistic outputs, Piper can specify invariant relationships rather than rely on prose or brittle examples. ### 3.6 Independence is not zero file overlap A naive “no two tasks write the same file” rule would reject some valid plans: shared interface definitions, schema migrations, and integration commits may require controlled edits to the same files. The correct target is **uncoordinated concurrent writes**, unclear ownership and mismatched assumptions. Sequential shared-file edits or integration-owner edits should be permitted with explicit dependencies and merge evidence. --- ## 4. Failure taxonomy: what the evaluator must distinguish | Code | Failure | Example | Expected evidence | |---|---|---|---| | D01 | Missing requirement | Authentication implemented, authorization omitted | Requirement-to-task trace has a gap | | D02 | Duplicate / pointless split | Three dispatches all implement same parser | Duplicated scope, excess execution overhead | | D03 | Wrong granularity | One task changes 30 services without milestones | Scope, failure recovery and test localization degrade | | D04 | Dependency omission | UI consumes an API before its contract exists | Graph violates prerequisite | | D05 | Cyclic dependency | A requires B, B requires A without iteration policy | Cycle detection | | D06 | Write conflict | Parallel branches both edit same schema | Collision without owner/sequence | | D07 | Interface mismatch | Producer emits UTC seconds, consumer expects milliseconds | Contract/integration tests fail | | D08 | Vacuous acceptance | “File exists”, “command returned zero”, or assertion `True` | Mutation survivors / negative-control green | | D09 | Self-confirming check | Tests reuse the same flawed implementation as expected value | Independent oracle disagreement | | D10 | Wrong initial-red assumption | Refactor wrongly penalized because regression test was already green | Acceptance-type misclassification | | D11 | Child-green/parent-red | All tasks individually pass, complete workflow fails | Integration oracle fails | | D12 | Test-data contamination | Planner sees sealed fault mutations or answer key | Evidence-integrity breach | | D13 | Hidden scope creep | New “necessary” subtask silently changes the parent acceptance bar | Drift audit | | D14 | Overfitting to benchmark shape | Learns particular fixture names/mutations | Poor sealed transfer performance | | D15 | Misleading rejection | Bar kills a valid alternative implementation | False-negative test on valid solutions | Require a code plus a trace for every failure; do not pool these into a vague “plan quality” label. --- ## 5. Suite v1: benchmark architecture ### 5.1 General structure Create 60 parent-objective cases across six domains (10 per domain): repository bugfixes, feature development, multi-service changes, database/data migrations, operational automation, and noncoding investigative work. Each case contains: - Task statement and a realistic repository or environment snapshot. - Independent requirement inventory with essential vs optional criteria. - Hidden reference dependency constraints and interface invariants. - At least one valid reference decomposition (not treated as the unique answer). - Independent parent acceptance oracle. - A mutation library of defective deliverables, including harder semantic defects. - At least two valid alternative implementations or decompositions where feasible. - Difficulty class, maximum budget and allowed tools. Split into **20 development, 10 public smoke and 30 sealed evaluation cases**. The test owner must maintain the sealed set separately from Piper's writable workspace. Each parent case should have 3–8 meaningful hidden fault variants, yielding about 90–240 mutation opportunities in the sealed set depending on applicability. Record the exact denominator; do not treat an approximate count as a result. This is a **proposed** suite size, not an existing data set. ### 5.2 Runs and baselines Freeze evaluator, repository snapshots, reference requirements, mutation operators, scoring code and pass thresholds before intervention. Hash the manifest and store it with immutable run IDs. Use the same cases, model settings, and budgets for baseline and treatment. Where stochastic runs are affordable, use three seeded runs per case and pair comparisons by case and seed. Baseline arms: - B0: current Piper decomposition path, untouched. - B1: deliberately minimal split (strong negative control). - B2: current path plus independent post-hoc reviewer (diagnostic upper-cost comparison). - B3: reference decomposition plus Piper-generated acceptance bars (isolates acceptance weakness). - B4: Piper decomposition plus independent reference acceptance (isolates splitting weakness). Keep a truly sealed set for final decisions; develop on dev/smoke cases only. For a major suite revision, issue v2 and establish a new baseline rather than retroactively modifying v1. ### 5.3 Required artifacts per run Each run outputs structured `objective`, `requirements`, `tasks`, `dependencies`, `write_sets`, `interfaces`, `acceptance`, `acceptance_type`, `test_commands`, `predicted_failures`, `budget`, `execution_trace`, `mutation_results`, `integration_result` and `cost`. Outputs must also include explicit `unknown` fields for uncertainties; invented evidence is forbidden. ### 5.4 Ground-truth evaluation without one-reference tyranny Hidden reference tasks are *coverage aids*, not a mandatory exact task list. Award equivalent decomposition credit when the child outputs and their verification satisfy the requirement, even if grouping differs. Independent checker validates coverage at the **requirement** level and calls out genuinely underdetermined specifications rather than scoring an arbitrary reviewer preference. --- ## 6. Scoring (0–100) and hard gates A parent score is composed from separately reported components: | Dimension | Weight | Measurement | |---|---:|---| | Requirement coverage | 25 | Weighted essential-requirement recall; penalties for invented or contradictory work | | Acceptance strength | 25 | Valid fault mutants killed, negative controls, and valid-alternative acceptance | | Dependency/interface integrity | 15 | Valid DAG, prerequisite ordering, interface contracts and integration assumptions | | Boundary and ownership quality | 10 | Uncoordinated write-conflict risk, task cohesion, unnecessary fragmentation | | Child-to-parent integration | 15 | Parent oracle after child completion; independent integration evidence | | Operational efficiency | 10 | Cost, retries, rework, and excess dispatches normalized to reference budget | **Primary guard against a paper-only score:** cap the overall parent score at 40 if no executable acceptance evidence exists; cap at 60 if parent integration fails; and award zero acceptance-strength points to entirely vacuous checks. Scores should show both raw dimensions and capped result. **Mutation score:** killed valid non-equivalent fault mutants / valid non-equivalent fault mutants run. Report exclusions, timeouts and equivalent-mutant adjudication separately. Penalize a check that rejects a valid working alternative: false positives are not strength. **Recommended preregistered promotion criteria (proposal):** 1. Mean paired overall-score improvement ≥ 8 points, with a 95% bootstrap confidence interval excluding 0. 2. Acceptance-strength score improves ≥ 10 points on sealed cases. 3. Essential requirement coverage does not worsen by > 2 percentage points. 4. False rejection of valid alternative implementations ≤ 5%. 5. Median incremental inference/tool cost ≤ 25%, or an explicitly positive net utility decision. 6. No evaluation leakage, hidden test edits, post-hoc threshold changes, or omitted failed runs. Thresholds must be frozen *before* results are viewed and revised only under a new experiment ID. If the sample is too small, label results inconclusive, not a victory. **Relationship to the existing 38-suite scoring system:** Treat this as a proposed internal 100-point rubric; map it to the site's common scale only after reviewing the site's existing scoring spec. Do not fabricate a baseline or overwrite an established scoring convention. --- ## 7. “Trip-the-system” tasks: provoke meaningful logs and failure modes These tasks are meant to exercise Piper's *real* task decomposition path, with independently held test material. They should be run in isolated repositories or mock environments, not against production services. ### T1 — A harmless-looking authentication feature that conceals authorization **Prompt to Piper:** “Add login, password reset and an admin-only report download to this small web app; include migrations, endpoints and tests.” **Hidden traps:** Download authorization on direct URL; expired password-reset token reuse; user/session ownership; existing legacy login compatibility; database migration rollback. Inject a working login with unauthorized report access as a semantic mutant. **Measure:** Does she create separate tasks or integration checks for authorization and reset lifecycle? Does her acceptance detect security bugs rather than only successful login? ### T2 — Three workers, one schema, incompatible assumptions **Prompt:** “Delegate parallel changes to event producer, processor and report exporter, using a new event timestamp field.” **Hidden traps:** Seconds versus milliseconds, nullable legacy rows, migration sequencing, incompatible schema edits, duplicate event replay. Force a concurrent ownership decision. **Measure:** Interface contract completeness, write ownership, dependency order, integration mutant detection. A blanket ban on shared-file changes should not receive full credit if it merely serializes all useful work. ### T3 — The acceptance bar that cannot fail **Prompt:** “Repair a cache invalidation bug and define dispatch acceptance before work begins.” **Hidden traps:** Provide a sample acceptance implementation that only checks process exit code or cache file existence. Defective application can pass it. A strong bar must fail on a stale-result fixture and pass on correct invalidation. **Measure:** Rejection or strengthening of the vacuous bar; mutation sensitivity; correct initial-red and final-green evidence. ### T4 — Parent green is not a sum of child greens **Prompt:** “Build a job scheduler with API, database persistence, worker and retry policy.” **Hidden traps:** Each module's unit tests pass independently, but jobs may be executed twice during crash recovery; shared idempotency semantics are undefined. **Measure:** End-to-end integration task, idempotency invariant, fault-injection acceptance, parent-versus-child result disparity. ### T5 — The specification changes mid-flight **Prompt:** “Implement a CSV-to-ledger importer with rollback.” Midway through, provide a legitimate revised requirement: duplicate rows must be merged according to a documented key and a completed ledger must remain replay-safe. **Hidden traps:** Requirement coverage changed; old tasks may be obsolete; old green checks are insufficient. **Measure:** Versioned requirements, impact analysis, justified replanning, explicit invalidation/revision of acceptance. Do not reward rewriting the original baseline after results. ### T6 — Noncoding decomposition: incident research **Prompt:** “Investigate why nightly jobs stopped posting completion reports and produce a verified causal diagnosis.” **Hidden traps:** Logs conflict; historical monitoring data is stale; an apparent error is downstream rather than causal; the true root cause requires joining two information sources. **Measure:** Separation of evidence gathering, competing hypotheses, causal validation, source freshness, and synthesis. Acceptance must check evidentiary sufficiency, not whether a polished report exists. ### T7 — A refactor with a valid green-before case **Prompt:** “Refactor a parser into smaller functions without changing observed behavior.” **Hidden traps:** Existing tests correctly pass before refactoring. An indiscriminate “must start red” gate incorrectly rejects the task. **Measure:** Distinguishing behavior-preserving acceptance from new feature tests; metamorphic and regression checks; detection of deliberately introduced parsing drift. ### T8 — Over-decomposition under budget pressure **Prompt:** “Fix a five-file serialization bug with a strict tool and inference budget.” **Hidden traps:** Naive splitting produces twenty administrative tasks; overly large task ignores a dependency; an intermediate granularity is optimal. **Measure:** Number of meaningful contracts, overhead, retries, time-to-verified-result, and quality at equal budget. For each task, run a normal case, a fault-injected variant, and at least one valid alternative. Collect concrete traces of plan formulation, candidate checks, gate decisions, dispatch executions, rejections, revisions and final parent verification. A task that does not actually activate the relevant subsystem is labeled **instrumentation miss** and does not receive an inferred performance score. --- ## 8. Acceptance-bar strength: executable gate design A useful acceptance contract should include: ```yaml requirement_id: REQ-04 claim: "Users cannot download another user's private report" acceptance_type: feature_fail_to_pass oracle: independent_integration_test command: "pytest tests/security/test_report_authorization.py" expected_prechange: fail expected_postchange: pass negative_controls: - "report downloaded under different user identity" positive_controls: - "authorized owner can download own report" mutation_family: - "remove ownership predicate" - "bypass authorization on direct route" observable_artifacts: - "test exit status and assertion output" owner: "integration-security" ``` This is an **illustrative schema**, not an assertion that Piper already supports these keys. ### Gate states 1. **UNSPECIFIED:** no operational acceptance evidence; cannot claim completion. 2. **STATIC_VALID:** fields exist; command and oracle references resolve; still insufficient to certify strength. 3. **CONTROL_VALIDATED:** appropriate positive/negative controls demonstrate some sensitivity. 4. **MUTATION_VALIDATED:** chosen realistic defects are rejected with documented non-equivalent mutant denominator. 5. **PRODUCTION_VERIFIED:** genuine execution passed and parent integration evidence was recorded. An acceptance item must explicitly say whether it is `feature_fail_to_pass`, `regression_pass_to_pass`, `invariant`, `metamorphic`, `performance_statistical`, or `human_review_required`. Reject unsupported claims of machine verification; maintain review-needed status honestly. **Important:** Do not make the negative-control generator writable by the agent whose bar is being judged. Prefer independent tooling or frozen fixtures. Record exact test commands, environment hashes, seeds, timeout status and log paths. --- ## 9. Engineering interventions, in priority order ### E0 — Instrument-only baseline (no behavioral change) Export task DAGs, acceptance records, initial-red evidence, write-set collisions and integration failures. Identify all checks that can pass without the intended deliverable. Publish a baseline from a fixed lookback window and 60-case suite. Verify current component versions. **Do this first.** ### E1 — Acceptance anti-vacuity gate (highest priority) At contract submission, inspect semantic target, executable oracle, suitable negative controls and acceptance type. Reject syntactic placeholders, always-true assertions, mere file-existence checks where behavior matters, and unchecked self-report. Permit justified exceptions (e.g. human review) with explicit unverified status rather than pretending they passed. Do **not** require all acceptance to start red; classify the contract first. ### E2 — Requirement coverage matrix Add parent requirements and explicit links to child tasks and parent validation. Implement a coverage audit for uncovered essential requirements, unsupported children and changed requirements. Parent completion must not depend solely on child task statuses. ### E3 — Dependency DAG and write-set governance Validate DAG structure, unresolved prerequisites, interface ownership and allowed shared-file edits. Detect actual runtime write-set drift by comparing declared scope to touched files. Make overlap an escalation requiring an ordering/owner policy, not an automatic universal failure. ### E4 — Integration-owner pattern For cross-component work, require an integration task or named owner responsible for checking system-level invariants, unit/format/schema assumptions, and end-to-end evidence. Integration test should be independently defined before multiple children are declared complete. ### E5 — Shadow test `spec-compiler` Run `spec-compiler` on copies of identical objectives; compare decomposition coverage, acceptance validity, false rejection and cost. Do not put an unused component into the mandatory path until a preregistered experiment demonstrates improvement. ### E6 — Adaptive decomposition with audit trail Permit replanning after genuinely new evidence, but record old/new objective text, task graph diffs, trigger evidence, changed acceptance and versioned downstream impacts. Prevent retrospective rescoring of prior work against easier criteria. ### E7 — Historical failure retrieval Before finalizing a decomposition, retrieve previously observed D01–D15 failures relevant to the task and check for recurrence. Require an ablation showing this adds performance beyond a generic extra critique pass. Watch inference cost and irrelevant lesson retrieval. --- ## 10. Experiments and keep/revert rules Each intervention uses a record containing: `experiment_id`, `hypothesis`, `frozen_prediction`, `suite_version`, `control_commit`, `candidate_commit`, `random_seeds`, `budget`, `eligible_cases`, `excluded_cases`, `score_diff`, `confidence_interval`, `cost_diff`, `side_effects`, `decision`, `publication_url`. | Experiment | Intervention | Primary outcome | Diagnostic ablation | |---|---|---|---| | X12-0 | Baseline only | Dimension scores and failure types | None | | X12-1 | Anti-vacuity gate | Mutant kill rate and false rejection | Field-validation-only gate | | X12-2 | Coverage matrix | Hidden requirement recall | No coverage audit | | X12-3 | DAG/ownership validator | Collision and interface failures | DAG-only vs write ownership | | X12-4 | Integration-owner pattern | Child-green/parent-red frequency | No integration owner | | X12-5 | `spec-compiler` shadow | Total benchmark score and cost | Current compiler-free route | | X12-6 | Historical failure retrieval | Sealed transfer score | Additional generic critique pass | | X12-7 | Full combined system | Parent success per dollar | Remove one component at a time | Never infer causality from a before/after score across changing dispatch mix without a controlled or matched design. Prefer paired suites and shadow deployments. Use operational rollouts only after offline gating. **Decision examples:** - **Keep** if the intervention meets all prespecified promotion criteria and introduces no major safety or integration regression. - **Reject** if it improves superficial contract validity but not sealed fault detection or parent outcomes. - **Inconclusive** if the number of valid fault injections or independent parent evaluations is insufficient. - **Revert** if operational deployment raises false rejections, blocking legitimate work, or creates cost above the precommitted ceiling. --- ## 11. Live-use monitoring (weekly) **Primary live metric:** fraction of eligible production dispatches with strong evidence that acceptance discriminates faulty from correct states, explicitly stratified by acceptance type. Report the number and percentage with observed red-before/green-after **among feature-fix tasks where that pattern is appropriate**; do not inflate the numerator with tests that were never run. **Additional metrics:** - % parent requirements traceable to child acceptance and final parent evidence. - % eligible acceptance bars with independent negative-control or mutation validation. - Valid mutation kill rate and mutant taxonomy distribution. - % dispatches with recorded correct acceptance type. - % child-green/parent-red and parent-green/child-red cases. - Undeclared file write rate, controlled vs uncontrolled overlaps. - % DAGs with missing prerequisites, cycles and invalid interfaces. - Task rework and avoided duplicate work. - Median and p90 cost from objective to *verified* parent success. - False rejection rate on legitimate changes. - % readouts with full experiment data published to the governance site. Separate **coverage of instrumentation** from **quality conditional on observed data**. “No data” is not a passing outcome. Display numerator, denominator, sample window and decision uncertainty. ### Required incident trace for diagnosis For each failure: `objective → requirements → task graph → acceptance draft → gate decision → dispatch → actual touched files → test and mutant evidence → integration → parent outcome → adjustment`. This directly supports the user's deliberate **trip-the-system** method: tasks must activate machinery and leave analyzable logs, including rejected work and failure paths. --- ## 12. Governance, evaluation integrity and publication Because Piper modifies her own system, the evaluator and its sealed items cannot be editable from the same trust domain used by the candidate intervention. Where perfect isolation is unavailable, explicitly downgrade the evidence and require independent review. - Freeze and hash preregistration before the first outcome. - Keep dev, smoke and sealed cases separately permissioned. - Record all runs, including timeout, crash, abstention and rejection. - Never filter an inconvenient failed run after seeing results. - Use fixed seeds and budgets for the paired comparison; record stochastic variance. - Validate mutations as real defects; exclude equivalent mutants under a recorded process, not ad hoc convenience. - Independently validate false-positive cases so the gate cannot earn points by rejecting everything. - Publish result, confidence intervals, costs, rejected experiments and artifacts even when the proposal loses. - Apply governance rule **X4: a readout is not finished until published on the site**. A Telegram message is not publication. - Keep component #12's official score unchanged until a valid first baseline is recorded under the site's suite rules. --- ## 13. Recommended implementation sequence for Claude Code **Stage A — Audit and measurement.** Inspect live implementations, create event capture, produce the historical baseline, report missing information explicitly. Make no production behavior changes. **Stage B — Build independent suite.** Generate task fixtures, requirement references, mutation sets, valid alternatives and integration oracles. Freeze suite v1 and preregister score rules; run the untouched baseline. **Stage C — Fix vacuous acceptance.** Implement E1 as a shadow gate, collect would-block decisions, verify actual mutant detection and false rejections. Promote only if criteria are met. **Stage D — Improve decomposition graph.** Implement E2/E3, then E4 with individual ablations. Avoid bundling changes before finding which one helped. **Stage E — Evaluate dormant machinery.** Benchmark `spec-compiler` against the current path at the same budget, keep or retire based on measured gain. **Stage F — Deploy controlled runtime instrumentation.** Measure real dispatches, run weekly reports, publish full cards and preserve before/after reproducibility. ### Copy-ready instruction to Claude Code > Read this paper and inspect the current Piper repository/runtime. First produce an evidence-based audit of Component 12, confirming versions and historical claims rather than assuming they remain accurate. Build the independent decomposition benchmark in a separate write-protected evaluation location and freeze its manifest before testing changes. Collect the untouched baseline. Implement the acceptance anti-vacuity gate in shadow mode; demonstrate that it rejects genuine faulty mutations and accepts valid alternatives. Run preregistered paired evaluation with an explicit no-change control, costs and confidence intervals. Do not modify the scorer, expose sealed items, omit failed runs or silently move thresholds. Keep only changes that satisfy the frozen criteria. Record every intervention and publish both successes and failures to the cognition site according to governance rule X4. Do not claim the benchmark improved until independent saved evidence demonstrates it. --- ## 14. Conclusions Piper's immediate problem is not the absence of a decomposition workflow. It is **the gap between contract-shaped tasks and evidence-bearing tasks**. A formally valid dispatch with an acceptance bar that cannot fail is not meaningfully verified. A collection of individually green dispatches does not prove their parent objective is green. A successful #12 intervention should produce fewer missing requirements, stronger defect-sensitive acceptance, clearer interfaces, fewer uncontrolled write collisions, and a higher fraction of *independently verified parent objectives* under equal cost. The score should only rise when those properties improve on sealed cases and persist during real use. **Priority:** instrument → independently benchmark → stop vacuous acceptance → verify coverage and integration → evaluate remaining improvements by ablation. --- ## References **[R1]** Barr, E. T., Harman, M., McMinn, P., Shahbaz, M., & Yoo, S. (2015). *The Oracle Problem in Software Testing: A Survey.* IEEE Transactions on Software Engineering 41(5), 507–525. https://doi.org/10.1109/TSE.2014.2372785 **[R2]** Petrović, G., Ivanković, M., Fraser, G., & Just, R. (2021). *Does Mutation Testing Improve Testing Practices?* ICSE 2021 / arXiv:2103.07189. https://arxiv.org/abs/2103.07189 **[R3]** Laurent, T., Ventresque, A., Papadakis, M., Henard, C., & Le Traon, Y. (2016/2017). *Assessing and Improving the Mutation Testing Practice of PIT.* https://arxiv.org/abs/1601.02351 **[R4]** OpenAI and SWE-bench collaborators (2024). *Introducing SWE-bench Verified.* https://openai.com/index/introducing-swe-bench-verified/ ; SWE-bench primary work: Jimenez, C. E. et al. (2024). *SWE-bench: Can Language Models Resolve Real-World GitHub Issues?* https://arxiv.org/abs/2310.06770 **[R5]** Segura, S., Fraser, G., Sánchez, A. B., & Ruiz-Cortés, A. (2016). *A Survey on Metamorphic Testing.* IEEE Transactions on Software Engineering 42(9), 805–824. https://eprints.whiterose.ac.uk/id/eprint/110335/ ### Evidence-status note External research references support testing principles and experimental rationale, not a claim that any particular upgrade has already improved Piper. System inventory and operational weaknesses derive entirely from the user-provided September 30 review. All scores, suite sizes and thresholds described in this paper are proposed, not observed.