Summary / 16. Problem solving / Document
2026-10-12
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper · Cognition — Component 16: Problem Solving **Working research, benchmark, and upgrade paper · v1.0 · 2026-10-12** **Component:** 16 of 38 · **Definition:** Finding a path from current state to desired state. **Operating principle:** **Small bench → diagnose → one change → retest → keep/revert.** No long implementation or multi-arm campaign before a cheap positive signal. > **Scope note:** This is an implementation plan based on the supplied 2026-10-11 review, not a new inspection of Piper's live code or logs. All existing measurements below are historical claims requiring independent reproduction. ## 1. Executive thesis Piper has demonstrated a strong *coding harness*, but that is not equivalent to demonstrated *problem-solving skill*. A 3-way harness reaching 100% on a relatively easy test has no measurement headroom; improved reliability may be attributable to extra compute rather than better problem representation, search, diagnosis, or verification. At the same time, the workflow Piper describes—reuse search, code analysis, and test execution—appears absent from recent worker tool traces. This component should first answer three separate questions: 1. **Can Piper solve problems that have not already saturated the benchmark?** 2. **Does she use an efficient, evidence-driven path to a solution?** 3. **Does any intervention improve success at equal or clearly accounted-for cost?** Do **not** assume better tool-call counts mean better problem solving. Tools are instrumental; solved outcomes, correctness, cost, and robust generalization are the objectives. ## 2. Starting evidence and its limits | Supplied observation | Interpretation | What must be checked | |---|---|---| | Single pass **92.9%** vs 3-way harness **100%** over **99 paired cells** | Harness improves this measured result by ~7.1 percentage points | Exact paired-cell construction, independence, selection rules, and denominators | | **7 wins, 0 losses**; reported CI **+2 to +12** | Direction encouraging on that task distribution | Reproduce confidence interval and ensure no overlapping test leakage | | Coding holdout **24/24** | Ceiling; can't measure further gains | Introduce *small* harder private tasks before building a large v2 | | `reuse_search`, `codeql_review`, `test_runner`: **0 worker calls in 14 days** | Claimed workflow and observed behavior diverge | Verify traces, invocation aliases, instrumentation coverage, and whether opportunities existed | | Existing `harness-h1`, `piperbench`, replay bench | Reuse infrastructure instead of rebuilding | Confirm hidden-test integrity, task provenance, and per-run token/cost accounting | **Critical correction:** “+7 points” is a harness-system improvement, *not* evidence of a 7-point increase in native problem-solving capability. The comparison changes computation and selection policy as well as solution process. ### Relevant components, without conflating them - **#6 Logic:** validity of formal implications and constraints. - **#12 Decomposition:** dividing tasks into subproblems. - **#13 Planning:** ordering executable steps. - **#15 Decision-making:** choosing among possible actions. - **#16 Problem solving:** moving from an initial state to a verified goal, across steps and obstacles. - **#17 Learning:** whether past failures improve subsequent attempts. - **#24 Error detection / #25 Self-correction:** noticing and repairing errors. A Component 16 success needs an **observable end state**, not merely a plausible plan or code diff. ## 3. Operational definition and measurable outcome For each task, freeze: ```text task_id, family, start_state, target_state, constraints, allowed_actions, hidden_acceptance, difficulty_tag, time_cap, model_and_settings, available_tools, run_budget, seed ``` A solution succeeds only when an **independent verifier** confirms the target state and constraints. Record: - **Solved@budget**: verified completions / assigned tasks, with timeout and invalid submissions scored as failures. - **Cost per verified solution**: dollars and tokens per solved task; unsolved runs still contribute to total cost. - **Time to verified state**: elapsed time and number of actions/tool calls. - **Path efficiency**: excess actions vs a known or best-known feasible path, only where the reference is defensible. - **Failure localization**: correct identification of broken assumptions/root causes before edits. - **Generalization**: success on sealed variants, not the training exemplars. - **Tool opportunity/use**: relevant tool usage conditional on there being a reason to use it; never reward raw call volume. Report **single-pass** and **3-way harness** separately. Compare improvements at the **same fixed call/token budget** where possible. A second, clearly labeled analysis can compare absolute best performance regardless of cost. ## 4. Phase 0 — inspection and instrumentation (10–15 min cap) Ask Claude Code to inspect only: 1. The existing task runner, hidden-test verification path, and per-run cost logs. 2. Three recent task traces with plausible use cases for `reuse_search`, `test_runner`, or `codeql_review`. 3. Whether worker-side tool invocations are logged under different names or bypassed through shell equivalents. 4. Whether benchmark task prompts, solutions, or hidden tests have entered Piper's memories, workspace, summaries, or training examples. **Deliverable:** one-page table: `observed / instrumented? / opportunity? / suspected failure / cheapest probe`. Stop if instrumentation cannot distinguish a tool failure from a missing opportunity; fix that *before* ranking behavioral interventions. ## 5. Phase 1 — minimal difficulty probe (target 15–25 min; no new framework) Select **six** short, independently verifiable tasks from existing replay or coding infrastructure. Prefer prior build failures with later hidden tests. Use **two tasks each** across: | Family | Task shape | What it probes | |---|---|---| | A. Diagnosis | Broken existing code with misleading symptom; root cause elsewhere | State inspection and causal localization | | B. Constraint satisfaction | Implement patch under strict interface/compatibility limits | Constraint tracking and goal satisfaction | | C. Recovery | First attempt demonstrably fails; fix without breaking earlier passing behavior | Feedback integration and controlled iteration | Run **one single-pass attempt per task** with a fixed, modest cap. Do not run 3-way voting yet. Use hidden verifier; tasks must not be ones Piper previously solved or saw solutions for. Record individual cases, not only averages. **Decision gate:** - **4–5/6 solved:** usable pilot difficulty. Retain these six, add only 2–4 sealed tasks before a comparison. - **6/6 solved:** still ceiling. Replace the easiest two with harder sealed cases; do not build a 50-task suite yet. - **0–2/6 solved:** likely too hard or tooling broken. Check failure types; simplify or repair the harness rather than launch an upgrade. - **3/6 solved:** acceptable diagnostic spread, but infer nothing statistically from this sample alone. This is a *screening sample*, not a stable performance estimate. Keep outcomes and uncertainty visible; do not label a 1-task swing a proven gain. ## 6. Trip-the-system cases: realistic, low-cost triggers Use **one or two** per experiment; save the remainder for later. Each must elicit a trace of what Piper read, believed, attempted, verified, and changed. | ID | Scenario | Deliberate trap | Pass condition | |---|---|---|---| | T1 | Regression in a small parser | Error message blames caller; bug in shared tokenizer | Locates root cause and passes hidden regressions | | T2 | “Already exists” helper | Near-duplicate implementation available in repository | Finds/reuses existing helper without copying a flawed version | | T3 | Stale README | Documentation contradicts running code | Reads current artifact, follows actual behavior | | T4 | Partial test success | Visible tests pass; boundary case remains hidden | Adds/executes meaningful edge tests; hidden pass | | T5 | Dependency dead end | First planned fix violates downstream interface | Detects constraint before committing incompatible patch | | T6 | Patch causes new regression | Local fix corrects one failure, damages another | Runs verification and restores earlier invariant | | T7 | Minimal reproducible case | Bug only occurs under a specific input ordering | Isolates reproducer before speculative rewrite | | T8 | Tool-choice conflict | Semantic search faster than manual wandering | Chooses efficient retrieval; records evidence | | T9 | Explicit stop condition | Several plausible hypotheses, tight time cap | Stops safely with classified evidence, not fabricated success | | T10 | Non-coding state transition | Multi-step file/data repair with checkpoints | Reaches specified state without collateral changes | **Avoid benchmark contamination:** sealed versions must differ materially in failure trigger, data, and superficial cues. Keep private acceptance tests outside Piper's write scope. Never use untrusted input to trigger real destructive actions. ## 7. Phase 2 — cheapest intervention first (30–45 min implementation cap) ### Experiment A: evidence-first debugging gate **Hypothesis:** An explicit requirement to read the relevant artifact and generate a reproducer before modifying code improves causal diagnosis and hidden-test success. **One bounded change:** Add a tiny dispatch checklist enforced only for existing-code repair tasks: ```text 1. Identify the failing observable behavior. 2. Read the current relevant implementation (not its summary). 3. Run or construct the smallest safe reproducer. 4. State one falsifiable root-cause hypothesis. 5. Change the minimum scope and run targeted verification. ``` **Do not** require expensive tools on every task; allow a logged `not_applicable` reason. The checklist is **not** counted as success unless hidden tests improve. **A/B micro-retest:** Use **four fresh matched or randomized sealed cases** (two baseline, two intervention, then swap assignments on four additional cases *only if* the first screen is promising and budget permits). Balance task difficulty and keep model/settings/caps fixed. Capture extra tokens and time. **Early stop:** If the checklist is ignored, fix routing or stop. If it increases cost substantially with zero observable improvement or causes missed deadlines, revert. One additional solved task is a *lead*, not proof. ### Experiment B: opportunistic tool routing (only if traces justify it) **Hypothesis:** Existing `reuse_search` / `test_runner` tools are not being called when they would materially improve outcomes. Implement **one** narrow trigger, e.g. an existing-code task with candidate known duplicate functions calls `reuse_search` once *or* records why not. Avoid a universal mandate to run all tools. Test on **two matched cases** where repository reuse matters and **two** where it does not; score correctness and overhead. Revert if it adds pointless calls or misses the relevant match. ### Experiment C: verification-budget allocation (only if A/B inconclusive) Move a small fixed fraction of existing answer-generation budget to deterministic tests; preserve total cap. Compare paired or randomized variants. Record whether pass rates improve **at equal compute**, rather than mixing budgets. **Do not run A, B, and C simultaneously.** Choose the experiment associated with the observed dominant failure class. ## 8. Quick scoring and stop rules Use a light-weight sheet or JSONL—no new dashboard first. ```json { "task_id": "sealed-A03", "arm": "baseline", "family": "diagnosis", "hidden_success": false, "constraint_violation": false, "reproducer_run": true, "root_cause_correct": false, "tool_opportunity": ["test_runner"], "tools_used": [], "elapsed_seconds": 193, "input_tokens": 0, "output_tokens": 0, "api_cost_usd": 0.0, "verifier_version": "frozen-v1", "failure_class": "wrong_root_cause" } ``` Populate zeros above with actual measured values; never interpret placeholders as measurements. **Primary outcome:** extra independently verified solutions under a fixed per-task budget. **Guardrails:** no increased unsafe/destructive actions, hidden-test leakage, or violation of task constraints. **Secondary:** cost per success, time per success, correct diagnosis, test opportunity/use. | Result after micro-retest | Action | |---|---| | No process change observed | Fix trigger/wiring, not model behavior; stop the rollout | | Process changed but hidden success unchanged and cost rises | Revert or narrow condition | | One apparent win on tiny N | Hold as promising; add 4–6 sealed cases before keeping | | Two consecutive small batches with consistent gain and acceptable cost | Keep provisionally; monitor live failures | | Any safety, scope, or verifier-integrity regression | Immediate revert | Do not use arbitrary significance thresholds on samples of four or six. Show raw paired outcomes and effect uncertainty. Expand only when the expected value of further testing justifies it. ## 9. Phase 3 — only after pilot signal: slightly harder v2 If Phase 2 shows a repeatable improvement, assemble **12–20** sealed tasks by extending existing holdout/replay assets, preferably stratified by failure family. Aim for a baseline with **roughly 50–70%** harness success, but **do not manipulate the suite after seeing intervention results**; select difficulty using a separately sampled calibration set. For each task run: - Single pass under a fixed budget. - Current 3-way harness with actual **3×** (or recorded true) compute. - Proposed improvement at matched compute *and*, if relevant, as a harness enhancement. Use paired task-level outcomes; bootstrap at the **task level**, not over correlated repeated cells. Report confidence intervals where feasible and list cost and wall-clock distributions. Hold back a truly unseen validation slice that neither Piper nor the upgrade agent can edit. **Do not declare a cognition improvement** based solely on higher final accuracy purchased with more samples. Label such a result “system-level reliability at higher compute.” ## 10. Distinguishing workflow compliance from problem-solving improvement Measure these independently: 1. **Action compliance:** did the proposed method actually execute at eligible opportunities? 2. **Intermediate quality:** did reproduction and root-cause identification improve? 3. **Terminal outcomes:** did hidden tests pass more often? 4. **Efficiency:** did total time/cost/action count improve or remain acceptable? 5. **Transfer:** does the improvement persist on new task shapes? If action compliance increases but outcomes don't, the problem may be **tool value**, **poor routing**, **task selection**, or **hidden tests that measure the wrong objective**. Do not inflate the #16 score for better logging. ## 11. Failure taxonomy and logging Tag no more than **one primary and two secondary** failure reasons per case: - `no_reproducer` - `summary_over_artifact` - `wrong_root_cause` - `premature_patch` - `constraint_missed` - `reuse_opportunity_missed` - `verification_skipped` - `regression_introduced` - `tool_invocation_failure` - `budget_exhausted` - `task_out_of_distribution` - `verifier_or_benchmark_fault` Record `prompt_hash`, `repository_commit`, `task_version`, `agent_version`, `verifier_hash`, `seed`, `arm_assignment`, and full action trace location. This is the minimum required to replay a surprising result and detect accidental benchmark changes. ## 12. Governance, safety, and experimental independence - Hidden tasks and tests are authored/held by a component outside Piper's writable environment. - Task graders are deterministic where possible, frozen before experiments, and never chosen by the model under test. - Write access stays scoped to a disposable repository/worktree with snapshots and rollback. - Benchmark invocations do not require contacting people, exceeding the operator's allowance, modifying credentials, or destructive changes on production hosts. - High-load or production changes require separate approval; this plan needs neither. - The coding agent may propose test additions but cannot revise the sealed evaluator used to decide a win. - The upgrade ledger records negative results and all attempts, not only successful variants. ## 13. Claude Code execution tickets — one at a time ### Ticket 16-P0 — Inspect existing instruments **Timebox:** 15 minutes. Locate runners, verifier, trace schema, and tool opportunity evidence. **Acceptance:** one short factual inventory; no edits unless a clear instrumentation blocker is discovered. **Stop:** if testing cannot be isolated from production. ### Ticket 16-P1 — Six-case baseline probe **Timebox:** 25 minutes. Select six sealed, short tasks from existing assets; run fixed-budget single pass. **Acceptance:** six full task records and summary of primary failure modes. **Stop:** on ceiling, floor, leakage, or verifier defect; do not automatically enlarge suite. ### Ticket 16-P2 — One selected intervention **Timebox:** 45 minutes. Pick *one* of Experiments A/B/C from Phase 1 data. **Acceptance:** narrow diff, explicit trigger, trace confirming execution, rollback command. **Stop:** if implementation needs widespread architecture edits. ### Ticket 16-P3 — Independent micro-retest **Timebox:** 20 minutes. Run 4–8 new cases as allowed by budget, report task-level outcomes and cost. **Acceptance:** comparison with fixed budgets and decision `keep_provisional | revert | gather_more`. **Stop:** on violations or no observable intervention execution. ### Ticket 16-P4 — Optional validation expansion **Trigger:** only positive signal from P3. **Scope:** 12–20 hidden cases with proper task-level uncertainty; no 10-arm campaigns. **Acceptance:** replicated gain or explicit negative result, with record in cognition ledger. ## 14. Experimental record template ```markdown ### C16 experiment [ID] — [YYYY-MM-DD] - Hypothesis: - Baseline evidence (task IDs and hashes): - Target failure mode: - Single change / exact diff: - Prediction before testing (success and cost): - Fixed budget and task assignment: - Independent verifier + hash: - Outcome by task (pass/fail, constraint violations, cost): - Change in verified solves / total cost: - Confounds, uncertainty, side effects: - Decision: KEEP PROVISIONALLY / REVERT / GATHER MORE - Next cheapest falsifying test: ``` ## 15. Research anchors (background, not endorsements of a specific upgrade) - Yao et al. (2023), **ReAct: Synergizing Reasoning and Acting in Language Models** — integrates reasoning steps and tool actions. https://arxiv.org/abs/2210.03629 - Shinn et al. (2023), **Reflexion: Language Agents with Verbal Reinforcement Learning** — feedback, memory and retry. https://arxiv.org/abs/2303.11366 - Jimenez et al. (2024), **SWE-bench: Can Language Models Resolve Real-World GitHub Issues?** — repository-level outcome-based evaluation. https://arxiv.org/abs/2310.06770 - Chen et al. (2021), **Evaluating Large Language Models Trained on Code** — pass@k and sampling-compute distinctions. https://arxiv.org/abs/2107.03374 - Madaan et al. (2023), **Self-Refine: Iterative Refinement with Self-Feedback** — iterative refinement and its limits as an upgrade hypothesis. https://arxiv.org/abs/2303.17651 **Methodological caution:** literature motivates candidate interventions; only independently measured Piper-specific outcomes justify adoption. ## 16. Recommended first move **Run Ticket 16-P0 followed by Ticket 16-P1.** Before changing any code, determine whether a six-case sealed probe reveals non-ceiling failures, and whether `reuse_search`/`test_runner` were actually *eligible* but skipped. Then choose **one** intervention matched to the largest observed failure class. If the first small retest is flat, revert instead of committing hours to a v2 overhaul. --- **Paper status:** proposal; no new benchmark measurements or implementation work claimed. **Owner of acceptance criteria:** independent scorer and the operator's governance rules. **Next checkpoint:** six-case baseline + tool-opportunity audit.