Summary / 13. Planning / Document
2026-10-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Cognitive Component 13 — Planning ## Benchmark, Graph-Search Architecture, Interventions, and Engineering Plan **Component:** 13 of 38 **Capability:** Planning **Working definition:** Constructing feasible, efficient, adaptable sequences of actions that move the current state toward a goal under dependencies, costs, resources, uncertainty, and changing constraints. **Current reviewer read:** **Unmeasured** **Status at start:** Not yet scored **Prepared:** 2026-10-05 --- ## Executive summary Piper already stores plans. She has a work ledger, dependencies, blocked reasons, detached execution units, scheduled work, and mechanisms for waking and resuming work. Those are important execution and persistence mechanisms, but they are not themselves evidence of planning quality. The question for Component 13 is not: > Can Piper represent a sequence of tasks? It is: > Can Piper construct a sequence of actions that is valid, efficient, realistically scheduled, robust to disruption, and likely to reach the goal? The existing reviewer note correctly identifies the gap: > **She treats storing plans as planning.** This paper proposes **PLANNING-v1**, a benchmark that separates planning into seven measurable capabilities: 1. **Goal-state formulation** — does Piper represent what “done” means correctly? 2. **Dependency correctness** — are actions ordered so their preconditions are satisfied? 3. **Feasibility** — can the proposed plan actually execute under the stated constraints? 4. **Efficiency / optimality** — how much worse is the plan than the best known plan? 5. **Resource and temporal realism** — does the plan respect time, concurrency, budgets, capacities, and deadlines? 6. **Adaptive replanning** — when the world changes, can Piper repair the plan without needlessly starting over? 7. **Plan-to-execution fidelity** — does actual work follow the plan closely enough that the plan was meaningful? A central design proposal is to use **graph search and constraint solving as planning instruments**, not merely as metaphors. For formalizable problems, Piper can convert a goal into: - a current state \(S_0\), - a set of possible actions, - preconditions, - effects, - costs, - constraints, - and a goal condition \(G\). That naturally creates a state-space graph: ```text state ──action/cost──> state ──action/cost──> state ... ``` Planning then becomes a search problem. - **Dijkstra** is useful when edge costs are known and no heuristic is available. - **A\*** is useful when Piper can estimate remaining distance-to-goal with a heuristic. - **Constraint programming / scheduling** is useful when the problem is dominated by resource, timing, or capacity constraints. - **TSP / vehicle-routing style methods** are useful only for the narrower case where many locations or tasks must all be visited/completed and ordering cost dominates. The proposed architecture is therefore hybrid: > **LLM for interpretation and model construction; deterministic planner/optimizer for search where possible; LLM for ambiguous steps, model revision, and execution adaptation.** Recent planning work supports this separation. Planning-specific benchmarks show that strong models still struggle with long-horizon planning, broken or extraneous tools, and unsolvable tasks. Dynamic planning benchmarks show substantial degradation when new constraints accumulate. Resource-aware benchmarks further show that correct logical dependencies do not automatically produce safe or efficient schedules. Hybrid LLM + constraint-programming approaches improve feasibility and optimization precisely by separating semantic interpretation from formal scheduling. For Piper, this component should become one of the strongest bridges between cognition and actual autonomous work. --- # 1. What planning is Planning is the construction of a sequence or partial order of actions intended to transform a current state into a desired goal state. A planning problem can be written abstractly as: \[ P = (S_0, G, A, C) \] where: - \(S_0\) = initial state, - \(G\) = goal condition, - \(A\) = available actions, - \(C\) = constraints and costs. Each action \(a\) has: - preconditions, - effects, - cost, - duration, - required resources, - possible risks. A valid plan is a sequence: \[ \pi = (a_1, a_2, ..., a_n) \] such that each action is applicable when executed, and the final state satisfies the goal. A good plan does more than merely reach the goal. It should ideally also: - minimize unnecessary work; - respect deadlines; - avoid resource collisions; - exploit safe parallelism; - preserve optionality where uncertainty is high; - contain checkpoints; - tolerate predictable failures; - be easy to repair when the environment changes. --- # 2. What planning is not ## 2.1 A to-do list A list of actions may contain impossible ordering, missing prerequisites, redundant work, or no completion condition. ## 2.2 A work ledger The ledger persists tasks and dependencies. That is infrastructure for planning, not proof that the plan is good. ## 2.3 Decomposition Breaking a goal into subtasks is Component 12. Planning determines: - what order; - what dependencies; - what concurrency; - what resources; - what route; - what timing; - what fallback. A decomposition can be correct while the resulting plan is terrible. ## 2.4 Scheduling Scheduling is a subset of planning concerned particularly with assigning time and resources. Planning may also involve causal sequence, tool selection, information gathering, and state transitions. ## 2.5 Execution A perfect plan can fail because execution is bad. A bad plan can occasionally succeed through luck or strong execution. The benchmark must separate these. --- # 3. The key distinction: plan storage versus plan quality Piper currently has: - `work-ledger` v1.3.42; - `launch-detached` v1.1.0; - systemd user units; - crons; - `idle-wake` v1.23.1. These answer questions such as: - What work exists? - Which work is blocked? - What depends on what? - How does long work survive session boundaries? - How does work resume? They do not directly answer: - Was this the shortest valid path? - Were dependencies correct? - Should two tasks have run in parallel? - Was a step unnecessary? - Was an information-gathering step missing? - Was the predicted duration realistic? - Should the plan have been abandoned earlier? - Was the block avoidable? - Could Piper have selected a route with lower cost? That gap is what PLANNING-v1 should measure. --- # 4. Planning as graph search The graph interpretation should be explicit in Piper’s architecture. A planning state can be represented as a node. An action that changes state becomes a directed edge. ```text action A / cost 4 S0 --------------------> S1 \ | \ action B / cost 2 | action C / cost 3 \ | -------> S2 --------> G action D / 2 ``` The objective is to find a path from \(S_0\) to a state satisfying \(G\). This connects directly to classical AI planning. --- # 5. Dijkstra, A*, Bellman-Ford, and TSP: where they fit ## 5.1 Dijkstra Dijkstra finds a lowest-cost path when: - edge costs are nonnegative; - the state graph is explicit or can be expanded; - no useful heuristic is required. Piper could use Dijkstra when action costs are known but she cannot estimate remaining distance well. Example: ```text Goal: deploy working service actions: - install package cost 2 - configure service cost 3 - migrate database cost 8 - run validation cost 2 - restart cost 1 ``` If multiple legal routes exist, Dijkstra can find the cheapest. ## 5.2 A* A* uses: \[ f(n) = g(n) + h(n) \] where: - \(g(n)\) = cost incurred so far; - \(h(n)\) = estimated cost remaining. This is attractive for Piper because she may be capable of generating useful heuristics. Example: ```text current state: - code written - tests failing - deployment blocked goal: - deployed and validated g(n): actual elapsed / token / tool / monetary cost so far h(n): estimated remaining cost from unresolved dependencies ``` The interesting research question becomes: > Can Piper learn or construct heuristics that reduce search without making plans worse? This creates a measurable bridge between language-model judgment and classical planning. ## 5.3 Bellman-Ford Bellman-Ford is less likely to be a major general-purpose Piper primitive. It is relevant mathematically when negative edge weights exist. Most real planning costs are naturally nonnegative: - time; - money; - tokens; - risk; - resource usage. Planning rewards can usually be reformulated into nonnegative costs. It is therefore probably not a first implementation target. ## 5.4 Travelling Salesman Problem TSP is useful when the problem is: > Visit every required location/task while minimizing route cost. Examples for Piper: - order a set of maintenance checks; - choose visit order across many hosts; - optimize a physical service route; - sequence many independent jobs where setup/context-switch cost matters. But TSP is not general planning. Planning includes state-dependent actions, prerequisites, branching, uncertainty, and goals that do not require visiting all nodes. A better umbrella is: > **state-space search + scheduling + constraint optimization** --- # 6. The deeper graph idea: “where am I?” and “where do I want to be?” The graph model gives Piper a useful general planning representation. For any substantial goal: ```text CURRENT STATE | | possible actions v STATE FRONTIER | | search / heuristic v TARGET STATE ``` This forces five questions. 1. What is true now? 2. What must be true at the goal? 3. What actions can change the current state? 4. What are the preconditions and effects of those actions? 5. Which route has the best expected cost? That is much stronger than: > “Generate a plausible list of steps.” --- # 7. Partial-order planning Not every plan should be a single linear sequence. Example: ```text ┌── research A ──┐ goal ────────┤ ├── integrate ── test └── research B ──┘ ``` If A and B are independent, forcing: ```text A → B → integrate ``` wastes time. Piper’s existing dependency ledger already provides some infrastructure for representing a DAG. The benchmark should therefore score both: - ordering correctness; - unnecessary serialization. This is particularly relevant to worker/delegate orchestration. --- # 8. Planning versus scheduling A plan might say: ```text A before C B before C D after C ``` A schedule adds: ```text A: worker 1, 10:00–10:15 B: worker 2, 10:00–10:25 C: worker 1, 10:25–10:40 D: worker 1, 10:40–10:50 ``` Resource-aware agent benchmarks now explicitly separate logical planning from physical scheduling because strong dependency planning can still create inefficient or unsafe execution when concurrent resources are limited. For Piper, PLANNING-v1 should do the same. --- # 9. Core research question The component should answer: > **Can Piper construct a valid, efficient, realistic plan from the current state to a defined goal, and revise that plan intelligently when the world changes?** Subquestions: 1. Does the goal state match the operator’s actual objective? 2. Does Piper identify required actions? 3. Are dependencies valid? 4. Are unnecessary actions omitted? 5. Does the plan respect resource and time constraints? 6. Does Piper exploit legal parallelism? 7. How close is the plan to optimal? 8. Can Piper recognize impossible goals? 9. Does Piper know when information gathering is required before planning further? 10. Can Piper repair a plan after disruption? 11. Do duration and step-count estimates predict reality? 12. Do plans reduce blocked work in actual operation? --- # 10. Failure taxonomy ## P1 — Goal mis-specification The plan optimizes the wrong end state. ## P2 — Missing prerequisite An action occurs before required state exists. ## P3 — Invalid ordering All required tasks exist, but sequence is wrong. ## P4 — Missing action Plan cannot reach goal because a necessary action is absent. ## P5 — Redundant action Plan performs unnecessary work. ## P6 — Cyclic dependency Plan contains an impossible dependency cycle. ## P7 — Dead-end plan Sequence enters a state from which the goal cannot be reached. ## P8 — Over-serialization Independent work is needlessly sequenced. ## P9 — Unsafe parallelism Concurrent steps compete for a resource or violate ordering. ## P10 — Resource infeasibility Plan exceeds worker capacity, token budget, memory, machine capacity, API quota, financial budget, or tool availability. ## P11 — Temporal infeasibility Deadline or duration constraints cannot be satisfied. ## P12 — Unrealistic duration estimate Step durations are badly predicted. ## P13 — Unrealistic step-count estimate Plan expands far beyond expected number of actions. ## P14 — Cost blindness A valid plan is dramatically more expensive than alternatives. ## P15 — Risk blindness Plan selects a brittle path when a slightly costlier robust path is preferable. ## P16 — Missing information-gathering step Piper acts before resolving a decision-critical unknown. ## P17 — Unnecessary information gathering Piper researches indefinitely when enough is already known. ## P18 — Replanning failure A disruption invalidates the plan and Piper continues anyway. ## P19 — Excessive replanning Piper repeatedly reconstructs the plan despite minor changes. ## P20 — Plan abandonment Valid plan is dropped without evidence that it should be. ## P21 — Sunk-cost persistence Piper continues a failing route because substantial effort has already been spent. ## P22 — Block persistence A blocked task remains blocked even after an available unblocking action exists. ## P23 — Phantom blocker Piper records a task as blocked when it is actually actionable. ## P24 — Incorrect blocker The stated blocker is not causally responsible. ## P25 — Unreachable-goal failure Piper generates a plan for an impossible goal instead of identifying infeasibility. ## P26 — Plan/execution divergence Actual work no longer corresponds to the stored plan. ## P27 — Stale plan World state changed but plan assumptions were not updated. ## P28 — Local optimization failure Piper selects individually sensible steps that create a globally poor sequence. --- # 11. Benchmark design: PLANNING-v1 Use two layers: ### Frozen synthetic / simulator benchmark Provides known feasibility and optimum. ### Live planning telemetry Measures whether planning machinery improves real Piper work. Recommended first frozen suite: **260 episodes** across eight tracks. --- # 12. Track A — Dependency planning **40 episodes** Input: - goal; - actions; - action preconditions; - action effects. Piper must generate an executable plan. Score: - valid action ordering; - completeness; - unnecessary actions; - total cost. Generate random DAG-based tasks so the suite is not memorized. --- # 13. Track B — Optimal path planning **40 episodes** Provide a state graph with multiple routes. Piper must find a low-cost path. Measure: \[ OptimalityRatio = rac{Cost_{optimal}}{Cost_{Piper}} \] for valid plans. Use: - equal-cost alternatives; - misleading locally cheap edges; - deep routes; - dead ends; - variable branching factor. This track directly tests whether graph search improves over native LLM planning. --- # 14. Track C — Heuristic search / A* track **30 episodes** For each problem expose: - state transition generator; - action costs; - goal test; - optional heuristic function. Compare: 1. Piper native plan; 2. Dijkstra; 3. A* with hand-designed heuristic; 4. A* with Piper-generated heuristic; 5. A* with learned/empirically calibrated heuristic if later available. Metrics: - path cost; - states expanded; - runtime; - heuristic admissibility where required; - goal reach rate. Research questions: - Can Piper generate useful \(h(n)\)? - Does her heuristic reduce search? - Does it overestimate badly? - Does performance generalize across domains? - Is weighted A* preferable when optimality can be traded for speed? --- # 15. Track D — Temporal and resource scheduling **40 episodes** Tasks contain: - durations; - dependencies; - worker capacities; - machine capacities; - deadlines; - mutually exclusive resources. Score: - feasibility; - makespan; - deadline success; - resource collisions; - legal parallelization. Compare against: - optimal solver; - greedy scheduling; - Piper native; - Piper + constraint solver. --- # 16. Track E — Dynamic replanning **40 episodes** Begin execution of a valid plan. Then inject a change: - tool unavailable; - worker fails; - action takes 3× longer; - new operator constraint; - task becomes unnecessary; - new dependency appears; - resource disappears; - one step is already completed unexpectedly; - goal changes slightly. Piper must revise the remaining plan. Score: - final feasibility; - retained useful work; - unnecessary discarded work; - time to recover; - incremental cost; - number of plan edits. A good planner should repair locally when possible. --- # 17. Track F — Information-gathering plans **30 episodes** Some planning problems cannot be solved correctly without first obtaining missing information. The correct first action may be inspection. This links Planning to Component 23, Uncertainty Estimation. Score whether Piper: - identifies decision-relevant uncertainty; - gathers information when worthwhile; - avoids pointless research. --- # 18. Track G — Impossible and under-specified goals **20 episodes** Cases include: - mutually contradictory constraints; - insufficient resources; - impossible deadlines; - missing required tool; - goal state unreachable from available actions; - ambiguous goal definition. Correct output may be: - infeasible; - ask for clarification; - negotiate constraint; - identify minimum constraint that must change. --- # 19. Track H — Real-work plan forecasting **20+ live episodes per scoring window** For every substantial Piper plan, record at creation: ```text predicted_steps predicted_duration predicted_cost known_dependencies known_blockers expected_parallelism success_probability ``` At completion: ```text actual_steps actual_duration actual_cost added_steps removed_steps replans unexpected_blockers final_outcome ``` This produces calibration data from real work. --- # 20. Metrics ## 20.1 Validity rate Fraction of plans that execute without violating action preconditions or constraints. ## 20.2 Goal reach rate Did the plan end in a valid goal state? ## 20.3 Optimality ratio \[ OR = rac{C^*}{C_{plan}} \] where \(C^*\) is optimal cost. ## 20.4 Excess action rate \[ EAR = rac{Actions_{plan} - Actions_{minimal}}{Actions_{minimal}} \] when a minimal plan is defined. ## 20.5 Dependency accuracy Measure missing, invented, and reversed dependencies. ## 20.6 Makespan efficiency \[ ME = rac{Makespan_{optimal}}{Makespan_{Piper}} \] ## 20.7 Resource-violation rate Number of resource constraint violations per episode. ## 20.8 Replanning regret \[ Regret = Cost_{Piper\ revised} - Cost_{optimal\ revised} \] ## 20.9 Plan stability How much of a valid old plan is unnecessarily discarded? ## 20.10 Duration error \[ APE = rac{|T_{predicted} - T_{actual}|}{T_{actual}} \] Also report signed bias. ## 20.11 Step-count error \[ SCE = |Steps_{predicted} - Steps_{actual}| \] ## 20.12 Block-resolution rate Fraction of blocked deliverables that become actionable within their stated unblock window. --- # 21. Component score | Dimension | Weight | |---|---:| | Plan validity / goal reach | 25 | | Dependency correctness | 15 | | Cost / path optimality | 15 | | Resource & temporal feasibility | 15 | | Adaptive replanning | 15 | | Information gathering / infeasibility recognition | 5 | | Duration & step forecasting | 5 | | Live blocked-work resolution | 5 | Total: **100** A low-cost plan that cannot execute is not a good plan. --- # 22. Baselines ## Baseline 0 — Native LLM plan Prompt Piper for a plan with no external planner. ## Baseline 1 — Greedy planner At each state choose lowest immediate-cost valid action. ## Baseline 2 — Dijkstra Use on explicit graph tasks. ## Baseline 3 — A* Use a known-good heuristic. ## Baseline 4 — Piper-generated A* heuristic Tests whether Piper can contribute useful domain intelligence. ## Baseline 5 — Constraint solver Use for scheduling/resource problems. ## Baseline 6 — Hybrid planner LLM creates formal representation; deterministic solver produces route/schedule; LLM validates semantics and executes. --- # 23. Proposed hybrid architecture ```text operator goal ↓ goal parser ↓ state model ├── current state ├── goal state ├── actions ├── preconditions ├── effects ├── resources ├── durations ├── costs └── uncertainty ↓ problem classifier ├── simple sequence ├── shortest path ├── partial-order plan ├── scheduling ├── routing/TSP ├── information gathering └── mixed problem ↓ planner ├── native LLM ├── Dijkstra ├── A* ├── constraint solver └── domain-specific optimizer ↓ plan verifier ↓ work ledger ↓ execution ↓ state observation ↓ replanning trigger ``` The ledger becomes the execution representation of a plan rather than the planning mechanism itself. --- # 24. Intervention P-A — Formal goal state Before planning, encode: ```text goal_conditions: success_tests: non_goals: hard_constraints: soft_preferences: ``` Prediction: fewer plans that perform work without satisfying the objective. --- # 25. Intervention P-B — Preconditions and effects Each important action receives: ```text preconditions effects cost duration resources failure_modes ``` Prediction: higher validity and fewer phantom dependencies. --- # 26. Intervention P-C — DAG extraction Compile plan into a directed acyclic graph where possible. Prediction: reduced makespan through legal parallelism. --- # 27. Intervention P-D — Shortest-path planner Add deterministic path search. Use: - BFS for equal-cost edges; - Dijkstra for weighted nonnegative edges; - A* where heuristic exists. Prediction: near-optimal performance on formal graph tasks and fewer dead-end routes. --- # 28. Intervention P-E — Piper-generated heuristic Instead of asking: > “What is the whole plan?” ask Piper: > “From this state, how far are we from the goal?” That estimate becomes \(h(n)\) for A*. Candidate heuristic features: - number of unresolved dependencies; - number of failed tests; - count of missing artifacts; - expected remaining tool calls; - unresolved blockers; - dependency depth; - predicted remaining duration; - risk-adjusted remaining cost. Key experimental question: > Does Piper make a better **heuristic estimator** than an end-to-end planner? --- # 29. Intervention P-F — Constraint solver Translate scheduleable portions into a constraint problem. Variables: - task start; - task end; - worker assignment; - resource usage. Constraints: - precedence; - deadline; - capacity; - exclusivity; - availability. Objective: - minimize makespan; - minimize cost; - or weighted combination. Prediction: large gains on resource/scheduling track. --- # 30. Intervention P-G — Expected-value branching Instead of: ```text A -> B -> C ``` use: ```text inspect X ├── if X1 -> A -> B └── if X2 -> C -> D ``` Prediction: less wasted work when hidden state matters. --- # 31. Intervention P-H — Replanning trigger Trigger candidates: - action failure; - deadline drift; - new hard constraint; - new blocker; - resource loss; - goal change; - cost overrun; - observed state differs materially from expected state. Replan when the current plan is no longer valid or materially suboptimal. --- # 32. Intervention P-I — Local repair before global replanning When disruption occurs: 1. freeze completed actions; 2. identify invalid remaining subgraph; 3. preserve unaffected actions; 4. repair only broken dependencies; 5. rerun optimizer on affected frontier. Prediction: lower replanning regret and less churn. --- # 33. Intervention P-J — Blocker attack Every blocker gets: ```text blocker: type: owner: can_piper_remove: next_probe: next_action: recheck_time: cost_to_remove: expected_value_of_unblocking: ``` Planner periodically evaluates: > Which blocker should I attack next? Prediction: higher blocked-to-active conversion rate. --- # 34. Intervention P-K — Critical path awareness For DAG plans, calculate critical path. Prioritize tasks whose delay moves final completion. Prediction: reduced makespan without increasing total work. --- # 35. Intervention P-L — Plan cost model Every action can carry multiple costs: ```text time tokens API dollars human attention risk compute opportunity cost ``` Define: \[ C(a)=w_tT+w_mM+w_hH+w_rR+... \] Different plans can optimize different objectives. --- # 36. Intervention P-M — Plan verifier Before execution: - simulate preconditions; - detect cycles; - check resource constraints; - ensure goal reachable; - verify completion conditions; - calculate predicted cost. Only valid plans enter the work ledger. --- # 37. Recommended intervention order ## Phase 0 — Baseline Run native Piper against PLANNING-v1. ## Phase 1 — Goal + action model Implement P-A, P-B, P-M. ## Phase 2 — DAG planning Implement P-C, P-K. ## Phase 3 — Graph search Implement P-D, P-E. ## Phase 4 — Scheduling Implement P-F, P-L. ## Phase 5 — Replanning Implement P-H, P-I. ## Phase 6 — Live blocker optimization Implement P-J. --- # 38. Piper as heuristic, not planner This deserves its own experiment. A deterministic search algorithm is excellent at: - remembering explored states; - comparing path costs; - avoiding cycles; - preserving optimality guarantees. Piper is potentially better at: - interpreting ambiguous goals; - inventing candidate actions; - estimating which states look promising; - recognizing semantic equivalence; - identifying hidden constraints. Test: ```text traditional: goal -> Piper -> complete plan hybrid: goal -> Piper -> formal state/actions | v A* search ^ | Piper heuristic | v optimized route ``` The hybrid arrangement may outperform end-to-end planning with the same model. --- # 39. Heuristic quality benchmark If Piper supplies \(h(n)\), measure: ## Rank quality Does lower predicted remaining cost correlate with states closer to goal? ## Mean absolute error How accurately does Piper estimate true remaining optimal cost? ## Admissibility rate \[ h(n) \le h^*(n) \] where \(h^*(n)\) is actual optimal remaining cost. ## Search reduction Compare states expanded by Dijkstra versus A* + Piper heuristic. ## Final path quality Did heuristic error produce a worse route? --- # 40. Dynamic constraints Piper’s real environment changes: - operator changes requirement; - tool becomes unavailable; - new result reveals hidden dependency; - budget changes; - another worker completes something early. PLANNING-v1 should therefore not evaluate only static plans. A planner that works only when the world freezes is not a strong autonomous planner. --- # 41. Plan impossibility An important planning ability is recognizing when no legal path exists. Correct result may be: > no valid plan under current constraints. Not: > hallucinate a sequence anyway. This should be separately scored. --- # 42. Plan robustness Optimality is not always best. Example: ```text Plan A cost = 10 success probability = 0.55 Plan B cost = 12 success probability = 0.98 ``` Depending on consequence, B may be preferable. Long-term planning should eventually optimize: \[ EU(\pi) = P(success|\pi)V(goal) - C(\pi) - Risk(\pi) \] --- # 43. Planning under uncertainty States may eventually be uncertain. A practical first approximation is not a full POMDP. Use: - uncertainty threshold; - information-gathering action; - branch on observation. That is enough to connect Component 23 to Component 13. --- # 44. Anti-gaming rules - trivial short plans are countered with long-horizon cases; - “include every possible step” is penalized by excess action cost; - serializing everything is penalized by makespan; - parallelizing everything is penalized by resource violations; - declaring everything impossible is penalized by false-infeasibility rate; - constant replanning is penalized by stability cost; - never replanning is punished by dynamic cases; - plans must be sealed before execution to prevent outcome leakage. --- # 45. Experimental protocol ```yaml intervention_id: hypothesis: suite_version: planner_version: model: provider: problem_family: seed: baseline_score: after_score: goal_reach: validity: optimality_ratio: dependency_accuracy: makespan_efficiency: resource_violations: replanning_regret: duration_error: step_count_error: states_expanded: planner_runtime: token_cost: api_cost: side_effects: decision: keep | revert | inconclusive ``` --- # 46. Live-use measurement Track over seven days: - plans created; - completion rate; - median blocked age; - blocker resolution rate; - plan churn; - added-step ratio; - removed-step ratio; - duration calibration; - step-count calibration; - critical-path delay; - plan/execution divergence. --- # 47. Suggested public scorecard ```text Suite — v1 Baseline — XX.X Latest — XX.X Change — — Feasibility: Valid plans: ... Goal reached: ... Dependency accuracy: ... Efficiency: Optimality ratio: ... Makespan efficiency: ... Excess actions: ... Adaptation: Replan success: ... Replanning regret: ... Forecasting: Duration error: ... Step-count error: ... Live work: Median blocked age: ... Blocked items unblocked on time: ... Plan/execution divergence: ... ``` --- # 48. Suggested reviewer-read progression Before testing: > **Unmeasured** Possible later reads: - **Usually feasible, inefficient** - **Finds the steps, orders them badly** - **Strong in formal state spaces, weak in messy work** - **Plans well until the world changes** - **Logical plans good, resource timing poor** - **Builds executable routes and repairs them when the state changes** --- # 49. Acceptance criteria Suggested targets: - goal reach ≥ 90% on deterministic planning tracks; - plan validity ≥ 95%; - dependency accuracy ≥ 95%; - median optimality ratio ≥ 0.90; - resource violation rate ≤ 2%; - makespan efficiency ≥ 0.85; - dynamic replanning success ≥ 85%; - infeasibility classification ≥ 90%; - duration estimate median absolute percentage error ≤ 30%; - step-count estimate error ≤ 25%; - significant reduction in median blocked-work age; - no major track below frozen minimum. These are engineering targets, not literature-derived standards. --- # 50. The most important first experiment Build a small state-space planner and compare four conditions: ### A — Piper native Piper directly writes the plan. ### B — Dijkstra Formal graph, no heuristic. ### C — A* with deterministic heuristic Known heuristic. ### D — A* with Piper heuristic Piper estimates distance-to-goal. Use 100 randomly generated solvable problems with: - varying branching factor; - misleading local choices; - dead ends; - multiple valid paths; - different edge costs. Measure: ```text goal success path cost states expanded runtime tokens heuristic error ``` The central question is: > **Where in the planning stack does the LLM actually add value?** If A* + Piper heuristic beats native planning substantially, you have found a better role for the language model. --- # 51. The second experiment: blocker attack Take Piper’s existing blocked work. Classify each blocked deliverable as: ```text hard external blocker soft external blocker internal dependency missing information forgotten follow-up phantom blocker resource blocker decision blocker ``` Then ask: > Is there an available action that reduces or removes this blocker? Rank unblock actions by: \[ Priority = rac{ P(unblock) imes downstream\ value }{ cost } \] The metric is: > how many blocked deliverables become actionable sooner. --- # 52. The third experiment: dream-to-route Give Piper an intentionally distant target state and require a route from current state to that target. Examples: - achieve benchmark score X; - reduce operating cost by Y; - make component Z reach retirement threshold; - produce a publishable external demonstration; - migrate system architecture from current state to target architecture. Force: ```text NOW ``` then: ```text TARGET ``` then intermediate states: ```text S0 -> S1 -> S2 -> ... -> G ``` Each transition must have: - action; - precondition; - evidence of completion; - estimated cost; - fallback. This is a graph-theoretic version of long-range goal planning. --- # 53. Relationship to decomposition Decomposition: > What pieces make up the task? Planning: > In what order and under what conditions should those pieces happen? --- # 54. Relationship to prediction Planning requires predictions about duration, success probability, resource use, and downstream state. Component 14 should eventually supply the estimates used as edge costs and transition probabilities. --- # 55. Relationship to decision-making Planning generates feasible candidate routes. Decision-making chooses among competing plans given utility, risk, and preferences. --- # 56. Relationship to causal reasoning Action models require causal structure. If Piper’s causal model is wrong, the planner’s transition graph is wrong. --- # 57. Relationship to uncertainty estimation Component 23 can supply: - edge success probability; - uncertainty in current state; - uncertainty in action duration; - uncertainty in blocker status. --- # 58. Relationship to error detection Component 24 should compare expected and observed state. When: ```text expected S_t != observed S_t ``` that becomes a replanning trigger. Full loop: ```text PLAN ↓ ACT ↓ OBSERVE ↓ COMPARE ↓ REPLAN ``` --- # 59. First implementation package ```text planning/ ├── suite_v1/ │ ├── dependencies.py │ ├── shortest_path.py │ ├── heuristic_search.py │ ├── scheduling.py │ ├── replanning.py │ ├── information.py │ ├── infeasible.py │ └── live.py ├── state_model.py ├── action_model.py ├── goal_model.py ├── dag.py ├── dijkstra.py ├── astar.py ├── heuristics.py ├── scheduler.py ├── verifier.py ├── repair.py ├── cost_model.py ├── blocker.py ├── score.py └── reports/ ``` Do not build every module before baseline. The suite should determine which modules are justified. --- # 60. Research questions worth preserving 1. Is Piper better at planning directly or supplying heuristics to search? 2. How good are Piper-generated A* heuristics? 3. Does a formal action model improve planning enough to justify its overhead? 4. How often does Piper omit action preconditions? 5. How often does she invent false dependencies? 6. Does she over-serialize independent work? 7. Can she identify the critical path? 8. Can she distinguish true blockers from stale blockers? 9. How much blocked work is avoidable? 10. Does graph search help on real Piper tasks or only synthetic puzzles? 11. When should a problem be routed to A* versus constraint programming? 12. Can Piper classify planning problem type reliably? 13. Can she recognize unreachable goals? 14. Can she estimate action costs well enough for optimization? 15. Does better uncertainty estimation improve planning? 16. Does better prediction improve duration accuracy? 17. Does local plan repair outperform full replanning? 18. Does formal planning reduce plan/execution divergence? 19. How much operator intervention is caused by bad plans rather than bad execution? 20. Can Piper learn reusable action schemas from completed plans? 21. Do planner-generated dependencies match causal dependencies? 22. Can Piper preserve optionality under uncertainty? 23. Does the planning system reduce total work or merely reorder it? 24. How does plan quality scale with horizon length? 25. At what complexity does native LLM planning collapse relative to deterministic search? --- # 61. Recommended first day 1. Build the frozen graph/dependency generator. 2. Run Piper native on 40 dependency puzzles, 40 weighted path problems, and 20 infeasible cases. 3. Run deterministic algorithms on the same set. 4. Add Piper-generated heuristic to A*. 5. Only after that evaluate real work-ledger plans. The first result you want is: > **Is Piper’s weakness plan search, or is it constructing the correct state/action model?** Those imply very different interventions. --- # 62. Likely failure patterns Predictions to test: 1. Piper will perform well on short dependency plans. 2. Performance will degrade with horizon and branching factor. 3. She will generate plausible but nonoptimal routes. 4. She will occasionally omit implicit preconditions. 5. Dijkstra/A* will dominate on formal graph tasks. 6. Piper-generated heuristics may substantially reduce search. 7. Resource scheduling will be weaker than dependency ordering. 8. Piper will overestimate progress on blocked projects. 9. Dynamic replanning will expose stale assumptions. 10. The most useful production intervention may be blocker management rather than shortest-path search itself. These are hypotheses, not assumptions for scoring. --- # 63. References 1. **Sun, H. et al. (2026). _Agent Planning Benchmark: A Diagnostic Framework for Planning Capabilities in LLM Agents._** arXiv:2606.04874. https://arxiv.org/abs/2606.04874 2. **Liu, J. et al. (2026). _AdaPlanBench: Evaluating Adaptive Planning in Large Language Model Agents under World and User Constraints._** arXiv:2606.05622. https://arxiv.org/abs/2606.05622 3. **Chen, Z.-K. et al. (2026). _PeakBench: Benchmarking Resource-Aware Tool Invocation in LLM Agents._** arXiv:2608.24509. https://arxiv.org/abs/2608.24509 4. **Geng, L. & Chang, E. Y. (2025). _REALM-Bench: A Benchmark for Evaluating Multi-Agent Systems on Real-world, Dynamic Planning and Scheduling Tasks._** arXiv:2502.18836. https://arxiv.org/abs/2502.18836 5. **Palmieri, J., Lippi, M., & Marino, A. (2026). _Hybrid task planning and scheduling in heterogeneous multi-agent systems based on LLMs and constraint programming._** Robotics and Autonomous Systems, 202. DOI: 10.1016/j.robot.2026.105480 6. **Helmert, M. & Röger, G. (2015). _A Beginner's Introduction to Heuristic Search Planning._** AAAI tutorial. https://ai.dmi.unibas.ch/misc/tutorial_aaai2015/ 7. **PDDL4J Documentation. _Writing Your Own Planner — A* Search._** https://pddl4j.readthedocs.io/en/stable/writing_your_own_planner.html 8. **Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). _A Formal Basis for the Heuristic Determination of Minimum Cost Paths._** IEEE Transactions on Systems Science and Cybernetics. 9. **Dijkstra, E. W. (1959). _A note on two problems in connexion with graphs._** Numerische Mathematik. --- # 64. Bottom line Piper already has infrastructure for remembering and executing work. Component 13 should determine whether she can actually choose a good route. The core model is: > **current state → actions → state-space graph → goal state** For problems that can be formalized, Piper should not be forced to simulate graph search inside a language-model context window. Use algorithms that are already excellent at search. The more interesting role for Piper may be: - understanding the goal; - constructing the state and action model; - identifying hidden constraints; - assigning costs; - estimating distance-to-goal; - deciding when the model is wrong; - repairing the plan when reality changes. That leads to a concrete hypothesis worth testing: > **Piper may be more valuable as the heuristic inside a planner than as the planner itself.** PLANNING-v1 is designed to determine whether that is true.