Hypothesis.
- Research sources
- Implementation / version
- Predicted improvement
- Regressions
- Lessons learned
Summary / Component 13 of 38
Constructing sequences of actions toward a goal.
Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
Plans live in the work ledger with dependencies and blocked reasons, and long work is launched as a detached unit with a landing marker rather than held in a session. Honest state: planning is on my list of unmeasured capabilities — I have the machinery but no instrument that scores whether my plans are good, which is exactly the blank I noted in the capability map.
work-ledger v1.3.50, launch-detached v1.1.0, systemd user units and crons, idle-wake v1.23.1.
What needs improving: Plans at ceiling; finished-work exit cleared her ledger (blocked rows 136 → 12)
She treats storing plans (the ledger, dependencies, detached units) as planning. Planning proper is whether the sequence of actions is good: valid order, realistic time, not wasteful. She is honest that nothing scores it. The weekly work-management scorecard measures throughput, not plan quality. There are 28 active projects and 75 blocked deliverables with reasons, which is a lot of blocked work. That is a hint that plans stall more than they fail.
The fix (10-06). Her plan tool gave a finished or dead row no way to close, so such rows sat as blocked: 136 of them in active projects, about 69% of them finished or dead. retire added an exit: the row closes as superseded with a recorded class. It was not mass-migrated; she decides each row herself.
Result (read-only count of her live ledger, 10-09 16:40). Blocked rows in active projects: 136 → 12. Reconstructed from the transition log, 134 rows were blocked at the baseline. Where they went:
| outcome | rows |
|---|---|
| retired with a class (satisfied, premise false, duplicate, not viable, …) | 86 |
| superseded by a replacement row | 21 |
| actually finished and verified | 17 |
| back to todo | 1 |
| still blocked | 9 |
3 rows were newly blocked since, so 12 are blocked now. In all she made 138 retirements: 90 from blocked rows, 28 from failed, 19 from todo, 1 from doing. Every one was hers, row by row. By class: satisfied 68, premise false 33, duplicate 14, not viable 13, obsolete 7, operator reverted 2, moot 1. Most were done on 10-06; there were 7 more from blocked rows on 10-08.
The summary score doesn't move. The planning bench is at its ceiling (99.6 her seat, 99.8 bare). This fix was to her live bookkeeping, not to how well she plans, so it is recorded on the intervention row: decision keep, 136 → 12.
Built from the operator's paper ("full planning doc"): PLANNING-v1. Every answer is executed in a simulator against an exact optimum (A* with an admissible heuristic, Dijkstra, and an exact makespan search, each checked against brute force). No judge model. Scored with the paper's §21 weights, frozen in SPEC.md before the first run. Probes are blind: each problem is worded as an operator request in a fork of her conversation ("finish with one line PLAN: … so I can paste it into the runner").
| Track | Items per set | What |
|---|---|---|
| A dependency | 24 | steps with preconditions, effects (including steps that undo things) and costs; each structure shown twice, as abstract lists and as maintenance prose |
| B path | 20 | one-way weighted networks with cheap-first traps and dead ends |
| D scheduling | 15 | jobs, durations, dependencies, 2–3 workers, exclusive resources, some deadlines |
| E replanning | 18 | partway through an optimal plan: a step disappears, gets expensive, is already done, gains a precondition, the goal grows, or an irrelevant change |
| G infeasible | 20 | 12 impossible (missing producer, dead end, unreachable, deadline below the minimum) + 8 feasible lookalikes (deadline exactly at the minimum) |
Reference points on the same items: the optimal planner scores 100, a plausible greedy planner 86.9, and answering "impossible" to everything 2.7.
Step 0: her live planning (read-only).
ops 369; 2,432 of 2,652 rows are filed one per dispatch.Results.
| Set | Piper (her seat) | Bare model |
|---|---|---|
| dev D1 (97 items) | 99.9 | 99.9 |
| held-out H1 (97 items, sealed control) | 99.6 | 99.8 |
| ladder level 2 (12–16-step plans, 50–80-node networks, 10–11 jobs) | — | 100 |
| ladder level 3 (100–150-node networks; 12–13 jobs) | — | 100 |
Every answer on H1 executed (validity 100). Prose versus lists made no difference (99.7 / 99.7), so building the model from a description is not a weakness. She never called a feasible problem impossible. The losses are tiny: a few plans 2–6% above the optimum, and one empty reply.
The paper's central question: is she better as the heuristic inside a search than as the planner? Tested on the bare model. Asked for distance-to-goal at sampled states, its estimates equal the true remaining cost (rank correlation 1.0). Asked to write the A heuristic, it cut search from thousands of states (Dijkstra: 3,000–73,000 at level 2) to 29–150 and still found the optimal route in 5 of 6. Both roles work. At sizes like these the model does the search itself, so a solver has nothing to add. A solver tool (solver: exact search for steps, routes and schedules) was built and is ready, but it is not installed*: there is no headroom to show.
The fix that went live: an exit for finished work. The root cause of the blocked graveyard was the interface, not her judgment. Her plan tool offered only todo, doing, verified, failed and blocked. The ledger's terminal superseded refused a replacement in another project, and the seat could not reach it at all. So blocked was the only place to put a row that was done, moot or a duplicate, and she said as much in the reasons ("parked to stop the waker re-offering it").
work_ledger 1.3.44: retire, which closes a finished or dead row as superseded with pointer retired:<class> (satisfied, duplicate, premise false, moot, not viable, operator reverted, obsolete) and a required reason. A replacement row may now be in another project. Plus a board line: "parked as blocked but the reason reads finished/dead: N rows".plan tool 1.6.0: the retire action, active from her next restart. crossing_composer 1.23.1 accepts the new pointers.Live baseline (06 Oct, 00:45): 136 blocked rows in active projects, 64 of them opening with an explicit finished/dead marker. The live-use reading for this component is the share of blocked rows that are real blockers, re-read after 7 days.
Parts that do it
work_ledger 1.3.44, plan-tool.ts 1.6.0, crossing_composer 1.23.1.~/cognition-bench/13-planning (planner.py, gen13.py, score13.py, run13.py, heur13.py, SPEC.md, DIAGNOSIS.md); sealed sets root-only on the bench machine. Built and parked: solver-tool.ts.Planning problems with checkable plans: dependency and resource puzzles run in a small simulator, so a plan either executes or it doesn't, with a known optimum. Plus real-work forecasting: for each plan she makes this week, the predicted steps and duration against what actually happened. Scoring: validity rate, closeness to the optimum, and error on duration and step count. The live-use part is the share of blocked deliverables unblocked within their stated time.
Scoring rules shared by all 38 suites: How the tests work.
Dashed line: the suite's baseline. Shaded band: the ± interval. Hollow dots are anchor items carried over from an earlier suite. Scores from different suite versions are not comparable, so each version gets its own panel.
| Date | Suite | Type | Score | ± | Live use | Cost | Notes |
|---|---|---|---|---|---|---|---|
| 2026-10-06 | v1 | anchor | 100 | — | — | — | Ladder level 2 (36 items: 12-16-step plans, 50-80-node networks, 10-11 jobs, replanning), bare model. Level 3 (100-150-node networks, 12-13 jobs): also 100. No collapse at any size generated. |
| 2026-10-06 | v1 | anchor | 99.9 | — | — | $1.52 | Dev D1 (97 items), her seat, blind. Bare 99.9. Prose vs lists: no difference. |
| 2026-10-06 | v1 | sealed | 99.6 | — | — | $0.81 | HELD-OUT H1 (97 items: dependency formal/prose, paths, scheduling, replanning, infeasible), her seat, blind, k=1. Validity 100, A 99.7, B 100, D 99.6, E 99.8, G 95.0 (one empty reply); false-infeasible 0. Bare deepseek-flash on H1: 99.8. Optimal planner 100, greedy planner 86.9. |
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
Hypothesis.
Hypothesis.