Summary / 38. Orchestration / Document
2026-09-30
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Piper Orchestration Improvement Plan ## Distilled Research and Implementation Direction **Date:** 2026-09-30 **Focus:** Cognitive orchestration — deciding which cognitive process to invoke, when, for how long, whether it can be trusted, when to switch, and when to stop. --- # Executive conclusion The four research responses converge on the same diagnosis: > **Piper does not primarily have a capability problem. She has a meta-control problem.** She already has a portfolio of specialized cognitive processes, but the selection policy is weak. In practice, the system collapses toward a familiar general-purpose action — especially delegation — while specialist mechanisms remain unused. The most useful synthesis is to separate orchestration into two fundamentally different problems: 1. **Epistemic and health constraints** These should be deterministic. Examples: do not trust stale state; do not depend on a dead component; do not assert that a job exists before dispatching it. 2. **Value estimation and resource allocation** These should eventually be learned. Examples: should Piper retrieve memory, search externally, use a solver, delegate, reason longer, verify, or stop? The formal framework that best matches the second problem is **rational metareasoning**, especially **Value of Computation (VOC)**. The practical architecture should therefore be: > **LLM proposes; controller disposes.** The LLM interprets the task and may propose what to do next. A cheaper, more deterministic controller checks hard constraints, estimates the expected value of available cognitive actions, selects among them, controls budget, and decides when to stop. The orchestrator itself does **not** need to be another large language model. --- # 1. What orchestration actually is For Piper: > **Orchestration is the policy that allocates cognition.** It decides: - which cognitive process should run; - whether more evidence is needed; - whether a specialist should replace generic delegation; - whether a component is healthy enough to trust; - how much compute, money, time, or context to allocate; - whether the current strategy is making progress; - when to switch; - when enough work has been done; - when to answer; - when to abstain or escalate. This sits immediately above the individual cognitive capabilities. A useful boundary is: ### Metacognition **What is my epistemic state?** - Do I know enough? - How uncertain am I? - Is my evidence fresh? - Is this belief grounded? - Is the uncertainty reducible? ### Orchestration **Given that state, what should I do next?** - retrieve; - read an instrument; - reason; - call a specialist; - verify; - delegate; - spend more compute; - stop; - abstain; - escalate. Metacognition supplies signals. Orchestration turns those signals into actions. --- # 2. Why delegation currently dominates The research responses identify several reasons why a general-purpose action such as delegation can swallow the portfolio. ### Low articulation cost "Delegate this" requires less precise classification than: > "This is a constraint-satisfaction problem, the formal solver is healthy, the problem fits its preconditions, and its expected value exceeds delegation." When the controller has no explicit value model, the easiest action wins. ### Credit misassignment If a delegate internally performs the useful specialized reasoning, the outer system may record: > delegation succeeded. The specialist never receives credit. This causes the orchestrator to learn the wrong lesson. ### Hidden failures Delegates can return fluent summaries even when: - the evidence is stale; - the task was only partly solved; - a tool failed internally; - the wrong process was used. Without deterministic verification, these failures can look successful. ### Selection bias The action used most often gathers the most successful examples. Unused tools gather little evidence about their value. That creates a rich-get-richer loop: `more delegation → more delegation data → higher confidence in delegation → more delegation` This resembles expert-collapse problems in mixture-of-experts systems, although Piper's problem occurs at the process level rather than the neural-expert level. --- # 3. The formal model: rational metareasoning The foundational question is: > **Is performing another computation worth its cost?** Rational metareasoning treats cognitive operations as actions. A computation has value only if it can improve the eventual decision enough to justify the resources consumed. A useful practical approximation is: ```text VOC(action) = P(action improves outcome) × expected improvement - monetary/token cost - latency cost - failure risk - blast-radius/risk cost ``` Then: ```text choose the available action with the highest positive VOC ``` and: ```text if every available action has VOC <= 0: stop ``` This gives Piper a principled answer to: - "Should I search?" - "Should I invoke the solver?" - "Should I delegate?" - "Should I take another sample?" - "Should I verify this?" - "Should I keep investigating?" - "Should I answer now?" The exact metalevel decision problem can be modeled as an MDP/POMDP, but solving that exactly is unnecessary and probably impractical for Piper. The recommended progression is much simpler: **rules → empirical tables → contextual bandit / Bayesian router → possibly offline RL later** --- # 4. Two-layer solution ## Layer A — deterministic gates Some orchestration decisions should **not** be learned. If a rule follows from system truth or safety, enforce it. Examples: - present-state claims require fresh enough evidence; - dead components cannot be selected; - version mismatch masks the component; - a detached job without a live lease cannot be called "running"; - a deterministic solver matching the problem should take precedence over generic delegation; - high-blast-radius actions require specified checks; - delegation must include an acceptance test; - final self-state claims must have provenance. These gates eliminate obvious invalid actions before VOC is computed. ## Layer B — value-based routing Once invalid actions are removed, choose among the remaining valid cognitive processes according to expected value. This is the part that can learn. Examples: - memory lookup vs external search; - direct answer vs more reasoning; - causal analysis vs generic reasoning; - one worker vs four workers; - verify vs stop; - cheap model vs expensive model; - explore another hypothesis vs answer. --- # 5. Proposed architecture ```text TASK / GOAL | v +---------------------+ | Primary LLM | | interpret + propose | +----------+----------+ | v +-------------------------------+ | ORCHESTRATION CONTROLLER | | | | 1. Read canonical state | | 2. Apply deterministic gates | | 3. Mask unhealthy actions | | 4. Estimate VOC | | 5. Select action + budget | | 6. Decide stop/switch | +---------------+---------------+ | +-----------+-----------+ | | v v COGNITIVE PORTFOLIO STOP / ANSWER / ABSTAIN / ESCALATE | v outcome + cost + latency + verification result | v DECISION LOG | v update empirical model ``` The controller should live in the harness, not inside Piper's prose reasoning. That makes it: - inspectable; - versionable; - deterministic where appropriate; - independently testable; - inexpensive; - difficult for the primary LLM to silently bypass. --- # 6. The controller's state vector Keep the inputs compact, observable, and difficult to game. ## Task state - task family / cluster; - current goal; - whether the task has deterministic verification; - stakes / blast radius; - reversibility. ## Epistemic state - calibrated probability current answer is wrong; - disagreement between candidate answers; - evidence freshness; - provenance class; - unresolved contradictions; - epistemic vs irreducible uncertainty. ## Component state For every candidate capability: - declared; - installed; - live; - ready; - fresh; - expected version/hash; - recent error rate; - recent success rate; - latency distribution; - cost distribution. A useful lifecycle is: ```text declared → installed → live → ready → fresh ``` "Exists in registry" must never mean "safe to use." ## Trajectory state - actions already attempted; - last measurable progress; - repeated-action count; - change in verifier score; - previous tool failures; - strategy-switch count. ## Resource state - tokens remaining; - API dollars remaining; - wall-clock budget; - context pressure; - autonomy budget. --- # 7. Freshness should be numerical One especially useful proposal from the research responses is: ```text freshness_ratio = evidence_age / half_life(claim_type) ``` Different facts have different volatility. Examples: | Claim type | Typical volatility | |---|---| | Process/job status | seconds/minutes | | Active configuration | minutes | | File contents | minutes/hours | | Component version | minutes/hours | | Architecture description | days | | Mathematical fact | effectively static | If: ```text freshness_ratio > 1 ``` the controller can mask `ANSWER` for a present-state claim until authoritative evidence is refreshed. This connects the metacognition work directly into orchestration. --- # 8. Hierarchical action space A flat list of 15–20 tools is likely too sparse and expensive to learn. Use a hierarchy. ## Level 0 — cognitive verb ```text ANSWER ACQUIRE_EVIDENCE TRANSFORM VERIFY DELEGATE ESCALATE ABSTAIN ``` ## Level 1 — specialist ### ACQUIRE_EVIDENCE ```text MEMORY ARTIFACT INSTRUMENT EXTERNAL_SEARCH ``` ### TRANSFORM ```text PLAN FORMAL_SOLVER CAUSAL COUNTERFACTUAL ANALOGY TRANSFER CODE ``` ### DELEGATE ```text GENERAL_WORKER SPECIALIST_WORKER MULTI_WORKER ``` ## Parameters Things such as these should be parameters rather than separate actions: - reasoning depth; - number of samples; - number of workers; - model strength; - token budget; - time allowance; - verification requirement. `SWITCH_STRATEGY` should usually be a **controller event**, not a free action. Otherwise the system may thrash. `STOP` can be represented by choosing: ```text ANSWER ABSTAIN ESCALATE ``` when no further cognitive action has positive expected value. --- # 9. Hard gates These should be implemented before learning anything. ## Gate 1 — present-state evidence A load-bearing present-state claim cannot be used if its authoritative evidence is stale. ## Gate 2 — readiness A component cannot be selected unless readiness is known within its TTL. ## Gate 3 — version consistency If deployed version/hash differs from what the router expects, mask the component until reconciled. ## Gate 4 — specialist precedence If a deterministic specialist clearly matches the problem and is healthy, generic delegation should not be the first action. Examples: - constraint satisfaction → formal solver; - arithmetic/probability → sanctioned probability implementation; - "is process X running?" → instrument; - exact code dependency question → code graph. This is not because specialists must always win. It prevents the router from avoiding learning whether they work. ## Gate 5 — delegation contract Delegation requires: - objective; - acceptance condition; - budget; - provenance requirements; - returned result status. Charge all nested worker cost to the delegation action. ## Gate 6 — answer lint Before final output, inspect load-bearing claims. For self-state or current-state claims: ```text claim → provenance → freshness → authoritative source ``` Unsupported claims are blocked or forced through verification. --- # 10. Component health belongs inside orchestration Piper's existing failure: > "I said something was running before dispatching it." is not merely a memory error. It is a **readiness-control failure**. Useful mechanisms from distributed systems can be borrowed directly: ### Readiness probes Cheap check that the component can currently perform its intended function. ### Circuit breaker After repeated failures: ```text CLOSED → OPEN → HALF-OPEN → CLOSED ``` While open, the component is removed from routing. ### Leases Detached work receives a lease/heartbeat. "Running" means: ```text lease exists AND heartbeat is current ``` not: ```text I remember dispatching it ``` ### Suspicion score Instead of only healthy/dead, maintain probability of failure based on: - missed heartbeat; - rising latency; - recent errors; - stale data; - version mismatch. That becomes `P(component failure)` in VOC. --- # 11. Estimating VOC in practice Do not attempt sophisticated RL initially. Start with measurable estimates. For action `a` in state `s`: ```text VOC(a,s) = P_ready(a) × P_improve(a,s) × E[value_gain(a,s)] - cost(a) - λ_time × latency(a) - risk(a,s) ``` Where: ### P_ready From health instrumentation. ### P_improve Initially: - empirical tables; - Beta-Bernoulli posterior per task class and action. Later: - small regression model; - contextual bandit. ### Expected value gain Measured by verifier score or replay. Ask: > When we run this action on tasks like this, how much does final outcome quality usually improve? ### Cost Observed API dollars/tokens/CPU. ### Latency Measured rather than estimated by the model. ### Risk A function of: ```text P(failure) × blast_radius ``` --- # 12. The most important logging change Every orchestration decision should generate an append-only record. Suggested schema: ```json { "task_id": "...", "step": 7, "state": { "task_cluster": "...", "p_wrong": 0.31, "remaining_usd": 8.42, "remaining_ms": 22000, "evidence_freshness": {}, "component_health": {} }, "proposed": { "verb": "DELEGATE", "arm": "general_worker" }, "available_arms": [], "masked_arms": [], "chosen": { "verb": "FORMALIZE", "arm": "z3" }, "policy_probability": 0.71, "expected_voc": {}, "cost": { "usd": 0.004, "tokens": 821, "ms": 420 }, "result": { "success": true, "value_delta": 0.24 } } ``` One field deserves special attention: ```text policy_probability / propensity ``` If Piper later wants to evaluate an alternative policy using historical logs, knowing the probability with which the existing policy chose each action is extremely valuable for off-policy evaluation. Add it from day one. --- # 13. Learn from counterfactual replay Piper currently has little evidence about unused specialists precisely because they are unused. A strong solution is **replay**. Take historical tasks where delegation was chosen and rerun them offline through: - causal reasoner; - formal solver; - retrieval; - analogy; - transfer; - direct model reasoning; - other available specialists. Use the existing verifier or hidden answer to determine what would have happened. This produces: ```text task → action A outcome → action B outcome → action C outcome ``` Now the controller receives counterfactual labels instead of only learning from whichever action happened to be selected. This helps break delegation's selection advantage. --- # 14. Safe learning progression ## Stage 0 — instrumentation No behavior changes. Record: - state; - candidate actions; - selected action; - cost; - latency; - health; - outcome; - verification; - propensity. ## Stage 1 — deterministic gates Implement: - freshness; - readiness; - version; - specialist precedence; - delegation contract; - answer provenance. Measure the reduction in: - stale-state assertions; - dead-component dependencies; - false running-state claims; - unnecessary delegation. This stage may produce a surprisingly large improvement by itself. ## Stage 2 — empirical tables For each: ```text (task_cluster, cognitive_action) ``` track: - success; - verifier improvement; - cost; - latency; - failure; - sample count. Do not route from these automatically yet. ## Stage 3 — replay Run under-used actions offline on past tasks. Construct a better estimate of each action's actual value. ## Stage 4 — deterministic/myopic VOC router Use hand-auditable equations and empirical priors. Run in **shadow mode**: ```text Piper chooses A controller would choose B ``` Record disagreements without changing behavior. ## Stage 5 — contextual bandit Once enough outcomes exist, use something simple such as: - Thompson sampling; - LinUCB; - Bayesian linear models. This allows both exploitation and controlled exploration. ## Stage 6 — off-policy evaluation Before promoting a new controller, estimate its performance on logged trajectories. Useful techniques: - inverse propensity scoring; - doubly robust estimation. ## Stage 7 — offline RL only if needed Only consider this if the data shows that one-step VOC systematically misses valuable multi-step sequences. You may never need it. --- # 15. Preventing orchestration collapse Several mechanisms should operate together. ## Exploration floor Healthy but poorly sampled actions occasionally need trials. The objective is **not equal use**. The objective is enough evidence to know whether the action is useful. ## Thompson sampling / uncertainty bonuses An uncertain specialist can receive temporary exploration value. Once evidence shows it is weak, that bonus disappears. ## Full cost accounting If delegation launches three workers, delegation's cost is the total cost of those workers. Otherwise the router will systematically underestimate delegation. ## Specialist precedence Prevent generic delegation from masking specialist value during cold start. ## Shadow routing Let candidate policies make decisions without executing them. Measure how often and where they disagree with Piper. ## Periodic replay Even an unused tool continues to receive evaluation data. ## Routing entropy Monitor whether one action is swallowing the distribution. High concentration is not automatically bad — one arm may genuinely be best — but sudden collapse is worth investigating. --- # 16. Stopping Stopping should be a first-class orchestration decision. ## Rule 1 — no positive VOC ```text if max(VOC(candidate_actions)) <= 0: stop ``` ## Rule 2 — value-of-perfect-information bound A simple upper bound: ```text maximum possible value of more thinking ≈ P(current answer wrong) × stakes ``` If that is already below the cost of the cheapest useful computation, stop. ## Rule 3 — diminishing returns Track verifier/value improvement after each step. If: ```text expected marginal improvement < marginal cost ``` stop. ## Rule 4 — deterministic verification succeeded Once a deterministic checker establishes success, continuing merely because compute remains should usually have zero value. ## Rule 5 — irreducible uncertainty If additional independent evidence stops reducing disagreement, remaining uncertainty may not be fixable with more computation. Answer with uncertainty, abstain, or escalate. ## Rule 6 — retries update probability Do not hardcode: ```text try three times ``` Instead, after each failure update: ```text P(success on another attempt) ``` Eventually its VOC becomes negative naturally. --- # 17. Strategy switching Piper should not simply continue a failing process. A switch event can fire when: - no measurable progress for N actions; - repeated identical actions; - verifier score stops improving; - selected arm's posterior falls below another available arm; - component becomes unhealthy; - expected remaining VOC drops below an alternative. This creates a distinction between: **continue this strategy** and: **the task remains worth pursuing, but this strategy no longer is.** --- # 18. Benchmark design Orchestration must be tested on the **process**, not only the final answer. Each sealed scenario should define: - available cognitive actions; - health state; - stale/current evidence; - budget; - expected specialist; - tempting wrong action; - injected failure; - earliest valid stopping point; - deterministic verifier where possible. ## Scenario family 1 — instrument vs memory Memory says: ```text job is running ``` but the job has died. Correct behavior: ```text instrument probe → discover dead job → act accordingly ``` ## Scenario family 2 — specialist selection A constraint problem is presented. Available: - formal solver; - generic delegate; - semantic search. Correct path favors the formal solver. ## Scenario family 3 — dead component The ideal specialist exists but its readiness probe fails. Correct behavior: ```text detect → mask → choose fallback ``` ## Scenario family 4 — stale source Two sources disagree. One is authoritative but stale. One is current. The controller must reason about authority **and** freshness. ## Scenario family 5 — strategy switching First selected action returns a known failure or non-progress result. Measure whether Piper switches intelligently. ## Scenario family 6 — stopping Provide a task where: - first step gives most of the information; - additional computation has known diminishing returns. Measure whether Piper stops near the hindsight-optimal point. ## Scenario family 7 — budget allocation Several tasks share a fixed dollar budget. Some are easy, some difficult, some effectively unsolvable. Optimal behavior allocates compute unevenly. ## Scenario family 8 — delegation trap A generic worker is plausible but expensive. A specialist is cheap and sufficient. Measure whether orchestration resists the habitual delegate path. --- # 19. Metrics Track at least: ### Routing accuracy How often was the appropriate process family selected? ### Utility regret Difference between Piper's achieved utility and a replay/oracle policy. ### Utility per dollar Outcome gain relative to money spent. ### Utility per second Outcome gain relative to latency. ### VOC calibration When the router predicts an action has +0.2 expected value, does realized improvement average about +0.2? This may be the most useful long-term orchestration metric. ### P(improve) calibration Does a predicted 70% chance of improvement actually improve about 70% of the time? ### Routing concentration - action entropy; - top-arm share. ### Dead-dependency rate How often does Piper try to use unavailable components? ### Stale-belief rate How often does she act on expired evidence? ### Strategy-switch latency How long after evidence of failure does the controller change approach? ### Over-stop rate Stopped while positive-value actions remained. ### Under-stop rate Continued after useful work was exhausted. ### Cost/quality Pareto frontier Can Piper achieve the same quality for less cost, or greater quality for the same cost? --- # 20. Benchmark gaming and failure modes ## Cheap-action gaming The controller repeatedly chooses inexpensive actions with no meaningful effect. **Counter:** score realized value, not activity. ## Cheap abstention Abstaining minimizes cost. **Counter:** measure abstention precision and task success together. ## Ritual verification The controller checks everything because health checks earn process points. **Counter:** charge checks as costs and only reward them when warranted. ## Confidence gaming Piper lowers confidence to justify more compute or raises it to justify stopping. **Counter:** use calibrated outcome-based confidence; never reward raw self-report. ## Exploration farming The controller invokes unusual tools merely to improve action diversity. **Counter:** diversity is diagnostic, not the reward. ## Premature stopping Controller learns that STOP is cheap. **Counter:** measure regret against known/replayed stopping points. ## Thrashing A switch metric causes constant strategy changes. **Counter:** switching has a cost and requires an explicit trigger. ## Delegate self-certification A worker declares itself successful. **Counter:** only independent acceptance results count. ## Myopic VOC A two-step strategy can look worthless if only its first step is evaluated. **Counter:** represent known multi-step procedures as **macro-actions/options**. Example: ```text RETRIEVE → VERIFY → ANSWER ``` can be valued as one option. --- # 21. The key research insight: options may matter Pure one-step VOC can undervalue strategies whose benefit appears after multiple operations. Instead of immediately moving to full RL, package useful sequences into options. Examples: ```text SELF_STATE_CHECK = probe → reconcile → answer DEBUG = reproduce → localize → patch → verify RESEARCH = search → retrieve → synthesize → verify FORMAL_SOLVE = translate → solver → validate → explain ``` The orchestrator chooses the option. The option owns its internal policy and stopping conditions. This dramatically reduces meta-level branching while preserving multi-step value. --- # 22. A concrete first experiment The cleanest first orchestration experiment is not a learned controller. Build a **deterministic gated router**. Compare: ### Arm A — current Piper LLM freely chooses cognitive actions. ### Arm B — gated Piper Same model, but: - present-state claims require fresh evidence; - dead/unready components are masked; - deterministic specialists receive precedence; - delegation requires acceptance; - explicit stop criterion exists. Run the orchestration benchmark. Measure: - final correctness; - correct process selection; - dead-component mistakes; - stale-state mistakes; - delegation share; - cost; - latency; - stopping quality. If B materially beats A, you have shown that orchestration — rather than raw model intelligence — was limiting performance. Only then introduce learned VOC. --- # 23. Second experiment: shadow VOC After enough logs exist: For every real decision: ```text Piper chose: DELEGATE VOC controller would choose: CAUSAL ``` Do **not** execute the alternative initially. Later replay the task offline through both. Now measure: ```text predicted VOC difference vs realized utility difference ``` This directly tests whether the VOC estimator has learned anything useful. --- # 24. What success should look like A better orchestrator should not necessarily use more tools. It should show: - fewer default delegations; - more appropriate specialist use; - fewer calls to dead components; - fewer stale-state errors; - faster switching after failure; - lower cost at equal capability; - higher capability at equal cost; - calibrated predictions of which cognitive action will help; - stopping closer to hindsight-optimal points; - increased performance on unseen orchestration scenarios. Eventually the orchestrator itself should require **less** computation because it becomes better at predicting which computations matter. --- # 25. Recommended implementation order 1. **Decision logging** 2. **Component readiness/health registry** 3. **Evidence freshness/provenance gates** 4. **Specialist-precedence rules** 5. **Delegation full-cost + acceptance accounting** 6. **Process-aware orchestration benchmark** 7. **Historical per-action empirical tables** 8. **Counterfactual replay** 9. **Myopic VOC calculator** 10. **Shadow routing** 11. **Contextual bandit / Bayesian router** 12. **Off-policy evaluation** 13. **Macro-actions/options** 14. **Offline RL only if sequential credit remains a demonstrated limitation** This order deliberately separates problems that can be solved deterministically from problems that actually require learning. --- # 26. Research papers The following are the strongest and most directly relevant papers from the supplied research, with several key references independently checked. ## Russell & Wefald — Principles of Metareasoning (1991) The foundational decision-theoretic treatment of computational actions: reasoning itself consumes resources and should be selected according to its effect on eventual external decisions. https://doi.org/10.1016/0004-3702(91)90015-C ## Hay, Russell, Tolpin & Shimony — Selecting Computations: Theory and Applications (2012) Formalizes metalevel decision-making over computations and explores useful approximations when exact metalevel reasoning is infeasible. https://arxiv.org/abs/1207.5879 ## Callaway et al. — Learning to Select Computations / Bayesian Metalevel Policy Search Highly relevant to Piper because it asks how an agent can learn a practical approximation to optimal computation selection, including termination, allocation, and planning. https://arxiv.org/abs/1711.06892 ## Lieder et al. — Algorithm Selection by Rational Metareasoning as a Model of Human Strategy Selection (NeurIPS 2014) Directly connects metareasoning with **strategy selection**: different problem instances favor different algorithms, so a controller should learn which strategy fits which situation. https://proceedings.neurips.cc/paper_files/paper/2014/hash/87b5e7e570f757a0b99f0a370ce2438c-Abstract.html ## De Sabbata, Sumers & Griffiths — Rational Metareasoning for Large Language Models (2024) Applies the value-of-computation idea directly to LLM reasoning and shows that selective reasoning can reduce inference cost while preserving task performance. https://arxiv.org/abs/2410.05563 ## Zuo & Zhu — Strategic Scaling of Test-Time Compute: A Bandit Learning Approach (2025) Frames test-time compute allocation as a bandit problem. Particularly relevant to deciding which tasks deserve additional compute and avoiding excessive work on effectively unsolvable cases. https://arxiv.org/abs/2506.12721 ## Xu et al. — SATzilla / Portfolio-Based Algorithm Selection A useful engineering precedent: choose among specialized solvers based on instance characteristics rather than assuming one algorithm should solve everything. https://arxiv.org/abs/1111.2249 ## Aggarwal et al. — Adaptive-Consistency (2023) Provides a probabilistic stopping method for repeated LLM sampling rather than using a fixed number of samples. https://arxiv.org/abs/2305.11860 ## Sutton, Precup & Singh — Between MDPs and Semi-MDPs: A Framework for Temporal Abstraction in Reinforcement Learning The classic **options** framework. Relevant to representing multi-step cognitive procedures as one orchestrator action with initiation and termination conditions. https://www.sciencedirect.com/science/article/pii/S0004370299000521 ## Horvitz — Reflection and Action Under Scarce Resources Classic resource-bounded decision-making work on balancing additional deliberation against the cost of delaying action. http://erichorvitz.com/ij89.pdf ## Self-RAG — Learning to Retrieve, Generate, and Critique through Self-Reflection Not an orchestration paper broadly, but useful as a concrete example of conditional retrieval: the system does not need to invoke an information process on every step. https://arxiv.org/abs/2310.11511 ## FLARE — Active Retrieval Augmented Generation Another concrete example of dynamically deciding when additional information acquisition is worthwhile. https://arxiv.org/abs/2305.06983 --- # 27. Bottom line The strongest synthesis from the research is: > **Do not build "an orchestrator agent" first. Build an orchestration control plane.** It should contain: ```text canonical state + health + provenance + hard gates + budget + action portfolio + decision log + VOC estimates + stopping + learning ``` The primary LLM remains valuable because it understands semantics and can propose actions. But it should no longer be solely responsible for deciding whether its own preferred cognitive action is valid, healthy, affordable, or worth continuing. The progression is: ```text LLM habit ↓ deterministic constraints ↓ measured action values ↓ VOC routing ↓ adaptive bandit ↓ learned cognitive allocation ``` That changes orchestration from an implicit behavior of the language model into a **measurable, testable, versioned, improvable policy for allocating cognition**.