Summary / 21. Multi-agent development / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 21 — Multi-agent Development (spawn, not summon) > Derived from `papers/21-multi-agent-development.md` — "Spawn, Not Summon: Artifact-Mediated, Evidence-Grounded, Conditionally-Activated Specialist Workers Under a Fully Deterministic Orchestration for a Weak-Model Coding Harness." > This document drives the build of the **multi-agent stage** as a *division-of-labor* layer, not a *division-of-attention* one. The > harness already owns every stage a naive multi-agent design would try to re-create: component 07 decides *what* a task decomposes > into, component 09 decides *how* a patch is produced, and component 18 decides *whether* a patch is accepted. This component > decides the one remaining question — *who* does which piece of work, under what context, tools, and authority, and how the pieces > are stitched back into one artifact without ever letting two models form an uncontrolled consensus loop. The spine distinction is > between "one agent that can call specialists" and "five copies of the same model in a chat room." Every downstream stage already > exists as a single well-tooled agent; this component decides, per task, whether to split that single agent into short-lived > specialist workers — and it demands that every split earn its keep in measured outcome, never in rhetorical vividness. The Piper > rule, made mechanical: **Don't build a room full of Pipers talking to each other. Build one Piper that can spawn narrow > specialists, give each exactly one job and exactly the evidence it needs, then kill the specialist and keep only its verified > artifact.** A role is a five-tuple of input schema, output schema, allowed tools, context policy, and authority — never a > persona. The orchestration (role instantiation, artifact typing and routing, state ownership, stuck-detection, escalation-on- > alignment-failure, evidence arbitration, and the spawn/retire decision) is fully deterministic software on CPU [D]; the control > plane holds **zero** learned entries. The specialist workers are the [L] entries, each a single-shot, stateless leaf making one > bounded model call that produces one typed artifact, then terminated; every inter-role handoff is mediated by the deterministic > orchestrator through a typed artifact, so no two models ever form a consensus or debate loop. The invariant that governs the > whole is **the L column contains at most one entry** — and, for this component specifically, **an empty L column is a permitted, > correct state**: a task that needs no specialist is handled by the single well-tooled worker and the deterministic path, and that > is the *default*, not the fallback. This spec pins the paper's design to exact primitives: the exact Role five-tuple schema, the > exact typed-artifact enum of four, the exact termination-contract schema of four bounds, the exact D/L invariant, the exact > no-worker-to-worker-channel discipline, the exact evidence-arbitrated judge, the exact conditional spawn policy and > escalation-on-alignment-failure, the exact honest A/B instruments (resolve rate + tokens + wall time + LLM calls + regressions + > invalid patches + context size, reported together), and an exact build sequence. --- ## 1. Purpose & Scope Build a **deterministic multi-agent orchestration** — a control plane that, given one task already decomposed by component 07, decides *per subtask* whether to instantiate a short-lived specialist worker (the L column's at-most-one entry) or to continue on the single well-tooled worker's deterministic path (an empty L column). When a specialist is spawned, the orchestration constructs it from a Role five-tuple, hands it exactly the typed artifacts its subtask needs, receives exactly one typed artifact back, validates that artifact against its schema, arbitrates it against deterministic evidence, and then **retires the worker immediately** — its context is discarded, its output merged into the single authoritative state, and nothing of it persists except the verified artifact. The component's job is not to make the single agent succeed more often; it is to make the *split* — when it happens at all — earn its keep, so that a weak model's scarcest resource (its small, precious context window) is spent on the one judgment a specialist was spawned to make, and never on reconstructing relevance from a long conversation. The scope is the division-of-labor layer only. Component 07 owns *what* a task decomposes into; component 09 owns *how* a patch is produced; component 18 owns *whether* a patch is accepted. This component owns the *who*: the decision to instantiate a role, the context that role is handed, the tools it may call, and the authority its output carries. Everything upstream and downstream is a deterministic stage that already knows what to do next and already holds the evidence it needs; the only question this component answers is whether one of those stages should be *subdivided into roles* — and the field's measured record (260 configurations reaching −70.0% on sequential, tool-heavy tasks; multi-agent systems underperforming a single agent across all three code-centric tasks) says the answer is "rarely, and only when you can measure that it helps." The paper's discipline compresses into three words that carry the entire design. **Modular** means a role is instantiated only where different subtasks genuinely benefit from different context, tools, objectives, or authority — never for a persona's vividness. **Artifact-mediated** means every inter-role handoff is a typed artifact with a schema and a parser, never a free-running conversation history; the model's prose is confined to the *inside* of an artifact field, and the structure, routing, and validation of the artifact are deterministic. **Conditional** means the decision to activate any additional role is a deterministic policy over task-shape features — never a model's judgment of whether a specialist "feels right" — and it is paid for in measured outcome, one role at a time. Three principles decide every design choice. **P1 Spawn, not summon** — a specialist is *spawned* for exactly one bounded judgment and *retired* the moment its typed artifact validates; a specialist that is "kept around" is a conversation that has not ended, and a persistent worker is the only kind whose context cannot be guaranteed clean. **P2 One authoritative state, one deterministic control plane** — there is exactly one repository checkout, one working tree, one task ledger, and one hypothesis/plan state, owned and mutated only by the orchestration; a worker that produces a patch produces a *candidate* artifact that the orchestration applies — or declines to apply. **P3 The D/L invariant** — role instantiation, artifact typing and routing, state ownership, stuck-detection, escalation decision, evidence arbitration, and the spawn/retire decision are all deterministic software on CPU; the L column holds at most the specialist worker itself, and **an empty L column is a permitted, correct state** — a no-specialist task is handled by the single well-tooled worker and the deterministic path, and that emptiness is the system working as designed, not a missing entry to be filled. The build lives in `build/21-multi-agent-development/`, writes nothing into the live harness, and is kept **only if** every role it adopts strictly earns its keep against the baseline of the single well-tooled agent plus the validator, on the honest measures of §9 — above all **Δsuccess per Δcost** (never resolve rate alone), with tokens, wall time, LLM calls, regressions, invalid patches, and context size reported *together*. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–9); **DEFERRED = later rounds** (rows 10–11), **not excluded** but gated by the same §9 A/B protocol. Every later round of this component still builds within the same `build/21-multi-agent-development/` tree. | # | Component (paper §2/§3/§4) | Classification | |---|---|---| | 1 | Orchestration core — one authoritative repository checkout, working tree, task ledger, hypothesis/plan state, owned only by the orchestration (§3.1) | **BUILD-NOW** | | 2 | Typed-artifact protocol + parsers — the closed set `spec`/`patch`/`test-report`/`review-verdict`, each with schema + parser, immutable once written (§3.2) | **BUILD-NOW** | | 3 | Role five-tuple contract — `input_schema` / `output_schema` / `allowed_tools[]` / `context_policy` / `authority` (§4) | **BUILD-NOW** | | 4 | Spawn/retire policy — deterministic fixed policy over task-shape features, empty-L default (§3.3) | **BUILD-NOW** | | 5 | Termination contracts — `max_model_calls` / `max_token_budget` / `max_wall_clock` / `max_rounds`, enforced not advisory (§3.6) | **BUILD-NOW** | | 6 | Stuck-detection — N rounds without an advancing artifact / M recurrences of one failure signature → replan / re-scope / abort (§3.4) | **BUILD-NOW** | | 7 | Alignment-check + conditional escalation — alignment checked every handoff, group review only on failure (RTADev) (§3.5) | **BUILD-NOW** | | 8 | Validator/judge worker — the one unconditionally-adopted separation, arbitrated against the deterministic `test-report` (§5.1) | **BUILD-NOW** | | 9 | Fault-injection + A/B driver — role-at-a-time ablations + the Δsuccess/Δcost decision rule (§9) | **BUILD-NOW** | | 10 | **Additional specialist roles** — localizer, reproducer, specifier, planner-specialist, and any further role beyond the validator/judge | **DEFERRED** — each is added singly via the §9 ablation, never shipped as a room; a role survives only on measured Δsuccess/Δcost | | 11 | **Learned spawn decision** — any model-based "should I spawn a specialist" or "which role" judgment | **DEFERRED / REJECTED-BY-DEFAULT** — the paper's deterministic-first stance forbids it; the spawn/retire decision is a fixed policy over task-shape features, never a model's judgment | **The cost rule (explicit).** The L column holds at most one entry — the specialist worker — and nothing else. In Round 1 that entry is produced by the harness's own weak model (`deepseek-v4-pro`) on exactly the same terms as the happy path; it is **not** a new learned node and it is **not** trained. The model is handed a clean, task-scoped context and asked to emit one typed artifact; its cost is bounded to one model call per instantiation (or a strictly bounded short sequence where the termination contract permits), and it never decides *that* a specialist should spawn, *which* role it should be, *what* authority it should carry, or *whether* its own output parses — those are all deterministic. The DEFERRED rows (additional roles, and any hypothetical learned spawn decision) must clear the same bar as in every other component: beat the deterministic baseline on the honest metrics (§9), on held-out tasks, at a defensible cost, or be rejected — and a learned spawn decision is the one addition that is **rejected by default**, because it puts L into the control plane and forfeits every guarantee the deterministic orchestration provides (§3.7). **Design decision: DETERMINISTIC-FIRST, SINGLE-L.** Round 1 puts no model anywhere in the control plane — not in role instantiation, not in artifact typing and routing, not in state ownership, not in stuck-detection, not in escalation decision, not in evidence arbitration, not in the spawn/retire decision. The full deterministic pipeline (rows 1–7) is the build; the single L entry is the specialist worker, produced only *after* the deterministic control plane has decided to spawn it, constructed its five-tuple, and assembled its clean context. --- ## 3. The Deterministic Orchestration (D) The orchestration is the control plane, and it is where the thesis's "modular, artifact-mediated, evidence-grounded, and conditional" clauses become software. Every function in this section is deterministic software on CPU; the L column contributes nothing here, and the invariant is stated at the outset of the architecture and re-stated at its close: **the L column contains at most one entry**, and the orchestration holds zero learned entries. Concretely, the control plane's pipeline is: ``` TASK ENTERS (single well-tooled worker on the deterministic path) → own_state [D] → one repository checkout, one working tree, one task ledger, one hypothesis/plan state (§3.1) → route_artifact [D] → type + parse + route every handoff (spec | patch | test-report | review-verdict) (§3.2) → spawn_or_retire [D] → deterministic policy over task-shape features; empty L = continue single-agent (§3.3) → detect_stuck [D] → N rounds without an advancing artifact, or M recurrences of one failure signature (§3.4) → check_alignment [D] → every handoff; escalate to conditional group review only on failure (§3.5) → enforce_termination [D] → max_model_calls / max_token_budget / max_wall_clock / max_rounds (§3.6) → arbitrate_evidence [D] → review-verdict checked against the deterministic test-report (§5.1) → (L) specialist_worker → the at-most-one learned entry: one bounded model call → one typed artifact, then terminated (§5) ``` The model never decides whether to spawn (the policy does), never decides which role to instantiate (the policy does), never decides what tools or authority a role carries (the five-tuple does), never decides whether an artifact parses (the parser does), and never decides whether a handoff aligned (the check does). All artifacts land in `build/21-multi-agent-development/`. ### 3.1 One authoritative repository and task state [D] **Input:** the decomposed task (component 07's output) and the current harness state. **Output:** a single, owned repository and task state. **No language model runs here.** There is exactly one repository checkout, one working tree, one task ledger, and one hypothesis/plan state, owned and mutated only by the orchestration. No worker ever touches the repository directly as its *authority*; a worker that needs the repository receives a read-only snapshot or a scoped view constructed by the orchestrator, and a worker that produces a patch produces a *candidate* artifact that the orchestration applies — or declines to apply — to the single state. The reason is the error-propagation finding measured at scale: architectures without centralized verification tend to propagate errors more, and multi-agent collaboration propagates identical errors because no agent detects the mistakes in another agent's output. A single authoritative state is the mechanism that makes propagation a *detected event* rather than an invisible accumulation: there is no "agent B's slightly different copy of the repo" for an error to hide in, because there are no other copies. ### 3.2 The typed-artifact protocol [D] **Input:** any worker's output. **Output:** a typed, parsed, routed artifact admitted to the authoritative state — or a rejection. **No language model runs here.** Every handoff is a typed artifact, never a conversation history. The protocol defines a small closed set of artifact types — `spec` (the task's requirements and acceptance criteria), `patch` (a candidate diff with a cited diff-test basis), `test-report` (the deterministic outcome of running the harness's own suite, component 17), and `review-verdict` (a structured accept/revise/reject with cited evidence) — each with a schema and a parser, each routed by the orchestrator to the next stage, and each immutable once written. The model's prose is confined to the *inside* of an artifact field; the structure, the routing, and the validation of the artifact are deterministic. This is the direct lesson of MetaGPT's publish–subscribe message pool: the framework that works is the one whose messages are *typed documents*, not chat bubbles. The difference between MetaGPT's publish–subscribe and CAMEL's free conversation is the difference between a pipeline and a cocktail party, and this component encodes the pipeline. The four types and their schema+parser pairs are specified exactly in §4. ### 3.3 Spawn and retire by task [D] **Input:** the current subtask and the artifact in front of it. **Output:** either "no spawn" (continue single-agent) or a constructed Role five-tuple for exactly one specialist. **No language model runs here.** The orchestration's only question at any decision point is: *does the current subtask benefit from a different context, toolset, objective, or authority than the subtask that produced the artifact in front of it?* If not — which is the common case — no specialist is spawned, and the single well-tooled worker continues on the deterministic path; **an empty L column is a permitted, correct state**. If so, the orchestration instantiates a specialist with exactly the five-tuple of §4, routes it exactly the typed artifacts it needs, receives exactly one typed artifact back, validates that artifact against its schema, and then **retires the worker immediately** — its context is discarded, its output is merged into the authoritative state, and nothing of it persists except the verified artifact. A worker is never kept around for the next task, because a persistent worker is a conversation that has not ended. The spawn/retire decision itself is deterministic: a fixed policy over task-shape features — e.g. the subtask's tool requirement differs from the producing subtask's (localize-then-fix), the subtask's objective differs from the producing subtask's (falsify-then-construct), or the subtask's authority differs (adjudicate, not edit) — not a model's judgment of whether a specialist "feels right." The task-shape features are enumerated in §5.2. ### 3.4 Stuck-detection [D] **Input:** the task ledger and its counters. **Output:** a deterministic response — replan, re-scope, or abort — or none. **No language model runs here.** The orchestration monitors the task ledger for the two failure signatures the literature names: *lack of progress* (rounds elapsing without a state mutation that advances the task) and *repeated failure* (the same failure class recurring across retries). This is Magentic-One's Task Ledger + Progress Ledger made mechanical: here the detection is a deterministic counter over the ledger — **N rounds without a typed artifact advancing the plan**, or **M recurrences of one failure signature** — and the response is deterministic too: **replan** (re-run decomposition with a narrower scope), **re-scope** (spawn a narrower specialist with a different context policy), or **abort** to the human/operator escalation path. No model decides it is stuck; the ledger decides. ### 3.5 Escalation on alignment failure [D] **Input:** the artifact just produced and the intent the previous stage established for it. **Output:** either "aligned" (continue) or an escalation to a conditional, ad hoc group review. **No language model runs here (the check itself).** The orchestration runs an alignment check at every handoff: does the artifact just produced still satisfy the intent that the previous stage established for it? The check is deterministic where it can be — a diff applies cleanly to the single state, a patch's stated scope matches the localizer's typed verdict, a spec's acceptance criteria are all addressable by the produced patch. Only when that check *fails* does the orchestration escalate, and it escalates to a **conditional, ad hoc review** rather than an always-on committee. This is RTADev's contribution taken at full strength: its design has "an alignment checking phase and a conditional ad hoc group review phase" — the review is *conditional*, i.e. only on failure. The transfer is the default-off committee: alignment is checked on every handoff (cheap, deterministic), and the expensive group review is paid for only when alignment has demonstrably broken, which is the thesis clause "check handoff alignment before propagating upstream mistakes" made into a gate. ### 3.6 Termination contracts [D] **Input:** a worker or loop about to run. **Output:** the bounds under which it runs, and enforcement. **No language model runs here.** Every worker, and every loop the orchestration may enter, is instantiated under a termination contract: a maximum number of model calls, a maximum token budget, a maximum wall-clock, and a maximum number of rounds in any multi-step loop. The contract is not advisory — the orchestration enforces it, and a worker that exceeds any bound is terminated and its partial output discarded or handed to the stuck-detection path. CAMEL made the necessity explicit: termination conditions existed to stop the infinite "thank you"/"goodbye" loop, and Contract Net's manager–bid–award protocol is the forty-year-old proof that a *contract*, not a conversation, is what bounds a distributed computation. The termination contract is the multi-agent instance of the program's wider blast-radius discipline: no learned loop runs unbounded, because an unbounded learned loop is where a weak model's most expensive failure lives. The four bounds are specified exactly in §4. ### 3.7 The D/L invariant, stated mechanically (D) **the L column contains at most one entry.** In this component the D column is the entire orchestration of §3.1–§3.6 — role instantiation, artifact typing and routing, state ownership, stuck-detection, escalation decision, evidence arbitration, and the spawn/retire decision — all deterministic software on CPU. The control plane holds zero learned entries. The L column holds exactly one entry: the specialist worker, a single-shot, stateless leaf. The consequence that must be stated explicitly, because the field keeps getting it wrong: the workers never talk to each other — every inter-role handoff is mediated by the deterministic orchestrator through a typed artifact, so no two models ever form a consensus or debate loop, which is the design that would leak L into the control plane and forfeit every guarantee the deterministic orchestration provides. **an empty L column is a permitted, correct state**: a task that needs no specialist is handled by the single well-tooled worker and the deterministic path, and that emptiness is the design working. A debate loop is not a checks-and-balances system; it is two weak models correlating toward the fluent middle, and the deterministic artifact protocol exists so that disagreement is arbitrated by *evidence*, not by *eloquence*. --- ## 4. The Core Schemas (the build's JSON contracts) The following three schemas are the load-bearing contracts of the build. Their shapes are EXACTLY these fields, no more and no fewer, and the label "EXACTLY these fields" is itself part of the contract — a field added for convenience is a spec violation. ### 4.1 The Role five-tuple (EXACTLY these fields) A role is a boundary, never a persona. "You are a senior staff engineer, be skeptical and rigorous" is a role's five-tuple replaced by a vibe, and a vibe is not a boundary: it does not restrict what the worker sees, does not restrict what it can do, does not type its output, and does not bound its authority. The whole failure mode has a name this component treats as its warning label — **role theatre**. The five-tuple is the machine-readable form of the boundary: ```json Role { input_schema: Schema // EXACTLY the typed facts the worker is allowed to see — and nothing else output_schema: Schema // EXACTLY the typed artifact the worker must emit (spec | patch | test-report | review-verdict), // validated by a parser before anything downstream reads it allowed_tools: string[] // the specific tools the worker may invoke — how a localizer differs from a fixer: // localizer = read repo + run tests, NO edit; fixer = edit, receives the localizer's typed verdict context_policy: { // the clean, task-scoped window the worker is instantiated with base: "authoritative_state" // built from the single authoritative state plus: ["routed_artifacts_only"] // + only the artifacts the orchestration routes to it — nothing else } authority: "propose" // a proposal the orchestrator arbitrates (a candidate artifact, applied-or-declined) | "edit" // a binding edit to the single repository state (the fixer) | "adjudicate" // falsify, not construct (the judge) | "execute" // run the harness's deterministic suite — no model judgment (the reproducer) | "read_only" // read + run tests, never mutate (the localizer) // EXACTLY these 5 values } ``` A specialist that is not defined by what it sees, what it emits, what it may call, what context it carries, and what its output may do is not a specialist; it is a second Piper, and the thesis says that is precisely what must not be built. ### 4.2 The typed-artifact protocol enum (EXACTLY these fields) Every handoff is a typed artifact. The set is closed — there is no fifth type, and a "chat message" is not an artifact: ```json TypedArtifact { type: "spec" | "patch" | "test-report" | "review-verdict" // EXACTLY these 4 values — the closed set schema: per_type_schema // one of the four below parser: per_type_parser // one of the four below body: <typed payload> // the model's prose is confined to the INSIDE of a field immutable: true // once written, never mutated; a mutated artifact is a spec violation } // the four types, each with schema + parser, named EXACTLY as the paper names them: spec: schema = { requirements: string, acceptance_criteria: string[] } parser = validate(requirements non-empty; every acceptance_criteria addressable by a produced patch) patch: schema = { diff: string, diff_test_basis: string } parser = validate(diff applies cleanly to the single authoritative state) test-report: schema = { suite: "harness_component_17", outcome: "pass" | "fail", failures: string[] } parser = none — deterministic; read from the harness's own runner, never a model's account review-verdict: schema = { verdict: "accept" | "revise" | "reject", evidence: string[] } parser = validate(verdict + cited evidence present; cross-checked against the test-report — the judge cannot outvote a green/red suite) ``` A worker whose output fails to parse does not get a second conversation — it is re-instantiated with the schema violation surfaced as part of its context; if it fails again it is retired and the task takes the single-agent path. ### 4.3 The termination contract (EXACTLY these fields) Every worker, and every loop the orchestration may enter, is instantiated under these four bounds. They are enforced, not advisory: ```json TerminationContract { max_model_calls: int // maximum number of learned invocations; exceeded → worker terminated, partial output discarded max_token_budget: int // maximum token budget; enforced by the orchestration, never advisory max_wall_clock: int // maximum wall-clock seconds; exceeded → worker terminated max_rounds: int // maximum rounds in any multi-step loop; exceeded → handed to the stuck-detection path } ``` A worker that exceeds any bound is terminated; a loop that exceeds `max_rounds` is not "kept trying" — it is re-planned, re-scoped, or aborted (§3.4). --- ## 5. The L Column — the specialist workers (AT MOST ONE entry) The specialist workers are the L column's at-most-one entry, and their entire design is the discipline of the leaf: **short-lived, clean-context, single-shot, typed-output, and with no worker-to-worker channel.** The single learned entry in the entire control plane is one specialist worker making one bounded model call that produces one typed artifact — then terminated. **Short-lived and clean-context.** A worker is instantiated for exactly one bounded judgment, given a context window assembled by the orchestration from the authoritative state plus only the typed artifacts the orchestrator routes to it, and terminated the moment its typed artifact validates. There is no persistent worker identity, no memory carried between instantiations, and no ambient conversation. Clean context is the mechanism that gives a specialist its only real advantage over the single agent: the single agent carries the whole task and the whole transcript, while the specialist carries *exactly* the slice of evidence its subtask needs, so its small budget is spent entirely on the judgment it was spawned to make rather than on reconstructing relevance from a long history. The failure this avoids is the planning agent's information overload, which produces overthinking and erroneous outputs. A clean-context leaf cannot be overloaded, because the orchestration decides what it sees. **Single-shot and typed-output.** A worker makes one bounded model call — or a strictly bounded short sequence, where the termination contract permits — and produces one typed artifact of a schema the orchestration has pre-declared. The artifact is parsed and validated deterministically before it is admitted to the authoritative state; a worker whose output fails to parse does not get a second conversation, it gets re-instantiated with the schema violation surfaced as part of its context, and if it fails again it is retired and the task takes the single-agent path. The single-shot constraint is what keeps a specialist from becoming a chat partner: the model is told "produce this one artifact," not "here is a role, discuss." **No worker-to-worker channel.** Workers do not send messages to one another. There is no bus they can address, no shared memory they can write, and no subscription graph that connects two models. Every inter-role handoff — localizer to fixer, fixer to validator — is mediated by the deterministic orchestrator, which reads one worker's typed artifact, checks alignment, and *routes a copy* to the next worker as part of that worker's clean context. The reason is the invariant itself: the moment two models converse directly, L has entered the control plane, and the harness has recreated the debate loop that the evidence of the paper's §5 shows costs more than it returns. Disagreement is resolved not by workers talking it out but by the next mechanism, which is deterministic. ### 5.1 The evidence-arbitrated judge — the one kind of separation that is always justified There is exactly one role separation this component adopts **unconditionally**, because it is the mechanism that keeps a patch generator from validating its own work — the thesis clause "prevent patch generators from validating their own work" and the single most important separation in the entire program. TraceRepair decomposes repair into three roles: a *probe* that characterizes the bug, a *repair* that proposes a patch, and a *judge* that accepts or rejects the proposal against evidence. The judge is the justified kind of specialist: it has a different *objective* (falsify, not construct), a different *authority* (`adjudicate`, not `edit`), and a different *context* (the patch plus the test evidence, not the generator's search trail). The self-validation trap — a generator grading its own output — is the most reliably harmful error in a single-agent pipeline, and the judge is the deterministic fix: the generator emits a typed `patch`, and the judge emits a typed `review-verdict` that the orchestration arbitrates against the `test-report` produced by the harness's own deterministic test runner. Note what makes this *evidence* arbitration rather than *peer* arbitration: **the judge's verdict is itself checked against the deterministic test report, so even the judge cannot outvote a green or red suite — it can only explain it.** ### 5.2 The conditional spawn policy (deterministic features, not a model judgment) The spawn/retire decision is a **fixed policy over task-shape features**, never a model's judgment of whether a specialist "feels right." The features are deterministic properties of the subtask and the artifact in front of it — the same properties the five-tuple makes explicit: a subtask whose **tool requirement** differs from the producing subtask's (localize-then-fix, where the localizer may read and run tests but not edit, and the fixer may edit but receives the localizer's typed verdict); a subtask whose **objective** differs (falsify, not construct — the judge); a subtask whose **authority** differs (`adjudicate` vs `edit`); and a subtask whose **context** requirement differs (a clean, task-scoped window instead of the whole transcript). If none of these features differs — the common case — no specialist is spawned, and the single well-tooled worker continues; **an empty L column is a permitted, correct state**, and it is the *default*, not the fallback. The one feature that triggers an *unconditional* spawn is the self-validation boundary (§5.1): a patch generator must never validate its own work, so the judge is adopted without ablation. ### 5.3 The empty L column A task that needs no specialist has an empty L column — the single well-tooled worker and the deterministic path handle it. That emptiness is not a gap in the architecture; it is the architecture correctly declining to invent a learned entry where none belongs. **the L column contains at most one entry** — and an empty L column is a permitted, correct state, the default rather than the fallback. --- ## 6. The Anti-Consensus Guarantee (concrete mechanics) The thesis promises that a weak-model harness never lets two models form an uncontrolled consensus loop. This section states *why*, as a guarantee rather than a hope. The guarantee rests on two properties working together — artifact-mediated handoffs and evidence arbitration — and it is the conjunction of the two, not either alone, that prevents the debate loop. **Artifact-mediated handoffs.** Every inter-role handoff is a typed artifact with a schema and a parser, routed by the deterministic orchestrator. The model's prose is confined to the *inside* of an artifact field; the structure, routing, and validation are deterministic. There is no free-running conversation history between two models, no shared channel they can both write, and no subscription graph connecting them — **no worker-to-worker channel exists**. The moment two models converse directly, L has entered the control plane, and the harness has recreated the exact debate loop the paper's evidence shows costs more than it returns (beyond a modest level of tit-for-tat exchange, the exchange itself becomes the cost). **Evidence arbitration.** Disagreement is resolved by deterministic evidence, never by eloquence. The generator emits a typed `patch`; the judge emits a typed `review-verdict`; the orchestration arbitrates the verdict against the `test-report` produced by the harness's own deterministic test runner. The judge cannot outvote a green or red suite — it can only explain it. The check is one-directional and deterministic: a green suite with a rejecting verdict means the verdict's *evidence* is examined, not overruled; a red suite with an accepting verdict means the acceptance is rejected, because the deterministic test report is the arbiter, not the judge's prose. The conjunction is the guarantee. Artifact-mediated handoffs mean no two models can *converse* (there is no channel); evidence arbitration means no two models can *collude into a fluent middle* (the arbiter is the test report, not another model's opinion). Take away either and the consensus loop returns in a different guise: without typed artifacts, the workers chat; without evidence arbitration, the workers vote. The guarantee is *structural* — a property of the orchestration's shape, not a property of any model's restraint. --- ## 7. Implementation Order (build sequence) Nine phases, each independently testable and reversible, landing in `build/21-multi-agent-development/` as standalone importable modules. The order is deliberate — it builds the deterministic orchestration from the state core outward, in the exact sequence the paper's mechanics require: **the orchestration core first, then the artifact protocol, then the role contract, then the spawn/retire policy, then the termination contracts, then stuck-detection, then alignment + escalation, then the judge, then the fault-injection + A/B driver.** Rollback = delete or flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Orchestration core** | `state.py` (§3.1 one repository checkout, one working tree, one task ledger, one hypothesis/plan state, owned only by the orchestration) | Assert exactly one authoritative state exists and only the orchestration mutates it; a worker receives a read-only snapshot or scoped view, never the authority; assert **no LLM consulted**. | Delete `state.py`. | | **2. Typed-artifact protocol + parsers** | `artifacts.py` (§3.2/§4.2 the closed set `spec`/`patch`/`test-report`/`review-verdict`, each with schema + parser, immutable once written) | A malformed `patch` fails its parser and is re-instantiated, not conversed with; a mutated artifact is rejected; assert the set is closed (no fifth type); assert **no LLM consulted**. | Delete `artifacts.py`. | | **3. Role five-tuple contract** | `role.py` (§4.1 the five-tuple: `input_schema`/`output_schema`/`allowed_tools[]`/`context_policy`/`authority`) | A role missing any of the five fields is rejected; `authority` outside the five-value enum is rejected; a persona string is not a role; assert **no LLM consulted**. | Delete `role.py`. | | **4. Spawn/retire policy** | `spawn.py` (§3.3/§5.2 the deterministic task-shape-feature policy; empty-L default; retire-immediately) | A subtask whose tool/objective/authority/context differs spawns exactly one specialist; a subtask with no differing feature spawns none (empty L); a worker is retired the moment its artifact validates; assert **no LLM consulted**. | Delete `spawn.py`. | | **5. Termination contracts** | `terminate.py` (§3.6/§4.3 the four bounds: `max_model_calls`/`max_token_budget`/`max_wall_clock`/`max_rounds`, enforced) | A worker exceeding any bound is terminated and its partial output discarded or handed to stuck-detection; assert the contract is enforced, not advisory; assert **no LLM consulted**. | Delete `terminate.py`. | | **6. Stuck-detection** | `stuck.py` (§3.4 N rounds without an advancing artifact / M recurrences of one failure signature → replan / re-scope / abort) | N idle rounds trigger replan; M recurrences of one signature trigger re-scope or abort; assert the response is deterministic and the ledger, not a model, decides; assert **no LLM consulted**. | Delete `stuck.py`. | | **7. Alignment-check + escalation** | `align.py` (§3.5 alignment checked every handoff; group review only on failure) | A clean diff + matching scope + addressable acceptance criteria pass the check with no escalation; a failed check escalates to a *conditional* review, never an always-on committee; assert **no LLM consulted (the check itself)**. | Delete `align.py`. | | **8. Validator/judge worker** | `judge.py` (§5.1 the one unconditionally-adopted separation; `adjudicate` authority; `review-verdict` cross-checked against `test-report`) | A generator's `patch` cannot self-validate; the judge's verdict is checked against the deterministic `test-report`; a green suite cannot be outvoted by a rejecting verdict; assert the judge cannot outvote a green/red suite. | Delete `judge.py`. | | **9. Fault-injection + A/B driver** | `inject.py` + `ab.py` (§9 baseline + add-roles-one-at-a-time + Δsuccess/Δcost decision rule + failure-mode audit) | `inject.py` forces each headline failure (role theatre, silent error propagation, debate drift, eager spawn, judge capture) and asserts the exact expected trace; `ab.py` runs the role-at-a-time ablation and the Δsuccess/Δcost comparison on held-out fixtures. | `ab.py` read-only; `inject.py` is a test harness, not live code. | The A/B runner (Phase 9) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The orchestration invents no new model, no new execution substrate, and no new diagnosis path; it reuses the harness's own decomposed task (component 07), generation (component 09), test suite (component 17), and validation (component 18), and adds only the deterministic division-of-labor control plane around them. --- ## 8. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of the harness's own stage primitives)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `enum` / `typing` | §4.1 the `authority` enum of five; §4.2 the artifact `type` enum of four; §4.3 the termination-contract fields | CPython stdlib. | **No** | | stdlib `json` | §4.1/§4.2/§4.3 the Role / TypedArtifact / TerminationContract schema serialization | CPython stdlib. | **No** | | stdlib `dataclasses` | §4.1/§4.2 the frozen Role and TypedArtifact (immutable, rebuilt not mutated) | CPython stdlib. | **No** | | stdlib `collections`/`itertools` | §3.4/§3.6 the stuck-detection counters and termination-contract bounds | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic services | In-repo, pinned interpreter. | **No** | | Harness component 07 (decomposition) | §1/§3.1 the decomposed task entering the orchestration | The harness's own stage, already vetted. | **No** | | Harness component 17 (test suite) | §3.2/§5.1 the `test-report` — the deterministic arbiter | The harness's own runner, reused. | **No** | | Harness component 18 (validation) | §1 the accept/reject stage the judge's verdict arbitrates with | The harness's own stage, reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §5 the single L entry — the specialist worker's one bounded call | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all services use stdlib `enum`/`json`/`dataclasses`/`collections`, and the decomposed task, the test suite, the validation stage, and the model call are **reused from the harness's own primitives**, not duplicated. **No new package is installed for Round 1.** The DEFERRED rows (§2 rows 10–11) are additional specialist roles and a hypothetical learned spawn decision — not new third-party Python installs. Any future trained-model download, fine-tuning run, or learned-judgment dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 9. Acceptance Criteria — the A/B Protocol Multi-agent development is the easiest component in the program to fool, because the naive metric — did the task eventually resolve? — is exactly the metric that hides a room of redundant Pipers. A harness that summons five near-identical models to restate the task, restate the restatement, and say "looks good" finally resolves the same task at five times the inference cost and scores the same on resolve rate as the single well-tooled agent. The measurement plan therefore exists to make the *split's worth* visible rather than the *task's outcome*, and it never reports resolve rate alone. **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as specs 01, 11, 12, 13, 14, 15, 16, 17, and 19** (Python-only, single-repo, no-Docker), each additionally annotated with the task-shape features the spawn policy reads (tool requirement, objective, authority, context requirement) so each spawn/no-spawn decision can be scored against ground truth. Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. Held-out tasks feed the role-ablation statistics; in-benchmark tasks feed nothing that scores the orchestration. **Baseline (concrete).** The **single well-tooled agent** — the full harness (decomposition, retrieval, generation, execution, validation, memory) with **no specialist spawns** — plus the **validator** (the evidence-arbitrated judge of §5.1, adopted unconditionally because it encodes "prevent patch generators from validating their own work"). This is the configuration the Scaling study and the Empirical Evaluation both found competitive or superior, and it is the floor every proposed role must clear. Everything is reported as a delta from this baseline. ### 9.1 Add roles one at a time, in earned order — the KEY protocol The start configuration is the single agent plus one validator. Every *further* role — a localizer, a reproducer, a specifier, a planner-specialist — is **added singly**, on top of the current best configuration, and measured against it. This is the anti-"room" discipline in procedural form: never ship a room at once, because a room cannot be ablated. A candidate role is a hypothesis to be falsified, one at a time. ### 9.2 The keep/retire criterion is Δsuccess per Δcost — never success alone A candidate role survives only if its marginal contribution is worthwhile on **both axes simultaneously**: the change in resolve rate it produces, and the cost it charges for that change. A role that raises resolve rate by 2 points while tripling the token budget is a cost with a plausible story, and it is **retired**. A role that raises resolve rate by 0 while *lowering* cost survives on cost alone (a cheaper equivalent is always worth keeping). A role that lowers resolve rate is **retired immediately**, regardless of how natural the org chart felt. ### 9.3 The instrument suite — always reported together No single number, and especially not resolve rate alone, decides anything; the seven together decide. **EXACTLY these seven, named as the paper names them:** 1. **Resolve rate** — the headline, never reported alone. 2. **Tokens** — total and per-artifact, because the token cost is the weak model's scarcest resource. 3. **Wall time** — because a sequential room of roles is a latency budget. 4. **LLM calls** — the number of learned invocations, the direct measure of how often L was entered. 5. **Regressions** — breaks to previously-passing behavior, the component-19 instrument carried forward. 6. **Invalid patches** — artifacts that failed schema validation or failed to apply to the single state, the direct measure of coordination waste. 7. **Context size** — per-worker window size, the direct measure of whether the clean-context promise is being kept. **State explicitly: "Never the end-to-end resolve rate alone."** A role that improves resolve while tripling tokens and wall time is **not adopted**. **State explicitly: "Never introspective fluency or model self-assessment."** A role whose removal changes none of the seven instruments is role theatre, and it is removed regardless of how persuasive its persona is. **These are the KEY measures — a role is KEPT only if its Δsuccess per Δcost is worthwhile on both axes simultaneously, measured on all seven instruments reported together, on held-out tasks.** ### 9.4 Ablation rule (mechanical) The instrument that enforces "never add a role because it sounds sensible" mechanically. One role removed at a time, everything else held fixed, the delta measured on the seven instruments of §9.3: 1. **Ablate the validator/judge** — allow the patch generator to self-validate: the direct test of the one *unconditional* separation. The delta must show regressions and invalid patches rising, and a green/red suite being outvoted by a model's prose — the self-validation trap caught by the harness's own test report. 2. **Ablate a candidate role (localizer / reproducer / specifier / planner-specialist)** — remove it and re-measure: the direct test of whether that role earns its keep. The delta must show which of the seven instruments the role was actually moving. 3. **Ablate the artifact protocol** — allow free conversation between two workers (no typed artifact, no parser): the direct test of the "artifact-mediated" clause. The delta must show token cost and LLM-call count rising without a resolve-rate gain. 4. **Ablate the termination contracts** — allow an unbounded worker/loop: the direct test of the "bounded" clause. The delta must show LLM-call count and wall time rising without bound. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. **Never add a role because it sounds sensible.** ### 9.5 Decision rule (explicit thresholds) Run role-vs-baseline on the *identical* corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). A candidate role is **kept iff**: ``` Δsuccess per Δcost worthwhile on BOTH axes simultaneously (KEY) — resolve-rate delta worthwhile, AND cost delta (tokens + wall time) not disproportionate — a role raising resolve 0 while lowering cost survives on cost alone; a role lowering resolve is retired immediately AND the seven instruments reported TOGETHER, never resolve rate alone (KEY) AND the role's removal degrades ≥1 of the seven instruments beyond noise (it is not role theatre) (ablation) AND fault injection all expected traces observed exactly once (companion) ``` The KEY measures must **strictly beat the single-agent-plus-validator baseline on Δsuccess per Δcost**; fault injection must pass with the exact expected traces. **Never the end-to-end resolve rate alone.** **Never introspective fluency or model self-assessment.** If a role wins it is adopted; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 9.6 Failure-mode audit (mechanical, the ways a multi-agent harness can lie) 1. **Role theatre** — a role whose removal changes none of the seven instruments: a persona wearing a five-tuple costume, and the ablation is how the costume is removed. 2. **Silent error propagation** — a handoff whose alignment check passed but whose downstream failure was never traced back: the "architectures without centralized verification propagate errors more" finding caught in the harness's own ledger. 3. **Debate drift** — two workers effectively conversing through the orchestrator's routing: the invariant violated in effect if not in letter, caught by the LLM-call count and the token profile. 4. **Eager spawn** — the orchestration instantiating specialists on tasks whose features did not meet the spawn policy's threshold: the "conditional" clause eroded by over-eagerness. 5. **Judge capture** — the validator's verdict drifting to mirror the generator's: caught by comparing the judge's `review-verdict` against the deterministic `test-report`, which is the one arbitration the judge cannot outvote. ### 9.7 Two honest caveats **First, the weak-model caveat:** the pro-case numbers of the paper's §5 — MASAI's 28.33% at $1.96, ChatDev's $0.2967, HyperAgent's role ablations — were measured on frontier models and large repositories a weak-model harness does not control; the weak model should expect materially smaller absolute effects while leaning *harder* on the deterministic orchestration and the typed-artifact protocol, which cost nothing per model scale. **Second, the held-out caveat:** all multi-agent metrics are reported on held-out and out-of-benchmark tasks, exactly as the earlier components require, so a role that has memorized a benchmark's answers cannot pass a resolve-rate ablation without its usefulness having been measured on problems it never saw. --- ## 10. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/21-multi-agent-development/`; it never writes into the live harness's source. - A **feature flag** (env var `MULTIAGENT=21|single`, default `single`) routes the A/B; the flag alone switches between the deterministic multi-agent orchestration and the single-agent baseline — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running role-vs-baseline is always possible in the same session, repo, model. - The spawn/retire ledger (every spawn decision, every typed artifact's parse outcome, every judge verdict vs test-report) is append-only and versioned; rolling the build back never loses it. **Invariants (must hold, checked at the end of every phase).** 1. **the L column contains at most one entry** — the specialist worker (§5), produced only after the deterministic control plane has decided to spawn it, constructed its five-tuple, and assembled its clean context; the model never decides *that* a specialist should spawn, *which* role it should be, *what* authority it should carry, or *whether* its output parses. **an empty L column is a permitted, correct state** — a no-specialist task is handled by the single well-tooled worker and the deterministic path, and the emptiness is the design working, never a violation. 2. **No worker-to-worker channel** — every inter-role handoff is mediated by the deterministic orchestrator through a typed artifact; two models never converse directly, and there is no bus, shared memory, or subscription graph connecting them. 3. **One authoritative state** — exactly one repository checkout, one working tree, one task ledger, and one hypothesis/plan state, owned and mutated only by the orchestration; a worker receives a read-only snapshot or scoped view, never the authority. 4. **Every handoff is a typed artifact with a parser** — a worker whose output fails to parse is re-instantiated with the schema violation surfaced, then retired on second failure; no artifact is admitted unvalidated. 5. **The judge cannot outvote a green/red suite** — the judge's `review-verdict` is arbitrated against the deterministic `test-report`, and the test report is the arbiter; the judge can only explain it. 6. **The spawn/retire decision is deterministic** — a fixed policy over task-shape features, never a model's judgment; the validator/judge is the one role adopted unconditionally, every other role is gated by the §9 Δsuccess/Δcost ablation. 7. **Every worker and loop runs under a termination contract** — `max_model_calls` / `max_token_budget` / `max_wall_clock` / `max_rounds`, enforced, never advisory. 8. **All metrics on held-out/out-of-benchmark tasks** — the seven instruments and every secondary are reported on held-out tasks, never tuning or in-benchmark data. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*