Summary / 22. Independent reviewer (critic) / Document
Spec 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 22 — Independent Reviewer / Critic (fresh-context adversarial hypothesis generation) > Derived from `papers/22-independent-reviewer-critic.md` — "Fresh-Context Adversarial Hypothesis Generation Under a Deterministic Acceptance Policy." > This document drives the build of the **independent reviewer** as a *fresh-context, adversarial hypothesis-generation stage*, > not a second model voting on whether a patch is correct. The paper's central claim, made structural here: the entire > orchestration — review-packet construction, blinding and stripping of the generator's rationale and confidence, atomic-edit > decomposition, claim-schema enforcement, falsification-procedure routing, and the final acceptance decision — is deterministic > software running on CPU and holds **zero** learned entries, while the critic itself is the single learned entry: one bounded > model call (or a short bounded sequence of calls) producing typed, falsifiable `ReviewFinding`s that *recommend* but never > *certify*. The harness, by component 21, already produces a candidate patch through generation, editing, execution, tracing, > and iterative repair, and gates it through a deterministic validation policy that owns final acceptance. This component answers > the one remaining question — *is there a role for a second model whose only job is to attack the patch before acceptance?* — and > the answer is "yes, but not the role a naive reading assigns it." The reviewer is **not there to tell Piper whether the patch > is good**; it is there to **manufacture the strongest falsifiable case that the patch is wrong**, so that whatever survives the > case is genuinely hard to break. The epistemic signature is deliberately inverted: every earlier component *increases* the > probability that a patch is correct; this one exists to *decrease* it — but only through falsifiable, evidence-backed claims > that the deterministic layer can then check. A reviewer that returns "this patch is fine" has contributed almost nothing > measurable; a reviewer that returns one verified defect has invalidated the patch. The word "independent" does **not** mean "a > second prompt to the same model": it means *different evidence access* and, at high stakes, *different weights* — with the > generator's self-assessment removed from the critic's context, so that Panickssery's self-preference and Sharma's sycophancy > are made structurally impossible to trigger rather than merely discouraged. The invariant that governs the whole design is > **the L column contains at most one entry** — everything that moves a packet, strips a rationale, decomposes an edit, enforces > a claim schema, routes a falsification procedure, or renders a final verdict is [D]; only the critic's generation of hypotheses > is [L]. And for this component specifically, **an empty L column is a permitted, correct state**: the critic is an *optional* > instrument engaged only when the marginal value of an adversarial pass exceeds its cost, so when the deterministic validator > alone suffices (small blast radius, fully covered by existing gates), no critic is spawned at all. This spec pins the paper's > design to exact primitives: the exact six-field `ReviewFinding` schema, the exact five-element review packet, the exact > four-field blinding contract, the exact D/L invariant, the exact asymmetry (weak-positive agreement / strong-negative verified > criticism), the exact honest measurement protocol (verified-defect recall vs over-veto false-positive rate, never "did it sound > good"), the exact A/B critic-on-vs-critic-off driver, the exact fix-guided counterfactual verification standard, the exact > train-on-repository-scar-tissue rule, and an exact build sequence. --- ## 1. Purpose & Scope Build a **deterministic review orchestration plus an optional adversarial critic** — a control plane that, given a candidate patch already produced and gated by the harness's earlier components, decides *whether* to engage a critic, constructs the review packet the critic sees, strips everything that would corrupt independence, and routes every criticism the critic emits through a deterministic verification gate that — and this is the load-bearing part — is the only thing whose outcome decides rejection, repair, or acceptance. The critic never votes; it *proposes falsifiable hypotheses*. The orchestration never trusts the critic's self-report; it *executes the falsification* and reads the outcome. The scope is the review layer only. Component 09 owns *how* a patch is produced; component 18 owns *whether* a patch is accepted (the deterministic validation policy that reserves final acceptance authority). This component owns the *attack*: the decision to engage a critic, the evidence that critic is handed, the schema its output must fill, the routing of each criticism to the specific procedure that would verify or refute it, and the assembly of the verification outcome into a signal the deterministic validator can consume. Everything upstream and downstream is a deterministic stage that already knows what to do next; the only question this component answers is whether a *fresh-context adversarial pass* adds verified defect-recall that the deterministic validator alone would miss — and the field's measured record (Huang's flatlined intrinsic self-correction, Panickssery's self-preference, Jin and Chen's 26.2%→73.2% false-negative escalation under over-correction, Knight and Leveson's correlated faults) says the answer is "only when you can measure it, and only through verified criticism, never through a second opinion." The paper's discipline compresses into a handful of load-bearing moves that carry the entire design. **Fresh context** means the critic is handed the specification, the diagnosed root cause, the preservation obligations, the impact evidence, and the candidate patch — and is *stripped of* the generator's rationale, its self-confidence, any peer votes, and all provenance that would let it detect that the patch came from a model sharing its own weights. **Adversarial hypothesis generation** means the critic's output is not APPROVE/REJECT but a set of typed, falsifiable findings, each naming a concrete defect, its supporting evidence, and — the Popperian core — the exact procedure that would prove the claim *wrong*. **Deterministic acceptance** means the critic's `confidence` is recorded but never trusted, its veto power is exactly zero, and the final acceptance authority belongs to the deterministic validation policy rather than to either LLM. Three principles decide every design choice. **P1 The critic RECOMMENDS; execution CERTIFIES** — the critic cannot accept a patch (it has no accept action, only "no blocking findings") and it cannot reject a patch (a blocking finding merely *forces* a deterministic verification step, whose outcome determines the disposition). **P2 Every blocking criticism carries a falsification procedure** — a criticism that cannot state how it would be shown wrong is not a scientific claim, it is a vibe, and it is downgraded to advisory or dropped, never blocking. **P3 The D/L invariant** — packet construction, blinding and stripping, edit-level decomposition, claim-schema enforcement, falsification-procedure routing, and the final acceptance authority are all deterministic software on CPU; the L column holds at most the critic itself, and **an empty L column is a permitted, correct state** — the critic is an optional instrument, and its absence is the design working as intended, not a missing entry to be filled. The build lives in `build/22-independent-reviewer-critic/`, writes nothing into the live harness, and is kept **only if** the critic strictly dominates its own absence — measured on the honest instruments of §9, above all **verified-defect recall against over-veto false-positive rate** (never "did it sound good"), A/B-tested against the deterministic validator alone, and settled by **fix-guided counterfactual execution** of each blocking finding's falsification procedure. --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. **BUILD-NOW = the Round 1 build** (rows 1–7); **DEFERRED = later rounds** (rows 8–9), **not excluded** but gated by the same §9 A/B protocol. Every later round of this component still builds within the same `build/22-independent-reviewer-critic/` tree. | # | Component (paper §3/§4/§6) | Classification | |---|---|---| | 1 | Review-packet constructor — the five evidentiary elements (specification, diagnosed root cause, preservation obligations, impact evidence, candidate patch) (§3.1) | **BUILD-NOW** | | 2 | Blinding and stripping — remove generator rationale, self-confidence, peer votes, provenance (§3.2) | **BUILD-NOW** | | 3 | Diff-parser — atomic-edit decomposition (each hunk an addressable unit) + whole-patch view (§3.1) | **BUILD-NOW** | | 4 | `ReviewFinding` six-field schema + parser — deterministic validation; empty/un-executable `falsification` downgraded or dropped, never blocking (§4.1) | **BUILD-NOW** | | 5 | Critic worker — the single bounded learned call producing typed falsifiable findings (§5) | **BUILD-NOW** | | 6 | Falsification-router — map each finding to test / static analysis / trace replay / counterfactual execution (§3.5) | **BUILD-NOW** | | 7 | Deterministic verification gate + A/B driver — run each falsification; critic-on vs critic-off; recall vs over-veto (§6, §9) | **BUILD-NOW** | | 8 | **Different-weights critic** — a distinct model, temperature, or decoding at high stakes (§1 "different weights at high stakes") | **DEFERRED** — Round 1 runs the critic on the harness's own weak model; a *different* model is adopted only if it strictly dominates on recall at no worse over-veto, per §9 | | 9 | **Learned critic tuning / fine-tuned critic on repository scar tissue** — CriticGPT-style training on past regressions and reverted commits (§6 "train on repository scar tissue") | **DEFERRED** — the training corpus rule is specified now, but the actual fine-tune run must clear the 72h-age + star/scrutiny + sandbox-test gate and the §9 strict-domination test before any weights are adopted | **The cost rule (explicit).** The L column holds at most one entry — the critic — and nothing else. In Round 1 that entry is produced by the harness's own weak model (`deepseek-v4-pro`) on exactly the same terms as the happy path; it is **not** a new learned node and it is **not** trained. The model is handed the blinded review packet and asked to emit a set of typed `ReviewFinding`s; its cost is bounded to one model call per critic pass (or a strictly bounded short sequence, each schema-validated), and it never decides *whether* a critic should spawn, *what* the schema is, *what* its own `confidence` means, or *whether* its findings are true — those are all deterministic. The DEFERRED rows (a different-weights critic, and a fine-tuned critic) must clear the same bar as in every other component: strictly dominate the deterministic baseline on verified-defect recall at no worse over-veto false-positive rate, on held-out tasks, at a defensible cost, or be rejected. **Design decision: DETERMINISTIC-FIRST, SINGLE-L.** Round 1 puts no model anywhere in the control plane — not in packet construction, not in blinding, not in edit decomposition, not in schema enforcement, not in falsification routing, not in the final acceptance decision. The full deterministic pipeline (rows 1–4, 6–7) is the build; the single L entry is the critic, produced only *after* the deterministic control plane has decided to engage it, constructed its blinded packet, and declared the schema its output must fill. --- ## 3. The Deterministic Orchestration (D) The orchestration is the control plane, and it is where the paper's "deterministic acceptance policy" becomes software. Every function in this section is deterministic software on CPU; the L column contributes nothing here, and the invariant is stated at the outset of the architecture and re-stated at its close: **the L column contains at most one entry**, and the orchestration holds zero learned entries. Concretely, the control plane's pipeline is: ``` CANDIDATE PATCH ENTERS (already generated and gated by components 09 → 18) → decide_engage [D] → blast-radius test: is a critic worth its cost? small radius + full gate coverage = empty L (§3.7) → build_packet [D] → assemble the five evidentiary elements: specification, diagnosed root cause, preservation obligations, impact evidence, candidate patch (§3.1) → blind_strip [D] → delete generator rationale, self-confidence, peer votes, provenance (§3.2) → decompose_edits [D] → diff-parser splits the patch into addressable atomic edits + whole-patch view (§3.1) → enforce_schema [D] → validate every ReviewFinding; empty/un-executable falsification → downgrade or drop (§4.1) → route_falsification [D]→ map each finding to test / static analysis / trace replay / counterfactual execution (§3.5) → run_verification [D] → execute each falsification procedure; the OUTCOME, not the critic, decides (§3.6) → render_acceptance [D] → hand the verification outcome to the deterministic validation policy (component 18) → (L) critic → the at-most-one learned entry: one bounded call → typed falsifiable ReviewFindings (§5) ``` The model never decides whether to engage a critic (the blast-radius test does), never decides what the critic sees (the packet and blinding stages do), never decides what shape its output must take (the schema does), never decides how its findings are verified (the router does), and never decides whether a finding is true (the verification gate does). All artifacts land in `build/22-independent-reviewer-critic/`. ### 3.1 Review-packet construction [D] **Input:** the candidate patch, the component-07 decomposed task, the component-06 code-graph impact data, and the harness's diagnostic record. **Output:** the review packet — exactly five evidentiary elements, nothing else. **No language model runs here.** The five elements, assembled by deterministic packet construction: 1. **The specification.** The task the patch was supposed to accomplish — the issue text, the target behavior, the acceptance contract — expressed independently of the generator's interpretation of it. The critic sees the requirement, not the generator's paraphrase of the requirement. 2. **The diagnosed root cause.** The *diagnostic* conclusion of the failure that motivated this patch: what the harness established was wrong (the failing test, the trace, the fault-localization ranking, the reproduced symptom). This is an evidentiary artifact, not an argument. 3. **The preservation obligations.** The explicit list of behaviors that must not change — the regression surface, the pinned contracts, the invariants the deterministic validator will re-check. The critic is asked to attack the patch *against these obligations*, so its criticism is testable rather than aesthetic. 4. **The impact evidence.** The computed blast radius and dependency-impact data (from the component-06 lineage): which callers, modules, and invariants the diff touches, derived deterministically from the code graph, not asserted by the generator. 5. **The candidate patch.** The diff itself, decomposed at the atomic-edit level (each hunk as a separate addressable unit) plus the whole-patch view. The decomposition is mechanical — a diff-parser splits the patch into atomic edits; no model is involved. Note the precise distinction the paper enforces: the generator's *diagnosis of the prior failure* is evidence and is included (element 2); the generator's *rationale for why the current patch is correct* is argument and is stripped (§3.2). The packet gives the critic everything it needs to *attack* the patch and nothing that would *recruit* it into agreement. ### 3.2 Blinding and stripping [D] **Input:** the assembled packet plus the generator's full artifacts. **Output:** the same packet with four things removed, so that no trace of the generator's self-assessment or identity survives. **No language model runs here.** Deterministic blinding removes, before the packet reaches the critic: - **The generator's rationale.** Any narrative "here is why this works" is deleted. The critic must reconstruct the patch's intent from the specification and the diff alone. - **The generator's self-confidence.** No "I am 0.9 sure." The critic has no number to anchor to. - **Peer votes.** No prior reviewer's agreement or disagreement is ever shown; each review is a fresh context. If multiple critic passes are run, none sees the others' findings. - **Provenance.** Nothing that would let the critic detect that the patch came from a model sharing its own weights, or from itself. The blinding is *the* implementation of the "independent means different evidence access" honesty: a fresh context is not a nicety, it is the mechanism by which Panickssery's self-preference and Sharma's sycophancy are made structurally impossible to trigger. A "reviewer" that shares the generator's weights, context, rationale, and confidence is not a reviewer; it is an echo — and the blind strip is what guarantees it cannot become one. ### 3.3 Edit-level decomposition (the diff-parser) [D] **Input:** the candidate patch. **Output:** the same diff split into addressable atomic edits, each with a stable id, plus the whole-patch view. **No language model runs here.** The decomposition is mechanical: a diff-parser splits the patch into atomic edits (each hunk a separate addressable unit), assigns each an id, and emits both the edit-level and the whole-patch view. This is what "require review at atomic-edit and whole-patch levels" means operationally: the packet is split so a defect can be pinned to a single hunk, and the whole patch is also reviewed as a unit so *emergent* defects — two hunks each locally fine but jointly wrong — are not missed. The atomic-edit id is the `location` field's vocabulary: a finding must name an atomic-edit id or the literal `"whole-patch"`. A criticism that cannot be located cannot be verified. ### 3.4 Claim-schema enforcement [D] **Input:** the critic's raw output. **Output:** a set of typed, schema-valid `ReviewFinding`s — or a rejection. **No language model runs here.** The critic's raw output is parsed and validated against the six-field schema of §4.1; malformed or missing fields are rejected and re-requested (a bounded retry), and a finding whose `falsification` field is empty or un-executable is *downgraded to advisory or dropped*, never blocking. The critic does not get to define the rules of its own acceptance; the deterministic parser does. ### 3.5 Falsification-procedure routing [D] **Input:** a schema-valid `ReviewFinding`. **Output:** a concrete, executable verification procedure — one of test, static analysis, trace replay, or counterfactual execution — bound to the finding's `location` and `claim`. **No language model runs here.** Each criticism is matched to *how* it will be verified: a claim about a removed null check routes to the contract's test; a claim about a type unsoundness routes to the static analyzer; a claim about a concurrency race routes to a trace replay; a claim about a behavior change routes to counterfactual execution (the edit reverted, or the critic's proposed repair applied). The router is deterministic and exhaustive: every blocking finding must land on exactly one procedure, and a finding that maps to nothing executable was already downgraded or dropped in §3.4. ### 3.6 Final acceptance [D] **Input:** the verification outcome for every finding. **Output:** a signal to the deterministic validation policy (component 18): no verified blocking defects → the patch proceeds to the validator's own gate; ≥1 verified blocking defect → the patch is rejected or repaired, *by the validator's authority, not the critic's*. **No language model runs here.** The critic's power is zero here: a blocking finding merely *forced* a verification step, and the step's outcome — not the critic's — determines the disposition. This is the structural answer to over-veto stall: the critic cannot filibuster, because it never holds a veto; it can only route evidence. ### 3.7 The D/L invariant, stated mechanically (D) **the L column contains at most one entry.** In this component the D column is the entire orchestration of §3.1–§3.6 — packet construction, blinding and stripping, edit-level decomposition, claim-schema enforcement, falsification-procedure routing, and the final acceptance authority — all deterministic software on CPU. The control plane holds zero learned entries. The L column holds exactly one entry: the critic, a bounded call producing typed falsifiable findings. The consequence that must be stated explicitly: **an empty L column is a permitted, correct state** — when the deterministic validator alone suffices to accept or reject the patch (small blast radius, fully covered by existing gates, nothing a critic could add that is not already checked), no critic is spawned at all. The critic is an *optional* instrument engaged only when the marginal value of an adversarial pass exceeds its cost. The control plane runs whether or not the critic exists; the critic never runs without the control plane. The critic **RECOMMENDS**; execution **CERTIFIES**. --- ## 4. The Core Schemas (the build's JSON contracts) The following schemas are the load-bearing contracts of the build. Their shapes are EXACTLY these fields, no more and no fewer, and the label "EXACTLY these fields" is itself part of the contract — a field added for convenience is a spec violation. ### 4.1 The ReviewFinding record (EXACTLY these fields) The critic never outputs a verdict. It outputs a set of typed, falsifiable findings, each a record with exactly six fields — the paper's exact six, named exactly as the paper names them. The `falsification` field is the Popperian core: a criticism that cannot state how it would be shown wrong is not a scientific claim, it is a vibe, and it is downgraded to advisory or dropped, never blocking. ```json ReviewFinding { location: string // atomic-edit id or "whole-patch" — mandatory and granular; a criticism that cannot be located cannot be verified claim: string // the concrete defect asserted, in one sentence severity: enum // {blocking, advisory} — EXACTLY these 2 values evidence: string // what in the packet supports the claim (spec clause, diff line, trace fragment) falsification: string // the procedure that would prove the claim WRONG (a test, a static check, a replayed trace, a counterfactual execution) confidence: float // 0..1, the critic's self-assessed probability the claim survives verification — recorded, never trusted } ``` Four design decisions are load-bearing and are enforced, not advisory: - **Location is mandatory and granular.** The critic must name an atomic edit or `"whole-patch"`. A finding without a valid `location` fails the parser. - **Every blocking criticism carries a falsification procedure.** A blocking finding whose `falsification` is empty or un-executable is downgraded to advisory or dropped, never blocking. The deterministic layer then executes exactly that refutation (§3.5–§3.6). - **The schema is enforced deterministically.** Malformed or missing fields are rejected and re-requested (a bounded retry). The critic does not get to define the rules of its own acceptance. - **Confidence is recorded but never trusted.** A 0.99-confidence finding and a 0.51-confidence finding face identical verification. The value is captured only as cheap metadata for calibration tracking. ### 4.2 The review packet (the five evidentiary elements) The review packet is the critic's entire evidence base — exactly these five elements, assembled by deterministic packet construction (§3.1) and carrying nothing else: ```json ReviewPacket { specification: string // the task the patch was to accomplish — the requirement, not the generator's paraphrase diagnosed_root_cause: string // the diagnostic conclusion of the motivating failure — evidence, not argument preservation_obligations: string[] // behaviors that must not change: regression surface, pinned contracts, invariants impact_evidence: object // computed blast radius + dependency impact from component 06 (callers/modules/invariants touched) candidate_patch: { atomic_edits: { edit_id: string, hunk: string }[] // each hunk as a separate addressable unit whole_patch: string // the full diff, also reviewed as a unit } } ``` The candidate patch carries both granularities deliberately: the atomic-edit level so a defect can be pinned to a single hunk, and the whole-patch view so *emergent* defects (two hunks each locally fine but jointly wrong) are not missed. ### 4.3 The blinding contract (the four stripped fields) Deterministic blinding removes exactly these four things before the packet reaches the critic (§3.2). The contract is a *deny list applied by construction* — the packet constructor never writes them into the packet, and the blind-strip stage deletes any that leak in: ```json BlindContract { strip: [ "generator_rationale", // any narrative "here is why this works" — the critic reconstructs intent from spec + diff alone "self_confidence", // no "I am 0.9 sure" — the critic has no number to anchor to "peer_votes", // no prior reviewer's agreement/disagreement — each review is a fresh context "provenance" // nothing that lets the critic detect shared weights or self-origin ] } ``` --- ## 5. The L Column — the critic (AT MOST ONE entry) The critic is the single learned entry in the entire component. This section specifies exactly what it is permitted to be, what it outputs, and what it is forbidden to do. **The critic is a hypothesis-generator, not a judge.** It is handed the blinded review packet (§4.2) and asked to emit a set of typed falsifiable `ReviewFinding`s (§4.1). It is optimized for **recall** — surfacing *candidate* defects, including speculative ones — while **precision** is provided by the deterministic verification stage that follows. This division of labor is what lets the critic be genuinely adversarial without being dangerous: it is encouraged to over-flag, because every flag is routed through a verification gate that discards false positives cheaply and deterministically. **RECOMMENDS, never CERTIFIES.** The critic has no accept action — only "no blocking findings" — and it cannot reject a patch either: a blocking finding merely *forces* a deterministic verification step, whose outcome determines whether the patch is rejected, repaired, or accepted. The critic's veto power is zero; its power is only to *route evidence*. This is the structural answer to over-veto stall: the critic cannot become a permanent filibuster, because it is never given a vote to cast. **One bounded model call.** The critic is one bounded model call (or a short bounded sequence of calls, each schema-validated), producing typed falsifiable `ReviewFinding`s. It never outvotes deterministic evidence, because it is never given a vote to cast. Recapitulating the invariant at its tightest form: **the L column contains at most one entry** — and that single entry is exactly this bounded critic call, nothing more. ### 5.1 Claim-granular output The critic's output is constrained to the schema of §4.1, and the deterministic parser enforces it. The critic does not emit a verdict, a score, or prose; it emits records, each one a falsifiable hypothesis. The four load-bearing decisions of §4.1 — mandatory granular `location`, falsification-procedure-on-every-blocking-claim, deterministic schema enforcement, and confidence-recorded-but-never-trusted — are the critic's entire contract with the control plane. ### 5.2 The empty L column A patch that needs no critic has an empty L column — the deterministic validator alone handles it. That emptiness is not a gap in the architecture; it is the architecture correctly declining to spawn a learned entry where none earns its cost. **the L column contains at most one entry** — and **an empty L column is a permitted, correct state**: the critic is optional, the deterministic validator alone suffices when blast radius is small, and the critic is kept only for as long as it is measurably better than its own absence (§9). --- ## 6. The Asymmetry (weak positive / strong negative) The component's epistemic asymmetry, stated once and enforced everywhere — the exact inverse of the naive design's error: - **Reviewer agreement (no blocking findings) is weak positive evidence.** Given rubber-stamping, self-preference, correlation, and sycophancy, a silent critic is only barely informative. It does not certify the patch; it merely fails to add a known defect. - **A verified criticism is strong negative evidence.** When a blocking finding's falsification procedure is *run* and the claim *survives* — the test fails, the static check trips, the trace confirms, the counterfactual execution reproduces the defect — the patch is, for acceptance purposes, broken. **One verified defect invalidates the patch** regardless of how many other findings failed verification. The naive design over-weights approval and under-verifies rejection; this component under-weights approval and over-verifies rejection. The reviewer is not there to tell Piper whether the patch is good; it is there to manufacture the strongest falsifiable case that the patch is wrong — and the deterministic layer, not the critic, decides whether that case holds. --- ## 7. Implementation Order (build sequence) Seven phases, each independently testable and reversible, landing in `build/22-independent-reviewer-critic/` as standalone importable modules. The order is deliberate — it builds the deterministic orchestration from the packet outward, in the exact sequence the paper's mechanics require: **the review-packet constructor + blinding first, then the diff-parser (atomic-edit decomposition), then the ReviewFinding schema + parser, then the critic worker (bounded call), then the falsification-router, then the deterministic verification gate, then the A/B driver.** Rollback = delete or flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Review-packet constructor + blinding** | `packet.py` (§3.1/§3.2/§4.2/§4.3 the five evidentiary elements + the four stripped fields) | Assemble a packet from a fixture patch and assert exactly the five elements are present; assert the four stripped fields (rationale, self-confidence, peer votes, provenance) are absent even when present in the input; assert **no LLM consulted**. | Delete `packet.py`. | | **2. Diff-parser (atomic-edit decomposition)** | `diffparse.py` (§3.3 split the patch into addressable atomic edits with stable `edit_id`s + the whole-patch view) | A two-hunk patch decomposes into two atomic edits each with a distinct id, plus a whole-patch view; assert the `location` vocabulary is `edit_id \| "whole-patch"`; assert **no LLM consulted**. | Delete `diffparse.py`. | | **3. ReviewFinding schema + parser** | `finding.py` (§4.1 the six-field schema, deterministic validation, downgrade-or-drop on empty/un-executable `falsification`) | A finding missing `location` or `claim` is rejected; a blocking finding with empty `falsification` is downgraded to advisory or dropped, never blocking; `severity` outside `{blocking, advisory}` is rejected; assert **no LLM consulted**. | Delete `finding.py`. | | **4. Critic worker (bounded call)** | `critic.py` (§5 the single L entry, one bounded model call producing typed `ReviewFinding`s) | The worker makes exactly one bounded model call and emits schema-valid findings; it never emits a verdict; a malformed output is re-requested under a bounded retry, then retired; assert the worker has no accept/reject action. | Delete `critic.py`. | | **5. Falsification-router** | `falsify.py` (§3.5 map each finding to test / static / trace / counterfactual, bound to `location` + `claim`) | A removed-null-check finding routes to the contract test; a type-unsoundness finding routes to the static analyzer; a race finding routes to trace replay; a behavior-change finding routes to counterfactual execution; assert every blocking finding maps to exactly one procedure; assert **no LLM consulted**. | Delete `falsify.py`. | | **6. Deterministic verification gate** | `verify.py` (§3.6 run each falsification procedure; the OUTCOME, not the critic, decides) | A finding whose falsification passes (claim survives) marks the patch broken; a finding whose falsification fails (claim killed) is discarded; one verified defect invalidates the patch regardless of other findings; assert the critic's `confidence` never influences the outcome; assert **no LLM consulted**. | Delete `verify.py`. | | **7. A/B driver** | `ab.py` (§9 critic-on vs critic-off, verified-defect recall vs over-veto false-positive rate, fix-guided counterfactual verification) | Run the identical patch population twice (validator alone vs critic-then-validator); report recall and over-veto together, never separately; assert the critic is kept only if it strictly dominates its absence. | `ab.py` read-only; never modifies live harness. | The A/B runner (Phase 7) is the only phase that ever *invokes* the current harness — only to measure it, never to modify it. The orchestration invents no new model, no new execution substrate, and no new diagnosis path; it reuses the harness's own decomposed task (component 07), generation (component 09), impact data (component 06), test suite (component 17), and validation (component 18), and adds only the deterministic review control plane and the optional critic around them. --- ## 8. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo + reuse of the harness's own stage primitives)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `enum` / `typing` | §4.1 the `severity` enum of two; §4.2/§4.3 the packet and blind-contract fields | CPython stdlib. | **No** | | stdlib `json` | §4.1/§4.2/§4.3 the ReviewFinding / ReviewPacket / BlindContract schema serialization | CPython stdlib. | **No** | | stdlib `dataclasses` | §4.1/§4.2 the frozen ReviewFinding and ReviewPacket (immutable, rebuilt not mutated) | CPython stdlib. | **No** | | stdlib `difflib` / `unidiff` (or in-repo diff parser) | §3.3 the atomic-edit decomposition | CPython stdlib or in-repo, already vetted. | **No** | | Python toolchain (`python3`, pinned) | §3 all deterministic services | In-repo, pinned interpreter. | **No** | | Harness component 06 (impact data) | §3.1/§4.2 the `impact_evidence` element (blast radius + dependency impact) | The harness's own stage, already vetted. | **No** | | Harness component 07 (decomposition) | §3.1/§4.2 the `specification` element | The harness's own stage, already vetted. | **No** | | Harness component 17 (test suite) | §3.5/§3.6 the test-based falsification procedures | The harness's own runner, reused. | **No** | | Harness component 18 (validation) | §3.6 the final acceptance authority the verification outcome feeds | The harness's own stage, reused. | **No** | | Harness weak model (`deepseek-v4-pro`) | §5 the single L entry — the critic's one bounded call | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: all services use stdlib `enum`/`json`/`dataclasses`/`difflib`, and the specification, impact data, test suite, validation stage, and the model call are **reused from the harness's own primitives**, not duplicated. **No new package is installed for Round 1.** The DEFERRED rows (§2 rows 8–9) are a different-weights critic and a fine-tuned critic — not new third-party Python installs. Any future trained-model download, fine-tuning run, or learned-judgment dependency must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — out of Round 1 scope entirely. --- ## 9. Acceptance Criteria — honest measurement, A/B, and ablation A component whose whole premise is "don't trust the critic's self-report" must hold *itself* to the same standard. This section specifies how the critic's value — or lack of it — is measured, so that the component earns its place in the harness by evidence rather than by assertion. ### 9.1 The two numbers: verified-defect recall vs over-veto false-positive rate **The metric is verified-defect recall against over-veto false-positive rate — not "did the critic sound good."** A critic is evaluated by exactly two numbers, both derived from the deterministic verification gate, never from the critic's own confidence: 1. **Verified-defect recall** — of the defects that actually existed in a held-out patch set, how many did the critic surface a falsifiable claim for, where the claim then *survived* verification. 2. **Over-veto false-positive rate** — of the blocking findings the critic emitted, how many were *killed* by their own falsification procedure. A critic that never blocks scores zero recall; a critic that blocks everything scores a false-positive rate near one. Both are failures. The sound-good metric — "the critique was well-written and thorough" — is explicitly *not* scored, because Zheng's verbosity bias and Sharma's sycophancy both live exactly on that channel. ### 9.2 A/B critic-on vs critic-off **The critic's marginal value is measured by running the identical patch population twice: once through the deterministic validator alone, once with the critic inserted before it.** The critic earns its compute only if the critic-on arm shows (a) more verified defects caught, or (b) the same defects caught with cheaper total verification (the critic routing verification effort toward the high-value checks). If critic-on is statistically indistinguishable from critic-off, the invariant's escape hatch fires: **an empty L column is a permitted, correct state**, and the critic is removed until the evidence changes. ### 9.3 Fix-guided counterfactual verification (the standard of proof) Following Jin and Chen's Fix-guided Verification Filter and CRITIC's tool-grounded critique, every blocking finding is settled by *executing* its falsification procedure, not by re-asking the critic. A finding claiming "this atomic edit removes a null check that the contract requires" is settled by running the contract's test on the edit and on the *counterfactual* (the edit reverted, or the critic's proposed repair applied); the outcome, not the critic, decides. This makes the critic's role in measurement exactly its role in operation: it *proposes falsifiable hypotheses*, and the deterministic layer *executes the falsification*. ### 9.4 Train on repository scar tissue CriticGPT's construction — training a critic on *known* inserted and historical bugs, with human preference over the *quality of criticism* — is the template for tuning this component's critic. The training corpus is the repository's own scar tissue: past regressions, past reverted commits, past defects that the deterministic gates caught late, each labeled with the evidence that *would have* caught it. A critic trained on this specific failure distribution is measurably better at surfacing *this repository's* correlated blind spots — which is, by Knight and Leveson, exactly the failure class a generic critic is worst at. The critic is therefore not a fixed off-the-shelf model but a *continually re-tuned* one, and each re-tuning is itself gated: a new critic is adopted only if it strictly dominates the incumbent on verified-defect recall at no worse false-positive rate. This is the DEFERRED row 9, specified here and gated by §9.6. ### 9.5 Ablation rule (mechanical) The instrument that enforces "keep the critic only if it strictly dominates its absence" mechanically. One mechanism removed at a time, everything else held fixed, the delta measured on the two numbers of §9.1 plus the harness's standard instruments: 1. **Ablate the critic** (critic-off) — remove the L entry entirely: the direct test of the critic's marginal value. The delta must show whether verified-defect recall drops and whether over-veto false positives were ever a cost worth paying. 2. **Ablate the blinding** — feed the critic the generator's rationale and self-confidence: the direct test of the rubber-stamping limb. The delta must show recall dropping (the critic recruited into agreement) with no over-veto gain. 3. **Ablate the falsification routing** — let blocking findings act as verdicts without running their falsification procedures: the direct test of the over-veto limb. The delta must show the false-positive rate spiking toward one (the critic as filibuster). 4. **Ablate the deterministic verification gate** — let the critic's own `confidence` or prose decide rejection: the direct test of the "confidence recorded but never trusted" rule. The delta must show the recall/over-veto trade collapsing into the self-report channel the component exists to avoid. A mechanism is **DELETED** iff its removal does **not** degrade the honest measures beyond noise. A mechanism whose *addition* degrades the measures is a demonstrated harm and is removed outright. **Never keep a critic because it sounds rigorous.** ### 9.6 Decision rule (explicit thresholds) Run critic-on vs critic-off on the *identical* held-out patch population with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The critic is **kept iff**: ``` verified-defect recall strictly higher with the critic, AND over-veto false-positive rate not worse (no more over-veto than the deterministic validator alone), AND the two numbers reported TOGETHER, never separately, never "did it sound good", AND the critic's removal degrades ≥1 honest measure beyond noise (it is not review theatre) (ablation) ``` **Strictly dominate the critic's absence** is the bar — a critic that is statistically indistinguishable from critic-off is removed until the evidence changes, and **an empty L column is a permitted, correct state** in the meantime. If the critic wins, it is adopted; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 9.7 Failure-mode audit (mechanical, the ways a review layer can lie) 1. **Rubber-stamp review** — the critic ratifying the generator's framing because blinding leaked: caught by the blinding ablation (§9.5 row 2) showing recall collapse when rationale is present. 2. **Self-preference capture** — the critic favoring its own generation because provenance survived the strip: caught by the provenance check in `packet.py`'s isolated test (Phase 1). 3. **Over-veto stall** — a critic with unbounded authority objecting to everything because objection is free: caught by the over-veto false-positive rate spiking when the verification gate is ablated (§9.5 row 3). 4. **Verdict-channel bias** — the critic emitting APPROVE/REJECT and triggering Zheng's position/verbosity/self-enhancement biases: structurally impossible, because the critic has no verdict channel — only falsifiable findings — and `critic.py`'s isolated test (Phase 4) asserts it. 5. **Confidence capture** — the critic's self-reported `confidence` influencing acceptance: caught by `verify.py`'s isolated test (Phase 6) asserting identical treatment of 0.99- and 0.51-confidence findings. ### 9.8 Two honest caveats **First, the weak-model caveat:** the pro-case numbers of the paper's §5 — CriticGPT's 63% human preference, CTRL's 64.3% JudgeBench, CRITIC's tool-grounded gains — were measured on frontier models a weak-model harness does not control; the weak model should expect materially smaller absolute effects while leaning *harder* on the deterministic verification gate, which costs nothing per model scale. **Second, the held-out caveat:** all critic metrics are reported on held-out patch sets, never tuning or in-benchmark data, so a critic that has memorized a benchmark's defects cannot pass a recall ablation without its usefulness having been measured on patches it never saw. --- ## 10. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/22-independent-reviewer-critic/`; it never writes into the live harness's source. - A **feature flag** (env var `CRITIC=22|off`, default `off`) routes the A/B; the flag alone switches between critic-on and critic-off — no code deletion required to revert. The default is `off`, because **an empty L column is a permitted, correct state**. - `ab.py` is a read-only driver: it *measures* the current harness, never modifies it. Running critic-on vs critic-off is always possible in the same session, repo, model. - The review ledger (every engage/decline decision, every packet's blind-strip outcome, every finding's schema-parse outcome, every falsification procedure's run outcome) is append-only and versioned; rolling the build back never loses it. **Invariants (must hold, checked at the end of every phase).** 1. **the L column contains at most one entry** — the critic (§5), produced only after the deterministic control plane has decided to engage it, constructed its blinded packet, and declared its output schema; the model never decides *whether* to engage, *what* it sees, *what* shape its output takes, *how* its findings are verified, or *whether* its findings are true. **an empty L column is a permitted, correct state** — a patch needing no critic is handled by the deterministic validator alone, and the emptiness is the design working, never a violation. 2. **The critic RECOMMENDS; execution CERTIFIES** — the critic has no accept action and no reject action; a blocking finding merely forces a deterministic verification step, and the step's outcome, not the critic, decides. 3. **Every blocking criticism carries a falsification procedure** — a finding whose `falsification` is empty or un-executable is downgraded to advisory or dropped, never blocking; the procedure is *run*, never re-asked. 4. **Confidence is recorded but never trusted** — a 0.99-confidence finding and a 0.51-confidence finding face identical verification; the critic's self-report has no authority anywhere in the control plane. 5. **The final acceptance authority is the deterministic validation policy** — not either LLM; the verification outcome is handed to component 18, which alone owns rejection, repair, and acceptance. 6. **The blinding contract holds** — generator rationale, self-confidence, peer votes, and provenance never reach the critic; the packet constructor never writes them, and the blind-strip stage deletes any that leak in. 7. **All metrics on held-out patch sets** — verified-defect recall and over-veto false-positive rate, and every secondary, are reported on held-out tasks, never tuning or in-benchmark data. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*