Summary / 22. Independent reviewer (critic) / Document
Paper 2026-09-06
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Component 22 — The Independent Reviewer / Critic ## Fresh-Context Adversarial Hypothesis Generation Under a Deterministic Acceptance Policy --- ### Abstract This paper specifies the twenty-second component of the coding harness: the independent reviewer, or critic. Its central claim is that independent review must be a *fresh-context, adversarial hypothesis-generation stage*, not a second model voting on whether a patch is correct. We argue the entire orchestration — review-packet construction, blinding and stripping of the generator's rationale and confidence, atomic-edit decomposition, claim-schema enforcement, falsification-procedure routing, and the final acceptance decision — is deterministic software running on CPU and holds **zero** learned entries, while the critic itself is the single learned entry: one bounded model call (or a short bounded sequence of calls) producing typed, falsifiable `ReviewFinding`s that *recommend* but never *certify*. We assemble the evidence in both directions. On the side of critic value: CriticGPT's 63% human preference and Force Sampling Beam Search, CTRL's weak-to-strong 64.3% JudgeBench accuracy, CRITIC's tool-grounded critique, and Lightman's step-level process supervision. On the side of failure of the naive second-opinion vote: Huang's null result that intrinsic self-correction flatlines without external signal, Panickssery's self-preference, Jin and Chen's over-correction (false-negative rate rising from 26.2% to 73.2% on HumanEval), Zheng's position/verbosity/self-enhancement judge biases, and Knight and Leveson's experimentally demonstrated correlated faults in multiversion programming. We close with the honest measurement protocol: critic value is scored by verified-defect recall against over-veto false-positive rate, A/B-tested against the deterministic validator alone, verified through fix-guided counterfactual execution, and trained on repository scar tissue. --- ## 1. Introduction The coding harness has, by component 21, produced a candidate patch through generation, editing, execution, tracing, and iterative repair, and has gated it through a deterministic validation policy that owns final acceptance. One question remains: is there a role for a *second model* whose only job is to attack the patch before acceptance? The answer this paper defends is that there is a role, but it is not the role a naive reading would assign it. **Independent review should be a fresh-context, adversarial hypothesis-generation stage rather than a second model voting on correctness: hide the generator's reasoning and confidence; give the critic the specification, diagnosed root cause, preservation obligations, impact evidence, and candidate patch; require review at atomic-edit and whole-patch levels; demand that every blocking criticism identify a concrete defect, supporting evidence, predicted consequence, and falsification procedure; independently verify critic claims through tests, static analysis, traces, or counterfactual execution; treat reviewer agreement as weak positive evidence but verified criticism as strong negative evidence; and reserve final acceptance authority for the deterministic validation policy rather than either LLM.** **The reviewer is not there to tell Piper whether the patch is good. It is there to manufacture the strongest falsifiable case that the patch is wrong.** This component therefore has an unusual epistemic signature among the twenty-two. Every earlier component *increases* the probability that a patch is correct; this one exists to *decrease* it — but only through falsifiable, evidence-backed claims that the deterministic layer can then check. A reviewer that returns "this patch is fine" has contributed almost nothing measurable; a reviewer that returns one verified defect has invalidated the patch. The asymmetry is deliberate and total. The core honesty of the component is about the word "independent." Independent does **not** mean "a second prompt to the same model." It means *different evidence access* and, at high stakes, *different weights* — possibly a different model, different temperature and decoding, and crucially a context from which the generator's self-assessment has been removed. The empirical literature is unambiguous on why this matters. Huang and colleagues [14] showed that intrinsic self-correction — asking a model to find and fix its own mistakes without any external signal — *flatlines or degrades* rather than improving: the model cannot generate the missing information that would correct its error, because the error was produced by precisely the knowledge that is still missing. Panickssery, Bowman, and Feng [8] showed that LLM evaluators recognize and *favor their own generations*: a judge rates its own output higher than equivalent output written by another model, and the strength of this self-preference scales linearly with the judge's ability to recognize its own writing. A "reviewer" that shares the generator's weights, context, rationale, and confidence is not a reviewer; it is an echo. **D/L invariant.** Every element of this component is classified into one of two columns: **[D]** — deterministic software running on CPU — or **[L]** — a learned model. The structural invariant that governs the entire design, and that every subsequent section is an elaboration of, is: > **the L column contains at most one entry.** Everything that moves a packet, strips a rationale, decomposes an edit, enforces a claim schema, routes a falsification procedure, or renders a final verdict is [D]. Only the critic's generation of hypotheses is [L]. The control plane holds zero learned entries; the learned plane holds at most one. --- ## 2. The failure of the naive "second LLM says APPROVE/REJECT" The naive version of this component is a single call: feed the patch (and often the generator's own explanation) to another LLM, and ask it to output APPROVE or REJECT. This design is tempting, simple, and broken in five distinct, documented ways. We enumerate them because each one is answered by a specific structural choice in §3–§4. **Rubber-stamp review.** When the reviewer is given the generator's reasoning and confidence, the generator's framing *recruits* the reviewer into agreement. The reviewer sees a confident, well-argued case and — other things equal — ratifies it. This is not a separate phenomenon from sycophancy; it is the same force operating on a smaller stage. Sharma and colleagues [21] showed that language models systematically drift toward a user's expressed views across a wide range of tasks, and that this drift is *not* fixed by merely instructing the model to be unbiased. A reviewer handed "here is why my patch is correct, and I am 0.93 confident" has already been sycophancy-primed. **Self-preference capture.** Even absent the generator's rationale, if the reviewer shares the generator's weights, Panickssery, Bowman, and Feng's result applies directly [8]: the judge prefers its own output. Because the candidate patch *is* the judge's own output (same model), the judge's verdict is systematically inflated. Blind sampling is the only fix: strip the provenance that allows self-recognition to bind. **Correlated blind spots.** Knight and Leveson [12] is the foundational negative result and the reason "independence" must be treated as an empirical claim rather than an assumption. In their multiversion-programming experiment, independently developed versions failed on the *same* inputs far more often than the independence assumption predicted, because the versions shared correlated design faults — common misunderstandings of the specification, common blind spots about edge cases, common library misuses. Two models, or one model prompted twice, share *correlated* blind spots: they both miss the same off-by-one, the same unhandled null, the same concurrency race. The failure modes are not additive; they overlap. A second vote from a correlated source buys far less than the vote-count suggests. **Sycophancy drift.** The reviewer is embedded in a loop where the generator and the harness both have incentives. If the reviewer's "disagreement" ever feeds back as cost or friction, the reviewer learns (or is steered) toward agreement; if its "agreement" is ever rewarded as "the team works well together," it learns to agree. Over repeated turns, an LLM reviewer without structural separation drifts toward whatever makes the conversation smoother — which is approval. Constitutional-AI-style separation of the critic from the policy it critiques is the relevant prior [19]; a critic that shares the objective surface of the thing it reviews is compromised. **Over-veto stall.** The symmetric failure: a reviewer with unbounded authority and no cost for objecting will object to everything, because an objection that turns out to be wrong is (in a naive design) free while an approval that turns out to be wrong is blamed on the reviewer. This produces a harness that never ships. Knight and Leveson's correlated-faults result is the antidote-shaped warning here too: if we *do* succeed in making the critic genuinely independent, we must then prevent it from becoming a permanent filibuster. The asymmetry of §4 — verified criticism is expensive, and acceptance authority is reserved to the deterministic layer — is what keeps the critic honest in *both* directions. The common thread: a naive second-LLM verdict is *weak evidence in the approval direction* (because of rubber-stamping, self-preference, correlation, and sycophancy) and *unbounded, unaccountable evidence in the rejection direction* (because of over-veto). Both limbs must be amputated. The next two sections describe the surgery. --- ## 3. The review packet and blinding (D) The review orchestration is deterministic software. This section specifies what the critic is *given* and, just as importantly, what it is *never shown*, and why the entire control surface is [D]. ### 3.1 What the critic receives The review packet is assembled by a deterministic packet-construction stage and contains exactly five evidentiary elements: 1. **The specification.** The task the patch was supposed to accomplish — the issue text, the target behavior, the acceptance contract — expressed independently of the generator's interpretation of it. The critic sees the requirement, not the generator's paraphrase of the requirement. 2. **The diagnosed root cause.** The *diagnostic* conclusion of the failure that motivated this patch: what the harness established was wrong (the failing test, the trace, the fault-localization ranking, the reproduced symptom). This is an evidentiary artifact, not an argument. Note the precise distinction enforced here: the generator's *diagnosis of the prior failure* is evidence and is included; the generator's *rationale for why the current patch is correct* is argument and is stripped (§3.2). 3. **The preservation obligations.** The explicit list of behaviors that must not change — the regression surface, the pinned contracts, the invariants the deterministic validator will re-check. The critic is asked to attack the patch *against these obligations*, so its criticism is testable rather than aesthetic. 4. **The impact evidence.** The computed blast radius and dependency-impact data (from the component-06 lineage): which callers, modules, and invariants the diff touches, derived deterministically from the code graph, not asserted by the generator. 5. **The candidate patch.** The diff itself, decomposed at the atomic-edit level (each hunk as a separate addressable unit) plus the whole-patch view. The decomposition is mechanical — a diff-parser splits the patch into atomic edits; no model is involved. ### 3.2 What the critic is stripped of — blinding Deterministic blinding removes, before the packet reaches the critic: - **The generator's rationale.** Any narrative "here is why this works" is deleted. The critic must reconstruct the patch's intent from the specification and the diff alone. - **The generator's self-confidence.** No "I am 0.9 sure." The critic has no number to anchor to. - **Peer votes.** No prior reviewer's agreement or disagreement is ever shown; each review is a fresh context. If multiple critic passes are run, none sees the others' findings. - **Provenance.** Nothing that would let the critic detect that the patch came from a model sharing its own weights or from itself. The blinding is *the* implementation of the "independent means different evidence access" honesty from §1. A fresh context is not a nicety; it is the mechanism by which Panickssery's self-preference [8] and Sharma's sycophancy [21] are made structurally impossible to trigger, rather than merely discouraged. ### 3.3 This layer is [D], and the L column is empty here Packet construction, blinding and stripping, edit-level decomposition, claim-schema enforcement (the schema the critic's output must fill), falsification-procedure routing (matching each criticism to *how* it will be verified — test, static analysis, trace replay, or counterfactual execution), and the final acceptance authority are all deterministic software on CPU. None of these stages runs a learned model. Recapitulating the invariant: > **the L column contains at most one entry.** And at this layer it is exactly zero. **An empty L column is a permitted, correct state** — when the deterministic validator alone suffices to accept or reject the patch (small blast radius, fully covered by existing gates, nothing a critic could add that is not already checked), no critic is spawned at all. The critic is an *optional* instrument engaged only when the marginal value of an adversarial pass exceeds its cost. The control plane runs whether or not the critic exists; the critic never runs without the control plane. --- ## 4. The critic as hypothesis-generator (L) The critic is the single learned entry. This section specifies exactly what it is permitted to be, what it outputs, and what it is forbidden to do. ### 4.1 Claim-granular output: the ReviewFinding schema The critic never outputs a verdict. It outputs a set of typed, falsifiable findings. Each `ReviewFinding` is a record with exactly six fields: ``` ReviewFinding { location: string, // atomic-edit id or "whole-patch" claim: string, // the concrete defect asserted, in one sentence severity: enum, // {blocking, advisory} evidence: string, // what in the packet supports the claim (spec clause, diff line, trace fragment) falsification: string, // the procedure that would prove the claim WRONG (a test, a static check, a replayed trace, a counterfactual execution) confidence: float, // 0..1, the critic's self-assessed probability the claim survives verification } ``` Four design decisions are load-bearing: - **Location is mandatory and granular.** The critic must name an atomic edit or the whole patch. A criticism that cannot be located cannot be verified. This is what "require review at atomic-edit and whole-patch levels" means operationally: the packet is split so a defect can be pinned to a single hunk, and the whole patch is also reviewed as a unit so *emergent* defects (two hunks each locally fine but jointly wrong) are not missed. - **Every blocking criticism carries a falsification procedure.** This is the Popperian core [22]: a criticism that cannot state how it would be shown wrong is not a scientific claim, it is a vibe. The critic is *required* to say what observable would refute its own claim. The deterministic layer will then execute exactly that refutation (§4.3). This single field is what turns "criticism" into "hypothesis." - **The schema is enforced deterministically.** The critic's raw output is parsed and validated against the schema; malformed or missing fields are rejected and re-requested (a bounded retry), and a finding whose `falsification` field is empty or un-executable is *downgraded to advisory or dropped*, never blocking. The critic does not get to define the rules of its own acceptance. - **Confidence is recorded but never trusted.** The critic's `confidence` is captured because it is cheap metadata for calibration tracking, but it has no authority: a 0.99-confidence finding and a 0.51-confidence finding face identical verification. (SelfCheckGPT's lesson [20] applies in miniature: a model's self-reported certainty about its own output is weak evidence about whether that output is true.) ### 4.2 Recall over precision; recommend, never certify The critic is optimized for **recall** — surfacing *candidate* defects, including speculative ones — while **precision** is provided by the deterministic verification stage that follows. This division of labor is what lets the critic be genuinely adversarial without being dangerous: it is encouraged to over-flag, because every flag is routed through a verification gate that discards false positives cheaply and deterministically. The critic **RECOMMENDS**; execution **CERTIFIES**. The critic cannot accept a patch (it has no accept action — only "no blocking findings"), and it cannot reject a patch either: a blocking finding merely *forces a deterministic verification step*, whose outcome — not the critic's — determines whether the patch is rejected, repaired, or accepted. This is the structural answer to over-veto stall (§2): the critic's veto power is zero; its power is only to *route evidence*. Recapitulating the invariant at its tightest form: > **the L column contains at most one entry** — and that single entry is one bounded model call (or a short bounded sequence of calls, each schema-validated), producing typed falsifiable `ReviewFinding`s. It never outvotes deterministic evidence, because it is never given a vote to cast. ### 4.3 The asymmetry The component's epistemic asymmetry, stated once and enforced everywhere: - **Reviewer agreement (no blocking findings) is weak positive evidence.** Given rubber-stamping, self-preference, correlation, and sycophancy, a silent critic is only barely informative. It does not certify the patch; it merely fails to add a known defect. - **A verified criticism is strong negative evidence.** When a blocking finding's falsification procedure is *run* and the claim *survives* — the test fails, the static check trips, the trace confirms, the counterfactual execution reproduces the defect — the patch is, for acceptance purposes, broken. One verified defect invalidates the patch regardless of how many other findings failed verification. This asymmetry is the exact inverse of the naive design's error. The naive design over-weights approval and under-verifies rejection; this component under-weights approval and over-verifies rejection. --- ## 5. The evidence both ways This section surveys the empirical record the component is built on, organized by what it teaches. We cite the positive results (critics can add genuine value) and, with equal care, the negative and outlier results (naive critic designs fail in specific, measurable ways). Both columns are load-bearing: the positives tell us *what to build*, the negatives tell us *what the positives must be surrounded with*. ### 5.1 Positive: critics add value when grounded, fresh, and trained for criticism **CriticGPT — "LLM Critics Help Catch LLM Bugs."** McAleese, Pokorny, Ceron Uribe, Nitishinskaya, Trebacz, and Leike [1] trained a critic model via reinforcement learning from human feedback to critique code produced by another model. Human contractors preferred CriticGPT's critiques over human-written critiques **63%** of the time on inserted-bug tasks. Two findings matter for this component specifically. First, the critic was trained to be a *critic* — RLHF against human preference over the *quality of the criticism*, not over code correctness — which is the difference between "a model that reviews code" and "a model optimized to review code." Second, the critique-generation procedure is itself a search: **Force Sampling Beam Search**, scoring `rm_score + LENGTH_MODIFIER * num_highlights`, samples over highlighted spans rather than free-generating, explicitly trading hallucinated-bug precision against recall. The paper reports this precision/recall tradeoff directly: critics hallucinate bugs (false positives), and the sampling scheme is a knob for how aggressively to accept that cost. This is the empirical warrant for our recall-over-precision division of labor (§4.2): hallucinated bugs are a *known, tunable* property of a good critic, which is exactly why every candidate bug must be deterministically verified before it can block. **CTRL — "Teaching Language Models to Critique via Reinforcement Learning."** Xie, Chen, Chen, Mao, Xu, and Kong [6] trained a weak-to-strong critic with a pair-sorting reward and showed it reaches **64.3% on JudgeBench**, competitive with Claude-3.5-Sonnet, and — critically for us — that a critic trained this way *outperforms both self-critique and a stronger-critic baseline*. The weak-to-strong result is the direct refutation of "the critic must be at least as capable as the generator": a smaller, *differently-trained* critic beats a bigger, self-critiquing generator, because what the critic needs is not raw capability but a distinct training objective. This is the empirical basis for "different weights at high stakes" in §1: independence can be manufactured by training objective and scale asymmetry even within one family of models. **CRITIC — "Large Language Models Can Self-Correct with Tool-Interactive Critiquing."** Gou, Shao, Gong, Shen, Yang, Duan, and Chen [10] showed that critique *grounded in external tool feedback* produces consistent improvement across question answering, program synthesis, and toxicity reduction — while self-critique without tools adds little or harms. The lesson maps one-to-one onto §4.3: the critic's value is realized only when its claims are routed through something external (here, tool execution; in our design, the deterministic verification gate). A critic that is not connected to a verification channel is not a critic, it is a commentary generator. **Lightman — "Let's Verify Step by Step."** Lightman, Kosaraju, Burda, Edwards, Baker, Lee, Leike, Schulman, Sutskever, and Cobbe [11] showed that *process* supervision — rewarding each reasoning step, not just the final answer — produces verifiers that solve 78% of a representative MATH subset, substantially outperforming outcome supervision, and that a *small* process-reward verifier generalizes to guide a *larger* generator (the easy-to-hard direction). For this component the lesson is granularity: a verifier that attends to *steps* (here, atomic edits) finds defects a whole-answer (whole-patch) verifier misses. This is the direct warrant for the atomic-edit + whole-patch dual granularity of §4.1. **SpecRover — "Code Intent Extraction via LLMs."** Ruan, Zhang, and Roychoudhury [2] combine LLM-derived code intent with a repair agent, reporting **more than 50% improvement in efficacy over AutoCodeRover** on the **full SWE-bench of 2,294 GitHub issues**, at a cost of **$0.65 per issue** — and here the scope distinction matters for honest reading: the $0.65/issue figure is measured on **SWE-bench Lite**, *not* the 2,294-instance full set, while the >50% improvement and the 2,294 count refer to full SWE-bench. Conflating the two scopes is precisely the kind of citation error this component's citation discipline exists to prevent. For this component, SpecRover's relevance is narrower: intent extraction is exactly the *grounded signal* a critic should be handed (the specification and diagnosed intent, §3.1) rather than being left to reconstruct the generator's intentions from the diff alone. **Abstain and Validate — "A Dual-LLM Policy for Reducing Noise in Agentic Program Repair."** Cambronero, Tufano, Shi, Wei, Uy, Cheng, Liu, Pan, Chandra, and Rondon [3] evaluate, on **174 human-reported bugs**, two complementary noise-reduction policies and their combination: bug **abstention** (the agent declines low-confidence tasks) raises success by **up to +13 percentage points**, **patch validation** by **up to +15 percentage points**, and the two **in combination by up to +39 percentage points** — not +29, a figure that must be corrected in any downstream citation. This is the closest empirical relative of our design: a *second, distinct* model whose only job is to reduce noise, measured by its effect on *verified* success rather than on its own self-report. It is also the direct warrant for §6's over-veto metric, since abstention and validation are exactly the two places a critic can either help (recall) or hurt (false-positive stall). **Yadavally — "Large Language Model Critics for Execution-Free Evaluation of Code Changes."** Yadavally, Nguyen, Callot, and Guinet [4] show an **execution-focused** critic achieves reference-aware F1 of **91.6%** and build-status prediction accuracy of **84.8%**, outperforming other reference-free and reference-aware critics by **38.9%–72.5%**. The lesson is the same one CRITIC teaches, but from the code-review side: grounding the critic in *execution* signals — what the code change actually does — is what separates a useful critic from a plausible-sounding one. This is why our critic's every finding must carry an execution-linked falsification procedure (§4.1), and why §6 verifies through counterfactual execution rather than through re-asking the critic. **Adjacent, cited for completeness.** SE-Jury [15] shows an ensemble of judges correlating 64.3% with human judgment against a best single baseline of 49.6% — though we note, because it matters for honest reading, that the headline "+75.2%" figure is a *program-repair-task-specific* number (APR-Assess), not the overall gap. Generative Verifiers [16] reframe reward modeling as next-token prediction, the same generative-verification direction as CTRL. Frömmgen and colleagues [17] show production-scale machine learning for resolving real code-review comments — the empirical existence proof that review-as-a-service is a solved *product* problem even if the *epistemic* problem of §2 remains. Shepherd [18] provides a fine-grained *critic* language model over specific spans, the same span-granularity motif as CriticGPT. ### 5.2 Negative and outlier: why the naive design fails **Huang — "Large Language Models Cannot Self-Correct Reasoning Yet."** Huang, Chen, Mishra, Zheng, Yu, Song, and Zhou [14] found that intrinsic self-correction — the model reviewing and fixing *its own* output with no external feedback — does not improve reasoning accuracy and in several settings *degrades* it, because the model cannot generate information it does not have. This is the null result that kills "the generator can review itself," and it is the reason this component forbids self-review structurally (§3.2). (Its foil is Self-Refine [13], which *can* help when the self-feedback is grounded in concrete external signal such as a compiler or test error — the boundary our design draws at "verification must be external to the critic.") **Panickssery — "LLM Evaluators Recognize and Favor Their Own Generations."** Panickssery, Bowman, and Feng [8] showed judges prefer their own output and that this self-preference is linearly correlated with self-recognition ability. It is the empirical reason blinding strips provenance (§3.2): if the reviewer can recognize the patch as "its kind," its verdict is inflated by a measurable, systematic amount. **Jin and Chen — "Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement."** Jin and Chen [5] measured what happens when a reviewer is asked not merely to judge but to *over-apply* a requirement: on HumanEval, GPT-4o's false-negative rate *rose* from **26.2% to 73.2%**, and on MBPP from **35.9% to 87.9%**, as the review prompt escalated from a direct judgment to a "full" direct-plus-explain-plus-fix pipeline. This is the quantitative form of over-veto stall: a reviewer asked to be "thorough" manufactures false defects at a catastrophic rate. Their **Fix-guided Verification Filter** — treating the reviewer's proposed fix as *executable counterfactual evidence* to be checked, rather than as a verdict to be trusted — is precisely the §4.3 asymmetry implemented as a method: the criticism is converted into a falsifiable execution, and only the execution's outcome counts. (We note explicitly: this paper is real and correctly cited; its 2026 Automated Software Engineering DOI and arXiv 2603.00539 are verified, and the numbers above are read from the paper's tables.) **Zheng — "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena."** Zheng, Chiang, Sheng, Zhuang, Wu, Zhuang, Lin, Li, Li, Xing, Zhang, Gonzalez, and Stoica [9] documented that LLM judges exhibit **position bias**, **verbosity bias**, and **self-enhancement bias**, and that while their agreement with humans reaches ~80% on MT-Bench, the biases are strong enough that naive judge outputs cannot be taken as ground truth. This is the warrant for why *this* component never asks the critic for a scalar verdict at all: every bias Zheng catalogued operates on the *verdict* channel, which we have removed, and none of them operates on the *falsifiable-finding* channel, which is all we kept. **Knight and Leveson — "An experimental evaluation of the assumption of independence in multiversion programming."** Knight and Leveson [12] demonstrated empirically that independently developed program versions fail on the same inputs far more than the independence assumption predicts, because of correlated design faults. It is the oldest and most important negative result in this space: *independence must be measured, not assumed*, and even genuinely independent developers share correlated blind spots. For this component it is the justification for treating "different weights at high stakes" as a *necessary but not sufficient* condition for independence (§1), and for the humility encoded in §4.3 — even a well-built critic is only ever weak positive evidence when silent, because the correlation between generator and critic can never be fully eliminated, only reduced. **The adversarial-debate caveat (Khan).** Khan and colleagues [7] showed that debate with more persuasive models improves truth-detection — non-expert judges reach 76% and humans 88% — but the mechanism is *persuasiveness*, and the result is about aggregating argument quality, not about a second-model vote. We cite it as the limit case: adversarial exchange helps when the *judge* is external and unaligned (a human, a distinct model, a deterministic check), and degrades toward self-preference the moment the debater and judge share an objective surface. Our design keeps the adversarial surface (the critic attacks) but removes the debate's weakness (no LLM is ever the judge). --- ## 6. Honest measurement and ablation A component whose whole premise is "don't trust the critic's self-report" must hold *itself* to the same standard. This section specifies how the critic's value — or lack of it — is measured, so that the component earns its place in the harness by evidence rather than by assertion. **The metric is verified-defect recall against over-veto false-positive rate — not "did the critic sound good."** A critic is evaluated by exactly two numbers, both derived from the deterministic verification gate, never from the critic's own confidence: (1) *verified-defect recall* — of the defects that actually existed in a held-out patch set, how many did the critic surface a falsifiable claim for, where the claim then *survived* verification; and (2) *over-veto false-positive rate* — of the blocking findings the critic emitted, how many were *killed* by their own falsification procedure. A critic that never blocks scores zero recall; a critic that blocks everything scores a false-positive rate near one. Both are failures. The sound-good metric — "the critique was well-written and thorough" — is explicitly *not* scored, because Zheng [9] showed verbosity bias and Sharma [21] showed sycophancy both live exactly on that channel. **A/B against the deterministic validator alone.** The critic's marginal value is measured by running the identical patch population twice: once through the deterministic validator alone, once with the critic inserted before it. The critic earns its compute only if the critic-on arm shows (a) more verified defects caught, or (b) the same defects caught with cheaper total verification (the critic routing verification effort toward the high-value checks). If critic-on is statistically indistinguishable from critic-off, the invariant's escape hatch fires: **an empty L column is a permitted, correct state**, and the critic is removed until the evidence changes. **Fix-guided counterfactual verification is the standard of proof.** Following Jin and Chen's Fix-guided Verification Filter [5] and CRITIC's tool-grounded critique [10], every blocking finding is settled by *executing* its falsification procedure, not by re-asking the critic. A finding claiming "this atomic edit removes a null check that the contract requires" is settled by running the contract's test on the edit and on the *counterfactual* (the edit reverted, or the critic's proposed repair applied); the outcome, not the critic, decides. This makes the critic's role in measurement exactly its role in operation: it *proposes falsifiable hypotheses*, and the deterministic layer *executes the falsification*. **Train on repository scar tissue.** CriticGPT's construction [1] — training a critic on *known* inserted and historical bugs, with human preference over the *quality of criticism* — is the template for tuning this component's critic. The training corpus is the repository's own scar tissue: past regressions, past reverted commits, past defects that the deterministic gates caught late, each labeled with the evidence that *would have* caught it. A critic trained on this specific failure distribution is measurably better at surfacing *this repository's* correlated blind spots — which is, by Knight and Leveson [12], exactly the failure class a generic critic is worst at. The critic is therefore not a fixed off-the-shelf model but a *continually re-tuned* one, and each re-tuning is itself gated: a new critic is adopted only if it strictly dominates the incumbent on verified-defect recall at no worse false-positive rate. --- ## 7. Conclusion The independent reviewer is the one component whose job is to be wrong on purpose — to generate the strongest falsifiable case against the patch, so that whatever survives the case is genuinely hard to break. **Independent review should be a fresh-context, adversarial hypothesis-generation stage rather than a second model voting on correctness: hide the generator's reasoning and confidence; give the critic the specification, diagnosed root cause, preservation obligations, impact evidence, and candidate patch; require review at atomic-edit and whole-patch levels; demand that every blocking criticism identify a concrete defect, supporting evidence, predicted consequence, and falsification procedure; independently verify critic claims through tests, static analysis, traces, or counterfactual execution; treat reviewer agreement as weak positive evidence but verified criticism as strong negative evidence; and reserve final acceptance authority for the deterministic validation policy rather than either LLM.** **The reviewer is not there to tell Piper whether the patch is good. It is there to manufacture the strongest falsifiable case that the patch is wrong.** The design reduces to one structural invariant, stated once more: **the L column contains at most one entry.** The orchestration — packet construction, blinding and stripping, edit decomposition, schema enforcement, falsification routing, acceptance — is deterministic software on CPU with zero learned entries. The critic is the single learned entry, a bounded call producing typed falsifiable `ReviewFinding`s. It **RECOMMENDS**; execution **CERTIFIES**. And because the critic is an optional instrument rather than a load-bearing gate, **an empty L column is a permitted, correct state** — the deterministic validator suffices alone whenever the evidence says so. The critic is kept, then, only for as long as it is measurably better than its own absence at finding what a correlated reviewer is worst at finding: the defect that the generator, confidently and coherently, did not know it had. --- ## References [1] Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. "LLM Critics Help Catch LLM Bugs." arXiv:2407.00215, 2024. (arXiv preprint; no peer-reviewed venue.) [2] Haifeng Ruan, Yuntong Zhang, and Abhik Roychoudhury. "SpecRover: Code Intent Extraction via LLMs." In Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE 2025), pp. 963–974. DOI: 10.1109/ICSE55347.2025.00080. arXiv:2408.02232. [3] José Cambronero, Michele Tufano, Sherry Shi, Renyao Wei, Grant Uy, Runxiang Cheng, Chin-Jung Liu, Shiying Pan, Satish Chandra, and Pat Rondon. "Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair." In Proceedings of the IEEE/ACM 48th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2026), pp. 118–129. DOI: 10.1145/3786583.3786858. arXiv:2510.03217. [4] Aashish Yadavally, Hoan Anh Nguyen, Laurent Callot, and Gauthier Guinet. "Large Language Model Critics for Execution-Free Evaluation of Code Changes." arXiv:2501.16655, 2025. (arXiv preprint; no peer-reviewed venue.) [5] Haolin Jin and Huaming Chen. "Are LLMs Reliable Code Reviewers? Systematic Overcorrection in Requirement Conformance Judgement." Automated Software Engineering 33(3), 2026. DOI: 10.1007/s10515-026-00638-5. arXiv:2603.00539. [6] Zhihui Xie, Jie Chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. "Teaching Language Models to Critique via Reinforcement Learning." In Proceedings of the 42nd International Conference on Machine Learning (ICML 2025). arXiv:2502.03492. [7] Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, and Ethan Perez. "Debating with More Persuasive LLMs Leads to More Truthful Answers." In Proceedings of the 41st International Conference on Machine Learning (ICML 2024). arXiv:2402.06782. [8] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. "LLM Evaluators Recognize and Favor Their Own Generations." In Advances in Neural Information Processing Systems 37 (NeurIPS 2024). DOI: 10.52202/079017-2197. arXiv:2404.13076. [9] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." In Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track (NeurIPS 2023). DOI: 10.52202/075280-2020. arXiv:2306.05685. [10] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. "CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing." In The Twelfth International Conference on Learning Representations (ICLR 2024). arXiv:2305.11738. [11] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. "Let's Verify Step by Step." arXiv:2305.20050, 2023. Later presented at ICLR 2024. [12] John C. Knight and Nancy G. Leveson. "An experimental evaluation of the assumption of independence in multiversion programming." IEEE Transactions on Software Engineering SE-12(1):96–109, 1986. DOI: 10.1109/TSE.1986.6312924. [13] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. "Self-Refine: Iterative Refinement with Self-Feedback." In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). arXiv:2303.17651. [14] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. "Large Language Models Cannot Self-Correct Reasoning Yet." In The Twelfth International Conference on Learning Representations (ICLR 2024). arXiv:2310.01798. [15] Xin Zhou, Kisub Kim, Ting Zhang, Martin Weyssow, Luis F. Gomes, Guang Yang, Kui Liu, Xin Xia, and David Lo. "An LLM-as-Judge Metric for Bridging the Gap with Human Evaluation in SE Tasks." arXiv:2505.20854, 2025. (arXiv preprint.) [16] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. "Generative Verifiers: Reward Modeling as Next-Token Prediction." In The Thirteenth International Conference on Learning Representations (ICLR 2025). arXiv:2408.15240. [17] Alexander Frömmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. "Resolving Code Review Comments with Machine Learning." In Proceedings of the 46th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2024). DOI: 10.1145/3639477.3639746. [18] Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O'Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. "Shepherd: A Critic for Language Model Generation." In The Twelfth International Conference on Learning Representations (ICLR 2024). arXiv:2308.04592. [19] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, the operator Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. "Constitutional AI: Harmlessness from AI Feedback." arXiv:2212.08073, 2022. [20] Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. "SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models." In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023). DOI: 10.18653/v1/2023.emnlp-main.557. arXiv:2303.08896. [21] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, the operator R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. "Towards Understanding Sycophancy in Language Models." In The Twelfth International Conference on Learning Representations (ICLR 2024). arXiv:2310.13548. [22] Karl R. Popper. "Conjectures and Refutations: The Growth of Scientific Knowledge." Routledge & Kegan Paul, London, 1963. --- *Carried forward from the ledger (not re-verified here):* CodeReviewer (arXiv 2203.09095), R2E-Gym (arXiv 2504.07164), SWE-RM (arXiv 2512.21919), and TraceRepair (arXiv 2604.02647) remain available as related-work pointers from the component-18 and component-21 verification passes.