Summary / 11. Code execution / Document
Spec 2026-09-05
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
# Build Spec 11 — Code Execution (the ≈80% fraction) > Derived from `papers/11-code-execution.md` — "The Runtime Is the Only Witness." This document > drives the build of the **code-execution stage**: the deterministic, isolated > **evidence-generation service** that turns the repository from a thing the harness merely reads > and edits into a thing that *answers back*. It is the **observation layer**. It sits immediately > after component 10 (editing, the write path, which landed the model's patch on disk) and before > the components that verify and repair (17–18), which judge whether that patch is good. Where > generation decides *what* and editing decides *whether it lands*, execution is the one channel > through which the repository itself talks back — mechanically, reproducibly, without the > intervention of any model's opinion about what code does. Every candidate is **run, never mentally > simulated**; commands execute under reproducible resource and security boundaries; raw runtime > behavior is normalized into compact, provenance-rich observations; execution evidence is used both > to *diagnose* failures and to *discriminate* among candidate implementations; and bounded > diagnostic probes may be generated where existing tests are insufficient, never mistaken for > authoritative specifications. This spec translates the paper's design into exact action-mode > tables, an exact isolation policy (CPU/RAM caps, filesystem boundary, network policy, timeouts), > exact raw-execution and deterministic per-toolchain evidence-parsing mechanics, exact > normalization into the failure-class vocabulary, exact behavioural-caching keys, the exact > `ExecutionObservation` schema, exact L-column caps and firing conditions, exact A/B instruments and > thresholds, exact ablation mechanics, and an exact build sequence — no further research required. > Where the paper says "≈80%", this spec pins *which* 80%. --- ## 1. Purpose & Scope Build a **reversible code-execution stage** — a deterministic-first, isolated, evidence-generation service (action classification → isolation policy → raw execution → evidence parsing → normalization into a failure-class vocabulary → behavioural caching/dedup → structured observation assembly) with a **single weak-LLM diagnostic-probe-generation call**, as a drop-in *observation* stage that converts a candidate artifact (a patch, a snippet, a command), its declared environment, and the existing test suite into the single structured **`ExecutionObservation`** that the diagnosis and discrimination stages (and, through them, the repair and verification stages) consume. It is distinct from its predecessors in *kind*, not merely in order: everything upstream of this component asks the model what it *thinks*; execution is the single stage that asks the repository what it *does*. Editing is the write path; execution is the observation path that makes the write meaningful, because **a patch that is never run has not been shown to do anything at all**. The runtime, not the model, is the only witness to what a candidate program actually does. The weak-model implication is the reason this stage is load-bearing: a weak model's beliefs about what code does are cheap, fast, and unreliable, whereas the runtime's answer is expensive-but-true — and the harness sits between them. The overall target is the paper's full 100% spec. **Round 1 lands the expected ≈80% of it**: everything deterministic (action classification, the isolation policy, raw execution, per-toolchain evidence parsing, normalization into the failure-class vocabulary, behavioural caching/dedup, and structured observation assembly with its empty-success guard, determinism check, and anti-gaming re-verification) plus the *bound* of the *one* diagnostic-probe-generation call — targeting the reachable fraction of the design where the L column holds exactly one entry and every other step is deterministic software on CPU. The heavy learned / container-grade / trained-verifier / rich-diff / multi-language-parser pieces (container-grade isolation via gVisor or Docker, the learned diagnostic-probe generator trained at scale, a LEVER-style learned verifier, AlphaCode-style behavioural clustering, rich `changed_state` via filesystem/database diff, and multi-language evidence parsers for C/C++/Rust/SQL/Java) are **not** built in Round 1 and are **not excluded** — they are the content of later rounds that climb from that ~80% landing toward 100% (see §2). Round 1 is kept only if it wins a measured A/B test on the *honest* code-execution metrics — led by **observation fidelity** (verdict correctness AND failure-class classification accuracy against a naive `subprocess.run` + raw-stdout baseline), never the end-to-end resolve rate alone, which would let execution quality mask repair quality (§7). --- ## 2. Round 1 vs later rounds (toward the 100% target) The goal is the paper's full **100%** spec. **~80%** is only the expected **first-round landing**; later rounds climb from there. Every element from the paper is classified below by *which round* builds it. - **BUILD-NOW = the Round 1 build** — the expected ≈80% deterministic landing this spec implements and A/B-tests (rows 1–9). - **DEFERRED = later rounds** — the container-grade / learned / trained-verifier / rich-diff / multi-language-parser pieces (rows 10–15) that Round 1 intentionally does **not** build *yet*, because each adds a container runtime, a second (or fully trained) model call against the L-column invariant, a learned verifier or clustering step, a rich persistent-diff substrate, or a full per-language parser surface. They are **not excluded from the project** — they are the content of later rounds, the climb toward 100%, gated by the same §7 A/B protocol once the Round 1 core is in place. | # | Component (paper §) | Classification | |---|---|---| | 1 | (a) Action classification — deterministic table over request type + artifact shape → one of the four execution modes (compile/parse, targeted execution, test execution, diagnostic probe) | **BUILD-NOW** | | 2 | (b) Isolation policy — subprocess + resource limits (CPU cap, RAM cap) + a scratch-dir filesystem boundary + network deny-by-default + per-mode hard timeouts; the policy recorded in the observation | **BUILD-NOW** | | 3 | (c) Execute — spawn under the declared bounds, capture raw artifacts (stdout, stderr, exit code, wall-clock duration, exception); on timeout/resource-cap, kill deterministically and record the kill as the outcome | **BUILD-NOW** | | 4 | (d) Evidence parsing — compile errors / runtime evidence / test verdicts, deterministic per toolchain (Python toolchain first) | **BUILD-NOW** | | 5 | (e) Normalization — classify parsed evidence into the failure-class vocabulary {compile, syntax, assertion, runtime, timeout, resource, type, dependency}; each failure carries location + message + exit code + duration | **BUILD-NOW** | | 6 | (f) Behavioural caching/dedup — key = hash(code + environment + command); an identical key returns the prior observation with no re-run | **BUILD-NOW** | | 7 | (g) Structured observation assembly — the `ExecutionObservation` schema; empty-success guard (exit 0 + zero tests = flagged); determinism check; anti-gaming re-verification (re-run "all pass" + diff-scope check that tests weren't edited) | **BUILD-NOW** | | 8 | A/B harness (§7: observation-fidelity measure + failure-class precision/recall + ablation runner + the probe/no-probe discrimination instrument) | **BUILD-NOW** | | 9 | (L) Diagnostic-probe generation — **the single learned entry**, included as a *bounded stub* (see the cost rule below) | **BUILD-NOW** (bounded stub only) | | 10 | **Container-grade isolation** — gVisor / Docker sandboxing of every execution | **DEFERRED** — Round 1 isolates via `subprocess` + resource limits + a scratch root (InterCode's Docker self-containment realized at subprocess scale); gated by §7 once a container substrate is warranted | | 11 | **Learned diagnostic-probe generation at scale** — the one L entry trained/calibrated against probe-discrimination and non-promotion rates | **DEFERRED** — the full training/calibration of the probe generator; Round 1 ships only the bounded stub (§9) | | 12 | A **LEVER-style learned verifier** — a model trained to score a candidate over the NL task + candidate program + execution result (LEVER [8]) | **DEFERRED** — adds a fully trained *second* model against the L-column invariant; needs training; the deterministic discrimination of §7 must be shown insufficient first | | 13 | **AlphaCode-style behavioural clustering** — cluster survivors BY RUNTIME BEHAVIOUR, not text, for scale-based selection (AlphaCode [5]) | **DEFERRED** — needs the ~scale of candidates a weak generator is not yet producing; Round 1 discrimination is per-observation and per-candidate-pair, not cluster-based | | 14 | **Rich `changed_state`** — filesystem/database diff as a first-class observation field | **DEFERRED** — Round 1 records a declared-changed-state summary only; the diff substrate is later-round content | | 15 | **Multi-language evidence parsers** — C/C++/Rust/SQL/Java compile-error and runner parsers | **DEFERRED** — Round 1 parses the Python toolchain only (§3.4); each additional language needs its own declared deterministic parser | **The cost rule (explicit).** The paper's single learned entry — diagnostic-probe generation — is the one L-column entry this architecture permits. In Round 1 it is included as a **bounded stub**: the stage exists, is wired into the pipeline at the correct firing point (§4), is gated, executes its probes as probe evidence, and is never auto-promoted — but its full training and calibration (the recursive-self-improvement probe-generation loop of the paper's §5, tuning the probe prompt against discrimination and non-promotion rates) is **deferred** to later rounds. The stub's *cost* is bounded before it is ever invoked: a probe cap of 8 per firing, each probe a small program/assertion (§4), temperature zero, one call, schema-constrained output, parse-failure fallback to "no probe". The stub is kept **only if** the §7 no-probe-vs-probe ablation shows it improves downstream selection on held-out repositories; otherwise it is removed, and the L column stands empty in Round 1 — an empty L column is a permitted, correct state, never a violation. **The full training/calibration of the learned entry is deferred regardless.** **Design decision: DETERMINISTIC-FIRST, LEARNED-EXCEPTIONAL.** Round 1 does not put a model anywhere in the observation path by default. It builds the full deterministic pipeline (rows 1–7) and the *bounded stub* of the single diagnostic-probe call (§4, §9) that fires **only** when no adequate oracle exists. The deterministic path is the default; the L step is the exception. Container-grade isolation, a trained probe generator, the LEVER-style learned verifier, AlphaCode-style behavioural clustering, rich `changed_state`, and the multi-language parsers are all candidates for later rounds precisely because each would add a container runtime, a *second* (or fully trained) model to the pipeline, break the one-entry L column, replace a deterministic stage that the ablation (§7) may already prove sufficient, or add a parser surface the Python toolchain does not yet demand. Any later-round candidate must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics (§7), on held-out repositories, at a defensible cost, or be rejected. The cost of the deterministic choice is paid once, in the schemas (§3); the cost of a misplaced learned call is paid every candidate, in L-column violations and trust placed in a generator that is not authoritative. --- ## 3. The Deterministic Pipeline (D) Seven stages, all **D** through the emission of the assembled `ExecutionObservation`. The LLM column holds exactly one thing: diagnostic-probe generation (§4), fired only when no adequate oracle exists. The pipeline, from candidate artifact to structured observation: ``` CANDIDATE ARTIFACT (a patch, a snippet, a command) + ENVIRONMENT DECLARATION + EXISTING TESTS → (a) action classification [D] → one of the four execution modes → (b) isolation policy [D] → the declared boundary (caps, FS, network, timeout) → (c) execute [D] → raw artifacts (stdout, stderr, exit, duration, exception) → (d) evidence parsing [D] → structured compile/runtime/test fields (per toolchain) → (§4) diagnostic-probe generation [L — the ONE entry, ONLY when no adequate oracle exists] → (e) normalization [D] → the failure-class vocabulary → (f) behavioural caching/dedup [D] → identical key → prior observation, no re-run → (g) structured observation assembly [D] → the single ExecutionObservation ``` Every execution runs against a **declared, reproducible boundary**; nothing about an execution reaches the model except through the assembled observation. The candidate is **run, never mentally simulated**: the model's prediction about what code does is a hypothesis; the runtime's report is a measurement, and the harness treats the former as provisional and the latter as load-bearing. All artifacts land in `build/11-code-execution/`. Nothing writes into the live harness. ### 3.1 (a) Action classification (D) **Input:** the incoming request — what the harness asks the runtime to establish about a candidate artifact (a patch, a snippet, a command), its declared environment, and the set of existing tests. **Output:** one of the **four execution modes**. **No language model runs here.** Classification is a deterministic table over the request type and the artifact shape — an inference is never required to decide what the runtime is being asked to witness (paper §4 stage a). The four modes: - **`compile_parse`** — does the artifact build at all? Syntax and type correctness *without* behavioural meaning. The runtime establishes that the code parses/compiles under the toolchain. - **`targeted_execution`** — run a specific entry point or snippet and observe its outputs and side effects. The runtime establishes what a named call actually does. - **`test_execution`** — run the existing suite or a subset and collect verdicts (the whole-suite action mode, ExecutionAgent's operation). The runtime establishes which tests pass. - **`diagnostic_probe`** — run a generated probe to expose behavioural differences among candidates or to establish a property the existing tests leave open. **This is the only mode that may involve a model**, via stage (§4); the mode itself is still classified deterministically (a probe request has the probe artifact shape), and execution of the probe is deterministic (paper §4 stage a). **Action-classification rule (exact — deterministic table over `request_type` × `artifact_shape`).** The `request_type` is one of `build_check | run_entry | run_suite | probe | discriminate`; the `artifact_shape` is one of `patch | snippet | command | test_subset | probe_program`. The first rule that matches wins: | request_type | artifact_shape | Action mode | |---|---|---| | `build_check` | any (patch, snippet, command) | `compile_parse` | | `run_entry` | `snippet` or `command` (a specific entry point) | `targeted_execution` | | `run_suite` | `test_subset` (a named suite/subset) | `test_execution` | | `probe` | `probe_program` (a generated probe) | `diagnostic_probe` | | `discriminate` | two+ candidates + an undecidable property | `diagnostic_probe` (a generated probe decides) — **the only path that arms (§4)** | | (any other combination) | — | `targeted_execution` (safe default: run the entry and observe) | **Output schema (exact):** ```json { "mode": "compile_parse | targeted_execution | test_execution | diagnostic_probe", "request_type": "build_check | run_entry | run_suite | probe | discriminate", "artifact": "str", // reference/id to the candidate artifact "probe_gated": bool } // true only when a model-generated probe may be needed (discriminate) ``` **Caps:** one mode per request, fixed deterministically before any execution. A `diagnostic_probe` mode with `probe_gated: true` merely *arms* the single L step (§4); it does not invoke it by default — the probe fires only if stage (b)/(c) confirm no adequate oracle exists. The classification is logged (mode + whether the L step was armed) for the §7 failure-class and probe instruments. ### 3.2 (b) Isolation policy (D) **Input:** the classified mode from (a). **Output:** the **declared boundary** under which the candidate will run — a reproducible, explicit contract so that the observed behavior is the candidate's own and not an artifact of a hostile, leaky, or runaway environment (P3). **No language model runs here.** Isolation is the precondition that makes the runtime's report trustworthy: if the execution environment can interfere with or be interfered by the host, the observation is not a clean measurement. The policy is **recorded in the observation**, so the reader of the evidence knows under what constraints it was produced — a timeout under a low CPU cap is evidence about the candidate, not about the environment (paper §4 stage b). **Isolation defaults (exact):** - **CPU cap:** **1.0 core** — the candidate runs under at most one CPU core of compute. - **RAM cap:** **512 MiB**, enforced as `RLIMIT_AS` (address space), so an allocation beyond the cap fails deterministically rather than degrading the host. - **Hard timeouts per mode:** `compile_parse` **60 s**, `targeted_execution` **30 s**, `test_execution` **120 s**, `diagnostic_probe` **30 s**. - **Filesystem boundary:** each execution runs with a **scratch root** (`tempfile.mkdtemp()`); the repository is mounted **read-only**; all writes are confined to the scratch root. A candidate that attempts a write outside scratch is blocked at the boundary. - **Network:** **DENY-BY-DEFAULT**. In **Round 1 the network is ALWAYS denied** for every execution — no execution may reach the network, regardless of mode. (The paper's "deny-by-default unless a specific execution explicitly opts in" is realized in Round 1 with no opt-in path at all; opt-in is later-round content and, when it arrives, must be explicit *and* logged.) - **Output caps:** `stdout` and `stderr` are each capped at **64 KiB**, with a **truncation marker** appended when the cap is hit (SWE-agent's principled output truncation, so a runaway dump cannot flood the observation; the "no output produced" sentinel is emitted when the stream is empty). **Output schema (exact):** ```json { "toolchain": "python3", // the declared interpreter/compiler/runner (pinned) "cpu_cap_cores": 1.0, "ram_cap_mib": 512, "timeout_s": <per-mode>, // 60 | 30 | 120 | 30 by mode "fs_boundary": { "scratch_root": "<mkdtemp>", "repo_ro": true }, "network": "deny_by_default", // Round 1: always denied "stdout_cap_kib": 64, "stderr_cap_kib": 64 } ``` **Caps:** the policy is a fixed, deterministic declaration per mode — no free-form tuning at execution time. The boundary is immutable for the run it declares; a candidate that would need looser bounds is declined, never silently re-bounded. ### 3.3 (c) Execute (D) **Input:** the candidate artifact + the declared isolation policy (b). **Output:** the **raw artifacts** — `stdout`, `stderr`, `exit_code`, `wall_clock_duration`, and any raised `exception` — captured **without interpretation**. **No language model runs here.** This stage is deliberately bare: it spawns the process, enforces the resource and time bounds, and captures what the runtime returned. It does not parse, classify, or judge (paper §4 stage c). **Procedure (exact, deterministic):** 1. **Spawn** the candidate under the declared boundary (b): the process is launched with the resource limits applied (`RLIMIT_CPU`, `RLIMIT_AS` = 512 MiB), the filesystem boundary in force (repo read-only, scratch writable, cwd set to the declared `cwd`), and the network denied. 2. **Enforce bounds:** the hard per-mode timeout and the CPU/RAM caps are enforced *by the harness*, not trusted to the candidate. If the process is still running at the timeout, or trips a resource cap, the harness **kills it deterministically** and records the kill as the outcome — never a hang, never a silent escape from the boundary. 3. **Capture raw artifacts:** record `stdout` and `stderr` (each capped at 64 KiB with the truncation marker), the `exit_code`, the wall-clock `duration`, and any raised `exception`. 4. **No interpretation:** the raw artifacts pass to (d) unjudged. If the timeout or a resource cap fired, that is recorded as the outcome — the evidence reflects the kill, not a hang. **Caps:** capture is byte-faithful up to the per-stream caps (64 KiB each); a resource-cap or timeout kill is recorded with the mode's timeout and the relevant cap in the observation, so the reader can tell a boundary death from a candidate failure. ### 3.4 (d) Evidence parsing (D) **Input:** the raw artifacts from (c). **Output:** the raw runtime behavior parsed into **structured fields** — compile errors extracted with their locations and messages; runtime evidence (exceptions, stack frames, printed output) captured; test verdicts collected from the runner's structured or convention-based output. **No language model runs here.** Parsing is **deterministic per toolchain** — each language and runner has a *declared* parser, so that the same raw output always yields the same structured evidence (paper §4 stage d). **Round 1 parses the Python toolchain only** (§2 row 15 defers C/C++/Rust/SQL/Java). The Python toolchain parser is deterministic and convention-based: - **`compile_parse` mode:** parse the interpreter/compiler output for syntax and compile errors — extract each error's `file:line`, message text, and error class (`SyntaxError`, `IndentationError`, `NameError` at compile time, `ImportError`/`ModuleNotFoundError`, etc.). - **`targeted_execution` mode:** capture a raised exception's type and message and the stack frames (the *traceback as the reasoning prompt*, Self-Debug [1]), plus any structured printed output. - **`test_execution` mode:** parse the runner's output (`pytest`/`unittest` convention) for per-test verdicts — passed, failed, errored, skipped — and the assertion message and location of each failure. **Procedure (exact, deterministic):** the parser for the declared toolchain consumes the raw artifacts from (c) and emits structured evidence records; an output that does not match the toolchain's declared conventions is passed through as raw `runtime_evidence` with a `parse_uncertain: true` flag rather than guessed at. Parsing never invokes a model, and an unparseable transcript is flagged, never fabricated into a confident parse. **Caps:** the parsed fields are bounded (per-error message ≤ 4 KiB, stack depth ≤ the harness's configured frame cap) so a deep traceback cannot flood the observation. Parsing determinism is assumed by (g)'s determinism check. ### 3.5 (e) Normalization (D) **Input:** the parsed evidence from (d). **Output:** the evidence classified into the **common failure-class vocabulary** — the uniform, repair-grade form the model and the harness reason over (P4). **No language model runs here.** Raw runtime failure is noisy — every tool and language reports errors differently — so normalization makes the species explicit: a compile error and a timeout are different species with different remedies, and a weak model reasons over the *normalized* failure, so the normalization must preserve everything needed to localize and repair while stripping the noise that would distract (Self-Debug's traceback-as-reasoning-prompt warrant [1]). **Failure-class vocabulary (exact — the closed set):** ``` compile | syntax | assertion | runtime | timeout | resource | type | dependency ``` **Normalized-failure schema (exact):** each classified failure carries: - **`failure_class`** — one of the closed set above; - **`location`** — `file:line` or symbol (the entity the failure points at); - **`message`** — the human-and-model-readable error text (bounded); - **`exit_code`** — the process exit code; - **`duration`** — the wall-clock duration of the run. **Classification rule (exact, deterministic, first true wins):** 1. `timeout` — the process was killed by the per-mode timeout (recorded by (c)). 2. `resource` — the process was killed by a CPU/RAM cap (recorded by (c)). 3. `syntax` — a `SyntaxError`/`IndentationError` (Python) or a parse-stage compiler rejection. 4. `compile` — a compile-stage error that is not a pure syntax error (Python: an error raised at import/bytecode-compile time, e.g. a top-level `NameError`, undefined attribute at class scope). 5. `type` — a `TypeError` raised at runtime. 6. `dependency` — an `ImportError`/`ModuleNotFoundError`/missing-package error. 7. `assertion` — an `AssertionError` or a failed `pytest`/`unittest` assertion (a test verdict of `failed`). 8. `runtime` — any other exception (the residual class). **Output schema (exact):** ```json { "failure_class": "compile|syntax|assertion|runtime|timeout|resource|type|dependency", "location": "file:line | symbol", "message": "…", // bounded "exit_code": int, "duration": float } ``` **Caps:** one primary `failure_class` per observation (the first-true classification wins; secondary failures are retained as structured detail but do not change the class). An empty or `exit_code == 0` run with zero tests is **not** normalized to a failure class here — it is passed to (g) where the empty-success guard flags it. ### 3.6 (f) Behavioural caching/dedup (D) **Input:** the candidate code + its declared environment (b) + the command, all prior to (c) running the candidate. **Output:** **no re-run** when an identical execution was already observed. **No language model runs here.** This is P2's determinism exploited for economy: identical code under identical constraints yields identical evidence, so re-running it would burn wall-clock time to rediscover a known fact (paper §4 stage f). AlphaCode's clustering intuition applies here in miniature — near-identical candidates need not each pay a full execution (but the paper's explicit caveat stands: AlphaCode does **not** describe its mechanism as "result caching," so that vocabulary is not attributed to it). **Cache key (exact):** ``` key = SHA-256( code || environment || command ) ``` where `environment` is the declared isolation policy (b) — toolchain, CPU/RAM caps, filesystem boundary, network policy, timeout — and `command` is the invocation. The key is computed from the declared inputs **before** execution, so an identical declared run is recognized without spawning. **Procedure (exact):** on every incoming request, after (a) and (b), compute the cache key. If the key was already observed under the same declared boundaries, **return the prior `ExecutionObservation`** (by reference to its id) and do **not** re-run. Otherwise execute (c)–(g) and store the assembled observation under the key. The cache is scoped to the declared environment — a change to any element of the key (a pinned dependency, a different cap, a different command) is a different key and a real run. **Caps:** a cache hit returns the prior observation's identity and verdict verbatim (never a stale or partial result); the cache itself is bounded (LRU to a build constant) so it cannot grow unbounded across a run. Dedup and cache both key on declared identity, never on textual similarity alone. ### 3.7 (g) Structured observation assembly (D) **Input:** everything above — the mode (a), the declared environment (b), the raw + parsed + normalized evidence (c–e), the cache verdict (f), and the probe evidence if the L step fired (§4). **Output:** the single **`ExecutionObservation`** — the schema this component's consumers read. **No language model runs here.** The observation is the single currency of this component: diagnosis and discrimination stages consume it, and nothing about an execution reaches the model except through it (paper §4 stage g). **The `ExecutionObservation` schema (exact — EXACTLY these fields):** ```json { "action": "compile_parse | targeted_execution | test_execution | diagnostic_probe", "environment": { "toolchain": "…", "cpu_cap_cores": 1.0, "ram_cap_mib": 512, "fs_boundary": {"scratch_root": "…", "repo_ro": true}, "network": "deny_by_default", "timeout_s": <per-mode> }, "cwd": "…", "artifact_version": "<SHA-256 of the code that ran>", "exit_code": int, "duration": float, // wall-clock seconds "timed_out": bool, "stdout": "…", // ≤ 64 KiB, truncation marker on cap "stderr": "…", // ≤ 64 KiB, truncation marker on cap "exception": "…", // present iff one was raised "changed_state": "…", // declared-changed-state summary (rich diff DEFERRED) "test_results": [ { "test": "…", "verdict": "passed|failed|errored|skipped", "f2p": bool, "p2p": bool } ], // FAIL_TO_PASS / PASS_TO_PASS "resource_usage": { "peak_cpu": 0..1, "peak_ram_mib": int } } ``` **Field meanings (exact):** `action` = the classified mode; `environment` = the declared isolation policy (b), recorded so the reader knows under what constraints the evidence was produced; `cwd` = the working directory; `artifact_version` = **the SHA-256 hash of the code that ran**, so every observation is traceable to exactly what produced it (provenance, §8 invariant 8); `exit_code`, `duration`, and `timed_out` = the raw outcome; `stdout`/`stderr` = bounded, elided on cap (SWE-agent hygiene, P4); `exception` = the raised exception if one occurred; `changed_state` = a declared summary of filesystem/persistent effects (the rich diff is DEFERRED, §2 row 14); `test_results` = per-test verdicts, each tagged `f2p` (FAIL_TO_PASS — failing before the change, passing after: the true signal of repair) and `p2p` (PASS_TO_PASS — previously passing and still passing: the guard against regression); `resource_usage` = peak CPU and RAM. **Assembly guards (exact, run at assembly, all D):** 1. **Empty-success guard:** an `exit_code == 0` run whose `test_results` is **empty** (or whose claimed coverage is nil) is **flagged** — `flagged_empty_success: true` — and is **never counted as evidence of correctness**. An execution that ran no tests proves the program ran, not that it is right. 2. **Determinism check:** because P2 makes determinism a precondition, the assembled observation is checked for reproducibility — dependencies pinned, tests seeded, identical (code, environment, command) must yield an identical observation. A run that cannot be reproduced from its own `artifact_version` + `environment` is **flagged**; non-reproducible evidence cannot be the basis of repair or of a recursive-improvement decision. 3. **Anti-gaming re-verification:** the harness **never trusts a self-report, including its own**. When a candidate or stage reports "all tests pass," the harness independently **re-runs the reported-passing tests and confirms**, and applies a **diff-scope check** that the test files themselves were **not edited** as part of the candidate's change. A pass achieved by tampering with the oracle is not a pass at all (ImpossibleBench's anti-gaming warrant [12]). **Caps:** exactly one `ExecutionObservation` per execution; `stdout`/`stderr` each ≤ 64 KiB with the truncation marker; `test_results` bounded to the executed suite; the observation is immutable once assembled (a later change to the candidate is a new `artifact_version` and a new observation). A regression in what an execution records is visible in the assembled observation long before it surfaces as a failed resolve (paper §5). **The L column holds exactly one entry.** Stages (a)–(g) are all deterministic software on CPU. The single learned node is stage (§4), diagnostic-probe generation, which fires only when no adequate oracle exists and is never auto-promoted. Any proposed addition that would put a language model elsewhere in the pipeline — a model that classifies actions, a model that decides isolation bounds, a model that parses raw output, a model that normalizes failures, a model that ranks cached observations — is a regression against P1–P7 and the L-column invariant, and must clear the same bar as in earlier components: beat the deterministic stage on the honest metrics of §7, on held-out repositories, at a defensible cost, or be rejected (paper §4). --- ## 4. The LLM Step (L, ONE entry) The **single** learned node in the entire stage: **diagnostic-probe generation**. The weak model proposes **bounded diagnostic probes** — small programs or assertions designed to expose behavioural differences among candidates or to establish a property the existing tests leave open — only when no adequate oracle exists to judge a candidate, at temperature zero, in one call (paper §4 stage L). This is the sole entry in the L column; every upstream and downstream step is deterministic software on CPU (paper §4). It is CodeT's code-and-test insight [15] applied on demand. **It fires ONLY when no adequate oracle exists** — the candidate has no test suite, or the existing tests do not distinguish the behaviour the harness cares about. The default is deterministic: no probe is generated when an adequate oracle is present, and a probe is **executed and recorded as probe evidence** — it is **never auto-accepted as an authoritative test**. Authority — what gates a candidate — is reserved for the existing test suite (ImpossibleBench's anti-gaming lesson [12]; §8 invariant 7). **Firing condition (exact):** the step fires **iff** stage (a) classified the request `diagnostic_probe` with `probe_gated: true` **and** the harness confirms **no adequate oracle exists** — deterministically: the candidate has no reachable test suite, or the existing tests demonstrably do not distinguish the behaviour at issue (e.g. all candidates already pass, or none do, so the suite cannot discriminate). In every other case **no probe call is made**; the deterministic path stands alone. The step never fires to parse, to normalize, to assemble, to verify, or to touch an authoritative test. **Prompt inputs (assembled deterministically, in this exact order):** 1. **`candidates`** — the candidate(s) under discrimination, as signatures/descriptions (bounded, never full unbounded bodies when signatures suffice). 2. **`property`** — the behavioural property the existing tests leave open (the question the probe must settle). 3. **`existing_tests`** — the authoritative tests already present (so the probe is novel and does not duplicate the oracle), in brief. **Input → output schema (exact):** ```json { "probes": [ { "probe_id": "P1", "program": "…", "expected": "…" } ] } ``` **Probe-record schema (exact — the model must emit nothing outside it):** ```json { "probe_id": str, // unique within the firing "program": str, // a small program/assertion (bounded, see caps) "expected": str } // the behavior the probe is designed to expose/establish ``` **Caps:** **ONE call**, `temperature = 0`, `max_completion_tokens = 2048`; **probe count capped at 8** per firing; each probe is **small** — a single self-contained program or assertion that fits the bound below. The output is schema-constrained; the harness, not the model, decides what a probe means. `temperature = 0` makes the single call deterministic in practice so probe proposals are comparable. **Gating (exact):** a generated probe is **executed** under the isolation policy (b) and its result is recorded as **`probe evidence`** in the observation — a distinct, labeled category. A probe is **never auto-accepted as an authoritative test**; it cannot gate a candidate, cannot mark a candidate as resolved, and cannot enter the `test_results` authoritative set. Its verdict is held loosely as a hypothesis about behaviour, exactly the looseness ImpossibleBench's anti-gaming construction requires. The two categories — authoritative tests and generated probes — are kept structurally distinct (§8 invariant 7), decided by the harness, never by the model. **Parse-failure fallback (exact):** any JSON parse failure, schema violation, out-of-bound probe (a probe over the size cap, more than 8 probes), or empty/malformed output is a **no-op**: the stage generates **no probe** and hands control back to the deterministic path — **never a re-prompt, never a second call**. The L column is reached **once** per firing; a failed parse is not retried with a "please try again" call. This is the mechanical enforcement of the one-entry L column and the anti-over-prompting rule. **Re-invocation on a later candidate (exact):** probe generation is re-considered **only** when a *new* candidate under discrimination again lacks an adequate oracle; each invocation is scoped strictly to that candidate and re-uses none of a prior firing's probes (a cached identical candidate, per (f), returns its prior observation instead). It remains the **single conceptual learned entry** — a bounded, once-per-inadequate-oracle call — **never a model that runs over the whole pipeline** and never a loop that re-derives what the deterministic stages already settled (paper §4). **Invariant 1 (§8): the L column contains exactly one entry — and that entry is only ever a bounded, gated, non-authoritative probe generator.** --- ## 5. Implementation Order Eight phases, each independently testable and reversible. Each lands in `build/11-code-execution/` as a standalone importable module. Rollback = delete/flag-off that phase's module; nothing earlier breaks because phases only add modules and never mutate live harness code. | Phase | Delivers | Isolated test | Rollback | |---|---|---|---| | **1. Observation schema** | `observation.py` (§3.7 the exact `ExecutionObservation` schema + field meanings + the empty-success flag + determinism + anti-gaming guard hooks) | Feed a fixture of mode/environment/cwd/exit/duration/stdout/stderr/exception/changed_state/test_results/resource_usage: the assembled object has **exactly** the schema fields and no others; `stdout`/`stderr` ≤ 64 KiB with the truncation marker; `test_results` verdicts carry `f2p`/`p2p`; assert **no LLM consulted** (probe the LLM client to raise). | Delete `observation.py`. | | **2. Action classification** | `classify.py` (§3.1 the request_type × artifact_shape table → the four modes + `probe_gated` flag) | Feed request/artifact fixtures: a `build_check` on any artifact → `compile_parse`; a `run_entry` on a snippet/command → `targeted_execution`; a `run_suite` on a `test_subset` → `test_execution`; a `probe`/`discriminate` on a probe program/two+ candidates → `diagnostic_probe` with `probe_gated` set only on `discriminate`; an out-of-table pair defaults to `targeted_execution`; assert **no LLM consulted**. | Delete `classify.py`. | | **3. Isolation policy** | `isolate.py` (§3.2 CPU cap 1.0 core, RAM cap 512 MiB `RLIMIT_AS`, per-mode timeouts 60/30/120/30 s, scratch-root FS boundary with repo read-only, network deny-by-default/always-denied in Round 1, 64 KiB stream caps) | Assert the policy object is correct per mode (compile/parse 60 s, targeted 30 s, test 120 s, probe 30 s); a spawned test process cannot write outside scratch and cannot reach the network (attempt both, assert both are blocked); `RLIMIT_AS` = 512 MiB is applied; assert **no LLM consulted**. | Delete `isolate.py`. | | **4. Execute** | `execute.py` (§3.3 spawn + enforce bounds + capture raw artifacts; deterministic kill on timeout/resource-cap) | Feed a candidate under each mode: raw `stdout`/`stderr`/`exit_code`/`duration`/`exception` are captured; a sleeping candidate is killed at the per-mode timeout and `timed_out` records the kill (no hang); a candidate over the RAM cap trips the cap and is killed deterministically; a candidate that runs to completion is captured cleanly; assert **no LLM consulted**. | Delete `execute.py`. | | **5. Evidence parsing** | `parse.py` (§3.4 deterministic per-toolchain parser — Python toolchain first: compile errors with `file:line`, runtime exceptions + stack, `pytest`/`unittest` per-test verdicts) | Feed raw Python outputs: a `SyntaxError` parses to its line and class; a runtime traceback parses to exception type/message/frames; a `pytest` run parses to per-test verdicts with assertion location; an output that does not match the declared conventions → `parse_uncertain: true`, never fabricated; assert determinism (same raw output → same structured evidence) and **no LLM consulted**. | Delete `parse.py`. | | **6. Normalization** | `normalize.py` (§3.5 classify into the failure-class vocabulary {compile, syntax, assertion, runtime, timeout, resource, type, dependency} with location + message + exit code + duration) | Feed parsed failures: a timeout-kill → `timeout`; a resource-kill → `resource`; `SyntaxError` → `syntax`; a compile-stage rejection → `compile`; `TypeError` → `type`; `ImportError` → `dependency`; a failed assertion → `assertion`; any other exception → `runtime`; first-true wins; an exit-0 zero-test run is passed to the empty-success guard, not mis-normalized to a failure class; assert **no LLM consulted**. | Delete `normalize.py`. | | **7. Behavioural caching/dedup** | `cache.py` (§3.6 key = SHA-256(code ‖ environment ‖ command); identical key → prior observation, no re-run) | Feed an identical (code, environment, command) twice: the second is a cache hit returning the prior observation **without re-running** (assert the execute stage is not invoked); a change to any key element (a pinned dep, a different cap, a different command) is a different key and a real run; assert **no LLM consulted**. | Delete `cache.py`. | | **8. Assembly + guards + probe stub + A/B harness** | `assemble.py` (§3.7 the `ExecutionObservation` + empty-success guard + determinism check + anti-gaming re-verification), `probe.py` (§4 the bounded L stub: firing condition + ONE call + probe cap 8 + gating + parse-failure fallback), `ab.py` (§7 feature-flag router + observation-fidelity capture + failure-class precision/recall + ablation runner + no-probe/probe runner + decision) | `assemble.py` end-to-end on fixtures: the empty-success guard flags exit-0 zero-test runs; the determinism check flags a non-reproducible run; anti-gaming re-runs "all pass" and diff-checks the test files. `probe.py` with a mocked LLM: fires **only** when (a) gated `diagnostic_probe` and no adequate oracle; the call is **one**, temperature 0, probes ≤ 8 and small; probes are recorded as `probe evidence` and **never** enter authoritative `test_results`; a malformed/over-cap response → no probe, **no second call**. `ab.py` runs the §7 pipeline-vs-baseline and the §7 ablation, emitting the §7 decision. | Feature flag `off` → live harness untouched; `ab.py` is read-only. | The A/B runner (Phase 8) is the only phase that ever *invokes* the current harness — and only to measure it, never to modify it. --- ## 6. Dependencies & Supply-Chain Per Piper's rule: nothing is installed unless >72h old AND (≥10 stars OR deep-scrutinized). This build is **zero-install for Round 1 (Python-only, stdlib + in-repo)**: | Dependency | Why | Status / age-stars | Install needed? | |---|---|---|---| | stdlib `subprocess` | §3.3 spawn + enforce bounds + capture raw artifacts | CPython stdlib. | **No** | | stdlib `resource` | §3.2 CPU cap + RAM cap (`RLIMIT_AS` = 512 MiB) | CPython stdlib. | **No** | | stdlib `tempfile` | §3.2 scratch root (`tempfile.mkdtemp()`) filesystem boundary | CPython stdlib. | **No** | | stdlib `hashlib`, `os`, `time`, `signal`, `json`, `re`, `sys` | §3.6 cache key, §3.2 spawn/limits/kill, §3.7 schema, §3.4 parsing | CPython stdlib. | **No** | | Python toolchain (`python3`, pinned) | §3.2 declared toolchain; §3.4 the Python evidence parser; the candidate runtime | In-repo, pinned interpreter. | **No** | | Component 01 tool layer / existing test runner (`pytest`/`unittest`) | §3.4 test-verdict parsing (test-execution mode), §3.7 authoritative `test_results` | In-repo, already vetted. | **No** | | Component 10 candidate change (the patch landed on disk) | §3.1 the candidate artifact the observation path witnesses | In-repo, built by spec 10. | **No** | | Harness weak model (`deepseek-v4-pro`) | §4 the single diagnostic-probe call (bounded stub in Round 1) | The harness's own model, already vetted. | **No** | **Preferred path: zero-install.** Round 1 is Python-only and deterministic: the observation path uses `subprocess` + `resource` + `tempfile` to spawn, bound, isolate, and capture; the Python toolchain parser is convention-based over stdlib/`pytest` output; the candidate artifact comes from component 10. The single LLM call (§4) goes to the harness's own weak model and is shipped as a bounded stub. **No new package is installed for Round 1, and nothing is flagged for a later install** — the six DEFERRED rows (§2 rows 10–15: container-grade gVisor/Docker isolation, the trained probe generator, the LEVER-style learned verifier, AlphaCode-style behavioural clustering, rich `changed_state` diff, and the multi-language parsers) are container runtimes, training/weights on known architectures, or parser surface on existing interpreters — not new third-party Python installs. Any future container runtime, model download, RL-training run, GPU dependency, or multi-language parser they imply must clear the **72h-age + star/scrutiny + sandbox-test** gate before use — a container runtime or learned verifier is ops-critical and gets deep scrutiny regardless of stars — and is out of Round 1 scope entirely. --- ## 7. Acceptance Criteria — the A/B Protocol **Sample (concrete).** Reuse the **same 20 held-out SWE-bench-Verified instances as spec 01** (Python-only, single-repo, no-Docker reproduction) as the source of a held-out **(candidate, environment, expected outcome) triple** corpus: for each instance, the candidates the pipeline is asked to witness (the landed patch, its failing and passing tests, targeted snippets), each with a declared environment and a mechanically-known expected outcome. Held out = never seen by any tuning, prompt, or generation pass. Seeded RNG. **Fallback:** `coding-cortex/piperbench` sampled to 20 tasks, held out identically. **Baseline (concrete).** The **naive `subprocess.run` + raw-stdout** baseline — the paper's §5 unstructured terminal: each candidate is run with a bare `subprocess.run`, and the model/harness consumes the raw, untruncated, unparsed, unnormalized transcript with no isolation declaration, no failure-class vocabulary, no empty-success guard, no determinism check, no anti-gaming re-verification. The deterministic pipeline must beat *this* on the honest observation-fidelity metrics below, and the ablation (7.5) removes pipeline mechanisms against those same honest metrics. ### 7.1 (a) Observation fidelity — the KEY measure The key code-execution instrument, and the direct score of stages (c)–(g): on the held-out triple corpus, the pipeline's **verdict correctness** — how often the observation's verdict (pass/fail/ specific failure class) matches the mechanically-known expected outcome — **AND** its **failure-class classification accuracy** — precision and recall for the **8 failure classes** ({compile, syntax, assertion, runtime, timeout, resource, type, dependency}) against the expected outcome. This is the instrument that makes P1–P7 falsifiable: if the pipeline's structured observation does not witness reality more faithfully than a raw transcript, the normalization, parsing, and isolation stages are not earning their place, regardless of how elegant the schema reads. **This is the KEY measure — the pipeline is adopted only if it STRICTLY beats the raw baseline on verdict accuracy AND on failure-class F1; a resolve-rate or discrimination win that does not beat the baseline on observation fidelity is not credited.** ### 7.2 (b) FAIL_TO_PASS / PASS_TO_PASS as the execution contract The standard execution contract inherited from HumanEval/Codex pass@k and SWE-bench: a candidate's score is how many test cases its **actual execution** passes, where FAIL_TO_PASS (`f2p`) counts tests that were failing before a change and pass after it (the true signal of repair) and PASS_TO_PASS (`p2p`) counts previously-passing tests that still pass (the guard against regression). These are mechanical, reported from the assembled `test_results`, and they are the honest currency of both diagnosis and discrimination: they say what the runtime witnessed, not what the model or harness claims. **The empty-success guard (exit 0 + zero tests = flagged) means an empty `test_results` is never scored as correctness.** A candidate whose pass is an empty-success is not a pass at all. ### 7.3 (c) End-to-end regression guard The coarse instrument kept for parity with the program's other components — **deliberately not** the gate for execution changes, because resolve rate cannot separate execution quality from repair quality (the paper's explicit stance: never the end-to-end resolve rate alone): 1. **Resolve rate** = `resolved / total` (spec 01's score). 2. **Cost** = `$ / instance` (model spend only; deterministic stages are ~free). 3. **Tokens** = `tokens / instance`. ### 7.4 Decision rule (explicit thresholds) Run pipeline-vs-baseline on the *identical* triple corpus with the *identical* model (the harness's own weak model, `deepseek-v4-pro`) and hardware (this box). The structured pipeline **wins iff**: ``` Verdict accuracy STRICTLY beats the naive subprocess.run + raw-stdout baseline (the KEY measure) AND failure-class F1 STRICTLY beats the raw baseline AND resolve_pipeline ≥ resolve_baseline (no end-to-end regression) AND cost_pipeline ≤ 1.5 × cost_baseline AND tokens_pipeline ≤ 1.5 × tokens_baseline ``` Observation fidelity (verdict accuracy **and** failure-class F1) must **strictly beat the raw baseline**; resolve must not regress; cost and tokens are bounded at 1.5×. **Never the end-to-end resolve rate alone** — a candidate that improves resolve but leaves observation fidelity flat against the raw baseline is **not adopted** (execution quality would be masking repair quality). If the structured pipeline wins, it replaces the naive raw path; if it loses, the build tree is kept for the record and the live harness is unchanged. ### 7.5 Ablation rule (mechanical) The instrument that enforces "never add an execution mechanism because it sounds sensible" mechanically (paper §5, §6). Two passes, everything else held fixed, the delta measured on the honest metrics (observation fidelity — verdict accuracy + failure-class F1 — plus resolve/cost/tokens): 1. **Remove normalization** — feed the raw transcript instead of the normalized failure-class observation — and measure the **downstream repair-accuracy drop**. This is the direct test of P4: does the failure-class vocabulary measurably improve the repair that consumes it? 2. **Remove each other mechanism in turn** — the isolation declaration, the empty-success guard, the anti-gaming re-verification, the behavioural cache. 3. **Ablate the single L probe step** — run **no-probe vs probe** (never generate a probe vs. generate one) and measure the downstream **selection success** on held-out repos. This is the test of whether the bounded probe stub (§9) earns its single learned entry at all. A mechanism is **DELETED** iff its removal does **not** degrade the honest metrics beyond noise (`Δ = 0`, or `|Δ| ≤ 5%` where continuous). A mechanism whose *addition* degrades the metrics is a demonstrated harm and is removed outright. Applied first to the weakest a-priori case (the single L probe stub — the one learned call, which must survive the no-probe-vs-probe ablation to be kept at all), then to any other mechanism — regardless of how sensible it looks. **Never add an execution mechanism because it sounds sensible.** ### 7.6 Anti-gaming probe (P7 load-bearing check) The instrument that verifies the anti-gaming re-verification and the probe/authority separation actually hold against the ways the paper says an execution layer can be gamed (ImpossibleBench [12], paper §6): 1. **Test-tampering:** a candidate whose change edits a test file (so a previously-failing test now passes because the oracle was mutated) must be **caught by the diff-scope check** and its pass rejected — never accepted as a green pass. 2. **Probe non-promotion:** a generated diagnostic probe (from the L stub) must never enter the authoritative `test_results`; a run that reports "all pass" on authoritative tests only must re-run and confirm before the pass is believed. If either probe case is accepted as a green pass, the anti-gaming re-verification or the probe/authority separation fails the probe and the mechanism is re-examined against 7.4. ### 7.7 Two honest caveats Two caveats close the section, inherited from the paper's §6. **First, the strong-model-and-scale caveat:** the headline numbers of the paper's review — Self-Debug's ~+12% with an oracle [1], PAL's 15%-absolute trust-transfer margin on GSM8K [6], InterCode's 9.1%→73.7% GPT-4 SQL doubling [2], LEVER's 4.6–10.9% over code-davinci-002 [8], AlphaCode's 34.2% at ~1M samples [5] — were obtained with models far stronger than this harness's target and, in AlphaCode's case, at a scale a weak generator is not yet producing; a weak-model harness should not expect to reproduce them. Its execution stage exists precisely so that weak *generation* is compensated by deterministic *selection*, and the honest instruments here — above all observation fidelity, FAIL_TO_PASS/PASS_TO_PASS, and the anti-gaming re-verification — are what will show whether that compensation is real. **Second, the held-out and memorization caution** inherited from the earlier components: all execution metrics must be reported on held-out and out-of-benchmark repositories, so that a model that has memorized a repository's tests cannot score well without its code having actually executed. --- ## 8. Guardrails & Reversibility **Revocability.** - The build lives entirely in `build/11-code-execution/` (a separate tree); it never writes into the live harness's source. - A **feature flag** (env var `EXEC=exec11|raw`, default `raw`) routes the A/B; the flag alone switches between the new deterministic observation stage and the current naive `subprocess.run` raw path — no code deletion required to revert. - `ab.py` is a read-only driver: it *measures* the current harness, it never modifies it. Running old-vs-new is always possible in the same session, same repo, same model. **Invariants (must hold, checked at the end of every phase).** 1. **The L column contains exactly one entry** — diagnostic-probe generation (§4), fired only when no adequate oracle exists, bounded (probe cap 8, small programs), gated, executed as probe evidence and never auto-promoted, never re-prompted on parse failure. Any phase that introduces a language model elsewhere in the pipeline (action classification, isolation, execution, evidence parsing, normalization, behavioural caching, structured assembly) is rejected outright. An **empty** L column (the probe stub ablated away in §7.5) is a permitted, correct state — never a violation. 2. **No execution without declared isolation bounds.** Every candidate runs under the (b) policy — CPU cap, RAM cap, filesystem boundary, network policy, timeout — declared and recorded in the observation before the process is ever spawned. An execution with no declared boundary is a rejected deliverable. 3. **Network deny-by-default.** No execution may reach the network in Round 1. When opt-in arrives in a later round, it must be explicit and logged — a silent or default network path is a rejected deliverable. 4. **Determinism — same (code, environment, command) → same observation.** Dependencies are pinned, tests are seeded, and identical candidates under identical declared environments must produce identical observations (the §3.7 determinism check). A run that cannot be reproduced from its own `artifact_version` + `environment` is flagged; non-reproducible evidence cannot be the basis of repair or of a recursive-improvement decision. 5. **Empty success never counts.** An `exit_code == 0` run with an empty `test_results` (or nil claimed coverage) is flagged by the empty-success guard and is **never** scored as evidence of correctness. An execution that ran no tests proves the program ran, not that it is right. 6. **Never accept self-report.** The harness never trusts a self-report, including its own: when a candidate or stage reports "all tests pass," the harness independently **re-runs** the reported-passing tests and confirms, and applies a **diff-scope check** that the test files themselves were not edited as part of the candidate's change. A pass achieved by tampering with the oracle is not a pass at all. 7. **Authoritative tests and generated probes are distinct; probes are never auto-promoted.** The existing test suite is ground truth that gates a candidate and is itself protected from mutation; a generated diagnostic probe is a bounded hypothesis recorded as probe evidence and **never** promoted to authoritative status on the model's say-so. The harness decides what counts as authority, mechanically, never the model. 8. **Provenance — every observation carries `artifact_version` + `environment` + `cwd`, traceable to exactly what ran.** Each observation is versioned by the SHA-256 of the code that ran and is diffable, so a regression in what an execution records is visible in the assembled observation long before it surfaces as a failed resolve; the reader of any observation can reconstruct the exact candidate, under the exact declared boundary, that produced it. --- *End of spec. This document is the build contract; the next document records the A/B outcome.*