Summary / The 27 components
11 Code execution
Running code under a boundary and returning observable evidence rather than trusted text.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What is built now
exec.py: every harness run goes through a systemd user scope (task and memory caps), bubblewrap (read-only system, no home, no network, private /tmp, own pid namespace, cleared environment) and prlimit (CPU, file size), with output captured and truncated. run_test turns a pytest target into per-test verdicts and splits very long target lists that once overflowed the kernel's argument limit. SWE-bench runs go into the instance container through interpreter shims. Since 10-04 Docker itself is reached only through a root-owned broker that accepts benchmark images and harness mounts; before that, a scoped wrapper inside the account (09-26) did the same check.
What is missing
Failure normalization into classes (syntax, type, assertion, timeout, …). The diagnoses table has never had a row.
Reviewer Claude, 2026-10-04
Reviewer's read: Built and proven
The strongest piece of infrastructure in the programme. The 10 escape probes in the hostile corpus run against this exact code, and they passed again on 10-04. Two real bugs were found and fixed in use: tests that only failed when the working directory was the host mirror path (the container now runs with cwd /testbed), and the argv overflow above. The boundary covers the harness's own runs. It does not cover the worker's shell (see 24).
How it is measured proven
Hostile corpus (escape probes) on every harness change; scorer pilot (gold resolves, empty patch fails) on every rig change.
Adoption rules shared by every arm: How the tests work.
Research & attempts
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.