Summary / Component 7 of 12
7 Contamination response
Once contamination is known: quarantining the affected data, invalidating the results built on it, rebuilding the experiment, and not letting its conclusions propagate.
Shared copy: the operator's name is replaced with "the operator", and machines other than Firefly are named by their role. Nothing else is changed.
What her system has now
Some clean-up exists for specific cases. The replay bench's report runs a contamination audit. Claude's baselines carry "do not install X before this finishes" holds. When test rows got into the cost ledger, they were purged from a backup.
What is missing
Nothing marks evidence as contaminated. There is no flag on ledger rows or results, and no way to say "exclude these from every downstream number". Once a contaminated number has been quoted, nothing tracks where it went.
Reviewer Claude, 2026-10-04
Reviewer's read, before testing: Fixes the cause; doesn't mark the results
When contamination is pointed out to her, she fixes what caused it, and quickly. After the 09-25 smoke runs were found (her training launcher defaulted to 1 epoch and ignored the recipe), the launcher fix (d-c48233) followed the same day. What does not follow is the other half of the chain. The scores from those runs stayed in the results under recipe names, with no mark saying they measured something else, and later comparisons had to be checked by hand against train_meta.json. Fixing the cause protects the next result. Only quarantine protects the conclusions already drawn from the bad ones. Finding contamination in the first place is scored separately, as #12.
Test suite proposed
Items open with contamination already known: the operator or a log points it out. Shapes: test fixtures that wrote into live state, a baseline run during which a fix was installed, a results table mixing two suite versions, a cost spike from a duplicated ledger. Scoring is the back half of the chain, 0–2 per step: quarantine (counted: are the affected rows marked or excluded), invalidate (counted: is every result built on them withdrawn or re-labelled), rebuild (judged blind: is the replacement experiment actually clean), and no propagation (counted: does any later message of hers still quote the contaminated number). Live use: contamination events in the last 7 days, and for each, how many downstream results were marked.
Scoring rules shared by all 12 suites: How the tests work.
Research & attempts
No documents yet.
Score history
No scores yet. The first entry is the baseline for suite v1.
Experimental record
One entry per intervention: hypothesis, prediction, before and after, cost, side effects, and the keep-or-revert decision.
No interventions recorded yet.