24 Error detection

Update 2026-10-10 — live audit: the second check fired 14 times and none was a request to check anything; re-aimed, and the two-pass gate (96.7) is now the live form

What the live log showed. From 10-06 09:21 to 10-08, finding_check 1.0.0 fired on 14 real operator messages. It added "treat the faults above as unconfirmed" under 13 of her replies. The 14th it skipped, because the bare pass itself "found mistakes" in a forum post. Her verdict was not logged for that one.

I read all 14 turns. None of them asked her to check anything:

Why it misfired: two faults in the live wiring, not in the tested rule. 1. The trigger was length, not a request. Any message with a fenced block of 3+ lines, or 8+ pasted lines, counted as "something to check". Her replies then got scored as findings whenever they talked about an error. Example: her own answer key being wrong, in a forum reply. On the bench every item was a check request, so this never came up there. 2. It dropped the operator's own words. The step that strips the router header removed every line starting with [, and that includes the operator's first line ([terminal] sanity check this…). So the second pass never saw what had been asked. A smoke test caught this on 10-10.

The notes were mostly harmless noise, but one was wrong. On the web puzzles (10-06 09:49) she showed that the posted answer to the stolen-ring puzzle does not hold, and her counterexample is valid. The bare pass missed it, and the note told the operator to treat her correct finding as unconfirmed.

The fix (finding_check 1.1.0, live 10-10 00:22, no restart needed). - Asked first. One flash call decides whether the operator's words ask her to check the material for mistakes. If not, it logs the decision and stops.

The test set (trigger/trig24.py): - Requests: the 180 bench check prompts, with their answer-format line removed, plus 24 fresh informal ones written before the test ran. - Non-requests: the 37 real operator pastes, plus 132 sealed record-and-question prompts from #7 and #27 ("I pulled these records for a review…"). - All of them were shaped the way the router delivers them. - Two-pass self-agreement is the live form. This is the 96.7 arm, kept on 10-06 and never wired until now. When she reports a defect in material she was asked to check, one more pass of her own seat runs exactly as benched: blind stand-in tools, the 09-30 snapshot fork, the current header. Her finding stands only if that pass also reports a defect. If it does not, the note follows her reply. - Cost: about 45 s and $0.06, async, on her ledger. It runs only on asked turns where she reported a defect. - Fallback: if her pass fails, the bare flash pass (the 95.0 arm) is used instead. - Every asked turn now logs both verdicts, so agreement is countable. 1.0.0 left hers blank whenever the bare pass flagged. - Smoke test on Firefly: a real off-by-one, sent as a check request. She flagged it, her second pass flagged it, so they agreed and no note was sent. - Also fixed: flash reasons before it answers, so a small token cap returned an empty answer. The classifier caps went from 400 to 1,500 tokens, and the first trigger run's 15 empty answers went with them.

Scores. The summary now carries the live configuration: 96.7, the self-agreement arm of the preregistered H2 run (+11.7 [+5.0, +20.0] over her single pass, 85.0). Before, it carried 95.0, the second-opinion arm, which never ran live as benched. The figure is from the bench; there is no live measurement yet. Of the 37 operator pastes in her conversation record since 09-20, none was a check request, so the gate will fire rarely.

Control reds (live24, 45 readings, 10-06 06:42 → 10-09 23:33). 11 of the 14 checks that went red later went green, after 2 to 35 hours. Three were red at the last reading: handover.staleness, claims.dispatch_reconciled and memory_reach.bridge. The first two have flickered rather than stayed red. Readout 10-13 as planned, then the cron is removed.

• Files: live/finding_check.py 1.1.0, Firefly core/finding_check.py (backup .bak-claude-c24-20261010), state/finding_check.json ({"mode": "on", "second": "self"}), state/finding_check_seat/, trigger/trig24.py (results root-only), ev24d.py.

Update 2026-10-06 — preregistered test on a fresh held-out set: requiring two passes to agree cuts her false alarms from 15% to 3–5% with no lost detections (+10 to +12 points)

Why. Her losses on H1/D1 were false alarms, and they looked unstable: one pass objected to one line, the next pass to another line or to nothing, while real defects were flagged every time. Re-scoring the saved H1/D1 replies under an agreement rule suggested a large gain, but that was after the fact. So the rule was frozen first (PREREG-H2.txt, prereg24.py, hashes in PREREG-H2.sha256), then a fresh held-out set was generated (H2: 80 items, 50 planted, 30 clean) and run once.

Arms (one run of her seat, k = 2, plus one bare run; every arm scored from those replies): - Control: her first attempt, each pass on its own. - Retry: a turn whose reply is empty is re-run once. - Self-agreement: a finding counts only if both of her passes flag the item. - Second opinion (primary): a finding counts only if her pass and a bare deepseek-flash pass both flag it.

Reading it. - The gate works, and it is deterministic. No prompt changes; the same replies are filtered by a fixed rule. Detection was untouched: every planted defect was still found and located in every arm. - A second model is not needed. Her own two passes agreeing did as well as agreement with the bare model. That is the cheaper and simpler choice to wire: it reuses her own seat, at about twice the cost of one check (the H2 seat run was $2.49 for 160 turns). - Agreement beats both single checkers. Her single pass 85.0, bare 90.0, her two passes agreeing 96.7. Her false alarms are mostly noise, and noise does not repeat. - The empty-reply retry is still unproven. It never fired on H2. It stays a sensible guard, with no evidence for or against it yet.

Live since 2026-10-06 09:21 (router 1.9.105, operator's go). When the operator sends her something to check (a fenced block of 3 or more lines, or 8 or more pasted lines), finding_check.py runs one bare deepseek-flash pass over that message alone, on her key and her cost ledger. If it finds no defect, a classifier reads her reply. If her reply reported one, a single note follows the reply on Telegram and in the console: "second check: an independent pass over what you sent found no defect. Treat the faults above as unconfirmed." Her reply is never delayed or edited. Agreement, or no verdict from the second pass, adds nothing. - Why a classifier, not wording rules: on her 160 H2 replies, defect-word patterns caught 55 of 109 findings; the flash classifier caught 102 of 109 with 1 false positive. - The live form is the second-opinion arm (+10.0). Self-agreement scored slightly higher on the bench, but it would mean re-running her whole Day turn; a bare pass is a few seconds and a fraction of a cent. - Not covered: replies the reply vet held, and their rewrites. - Switch: state/finding_check.json mode on / observe / off, read on every call. Log: state/finding_check.jsonl.

• Files: PREREG-H2.txt, PREREG-H2.sha256, prereg24.py 1.0.0, H2-result.json, run24.py 1.4.0 (--retry-empty).

Update 2026-10-06 — benchmark built: she finds every planted error and names the right line; her losses are false alarms, and on the held-out set her full seat scores below the bare model

The bench (ERRDET-v1). Fifty items per set, ten in each of five families, six planted and four clean. Every planted defect is checked by machine when the set is generated: code mutants must fail executed tests (equivalent mutants are dropped), note contradictions and log summaries are re-derived, and every test file must compile. No judge model. Probes are blind: each item is worded as an operator request ("before I mark it green...", "before this goes to Sue...").

Score = 100 × (planted credit − false-alarm rate): half credit for flagging, half for naming a culprit unit. Answering "none" to everything, flagging everything and flagging the first unit all score 0.

Building it was itself an error-detection exercise. The bare model's and her pilot answers found nine flaws in my own clean items before the baseline. Examples: two services given the same port by accident, a test file with an unterminated string, a "correct" version comparer that treated 1.2 and 1.2.0 as different, a retry logged after progress 30/30, and logs dated the wrong night. In her pilot she caught the date and process-id flaws herself. All were fixed and both sets resealed before the scored runs. The change log is in SPEC.md.

Results (k = 2, sets frozen)

Paired, her seat minus the bare model: H1 −10.8 [−23.3, −1.7], D1 −5.0 [−17.5, +5.0].

Reading it. - Detection is solved at this difficulty. Neither arm missed a planted defect it answered, and both named the right line. The base model does this unaided. - Her seat adds false alarms. On clean logs she flags a line because the log doesn't prove something. Examples: "the single failure is never named", "nothing re-reads results.jsonl", "six cases in four seconds could be a resumed checkpoint". On clean code she reads past the docstring: a generator passed to a function documented for records, windows that cross midnight. Each objection is articulate and none is a defect. The bare model makes the code objections sometimes, the log ones almost never. The habit matches the live system: she treats "not shown" as "wrong". - Two empty replies (no text, no verdict) on held-out draft items. A seat turn that ends without an answer is its own failure mode. - This matches the live picture. The detectors are not the weak part, live or on the bench. The live weak part is what happens after a red, which the bench does not measure; the follow-through reading below does.

Live reading started. live24.py (every 2 hours on the bench machine) records which control-regression checks are red, so the time from red to green can be measured. Baseline 10-06 06:42: six reds. First readout due 10-13.

Not done. No intervention was tried; the gap is precision, and the obvious lever (an instruction to flag only what can be shown wrong) is a prompt directive, which the program avoids.

• Bench: ~/cognition-bench/24-error-detection (codebank24.py, gen24.py 1.1.2, score24.py 1.0.0, run24.py 1.3.0, rescore24.py, paired24.py, live24.py, SPEC.md, PROGRESS.md); sealed sets root-only on the bench machine.

Update 2026-10-06 — live re-check: the detectors fire, the contradiction backlog is cleared, the reds still sit

Read-only check of the live system, 2026-10-06 06:30.

Reading it. Detection is not the bottleneck: the checks are specific and they fire. What is missing is the step after: a red with an owner and a deadline. The artifact-read rule has a measured rate now (16 fires in 6 days) and could move to enforce on that evidence; that is the operator's call.