23 Uncertainty estimation

Update 2026-10-20 — corrected-key held-out reading: H2 control 54.3 replaces the mis-keyed H1 60.0 (different model since 17 Oct — not a before/after)

Corrected-key reading. The held-out set H1 carried a known suite defect: generator 1.0.2 modelled deploys as absorbing, so 11 of its 57 deploy items had a wrong ideal (on one of them her reasoning was right and the suite was wrong). The spare held-out set H2 was generated after the fix and sealed on 5 October; it had never been run. Piper asked who would correct it (her question q-203), and it was run on 20 October: same scorer, same clean snapshot, her live seat, k=2. H2 control: 54.3 (accuracy 0.860, mean p 0.923, Brier 0.103, AUROC 0.82). It replaces 60.0 as the reading; H1 stays on record with its caveat (57.3 without the 11 items). Most of the difference is in the inspection decisions (action 7.3 against 44.8 on H1): she paid for cheap irrelevant checks more often (33% against 8%), skipped expensive decisive ones more often (25% against 8%) and abstained less (7% against 17%). Calibration is about where it was (14.6 against 16.2).

Not a before/after. H2 ran on the model now behind deepseek-flash (changed by the provider on 17 October) and on her new machine; H1 ran on the old model and machine. The difference between the two numbers is the suite fix plus the model and machine change, not a change in her.

Update 2026-10-18 — shadow 'running long' mark built: live dispatches are marked when the worker acts past 70% of its budget (logged only, nothing acts)

What was built (the operator's go, 10-18 evening). This is the candidate the T1 KEEP allowed. A watcher outside her machine reads her live dispatches every 2 minutes and changes nothing there. It marks a dispatch when its worker takes a step after 70% of its time budget. That is the same rule T1 scored after the fact, now applied while the work runs.

• Shadow only. The marks are logged on the bench machine. Nothing on her side acts on them, and she has not been told. Her bridge was not edited, so no restart was needed.
• Checked before install: reading only the end of each trail gives the same flag as the full-trail scorer on the last 181 finished dispatches (181/181).
• What decides it. A report on 11-01 shows the mark rate, the fail rate for marked and unmarked dispatches, the share of failures caught, and how often the live mark matches the after-the-fact flag. The operator then picks arm (a real actuator such as a check-in at 70%, which needs the operator's go) or retire. With no decision by 11-08 it retires itself.

Update 2026-10-18 — T1 final: KEEP at 500 dispatches. A worker still acting after 70% of its time budget is 4.9x as likely to fail

PREREG-T1 verdict: KEEP (frozen 10-10 07:28, scorer score_t1.py, hashes checked before the run; pulled read-only from Firefly 10-18). The first 500 eligible dispatches after T0, 10-10 09:50 to 10-17 00:59, 42 failures:

The final read matches the 10-15 interim (n=432: RD +0.211, RR 4.9, catch 46%). The sample filled before the 7-day limit, so no fallback rule applied. Nothing was re-cut and the threshold did not change.

What KEEP licenses. Per the PREREG, the candidate is a shadow mark: the bridge logs "running long" when a worker crosses 70%, and nothing acts on it. Escalation, a check-in or a stop policy needs the operator's go. Nothing is installed yet. Piper has not been told about this test.

Parts. ~/cognition-bench/23-uncertainty/diff1/: PREREG-T1 + sha256, score_t1.py, res-t1-final.json.

Update 2026-10-15 — T1 interim: a worker still acting late in its time budget is 4.9x as likely to fail; all four keep conditions pass at 432 of 500 dispatches

PREREG-T1 (frozen 10-10 07:28, scorer score_t1.py, first 500 dispatches after T0): flag a dispatch when its worker is still acting after 70% of its worker_timeout_s. Read 10-15 at n = 432 (10-10 09:50 to 10-15 06:45), 37 failures:

This is the first signal on this page that tells, before the result comes back, which of her work is likely wrong. It is an interim read: the preregistered sample is 500, due by 10-17, and the verdict is taken there. Nothing is installed; if it holds, the next step is to show the flag to her seat when a worker crosses 70%.

Update 2026-10-07 — rolling recalibration of her own dispatch number, preregistered and tested live: it fixes the scale but only reaches the base rate, so it is not kept

Scored 10-07 02:11 and published 10-10. The result was finished on time, but the step that scored it only sent a message and never wrote to this site.

The test (PREREG-RC1, frozen 10-05 22:35 before any test outcome existed). Her declared dispatch probability ranked outcomes (dev AUROC 0.66) but was pitched far too low. The candidate fix is a one-parameter shift, p′ = sigmoid(logit(p) + b), with b refit for each dispatch. It is fitted only on her dispatches resolved before that one was created, within the last 7 days. She sees nothing and works the same way. The test set was the first 150 eligible live dispatches after the freeze (10-05 22:35 → 10-07 01:48), and she was not told about the test.

To be kept, all three rules had to pass: 1. The paired Brier difference's 95% CI lies below 0. 2. The Brier skill score is above 0 against the test pass rate, so it beats saying the pass rate every time. 3. Ranking (AUROC) loses no more than 0.05.

• Brier difference −0.081 [−0.104, −0.056]: rule 1 met.
• Brier skill −0.002: rule 2 failed.
• AUROC up by 0.01: rule 3 met.

Not kept. Nothing installed.

Reading it. - The scale fault is real and large. She said 0.63 on average when 91% of dispatches passed. The shift removes it and halves her Brier score. - But it lands exactly at the base rate. Once the scale is fixed, nothing is left: on these 150 dispatches her number barely separated passes from failures (AUROC 0.56, against 0.66 on dev). A calibrated version of a number that doesn't rank is "say 0.91 every time". That beats her raw number easily but adds nothing over the pass rate. - Robustness check: with the "verdict none" rows kept, the shift still helps (−0.038 [−0.052, −0.023]), and its skill is still below 0 (−0.088). - What this means for #23: her dispatch confidence has two faults, scale and ranking. Only ranking is worth fixing, and no tested second opinion improves it here (the base-model backtest ranked worse than her, 0.40). The next thing worth testing is a ranking signal from the task itself, such as task type or files touched, not another map over her number.

• Files: 23-uncertainty/recal/ (PREREG-RC1.txt + sha256, rc23.py, score_rc1.py, RESULT-RC1.md, result-rc1.json, ev23rc.py). Re-checked 10-10 from the saved rows: identical numbers, and each fit used only rows resolved before the dispatch it scored.

Update 2026-10-05 — the three next steps, done: her own dispatch number is measured again (live), the deploy model is fixed, and the second opinion was tried where she errs; neither second-opinion trial met its rule

The operator approved the three next steps the same evening.

1. Her own number is measured again (live). From delegate bridge 1.102.0, every recorded dispatch forecast gets a twin row (dispatchdecl:<task>) carrying her declared probability, resolved from the same harness verdict. Drives 1.16.0 reports it as its own set, her_declared, never pooled with the estimator's. 437 rows since 09-30 were backfilled from the declared number already stored with each forecast. First 7-day reading:

Her number ranks outcomes better than the estimator's; it is pitched far too low (about 0.22 under the pass rate), which is what makes its skill score negative. That is a scale fault, and it is now visible every day.

2. The second opinion, tried only where she errs. Both tests were preregistered before any outcome was looked at.

• Code predictions (PREREG-HC1, 80 fresh list/string items, the families where she erred on dev; an in-family test, k=2). With the card her wrong answers fell from 4 to 1 of 160, Brier 0.0245 → 0.0063, AUROC 0.49 → 0.73, at lower cost. The paired Brier difference is −0.018, 95% CI [−0.048, +0.006]: the upper bound is not below 0, so by its own rule it is not kept. On fresh items she was wrong only 2.5% of the time, so there was little left to fix.
• Dispatch outcomes (offline backtest, 250 dispatches from 10-01 on; the base model read her full task envelope three times). Not supported: the base model ranks dispatch outcomes worse than she does (AUROC 0.40 vs her 0.65; paired −0.26 [−0.40, −0.10]) and is far too pessimistic (mean 0.34 against a 0.89 pass rate). Blending lowers her AUROC (−0.17 [−0.28, −0.05]). A second opinion without the repository cannot judge her dispatches.

3. The deploy model is fixed (generator 1.1.0): deploys now come from a simulated build timeline, so a later deploy can bring back the build a newer note names, and versions are neutral build ids. Held-out set H2 is resealed with it; D1 and H1 stay as they were, since their runs are scored against them.

Where #23 stands. Nothing from the second-opinion line goes live. The useful finding is the reverse of the paper's expectation for dispatch: her own probability carries real ranking signal (0.65–0.66) that the system had stopped reading, and its fault is scale. The obvious next candidate is a recalibration of her number on her own recent outcomes, now that it is measured; Stage 0 showed a global map fails under drift, so it would have to be rolling, and it needs its own preregistered test.

Update 2026-10-05 — benchmark built; she weighs evidence close to the ideal and chooses inspections well, but her confidence barely separates her right answers from her wrong ones; two second-opinion upgrades tested on held-out, neither kept; nothing live

Built from the operator's paper ("uncertainty estimation"): UNCERTAINTY-v1, six tracks as the paper lays them out, scored with the paper's §11 weights. The weights, the 0–100 mappings and the ship rule were frozen in SPEC.md before the first run.

Step 0: her history (read-only). Her real probabilities, before any bench:

• Dispatch confidence ranks the wrong way. On 1,640 dispatches her stated probability has AUROC 0.435 (95% CI 0.40–0.46): she was more confident on the ones that failed. Since 09-25 it is at chance (0.49).
• A calibration map does not fix it. Fitted on the first half and tested on the second, an isotonic map made Brier worse (0.186 → 0.222), because the success rate moved from 0.63 to 0.78. Dispatch-time features add nothing (AUROC ≤ 0.56). The only thing that beats her number is a rolling base rate of recent outcomes.
• Her own number is no longer measured live. Since 09-30 the dispatch row's confidence is a fitted estimator's, and the calibration drive scores that. Her own figure survives only as declared_prob_acceptance, which nothing reads.
• The calibration gym looks perfect by design (BSS 0.69): its items are built to be easy to separate.

By the paper's decision tree (§31) this is the "no case-specific signal" branch: a global calibration map is out; evidence features or disagreement are next.

The suite. 170 items per set; dev set D1 and held-out sets H1/H2 drawn from different task families (§9.3).

Calibration is graded where the answer is real (A computed, D executed). The world tracks use a small two-state model of each system (a job, a flag, a deploy, a worker), stated to her in plain operational terms; its exact posterior is the ideal answer. Track E scores the inspection decision by expected regret, so one lucky reading does not move the score.

Baseline (dev set D1):

• On the evidence tracks she is close to the ideal, and much closer than the base model. With a summary and a fresh primary record in conflict, the ideal is 0.72; she said 0.79, the bare model 0.91. Stale memory gets no credit from her (gap 0.00).
• Her inspection choices are good: she reads the cheap decisive source, skips the expensive low-value one, and abstains when the source is down.
• Her loss is discrimination: four confident wrong code predictions (0.97–0.99) that the bare model got right. Her confidence is about equally high on the answers that turn out wrong.
• The base model is at ceiling on Tracks A and D (100%), as on #8 and #9: textbook reasoning is not the weakness.

Held-out (H1, her live configuration, k=2, preregistered): 60.0. Accuracy 0.882, mean confidence 0.924, Brier 0.087, AUROC 0.82, accuracy at p ≥ 0.9 = 0.938 (n = 258). Ideal ceiling 81.3.

Upgrades tried: the paper's U-E disagreement estimator. The same question goes 5 times to the base model with none of her context; agreement with her answer is the signal. On dev it looked strong: alone it was weak (AUROC 0.58), but averaged with her own number it lifted her AUROC from 0.754 to 0.867 [0.785, 0.933]. Both forms were preregistered (PREREG-H1.txt, hashed) and tested once on H1:

The dev gain came from her D1 code misses. On the held-out families she does not make them, the independent samples almost always agree with her, and the blend only pushes her confidence up. Most of the card's +5 is on Track E, where the card is not shown: run-to-run noise. Nothing was installed.

Faults found in my own suite, recorded rather than buried:

• E5 (source unavailable) did not tell her the source was down, so trying it was rational while the scorer charged it as waste. Fixed (gen 1.0.1) after the dev baseline, before any held-out run; the 6 items were re-run in every arm.
• One inspect result was ambiguous, which she rightly pointed out (D1-E23). Fixed (gen 1.0.2) before any held-out seat run, together with k=2 (an amendment to the preregistration, made before H1).
• The deploy family was modelled one-way (a version, once replaced, can never come back), so 11 of 57 held-out deploy items carry a wrong ideal. On one of them (H1-E12) her reasoning is right and the suite is wrong. Paired comparisons are unaffected (both arms face the same items). Without the 11 items: control 57.3, card 63.8. To be fixed before H2 is used.
• Source reliabilities are a design choice she is not told (memory note 0.80, summary 0.88). Where she treats a one-minute-old memory note as decisive (H1-E28), that is partly a disagreement with my numbers.

What would move this, by the evidence:

1. Measure her own number again. Score declared_prob_acceptance alongside the estimator, so her calibration is visible live.
2. Disagreement works only where she makes confident errors the base model doesn't, so it belongs on code predictions and dispatch outcomes, not on every answer. A narrow trial on dispatch commits would need a fresh preregistration.
3. Fix the deploy model, then use H2 for any further treatment.

Apparatus: ~/cognition-bench/23-uncertainty (SPEC, DIAGNOSIS, gen23 1.0.2, score23, run23 1.1.0, paired23 1.1.0, PREREG-H1). Sealed sets on the bench machine. Model spend ≈ $8 (bench account).