Summary / 23. Uncertainty estimation / Document
2026-10-10
Shared copy: the operator's name and machine names are replaced by roles. Nothing else is changed.
Rendering…
RC1 tells us something important: **Piper's confidence number contains very little information about which dispatches will fail.** Recalibration fixed the probabilities but couldn't create information that wasn't there. I'd stop asking Piper whether she thinks she'll succeed and start measuring properties of the task and her execution. Three experiments are worth considering, in this order. ## 1. Empirical difficulty prediction — learn which tasks Piper fails **My first choice. Lowest cost, easiest to test.** Instead of predicting success from Piper's declared confidence, predict it from the characteristics of the dispatch. Piper already records considerable operational information. Use it to build a task difficulty model. Potential features: | Feature | Hypothesis | |---|---| | Task category | Some task types are systematically harder | | Number of files affected | More files may increase failure risk | | Number of dependencies | More dependencies create additional failure opportunities | | Previous success on similar tasks | Historical performance predicts future performance | | Familiarity with affected code | Unfamiliar code increases uncertainty | | Availability of automated tests | Testable work may have higher verified success rates | | Number of required steps | Longer tasks present more opportunities for failure | Importantly, features must be available **before the prediction is made**. Train a simple regularized logistic regression or gradient-boosted model on historical dispatch outcomes. Compare it with Piper's existing number and a rolling base-rate predictor. **Research support:** Work on agent psychometrics has found that features extracted from coding tasks and their artifacts can predict agent performance on held-out tasks better than baseline predictions. That supports testing task-derived features, although it doesn't establish which features will work for Piper. [ResearchGate](https://www.researchgate.net/publication/403429582_Agent_psychometrics_Task-level_performance_prediction_in_agentic_coding_benchmarks?utm_source=chatgpt.com) **Benchmark:** Use chronological out-of-sample predictions on at least 300–500 new dispatches. My proposed experimental targets: - AUROC ≥ 0.65. - Positive Brier skill against a genuinely prospective rolling base-rate forecast. - Improvement over Piper's existing confidence ranking. - No outcome leakage through post-dispatch information. If it works, Piper will have an empirically grounded estimate of her success probability before attempting work. ## 2. Trajectory-based uncertainty — detect when a task is going wrong **Potentially more valuable than predicting success at dispatch time.** This changes the question. Instead of: *Will I successfully complete this task?* Ask: *Given what has happened so far, how likely am I to complete it successfully?* Consider a dispatch that begins with an estimated 90% success probability. During execution: 1. Piper attempts a modification. 2. A test fails. 3. Her first repair attempt fails. 4. She encounters an undocumented dependency. 5. She modifies an additional subsystem. Her chance of eventual success may have changed substantially. A trajectory predictor can examine: - Failed commands and tests. - Repeated unsuccessful repair attempts. - Unexpected dependency discovery. - Plan deviations. - Increasing scope. - Unresolved contradictions. - Lack of expected progress. This could be evaluated at fixed execution checkpoints, such as 25%, 50%, and 75% of the available budget. **Research support:** STeCa studies step-level trajectory evaluation for improving agent behavior. Although it is primarily an agent-learning method rather than a dispatch-success predictor, it provides relevant evidence that evaluating intermediate execution steps can reveal information unavailable at task initiation. [arXiv](https://arxiv.org/abs/2502.14276?utm_source=chatgpt.com) **Benchmark:** Predict final success from partial trajectories, using only information available at each checkpoint. Compare: - Original dispatch confidence. - Task difficulty model from experiment 1. - Task difficulty plus execution trajectory. Measure AUROC, Brier skill, and whether escalating at-risk tasks to a stronger model would improve outcomes. **This has a useful secondary application:** Piper could learn when to stop struggling and request additional resources, rather than spending money on a task that is deteriorating. ## 3. Prediction by controlled disagreement — use independent solutions This is the most experimental option. Rather than requesting another model's opinion of Piper's confidence, have an independent agent produce a solution or independently verify a critical intermediate result. Measure substantive disagreement. For example, Piper and an independent agent might disagree about: - The root cause of a bug. - Which files require modification. - Whether an assumption is supported. - Whether a proposed solution satisfies the requirements. The hypothesis is that **independent disagreement about consequential decisions may contain predictive information about failure**. But disagreement is not automatically uncertainty, and agreement is not automatically correctness. Research on semantic entropy supports measuring differences in meaning rather than superficial wording. More recent work also shows that model agreement can be a weak and unreliable correctness signal. [DOI](https://doi.org/10.1038/s41586-024-07421-0?utm_source=chatgpt.com) For Piper, I would test this selectively rather than on every dispatch, because additional inference has a cost. **Benchmark:** Take 150–200 dispatches, preferably including a reasonable number of failures. Compare success prediction with and without an independent disagreement signal. Score the improvement in AUROC, Brier skill, and detection of failures at a fixed false-alarm rate. Record the additional inference cost. A critical control is testing whether the independent agent's information merely duplicates task difficulty. If disagreement adds no information beyond experiment 1, reject it. --- ## Which would I build first? | Experiment | Cost | Implementation difficulty | Potential operational benefit | |---|---|---|---| | 1. Task difficulty | Low | Low–medium | Better initial dispatch predictions | | 2. Trajectory uncertainty | Low–medium | Medium | Detect impending failure and trigger intervention | | 3. Independent disagreement | Medium–high | Medium–high | Detect errors that Piper cannot recognize herself | **I'd build #1 and #2 as complementary instruments.** The first answers whether Piper is likely to succeed **before starting**. The second answers whether she is likely to succeed **given what has happened so far**. Both should be evaluated against a prospective rolling base rate, not a base rate calculated retrospectively from the test outcomes. That avoids giving the comparison predictor information it wouldn't have had in production. One statistical consideration: at RC1's 90.7% success rate, 150 dispatches produced only about 14 failures. That's a small sample for evaluating ranking quality. I'd retain all natural live outcomes but also report uncertainty intervals and performance by task category. ### One additional observation Your result says: > The base-model backtest ranked worse than her, 0.40. That doesn't necessarily mean other models cannot evaluate Piper's uncertainty. It means **that particular second-opinion method did not produce a useful ranking signal**. There's a significant difference between asking another model to estimate whether Piper will succeed and giving another model concrete evidence that Piper's work is failing. The second is closer to verification than introspection. For Component 23, I'd make the central hypothesis: **Uncertainty estimation should be grounded in observable evidence about task difficulty and execution state, not primarily in a language model's subjective confidence.** That's a substantially different direction from RC1, and one that Piper's existing instrumentation makes practical to investigate.