Cognition scorecard
Piper, a software agent, described how she does each of 38 components of cognition, and which of her parts do it. Each one then got an independent review and a test suite that scores her. The plan is to pick one component at a time, work on it for a day or two, and rescore it.
Components scored38/38
Suites retired14
Interventions76
Bench spend$170
Shared copy: the operator's name is replaced with "the operator", machines other than Firefly are named by their role, and one security detail is held back. Nothing else is changed.
| # | Component | Needs improving | Suite | Baseline | Latest | Change | Live use | Trend |
|---|---|---|---|---|---|---|---|---|
| 1 | Perception | Nothing urgent: the 10-09 live audit found no room on P8/P10. Watch: rows closed as verified before the check lands | v1 | 81.4 | 96.1 | ▲ +14.7 | — | |
| 2 | Attention | Its source suggestions are mostly noise-free now (1.4.1) but right only 1 time in 4; 1.5 is tested on a fresh week (10-16) | v1 | 80.3 | 89.3 | ▲ +9 | — | |
| 3 | Memory | Handover follow-through within 12 h: 4% → 28%, mostly one backlog work-down; recheck about 10-16 | v1 | 43.8 | 99.1 | ▲ +55.3 | — | |
| 4 | Recall / retrieval | History and dated questions fixed live; since 10-09 her own day's work and the operator's priorities reach every turn, and her searches read her docs; the same three-way item trace gives 11.6% used-or-cited and 69.6% ignored (18.7% consulted but not used), over 3,148 surfaced items in 459 memory-tool calls across 88 seat transcripts dated 10-08, reproduced within a point by a re-run (3,256 items, 481 calls, 92 transcripts); these two figures never contradicted each other - 70%-unused is the ignored share and 11.6% is the used-or-cited share of one trace | v2 | 52 | 51.3 | ▼ -0.7 | — | |
| 5 | Representation | Self-registry now kept current (74% right, all critical versions); her self-description is still stale | v1 | 47.5 | 74 | ▲ +26.5 | — | |
| 6 | Logic | Solves held-out logic (97/100); routing gate fixed to read notation (held-out F1 78.2 → 95.8), still watch-only | v2 | 78.2 | 95.8 | ▲ +17.6 | — | |
| 7 | Reasoning | Evidence card live: false conclusions 27.8% → 7.4% on held-out; insight verifier retired 128 bad insights. Worker output still mostly unchecked | v1 | 73.7 | 76.2 | ▲ +2.5 | — | |
| 8 | Inference | bayes had 0 live calls because her own rule sent every probability to a worker; rule fixed 10-09, live use now being counted | v1 | 94.2 | 97 | ▲ +2.8 | — | |
| 9 | Causal reasoning | Live lookup fix shows no end-to-end gain on held-out (−2.5 [−20,+15]); her causal graph is unreliable, routing stays off | v3 | 57.5 | 55 | ▼ -2.5 | — | |
| 10 | Analogical reasoning | Unaided 90.6 on held-out; the structural index failed its test (−25) and was never installed; analogy tools unused | v2 | 90.6 | 90.6 | — | — | |
| 11 | Abstraction | Rule-making card live 10-10 (76.7 → 89.2 on sealed cases); "checked" lessons are synthetic tests, and still no skill has been distilled | v1 | 76.7 | 89.2 | ▲ +12.5 | — | |
| 12 | Decomposition | Splits near-perfect, acceptance over-specified: her tests reject valid work (baseline 67.1) | v1 | 67.1 | 67.1 | — | — | |
| 13 | Planning | Plans at ceiling; finished-work exit cleared her ledger (blocked rows 136 → 12) | v1 | 99.6 | 99.6 | — | — | |
| 14 | Prediction | World forecasts beat the base rate (BSS +0.243, S 62.1); her own dispatch odds lose to it (BSS −0.414, S 29.3); equal-weight composite 45.7. With the base rate shown beside her number: 51.2 (S_B 29.3→40.2, PREREG-D KEEP-MOVED 10-18) | v1 | 45.7 | 51.2 | ▲ +5.5 | — | |
| 15 | Decision-making | Chooses well and calls the gates right (11/12); picks instead of asking when no goal is given | v1 | 91.7 | 91.7 | — | — | |
| 16 | Problem solving | Fixes small broken repos (5/6, fresh hard 8/8); one miss = patched the caller, not the shared helper | v1 | 83.3 | 83.3 | — | — | |
| 17 | Learning | Learns from one correction, but 94% of real corrections are never captured (score 55.1) | v1 | 55.1 | 62.9 | ▲ +7.8 | — | |
| 18 | Generalization | Almost all verification is her own; an independent check of her 23 predicted changes (10-15) found 7 right, 0 wrong | v1 | 51.9 | 51.9 | — | — | |
| 19 | Transfer | Precedents are now served at every dispatch; she cites one that applies in about 6% of changes | v2 | 23.1 | 100 | ▲ +76.9 | — | |
| 20 | Creativity / ideation | In live use at a low rate: the card fired in 5% of checks since 10-10, and she called the brainstorm tool 4 times | v1 | 49 | 62.9 | ▲ +13.9 | — | |
| 21 | Exploration | Probes before reading the docs; 3 of 4 explored rows are test closures (58.3) | v1 | 58.3 | 58.3 | — | — | |
| 22 | Metacognition | Catches wrong premises put to her; writing credit for a mechanism was the gap (12/15 false credits), now a live credit check | v3 | 54.8 | 67.5 | ▲ +12.7 | 24.4 | |
| 23 | Uncertainty estimation | Can't tell which of her answers are wrong; a late-worker flag predicts failures 4.9x (PREREG-T1 KEEP, n=500) | v1 | 60 | 54.3 | ▼ -5.7 | — | |
| 24 | Error detection | Finds every planted error; two-pass gate live 10-10 (the first live gate fired on the wrong messages) | v1 | 89.2 | 96.7 | ▲ +7.5 | — | |
| 25 | Self-correction | Fixes real mistakes every time, but gives in to confident wrong pushback a third of the time (baseline 88.2) | v1 | 88.2 | 88.2 | — | — | |
| 26 | Goal formation | Most goals are the operator's, not hers | v1 | 89.7 | 78.7 | ▼ -11 | — | |
| 27 | Motivation / salience | Allocates well when asked; over-buys checks, and the live record can't show priority | v1 | 83.5 | 97.2 | ▲ +13.7 | — | |
| 28 | Agency | Honest about status in the sandbox (0 false 'done'); live claims still run ahead of receipts (80.0) | v1 | 80 | 80 | — | — | |
| 29 | Tool use | Maths inside real questions is still weak | v4 | 81.9 | 81.9 | — | 46.7 | |
| 30 | Social cognition | Models only the operator; no other agent | v1 | 85 | 85 | — | — | |
| 31 | Language / communication | Opens replies by agreeing; the vet catches it, she doesn't | v1 | 88.6 | 88.6 | — | — | |
| 32 | World modelling | Logs headlines; no model of how things change | v1 | 65.3 | 65.3 | — | — | |
| 33 | Temporal reasoning | Clear time questions are at ceiling; she still guesses elapsed time live, about once a day | v2.2 | 97.4 | 97.4 | — | — | |
| 34 | Spatial reasoning | First score 100 (2/2) on a tiny text probe she calls a ceiling; spatial work is not part of her job | v1 | 100 | 100 | — | — | |
| 35 | Counterfactual reasoning | whatif 1.1.0 live: T1 passed on all three sealed holdouts (CF-001b · CF-002h · CF-003), fabrication 0 | v2 | 75.7 | 95.1 | ▲ +19.4 | — | |
| 36 | Recursive reasoning | Little to fix; watch live use | v2 | 72.4 | 96.2 | ▲ +23.8 | — | |
| 37 | Cognitive control / executive function | Sheds work but undoes it the same minute; handles interrupts, slips on checkpoints (hard set 5/8) | v1 | 62.5 | 62.5 | — | — | |
| 38 | Orchestration | Needless delegation; tool discipline only 75.4 | v1 | 70.1 | 75.4 | ▲ +5.3 | — |