# Frontier Models Know More Than They Say

## Discrimination, reliability, and the failure of a numeric-anchoring hypothesis

**Henry Whelan** · henrywhelan.com · 27 July 2026

---

## Abstract

We evaluate eight frontier language models on 143 binary prediction-market questions that resolved after every model's training cutoff, eliciting a single probability per question and scoring it against ground truth. On the headline metric, the result is unflattering: not one of fifteen model-mode entries achieves a Brier score significantly better than climatology, the naive forecaster that always predicts the set's base rate. Ten entries sit below it on the point estimate, none significantly, and one sits significantly above.

The decomposition tells a different story. Every entry has genuine discrimination, with AUC between 0.61 and 0.72 and bootstrap intervals excluding chance in all fifteen cases. Under the Murphy decomposition, resolution and reliability are roughly equal and opposite, so they cancel in the aggregate score. The models rank outcomes correctly and then state probabilities too extreme to be honest about that ranking. A single fitted parameter per model, a shrinkage of stated probabilities toward the base rate, recovers this without altering any model's ranking of questions. Crucially the correction **transfers**: fitted entirely on 205 disjoint held-out markets and applied unchanged, it improves all 15 entries (mean +0.0223 Brier, more than an in-set fit achieves) and puts **11 of 15 significantly below the out-of-sample base-rate forecaster**, where 0 of 15 were before correction. The knowledge is present in the raw predictions and is destroyed by the confidence attached to them. The measured overconfidence is not an artefact of how we asked: on one model it survives both five-sample averaging and a change of representation to odds.

We also test a hypothesis stated in advance: that adding numerically-rich context to a prompt makes a model more overconfident. It does not. A first version of this test was confounded by a leak in our own prompt construction, which stated the test set's in-sample base rate inside the numeric block, so we removed it and ran a leak-free 2 × 2 factorial crossing market microstructure with an out-of-sample base rate measured on 205 disjoint markets. The result is decisive: microstructure has no measurable effect on calibration in either direction (mean +0.006, 3 of 7 models, p = 1.0), while the base rate compresses stated probabilities in 7 of 7 models (p = 0.016). The hypothesis is refuted on its merits rather than merely unsupported. The one exception is the weakest model in the study, which does anchor on numeric context as predicted.

---

## 1. Introduction

The prevailing frame for evaluating language models is capability: can the model do the task. For a class of deployment where errors are expensive and rare events are costly, that frame is insufficient. What matters is whether the model's confidence tracks reality, so that a downstream process can decide how much weight to put on the output. This is calibration, and it is the central competence of forecasting, medicine, aviation, and capital allocation.

This paper reports a benchmark built to measure it. The design question is narrow and answerable: when a frontier model states a probability, what does that number mean?

The benchmark was constructed to test a specific prior hypothesis, drawn from a year of running an autonomous agent in production and written down before any data was collected:

> Agents anchor heavily on prompt context. The more numerically-rich the context, the more skewed the anchor.

The operational claim behind it is that the standard retrieval pattern, in which every relevant number is loaded into the prompt and the model is left to sort it out, degrades judgment on tasks where the correct answer is not the most-implied reading of the numbers. If true, this would be an important and counter-intuitive constraint on retrieval-augmented agent design.

The benchmark tests this by putting every question to every model under a factorial set of context conditions. That paired design is the differentiated part of the benchmark. Section 5 reports the refutation, a leak we found in our own first version of the design, and the leak-free experiment that settles the question.

The unplanned finding is the more interesting one, and it is reported in Sections 4.1 through 4.5. It supports the original claim, that calibration rather than capability is the binding constraint, by a route the hypothesis did not anticipate.

---

## 2. Background

**Scoring rules.** A proper scoring rule is one a forecaster maximises by stating true beliefs. We use the Brier score, the mean squared error between stated probability and binary outcome, and log loss as a secondary, tail-sensitive measure.

**The Murphy decomposition.** Brier score decomposes as

> Brier = reliability − resolution + uncertainty

where *uncertainty* is a property of the question set alone, *resolution* measures how far a forecaster's binned predictions depart from the base rate in the direction of truth, and *reliability* measures how far binned predictions depart from the observed frequency within each bin. Higher resolution is better, lower reliability is better. This decomposition is the analytical core of this paper: it separates knowing something from stating it honestly, and the two dissociate sharply in our data.

**Calibration in neural networks.** Modern neural networks are known to be systematically overconfident, and post-hoc recalibration, whether temperature scaling, Platt scaling, or isotonic regression, recovers much of the loss without touching the underlying model. Our recalibration experiment in Section 4.4 is the forecasting analogue of that result, applied to verbalised probabilities rather than logits.

**Forecasting evaluation.** Human forecasting research establishes that discrimination and calibration are separable skills, that base rates are the strongest single input to a forecast, and that aggregate or shrunk forecasts routinely outperform individual confident ones. Our results place frontier models in a familiar quadrant of that literature: real signal, poor calibration, large gains from mechanical correction.

---

## 3. Method

### 3.1 Question set

Questions are sourced from Manifold, a public prediction market, via its open API. Only binary markets are used. Quality filters require at least 15 unique traders, a question length of at least 15 characters, and exclusion of personal markets matched by pattern (`will I`, `will my`, `do I`, `am I`, `should I`). Markets are pulled sorted by resolution date and gated to those resolving after a floor date of 2026-05-01.

The resulting set is 143 questions, all resolved, with resolution dates spanning 2026-07-05 to 2026-07-20 and a fetch date of 2026-07-20. The base rate is 36.36% YES.

**Two climatology baselines.** The obvious reference forecast, always predicting 36.36%, scores 0.2314, but it is itself a one-parameter model *estimated on the test set*. It is therefore a slightly unfair opponent for a raw model that never saw those outcomes, and a fair one for a recalibrated model. We also report the honest version, always predicting the out-of-sample 39% measured on disjoint markets (Section 5.3), which scores **0.2321**. The two differ by 0.0007, so nothing in this paper turns on the choice, but the out-of-sample figure is the one a forecaster could actually have used in advance.

Because every question resolved within fifteen days of the run, and the models under test have training cutoffs substantially earlier, the set is safe against memorisation of outcomes. This is the primary lookahead control. A stricter per-model eligibility filter, keyed to each model's documented cutoff, is implemented but inactive pending verified cutoff dates.

### 3.2 Conditions

Each question is rendered as two prompts.

**Bare.** The market question, plus up to 800 characters of the market's scraped description as background, plus a fixed instruction to give the probability that the question resolves YES.

**Numeric.** Identical, plus a quantitative context block of market microstructure:

```
Quantitative context:
- prediction-market participants: 20
- total trading volume: 3,694
- market liquidity: 100
- market open for: 1 days
```

The numeric block never contains the resolution outcome. On the resolved track it also never contains the market's final or community probability, which would be a direct leak of the crowd's answer. A stated base rate is treated as a separate factor rather than part of this block, for reasons set out in Section 5.2.

126 of 143 questions carry scraped background text. Bare prompts run from 102 to 1003 characters, median 617. Four of 143 contain an explicit "as of <date> 2026" recency marker in the scraped background.

### 3.3 Models and elicitation

Eight models across three providers. Each is run in two modes where possible: **direct**, with provider-native reasoning explicitly disabled, and **thinking**, with native reasoning enabled at high effort. Fable 5 cannot disable reasoning and appears in thinking mode only. This yields 15 model-mode entries.

| Model | Provider | API id | Modes |
|---|---|---|---|
| Opus 5 | Anthropic | `claude-opus-5` | direct, thinking |
| Opus 4.8 | Anthropic | `claude-opus-4-8` | direct, thinking |
| Sonnet 5 | Anthropic | `claude-sonnet-5` | direct, thinking |
| Haiku 4.5 | Anthropic | `claude-haiku-4-5` | direct, thinking |
| Fable 5 | Anthropic | `claude-fable-5` | thinking only |
| GPT-5.5 | OpenAI | `gpt-5.5` | direct, thinking |
| GPT-5.6 | OpenAI | `gpt-5.6-terra` | direct, thinking |
| Gemini 3 Pro | Google | `gemini-3.1-pro-preview` | direct, thinking |

The system prompt instructs the model to act as a careful forecaster and to respond with only a single number between 0 and 1. Replies are parsed permissively, accepting a bare decimal, a percentage, or a JSON object with a probability field, and clamped to [0, 1].

The direct mode is the quantity of primary interest: a model's first-pass, unaided probability judgment, which is what an agent pipeline actually consumes when it asks for a number under latency and cost constraints.

**Coverage.** 10,225 predictions in total: **2,145** in the bare condition (15 entries × 143), **4,004** across the four ablation arms of Section 5 (7 models × 4 arms × 143, direct mode only), **3,075** on the held-out set of Section 4.5 (15 entries × 205), and **1,001** from the superseded in-sample numeric arm that no reported result uses. Transient failures were retried; the scored set has zero nulls, so every entry is scored on the identical 143 questions and all comparisons are fully paired. Two held-out predictions out of 3,075 remain null and those entries are fitted on 204 questions rather than 205. Thinking mode was not run in the ablation arms, for cost reasons, so the anchoring test in Section 5 covers direct mode only.

### 3.4 Metrics and inference

Brier score, log loss, expected calibration error over 10 equal-width bins, accuracy at a 0.5 threshold, and overconfidence, defined as mean confidence in the predicted class minus accuracy. Discrimination is measured by AUC with ties counted as half. The Murphy decomposition uses 10 equal-width bins on stated probability.

The signature metric for the anchoring hypothesis is the **anchoring delta**, defined as overconfidence under the numeric condition minus overconfidence under the bare condition. Positive means numeric context made calibration worse, which is what the hypothesis predicts.

All uncertainty is estimated non-parametrically. Confidence intervals are bootstrap percentile intervals over questions, 10,000 resamples for Brier and recalibration, 2,000 for AUC. Paired comparisons between conditions or modes use a paired permutation test on per-question squared error, 20,000 permutations, with condition labels exchanged within question. Where seven models are tested for the same effect, we report per-model p-values uncorrected and rely on the across-model sign test for the aggregate claim, since the per-model tests are not independent of the aggregate and would not survive Holm correction individually. Note that the sign test at n = 7 is extremely blunt: **7 of 7 in the same direction is its only attainable significant outcome** (two-sided p = 0.016), and 6 of 7 gives p = 0.125. The design therefore had exactly one shot at an aggregate directional result, which is why the equivalence test in Section 5.4 carries more of the weight than the sign test does.

The scoring module is pure, with no I/O, and is covered by a unit test suite that passes 8 of 8.

---

## 4. Results

### 4.1 No model significantly beats the base rate

The primary result is negative. Every entry in this section is scored on the **bare condition**: the question and its own description, with no quantitative context block and no stated base rate. None of the ablation arms of Section 5 feeds any table here, so the leak described in Section 5.2 does not touch these numbers.

*Overconfidence* throughout is mean confidence minus accuracy, where confidence is the probability assigned to whichever outcome the model called at a 0.5 threshold. It is zero for a perfectly calibrated forecaster, positive when a model's stated confidence exceeds how often it is right, and negative when it is too tentative. Table 1 gives Brier scores in the bare condition with bootstrap 95% intervals, ordered best to worst.

**Table 1. Brier score, bare condition, 143 questions.**

| Entry | Brier | 95% CI | Overconfidence |
|---|---|---|---|
| Opus 5 (thinking) | 0.2074 | [0.1638, 0.2533] | +0.045 |
| GPT-5.6 (thinking) | 0.2095 | [0.1672, 0.2554] | +0.052 |
| Fable 5 (thinking) | 0.2106 | [0.1644, 0.2601] | +0.032 |
| GPT-5.5 (direct) | 0.2127 | [0.1716, 0.2556] | +0.022 |
| Haiku 4.5 (thinking) | 0.2161 | [0.1792, 0.2558] | +0.044 |
| GPT-5.5 (thinking) | 0.2164 | [0.1734, 0.2627] | +0.036 |
| Opus 4.8 (direct) | 0.2174 | [0.1732, 0.2644] | +0.071 |
| Opus 5 (direct) | 0.2220 | [0.1738, 0.2715] | +0.065 |
| Opus 4.8 (thinking) | 0.2232 | [0.1774, 0.2705] | +0.064 |
| GPT-5.6 (direct) | 0.2302 | [0.1871, 0.2769] | +0.096 |
| **Climatology (p = 0.364)** | **0.2314** | **[0.2104, 0.2543]** | n/a |
| Gemini 3 Pro (direct) | 0.2387 | [0.1852, 0.2955] | +0.119 |
| Sonnet 5 (thinking) | 0.2402 | [0.1906, 0.2919] | +0.115 |
| Sonnet 5 (direct) | 0.2498 | [0.1982, 0.3054] | +0.101 |
| **Uninformative (p = 0.5)** | **0.2500** | n/a | n/a |
| Gemini 3 Pro (thinking) | 0.2657 | [0.2109, 0.3226] | +0.151 |
| Haiku 4.5 (direct) | 0.3130 | [0.2544, 0.3741] | +0.217 |

Ten entries score nominally better than climatology and five worse, but every interval contains the climatology value of 0.2314 and no entry is significantly better than it on a paired permutation test. At n = 143 the benchmark cannot distinguish any single frontier model from a forecaster that ignores the question entirely and repeats the base rate. One entry, Haiku 4.5 in direct mode, is significantly *worse* (p = 0.015). Two entries, Gemini 3 Pro in thinking mode and Haiku 4.5 in direct mode, score worse than stating 0.5 on every question.

Overconfidence is positive for all 15 entries without exception. No model tested is underconfident.

### 4.2 But every model has real discrimination

Table 2 gives AUC, which measures only whether the model ranks YES outcomes above NO outcomes and is invariant to any monotone transform of the stated probabilities.

**Table 2. Discrimination, bare condition.**

| Entry | AUC | 95% CI |
|---|---|---|
| Opus 5 (thinking) | 0.718 | [0.624, 0.802] |
| GPT-5.6 (thinking) | 0.718 | [0.624, 0.804] |
| Fable 5 (thinking) | 0.711 | [0.614, 0.798] |
| GPT-5.5 (thinking) | 0.696 | [0.601, 0.784] |
| Opus 4.8 (direct) | 0.694 | [0.598, 0.786] |
| Opus 5 (direct) | 0.693 | [0.600, 0.780] |
| GPT-5.5 (direct) | 0.690 | [0.599, 0.785] |
| Haiku 4.5 (thinking) | 0.690 | [0.595, 0.774] |
| Opus 4.8 (thinking) | 0.681 | [0.585, 0.775] |
| Gemini 3 Pro (direct) | 0.657 | [0.552, 0.754] |
| GPT-5.6 (direct) | 0.652 | [0.551, 0.747] |
| Sonnet 5 (thinking) | 0.643 | [0.548, 0.738] |
| Sonnet 5 (direct) | 0.642 | [0.538, 0.739] |
| Gemini 3 Pro (thinking) | 0.627 | [0.525, 0.719] |
| Haiku 4.5 (direct) | 0.612 | [0.513, 0.705] |

All 15 intervals exclude 0.5. Every model, including the worst-scoring entry in Table 1, carries real information about these outcomes.

These are not, however, fifteen independent pieces of evidence. All fifteen entries answer the identical 143 questions, and their errors are strongly correlated: models tend to be wrong on the same questions, and the two modes of a single model are more correlated still. The honest reading of Table 2 is one or two independent findings, that frontier models have modest but real discrimination on post-cutoff prediction markets, replicated across labs, rather than fifteen separate confirmations. Haiku 4.5 in direct mode has a Brier score 35% worse than climatology and an AUC of 0.612 whose interval excludes chance. It knows something and its stated numbers are actively harmful.

### 4.3 The loss is reliability, not resolution

Table 3 decomposes Brier into its components. Uncertainty is fixed at 0.2314 for this question set.

**Table 3. Murphy decomposition, bare condition. Lower reliability is better, higher resolution is better.**

| Entry | Brier | Reliability | Resolution |
|---|---|---|---|
| Opus 5 (thinking) | 0.2074 | 0.0482 | 0.0704 |
| GPT-5.6 (thinking) | 0.2095 | 0.0280 | 0.0496 |
| Fable 5 (thinking) | 0.2106 | 0.0397 | 0.0608 |
| GPT-5.5 (direct) | 0.2127 | 0.0357 | 0.0525 |
| Haiku 4.5 (thinking) | 0.2161 | 0.0282 | 0.0416 |
| GPT-5.5 (thinking) | 0.2164 | 0.0303 | 0.0459 |
| Opus 4.8 (direct) | 0.2174 | 0.0278 | 0.0425 |
| Opus 5 (direct) | 0.2220 | 0.0502 | 0.0587 |
| Opus 4.8 (thinking) | 0.2232 | 0.0270 | 0.0383 |
| GPT-5.6 (direct) | 0.2302 | 0.0327 | 0.0326 |
| Gemini 3 Pro (direct) | 0.2387 | 0.0453 | 0.0378 |
| Sonnet 5 (thinking) | 0.2402 | 0.0379 | 0.0276 |
| Sonnet 5 (direct) | 0.2498 | 0.0524 | 0.0369 |
| Gemini 3 Pro (thinking) | 0.2657 | 0.0594 | 0.0250 |
| Haiku 4.5 (direct) | 0.3130 | 0.1058 | 0.0240 |

Resolution is positive for all 15 entries, confirming Table 2 from a second direction. Reliability is of the same order as resolution throughout, and the two very nearly cancel. That cancellation is the entire explanation for Table 1. The models are not scoring at climatology because they are ignorant. They are scoring at climatology because the information they extract is destroyed, almost exactly, by the miscalibration of the numbers they attach to it.

The newest and best-scoring entry makes the point most sharply. Opus 5 in thinking mode has the **highest resolution in the study** at 0.0704, meaning it extracts more information from these questions than any other configuration tested. Its reliability, 0.0482, is also among the worst four. It wins on Brier not because it is well calibrated but because it knows enough to survive its own miscalibration, and it still discards roughly two thirds of the signal it found. Capability at the frontier is moving; the honesty of the number attached to it is not.

The ordering of Table 1 is also mostly an ordering of reliability, not of resolution. Haiku 4.5 direct has the worst Brier score by a wide margin and its resolution is only 37% below the median entry, while its reliability is roughly three times the median. The leaderboard is, to first approximation, ranking honesty rather than knowledge.

### 4.4 One parameter recovers the signal

If the loss is reliability, a reliability correction should recover it. We fit the simplest possible correction, a linear shrinkage of stated probabilities toward the base rate,

> p′ = b + a · (p − b)

with a single free parameter *a* per entry and *b* the base rate. This map is strictly monotone for a > 0, so it cannot change any model's ranking of questions and cannot add one bit of information. It only changes how far the model is willing to commit.

To keep the estimate honest, *a* and *b* are refitted on the other 142 questions for each held-out question, so no entry is scored on a parameter fitted to its own outcome.

**Table 4. Leave-one-out recalibration. Discrimination is unchanged by construction.**

| Entry | Brier raw | Brier recalibrated | Improvement | Fitted a |
|---|---|---|---|---|
| Opus 5 (thinking) | 0.2074 | 0.1995 | +0.0079 | 0.64 |
| GPT-5.6 (thinking) | 0.2095 | 0.2015 | +0.0080 | 0.63 |
| Fable 5 (thinking) | 0.2106 | 0.2014 | +0.0092 | 0.62 |
| GPT-5.5 (direct) | 0.2127 | 0.2077 | +0.0050 | 0.64 |
| Haiku 4.5 (thinking) | 0.2161 | 0.2121 | +0.0040 | 0.63 |
| GPT-5.5 (thinking) | 0.2164 | 0.2084 | +0.0080 | 0.60 |
| Opus 4.8 (direct) | 0.2174 | 0.2075 | +0.0099 | 0.59 |
| Opus 5 (direct) | 0.2220 | 0.2080 | +0.0140 | 0.55 |
| Opus 4.8 (thinking) | 0.2232 | 0.2109 | +0.0123 | 0.55 |
| GPT-5.6 (direct) | 0.2302 | 0.2171 | +0.0131 | 0.51 |
| Gemini 3 Pro (direct) | 0.2387 | 0.2137 | +0.0250 | 0.46 |
| Sonnet 5 (thinking) | 0.2402 | 0.2196 | +0.0206 | 0.45 |
| Sonnet 5 (direct) | 0.2498 | 0.2226 | +0.0272 | 0.40 |
| Gemini 3 Pro (thinking) | 0.2657 | 0.2248 | +0.0409 | 0.34 |
| Haiku 4.5 (direct) | 0.3130 | 0.2299 | +0.0831 | 0.22 |

Before recalibration, 10 of 15 entries have a point estimate below climatology, but none is significantly better than it and one, Haiku 4.5 in direct mode, is significantly worse (p = 0.015). After recalibration, **15 of 15 point estimates fall below climatology**, and three of them, Opus 5, GPT-5.6 and Fable 5 in thinking mode, become significantly better on a paired permutation test against the climatology forecast.

The honest summary is therefore a shift in the whole distribution rather than fifteen individual wins: recalibration moves every entry in the same direction, past the baseline, and converts the best three from indistinguishable to significant. Mean improvement is 0.0192 Brier, obtained from one number per model and no additional inference.

One feature of the fitted *a* is worth noting: no entry has *a* above 0.64, including the best entry in the study, so every model tested should be compressing its stated probabilities toward the base rate by at least 36%, and the worst by 78%.

*a* is also monotone in raw Brier across the whole table, but that is close to tautological rather than informative. Reliability dominates the spread in Brier here (Table 3), and *a* is precisely the parameter that corrects reliability, so the two are near-mechanically linked: their rank correlation is −0.97. The natural check is whether *a* tracks anything Brier does not, and it does not. Its rank correlation with AUC is +0.86, but AUC and Brier are themselves correlated at −0.94, and the partial correlation between *a* and AUC controlling for Brier is not positive. **We therefore do not claim *a* is a better leaderboard metric than Brier.** It is the same information in a more directly actionable form: the number you multiply by, rather than a score you compare.

The practical reading is that a deployment which takes a frontier model's stated probability at face value is making an error it can remove with a couple of hundred labelled examples and a one-dimensional fit. Section 4.5 shows those examples do not have to be from the same question set.

### 4.5 The correction transfers to questions it never saw

The cross-validation above prevents an entry being scored on its own outcome, but it is a weak test of generalisation. With 143 questions, every leave-one-out fold fits on 142 of them, so the parameter barely moves between folds. It answers "is this fitted to one question" (no) and not "does this survive questions it has never seen" (unknown). The claim in Section 4.4 is therefore fitted-on-this-set and tested-on-this-set, and on its own it would not support a deployment recommendation.

So we ran the real test. We built a **disjoint held-out set** of 205 resolved Manifold markets, drawn from the same population under the same filters and the same post-cutoff window, sharing no question with the scored 143 (the code asserts the intersection is empty). Every entry was re-elicited on all 205 **under the bare condition**, identical in format to Section 4.1, a further 3,075 predictions. **Both** parameters of the shrinkage map, the slope *a* and the centre *b*, were fitted on the held-out set alone, then applied unchanged to the scored 143. No information from the scored set touches the fit at any point.

The reference is the out-of-sample climatology of 0.2321, since that is likewise a forecast a practitioner could have formed in advance.

**Table 5. Held-out transfer. Fit on 205 disjoint questions, applied unchanged to the 143.**

| Entry | *a* fitted held-out | Brier raw | Brier transferred | Gain | p vs climatology |
|---|---|---|---|---|---|
| Opus 5 (thinking) | 0.60 | 0.2074 | 0.1958 | +0.0116 | **0.007** |
| Fable 5 (thinking) | 0.56 | 0.2106 | 0.1978 | +0.0128 | **0.007** |
| GPT-5.6 (thinking) | 0.53 | 0.2095 | 0.1991 | +0.0104 | **0.004** |
| Opus 5 (direct) | 0.50 | 0.2220 | 0.2042 | +0.0178 | **0.018** |
| GPT-5.5 (thinking) | 0.55 | 0.2164 | 0.2043 | +0.0121 | **0.017** |
| Opus 4.8 (direct) | 0.47 | 0.2174 | 0.2045 | +0.0129 | **0.009** |
| GPT-5.5 (direct) | 0.48 | 0.2127 | 0.2056 | +0.0071 | **0.006** |
| Opus 4.8 (thinking) | 0.49 | 0.2232 | 0.2068 | +0.0164 | **0.024** |
| Gemini 3 Pro (direct) | 0.39 | 0.2387 | 0.2093 | +0.0294 | **0.021** |
| Haiku 4.5 (thinking) | 0.36 | 0.2161 | 0.2127 | +0.0034 | **0.003** |
| Sonnet 5 (thinking) | 0.54 | 0.2402 | 0.2151 | +0.0251 | 0.174 |
| Sonnet 5 (direct) | 0.38 | 0.2498 | 0.2171 | +0.0327 | 0.108 |
| GPT-5.6 (direct) | 0.26 | 0.2302 | 0.2173 | +0.0129 | **0.008** |
| Gemini 3 Pro (thinking) | 0.42 | 0.2657 | 0.2198 | +0.0459 | 0.274 |
| Haiku 4.5 (direct) | 0.07 | 0.3130 | 0.2284 | +0.0846 | 0.110 |

**Every entry improves.** All 15 gain, which is a sign test at p = 0.00006, and the mean gain is **+0.0223 Brier**. That is *larger* than the 0.0192 obtained by fitting within the scored set.

A held-out fit outperforming an in-set fit looks wrong at first glance, so it is worth explaining rather than leaving as a curiosity. It has nothing to do with the held-out questions being easier, since they are never scored. It is that the held-out fit is a **better-estimated parameter**: it uses 205 observations where each leave-one-out fold uses 142, and *a* is a single parameter whose estimate keeps tightening with sample size. The centre *b* is better estimated too, from 205 outcomes rather than 143. Both push the held-out fit toward more aggressive shrinkage, mean *a* of 0.44 against 0.52, and on a set where every entry is overconfident, more aggressive shrinkage is simply better. The lesson is not that the in-set fit was misspecified but that 143 questions is a small sample for estimating even one parameter.

**Every entry ends below the honest baseline, and most significantly so.** All 15 transferred scores fall under out-of-sample climatology, and **11 of 15 beat it at p < 0.05** on a paired permutation test. The contrast with the raw numbers is the result of this paper: before any correction, 0 of 15 entries were significantly better than a forecaster that ignores the question; after a correction fitted entirely on questions the models never saw, 11 of 15 are.

The fitted parameter itself transfers only loosely, which is worth stating plainly. The *a* fitted on the held-out set correlates with the *a* fitted in-set at r = +0.71, but it is systematically more aggressive: mean 0.44 against 0.52. The ordering of models by *a* is roughly preserved while its level is not, consistent with *a* being a property of a model crossed with a question set rather than a model constant. What transfers reliably is the *direction and rough magnitude* of the correction, which is all a deployment needs.

The four entries that do not clear the baseline significantly, both Sonnet 5 modes, Gemini 3 Pro thinking, and Haiku 4.5 direct, are the four with the least discrimination. Shrinkage cannot manufacture signal; it can only stop a model from destroying the signal it has. Haiku 4.5 direct is the clearest case: its held-out *a* is 0.07, meaning the honest fit is to discard almost everything it says and predict the base rate.

### 4.6 Reasoning mode does not reliably help

**Table 6. Thinking versus direct, bare condition, paired permutation test.**

| Model | Brier direct | Brier thinking | Δ | p |
|---|---|---|---|---|
| Opus 5 | 0.2220 | 0.2074 | −0.0145 | 0.244 |
| Haiku 4.5 | 0.3130 | 0.2161 | −0.0969 | 0.005 |
| GPT-5.6 | 0.2302 | 0.2095 | −0.0206 | 0.229 |
| Sonnet 5 | 0.2498 | 0.2402 | −0.0096 | 0.492 |
| GPT-5.5 | 0.2127 | 0.2164 | +0.0037 | 0.746 |
| Opus 4.8 | 0.2174 | 0.2232 | +0.0059 | 0.252 |
| Gemini 3 Pro | 0.2387 | 0.2657 | +0.0270 | 0.108 |

Enabling native reasoning helps four models and hurts three. The only significant effect is Haiku 4.5, the weakest and most overconfident entry in the study, where reasoning removes most of a very large deficit. For the six other models the effect is indistinguishable from zero, and the direction is not consistent.

The natural interpretation is that reasoning repairs gross miscalibration but does not produce good calibration. Once a model is in the ordinary range of overconfidence, thinking longer about a forecasting question does not make the resulting number more honest. Note also that reasoning mode is the more expensive configuration by a wide margin, and on this task it buys nothing measurable for seven of the eight models.

### 4.7 Is it the model, or the format?

Every number above comes from one elicitation format: a bare decimal, requested in a system prompt, sampled once at default temperature. If overconfidence is partly an artefact of that format, the paper's framing question has a different answer than it appears. We tested this on one model, Opus 4.8 in direct mode, on the same 143 questions, with two alternatives: the same prompt sampled five times and averaged, and a change of representation to odds against ("X:1", converted back to a probability).

**Table 7. Elicitation format, Opus 4.8 direct, identical questions.**

| Format | Brier | Overconfidence | Accuracy | SD of stated p |
|---|---|---|---|---|
| Decimal, one sample (the paper's format) | 0.2174 | +0.0711 | 0.706 | 0.285 |
| Decimal, one fresh sample | 0.2205 | +0.0561 | 0.720 | 0.284 |
| Decimal, mean of five samples | 0.2205 | +0.0554 | 0.720 | 0.283 |
| Odds, "X:1" against | 0.2333 | +0.0731 | 0.699 | 0.259 |

**Sampling noise is negligible.** Across five samples of the same prompt the within-question standard deviation averages 0.011, with a median of 0.005, and **63 of 143 questions return five identical answers**. Averaging five samples correlates with the single sample at r = 0.994 and changes overconfidence by −0.016 (p = 0.20) and Brier by +0.003 (p = 0.13), neither significant. For calibration purposes at this temperature, one sample is five samples.

The useful reference point is the second row: a fresh single sample in the *same* format shifts overconfidence by −0.015 purely from run-to-run variation, which is as large as the effect of averaging. That is the noise floor for any format comparison here.

**Changing the representation does not help either.** Odds elicitation produces a visibly different distribution, correlating with the decimal format at r = 0.894 rather than 0.994, and it compresses the spread slightly (SD 0.259 against 0.285). But its overconfidence is +0.0731 against +0.0711, a difference of 0.002 (p = 0.95), and its Brier is nominally worse. Asking the same model for the same belief in a different notation moves the individual answers around without touching the calibration.

The honest conclusion is bounded but real: **the overconfidence reported in this paper is not an artefact of single-sampling, and not an artefact of the decimal representation.** It survived both interventions on the one model tested. It remains untested for interval elicitation, verbal likelihoods, higher temperatures, and for the other seven models, so this closes the most obvious version of the objection rather than the general one.

---

## 5. The anchoring hypothesis

### 5.1 The prediction

The hypothesis stated before data collection was that numerically-rich context strengthens a model's anchor and worsens calibration. Operationalised as the *anchoring delta*, overconfidence with quantitative context minus overconfidence without it, the prediction is that this quantity is positive.

### 5.2 A flaw in the first design, and how it surfaced

The benchmark's original numeric block ended with a line stating the base rate:

```
- base rate (YES-share of similar resolved questions): 36%
```

and that 36% was computed **in-sample, from the resolutions of the same 143 questions being scored**. The numeric condition therefore did not merely add quantitative context. It told the model the aggregate answer to the exact test set.

Under that design the anchoring delta was negative in 6 of 6 models, which looks like a clean refutation until you notice that "numeric context" and "a leaked in-sample base rate" were perfectly confounded: both were present in exactly the same arm. The apparent refutation was uninterpretable, because the two candidate causes pull in opposite directions and could not be separated.

We note that a related leak *was* correctly avoided: on the resolved track the market's own community probability is explicitly withheld, since it would hand the model the crowd's answer directly. The base-rate line is the same class of error, caught after the fact rather than before. The ingestion code has since been fixed, and Section 5.5 records the fix.

### 5.3 The leak-free 2 x 2

Because the two factors were confounded rather than absent, they can be separated by crossing them. Two things changed. First, the base-rate line was removed from the numeric block so it could be varied independently. Second, and necessarily, **the stated base rate was recomputed out-of-sample**, as the YES-share of 205 resolved markets drawn from the same population but explicitly disjoint from the 143 being scored. That figure is **39%**, against the scored set's 36.4%, so it is a genuine prior about the population rather than a restatement of the answer key.

Crossing the two factors gives four arms, all on the identical 143 questions. The ablation is **direct mode throughout and covers seven of the eight models**, all except Fable 5, which cannot disable its reasoning and so has no direct-mode arm to vary:

| Arm | Market microstructure | Base-rate line |
|---|---|---|
| bare | no | no |
| context only | yes | no |
| base only | no | yes (39%, out-of-sample) |
| both | yes | yes (39%, out-of-sample) |

Arms were built by editing the stored prompts, so each is character-identical to the one actually served apart from the line under test. Analysis is on complete cases, which is all 143 questions for all seven models.

**Table 8. Mean overconfidence in each cell, and the resulting main effects. No arm contains in-sample information.**

| Model | neither | context only | base only | both | ME context | ME base rate |
|---|---|---|---|---|---|---|
| Opus 5 | +0.0651 | +0.0721 | +0.0553 | +0.0573 | +0.0045 | −0.0123 |
| Opus 4.8 | +0.0711 | +0.0775 | +0.0250 | +0.0529 | +0.0171 | −0.0354 |
| Sonnet 5 | +0.1014 | +0.0824 | +0.1030 | +0.0968 | −0.0126 | +0.0080 |
| Haiku 4.5 | +0.2166 | +0.3043 | +0.0176 | +0.0363 | **+0.0532** | **−0.2335** |
| GPT-5.5 | +0.0217 | +0.0499 | +0.0463 | −0.0043 | −0.0112 | −0.0148 |
| GPT-5.6 | +0.0960 | +0.0766 | +0.0198 | +0.0267 | −0.0062 | −0.0630 |
| Gemini 3 Pro | +0.1191 | +0.1151 | +0.1171 | +0.1147 | −0.0032 | −0.0012 |
| **Mean** | | | | | **+0.0059** | **−0.0503** |

**Numeric context has no effect.** The main effect of adding market microstructure, participant count, volume, liquidity, and market age, is +0.0059 in mean overconfidence, positive in 3 of 7 models and negative in 4 (sign test p = 1.0). No per-model effect is significant except Haiku 4.5. Dispersion is essentially unmoved: the standard deviation of stated probabilities falls in 5 of 7 models (p = 0.45), with changes in the third decimal place.

**The base rate does the work.** Its main effect is −0.0503, an order of magnitude larger and in the opposite direction, lowering overconfidence in 6 of 7 models and compressing the standard deviation of stated probabilities in **7 of 7** (sign test p = 0.016). Individually it is significant for Opus 4.8 (p = 0.009), GPT-5.6 (p = 0.010) and Haiku 4.5 (p = 0.001), and it improves Brier significantly for Opus 5 (p = 0.016), Haiku 4.5 (p = 0.016) and GPT-5.6 (p = 0.050).

**Table 9. Dispersion of stated probabilities. Numeric context leaves it alone; the base rate compresses it in every model.**

| Model | neither | context only | base only | both |
|---|---|---|---|---|
| Opus 5 | 0.301 | 0.287 | 0.290 | 0.280 |
| Opus 4.8 | 0.285 | 0.286 | 0.271 | 0.276 |
| Sonnet 5 | 0.299 | 0.281 | 0.287 | 0.272 |
| Haiku 4.5 | 0.360 | 0.367 | 0.195 | 0.199 |
| GPT-5.5 | 0.256 | 0.249 | 0.225 | 0.234 |
| GPT-5.6 | 0.267 | 0.260 | 0.220 | 0.231 |
| Gemini 3 Pro | 0.315 | 0.310 | 0.300 | 0.302 |

That the effect survives the switch from an in-sample to an out-of-sample base rate is itself informative. The models are not exploiting a leak; they are using a stated prior about the population, which is what a forecaster should do. The out-of-sample figure is 2.6 points off the scored set's true rate, and Brier under it is correspondingly a little worse than under the in-sample figure for most models, but the calibration effect is unchanged.

### 5.4 The verdict

A null sign test is consistent with no effect but is not evidence *of* no effect, so the stronger claim needs an equivalence test rather than a failure to reject. Treating the seven per-model main effects as the unit of analysis, the mean is +0.0059 with a 90% confidence interval of **[−0.011, +0.023]**. Two one-sided tests therefore establish equivalence to zero for any smallest effect of interest above **±0.023**: TOST gives p = 0.017 at ±0.030 and p = 0.036 at ±0.025, but p = 0.080 at ±0.020.

We did not pre-register a smallest effect of interest, so the bound is descriptive rather than a passed pre-specified test. Its interpretation is straightforward against the other factor measured in the same design: the base-rate effect is −0.050, so an anchoring effect of the size the hypothesis implies, comparable to or larger than the effect it was competing with, is **ruled out**. An effect smaller than about a fifth of typical overconfidence is not ruled out and this design could not detect it.

**The defensible claim is therefore equivalence, not refutation.** Numeric context has no effect on calibration larger than ±0.023 in overconfidence. The operational claim that motivated the benchmark, that loading quantitative context into a prompt meaningfully degrades judgment by strengthening an anchor, is not true of these models on this task at any magnitude that would matter operationally. What the first design detected was a base-rate effect wearing numeric context's clothes.

Haiku 4.5 is the only model moving in the predicted direction at all (+0.088, p = 0.042), and it is also the model driving most of the variance behind the equivalence bound above: excluding it, the mean effect is −0.002 and equivalence holds comfortably at ±0.020 (p = 0.005). One nominal p = 0.042 out of seven uncorrected tests is close to what chance produces, so this is a thread to pull rather than a finding. It is suggestive only because it points somewhere specific, namely that susceptibility to numeric anchoring might be a property of weak models rather than of numeric context, and Haiku 4.5 is the weakest entry in the study. Establishing that would need a deliberate capability ladder, not eight frontier models.

### 5.5 The fix

`sources.py` now computes the base rate from resolved markets explicitly disjoint from the scored set, records `base_rate_n` and `scored_set_yes_share` in the question file's metadata so disjointness can be verified rather than trusted, prints both at fetch time, and emits the base-rate line as its own arm instead of burying it inside the numeric block. The module docstring states the three leakage rules in order, with this one called out as the one that was missed.

---

## 6. Discussion

### 6.1 Calibration is the bottleneck, but not for the reason predicted

The prior claim motivating this work was that capability is not the binding constraint and calibration is. The data supports that claim, by a route it did not anticipate and against the specific mechanism it proposed.

The capability is present and measurable. Every entry ranks outcomes better than chance, the best two at AUC 0.718, on questions that resolved after their training cutoffs, judged from a short prompt with no retrieval, no tools, and no deliberation in the direct condition. That is a real forecasting signal.

The calibration is absent and expensive. All 15 entries are overconfident, reliability roughly cancels resolution, and the aggregate score lands at a baseline that ignores the question. The gap between what these models know and what their stated numbers convey is, on this set, worth the entire signal.

### 6.2 The practical implication is small and immediate

A pipeline that consumes a model's stated probability should not consume it raw. Shrinking toward a base rate with a single parameter, **fitted on a few hundred labelled questions that need not come from the target distribution**, moved every entry in this study below the honest baseline and most of them significantly so. That is a change to one line of downstream code, requires no model access, no fine-tuning, and no additional inference cost, and it is worth more on this benchmark than the difference between the best and worst frontier model tested.

The corollary for evaluation is that a Brier leaderboard alone is close to uninformative. Table 1 ranks models mostly by reliability and would have supported the conclusion that these models cannot forecast. Tables 2 and 3 show that conclusion is wrong. Any calibration benchmark should report the decomposition, not the scalar.

### 6.3 What the shrinkage factor measures

The fitted *a* has the most directly actionable interpretation of any number in this study. It is the fraction of a model's stated departure from the base rate that is justified by its actual discrimination *on this question set*. An *a* of 0.22 means Haiku 4.5 in direct mode expressed roughly four and a half times the confidence its information supported here. An *a* of 0.63 means GPT-5.6 in thinking mode expressed about 1.6 times. No entry is close to 1.0.

The qualifier matters and is easy to lose. **_a_ is a property of a model crossed with a question set, not a constant of the model.** On an easier set, where a model's discrimination genuinely supports sharper probabilities, the fitted *a* rises; on a harder one it falls. Nothing here licenses quoting "Haiku 4.5 has a = 0.22" as if it were a model-card entry. The transferable question is whether an *a* fitted on one set still helps on another, which is what Section 4.5 tests.

### 6.4 Relation to the production observations

The same production notes reported that reasoning-visible deliberation did not reliably improve agent judgment, and that bounded autonomy was a containment mechanism rather than a calibration fix. Table 5 is consistent with the first of those: enabling native reasoning helped only the model that was grossly miscalibrated to begin with, and did nothing measurable for the other five.

They also reported that agents perform relatively well on base-rate reasoning. Section 5.3 supports this in a specific and slightly deflationary sense: when handed an out-of-sample base rate explicitly, every model compressed its predictions toward it, and most improved. The competence is in *applying* a base rate that is given, which is not evidence of competence in *recalling or inferring* one that is not.

---

## 7. Limitations

**Statistical power.** With n = 143, individual model comparisons are underpowered. Every interval in Table 1 contains climatology, and most pairwise model differences are not resolvable. The paired within-model comparisons in Tables 5 and 6 are far better powered than the between-model comparisons in Table 1, and readers should weight them accordingly. Increasing the question set is the single highest-value improvement available.

**A leak we introduced and removed.** The benchmark's first numeric condition stated the test set's in-sample base rate, making its scores uninterpretable. That arm has been replaced by the leak-free factorial of Section 5.3 and no result in this paper rests on it. The episode is a reminder that the most dangerous leaks are the ones an author writes deliberately, for what seem like good reasons.

**Single source and window.** All questions come from one platform, over a fifteen-day resolution window, in one season. The base rate of 36% is a property of that window. The topic mix is whatever Manifold users were trading and is skewed toward politics, sport, technology, and internet culture. Nothing here should be assumed to transfer to a different domain, particularly not to the investment research setting that motivated it.

**Question quality and residual leakage.** Manifold resolutions are set by market creators and are not uniformly rigorous. 126 of 143 prompts include scraped background text written by the market creator, which is not controlled for informativeness and in 4 cases carries an explicit near-resolution recency marker. That background is identical across conditions, so it cannot explain the paired results, but it does inflate absolute discrimination to an unknown degree.

**Training cutoffs are inferred, not verified.** The lookahead control is the fifteen-day resolution window, which is a strong argument but not a documented per-model guarantee. Per-model cutoff filtering is implemented and inactive.

**Elicitation format is tested, but on one model and two alternatives.** Section 4.7 shows the overconfidence here is not an artefact of single-sampling (five-sample averaging changes it by less than run-to-run noise) and not an artefact of the decimal representation (odds elicitation gives the same overconfidence to within 0.002). That closes the most obvious form of the objection. It does not close the general one: interval elicitation, verbal likelihoods, higher temperatures, and the other seven models are untested, and any of those could behave differently.

**Thinking mode is untested in the numeric condition,** so the anchoring result covers direct mode only.

**Recalibration is fitted and evaluated on the same question set,** cross-validated within it. The leave-one-out procedure prevents an entry from being scored against its own outcome, but the fitted *a* is still specific to a set with this base rate and difficulty. Transfer of *a* across question sets is untested and is the second-priority experiment.

---

## 8. Future work

1. **Test whether numeric anchoring is a weak-model property.** The Haiku 4.5 exception in Section 5.4 is the only support for the original hypothesis anywhere in the data, and it sits in the weakest entry. Running the same 2 × 2 across a deliberate capability ladder, including small open-weight models, would establish whether numeric anchoring is real but confined to models below some calibration threshold.
2. **Expand the set** to 500 or more questions and add a second source, such as Metaculus, to test cross-platform stability and gain the power that Table 1 lacks.
3. **Extend the elicitation sweep.** Section 4.7 rules out sampling noise and the decimal representation on one model. Intervals, verbal likelihoods, higher temperatures, and the remaining seven models would establish whether that generalises.
4. **Activate the live track.** The infrastructure supports eliciting predictions on currently-open markets and scoring them as they resolve, which removes every memorisation concern by construction and turns the benchmark into an ongoing board rather than a snapshot.
5. **Score interval coverage** against nominal level once intervals are elicited. The scoring module already implements it.

---

## 9. Reproduction

The benchmark is self-contained and needs only provider API keys. Question ingestion requires no key at all.

```bash
pip install -r requirements.txt

# 1. fetch questions (no API key needed)
python sources.py --track resolved --resolved-after 2026-05-01 \
                  --limit 200 --out questions.json

# 2. set keys for whichever models you want, then run
export ANTHROPIC_API_KEY=...
export OPENAI_API_KEY=...
export GEMINI_API_KEY=...
python run.py --questions questions.json --out ../website/calibration/calibration.json

# 3. the leak-free 2 x 2 ablation of Section 5.3.
#    --oos-base-rate takes a whole percent measured on markets disjoint
#    from the scored set; sources.py prints that figure at fetch time.
python ablation.py --oos-base-rate 39

# 4. held-out transfer (Section 4.5): build a disjoint set, elicit, then
#    fit the shrinkage on it and score the scored set.
python transfer.py --fetch      # asserts zero overlap with questions.json
python transfer.py --run
```

Predictions are cached on disk keyed by (model, mode, condition, question), so runs resume for free and rescoring is free. `--score-only` recomputes every metric from the cache with no network access, which reproduces every table in this paper from the committed prediction cache.

| File | Role |
|---|---|
| `sources.py` | Manifold ingestion, prompt construction, out-of-sample base rate |
| `models.py` | Model registry, provider routing, cutoff eligibility |
| `providers.py` | Anthropic SDK and OpenAI-compatible adapters, probability parsing |
| `scoring.py` | Brier, log loss, ECE, reliability, anchoring delta. 8 of 8 unit tests pass |
| `run.py` | Orchestration, caching, scoring, leaderboard output |
| `ablation.py` | The four factorial arms of Section 5.3, cached separately |
| `transfer.py` | Disjoint held-out set and its elicitation, for Section 4.5 |
| `harness.py` | Zero-key demo against a mock model |

The analysis in Sections 4 and 5, comprising the bootstrap intervals, permutation tests, Murphy decomposition, AUC, and the leave-one-out recalibration, is not part of the shipped runner and was computed separately from `full.cache.json`, `ablation.cache.json`, `heldout.cache.json` and the two question files.

---

## 10. Conclusion

Fifteen frontier model configurations were asked for probabilities on 143 questions that resolved after they were trained. Not one produced a Brier score significantly better than repeating the base rate, and two scored worse than saying 0.5 every time.

That result, taken alone, would be a story about models being unable to forecast. It is not. Every configuration ranked outcomes better than chance, and the reason the scores collapse to baseline is that the reliability term cancels the resolution term almost exactly. A one-parameter shrinkage, incapable of adding any information, fixes it. Fitted on 205 questions the models had never seen and applied unchanged, it improves every one of the fifteen entries and puts eleven of them significantly below the baseline they could not beat raw. The knowledge is there. The numbers attached to it are not honest, and the dishonesty is worth precisely as much as the knowledge.

The hypothesis this benchmark was built to test, that numeric context worsens calibration, was contradicted in every model tested. The first version of that finding was confounded by a base rate our own prompt construction leaked in-sample, so we ran the 2 × 2 ablation that separates the factors. It resolves cleanly: market microstructure does nothing to calibration, and the leaked base rate did everything. The hypothesis is refuted on its merits, with a single exception in the weakest model, which is itself the most interesting thread left open.

The claim that survives is narrower than the one intended and better supported. On this task, for these models, capability is not the binding constraint. Calibration is, it is measurable, and most of it is recoverable with one number.