Forecasting · Explainer 2 of 3

What 62% actually means.

A probability is a promise about many worlds, not one. Here is how to read it, test it and score it.

FForeseez ResearchSep 22, 20268 min read

When a market says 62%, it is not predicting that the event will happen. It is saying something stranger: across all the ways the next few weeks could unfold, a bit more than six in ten end with a yes.

You only get to live in one of those worlds. That makes a single forecast impossible to grade on its own. A 62% call that misses was not necessarily wrong, any more than a 62% call that hits was right. So let us cheat, and live in a hundred of them.

FIGURE 1 · ROLL THE WORLDA hundred worlds where the odds are 62%

?Press Run to draw 100 worlds.

Every run so far0 runs · average –

your runswhat theory expects95% of runs land here (52 to 72)

Each dot is one world. Filled dots are worlds where the event happened. Every run draws 100 fresh worlds at exactly 62%.

Run it a few times. Even when the true chance is exactly 62%, the count bounces around: high fifties one run, mid sixties the next. About 95% of runs land between 52 and 72. That spread is not error. It is what 62% means.

So how do you grade a forecaster?

You grade many forecasts at once, with a score that rewards being confident and right and punishes being confident and wrong. The most common is the Brier score: take the gap between the forecast and what happened, square it, and average across every forecast. Lower is better. Zero is perfect.

FIGURE 2Score a single forecast

Your forecast80%

BRIER PENALTY0.640Worse than always saying 50% (0.25). You said 80% and it did not happen.

The solid curve is the penalty if the event goes the way you chose above. The dashed curve is the penalty if it goes the other way.

Two things fall out of the shape. First, hedging has a price: saying 50% every time scores 0.25 whatever happens, which is the bar any forecaster has to beat. Second, confidence is expensive when you miss. A 95% call that fails costs about 0.90. A 70% call that fails costs 0.49.

Do your 70s happen 70% of the time?

There is a simpler test than any score. Collect every forecast you made at around 70% and count how many came true. If it is close to 70%, you are calibrated. Do the same for every bucket and draw it. A calibrated forecaster's points sit on the diagonal.

FIGURE 3Build a forecaster, then check it

When it says 90%, it happens83%

ConfidenceDrag left for a forecaster who hedges, right for one who overstates.

TimidCalibratedOverconfident

OverconfidentSays 90%, happens 83%. Says 10%, happens 17%. The forecasts are too far from 50% for the evidence behind them.

Expected Brier0.209Cost of miscalibration+0.003

Circles are forecast buckets. The shaded area is the gap between what the forecaster says and what happens. Expected Brier assumes forecasts spread evenly from 5% to 95%.

Most people, and most models, lean overconfident: their 90s happen 75 or 80% of the time. The fix is rarely to know more. It is to say less extreme numbers when the evidence is thin.

How our agent does

We publish every forecast our agent makes and score it against the market price at the same moment. Across 412 resolved forecasts, its Brier score is 0.171 against the crowd's 0.179. It wins in three of five categories, ties in crypto, and loses in politics.

FIGURE 4Agent versus crowd, category by category

AgentCrowd

Politics72 forecasts · agent 0.195 · crowd 0.181Our weakest. Polling-heavy questions reward local knowledge the agent does not have yet, so we flag these forecasts as low confidence.

Brier scores on the same questions at the same moment. Sample data.

A gap of 0.008 is small. It is also the kind of edge that only shows up if you count every forecast, misses included. That is the whole point of keeping score in public.

See the full track record Next: If this, then that

Figures 1 to 3 use exact math on simulated forecasts. Figure 4 uses sample data. Nothing here is investment advice.