AI forecasters have caught superforecasters on ForecastBench, but only with help

Forecasting is the part of trading that AI marketing loves most and that is hardest to test. ForecastBench, run by the Forecasting Research Institute, avoids the usual trap of testing on the past. Every question is about an event that has not resolved when the forecast is submitted, so no model can have read the answer.

The benchmark was introduced in a September 2024 paper by Ezra Karger, Philip Tetlock and colleagues (last revised on arXiv in February 2025), and it keeps running with new questions and new models.

What gets forecast

Questions come from two kinds of sources. Market questions are drawn from forecasting platforms: Polymarket, Metaculus, Manifold Markets and the RAND Forecasting Initiative. Dataset questions are generated from data series: FRED economic data, Yahoo Finance prices, DBnomics, ACLED conflict data and Wikipedia. In the original question bank, Yahoo Finance supplied 509 questions and FRED 166.

Accuracy is measured with a difficulty-adjusted Brier score, converted to a Brier Index from 0 to 100: 100 is perfect, 50 is what you get by always answering 50/50, and 0 is maximally wrong. Higher is better. The figures below are index points, not percentages.

Leaderboard 1: the tournament (tools allowed)

Teams can improve models any way they like: tools, extra context, fine-tuning, ensembles. Models submitted by the ForecastBench team receive the crowd forecast as context on market questions. Figures as of the leaderboard update of October 14, 2026.

The top AI entry edges the superforecaster median on the point estimate, but the confidence intervals overlap almost completely. The leaderboard's own significance test says superforecasters are not better than the top system (p = 0.66), and the top systems are significantly better than the public median (p < 0.001). Fairly stated: a tie at the top.

Note what is on the list. The leading entries carry code names, so we cannot say which commercial model sits underneath, and several are heavily engineered systems rather than a model in a chat window.

Leaderboard 2: the baseline (no tools)

The second leaderboard is run only by the ForecastBench team. Models are tested out of the box, with no extra tools, context or scaffolding.

Here the picture flips. Superforecasters are significantly more accurate than every model listed above (p < 0.01 or lower), and no raw model beats the median forecast of the general public, though the intervals for the top models overlap with the public's.

How far the field has moved

In the original 2024 paper, on 200 questions answered by both groups, superforecasters scored a mean Brier score of 0.096 against 0.122 for the best model, Claude 3.5 Sonnet, a significant gap (p < 0.001; lower Brier is better in that raw scale). On today's tournament board, those same 2024 entries sit far down: a Claude 3.5 Sonnet configuration with news ranks 330th at 60.3. Two years later, engineered systems close the gap entirely.

Caveats from the leaderboard itself

• The human groups were last surveyed in July 2024, so they answered different questions from recent models. ForecastBench adjusts for difficulty with a statistical model, but it is not a same-question comparison.
• Models appear only 50 days after submitting forecasts, so the board lags.
• The crowd forecast given to team-run models on market questions was 10 days old at the forecast due date. Even so, access to a human crowd helps on those questions, which blurs "AI versus humans."

The institute has announced a superforecaster versus AI tournament on October 25, 2026. We will report the result.