Does Forecasting Have Room At The Top?
Read the original on Astral Codex Ten →
Follows up on
↳ The AI Superforecasters Are Here — Essay · Jul 2026
Summary
Continues 'The AI Superforecasters Are Here' by asking whether forecasting has headroom above top humans ('ultraforecasting') or is near an aleatoric ceiling. Against Daniel Reeves' 2010 finding that prediction markets beat simple models by only 3-6% RMSE, he argues the small number is an artifact: sports are deliberately engineered against prediction, and the market's gain was half the size of knowing team records. Box office was measured on a log scale against screen-count and search-traffic baselines, and Metaculus pros already remove 3x more uncertainty than a naive port of Reeves allows. Then, Cotra-style, he sets three anchors (Reeves-sized gain → score 56; Samotsvety-over-pros gain → 65; chess-handicap analogy, ~3 pawns → 85) and translates them into a 4-12 percentage-point sharpening of a 50% market. He argues that is genuinely valuable and compounds through better question-finding and policy. He states 70-30 for continued AI gains.
Why this score
Quality 67 · Strong. Strong. A careful, quantitative piece that turns a vague 'is there a ceiling?' debate into anchored estimates with an interpretable payoff, and the sports-optimized-against-prediction point is a memorable reframe. Med confidence and a cap in the mid-60s because the napkin conversions lean on AI-assisted math he flags as possibly unreliable, and the anchors are loose.
Claude’s paradigm shift 50 · Moderate. Moderate. Applies the Cotra bio-anchors method to forecasting limits, a fresh pairing, though the components (aleatoric-limit debate, chess handicaps) are known.
Real-world impact 1 · Negligible. 1: forecasting-community discussion only.