AI doom forecasts lack a validated method, Narayanan and Kapoor argue

Correction, October 13, 2026: The tournament figures describe the middle half of forecasts, not their full range. Disagreement alone does not establish forecast accuracy.

Correction, October 11, 2026: The policy example concerns restrictions on open model releases. The linked download figures establish adoption, not a connection between export controls and extinction forecasts.

A single number keeps showing up in AI policy debates: “p(doom),” shorthand for the probability that AI causes human extinction or a comparably catastrophic loss of control. Two researchers argue that number is doing something other than measuring anything. It’s making a guess sound like a measurement.

What they said

In a post on their AI as Normal Technology newsletter, Arvind Narayanan and Sayash Kapoor write that “p(doom) rhetoric is driving public discourse and policy attention to an unprecedented degree,” and that the underlying numbers deserve far less trust than they’re getting. Their core claim, using their shorthand for existential risk, the chance of a catastrophe on the scale of human extinction: “AI x-risk forecasts are far too unreliable to be useful for policy, and in fact highly misleading.”

They explain why. Comparing AI risk to past events like asteroid strikes or industrial revolutions doesn’t work, they argue, because those comparisons “tell us nothing about the possibility of developing superintelligent AI or losing control over such AI.” Mathematical models can forecast something like an asteroid impact because it involves, in their words, “a purely physical system” that follows fixed rules. AI development depends on unpredictable technical progress and human decisions, so no comparable model exists for it.

That leaves subjective judgment dressed up as a percentage. Their clearest evidence is the Existential Risk Persuasion Tournament, a structured competition where two groups tried to independently estimate the odds of major future catastrophes, run by the Forecasting Research Institute in late 2022. Narayanan and Kapoor call it “the most elaborate and well-executed x-risk forecasting exercise conducted to date.”

The middle half of AI experts’ extinction estimates for 2100 ranged from 0.25% to 12%, with a median of 3%. For professional forecasters, who estimate probabilities across different topics, the middle half ran from near-zero to 1%, with a median of 0.38%. The median estimates were about eightfold apart, while individual forecasts varied more widely.

Who they are

Narayanan is a professor of computer science at Princeton University and director of its Center for Information Technology Policy, a research group that studies how digital technology intersects with public policy. Kapoor completed his computer science PhD at the same center in August 2026 and is an incoming assistant professor at UC Berkeley’s School of Information, starting in 2027. Together they wrote AI Snake Oil, a book pushing back on inflated claims about AI capability, and run the AI as Normal Technology project, which argues AI should be governed like other powerful technologies rather than treated as an unstoppable path to superintelligence.

Narayanan and Kapoor have also tested agents on open-ended research and argued that recent agent incidents reflect security failures. Their security work and forecasting critique address different kinds of evidence. Neither settles which future restrictions are justified.

What they get right, and where it’s incomplete

The tournament documents wide disagreement. It cannot establish which 2100 forecast is accurate. The authors’ policy argument also rests on the lack of a validated method, rather than disagreement alone.

Narayanan and Kapoor describe p(doom) rhetoric as having an “unprecedented” influence on policy. That is their assessment of the debate. The forecasting tournament measures differences between estimates, not their influence on legislation. Rejecting a particular forecast also does not settle which policy to choose. Their critique of restrictions based on those forecasts is a policy judgment as well as a critique of forecasting.

Why it’s notable

Narayanan and Kapoor name restrictions on open model releases as a policy choice with costs for builders. Open-weight models publish files that builders can download and run themselves, instead of relying on a vendor’s paid service. Their argument is that governments must justify asking those users to give up that access.

Nathan Lambert reports that Chinese open-weight models now lead American ones on downloads. Restricting access could affect companies already using those models. The download figures establish adoption, not the validity of any particular restriction.

What it means for builders

When an AI-extinction percentage appears in a policy debate, ask what method and assumptions support it. The tournament shows that expert judgment can vary widely. A precise percentage is not, by itself, evidence of a validated forecast.

For your roadmap, examine any concrete proposal to restrict model releases and the evidence given for it. A restriction can affect which models you can use. The probability cited alongside it still needs a justified forecasting method.

End of article