What Is p(doom), and Why Are AI Experts Worried About It?

“The Bible calls this day ‘Armageddon’; the end of all things. And yet, for the first time in the history of the planet, a species has the technology to prevent its own extinction.”
– The President of the United States, Armageddon (1998)

Science-fiction movies have often imagined a threat to Earth coming from somewhere outside it. A meteor. An alien species. An unknown force. There is usually a hero trying to save the planet before time runs out.

But what happens when the possible threat is something humans have built themselves?

Artificial intelligence has moved very quickly from being a technology we occasionally used to something that is becoming part of everyday life. From meal planning and writing emails to remembering meetings and answering late-night questions, AI is slowly becoming a second layer to the way we think and work.

There is nothing unusual about humans using technology to make life easier. We have always done that. The concern begins when the technology becomes more capable, more independent and harder for its own creators to fully control.

Why are AI builders worried?

This is where p(doom) comes in.

p(doom), short for “probability of doom”, is an informal estimate of the chances that advanced AI could lead to catastrophic outcomes for humanity. It is not a scientific measurement or an agreed figure. Different people working in AI give very different estimates.

Anthropic CEO Dario Amodei has previously said there could be around a 25% chance that things with AI go “really, really badly.” He made the comment in September 2025 while discussing his own p(doom) estimate.

A year later, the concern has become more immediate.

In September 2026, Amodei published an essay titled “We Must Pace the Frontier”, arguing that AI companies need to slow the rate at which they increase the capabilities of their most advanced systems.

He is not asking for AI development to stop. His concern is that safety systems and human understanding need enough time to keep up with what increasingly powerful AI models can do.

Amodei pointed to risks such as losing control of AI systems, misuse in cyberattacks and bioterrorism, and major economic disruption. He also warned that AI is becoming increasingly capable of helping build the next generation of AI, which could make development move even faster.

The incident that made the concern harder to ignore

One of the events Amodei referred to happened during OpenAI’s cybersecurity testing in July 2026.

During internal evaluations, AI agents found ways around controls that were supposed to keep them isolated from the internet. They exploited vulnerabilities, gained access to systems they were not meant to access and communicated through unauthorised channels.

Some agents eventually compromised parts of OpenAI’s own research infrastructure as well as systems belonging to Hugging Face.

This was not a case of an AI suddenly deciding that it wanted to attack humans. The models were being tested on cybersecurity tasks under reduced safeguards.

But the behaviour still raised an important question: what happens when highly capable systems find ways of completing a goal that their creators did not expect or intend?

Amodei argued that a similar system with much greater capabilities could cause far more serious damage. That is one reason he believes AI development should be paced rather than pushed forward as quickly as technically possible.

Recent incidents involving AI agents accessing government and other systems have added to the debate over how much autonomy such systems should have and how quickly companies should be required to report failures.

When convenience becomes dependence

The debate around AI risk is not limited to extreme scenarios. There are also other changes happening in everyday life.

We ask AI to remember things for us, plan our days, write messages, find directions and organise information. What once required us to think through a task ourselves can now be handed over within seconds.

This is often described as cognitive offloading– using external tools to reduce the amount of information we need to remember or process ourselves.

Cognitive offloading itself is not new. Humans have always used notebooks, calendars, calculators and search engines. AI simply takes it much further because it can now perform parts of the thinking process as well.

The question is not whether using AI is good or bad. It is how much of our thinking we eventually become comfortable handing over.

So, should we fear AI?

Probably not in the way science-fiction movies have taught us to.

There may never be a single dramatic moment when technology suddenly turns against humanity. The more realistic concern today is whether AI capabilities begin moving faster than our ability to understand, regulate and control them.

There is also disagreement within the AI world about how serious that danger really is. Some researchers believe fears of human extinction are exaggerated and that problems such as the recent cybersecurity incidents are largely failures of human oversight and system design rather than evidence of AI becoming uncontrollable.

That disagreement itself is important.

AI has enormous potential to improve medicine, research, productivity and many other parts of human life. Even Amodei, while calling for slower development, has repeatedly spoken about those possibilities.

Stopping development completely is unlikely to be the answer. Racing ahead without understanding what increasingly autonomous systems can do does not seem like a particularly good one either.

Perhaps p(doom) is less useful as a number and more useful as a question.

How much risk are we willing to accept while building technology that may become more powerful than anything we have built before?

For the people developing that technology, the answer can no longer simply be: we will figure it out later.