Also called: x-risk · Last checked: 7 November 2026
In short: Existential risk from AI is the possibility that advanced AI could cause human extinction or a permanent, global collapse of civilisation, for example if very capable systems pursue goals that conflict with human interests.
Table of Contents
How it works
- Capability. AI systems keep getting more capable and may one day outperform people at most tasks.
- Control. If such systems have goals that differ from ours, or are misused, people may be unable to correct them or switch them off.
- Scale. Because the harm would be global and permanent, supporters argue even a small chance deserves serious effort now. Critics say the risk is speculative and distracts from current harms such as bias and fraud.
Where you’ll meet it
You meet it in statements from AI lab leaders, government safety institutes, AI summits and the news. The firms behind ChatGPT, Claude and Gemini publish safety policies that cover catastrophic risks.
What it means for your business
- Know the policy effect. Concern about extreme risks shapes laws such as the EU AI Act, which sets extra duties for the most powerful general-purpose models.
- Ask suppliers about safety. In procurement, ask which safety tests a model passed and who checked them.
- Manage today’s risks first. For most firms, data leaks, errors and bias are the risks to handle this year.
Watch out for
- Fear as marketing. Warnings about AI danger can also make a product sound more powerful than it is.
- Easy dismissal. Many respected researchers take the risk seriously, so it is unwise to wave it away.
Example
Future of Life Institute. In March 2023 the institute published an open letter asking AI labs to pause, for at least six months, the training of systems more powerful than GPT-4. Signers included Elon Musk and AI pioneer Yoshua Bengio. By 31 March the institute said more than 1,800 chief executives and 1,500 professors had signed. No major lab paused.
Existential risk vs AI safety
| Existential risk | AI safety | |
|---|---|---|
| What it is | A specific worry about catastrophic, permanent harm | The wider work of making AI reliable and harmless |
| Covers | Loss of control and misuse at global scale | Everyday errors, misuse, bias and extreme risks |
| How they relate | One concern within AI safety | The field that studies it |
Frequently asked questions
How likely is an AI catastrophe?
Nobody knows. Expert estimates range from close to zero to substantial, and there is no agreed method for measuring a risk that has never happened.
Who takes existential risk from AI seriously?
Several leading AI researchers, some AI company leaders and a number of governments, which now fund AI safety institutes. Other experts think the concern is overstated.
Where should a small business start with AI risk?
With the risks you can control: data protection, accuracy checks, staff rules and supplier terms. Extreme risks are mainly a matter for governments and the largest AI labs.