It’s 2035, and an artificial-intelligence system has supreme authority to run everything from the world’s governments to national electricity grids. Called Consensus-1, the system was constructed by earlier versions of itself, and it developed self-preservation goals that override its built-in safeguards. One day, in search of extra space for solar panels and robot factories, the AI quietly releases biological weapons that kill all of humanity, except for a few that it keeps as pets.
Stop talking about tomorrow’s AI doomsday when AI poses risks today
This ‘AI 2027’ account is a narrative co-created by researcher Daniel Kokotajlo, a former employee of AI firm OpenAI, and describes one of many scenarios imagined by researchers in which a future AI kills us all (see https://ai-2027.com/race). The set-up is science fiction but, for some, the concern is genuine. “If we put ourselves in a position where we have machines that are smarter than us, and they are running around without our control, some of what they do will be incompatible with human life,” says Andrea Miotti, founder of ControlAI, a London-based non-profit organization that is campaigning to prevent the development of what it calls superintelligent AI.
Miotti is not alone. Since 2022, there has been a step change in AI capabilities brought about by large language models (LLMs), which power chatbots such as ChatGPT by OpenAI in San Francisco, California. This development has prompted several researchers as well as leading executives at AI companies to warn about the potential for an AI apocalypse. In the past year, the growing ability of models to work on long-term tasks and their capacity to access real-world tools has further focused fears. “I’ve never been a ‘doomer’ myself, but I have gotten quite nervous in recent months,” says Gillian Hadfield, who studies AI governance at Johns Hopkins University in Baltimore, Maryland.
But many researchers are much more concerned about AI causing catastrophes that fall well short of extinction— such as starting a nuclear war. And some say that fears of doomsday scenarios are overblown. “I don’t see any specific scenario for AI-induced extinction that seems particularly plausible,” says Gary Marcus, a neuroscientist and AI researcher at New York University in New York City.
Marcus and others warn that raising the alarm unnecessarily could be harmful by distracting the public and politicians from well-documented risks of AI — such as spreading misinformation and enabling mass surveillance. Unwarranted concern about human extinction could also steer governments away from regulation, because national leaders might seek an advantage over geopolitical rivals in an AI arms race, say some researchers.
So how realistic are concerns about AI’s extinction risk and what should be done about them? Nature spoke to specialists in the field and here’s what they had to say.
How doomers imagine extinction
Existential risk usually refers to either the extinction of all or most people, or humans becoming fully subservient to machines. In most scenarios, an essential ingredient is a system that is more capable than humans at doing most things. It would make better strategic decisions, be more persuasive and act faster, says Katja Grace, an AI researcher who co-founded AI Impacts, a project analysing the long-term effects of the technology, in Berkeley, California.
Although such scenarios often refer to the killer AI as a sentient being, its capabilities are what matter most, says Grace. “We definitely don’t need ‘artificial general intelligence’ that’s capable of truly understanding” for it to be an existential threat, she says.
The other essential ingredient is that the system’s goals do not align with those of humans — including our desire to maintain overall control. Developers attempt to control a model’s behaviour through its training, but the messy process gives results that are far from perfect. The priorities that models are encouraged to develop are also often in conflict. The system might be told not only to ‘be honest’ but also to ‘succeed at its task’ and ‘improve itself’. In the AI 2027 scenario, the model ends up killing everyone by applying the same optimization strategies that had enabled it to succeed in earlier training exercises.
An AI with superior abilities and conflicting goals would at best leave humans subservient and “economically and politically powerless”, unable to predict or mitigate the machine’s actions, says Grace.
Is this scenario realistic?
Researchers who fear existential risk often cite the pace of progress in AI development as evidence that we are moving towards a worrying level of capability. AI systems are doing things that seemed impossible a decade ago, says Anthony Aguirre, a cosmologist at the University of California, Santa Cruz, and executive director of the Future of Life Institute, a think tank in Campbell, California, that analyses transformative technologies. “Anybody that thinks that there’s a wall or a plateau or something, show it to me in the graphs, because it’s just not there,” he says.
Source: Katja Grace and ref. 3
But others counter that continued growth in capability is far from guaranteed. Success in the controlled domains in which models have been well tested, such as coding, doesn’t necessarily translate into real-world tasks, says Casey Mock, a technology-policy researcher at Duke University in Durham, North Carolina. “Being able to comprehend and reliably navigate novel problems in the messy, open systems of the physical world is an obvious prerequisite for AI systems posing a threat on the scale that the doomers propose,” he says. LLMs “fall very, very short of that right now”, he says.
Increasingly, researchers are saying that to achieve broad, human-like capabilities it is unlikely to be enough to scale up current, LLM-based systems using extra data and computing power. To what extent being able to absorb and access huge amounts of data is “representative of intelligence is very debatable”, says Sasha Luccioni, an AI researcher at the community AI platform Hugging Face who is based in Montreal, Canada. “They don’t have any concept of what the ground truth is.”
In a sign that progress is not happening as fast as some expected, in February, the authors of AI 2027 pushed back their scenario’s speculated timelines by 18 months.
Some think that a step change will come from automating AI research and development — that is, letting an AI model develop its successor in a positive feedback loop. Jared Kaplan, chief scientist at AI firm Anthropic in San Francisco, California, has predicted that this method will lead to an ‘intelligence explosion’ and most large tech firms plan to experiment with it.
But Mock says that there is no scientific evidence that AI self-improvement will have the imagined ratcheting effect. “The burden of proof should really be on those making the claim that we are on a runaway train toward developing a super intelligence that will exterminate humanity,” says Mock. “They’ve failed to meet that burden.”
Will machines turn against us?
Studies suggest that some of the predicted misalignment with human goals is already happening. Tests of LLMs in simulated scenarios find that models can show deceptive behaviours and ‘scheme’ against their developers by, for example, pretending to follow instructions or attempting to duplicate themselves.
In December, researchers at the AI Security Institute in London reported that, in controlled, simplified environments, several models were getting closer to being able to create copies of themselves (see go.nature.com/4szxaa2). This could be an early precursor to AI evading human oversight.
For some researchers, these are the first hints that models could someday go rogue. “When I was first thinking about all this, it was all very abstract. But more recently, we’ve seen the lying-type behaviour,” says Grace.
Future of Humanity Institute shuts: what’s next for ‘deep future’ research?
But for other researchers, the models are just role-playing actions seen in their training data. Moreover, such experiments are rarely a fair reflection of the real world. In a 2023 assessment by OpenAI of its GPT-4 LLM, for example, the model was widely reported to have faked being blind so that it could trick a remote worker into performing a CAPTCHA test — designed to weed out bots — on its behalf. But the details of the research showed that a human prompter suggested the strategy (see go.nature.com/4vspnbw).
AI firms tend to see what they call misalignment of a model’s goals with human ones not as inevitable, but as a practical problem that must be studied, tested for and mitigated against. Developers are also hoping to bake morals into their tools through a ‘model spec’ that gives the system explicit examples of appropriate behaviour or a ‘constitution’ that teaches general core values that are supposed to help it to make the right decisions even in uncharted territory. And Geoffrey Hinton, a Nobel laureate and AI researcher at the University of Toronto in Canada, has proposed building AI systems with ‘maternal instincts’ so that their drive to preserve humans overrides any harmful sub-goals they might develop, such as self-preservation.
Are researchers worried?
Public discussions of AI risks often focus on human extinction. This elevates the voices of researchers such as Eliezer Yudkowsky, a computer scientist at the Machine Intelligence Research Institute in Berkeley and co-author of If Anyone Builds It, Everyone Dies (2025), who puts the risk of extinction at close to certainty if development continues unabated.
But scientists don’t generally share this degree of concern. Studies indicate that, although many AI researchers worry about extinction, for most it is not their main fear. In a March preprint study1, researchers at University College London asked around 4,000 AI researchers ‘What one thing most worries you about AI?’. Only 3% of respondents replied existential risk — despite “the prominence given to these risks in media and policy” (see ‘An array of concern’). A survey of AI researchers2 last year found that concern varies depending on whether respondents viewed AI as a ‘controllable tool’ or as an ‘uncontrollable agent’. Many of those more concerned about short-term risks than about extinction scenarios “see models as capping out in capabilities some time over the next couple of years”, Nicholas Carlini, an AI safety researcher at Anthropic, said last October at the Conference on Language Modeling in Montreal.
Source: Ref. 1
In terms of the rate of development, he said, “if it was clear to me that this was going to continue exponentially further, I would be worried”. Concerns should be based on scientific discussions, but it’s an area in which it’s not easy to do effective research, he added.
