calculation and qualitative, value-based judgement.[264]
It has been argued AI will become so powerful that humanity may irreversibly lose control of it. This
could, as physicist Stephen Hawking stated, "spell the end of the human race".[265] This scenario has been
common in science fiction, when a computer or robot suddenly develops a human-like "self-awareness"
(or "sentience" or "consciousness") and becomes a malevolent character.[q] These sci-fi scenarios are
misleading in several ways.
First, AI does not require human-like sentience to be an existential risk. Modern AI programs are given
specific goals and use learning and intelligence to achieve them. Philosopher Nick Bostrom argued that if
one gives almost any goal to a sufficiently powerful AI, it may choose to destroy humanity to achieve it
(he used the example of a paperclip factory manager).[267] Stuart Russell gives the example of household
robot that tries to find a way to kill its owner to prevent it from being unplugged, reasoning that "you
can't fetch the coffee if you're dead."[268] In order to be safe for humanity, a superintelligence would have
to be genuinely aligned with humanity's morality and values so that it is "fundamentally on our side".[269]
Second, Yuval Noah Harari argues that AI does not require a robot body or physical control to pose an
existential risk. The essential parts of civilization are not physical. Things like ideologies, law,
government, money and the economy are built on language; they exist because there are stories that
billions of people believe. The current prevalence of misinformation suggests that an AI could use
language to convince people to believe anything, even to take actions that are destructive.[270]
The opinions amongst experts and industry insiders are mixed, with sizable fractions both concerned and
unconcerned by risk from eventual superintelligent AI.[271] Personalities such as Stephen Hawking, Bill
Gates, and Elon Musk,[272] as well as AI pioneers such as Yoshua Bengio, Stuart Russell, Demis
Hassabis, and Sam Altman, have expressed concerns about existential risk from AI.
In May 2023, Geoffrey Hinton announced his resignation from Google in order to be able to "freely speak
out about the risks of AI" without "considering how this impacts Google."[273] He notably mentioned
risks of an AI takeover,[274] and stressed that in order to avoid the worst outcomes, establishing safety
guidelines will require cooperation among those competing in use of AI.[275]
In 2023, many leading AI experts endorsed the joint statement that "Mitigating the risk of extinction from
AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear
war".[276]
Some other researchers were more optimistic. AI pioneer Jürgen Schmidhuber did not sign the joint
statement, emphasising that in 95% of all cases, AI research is about making "human lives longer and
healthier and easier."[277] While the tools that are now being used to improve lives can also be used by
bad actors, "they can also be used against the bad actors."[278][279] Andrew Ng also argued that "it's a
mistake to fall for the doomsday hype on AI—and that regulators who do will only benefit vested
interests."[280] Yann LeCun "scoffs at his peers' dystopian scenarios of supercharged misinformation and
even, eventually, human extinction."[281] In the early 2010s, experts argued that the risks are too distant in
Existential risk