dw2

Should we be seriously concerned that forthcoming new software systems might cause a global catastrophe?

Or are there, instead, good reasons to dismiss any such concern?

(image by Midjourney)

It’s a vitally important public debate. Alas, this debate is bedevilled by false turnings.

For example, dismissers often make claims with this form:

• The argument for being concerned assumes that such-and-such a precondition holds
• But that precondition is suspect (or false)
• Therefore the concern can be dismissed.

Here’s a simple example – which used to be common, though it appears less often these days:

• The argument for being concerned assumes that Moore’s Law will hold for the next three decades
• But Moore’s Law is slowing down
• Therefore the concern can be dismissed.

Another one:

• The argument for being concerned assumes that deep learning systems understand what they’re talking about
• But by such-and-such a definition of understanding, these systems lack understanding
• (They’re “just stochastic parrots”)
• Therefore the concern can be dismissed.

Or a favourite:

• You call these systems AI, meaning they’re (supposedly) artificially intelligent
• But by such-and-such a definition of intelligence, these systems lack intelligence
• Therefore the concern can be dismissed.

Perhaps the silliest example:

• Your example of doom involves a software system that is inordinately obsessed with paperclips
• But any wise philosopher would design an AI that has no such paperclip obsession
• Therefore the concern can be dismissed.

My conclusion: those of us who are seriously concerned about the prospects of a software-induced global catastrophe should clarify what are the minimum conditions that would give rise to such a catastrophe.

To be clear, these minimum conditions don’t include the inexorability of Moore’s Law. Nor the conformance of software systems to particular academic models of language understanding. Nor that a fast take-off occurs. Nor that the software system becomes sentient.

Here’s my suggestion of these minimum conditions:

1. A software system that can influence, directly or indirectly (e.g. by psychological pressure) what happens in the real world
2. That has access, directly or indirectly, to physical mechanisms that can seriously harm humans
3. That operates in ways which we might fail to understand or anticipate
4. That can anticipate actions humans might take, and can calculate and execute countermeasures
5. That can take actions quickly enough (and/or stealthily enough) to avoid being switched off or reconfigured before catastrophic damage is done.

Even more briefly: the software system operates outside our understanding and outside our control, with potential devastating power.

I’ve chosen to use the term “software” rather than “AI” in order to counter a whole posse of dismissers right at the beginning of the discussion. Not even the smuggest of dismissers denies that software exists and can, indeed, cause harm when it contains bugs, is misconfigured, is hacked, or has gaps in its specification.

Critically, note that software systems often do have real-world impact. Consider the Stuxnet computer worm that caused centrifuges to speed up and destroy themselves inside Iran’s nuclear enrichment facilities. Consider the WannaCry malware that disabled critical hospital equipment around the world in 2017.

Present-day chatbots have already influenced millions of people around the world, via the ideas emerging in chat interactions. Just as people can make life-changing decisions after talking with human therapists or counsellors, people are increasingly taking life-changing decisions following their encounters with the likes of ChatGPT.

Software systems are already involved in the design and operation of military weapons. Presently, humans tend to remain “in the loop”, but military leaders are making the case for humans instead being just “on the loop”, in order for their defence systems to be able to move “at the speed of relevance”.

So the possibility of this kind of software shouldn’t be disputed.

It’s not just military weapons where the potential risk exists. Software systems can be involved with biological pathogens, or with the generation of hate-inducing fake news, or with geoengineering. Or with the manipulation of parts of our infrastructure that we currently only understand dimly, but which might turn out to be horribly fragile, when nudged in particular ways.

Someone wanting to dismiss the risk of software-induced global catastrophe therefore needs to make one or more of the following cases:

1. All such software systems will be carefully constrained – perhaps by tamperproof failsafe mechanisms that are utterly reliable
2. All such software systems will remain fully within human understanding, and therefore won’t take any actions that surprise us
3. All such software systems will fail to develop an accurate “theory of mind” and therefore won’t be able to anticipate human countermeasures
4. All such software systems will decide, by themselves, to avoid humans experiencing significant harm, regardless of which other goals are found to be attractive by the alien mental processes of that system.

If you still wish to dismiss the risk of software global catastrophe, which of these four cases do you wish to advance?

Or do you have something different in mind?

And can you also be sure that all such software systems will operate correctly, without bugs, configuration failures, gaps in their specification, or being hacked?

Case 2, by the way, includes the idea that “we humans will merge with software and will therefore remain in control of that software”. But in that case, how confident are you that:

• Humans can speed up their understanding as quickly as the improvement rate of software systems that are free from the constraints of the human skull?
• Any such “superintelligent” humans will take actions that avoid the same kinds of global catastrophe (after all, some of the world’s most dangerous people have intelligence well above the average)?

Case 4 includes the idea that at least some aspects of morality are absolute, and that a sufficiently intelligent piece of software will discover these principles. But in that case, how confident are you that:

• The software will decide to respect these principles of morality, rather than (like many humans) disregarding them in order to pursue some other objectives?
• That these fundamental principles of morality will include the preservation and flourishing of eight billion humans (rather than, say, just a small representative subset in a kind of future “zoo”)?

Postscript: My own recommendations for how to address these very serious risks are in The Singularity Principles. Spoiler alert: there are no magic bullets.