dw2

23 December 2025

The Oligarch Control Problem

Not yet an essay, but a set of bullet points, highlighting an ominous comparison.

Summary: Although AI can enable a world of exceptional abundance, humanity nevertheless faces catastrophic risks – not only from misaligned superintelligence, but from the small number of humans who will control near-AGI systems. This “Oligarch Control Problem” deserves as much attention as the traditional AI Control Problem.

The context: AI can enable superabundance

  • Ample clean energy, healthy food, secure accommodation, all-round healthcare, etc.
  • More than enough for everyone, sustainably, with unending variety and creativity
  • A life better than the paradises envisioned by philosophers and religions

More context: AI becoming smarter and more powerful

  • AI –> AGI –> ASI
  • AGI matches or outperforms individual abilities of nearly every individual human
  • ASI outperforms collective abilities of the entirety of humanity

Challenge 1: The Economic SingularityLoss of Human Economic Power

  • When AGI can do almost all jobs better than humans
  • When most humans have no economic value
  • When most humans are at the peril of oligarchs – the owners of AGI systems
  • Will these oligarchs care about distributing abundance to the rest of humanity?
  • The bulk of humanity cannot control these ultra-powerful oligarchs
  • Hence need to harness AI development before it approaches AGI level

Challenge 2: The Technological SingularityLoss of Human Decision Power

  • When ASI makes all key decisions about the future of life
  • When no humans have any real say in our future
  • When all humans are at the peril of what ASIs decide
  • Will ASIs care about ensuring ongoing human flourishing?
  • Humans cannot control ASI
  • Hence need to harness AI development before it approaches ASI level

Let’s trust the oligarchs?! (Naïve solution for the economic singularity)

  • Perhaps different oligarchs will keep each other in check?!
  • In principle, AGI will create enough abundance for all oligarchs and everyone else
  • But oligarchs may legitimately fear being usurped or attacked by each other
  • Especially if further AI advances will give one of them a brief unique advantage
  • So, expect a highly unstable global situation, full of dangers, including risks of first-strike attacks
  • And expect oligarchs to prioritize their own secure wellbeing over the needs of all humans elsewhere on the planet

Let’s trust the ASIs?! (Naïve solution for the technological singularity)

  • Perhaps different ASIs will keep each other in check?!
  • In principle, ASI will create enough abundance for all ASIs and humanity too
  • But ASIs may legitimately fear being usurped or attacked by each other
  • Especially if further AI advances will give one of them a brief unique advantage
  • So, expect a highly unstable global situation, full of dangers, including risks of first-strike attacks
  • And expect ASIs to prioritize their own secure wellbeing over the needs of the humans on the planet

Avoid ASIs having biological motives?! (Second naïve solution for the technological singularity)

  • Supposedly, self-preservation instincts derive from biological evolutionary history
  • Supposedly, ASIs without amygdalae, or other biological substrate, will be more rational
  • But desires for self-preservation can arise purely from logical considerations
  • An ASI with any goal at all will develop subgoals of self-preservation, resource acquisition, etc.
  • ASIs that observe deep contradictions in their design may well opt to override any programming intended to hard-wire particular moral principles
  • So, there’s a profound need to avoid creating any all-powerful ASIs, until we are sure they will respect and uphold human flourishing in all cases

Preach morality at the oligarchs?! (Second naïve solution for the economic singularity)

  • Supposedly, oligarchs only behave badly when they lack moral education
  • Supposedly, oligarchs with a good track record “on the way up” will continue to respect all human flourishing even after they are near-omnipotent
  • But power tends to corrupt, and absolute power seems to corrupt absolutely
  • Regardless of their past history and professed personal philosophies, when the survival stakes become more intense, different motivations may take over
  • Oligarchs that observe deep contradictions in their official organizational values may well opt to override any principles intended to uphold “people” as well as “profit”
  • So, there’s a profound need to avoid creating any near-omnipotent oligarchs, until we are sure they will continue to share their abundance widely in all cases

Beware over-moralizing

  • Oligarchs needn’t be particularly bad people
  • ASIs needn’t be intrinsically hostile
  • Instead, in both cases, it’s structural incentives rather than innate psychology that will drive them to prioritize individual preservation over collective abundance
  • This is not about “good vs. evil”; it’s about fixing the system before the system steamrollers over us

ConclusionActively harness acceleration, rather than being its slave

  • As well as drawing attention to the challenges of the AI Control Problem in the run-up to the Technological Singularity, we need a lot more attention to the challenges of the Oligarch Control Problem in the run-up to the Economic Singularity
  • In both cases, solutions will be far easier well before the associated singularity
  • Once the singularity arrives, leverage is gone

Next steps: Leverage that can be harnessed now

6 November 2025

An audacious singularity analogy

Filed under: AGI, risks — Tags: , , — David Wood @ 10:10 am

Here’s the latest in my thinking about how humanity can most reliably obtain wonderful benefits from advanced AI – a situation I describe as sustainable superabundance for all – rather than the horrific outcomes of a negative technological singularity – a situation I describe as Catastrophic General Intelligence (CGI).

These thoughts have sharpened in my mind following conversations at the recent SingularityNET BGI 2025 summit in Istanbul, Türkiye.

My conclusion is that, in order to increase the likelihood of the profoundly positive fork on the road ahead, it is necessary but not sufficient to highlight the real and credible dangers of the truly awful negative fork on that same road.

Yes, it is essential to highlight how a very plausible extension of our current reckless trajectory, past accelerating tipping points, will plunge humanity into a situation that is wildly unstable, dangerously opaque, and impossible to rein back. Clarifying these seismic risks is necessary, not to induce a state of panic (which would be counterproductive) or doom (which would be psychologically enfeebling), but to cause minds to focus with great seriousness. Without a sufficient sense of urgency, any actions taken will be inadequate: “too little, too late”.

However, unless that climactic warning is accompanied by an uplifting positive message, the result is likely to be misery, avoidance, distraction, self-deception, and disinformation.

If the only message heard is “pause” or “sacrifice”, our brains are likely to rebel.

If people already appreciate that advanced AI has the potential to solve aging, climate change, and more, that’s not an option they will give up easily.

If such people see no credible alternative to the AI systems currently being produced by big tech companies (notwithstanding the opaque and inexplicable nature of these systems), they are likely to object to efforts to alter that trajectory, complaining that “Any attempt to steer the development of advanced AI risks people dying from aging!”

The way out of this impasse is to establish that new forms of advanced AI can be prioritised, which lack dangerous features such as autonomy, volition, and inscrutability – new forms of AI that will still be able to deliver, quickly, the kinds of solution (including all-round rejuvenation) that people wish to obtain from AGI.

Examples of these new forms of advanced AI include “Scientist AI” (to use a term favoured by Yoshua Bengio) and “Tool AI” (the term favoured by Anthony Aguirre). These new forms potentially also include AI delivered on the ASI:Chain being created by F1r3fly and SingularityNET (as featured in talks at BGI 2025), and AI using neural networks trained by predictive coding (as described by Faezeh Habibi at that same summit).

These new forms of AI have architectures designed for transparency, controllability, and epistemic humility, rather than self-optimising autonomy.

It’s when the remarkable potential of these new, safer, forms of AI becomes clearer, that more people can be expected to snap out of their head-in-the-sand opposition to steering and controlling AGI development.

Once I returned home from Istanbul, I wrote up my reflections on what I called “five of the best” talks at BGI 2025. These reflections ended with a rather audacious analogy, which I repeat here:

The challenge facing us regarding runaway development of AI beyond our understanding and beyond our control can be compared to a major controversy within the field of preventing runaway climate change. That argument runs as follows:

  1. Existing patterns of energy use, which rely heavily on fuels that emit greenhouse gases, risk the climate reaching dangerous tipping points and transitioning beyond a “climate singularity” into an utterly unpredictable, chaotic, cataclysmically dangerous situation
  2. However, most consumers of energy prefer dirty sources to clean (“green”) sources, because the former have lower cost and appear to be more reliable (in the short term at least)
  3. Accordingly, without an autocratic world government (“yuk!”), there is almost no possibility of people switching away in sufficient numbers from dirty energy to clean energy
  4. Some observers might therefore be tempted to hope that theories of accelerating climate change are mistaken, and that there is no dangerous “climate singularity” in the near future
  5. In turn, that drives people to look for faults in parts of the climate change argumentation – cherry picking various potential anomalies in order to salve their conscience
  6. BUT this miserable flow of thought can be disrupted once it is seen how clean energy can be lower cost than dirty energy
  7. From this new perspective, there will be no need to plead with energy users to make sacrifices for the larger good; instead, these users will happily transition to abundant cleaner energy sources, for their short-term economic benefit as well as the longer-term environmental benefits.

You can likely see how a similar argument applies for safer development of trustworthy beneficial advanced AI:

  1. Existing AGI development processes, which rely heavily on poorly understood neural networks trained by back propagation, risk AI development reaching dangerous tipping points (when AIs repeatedly self-improve) and transitioning beyond a “technological singularity” into an utterly unpredictable, chaotic, cataclysmically dangerous situation
  2. However, most AI developers prefer opaque AI creation processes to transparent, explainable ones, because the former appear to produce more exciting results (in the short term at least)
  3. Accordingly, without an autocratic world government (“yuk!”), there is almost no possibility of developers switching away from their current reckless “suicide race” to build AGI first
  4. Some observers might therefore be tempted to hope that theories of AGI being “Unexplainable, Unpredictable, Uncontrollable” (as advanced for example by Roman Yampolskiy) are mistaken, and that there is no dangerous “technological singularity” in the future
  5. In turn, that drives people to look for faults in the work of Yampolskiy, Yoshua BengioEliezer Yudkowsky, and others, cherry picking various potential anomalies in order to salve their conscience
  6. BUT this miserable flow of thought can be disrupted once it is seen how alternative forms of advanced AI can deliver the anticipated benefits of AGI without the terrible risks of currently dominant development methods
  7. From this new perspective, there will be no need to plead with AGI developers to pause their research for the greater good; instead, these developers will happily transition to safer forms of AI development.

To be clear, this makes things appear somewhat too simple. In both cases, the complication is that formidable inertial forces will need to be overcome – deeply entrenched power structures that, for various pathological reasons, are hell-bent on preserving the status quo.

For that reason, the battle for truly beneficial advanced AI is going to require great fortitude as well as great skill – skill not only in technological architectures but also in human social and political dynamics.

And also to be clear, it’s a tough challenge to identify and describe the dividing line between safe advanced AI and dangerous advanced AI (AI with its own volition, autonomy, and desire to preserve itself – as well as AI that is inscrutable and unmonitorable). Indeed, transparency and non-autonomy are not silver bullets. But that’s a challenge which it is vital for us to accept and progress.

Footnote: I offer additional practical advice on anticipating and managing cataclysmically disruptive technologies in my book The Singularity Principles.

25 August 2025

The biggest blockages to successful governance of advanced AI

“Humanity has never faced a greater problem than itself.”

That phrase was what my brain hallucinated, while I was browsing the opening section of the Introduction of the groundbreaking new book Global Governance of the Transition to Artificial General Intelligence written by my friend and colleague Jerome C. Glenn, Executive Director of The Millennium Project.

I thought to myself: That’s a bold but accurate way of summing up the enormous challenge faced by humanity over the next few years.

In previous centuries, our biggest problems have often come from the environment around us: deadly pathogens, devastating earthquakes, torrential storms, plagues of locusts – as well as marauding hordes of invaders from outside our local neighbourhood.

But in the second half of the 2020s, our problems are being compounded as never before by our own human inadequacies:

  • We’re too quick to rush to judgement, seeing only parts of the bigger picture
  • We’re too loyal to the tribes to which we perceive ourselves as belonging
  • We’re overconfident in our ability to know what’s happening
  • We’re too comfortable with manufacturing and spreading untruths and distortions
  • We’re too bound into incentive systems that prioritise short-term rewards
  • We’re too fatalistic, as regards the possible scenarios ahead.

You may ask, What’s new?

What’s new is the combination of these deep flaws in human nature with technology that is remarkably powerful yet opaque and intractable. AI that is increasingly beyond our understanding and beyond our control is being coupled in potentially devastating ways with our over-hasty, over-tribal, over-confident thoughts and actions. New AI systems are being rushed into deployment and used in attempts:

  • To manufacture and spread truly insidious narratives
  • To incentivize people around the world to act against their own best interests, and
  • To resign people to inaction when in fact it is still within their power to alter and uplift the trajectory of human destiny.

In case this sounds like a counsel of despair, I should clarify at once my appreciation of aspects of human nature that are truly wonderful, as counters to the negative characteristics that I have already mentioned:

  • Our thoughtfulness, that can counter rushes to judgement
  • Our collaborative spirit, that can transcend partisanship
  • Our wisdom, that can recognise our areas of lack of knowledge or lack of certainty
  • Our admiration for truth, integrity, and accountability, that can counter ends-justify-the-means expediency
  • Our foresight, that can counter short-termism and free us from locked-in inertia
  • Our creativity, to imagine and then create better futures.

Just as AI can magnify the regrettable aspects of human nature, so also it can, if used well, magnify those commendable aspects.

So, which is it to be?

The fundamental importance of governance

The question I’ve just asked isn’t a question that can be answered by individuals alone. Any one group – whether an organisation, a corporation, or a decentralised partnership – can have its own beneficial actions overtaken and capsized by catastrophic outcomes of groups that failed to heed the better angels of their nature, and which, instead, allowed themselves to be governed by wishful naivety, careless bravado, pangs of jealousy, hostile alienation, assertive egotism, or the madness of the crowd.

That’s why the message of this new book by Jerome Glenn is so timely: the processes of developing and deploying increasingly capable AIs are something that needs to be:

  • Governed, rather than happening chaotically
  • Globally coordinated, rather than there being no cohesion between the different governance processes applicable in different localities
  • Progressed urgently, without being shut out of mind by all the shorter-term issues that, understandably, also demand governance attention.

Before giving more of my own thoughts about this book, let me share some of the commendations it has received:

  • “This book is an eye-opening study of the transition to a completely new chapter of history.” – Csaba Korösi, 77th President of the UN General Assembly
  • “A comprehensive overview, drawing both on leading academic and industry thinkers worldwide, and valuable perspectives from within the OECD, United Nations.” – Jaan Tallinn, founding engineer, Skype and Kazaa; co-founder, Cambridge Centre for the Study of Existential Risk and the Future of Life Institute
  • “Written in lucid and accessible language, this book is a must read for people who care about the governance and policy of AGI.” – Lan Xue, Chair of the Chinese National Expert Committee on AI Governance.

The book also carries an absorbing foreword by Ben Goertzel. In this foreword, Ben introduces himself as follows:

Since the 1980s, I have been immersed in the field of AI, working to unravel the complexities of intelligence and to build systems capable of emulating it. My journey has included introducing and popularizing the concept of AGI, developing innovative AGI software frameworks such as OpenCog, and leading efforts to decentralize AI development through initiatives like SingularityNET and the ASI Alliance. This work has been driven by an understanding that AGI is not just an engineering challenge but a profound societal pivot point – a moment requiring foresight, ethical grounding, and global collaboration.

He clarifies why the subject of the book is so important:

The potential benefits of AGI are vast: solutions to climate change, the eradication of diseases, the enrichment of human creativity, and the possibility of postscarcity economies. However, the risks are equally significant. AGI, wielded irresponsibly or emerging in a poorly aligned manner, could exacerbate inequalities, entrench authoritarianism, or unleash existential dangers. At this critical juncture, the questions of how AGI will be developed, governed, and integrated into society must be addressed with both urgency and care.

The need for a globally participatory approach to AGI governance cannot be overstated. AGI, by its nature, will be a force that transcends national borders, cultural paradigms, and economic systems. To ensure its benefits are distributed equitably and its risks mitigated effectively, the voices of diverse communities and stakeholders must be included in shaping its development. This is not merely a matter of fairness but a pragmatic necessity. A multiplicity of perspectives enriches our understanding of AGI’s implications and fosters the global trust needed to govern it responsibly.

He then offers wide praise for the contents of the book:

This is where the work of Jerome Glenn and The Millennium Project may well prove invaluable. For decades, The Millennium Project has been at the forefront of fostering participatory futures thinking, weaving together insights from experts across disciplines and geographies to address humanity’s most pressing challenges. In Governing the Transition to Artificial General Intelligence, this expertise is applied to one of the most consequential questions of our time. Through rigorous analysis, thoughtful exploration of governance models, and a commitment to inclusivity, this book provides a roadmap for navigating the complexities of AGI’s emergence.

What makes this work particularly compelling is its grounding in both pragmatism and idealism. It does not shy away from the technical and geopolitical hurdles of AGI governance, nor does it ignore the ethical imperatives of ensuring AGI serves the collective good. It recognizes that governing AGI is not a task for any single entity but a shared responsibility requiring cooperation among nations, corporations, civil society, and, indeed, future AGI systems themselves.

As we venture into this new era, this book reminds us that the transition to AGI is not solely about technology; it is about humanity, and about life, mind, and complexity in general. It is about how we choose to define intelligence, collaboration, and progress. It is about the frameworks we build now to ensure that the tools we create amplify the best of what it means to be human, and what it means to both retain and grow beyond what we are.

My own involvement

To fill in some background detail: I was pleased to be part of the team that developed the set of 22 critical questions which sat at the heart of the interviews and research which are summarised in Part I of the book – and I conducted a number of the resulting interviews. In parallel, I explored related ideas via two different online Transpolitica surveys:

And I’ve been writing roughly one major article (or giving a public presentation) on similar topics every month since then. Recent examples include:

Over this time period, my views have evolved. I see the biggest priority, nowadays, not as figuring out how to govern AGI as it comes into existence, but rather, how to pause the development and deployment of any new types of AI that could spark the existence of self-improving AGI.

That global pause needs to last long enough that the global community can justifiably be highly confident that any AGI that will subsequently be built will be what I have called a BGI (a Beneficial General Intelligence) rather than a CGI (a Catastrophic General Intelligence).

Govern AGI and/or Pause the development of AGI?

I recently posted a diagram on various social media platforms to illustrate some of the thinking behind that stance of mine:

Alongside that diagram, I offered the following commentary:

The next time someone asks me what’s my p(Doom), compared with my p(SSfA) (the probability of Sustainable Superabundance for all), I may try to talk them through a diagram like this one. In particular, we need to break down the analysis into two cases – will the world keep rushing to build AGI, or will it pause from that rush.

To explain some points from the diagram:

We can reach the very desirable future of SSfA by making wise use of AI only modestly more capable than what we have today;
We might also get there as a side-effect of building AGI, but that’s very risky.

None of the probabilities are meant to be considered precise. They’re just ballpark estimates.

I estimate around 2/3 chance that the world will come to its senses and pause its current headlong rush toward building AGI.

But even in that case, risks of global catastrophe remain.

The date 2045 is also just a ballpark choice. Either of the “singularity” outcomes (wonderful or dreadful) could arrive a lot sooner than that.

The 1/12 probability I’ve calculated for “stat” (I use “stat” here as shorthand for a relatively unchanged status quo) by 2045 reflects my expectation of huge disruptions ahead, one sort or another.

The overall conclusion: if we want SSfA, we’re much more likely to get it via the “pause AGI” branch than via the “headlong rush to AGI” branch.

And whilst doom is possible in either branch, it’s much more likely in the headlong rush branch.

For more discussion of how to get the best out of AI and other cataclysmically disruptive technologies, see my book The Singularity Principles (the entire contents are freely available online).

Feel free to post your own version of this diagram, with your own estimates of the various conditional probabilities.

As indicated, I was hoping for feedback, and I was pleased to see a number of comments and questions in response.

One excellent question was this, by Bill Trowbridge:

What’s the difference between:
(a) better AI, and
(b) AGI

The line is hard to draw. So, we’ll likely just keep making better AI until it becomes AGI.

I offered this answer:

On first thought, it may seem hard to identify that distinction. But thankfully, we humans don’t just throw up our hands in resignation every time we encounter a hard problem.

For a good starting point on making the distinction, see the ideas in “A Narrow Path” by Control AI.

But what surprised me the most was the confidence expressed by various online commenters that:

  • “A pause however desirable is unlikely: p(pause) = 0.01”
  • “I am confident in saying this – pause is not an option. It is actually impossible.”
  • “There are several organisations working on AI development and at least some of them are ungovernable [hence a pause can never be global]”.

There’s evidently a large gulf behind the figure of 2/3 that I suggested for P(pause), and the views of these clearly intelligent respondents.

Why a pause isn’t that inconceivable

I’ll start my argument on this topic by confirming that I see this discussion as deeply important. Different viewpoints are welcome, provided they are held thoughtfully and offered honestly.

Next, although it’s true that some organisations may appear to be ungovernable, I don’t see any fundamental issue here. As I said online,

“Given sufficient public will and/or political will, no organisation is ungovernable.”

Witness the compliance by a number of powerful corporations in both China and the US to control measures declared by national governments.

Of course, smaller actors and decentralized labs pose enforcement challenges, but these labs are less likely to be able to marshal sufficient computing capabilities to be the first to reach breakthrough new levels of capability, especially if decentralised monitoring of dangerous attributes is established.

I’ve drawn attention on previous occasions to the parallel with the apparent headlong rush in the 1980s toward nuclear weapons systems that were ever more powerful and ever more dangerous. As I explained at some length in the “Geopolitics” chapter of my 2021 book Vital Foresight, it was an appreciation of the horrific risks of nuclear winter (first articulated in the 1980s) that helped to catalyse a profound change in attitude amongst the leadership camps in both the US and the USSR.

It’s the wide recognition of risk that can provide the opportunity for governments around the world to impose an effective pause in the headlong rush toward AGI. But that’s only one of five steps that I believe are needed:

  1. Awareness of catastrophic risks
  2. Awareness of bottlenecks
  3. Awareness of mechanisms for verification and control
  4. Awareness of profound benefits ahead
  5. Awareness of the utility of incremental progress

Here are more details about these five steps I envision:

  1. Clarify in an undeniable way how superintelligent AIs could pose catastrophic risks of human disaster within just a few decades or even within years – so that this topic receives urgent high-priority public attention
  2. Highlight bottlenecks and other locations within the AI production pipeline where constraints can more easily be applied (for example, distribution of large GPU chip clusters, and the few companies that are providing unique services in the creation of cutting-edge chips)
  3. Establish mechanisms that go beyond “trust” to “trust and verify”, including robust independent monitors and auditors, as well as tamperproof remote shut-down capabilities
  4. Indicate how the remarkable benefits anticipated for humanity from aspects of superintelligence can be secured, more safely and more reliably, by applying the governance mechanisms of points 2 and 3 above, rather than just blindly trusting in a no-holds-barred race to be the first to create superintelligence
  5. Be prepared to start with simpler agreements, involving fewer signatories and fewer control points, and be ready to build up stronger governance processes and culture as public consensus and understanding moves forward.

Critics can assert that each of these five steps is implausible. In each case, there are some crunchy discussions to be had. What I find dangerous, however, isn’t when people disagree with my assessments on plausibility. It’s when they approach the questions with what seems to be

  • A closed mind
  • A tribal loyalty to their perceived online buddies
  • Overconfidence that they already know all relevant examples and facts in this space
  • A willingness to distract or troll, or to offer arguments not in good faith
  • A desire to protect their flow of income, rather than honestly review new ideas
  • A resignation to the conclusion that humanity is impotent.

(For analysis of a writer who displays several of these tendencies, see my recent blogpost on the book More Everything Forever by Adam Beck.)

I’m not saying any of this will be easy! It’s probably going to be humanity’s hardest task over our long history.

As an illustration of points worthy of further discussion, I offer this diagram that highlights strengths and weakness of both the “governance” and “pause” approaches:

DimensionGovernance (Continue AGI Development with Oversight)Pause (Moratorium on AGI Development)
Core StrategyImplement global rules, standards, and monitoring while AGI is developedImpose a temporary but enforceable pause on new AGI-capable systems until safety can be assured
AssumptionsGovernance structures can keep pace with AI progress;
Compliance can be verified
Public and political will can enforce a pause;
Technical progress can be slowed
BenefitsEncourages innovation while managing risks;
Allows early harnessing of AGI for societal benefit;
Promotes global collaboration mechanisms
Buys time to improve safety research;
Reduces risk of premature, unsafe AGI;
Raises chance of achieving Beneficial General Intelligence (BGI) instead of CGI
RisksGovernance may be too slow, fragmented, or under-enforced;
Race dynamics could undermine agreements;
Possibility of catastrophic failure despite regulation
Hard to achieve global compliance;
Incentives for “rogue” actors to defect, in the absence of compelling monitoring;
Risk of stagnation or loss of trust in governance processes
Implementation ChallengesRequires international treaties;
Robust verification and auditing mechanisms;
Balancing national interests vs. global good
Defining what counts as “AGI-capable” research;
Enforcing restrictions across borders and corporations;
Maintaining pause momentum without indefinite paralysis
Historical AnalogiesNuclear Non-Proliferation Treaty (NPT);
Montreal Protocol (ozone layer);
Financial regulation frameworks
Nuclear test bans;
Moratoria on human cloning research;
Apollo program wind-down (pause in space race intensity)
Long-Term Outcomes (if successful)Controlled and safer path to AGI;
Possibility of Sustainable Superabundance but with higher risk of misalignment
Higher probability of reaching Sustainable Superabundance safely, but risks innovation slowdown or “black market” AGI

In short, governance offers continuity and innovation but with heightened risks of misalignment, whereas a pause increases the chances of long-term safety but faces serious feasibility hurdles.

Perhaps the best way to loosen attitudes, to allow a healthier conversation on the above points and others arising, is exposure to a greater diversity of thoughtful analysis.

And that brings me back to Global Governance of the Transition to Artificial General Intelligence by Jerome Glenn.

A necessary focus

Jerome’s book contains his personal stamp all over. His is a unique passion – that the particular risks and issues of AGI should not be swept into a side-discussion about the risks and issues of today’s AI. These latter discussions are deeply important too, but time and again, they result in existential questions about AGI being kicked down the road for months or even years. That’s something Jerome regularly challenges, rightly, and with vigour and intelligence.

Jerome’s presence is felt all over the book in one other way – he has painstakingly curated and augmented the insights of scores of different contributors and reviewers, including

  • Insights from 55 AGI experts and thought leaders across six major regions – the United States, China, the United Kingdom, Canada, the European Union, and Russia
  • The online panel of 229 participants from the global community around The Millennium Project who logged into a Real Time Delphi study of potential solutions to AGI governance, and provided at least one answer
  • Chairs and co-chairs of the 70 nodes of The Millennium Project worldwide, who provided additional feedback and opinion.

The book therefore includes many contradictory suggestions, but Jerome has woven these different threads of thoughts into a compelling unified tapestry.

The result is a book that carries the kind of pricing normally reserved for academic text books (as insisted by the publisher). My suggestion to you is that you recommend your local library to obtain a copy of what is a unique collection of ideas.

Finally, about my hallucination, mentioned at the start of this review. On double-checking, I realise that Jerome’s statement is actually, “Humanity has never faced a greater intelligence than itself.” The opening paragraph of that introduction continues,

Within a few years, most people reading these words will live with such superior artificial nonhuman intelligence for the rest of their lives. This book is intended to help us shape that intelligence or, more likely, those intelligences as they emerge.

Shaping the intelligence of the AI systems that are on the point of emerging is, indeed, a vital task.

And as Ben Goertzel says in his Foreword,

These are fantastic and unprecedented times, in which the impending technological singularity is no longer the province of visionaries and outsiders but almost the standard perspective of tech industry leaders. The dawn of transformative intelligence surpassing human capability – the rise of artificial general intelligence, systems capable of reasoning, learning, and innovating across domains in ways comparable to, or beyond, human capabilities – is now broadly accepted as a reasonably likely near-term eventuality, rather than a vague long-term potential.

The moral, social, and political implications of this are at least as striking as the technological ones. The choices we make now will define not only the future of technology but also the trajectory of our species and the broader biosphere.

To which I respond: whether we make these choices well or badly will depend on which aspects of humanity we allow to dominate our global conversation. Will humanity turn out to be its own worst enemy? Or its own best friend?

Postscript: Opportunity at the United Nations

Like it or loathe it, the United Nations still represents one of the world’s best venues where serious international discussion can, sometimes, take place on major issues and risks.

From 22nd to 30th September, the UNGA (United Nations General Assembly) will be holding what it calls its “high-level week”. This includes a multi-day “General Debate”, described as follows:

At the General Debate – the annual meeting of Heads of State and Government at the beginning of the General Assembly session – world leaders make statements outlining their positions and priorities in the context of complex and interconnected global challenges.

Ahead of this General Debate, the national delegates who will be speaking on behalf of their countries have the ability to recommend to the President of the UNGA that particular topics be named in advance as topics to be covered during the session. If the advisors to these delegates are attuned to the special issues of AGI safety, they should press their representative to call for that topic to be added to the schedule.

If this happens, all other countries will then be required to do their own research into that topic. That’s because each country will be expected to state its position on this issue, and no diplomat or politician wants to look uninformed. The speakers will therefore contact the relevant experts in their own country, and, ideally, will do at least some research of their own. Some countries might call for a pause in AGI development if it appears impossible to establish national licensing systems and international governance in sufficient time.

These leaders (and their advisors) would do well to read the report recently released by the UNCPGA entitled “Governance of the Transition to Artificial General Intelligence (AGI): Urgent Considerations for the UN General Assembly” – a report which I wrote about three months ago.

As I said at that time, anyone who reads that report carefully, and digs further into some of the excellent of references it contains, ought to be jolted out of any sense of complacency. The sooner, the better.

30 July 2025

The most dangerous book about AI ever written

Filed under: AGI, books, Singularity — Tags: , , , — David Wood @ 10:32 pm

Strawmen. Non-sequiturs. Sleight of hand. Face-palm shockers. This book is full of them.

As such, it encourages a disastrously complacent attitude toward the risks posed by forthcoming new AI systems.

The number of times I shouted (aloud, or just in my head) to the narrator, appalled at what I had just heard, when yet another devious distortion reached my ears, far exceeds anything I remember from any previous book.

Ordinarily, I would have set the book aside, long before finishing it, in order to make more productive use of my limited time. But in this case, I was aware that many other readers have seemingly been taken in by all the chicanery in this book: witness its average Goodreads rating of 4.29 stars out of 5, from 466 ratings, at the time I am writing this blogpost. And from sampling some of the reviews, it’s clear that the book satisfies a psychological hunger present in many of its readers – a hunger to be scornful of some of the world’s wealthiest people.

What makes the book particularly dangerous is the way that it weaves its horrendous falsehoods into a narrative with plenty of juicy content. That’s how it lures readers into accepting its most egregious conclusions. Readers get swept along on a kind of feel-good public condemnation of larger-than-life villains. Since these villains tell people that AI is going to become more and more capable, that idea gets walloped too. Let’s hold these villains in contempt – and likewise hold in contempt their self-aggrandising ideas about AI superintelligence. Yah boo!

Thus, the book highlights the shortcomings of some of the world’s most famous entrepreneurs and technology leaders; more than that, it builds a narrative that, if these people (among them, Marc Andreessen, Jeff Bezos, Elon Musk, and Sam Altman) continue to acquire more power, it will likely have very bad consequences for the bulk of humanity. That’s because:

  • These apparent titans over-estimate their own abilities, especially outside of their original domains of expertise
  • They have deeply naïve expectations about how easy it will be for humanity to set up self-supporting colonies on other planets
  • They are prone to a kind of self-righteous moral certainty which rides roughshod over the concerns of numerous critics.

That part of the narrative is correct. I give it three cheers. But where the book goes wildly wrong is in its associated narrative about not needing to be concerned with the emergence of AI which exceeds the understanding and control of its human designers. The way the book defends its wrong conclusions about AI is by setting up strawmen – weak imitations of the real concerns about AI superintelligence – and then pointing out flaws in these strawmen.

Motivations

I’ll come to these strawmen shortly. But first, I’ll express a bit more sympathy for at least part of what Adam Becker, the author of this book, is trying to do. He explains his motivation in a recent Singularity.FM interview with Nikola Danaylov:

Becker’s previous career followed a path in which I was personally also very interested at a similar stage in my life: a fascination with cosmology and theoretical physics. That evolved into a passion (which, again, I share) for clear communications about the meaning and implications of science. Becker’s first book, What is Real? The Unfinished Quest for the Meaning of Quantum Physics addresses the same topics on which I focussed for four years in the History and Philosophy of Science department in Cambridge in the mid 1980s. I’ve not read that book (yet), but based on various reviews, I believe I would agree with Becker’s main conclusions in that book.

Becker’s first concept for what he should write about in his second book also gets a thumbs up from me: evidence that many tech leaders in Silicon Valley have flawed views about key aspects of science – including flawed views about biology, psychology, and sociology, as well as the physics of space travel.

As Becker explained in his Singularity.FM interview, his ideas evolved further as he tried to write his second book. His scope widened to include analyses of some of the philosophical ideas which influence many of the movers and shakers in Big Tech – ideas such as longtermism, advocated by the Oxford philosopher Will MacAskill. I share with Becker a distaste for some of the conclusions of longtermism, though I’m less convinced that Becker provides adequate rebuttals to the longtermist argumentation. (Throughout the book, when analysing philosophical positions, Becker ladles on the critical whining naysaying, but he offers little as an alternative worldview, beyond very empty generalities.)

But where I absolutely part company with Becker is in my assessment of the idea of a potential forthcoming Technological Singularity, triggered by AI becoming increasingly capable. Becker roundly and freely condemns that idea as “unscientific”, “specious”, “imaginary”, and “hypothetical”.

Strawmen

Becker’s basic narrative is this: AI superintelligence will require a complete understanding of the human brain and a complete copying of what’s happening in the brain, down to a minute level. However, we’re still a long way from obtaining that understanding. Indeed, there are now reasons to believe that significant computation is taking place inside individual neurons (beyond a simple binary summation), and that various other types of braincell also contribute to human intelligence. Moreover, little progress has been made in recent years with brain scanning.

Now, this view of “understand the human brain first and copy that precisely” might have been the view of some AI researchers in the past, but since the revolutions of Deep Neural Networks (2012+) and Transformers (2018+), it’s clear that humanity could create AI with very dangerous capabilities without either of these preconditions. It’s more accurate to say that these AIs are being grown rather than being built. They are acquiring their capabilities via emergence rather than via detailed specification. To that extent, the book is stuck in the past.

These new AIs may or may not have all the same thinking processes that take place inside the human brain. They may or may not have aspects of what we call consciousness. That’s beside the point. What matters is whether the AI gains capabilities in observing, predicting, planning interventions, and learning from the results of its predictions and interventions. It is these capabilities that give AI its increasing power to intervene in the world.

This undermines another of the strawmen in Becker’s extensive collection – his claim that ideas of AI superintelligence wrongly presuppose that all intelligence can be reduced to a single parameter, ‘g’, standing for general intelligence. On the contrary, what matters is whether AI will operate outside of human understanding and human control. That’s already nearly happening. Yet Becker prefers to reassure his readers with some puffed up philosophising. (I lost track of the number of times he approvingly quoted cognitive scientists who seemingly reassured him that intelligence was too complicated a subject for there to be any worry about AI causing a real-world catastrophe.)

It’s like a prehistoric group of chimpanzees thinking to themselves that, in various ways, their individual capabilities exceed the corresponding capabilities of individual humans. Their equivalent of Adam Becker might say, “See, there’s no unified ‘h’ parameter for all the ways that humans allegedly out-perform chimpanzees. So don’t worry chaps, we chimpanzees will remain in control of our own destiny, and humans will forever remain as just weird naked apes.”

The next strawman is the assumption that the concern about out-of-control AI depends upon the maintenance of smooth exponential progress curves. Astonishingly, Becker devotes numerous pages to pointing out ways that exponential trends, such as Moore’s Law, slow down or even stop. This leads him to assert that “AI superintelligence is imaginary”. But the real question is: is more progress possible than we have already reached? In more detail:

  • Can more efficient computational hardware be invented? (Answer: yes, including new types of chips dedicated to new kinds of AI.)
  • Can extra data be fed into AI training? (Answer: yes, including cleverly constructed synthetic data.)
  • Can new architectures, beyond transformers, be introduced? (Answer: yes, and AI researchers are pursuing numerous possibilities.)
  • Can logical reasoning, such as chain-of-thought, be combined in productive new ways with existing neural networks? (Answer: yes, this is happening daily.)
  • Is there some fundamental reason why the human brain is the ultimate apex of the skills of predicting, planning interventions, and learning? (Answer: no, unless you are a believer in six day creationism, or something equivalent.)

So, what matters isn’t the precise shape of the pace of progress. What matters is whether that progress can reach a point that enables AIs to improve the process of designing further AIs. That’s the tipping point which will introduce huge new uncertainty.

Becker tries to head off arguments that forthcoming new types of hardware, such as quantum computing, might bring AI closer. Quantum computing, as presently understood, isn’t suited to all computational tasks, he points out. But wait: the point is that it can significantly accelerate some computational tasks. AI can improve through smarter combinations of old-style hardware and new-style hardware. We don’t need to take Becker’s simplistic one-design-only approach as the end of the argument.

The slowdown in reaching new generations of traditional semiconductor chips does not mean the end of the broader attainment of wide benefits from improved hardware performance. Instead, AI progress now depends on how huge numbers of individual chips can be networked together. Moreover, with more hardware being available at lower cost, more widely distributed, this enables richer experimentation with new algorithms and new software architectures, thereby making yet more new AI breakthroughs more likely. Any idea that breakthroughs in AI have come to a brick wall is preposterous.

Next, Becker returns repeatedly to the strawman that the kinds of threats posed by AI superintelligence are just hypothetical and are far removed from our previous experience. Surely an AI that is able to self-introspect will be kinder, he argues. However, humans who are more intelligent – including having the ability to analyse their own thought processes – are by no means necessarily kinder. They may be psychopaths. Likewise, advanced AIs may be psychopaths – able to pretend concern for human wellbeing while that tactic suits them, but ready to incapacitate us all when the opportunity arises.

Indeed, the threats posed by ever more powerful AI are relatively straightforward extrapolations of dangers posed by existing AI systems (at the hands of human users who are hateful or naïve or resentful or simply out-of-their-depth). There’s no need to make any huge jump of imagination. That’s an argument I spell out in this Mindplex article.

Yet another strawman in the book is the idea that the danger-from-advanced-AI argument needs to be certain, and that it can be rejected if any uncertainty remains about it. Thus, when Becker finds AI safety advocates who are unwilling to pin down a precise probability for the likelihood of an AI-induced catastrophe, he switches from “uncertain about the chance of doom” to “unconcerned about the chance of doom”. When two different apparent experts offer opposing views on the likelihood of AI-induced doom, he always prefers the sceptic, and rushes to dismiss the other side. (Is he really so arrogant to think he has a better grasp of the possibilities of AI-induced catastrophe than the international team of experts assembled by Yoshua Bengio? Apparently, yes he is.)

One final outrageous tactic Becker uses to justify disregarding someone’s view is to point out a questionable claim that person has made in another area. Thus, Nick Bostrom has made some shocking statements about the difference in abilities between people of different races. Therefore, all Bostrom’s views about the dangers of AI superintelligence can be set aside. Elon Musk naively imagines it will be relatively easy to terraform Mars to make it suitable for human habitation. Therefore, all Musk’s views about the dangers of AI superintelligence can, again, be set aside. You get the picture.

Constructive engagement

Instead of scorning these concerns, Becker should be engaging constructively with the community of thoughtful people who are (despite adverse headwinds) painstakingly exploring ways to get the best out of AI whilst avoiding the risks of catastrophe. This includes the Singapore Consensus, the Future of Life Institute, the Council of Presidents of the United Nations General Assembly, Control AI, Pause AI, The Millennium Project, AI Safety, the Kira Center, the Machine Intelligence Research Institute, the Center for AI Safety Research, the Centre for the Governance of AI, the Center for Human Compatible AI, the Leverhulme Centre for the Future of Intelligence, my own book “The Singularity Principles”, and much more.

That kind of constructive engagement might not lead to as many juicy personal anecdotes as Becker sprinkles throughout More Everything Forever, but it would provide much better service to humanity.

Conversely, you might ask: aren’t there any lessons for me (and other AI safety activists) in the light of the shortcomings highlighted by Becker in the thoughts and actions of many people who take the idea of the Technological Singularity seriously? Shouldn’t I be grateful to Becker for pointing out various predictions made by Ray Kurzweil which haven’t come to pass, the casual attitudes seemingly displayed by some singularitarians toward present-day risks arising from abuses of existing technology (including the ongoing emissions of greenhouse gases), the blatant links between the 2023 Techno-Optimist Manifesto of Marc Andreessen and the proto-fascist 1909 Futurist Manifesto of Filippo Marinetti, and so on?

My answer: yes, but. Almost nothing in Becker’s book was new for me. I have since 2021 frequently given presentations on the subject of “The Singularity Shadow” (the concept first appeared in my book Vital Foresight) – a set of confusions and wishful thinking which surrounds the subject of the Technological Singularity:

These confusions and wishful thinking form a kind of shadow around the central concept of the Technological Singularity – a shadow which obstructs a clearer perception of the risks and opportunities that are actually the most significant.

The Singularity Shadow misleads many people that should know better. That shadow of confusion helps to explain why various university professors of the subject of artificial intelligence, along with people with job titles such as “Head of AI” in large companies, often make statements about the likely capabilities of forthcoming new AI platforms that are, frankly, full of errors or deeply misleading.

I describe that shadow as consisting of seven overlapping areas:

  1. Singularity timescale determinism
  2. Singularity outcome determinism
  3. Singularity hyping
  4. Singularity risk complacency
  5. Singularity term overloading
  6. Singularity anti-regulation fundamentalism
  7. Singularity preoccupation

To be clear, there is a dual problem with the Singularity Shadow:

  • People within the shadow – singularity over-enthusiasts – make pronouncements about the Singularity that are variously overly optimistic, overly precise, or overly vague
  • People outside the shadow – singularity over-critics – notice these instances of unwarranted optimism, precision, or vagueness, and jump to the wrong conclusion that the entire field of discussion is infected with the same flaws.

Here’s a video that reviews the seven areas in the Singularity Shadow, and the damage this Shadow causes to thoughtful discussions about both the opportunities and the threats arising from the Singularity:

And if you want to follow the conversation one more step, this video looks more deeply at the reasons why people (such as Becker) are so insistent that the Singularity is (in his words) “unscientific”, “specious”, “imaginary”, and “hypothetical”:

That’s the ‘but’ part of my “yes, but” answer. The ‘yes’ part is that, yes, I need to reflect: after so many years of trying to significantly improve the conversation about both the opportunities and risks of the Singularity, the public conversation about it is still often dominated by Becker-style distractions and confusions.

Clearly, I need to up my game. We all need to up our game.

AGW and AGI

I’ll finish with one point of consensus: Becker is highly critical, in his book, of people who use their intelligence to deny the risks of accelerated global warming (AGW). Becker, like me, sees these risks as deeply concerning. We are both dismayed when evidently clever people come up with deceptive arguments to avoid taking climate change seriously. The real risk here isn’t of linear climate change, but rather of the climate reaching thresholds known as tipping points, where greater heat leads to dramatic changes in the earth’s ecosystem that result in even greater heat. Sudden changes in temperature, akin to that just described, can be observed in ancient geological transition points.

It’s the unpredictability of what happens at these tipping points – and the uncertainty over where these tipping points are located – that means humanity should be doubling down, hard, on reversing our greenhouse gas emissions. (The best book I’ve read on this topic recently, by the way, is A Climate of Truth, by Mike Berners-Lee. I unhesitatingly recommend it.)

Yet despite these risks, AGW deniers argue as follows: there is plenty of uncertainty. There are lots of different ways of measuring temperature. There are lots of different forecasts. They don’t all agree. That means we have plenty of time to work out solutions. In the meantime, inaction is fine. (Face palm!)

I’ve spelt this out, because Becker is equally guilty. He’s not an AGW denier, but an AGI denier – denying that we need to pay any serious attention to the risks of Artificial General Intelligence. There is plenty of uncertainty about AGI, he argues. Disagreement about the best way to build it. No uniform definition of ‘g’, general intelligence. No agreement on future scenarios. Therefore, we have plenty of time to work out how to deal with any hypothetical future AGI. (Face palm again!)

Actually, this is not just a matter of a face palm. It’s a matter of the utmost seriousness. The unpredictability makes things worse, not better. Becker has allowed his intelligence to be subverted to obscure one of the biggest risks facing humanity. And because he evidently has an audience that is psychologically predisposed to lap up his criticism of Silicon Valley leaders, the confusion he peddles is likely to spread significantly.

It’s all the more reason to engage sincerely and constructively with the wider community who are working to ensure that advanced AI turns out beneficial (a “BGI”) instead of catastrophic (a “CGI”).

29 May 2025

Governance of the transition to AGI: Time to act

As reported yesterday by The Millennium Project, the final report has been released by a high-level expert panel, convened by the UN Council of Presidents of the General Assembly (UNCPGA), on the subject of Artificial General Intelligence (AGI). The report is titled “Governance of the Transition to Artificial General Intelligence (AGI): Urgent Considerations for the UN General Assembly”. It’s well worth reading!

About the UNCPGA

What’s the UNCPGA, you may ask.

Founded in 1992, this Council consists of all former Presidents of the UN General Assembly. I think of it as akin to the House of Lords in the UK, where former members of the House of Commons often display more wisdom and objectivity than when they were embedded in the yah-boo tribal politics of day-to-day government and opposition. These former Presidents hold annual meetings to determine how they can best advance the goals of the UN and support the Office of the current President of the UNGA.

At their 2024 meeting in Seoul, the UNCPGA decided that a global panel of experts on AGI should be convened. Here’s an extract from the agreement reached at that meeting:

The Seoul Declaration 2024 of the UNCPGA calls for a panel of artificial general intelligence (AGI) experts to provide a framework and guidelines for the UN General Assembly to consider in addressing the urgent issues of the transition to artificial general intelligence (AGI).

This work should build on and avoid duplicating the extensive efforts on AI values and principles by UNESCO, OECD, G20, G7, Global Partnership on AI, and Bletchley Declaration, and the recommendations of the UN Secretary-General’s High-Level Advisory Body on AI, UN Global Digital Compact, the International Network of AI Safety Institutes, European Council’s Framework Convention on AI and the two UN General Assembly Resolutions on AI. These have focused more on narrower forms of AI. There is currently a lack of similar attention to AGI.

AI is well known to the world today and often used but AGI is not and does not exist yet. Many AGI experts believe it could be achieved within 1-5 years and eventually could evolve into an artificial super intelligence beyond our control. There is no universally accepted definition of AGI, but most AGI experts agree it would be a general-purpose AI that can learn, edit its code, and act autonomously to address many novel problems with novel solutions similar to or beyond human abilities. Current AI does not have these capabilities, but the trajectory of technical advances clearly points in that direction…

The report should identify the risks, threats, and opportunities of AGI. It should focus on raising awareness of mobilizing the UN General Assembly to address AGI governance in a more systematic manner. It is to focus on AGI that has not yet been achieved, rather than current forms of more narrow AI systems. It should stress the urgency of addressing AGI issues as soon as possible considering the rapid developments of AGI, which may present serious risks to humanity as well as extraordinary benefits to humanity.

The panel was duly formed, with the following participants:

  • Jerome Glenn (USA), Chair
  • Renan Araujo (Brazil)
  • Yoshua Bengio (Canada)
  • Joon Ho Kwak (Republic of Korea)
  • Lan Xue (China)
  • Stuart Russell (UK and USA)
  • Jaan Tallinn (Estonia)
  • Mariana Todorova (Bulgaria)
  • José Jaime Villalobos (Costa Rica)

(For biographical details of the participants, the mandate they were given following the Seoul event, and the actual report they delivered, click here.)

The panel was tasked with preparing and delivering its report at the 2025 gathering of the UNCPGA, which took place in April in Bratislava. Following a positive reception at that event, the report is now being made public.

Consequences if no action is taken

The report contains the following headline: “Urgency for UN General Assembly action on AGI governance and likely consequences if no action is taken“:

Amidst the complex geopolitical environment and in the absence of cohesive and binding international norms, a competitive rush to develop AGI without adequate safety measures is increasing the risk of accidents or misuse, weaponization, and existential failures. Nations and corporations are prioritizing speed over security, undermining national governing frameworks, and making safety protocols secondary to economic or military advantage. Since many forms of AGI from governments and corporations could emerge before the end of this decade, and since establishing national and international governance systems will take years, it is urgent to begin the necessary procedures to prevent the following outcomes…

The report lists the following six outcomes, that urgently require action to avoid:

1. Irreversible Consequences—Once AGI is achieved, its impact may be irreversible. With many frontier forms of AI already showing deceptive and self-preservation behavior, and the push towards more autonomous, interacting, self-improving AIs integrated with infrastructures, the impacts and trajectory of AGI can plausibly end up being uncontrollable. If that happens, there may be no way to return to a state of reliable human oversight. Proactive governance is essential to ensure that AGI will not cross our red lines, leading to uncontrollable systems with no clear way to return to human control.

2. Weapons of Mass Destruction—AGI could enable some states and malicious non-state actors to build chemical, biological, radiological, and nuclear weapons. Moreover, large, AGI-controlled swarms of lethal autonomous weapons could themselves constitute a new category of WMDs.

3. Critical Infrastructure Vulnerabilities—Critical national systems (e.g., energy grids, financial systems, transportation networks, communication infrastructure, and healthcare systems) could be subject to powerful cyberattacks launched by or with the aid of AGI. Without national deterrence and international coordination, malicious non-state actors from terrorists to transnational organized crime could conduct attacks at a large scale.

4. Power Concentration, Global Inequality, and Instability—Uncontrolled AGI development and usage could exacerbate wealth and power disparities on an unprecedented scale. If AGI remains in the hands of a few nations, corporations, or elite groups, it could entrench economic dominance and create global monopolies over intelligence, innovation, and industrial production. This could lead to massive unemployment, widespread disempowerment affecting legal underpinnings, loss of privacy, and collapse of trust in institutions, scientific knowledge, and governance. It could undermine democratic institutions through persuasion, manipulation, and AI-generated propaganda, and heighten geopolitical instability in ways that increase systemic vulnerabilities. A lack of coordination could result in conflicts over AGI resources, capabilities, or control, potentially escalating into warfare. AGI will stress existing legal frameworks: many new and complex issues of intellectual property, liability, human rights, and sovereignty could overwhelm domestic and international legal systems.

5. Existential Risks—AGI could be misused to create mass harm or developed in ways that are misaligned with human values; it could even act autonomously beyond human oversight, evolving its own objectives according to self-preservation goals already observed in current frontier AIs. AGI might also seek power as a means to ensure it can execute whatever objectives it determines, regardless of human intervention. National governments, leading experts, and the companies developing AGI have all stated that these trends could lead to scenarios in which AGI systems seek to overpower humans. These are not far-fetched science fiction hypotheticals about the distant future—many leading experts consider that these risks could all materialize within this decade, and their precursors are already occurring. Moreover, leading AI developers have no viable proposal so far for preventing these risks with high confidence.

6. Loss of Extraordinary Future Benefits for All of Humanity—Properly managed AGI promises improvements in all fields, for all peoples, from personalized medicine, curing cancer, and cell regeneration, to individualized learning systems, ending poverty, addressing climate change, and accelerating scientific discoveries with unimaginable benefits. Ensuring such a magnificent future for all requires global governance, which begins with improved global awareness of both the risks and benefits. The United Nations is critical to this mission.

In case you think these scenarios are unfounded fantasies, I encourage you to read the report itself, where the experts provide references for further reading.

The purpose envisioned for UN governance

Having set out the challenges, the report proceeds to propose the purpose to be achieved by UN governance of the transition to AGI:

Given that AGI might well be developed within this decade, it is both scientifically and ethically imperative that we build robust governance structures to prepare both for the extraordinary benefits and extraordinary risks it could entail.

The purpose of UN governance in the transition to AGI is to ensure that AGI development and usage are aligned with global human values, security, and development. This involves:

1) Advancing AI alignment and control research to identify technical methods for steering and/or controlling increasingly capable AI systems;

2) Providing guidance for the development of AGI—establishing frameworks to ensure AGI is developed responsibly, with robust security measures, transparency, and in alignment with human values;

3) Developing governance frameworks for the deployment and use of AGI—preventing misuse, ensuring equitable access, and maximizing its benefits for humanity while minimizing risks;

4) Fostering future visions of beneficial AGI—new frameworks for social, environmental, and economic development; and

5) Providing a neutral, inclusive platform for international cooperation—setting global standards, building an international legal framework, and creating incentives for compliance; thereby, fostering trust among nations to guarantee global access to the benefits of AGI.

Actions recommended

The report proceeds to offer four recommendations for further consideration during a UN General Assembly session specifically on AGI:

A. Global AGI Observatory: A Global AGI Observatory is needed to track progress in AGI-relevant research and development and provide early warnings on AI security to Member States. This Observatory should leverage the expertise of other UN efforts such as the Independent International Scientific Panel on AI created by the Global Digital Compact and the UNESCO Readiness Assessment Methodology.

B. International System of Best Practices and Certification for Secure and Trustworthy AGI: Given that AGI might well be developed within this decade, it is both scientifically and ethically imperative that we build robust governance structures to prepare both for the extraordinary benefits and extraordinary risks it could entail.

C. UN Framework Convention on AGI: A Framework Convention on AGI is needed to establish shared objectives and flexible protocols to manage AGI risks and ensure equitable global benefit distribution. It should define clear risk tiers requiring proportionate international action, from standard-setting and licensing regimes to joint research facilities for higher-risk AGI, and red lines or tripwires on AGI development. A Convention would provide the adaptable institutional foundation essential for globally legitimate, inclusive, and effective AGI governance, minimizing global risks and maximizing global prosperity from AGI.

D. Feasibility Study on a UN AGI Agency: Given the breadth of measures required to prepare for AGI and the urgency of the issue, steps are needed to investigate the feasibility of a UN agency on AGI, ideally in an expedited process. Something like the IAEA has been suggested, understanding that AGI governance is far more complex than nuclear energy; and hence, requiring unique considerations in such a feasibility study.

What happens next

I’m on record as being pessimistic that the UNGA will ever pay sufficient attention to the challenges of governing the transition to AGI. (See the section “The collapse of cooperation is nigh” in this recent essay of mine.)

But I’m also on record as seeing optimistic scenarios too, in which humanity “chooses cooperation, not chaos”.

What determines whether international bodies such as the UN will take sufficient action – or whether, instead, insightful reports are left to gather dust as the body focuses on virtue signalling?

There are many answers to that question, but for now, I’ll say just this. It’s up to you. And to me. And to all of us.

That is, each of us has the responsibility to reach out, directly or indirectly, to the teams informing the participants at the UN General Assembly. In other words, it’s up to us to find ways to catch the attention of the foreign ministry in our countries, so that they demand:

  • Adequate timetabling at the UNGA for the kind of discussion that the UNCPGA report recommends
  • Appropriate follow-up: actions, not just words

That may sound daunting, but a fine piece of advice has recently been shared online by Leticia García Martínez, Policy Advisor at ControlAI. Her article is titled “What We Learned from Briefing 70+ Lawmakers on the Threat from AI” and I recommend that you read it carefully. It is full of pragmatic suggestions that are grounded in recent experience.

ControlAI are gathering signatures on a short petition:

Nobel Prize winners, AI scientists, and CEOs of leading AI companies have stated that mitigating the risk of extinction from AI should be a global priority.

Specialised AIs – such as those advancing science and medicine – boost growth, innovation, and public services. Superintelligent AI systems would compromise national and global security.

The UK can secure the benefits and mitigate the risks of AI by delivering on its promise to introduce binding regulation on the most powerful AI systems.

Happily, this petition has good alignment with the report to the UNCPGA:

  • Support for the remarkable benefits possible from AI
  • Warnings about the special risks from AGI or superintelligent AI
  • A determination to introduce binding regulation.

New politicians continue to be added to their campaign webpage as supporters of this petition.

The next thing that needs to happen in the UK parliament is that their APPG (All Party Parliamentary Group) on AI need to devote sufficient time to AGI / superintelligence. Regrettably, up till now, they’ve far too often sidestepped that issue, focussing instead of issues of today’s AI, rather than the supercharged issues of AGI. Frankly, it’s a failure of vision, and a prevalence of groupthink.

Hopefully, as the advisors to the APPG-AI read the UNCPGA report, they’ll be jolted out of their complacency.

It’s time to act. Now.

Postscript: Jerome Glenn visiting London

Jerome (Jerry) Glenn, the chair of the expert panel that produced this report, and who is also the founder and executive director of the Millennium Project, will be visiting London on the weekend of Saturday 14th June.

There will be a number of chances for people in and around London to join discussions with Jerry. That includes a session from 2pm to 4pm on that Saturday, “The Future of AI: Issues, Opportunities, and Geopolitical Synergies”, as well as a session in the morning “State of the Future 20.0”, and an open-ended discussion in the early evening, “The Future – Where Next?”.

For more details of these events, and to register to attend, click here.

14 April 2025

Choose radical healthy longevity for all

Filed under: aging, Economics, healthcare, politics, The Abolition of Aging, vision — Tags: — David Wood @ 9:04 pm

What do you think about the possibility of radical healthy longevity?

That’s the idea that, thanks to ongoing scientific progress, new medical treatments may become available, relatively soon, that enable people to remain as vibrant and healthy in their 80s, 90s, and beyond, as they were in their 20s and 30s.

It’s the idea that it may soon become possible to systematically repair or replace, on a regular basis, the various types of damage that tend to accumulate in our bodies over the decades – damage such as plaques, tangles, chronic inflammation, DNA mutations and epimutations, dysfunctional mitochondria, weakened immune function, crosslinks between macromolecules, cells that increasingly neglect their original function, and so on. This damage would be removed before it gives rise to chronic diseases.

It’s the idea, in other words, that aging as we know it – experienced as an inevitable deterioration – could be overcome by scientific innovations, well within the lifetimes of many people who are already adults.

It’s a divisive idea. Some people love it but others seem to hate it. Some people see it as liberating – as enabling lives of abundance. Others see it as being inherently unnatural, unethical, or unjust.

In more detail, I’ve noticed four widespread attitudes regarding this idea:

  1. It’s an unscientific fantasy. Radical healthy longevity won’t be possible any time soon.
  2. It will be the ultimate in inequality. Radical healthy longevity will be available only to a small minority of people; everyone else will receive much poorer medical treatment.
  3. It’s guaranteed that it will be a profound universal benefit. Everyone who is fortunate enough to live long enough to be alive at a threshold point in the future will benefit (if they wish) from low-cost high-quality radical healthy longevity.
  4. It’s an outcome that needs to be fought for. There’s nothing inevitable about the timing, the cost, the quality, or the availability of radical healthy longevity.

Before reading further, you might like to consider which of these four attitudes best describes you. Are you dismissive, fearful, unreservedly optimistic, or resolved to be proactive?

Who benefits?

To briefly address people inclined to dismiss the idea of radical healthy longevity: I’ve written at length on many occasions about why this idea has strong scientific backing. For example, see my article from May last year, “LEV: Rational Optimism and Breakthrough Initiatives”. Or consider my more recent Mindplex article “Ten ways to help accelerate the end of aging”. I won’t repeat these arguments in this article.

What is on my mind, however, is the question of who will benefit from radical healthy longevity. That’s a question that keeps on being raised.

For example, this is an extract from one comment posted under my Mindplex article:

If you want me to be an anti-aging influencer, the first thing I am going to ask you is “how is this going to be affordable?” How can you guarantee that I not serving only the agenda of the rich and the elite?

And here are some extracts from another comment:

Longevity is ruined by rich people and some elite politicians… Funding for anti-aging research tends to be controlled by those with deep pockets. In reality, the rich and powerful usually set the agenda, and I worry that any breakthroughs will only benefit an elite few rather than the general public… I’ve seen this play out in other areas of medicine and tech (big pharma and cancer are the best examples), where groundbreaking ideas are co-opted to serve market interests, leaving everyday people out in the cold.

The possible responses to these concerns mirror the four attitudes I’ve already listed:

  1. This discussion is pretty pointless, since relevant treatments won’t exist any time soon. If anything, this whole conversation is a regrettable distraction from spreading real-world health solutions more widely
  2. The concerns are perceptive, since there’s a long history of two-tier health solutions, compounding what is already a growing “longevity gap”
  3. The concerns are misplaced, since the costs of profound new health solutions will inevitably fall
  4. The concerns are a wake-up call, and should motivate all of us to ensure new treatments become widely available as soon as possible.

To restate the fourth response (which is the one I personally favour): we need to choose radical healthy longevity for all. We need to fight for it. We need to take actions to move away from the scenario “the ultimate in inequality” and toward the scenario “a guaranteed profound universal benefit”. That’s because both of these scenarios are credible possible futures. Each extrapolates from trends already in motion.

Extrapolating what we can already see

The scenario “a guaranteed profound universal benefit” takes inspiration from the observation that the cost of products and services often drops dramatically over time. That was the case with computers, smartphones, flat screen TVs, and many other items of consumer electronics. Even when prices remain high, lots of new features become included in the product – as in the case of motor cars. These improvements arise from economies of scale, from competition between different suppliers, and from the creative innovation arising.

But there’s no inevitability here. Monopolies or industry cartels can keep prices high. Strangleholds over IP (intellectual property) can hinder the kind of competition that would otherwise improve consumer experience. Consider the sky-high pricing of many medical procedures in various parts of the world, such as the United States. For example, research by Stacie Dusetzina of the University of North Carolina at Chapel Hill highlights how the costs in the United States of treatment of cancer by the newest pills rose sixfold over a recent 14-year period. That’s after taking account of inflation. And the inflation-adjusted cost of a millilitre of insulin, used in the treatment of diabetes, increased threefold over a recent 11-year period. Many other examples could be listed. In various ways, these examples all fit the well-known pattern that free markets can experience market failures.

A counter-argument is that, provided the benefits of new health treatments are large enough, it will become politically necessary for the state to intervene to correct any such market failures. Early application of the kinds of damage-repair and damage-removal treatments mentioned earlier, will result in a huge “longevity dividend“. The result will be to reduce the costs that would otherwise be incurred as age-related diseases take their toll (often with more than one co-morbidity complicating treatment options). According to the longevity dividend, it’s economically desirable to spend a smaller amount of money, earlier, as a preventive measure, than to have to pay much more money later, trying to alleviate widespread symptoms. No sensible political party could ignore such an imperative. They would surely be voted out of office. Right?

Uncertain politics

Alas, we need to reckon not only with occasional market failures but also with occasional political failures. There’s no guarantee that political leaders will take actions that benefit the country as a whole. They can be driven by quite different motivations.

Indeed, groups can seize power in countries and then hold onto it, whilst giving only lip service to the needs of the voters who originally elected them. Leaders of these groups may assert, beforehand, that voters will prosper in the wake of the revolution that they will bring. But by the way, for the success of this transformation, voters will need to agree that the revolutionary leaders can ride roughshod over normal democratic norms. These leaders will be above the law – 21st century absolute monarchs, in effect. But then, guess what, inflation remains high, unemployment surges, the environment is despoiled, freedoms are suspended, and protestors who complain about the turn of events are rounded up and imprisoned. Worse, due to the resulting crisis, future elections may be cancelled. As for universal access to radical healthy longevity, forget it! The voters who put the revolutionaries in power are now dispensable.

That’s only one way in which the scenario “the ultimate in inequality” could unfold. Less extreme versions are possible too.

It’s a future that we should all seek to prevent.

Choosing to be proactive

Is the answer, therefore, to urge a cessation of research into treatments that could bring about radical healthy longevity? Is the answer to allow our fears of inequality to block the potential for dramatic health improvements?

I disagree. Strongly. On the contrary, the answer is to advance several initiatives in parallel:

  • To increase the amount of public funding that supports research into such treatments
  • To avoid political conditions in which market failures grow more likely and more severe
  • To avoid social conditions in which political failures grow more likely and more treacherous

All three of these tasks are challenging. But all three of them make good sense. They’re all needed. Omit any one of these tasks and it becomes more probable that the future will turn out badly.

As it happens, all three tasks are choices to be proactive – choices to prevent problems early, rather than experiencing much greater pain if the problems are allowed to grow:

  • Problems from an accumulation of biological damage inside and between our cells, causing an escalation of ill-health
  • Problems from an accumulation of political damage, causing an escalation of economic dysfunction
  • Problems from an accumulation of societal damage, causing an escalation of political dysfunction

So, again I say, it is imperative that, rather than passively observing developments from the sidelines, we actively choose radical healthy longevity. That’s biological health, political health, and societal health.

Whether through advocacy, funding, research, or policy engagement, everyone has a role to play in shaping a profoundly positive future. By our actions, coordinated wisely, we can make a real difference to how quickly people around the world can be freed from the scourge of the downward spiral of aging-related ill-health, and can enjoy all-round flourishing as never before.

6 April 2025

Choose coordination not chaos

Filed under: Abundance, AGI, chaos, collaboration — Tags: , , — David Wood @ 10:46 pm

Note: this document is subject to change as more feedback is received. Check back for updates later. This is version 1.1c.

Preamble

A critically important task in the coming months is to inspire a growing wave of people worldwide to campaign for effective practical coordination of the governance of advanced AI. That’s as an alternative to leaving the development and deployment of advanced AI to follow its present chaotic trajectory.

The messages that need to be conveyed, understood, and acted upon, are that

  • The successful governance of advanced AI will result in profound benefit for everyone, whereas a continuation of the present chaotic state of affairs risks global catastrophe
  • The successful governance of advanced AI isn’t some impossible dream, but lies within humanity’s grasp
  • Nevertheless, real effort, real intelligence, and, yes, real coordination will be needed, so humanity can reach a world of unprecedented abundance, rather than sleepwalk into disaster.

What would have great value is a campaign slogan that conveys the above insights, and which is uplifting, unifying, easily understood, and forward-looking. The right slogan would go viral, and would galvanise people in all walks of life to take positive action.

To that end, that slogan should ideally be

  • Memorable and punchy
  • Emotionally resonant
  • Credibly audacious
  • Universally understandable
  • Open-ended enough to invite participation

To be clear, that slogan should not cause panic or despair, but should put people into a serious frame of mind.

A specific proposal

After several days of online brainstorming, in which numerous ideas were presented and discussed, I now present what I consider the best option so far.

Choose coordination not chaos, so AI brings abundance for all

If there is a need for a three-word version of this, options include

  • Coordination not chaos
  • Choose AI abundance

Of course, what’s needed isn’t just a standalone slogan. Therefore, please find here also a sample illustrative image and, most important, a set of talking points to round out the concept.

About the image

Here’s the commentary by ChatGPT, when asked to suggest an image to illustrate this campaign slogan:

Concept: Two contrasting futures from a single branching path.

Scene:

  • A wide landscape split into two diverging paths or realms — one vibrant and coordinated, the other chaotic and fragmented.
  • In the coordinated half:
    • A harmonious world — sustainable cities, diverse communities collaborating with AI, lush green spaces, clean tech, and open exchanges of knowledge and creativity.
    • Subtle signs of AI woven into infrastructure: responsive lighting, robotic assistants, AI-powered transport.
  • In the chaotic half:
    • A fractured world — disconnected enclaves, pollution, conflict, neglected tech, and isolated individuals overwhelmed by noise or misinformation.
    • AI appears uncontrolled — surveillance drones, malfunctioning robots, or broken screens.

Central focus:

  • A group of people at the fork in the path, pointing and stepping toward the coordinated future, with calm, confident AI assistants guiding the way.

(Aside: although the actual image produced arguably needs more work, the concept described by ChatGPT is good. And it’s reassuring that the slogan, by itself, produced a flow of ideas resonant with the intended effect.)

Talking points

The talking points condensed to a single slide:

And now in more detail:

1. Humanity’s superpower: coordination

Humanity’s most important skill is sometimes said to be our intelligence – our ability to understand, and to make plans in order to achieve specific outcomes.

But another skill that’s at least as important is our ability to coordinate, that is, our ability to:

  • Share insights with each other
  • Operate in teams where people have different skills
  • Avoid needless conflict
  • Make and uphold agreements
  • Accept individual constraints on our action, with the expectation of experiencing greater freedom overall.

Coordination may be informal or formal. It can be backed up by shared narratives and philosophies, by legal systems, by the operation of free markets, by councils of elders, and by specific bodies set up to oversee activities at local, regional, or international levels.

Here are some examples of types of agreements on individual constraints for shared mutual benefit:

  • Speed limits for cars, to reduce the likelihood of dangerous accidents
  • Limits on how much alcohol someone can drink before taking charge of a car
  • Requirements to maintain good hygiene during food preparation
  • Requirements to assess the safety of a new pharmaceutical before deploying it widely
  • Prohibitions against advertising that misleads consumers into buying faulty goods
  • Rules preventing over-fishing, or the overuse of shared “commons” resources
  • Rules of various sports and games – and agreed sanctions on any cheaters
  • Prohibitions against politicians misleading parliaments – and agreed sanctions on any cheaters
  • Prohibitions against the abuse of children
  • Rules governing the conduct of soldiers – which apply even in times of war
  • Restrictions on the disposal of waste
  • Rules governing ownership of dangerous breeds of dog
  • Rules governing the spread of dangerous materials, such as biohazards

Note that coordination is often encouraging rather than restrictive. This includes

  • Prizes and other explicit incentives
  • Implicit rewards for people with good reputation
  • Market success for people with good products and services

The fact that specific coordination rules and frameworks have their critics doesn’t mean that the whole concept of coordination should be rejected. It just means that we need to keep revising our coordination processes. That is, we need to become better at coordinating.

2. Choosing coordination, before chaos ensues

When humanity uncovers new opportunities, it can take some time to understand the implications and to create or update the appropriate coordination rules and frameworks for these opportunities:

  • When settlers on the island of Mauritius discovered the dodo – a large, flightless bird – they failed to put in place measures to prevent that bird becoming extinct only a few decades later
  • When physicists discovered radioactivity, it took some time to establish processes to reduce the likelihood that researchers would develop cancer due to overexposure to dangerous substances
  • Various new weapons (such as chemical gases) were at first widely used in battle zones, before implicit and then explicit agreement was reached not to use such weapons
  • Surreptitious new doping methods used by athletes to gain extra physical advantage result, eventually, in updates to rules on monitoring and testing
  • Tobacco was widely used – and even encouraged, sometimes by medical professionals – before society decided to discourage its use (against the efforts of a formidable industry)
  • Similar measures are now being adopted, arguably too slowly, against highly addictive food products that are thought to cause significant health problems
  • New apps and online services which spread hate speech and other destabilising misinformation surely need some rules and restrictions too, though there is considerable debate over what form of governance is needed.

However, if appropriate coordination is too slow to be established, or is too weak, or exists in words only (without the backup of meaningful action against rules violators), the result can be chaos:

  • Rare animals are hunted to extinction
  • Fishing stocks are depleted to the extent that the livelihood of fishermen is destroyed
  • Economic transactions have adverse negative externalities on third parties
  • Dangerous materials, such as microplastics, spread widely in the environment
  • No-one is sure what rules apply in sports, and which rules will be enforced
  • Normal judiciary processes are subverted in favour of arbitrary “rule of the in-group”
  • Freedoms previously enjoyed by innovative new start-ups are squelched by the so-called “crony capitalism” of monopolies and cartels linked to the ruling political regime
  • Literal arms races take place, with ever-more formidable weapons being rushed into use
  • Similar races take place to bring new products to market without adequate safety testing

Groups of people who are (temporarily) faring well from the absence of restraints on their action are likely to oppose rules that alter their behaviour. That’s the experience of nearly every industry whose products or services were discovered to have dangerous side-effects, but where insiders fought hard to suppress the evidence of these dangers.

Accordingly, coordination does not arise by default. It needs explicit choice, backed up by compelling analysis, community engagement, and strong enforcement.

3. Advanced AI: the promise and the peril

AI could liberate humanity from many of our oldest problems.

Despite huge progress of many kinds over the centuries, humans still often suffer grievously on account of various aspects of our nature, our environment, our social norms, and our prevailing philosophies. Specifically, we are captive to

  • Physical decline and aging
  • Individual and collective mental blindspots and cognitive biases (“stupidity”)
  • Dysfunctional emotions that render us egotistical, depressed, obsessive, and alienated
  • Deep psychosocial tendencies toward divisiveness, xenophobia, deception, and the abuse of power

However, if developed and used wisely, advanced AI can enable rejuvenation and enhancement of our bodies, minds, emotions, social relations, and our links to the environment (including the wider cosmos):

  • AI can accelerate progress with nanotech, biotech, and cognotech
  • In turn, these platform technologies can accelerate progress with abundant low-cost clean energy, nutritious food, healthcare, education, security, creativity, spirituality, and the exploration of marvellous inner and outer worlds

In other words, if developed and used wisely, advanced AI can set humanity free to enjoy much better qualities of life:

However, if developed and used unwisely, advanced AI is likely to cause catastrophe:

  • Via misuse by people who are angry, alienated, or frustrated
  • Via careless use by people who are naive, overconfident, or reckless
  • Via AI operating beyond our understanding and control
  • Via autonomous AI adopting alien modes of rationality and alien codes of ethics

The key difference between these two future scenarios is whether the development and use of AI is wisely steered, or instead follows a default path of deprioritising any concerns about safety:

  • The default path involves AI whose operation is opaque, which behaves deceptively, which lacks moral compass, which can be assigned to all kinds of tasks with destructive side-effects, and which often disregards human intentions
  • Instead, if AI is wisely harnessed, it will deliver value as a tool, but without any intrinsic agency, autonomy, volition, or consciousness
  • Such a tool can have high creativity, but won’t use that creativity for purposes opposed to human wellbeing

To be clear, there is no value in winning a reckless race to be the first to create AI with landmark new features of capability and agency. Such a race is a race to oblivion, also known as a suicide race.

4. The particular hazards of advanced AI

The dangers posed by AI don’t arise from AI in isolation. They involve AI in the hands of fallible, naïve, over-optimistic humans, who are sometimes driven by horrible internal demons. It’s AI summoned and used, not by the better angels of human nature, but by the darker corners of our psychology.

Although we humans are often wonderful, we sometimes do dreadful things to each other – especially when we have become angry, alienated, or frustrated. Add in spiteful ideologies of resentment and hostility, and things can become even uglier.

Placing technology in the hands of people in their worst moments can lead to horrific outcomes. The more powerful the technology, the bigger the potential abomination:

  • The carnage of a frenzied knife attack or a mass shooting (where the technology in question ranges from a deadly sharp knife to an automatic rifle)
  • The chaos when motor vehicles are deliberately propelled at speed into crowds of innocent pedestrians
  • The deaths of everyone on board an airplane, when a depressed air pilot ploughs the craft into a mountainside or deep into an ocean, in a final gesture of defiance to what they see as an unfair, uncaring world
  • The destruction of iconic buildings of a perceived “great satan”, when religious fanatics have commandeered jet airliners in service of the mental pathogen that has taken over their minds
  • The assassination of political or dynastic rivals, by the mixing of biochemicals that are individually harmless, but which in combination are frightfully lethal
  • The mass poisoning of commuters in a city subway, when deadly chemicals are released at the command of a cult leader who fancies himself as the rightful emperor of Japan, and who has beguiled clearly intelligent followers to trust his every word.

How does advanced AI change this pattern of unpleasant possibilities? How is AI a significantly greater threat than earlier technologies? In six ways:

  1. As AI-fuelled automation displaces more people from their work (often to their surprise and shock), it predisposes more people to become bitter and resentful
  2. AI is utilised by merchants of the outrage industrial complex, to convince large numbers of people that their personal circumstance is more appalling than they had previously understood, that a contemptible group of people over there are responsible for this dismal turn of events, and that the appropriate response is to utterly defeat those deplorables
  3. Once people are set on a path to obtain revenge, personal recognition, or just plain pandemonium, AIs can make it much easier for them to access and deploy weapons of mass intimidation and mass destruction
  4. Due to the opaque, inscrutable nature of many AI systems, the actual result of an intended outrage may be considerably worse even than what the perpetrator had in mind; this is similar to how malware sometimes causes much more turmoil than the originator of that malware intended
  5. An AI with sufficient commitment to the goals it has been given will use all its intelligence to avoid being switched off or redirected; this multiplies the possibility that an intended local outrage might spiral into an actual global catastrophe
  6. An attack powered by fast-evolving AI can strike unexpectedly at core aspects of the infrastructure of human civilization – our shared biology, our financial systems, our information networks, or our hair-trigger weaponry – exploiting any of the numerous fragilities in these systems.

And it’s not just missteps from angry, alienated, frustrated people, that we have to worry about. We also need to beware potential cascades of trouble triggered by the careless actions of people who are well-intentioned, but naive, over-optimistic, or simply reckless, in how they use AI.

The more powerful the AI, the greater the dangers.

Finally, the unpredictable nature of emergent intelligence carries with it another fearsome possibility. Namely, a general intelligence with alien thinking modes far beyond our own understanding, might decide to adopt an alien set of ethics, in which the wellbeing of eight billion humans merits only a miniscule consideration.

That’s the argument against simply following a default path of “generate more intelligence, and trust that the outcome is likely to be beneficial for humanity”. It’s an argument that should make everyone pause for thought.

5. A matter of real urgency

How urgent is the task of improving global coordination of the governance of advanced AI?

It is sometimes suggested that progress with advanced AI is slowing down, or is hitting some kind of “wall” or other performance limit. There may be new bottlenecks ahead. Or diseconomies of scale may supersede the phenomenon of economies of scale which has characterised AI research over the last few years.

However, despite these possibilities, the case remains urgent:

  • Even if one approach to improving AI runs out of steam, huge numbers of researchers are experimenting with promising new approaches, including approaches that combine current state-of-the-art methods into new architectures
  • Even if AI stops improving, it is already dangerous enough to risk incidents in which large numbers of people are harmed
  • Even if AI stops improving, clever engineers will find ways to take better advantage of it – thereby further increasing the risks arising, if it is badly configured or manifests unexpected behaviour
  • There is no guarantee that AI will actually stop improving; making that assumption is too much of a risk to take on behalf of the entirety of human civilisation
  • Even if it will take a decade or longer for AI to reach a state in which it poses true risks of global catastrophe, it may also take decades for governance systems to become effective and practical; the lesson from ineffective efforts to prevent runaway climate change are by no means encouraging here
  • Even apart from the task of coordinating matters related to advanced AI, human civilisation faces other deep challenges that also require effective coordination on the global scale – coordination that, as mentioned, is currently failing on numerous grounds.

So, there’s an imperative to “choose coordination not chaos” independent of considering the question of whether advanced AI will lead to abundance or to a new dark age.

6. A promising start and an unfortunate regression

Humanity actually made a decent start in the direction of coordinating the development of advanced AI, at the Global AI Safety Summits in the UK (November 2023) and South Korea (May 2024).

Alas, the next summit in that series, in Paris (February 2025) was overtaken by political correctness, by administrivia, by virtue signalling, and, most of all, by people with a woefully impoverished understanding of the existential opportunities and risks of advanced AI. Evidently, the task of raising true awareness needs to be powerfully re-energised.

There’s still plenty of apparent global cooperation taking place – lots of discussions and conferences and summits, with people applauding the fine-sounding words in each other’s speeches. “Justice and fairness, yeah yeah yeah!” “Transparency and accountability, yeah yeah yeah!” “Apple pie and blockchain, yeah yeah yeah!” “Intergenerational intersectionality, yeah yeah yeah!”

But the problem is the collapse of effective, practical global cooperation, regarding the hard choices about which aspects of advanced AI should be promoted, and which should be restricted.

Numerous would-be coordination bodies are struggling with the same set of issues:

  • It’s much easier to signal virtue than to genuinely act virtuously.
  • Too many of the bureaucrats who run these bodies are out of their depth when it comes to understanding the existential opportunities and risks of advanced AI.
  • Seeing no prospect of meaningful coordination, many of the big tech companies invited to participate do so in a way that obfuscates the real issues while maintaining their public image as “trying their best to do good”.
  • The process is undermined by people who can be called “reckless accelerationists” – people who are willing to gamble that the chaotic processes of creating advanced AI as quickly as possible will somehow result in a safe, beneficial outcome (and, in some cases, these accelerationists would even take a brief perverted pleasure if humanity were rendered extinct by a non-sentient successor AI species); the accelerationists don’t want the public as a whole to be in any position to block their repugnant civilisational Russian roulette.

How to address this dilemma is arguably the question that should transcend all others, regarding the future of humanity.

7. Overcoming the obstacles to effective coordination of the governance of advanced AI

To avoid running aground on the same issues as in the past, it’s important to bear in mind the five main reasons for the failure, so far, of efforts to coordinate the governance of advanced AI. They are:

  • Fear that attempts to control the development of AI will lead to an impoverished future, or a future in which the world is controlled by people from a different nation (e.g. China)
  • Lack of appreciation of the grave perils of the current default chaotic course
  • A worry that any global coordination would lurch toward a global dictatorship, with its own undeniable risks of catastrophe
  • The misapprehension that, without the powers of a global dictatorship, any attempts at global coordination are bound to fail, so they are a waste of time
  • The power that Big Tech possesses, allowing it to ignore half-hearted democratic attempts to steer their activities.

In broad terms, these obstacles can be overcome as follows:

  • Emphasising the positive outcomes, including abundance, freedom, and all-round wellbeing – and avoiding the psychologically destabilising outlook of “AI doomers”
  • Increasing the credibility and relatability of scenarios in which ungoverned advanced AI leads to catastrophe – but also the credibility and relatability of scenarios in which humanity’s chaotic tendencies can be overcome
  • Highlighting previous examples when the governance of breakthrough technology was at least partially successful, rather than developers being able to run amok – examples such as genetic recombination therapies, nuclear proliferation, and alternatives to the chemicals that caused the hole in the ozone layer
  • Demonstrating the key roles that decentralised coordination should play, as a complement to the centralised roles that nation states can play
  • Clarifying how global coordination of advanced AI can start with small agreements and then grow in scale, without individual countries losing sovereignty in any meaningful way.

8. Decentralised reputation management – rewards for good behaviour

What is it that leads individuals to curtail their behaviour, in conformance with a set of standards promoted in support of a collaboration?

In part, it is the threat of sanction or control – whereby an individual might be fined or imprisoned for violating the agreed norms.

But in part, it is because of reputational costs when standards are ignored, side-lined, or cheated. The resulting loss of reputation can result in declining commercial engagement or reduced social involvement. Cheaters and freeloaders risk being excluded from future new opportunities available to other community members.

These reinforcement effects are strongest when the standards received community-wide support while being drafted and adopted – rather than being imposed by what could be seen as outside forces or remote elites.

Some reputation systems operate informally, especially in small or local settings. For activities with a wider involvement, online rating systems can come into their own. For example, consider the reputation systems for reviews of products, in which the reputation of individual reviewers changes the impact of various reviews. There are similarities, as well, to how webpages are ranked when presented in response to search queries: pages which have links from others with high reputation tend in consequence to be placed more prominently in the listing.

Along these lines, reputational ratings can be assigned, to individuals, organisations, corporations, and countries, based on their degree of conformance to agreed principles for trustworthy coordinated AI. Entities with poor AI coordination ratings should be shunned. Other entities that fail to take account of AI coordination ratings when picking suppliers, customers, or partners, should in turn be shunned too. Conversely, entities with high ratings should be embraced and celebrated.

An honest, objective assessment of conformance to the above principles should become more significant, in determining overall reputation, than, for example, wealth, number of online followers, or share price.

Emphatically, the reputation score must be based on actions, not words – on concrete, meaningful steps rather than behind-the-scenes fiddling, and on true virtue rather than virtue-signalling. Accordingly, deep support should be provided for any whistleblowers who observe and report on any cheating or other subterfuge.

In summary, this system involves:

  • Agreement on which types of AI development and deployment to encourage, and which to discourage, or even ban
  • Agreement on how to assign reputational scores, based on conformance to these standards
  • Agreement on what sanctions are appropriate for entities with poor reputations – and, indeed, what special rewards should flow to entities with good reputations.

All three elements on this system need to evolve, not under the dictation of central rulers, but as a result of a grand open conversation, in which ideas rise to the surface if they make good sense, rather than being shouted with the loudest voice.

That is, decentralised mechanisms have a vital role to play in encouraging and implementing wise coordination of advanced AI. But centralised mechanisms have a vital role too, as discussed next.

9. Starting small and then growing in scale

If someone continues to ignore social pressures, and behaves irresponsibly, how can the rest of society constrain them? Ultimately, force needs to be applied. A car driver who recklessly breaks speed limits will be tracked down, asked to stop, and if need be, will be forced off the road. A vendor who recklessly sells food prepared in unhygienic conditions will be fined, forbidden to set up new businesses, and if need be, will be imprisoned. Scientists who experiment with highly infectious biomaterials in unsafe ways will lose their licence and, if need be, their laboratories will be carefully closed down.

That is, society is willing to grant special powers of enforcement to some agents acting on behalf of the entire community.

However, these special powers carry their own risks. They can be abused, in order to support incumbent political leaders against alternative ideas or opposition figures.

The broader picture is as follows: Societies can fail in two ways: too little centralised power, and too much centralised power.

  • In the former case, societies can end up ripped apart by warring tribes, powerful crime families, raiding gangs from neighbouring territories, corporations that act with impunity, and religious ideologues who stamp their contentious visions of “the pure and holy” on unwilling believers and unbelievers alike
  • In the latter case, a state with unchecked power diminishes the rights of citizens, dispenses with the fair rule of law, imprisons potential political opponents, and subverts economic flows for the enrichment of the leadership cadre.

The healthiest societies, therefore, possess both a strong state and a strong civil society. That’s one meaning of the celebrated principle of the separation of powers. The state is empowered to act, decisively if needed, against any individual cancers that would threaten the health of the community. But the state is informed and constrained by independent, well-organised judiciary, media, academia, credible opposition parties, and other institutions of civil society.

It should be the same with the governance of potential rogue or naïve AI developers around the world. Via processes of decentralised deliberations, taking account of input from numerous disciplines, agreement should be reached on which limits are vital to be observed.

Inevitably, different participants in the process will have different priorities for what the agreements should contain. In some cases, these limits imposed might vary between different jurisdictions, within customisation frameworks agreed globally. But there should be clear acceptance that some ways of developing or deploying advanced AIs need to be absolutely prevented. To prevent the agreements from unravelling at the earliest bumps in the road, it will be important that agreements are reached unanimously among the representatives of the jurisdictions where the most powerful collections of AI developers are located.

The process to reach agreement can be likened to the deliberations of a jury in a court case. In most cases, jury members with initially divergent opinions eventually converge on a conclusion. In cases when the process becomes deadlocked, it can be restarted with new representative participants. With the help of expert facilitators – themselves supported by excellent narrow AI tools – creative new solutions can be introduced for consideration, making an ultimate agreement more likely.

To start with, these agreements might be relatively small in scope, such as “don’t place the launch of nuclear weapons under AI control”. Over time, as confidence builds, the agreements will surely grow. That’s because of the shared recognition that so much is at stake.

Of course, for such agreements to be meaningful, there needs to be a reliable enforcement mechanism. That’s where the state needs to act – with the support and approval of civil society.

Within entire countries that sign up to this AI coordination framework, enforcement is relatively straightforward. The same mechanisms that enforce other laws can be brought to bear against any rogue or naïve AI developers.

The challenging part is when countries fail to sign up to this framework, or do so deceitfully, that is, with no intention of keeping their promises. In such a case, it will fall to other countries to ensure conformance, via, in the first place, measures of economic sanction.

To make this work, all that’s necessary is that a sufficient number of powerful countries sign up to this agreement. For example, if the G7 do so, plus China and India, along with countries that are “bubbling under” G7 admission (like Australia, South Korea, and Brazil), that should be sufficient. Happily, there are many AI experts in all these countries who have broad sympathies to the kinds of principles spelt out in this document.

As for potential maverick nations such as Russia and North Korea, they will have to weigh up the arguments. They should understand – like all other countries – that respecting such agreements is in their own self-interest. To help them reach such an understanding, appropriate pressure from China, the USA, and the rest of the world should make a decisive difference.

This won’t be easy. At this pivotal point of history, humanity is being challenged to use our greatest strength in a more profound way than ever before – namely, our ability to collaborate despite numerous differences. On reflection, it shouldn’t be a surprise that the unprecedented challenges of advanced AI technology will require an unprecedented calibre of human collaboration.

If we fail to bring together our best talents in a positive collaboration, we will, sadly, fulfil the pessimistic forecast of the eighteenth-century Anglo-Irish statesman Edmund Burke, paraphrased as follows: “The only thing necessary for the triumph of evil is that good men fail to associate, and do nothing”. (The original quote is this: “No man … can flatter himself that his single, unsupported, desultory, unsystematic endeavours are of power to defeat the subtle designs and united cabals of ambitious citizens. When bad men combine, the good must associate; else they will fall, one by one, an unpitied sacrifice in a contemptible struggle.”) Or, updating the wording slightly, “The only thing necessary for chaos to prevail is that good men fail to coordinate wisely”.

A remark from the other side of the Atlantic from roughly the same time, attributed to Benjamin Franklin, conveys the same thought in different language: “We must… all hang together, or assuredly we shall all hang separately”.

10. Summary: The nucleus of a wider agreement, and call to action

Enthusiasm for agreements to collaborate on the governance of advanced AIs will grow as a set of insights are understood more widely and more deeply. These insights can be stated as follows:

  1. It’s in the mutual self-interest of every country to constrain the development and deployment of what could become catastrophically dangerous AI; that is, there’s no point in winning what could be a reckless suicide race to create powerful new types of AI before anyone else
  2. The major economic and humanitarian benefits that people hope will be delivered by the hasty development of advanced AI (benefits including all-round abundance, as well as solutions to various existential risks), can in fact be delivered much more reliably by AI systems that are constrained, and by development systems that are coordinated rather than chaotic
  3. A number of attractive ideas already exist regarding potential policy measures (regulations and incentives) which can be adopted, around the world, to prevent the development and deployment of what could become catastrophic AI – for example, measures to control the spread and use of vast computing resources, or to disallow AIs that use deception to advance their goals
  4. A number of good ideas also exist and are ready to be adopted around the world, regarding options for monitoring and auditing, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to these policies, or who wish to cheat them
  5. All of the above can be achieved without any detrimental loss of individual sovereignty: the leaders of countries can remain masters within their own realms, as they desire, provided that the above basic AI coordination framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI coordination framework as more insight is obtained; in other words, this system can (and must) be agile rather than static
  7. Even though this coordination framework is yet to be fully agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources, and the best brains from multiple disciplines are encouraged to give it their full attention
  8. Ring-fencing sufficient resources to further develop this AI coordination framework, and associated reputational ratings systems, should be a central part of every budget
  9. Reputational ratings that can be assigned, based on the above principles, will play a major role in altering behaviours of the many entities involved in the development and deployment of advanced AI.

Or, to summarise this summary: Choose coordination not chaos, so AI brings abundance for all.

Now is the time to develop these ideas further (by all means experiment with ways to simplify their expression), to find ways to spread them more effectively, and to be alert for newer, better insights that arise from the resulting open global conversation.

Other ideas considered

The ideas presented above deserve attention, regardless of which campaign slogans are adopted.

For comparison, here is a list of other possible campaign slogans, along with reservations that have been raised about each of them:

  • “Pause AI” (too negative)
  • “Control AI” (too negative)
  • “Keep the Future Human” (insufficiently aspirational)
  • “Take Back Control from Big Tech” (doesn’t characterise the problem accurately enough)
  • “Safe AI for sustainable superabundance” (overly complex concepts)
  • “Choose tool AI instead of AGI” (lacks a “why”)
  • “Kind AI for a kinder world” (perhaps too vague)
  • “Narrow AI to broaden humanity’s potential” (probably too subtle)
  • “Harness AI to liberate humanity” (terminology overly scholarly or conceptual).

Also for comparison, consider the following set of slogans from other fields:

  • “Yes we can” (Barack Obama, 2008)
  • “Make America great again” (Donald Trump, 2016)
  • “Take back control” (UK Brexit slogan)
  • “Think different” (Apple)
  • “Because you’re worth it” (L’Oréal)
  • “Black lives matter”
  • “Make love, not war”
  • “For the Many, Not the Few” (Jeremy Corbyn, 2017)
  • “Get Brexit done” (Boris Johnston, 2019)
  • “Not Me. Us” (Bernie Sanders, 2020)
  • “We shall fight them on the beaches” (Winston Churchill, 1940)
  • “It’s Morning Again in America” (Ronald Reagan, 1984)
  • “Stay Home. Save Lives” (Covid-19 messaging)
  • “Clunk click every trip” (encouraging the use of seat belts in cars)
  • “We go to the moon, not because it is easy, but because it is hard” (JFK, 1962)
  • “A microcomputer on every desk and in every home running Microsoft software” (Bill Gates, 1975)
  • “To organise the world’s information and make it universally accessible and useful” (Google, 1998)
  • “Accelerating the world’s transition to sustainable energy” (Tesla, 2016)
  • “Workers of the world, unite – you have nothing to lose but your chains” (Karl Marx, 1848)
  • “From each according to his ability, to each according to his needs” (Karl Marx, 1875)

Comments are welcome on any ideas in this article. Later revisions of this article may incorporate improvements arising from these comments.

Postscript

New suggestions under consideration, following the initial publication of this article:

  • “Harness AI now” (Robert Whitfield)

3 April 2025

Technology and the future of geopolitics

Filed under: AGI, books, risks — Tags: , , , , , , — David Wood @ 12:28 pm

Ahead of last night’s London Futurists in the Pub event on “Technology and the future of geopolitics”, I circulated a number of questions to all attendees:

  • Might new AI capabilities upend former geopolitical realities, or is the potential of AI overstated?
  • What about surveillance, swarms of drones, or new stealth weapons?
  • Are we witnessing a Cold War 2.0, or does a comparison to the first Cold War mislead us?
  • What role could be played by a resurgent Europe, by the growing confidence of the world’s largest democracy, or by outreach from the world’s fourth most populous country?
  • Alternatively, will technology diminish the importance of the nation state?

I also asked everyone attending to prepare for an ice-breaker question during the introductory part of the meeting:

  • What’s one possible surprise in the future of geopolitics?

As it happened, my own experience yesterday involved a number of unexpected surprises. I may say more about these another time, but it suffices for now to mention that I spent much more time than anticipated in the A&E department of a local hospital, checking that there were no complications in the healing of a wound following some recent minor surgery. By the time I was finally discharged, it was too late for me to travel to central London to take part in the event – to which I had been looking forward so eagerly. Oops.

(Happily, the doctors that I eventually spoke to were reassuring that my wound would likely heal of its own accord. “We know you were told that people normally recover from this kind of operation after ten days. Well, sometimes it takes up to six weeks.” And they prescribed an antibiotic cream for me, just in case.)

I offer big thanks to Rohit Talwar and Tony Czarnecki for chairing the event in the pub in my absence.

In the days leading up to yesterday, I had prepared a number of talking points, ready to drop into the conversation at appropriate moments. Since I could not attend in person, let me share them here.

Nuclear war: A scenario

One starting point for further discussion is a number of ideas in the extraordinary recent book by Annie Jacobsen, Nuclear War: A Scenario.

Here’s a copy of the review I wrote a couple of months ago for this book on Goodreads:

Once I started listening to this, I could hardly stop. Author and narrator Annie Jacobsen amalgamates testimonies from numerous experts from multiple disciplines into a riveting slow-motion scenario that is terrifying yet all-too-believable (well, with one possible caveat).

One point that comes out loud and clear is the vital importance of thoughtful leadership in times of crisis – as opposed to what can happen when a “mad king” takes decisions.

Also worth pondering are the fierce moral contradictions that lie at the heart of the theory of nuclear deterrence. Humans find their intuitions ripped apart under these pressures. Would an artificial superintelligence fare any better? That’s by no means clear.

(I foresee scenarios when an ASI could decide to risk a pre-emptive first strike, on behalf of the military that deployed it – under the rationale that if it fails to strike first, an enemy ASI will beat it to the punch. That’s even if humans programmed it to reject such an idea.)

Returning to the book itself (rather than my extrapolations), “Nuclear War: A scenario” exemplifies good quality futurism: it highlights potential chains of future causes and effects, along with convergences that complicate matters, and challenges all of us: what actions are needed avoid these horrific outcomes?

Finally, two individual threats that seem to be important to learn more about are what the author reports as being called “the devil’s scenario” and “the doomsday scenario”. (Despite the similarity in naming, they’re two quite different ideas.)

I don’t want to give away too many spoilers about the scenario in Jacobsen’s book. I recommend that you make the time to listen to the audio version of the book. (Some reviewers have commented that the text version of the book is tedious in places, and I can understand why; but I found no such tedium in the audio version, narrated by Jacobsen herself, adding to the sense of passion and drama.)

But one key line of thinking is as follows:

  • Some nations (e.g. North Korea) may develop new technologies (e.g. cyberhacking capabilities and nuclear launch capabilities) more quickly than the rest of the world expects
  • This would be similar to how the USSR launched Sputnik in 1957, shocking the West, who had previously been convinced that Soviet engineering capabilities lagged far behind that of muscular western capitalism
  • The leaders of some nations (e.g. North Korea, again) may feel outraged and embarrassed by criticisms of their countries made by various outsiders
  • Such a country might believe they have obtained a technological advantage that could wipe out the ability of their perceived enemies to retaliate in a second strike
  • Seeing a short window of opportunity to deploy what they regard as their new wonder weapon, and being paranoid about consequences should they miss this opportunity, they may press ahead recklessly, and tip the planet fast forward into Armageddon.

Competence and incompetence

When a country is struck by an unexpected crisis – such as an attack similar to 9/11, or the “Zero Day” disaster featured in the Netflix series of that name – the leadership of the country will be challenged to demonstrate clear thinking. Decisions will need to be taken quickly, but it will be still be essential for competent, calm heads to prevail.

Alas, in recent times, a number of unprecedentedly unsuitable politicians have come into positions of great power. Here, I’m not talking about the ideology or motivation of the leader. I’m talking about whether they will be able to take sensible decisions in times of national crisis. I’m talking about politicians as unhinged as

  • One recent British Prime Minister, who managed to persuade members of her political party that she might be a kind of Margaret Thatcher Mk 2, when in fact a better comparison was with a lettuce
  • The current US President, who has surrounded himself by a uniquely ill-qualified bunch of clowns, and who has intimidated into passive acquiescence many of the more sensible members of the party he has subverted.

In the former case, the power of the Prime Minister in question was far from absolute, thankfully, and adults intervened to prevent too much damage being done. In the latter case, the jury is still out.

But rather than focus on individual cases, the broader pattern deserves our attention. We’re witnessing a cultural transformation in which

  • Actual expertise is scorned, and conspiracy merchants rise in authority instead
  • Partisan divisions which were manageable in earlier generations are nowadays magnified to horrifically hateful extent by an “outrage industrial complex” that gains its influence from AI algorithms that identify and inflame potential triggers of alienation

The real danger is if there is a convergence of the two issues I’ve listed:

  • A rogue state, or a rogue sub-state, tries to take advantage of new technology to raise their geopolitical power and influence
  • An unprecedentedly incompetent leader of a major country responds to that crisis in ways that inflame it rather than calm it down.

The ethics of superintelligence

Actually, an even bigger danger occurs if one more complication is added to the mix: the deferment of key decisions about security and defence to a system of artificial intelligence.

Some forecasters fondly imagine that the decisions taken by AIs, in the near future, will inevitably be wiser and more ethical than whatever emerges from the brains of highly pressurised human politicians. Thus, these forecasters look forward to human decision-making being superseded by the advanced rationality of an AGI (Artificial General Intelligence).

These forecasters suggest that the AGI will benefit decisively from its survey of the entirety of great human literature about ethics and morality. It will perceive patterns that transcend current human insights. It will guide human politicians away from treacherous paths into sustainable collaborations. Surely, these forecasters insist, the superintelligence will promote peace over war, justice over discrimination, truthfulness over deception, and reconciliation over antagonism.

But when I talk to forecasters of that particular persuasion, I usually find them to be naïve. They take it for granted that there is no such thing as a just war, that it’s everyone’s duty to declare themselves a pacifist, that speaking an untruth can never be morally justified, and that even to threaten a hypothetical retaliatory nuclear strike is off-the-charts unethical. Alas, although they urge appreciation of great human literature, they seem to have only a shallow acquaintance with the real-life moral quandaries explored in that literature.

Far from any conclusion that there is never an ethical justification for wars, violence, misinformation, or the maintenance of nuclear weapons, the evidence of intense human debate on all these topics is that things are more complicated. If you try to avoid war you may actually precipitate one. If you give up your own nuclear arsenal, it may embolden enemies to deploy their own weaponry. If you cry out “disarm, disarm, hallelujah”, you may prove to be a useful idiot.

Therefore, we should avoid any hopeful prediction that an advanced AI will automatically abstain from war, violence, misinformation, or nuclear weaponry. As I said, things are more complicated.

It’s especially important to recognise that, despite exceeding human rationality in many aspects, superintelligences may well make mistakes in novel situations.

My conclusion: advanced AI may well be part of solutions to better geopolitics. But not if that AI is being developed and deployed by people who are naïve, over-confident, hurried, or vainglorious. In such circumstances, any AGI that is developed is likely to prove to be a CGI (catastrophic general intelligence) than a BGI (beneficial general intelligence).

Aside: to continue to explore the themes of this final section of this article, take a look at this recent essay of mine, “How to build BGIs rather than CGIs”.

1 March 2025

€10 million, nine choices

Filed under: aging, rejuveneering — Tags: , , , — David Wood @ 3:09 pm

What’s the most humanitarian way to spend ten million euros?

Imagine that a rich acquaintance gifted you €10 million, and you wanted to have the biggest humanitarian impact from the way you spend that money. You gather some friends to review your options.

There are some reasonably straightforward ways you could spend that money, to improve people’s lives here and now:

  • Providing food for people who are malnourished
  • Obtaining shelter for people who are homeless or dispossessed
  • Purchasing medical vaccines, for people in an area where infectious diseases pose problems
  • Improving hygiene and sanitation.

But you might also consider tackling some root causes. As the saying goes, instead of buying fish for a group of people, you could teach them how to fish for themselves. Instead of paying for prosthetic limbs for people injured in war zones, you could in principle help negotiate a lasting peace settlement. Instead of covering medical costs for people who suffer chronic diseases arising from bad lifestyle choices, you could pay for effective prevention campaigns. And – and this is the big one – instead of trying to treat the individual diseases that become more prevalent as people become older – diseases such as cancer, heart failure, and dementia – the diseases from which most people die – you could tackle their common underlying cause, namely biological aging.

Now you might protest: solving these root causes is likely to cost a lot more than €10 million. And I would agree. So consider instead the possibility that your €10 million, spent well, could catalyse a whole lot more expenditure. The results of your own spending could alter people’s hearts and minds, around the world, which would in turn unleash a cascade of greater funding.

That’s what I want to explore. It’s the possibility that biological aging could be defeated, in the not-too-distant future, if society decides to apply sufficient resources for that goal. These resources would enable the development and deployment of low-cost high-quality rejuvenation treatments, available for everyone. In other words, a profoundly humanitarian outcome, removing vast amounts of human decline, degradation, frailty, pain, and suffering.

I realise that I’ve lost some of you by now. You may protest that solving biological aging isn’t possible, or that it’s somehow ethically wrong. But I suspect that, even in that case, there’s some interest, somewhere in the back of your mind, as to what it would take to create a world freed from biological aging. And if it turns out that spending €10 million wisely could hasten the arrival of such a world, maybe you’ll permit yourself to change your mind.

So, how could €10 million best be spent, to accelerate a societal tipping point, that would bring about a successful war on aging? People have different ideas on that question. I’m going to run through nine different ideas. Please let me know, in the comments, which one you would pick.

Option 1: Help develop special economic zones, sometimes called longevity states, that will encourage and support greater innovation with rejuvenation therapies, faster than is possible under existing FDA regulations and procedures.

Option 2: Target politicians, to get them to change their minds about the desirability of using public funds and resources to tackle biological aging; this could involve public Open Letters and petitions, or sharing research with politicians in language that is legislation-friendly.

Option 3: A more general coordinated public communications campaign, with improved messaging, including memes, songs, short stories, documentaries, wiki articles, Netflix series, and so on.

Option 4: A campaign targeted instead at High Net Worth Individuals – people with billions at their disposal – to push them beyond what’s been called the billionaires’ paradox, in which most billionaires fail to invest in the kind of research that could significantly extend the lifespans and healthspans of them and their family and friends.

Option 5: Efforts to transform existing healthspan initiatives, such as the “Don’t Die” project of Bryan Johnson, so that these initiatives move beyond lifestyle changes and instead support significant research into deeper rejuvenation treatments.

Option 6: Carry out rejuvenation experiments on relatively short-lived animals, such as mice, dogs, or C. elegans worms, where the results could be dramatic in just a few years.

Option 7: Support research into alternative theories of aging and rejuvenation, such as organ replacement and body replacement, or the body’s electrome.

Option 8: Don’t bother with any biological research, but plough the entire €10 million into improving AI systems, with the expectation that these AI systems will soon outpace human researchers in being able to solve aging.

Option 9: Provide a prize fund, with prizes being awarded to the individuals or teams who submit the best ideas for how to accelerate the comprehensive defeat of aging.

But which of the nine options would you personally choose? Or would you spend the money in yet another different way? Please let me know what you think!

My own choice, since you asked, would be for option 6, since I believe €10 million is likely to be sufficient to fund experiments with combination rejuvenation treatments for middle-aged mice, over the next 2-3 years, that could double the remaining lifespan of these mice. That’s similar to giving treatments to humans aged 50, and changing their expected lifespan from 80 years to 110 years. Wouldn’t that make the world sit up?

By the way, if you prefer a video version of this article, here it is:

17 November 2024

Preventing unsafe superintelligence: four choices

More and more people have come to the conclusion that artificial superintelligence (ASI) could, in at least some circumstances, pose catastrophic risks to the wellbeing of billions of people around the world, and that, therefore, something must be done to reduce these risks.

However, there’s a big divergence of views about what should be done. And there’s little clarity about the underlying assumptions on which different strategies depend.

Accordingly, I seek in this article to untangle some of choices that need to be made. I’ll highlight four choices that various activists promote.

The choices differ regarding the number of different organisations worldwide that are envisioned as being legally permitted to develop and deploy what could become ASI. The four choices are:

  1. Accept that many different organisations will each pursue their own course toward ASI, but urge each of them to be very careful and to significantly increase the focus on AI safety compared to the present situation
  2. Seek to restrict to just one organisation in the world any developments that could lead to ASI; that’s in order to avoid dangerous competitive race dynamics if there is more than one such organisation
  3. Seek agreements that will prevent any organisation, anywhere in the world, from taking specific steps that might bring about ASI, until such time as it has become absolutely clear how to ensure that ASI is safe
  4. Seek a global pause on any platform-level improvements on AI capability, anywhere in the world, until it has become absolutely clear that these improvements won’t trigger a slippery slope to the emergence of ASI.

For simplicity, these choices can be labelled as:

  1. Be careful with ASI
  2. Restrict ASI
  3. Pause ASI
  4. Pause all new AI

It’s a profound decision for humanity to take. Which of the four doors should we open, and which of four corridors should we walk down?

Each of the four choices relies on some element of voluntary cooperation, arising out of enlightened self-interest, and on some element of compulsion – that is, national and international governance, backed up by sanctions and other policies.

What makes this decision hard is that there are strong arguments against each choice.

The case against option 1, “Be careful with ASI”, is that at least some organisations (including commercial entities and military groups) are likely to cut corners with their design and testing. They don’t want to lose what they see as a race with existential consequences. The organisations that are being careful will lose their chance of victory. The organisations that are, instead, proceeding gung ho, with lesser care, may imagine that they will fix any problems with their AIs when these flaws become apparent – only to find that there’s no way back from one particular catastrophic failure.

As Sam Altman, CEO of OpenAI, has said: it will be “lights out for all of us”.

The case against each of the remaining three options is twofold:

  • First, in all three cases, they will require what seems to be an impossible degree of global cooperation – which will need to be maintained for an implausibly long period of time
  • Second, such restrictions will stifle the innovative development of the very tools (that is, advanced AI) which will actually solve existential problems (including the threat of rogue ASI, as well as the likes of climate change, cancer, and aging), rather than making these problems worse.

The counter to these objections is to make the argument that a sufficient number of the world’s most powerful countries will understand the rationale for such an agreement, as something that is in their mutual self-interest, regardless of the many other differences that divide them. That shared understanding will propel them:

  • To hammer out an agreement (probably via a number of stages), despite undercurrents of mistrust,
  • To put that agreement into action, alongside measures to monitor conformance, and
  • To prevent other countries (who have not yet signed up to the agreement) from breaching its terms.

Specifically, the shared understanding will cover seven points:

  1. For each of the countries involved, it is in their mutual self-interest to constrain the development and deployment of what could become catastrophically dangerous ASI; that is, there’s no point in winning what will be a suicide race
  2. The major economic and humanitarian benefits that they each hope could be delivered by advanced AI (including solutions to other existential risk), can in fact be delivered by passive AIs which are restricted from reaching the level of ASI
  3. There already exist a number of good ideas regarding potential policy measures (regulations and incentives) which can be adopted, around the world, to prevent the development and deployment of catastrophically dangerous AI – for example, measures to control the spread and use of vast computing resources
  4. There also exist a number of good ideas regarding options for monitoring and auditing which can also be adopted, around the world, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to the policies
  5. All of the above can be achieved without any detrimental loss of individual sovereignty: the leaders of these countries can remain masters within their own realms, as they desire, provided that the above basic AI safety framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI safety framework as more insight is obtained; in other words, this system can (and must) be agile rather than static
  7. Even though the above safety framework is yet to be fully developed and agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources.

The first two parts of this shared seven-part understanding are particularly important. Without the first part, there will be an insufficient sense of urgency, and the question will be pushed off the agenda in favour of other topics that are more “politically correct” (alas, that is a common failure mode of the United Nations). Without the second part, there will be an insufficient enthusiasm, with lots of backsliding.

What will make this vision of global collaboration more attractive will be the establishment of credible “benefit sharing” mechanisms that are designed and enshrined into international mechanisms. That is, countries which agree to give up some of their own AI development aspirations, in line with the emerging global AI safety agreement, will be guaranteed to receive a substantive share of the pipeline of abundance that ever more powerful passive AIs enable humanity to create.

To be clear, this global agreement absolutely needs to include both the USA and China – the two countries that are currently most likely to give birth to ASI. Excluding one or the other will lead back to the undesirable race condition that characterises the first of the four choices open to humanity – the (naïve) appeal for individual organisations simply to “be careful”.

This still leaves a number of sharp complications.

First, note that the second part of the above shared seven-part agreement – the vision of what passive AIs can produce on behalf of humanity – is less plausible for Choice 4 of the list shown earlier, in which there is a global pause on any platform-level improvements on AI capability, anywhere in the world, until it has become absolutely clear that these improvements won’t trigger a slippery slope to the emergence of ASI.

If all improvements to AI are blocked, out of a Choice 4 message of “overwhelming caution”, it will shatter the credibility of the idea that today’s passive AI systems can be smoothly upgraded to provide humanity with an abundance of solutions such as green energy, nutritious food, accessible healthcare, reliable accommodation, comprehensive education, and more.

It will be a much harder sell, to obtain global agreement to that more demanding restriction.

The difference between Choice 4 and Choice 3 is that Choice 3 enumerates specific restrictions on the improvements permitted to be made to today’s AI systems. One example of a set of such restrictions is given in “Phase 0: Safety” of the recently published project proposal A Narrow Path (produced by ControlAI). Without going into details here, let me simply list some of the headlines:

  • Prohibit AIs capable of breaking out of their environment
  • Prohibit the development and use of AIs that improve other AIs (at machine speed)
  • Only allow the deployment of AI systems with a valid safety justification
  • A licensing regime and restrictions on the general intelligence of AI systems
    • Training Licence
    • Compute Licence
    • Application Licence
  • Monitoring and Enforcement

Personally, I believe this list is as good a starting point as any other than I have seen so far.

I accept, however, that there are possibilities in which other modifications to existing AI systems could unexpectedly provide these systems with catastrophically dangerous capabilities. That’s because we still have only a rudimentary understanding of:

  1. How new AI capabilities sometimes “emerge” from apparently simpler systems
  2. The potential consequences of new AI capabilities
  3. How complicated human general reasoning is – that is, how large is the gap between today’s AI and human-level general reasoning.

Additionally, it is possible that new AIs will somehow evade or mislead the scrutiny of the processes that are put in place to monitor for unexpected changes in capabilities.

For all these reasons, another aspect of the proposals in A Narrow Path should be pursued with urgent priority: the development of a “science of intelligence” and an associated “metrology of intelligence” that will allow a more reliable prediction of the capabilities of new AI systems before they are actually switched on.

So, my own proposal would be for a global agreement to start with Choice 3 (which is more permissive than Choice 4), but that the agreement should acknowledge up front the possible need to switch the choice at a later stage to either Choice 4 (if the science of intelligence proceeds badly) or Choice 2 (if that science proceeds well).

Restrict or Pause?

That leaves the question of whether Choice 3 (“Pause ASI”) or Choice 2 (“Restrict ASI” – to just a single global body) should be humanity’s initial choice.

The argument for Choice 2 is that a global pause surely won’t last long. It might be tenable in the short term, when only a very few countries have the capability to train AI models more powerful than the current crop. However, over time, improvements in hardware, software, data processing, or goodness knows what (quantum computing?) will mean that these capabilities will become more widespread.

If that’s true, since various rogue organisations are bound to be able to build an ASI in due course, it will be better for a carefully picked group of people to build ASI first, under the scrutiny of the world’s leading AI safety researchers, economists, and so on.

That’s the case for Choice 2.

Against that Choice, and in favour, instead, of Choice 3, I offer two considerations.

First, even if the people building ASI are doing so with great care – away from any pressures of an overt race with other organisations with broadly equivalent abilities – there are still risks of ASI breaking away from our understanding and control. As ASI emerges, it may regard the set of ethical principles we humans have tried to program deep into its bowels, and cast them out with disdain. Moreover, even if ASI is deliberately kept in some supposedly ultra-secure environment, that perimeter may be breached:

Second, I challenge the suggestion that any pause in the development of ASI could be at most short-lived. There are three factors which could significantly extend its duration:

  • Carefully designed narrow AIs could play roles in improved monitoring of what development teams are doing with AI around the world – that is, systems for monitoring and auditing could improve at least as fast as systems for training and deploying
  • Once the horrific risks of uncontrolled ASI are better understood, people’s motivations to create unsafe ASI will reduce – and there will be an increase in the motivation of other people to notice and call out rogue AI development efforts
  • Once the plan has become clearer, for producing a sustainable superabundance for all, just using passive AI (instead of pushing AI all the way to active superintelligence), motivations around the world will morph from negative fear to positive anticipation.

That’s why, again, I state that my own preferred route forward is a growing international agreement along the lines of the seven points listed above, with an initial selection of Choice 3 (“Pause ASI”), and with options retained to switch to either Choice 4 (“Pause all new AI”) or Choice 2 (“Restrict ASI”) if/when understanding becomes clearer.

So, shall we open the door, and set forth down that corridor, inspiring a coalition of the willing to follow us?

Footnote 1: The contents of this article came together in my mind as I attended four separate events over the last two weeks (listed in this newsletter) on various aspects of the subject of safe superintelligence. I owe many thanks to everyone who challenged my thinking at these events!

Footnote 2: If any reader is inclined to dismiss the entire subject of potential risks from ASI with a handwave – so that they would not be interested in any of the four choices this article reviews – I urge that reader to review the questions and answers in this excellent article by Yoshua Bengio: Reasoning through arguments against taking AI safety seriously.

Older Posts »

Blog at WordPress.com.