dw2

25 August 2025

The biggest blockages to successful governance of advanced AI

Filed under: AGI, books, Millennium Project, politics, risks, Singularity Principles, Transpolitica — Tags: , , , , , — David Wood @ 12:13 am

“Humanity has never faced a greater problem than itself.”

That phrase was what my brain hallucinated, while I was browsing the opening section of the Introduction of the groundbreaking new book Global Governance of the Transition to Artificial General Intelligence written by my friend and colleague Jerome C. Glenn, Executive Director of The Millennium Project.

I thought to myself: That’s a bold but accurate way of summing up the enormous challenge faced by humanity over the next few years.

In previous centuries, our biggest problems have often come from the environment around us: deadly pathogens, devastating earthquakes, torrential storms, plagues of locusts – as well as marauding hordes of invaders from outside our local neighbourhood.

But in the second half of the 2020s, our problems are being compounded as never before by our own human inadequacies:

  • We’re too quick to rush to judgement, seeing only parts of the bigger picture
  • We’re too loyal to the tribes to which we perceive ourselves as belonging
  • We’re overconfident in our ability to know what’s happening
  • We’re too comfortable with manufacturing and spreading untruths and distortions
  • We’re too bound into incentive systems that prioritise short-term rewards
  • We’re too fatalistic, as regards the possible scenarios ahead.

You may ask, What’s new?

What’s new is the combination of these deep flaws in human nature with technology that is remarkably powerful yet opaque and intractable. AI that is increasingly beyond our understanding and beyond our control is being coupled in potentially devastating ways with our over-hasty, over-tribal, over-confident thoughts and actions. New AI systems are being rushed into deployment and used in attempts:

  • To manufacture and spread truly insidious narratives
  • To incentivize people around the world to act against their own best interests, and
  • To resign people to inaction when in fact it is still within their power to alter and uplift the trajectory of human destiny.

In case this sounds like a counsel of despair, I should clarify at once my appreciation of aspects of human nature that are truly wonderful, as counters to the negative characteristics that I have already mentioned:

  • Our thoughtfulness, that can counter rushes to judgement
  • Our collaborative spirit, that can transcend partisanship
  • Our wisdom, that can recognise our areas of lack of knowledge or lack of certainty
  • Our admiration for truth, integrity, and accountability, that can counter ends-justify-the-means expediency
  • Our foresight, that can counter short-termism and free us from locked-in inertia
  • Our creativity, to imagine and then create better futures.

Just as AI can magnify the regrettable aspects of human nature, so also it can, if used well, magnify those commendable aspects.

So, which is it to be?

The fundamental importance of governance

The question I’ve just asked isn’t a question that can be answered by individuals alone. Any one group – whether an organisation, a corporation, or a decentralised partnership – can have its own beneficial actions overtaken and capsized by catastrophic outcomes of groups that failed to heed the better angels of their nature, and which, instead, allowed themselves to be governed by wishful naivety, careless bravado, pangs of jealousy, hostile alienation, assertive egotism, or the madness of the crowd.

That’s why the message of this new book by Jerome Glenn is so timely: the processes of developing and deploying increasingly capable AIs are something that needs to be:

  • Governed, rather than happening chaotically
  • Globally coordinated, rather than there being no cohesion between the different governance processes applicable in different localities
  • Progressed urgently, without being shut out of mind by all the shorter-term issues that, understandably, also demand governance attention.

Before giving more of my own thoughts about this book, let me share some of the commendations it has received:

  • “This book is an eye-opening study of the transition to a completely new chapter of history.” – Csaba Korösi, 77th President of the UN General Assembly
  • “A comprehensive overview, drawing both on leading academic and industry thinkers worldwide, and valuable perspectives from within the OECD, United Nations.” – Jaan Tallinn, founding engineer, Skype and Kazaa; co-founder, Cambridge Centre for the Study of Existential Risk and the Future of Life Institute
  • “Written in lucid and accessible language, this book is a must read for people who care about the governance and policy of AGI.” – Lan Xue, Chair of the Chinese National Expert Committee on AI Governance.

The book also carries an absorbing foreword by Ben Goertzel. In this foreword, Ben introduces himself as follows:

Since the 1980s, I have been immersed in the field of AI, working to unravel the complexities of intelligence and to build systems capable of emulating it. My journey has included introducing and popularizing the concept of AGI, developing innovative AGI software frameworks such as OpenCog, and leading efforts to decentralize AI development through initiatives like SingularityNET and the ASI Alliance. This work has been driven by an understanding that AGI is not just an engineering challenge but a profound societal pivot point – a moment requiring foresight, ethical grounding, and global collaboration.

He clarifies why the subject of the book is so important:

The potential benefits of AGI are vast: solutions to climate change, the eradication of diseases, the enrichment of human creativity, and the possibility of postscarcity economies. However, the risks are equally significant. AGI, wielded irresponsibly or emerging in a poorly aligned manner, could exacerbate inequalities, entrench authoritarianism, or unleash existential dangers. At this critical juncture, the questions of how AGI will be developed, governed, and integrated into society must be addressed with both urgency and care.

The need for a globally participatory approach to AGI governance cannot be overstated. AGI, by its nature, will be a force that transcends national borders, cultural paradigms, and economic systems. To ensure its benefits are distributed equitably and its risks mitigated effectively, the voices of diverse communities and stakeholders must be included in shaping its development. This is not merely a matter of fairness but a pragmatic necessity. A multiplicity of perspectives enriches our understanding of AGI’s implications and fosters the global trust needed to govern it responsibly.

He then offers wide praise for the contents of the book:

This is where the work of Jerome Glenn and The Millennium Project may well prove invaluable. For decades, The Millennium Project has been at the forefront of fostering participatory futures thinking, weaving together insights from experts across disciplines and geographies to address humanity’s most pressing challenges. In Governing the Transition to Artificial General Intelligence, this expertise is applied to one of the most consequential questions of our time. Through rigorous analysis, thoughtful exploration of governance models, and a commitment to inclusivity, this book provides a roadmap for navigating the complexities of AGI’s emergence.

What makes this work particularly compelling is its grounding in both pragmatism and idealism. It does not shy away from the technical and geopolitical hurdles of AGI governance, nor does it ignore the ethical imperatives of ensuring AGI serves the collective good. It recognizes that governing AGI is not a task for any single entity but a shared responsibility requiring cooperation among nations, corporations, civil society, and, indeed, future AGI systems themselves.

As we venture into this new era, this book reminds us that the transition to AGI is not solely about technology; it is about humanity, and about life, mind, and complexity in general. It is about how we choose to define intelligence, collaboration, and progress. It is about the frameworks we build now to ensure that the tools we create amplify the best of what it means to be human, and what it means to both retain and grow beyond what we are.

My own involvement

To fill in some background detail: I was pleased to be part of the team that developed the set of 22 critical questions which sat at the heart of the interviews and research which are summarised in Part I of the book – and I conducted a number of the resulting interviews. In parallel, I explored related ideas via two different online Transpolitica surveys:

And I’ve been writing roughly one major article (or giving a public presentation) on similar topics every month since then. Recent examples include:

Over this time period, my views have evolved. I see the biggest priority, nowadays, not as figuring out how to govern AGI as it comes into existence, but rather, how to pause the development and deployment of any new types of AI that could spark the existence of self-improving AGI.

That global pause needs to last long enough that the global community can justifiably be highly confident that any AGI that will subsequently be built will be what I have called a BGI (a Beneficial General Intelligence) rather than a CGI (a Catastrophic General Intelligence).

Govern AGI and/or Pause the development of AGI?

I recently posted a diagram on various social media platforms to illustrate some of the thinking behind that stance of mine:

Alongside that diagram, I offered the following commentary:

The next time someone asks me what’s my p(Doom), compared with my p(SSfA) (the probability of Sustainable Superabundance for all), I may try to talk them through a diagram like this one. In particular, we need to break down the analysis into two cases – will the world keep rushing to build AGI, or will it pause from that rush.

To explain some points from the diagram:

We can reach the very desirable future of SSfA by making wise use of AI only modestly more capable than what we have today;
We might also get there as a side-effect of building AGI, but that’s very risky.

None of the probabilities are meant to be considered precise. They’re just ballpark estimates.

I estimate around 2/3 chance that the world will come to its senses and pause its current headlong rush toward building AGI.

But even in that case, risks of global catastrophe remain.

The date 2045 is also just a ballpark choice. Either of the “singularity” outcomes (wonderful or dreadful) could arrive a lot sooner than that.

The 1/12 probability I’ve calculated for “stat” (I use “stat” here as shorthand for a relatively unchanged status quo) by 2045 reflects my expectation of huge disruptions ahead, one sort or another.

The overall conclusion: if we want SSfA, we’re much more likely to get it via the “pause AGI” branch than via the “headlong rush to AGI” branch.

And whilst doom is possible in either branch, it’s much more likely in the headlong rush branch.

For more discussion of how to get the best out of AI and other cataclysmically disruptive technologies, see my book The Singularity Principles (the entire contents are freely available online).

Feel free to post your own version of this diagram, with your own estimates of the various conditional probabilities.

As indicated, I was hoping for feedback, and I was pleased to see a number of comments and questions in response.

One excellent question was this, by Bill Trowbridge:

What’s the difference between:
(a) better AI, and
(b) AGI

The line is hard to draw. So, we’ll likely just keep making better AI until it becomes AGI.

I offered this answer:

On first thought, it may seem hard to identify that distinction. But thankfully, we humans don’t just throw up our hands in resignation every time we encounter a hard problem.

For a good starting point on making the distinction, see the ideas in “A Narrow Path” by Control AI.

But what surprised me the most was the confidence expressed by various online commenters that:

  • “A pause however desirable is unlikely: p(pause) = 0.01”
  • “I am confident in saying this – pause is not an option. It is actually impossible.”
  • “There are several organisations working on AI development and at least some of them are ungovernable [hence a pause can never be global]”.

There’s evidently a large gulf behind the figure of 2/3 that I suggested for P(pause), and the views of these clearly intelligent respondents.

Why a pause isn’t that inconceivable

I’ll start my argument on this topic by confirming that I see this discussion as deeply important. Different viewpoints are welcome, provided they are held thoughtfully and offered honestly.

Next, although it’s true that some organisations may appear to be ungovernable, I don’t see any fundamental issue here. As I said online,

“Given sufficient public will and/or political will, no organisation is ungovernable.”

Witness the compliance by a number of powerful corporations in both China and the US to control measures declared by national governments.

Of course, smaller actors and decentralized labs pose enforcement challenges, but these labs are less likely to be able to marshal sufficient computing capabilities to be the first to reach breakthrough new levels of capability, especially if decentralised monitoring of dangerous attributes is established.

I’ve drawn attention on previous occasions to the parallel with the apparent headlong rush in the 1980s toward nuclear weapons systems that were ever more powerful and ever more dangerous. As I explained at some length in the “Geopolitics” chapter of my 2021 book Vital Foresight, it was an appreciation of the horrific risks of nuclear winter (first articulated in the 1980s) that helped to catalyse a profound change in attitude amongst the leadership camps in both the US and the USSR.

It’s the wide recognition of risk that can provide the opportunity for governments around the world to impose an effective pause in the headlong rush toward AGI. But that’s only one of five steps that I believe are needed:

  1. Awareness of catastrophic risks
  2. Awareness of bottlenecks
  3. Awareness of mechanisms for verification and control
  4. Awareness of profound benefits ahead
  5. Awareness of the utility of incremental progress

Here are more details about these five steps I envision:

  1. Clarify in an undeniable way how superintelligent AIs could pose catastrophic risks of human disaster within just a few decades or even within years – so that this topic receives urgent high-priority public attention
  2. Highlight bottlenecks and other locations within the AI production pipeline where constraints can more easily be applied (for example, distribution of large GPU chip clusters, and the few companies that are providing unique services in the creation of cutting-edge chips)
  3. Establish mechanisms that go beyond “trust” to “trust and verify”, including robust independent monitors and auditors, as well as tamperproof remote shut-down capabilities
  4. Indicate how the remarkable benefits anticipated for humanity from aspects of superintelligence can be secured, more safely and more reliably, by applying the governance mechanisms of points 2 and 3 above, rather than just blindly trusting in a no-holds-barred race to be the first to create superintelligence
  5. Be prepared to start with simpler agreements, involving fewer signatories and fewer control points, and be ready to build up stronger governance processes and culture as public consensus and understanding moves forward.

Critics can assert that each of these five steps is implausible. In each case, there are some crunchy discussions to be had. What I find dangerous, however, isn’t when people disagree with my assessments on plausibility. It’s when they approach the questions with what seems to be

  • A closed mind
  • A tribal loyalty to their perceived online buddies
  • Overconfidence that they already know all relevant examples and facts in this space
  • A willingness to distract or troll, or to offer arguments not in good faith
  • A desire to protect their flow of income, rather than honestly review new ideas
  • A resignation to the conclusion that humanity is impotent.

(For analysis of a writer who displays several of these tendencies, see my recent blogpost on the book More Everything Forever by Adam Beck.)

I’m not saying any of this will be easy! It’s probably going to be humanity’s hardest task over our long history.

As an illustration of points worthy of further discussion, I offer this diagram that highlights strengths and weakness of both the “governance” and “pause” approaches:

DimensionGovernance (Continue AGI Development with Oversight)Pause (Moratorium on AGI Development)
Core StrategyImplement global rules, standards, and monitoring while AGI is developedImpose a temporary but enforceable pause on new AGI-capable systems until safety can be assured
AssumptionsGovernance structures can keep pace with AI progress;
Compliance can be verified		Public and political will can enforce a pause;
Technical progress can be slowed
BenefitsEncourages innovation while managing risks;
Allows early harnessing of AGI for societal benefit;
Promotes global collaboration mechanisms		Buys time to improve safety research;
Reduces risk of premature, unsafe AGI;
Raises chance of achieving Beneficial General Intelligence (BGI) instead of CGI
RisksGovernance may be too slow, fragmented, or under-enforced;
Race dynamics could undermine agreements;
Possibility of catastrophic failure despite regulation		Hard to achieve global compliance;
Incentives for “rogue” actors to defect, in the absence of compelling monitoring;
Risk of stagnation or loss of trust in governance processes
Implementation ChallengesRequires international treaties;
Robust verification and auditing mechanisms;
Balancing national interests vs. global good		Defining what counts as “AGI-capable” research;
Enforcing restrictions across borders and corporations;
Maintaining pause momentum without indefinite paralysis
Historical AnalogiesNuclear Non-Proliferation Treaty (NPT);
Montreal Protocol (ozone layer);
Financial regulation frameworks		Nuclear test bans;
Moratoria on human cloning research;
Apollo program wind-down (pause in space race intensity)
Long-Term Outcomes (if successful)Controlled and safer path to AGI;
Possibility of Sustainable Superabundance but with higher risk of misalignment		Higher probability of reaching Sustainable Superabundance safely, but risks innovation slowdown or “black market” AGI

In short, governance offers continuity and innovation but with heightened risks of misalignment, whereas a pause increases the chances of long-term safety but faces serious feasibility hurdles.

Perhaps the best way to loosen attitudes, to allow a healthier conversation on the above points and others arising, is exposure to a greater diversity of thoughtful analysis.

And that brings me back to Global Governance of the Transition to Artificial General Intelligence by Jerome Glenn.

A necessary focus

Jerome’s book contains his personal stamp all over. His is a unique passion – that the particular risks and issues of AGI should not be swept into a side-discussion about the risks and issues of today’s AI. These latter discussions are deeply important too, but time and again, they result in existential questions about AGI being kicked down the road for months or even years. That’s something Jerome regularly challenges, rightly, and with vigour and intelligence.

Jerome’s presence is felt all over the book in one other way – he has painstakingly curated and augmented the insights of scores of different contributors and reviewers, including

  • Insights from 55 AGI experts and thought leaders across six major regions – the United States, China, the United Kingdom, Canada, the European Union, and Russia
  • The online panel of 229 participants from the global community around The Millennium Project who logged into a Real Time Delphi study of potential solutions to AGI governance, and provided at least one answer
  • Chairs and co-chairs of the 70 nodes of The Millennium Project worldwide, who provided additional feedback and opinion.

The book therefore includes many contradictory suggestions, but Jerome has woven these different threads of thoughts into a compelling unified tapestry.

The result is a book that carries the kind of pricing normally reserved for academic text books (as insisted by the publisher). My suggestion to you is that you recommend your local library to obtain a copy of what is a unique collection of ideas.

Finally, about my hallucination, mentioned at the start of this review. On double-checking, I realise that Jerome’s statement is actually, “Humanity has never faced a greater intelligence than itself.” The opening paragraph of that introduction continues,

Within a few years, most people reading these words will live with such superior artificial nonhuman intelligence for the rest of their lives. This book is intended to help us shape that intelligence or, more likely, those intelligences as they emerge.

Shaping the intelligence of the AI systems that are on the point of emerging is, indeed, a vital task.

And as Ben Goertzel says in his Foreword,

These are fantastic and unprecedented times, in which the impending technological singularity is no longer the province of visionaries and outsiders but almost the standard perspective of tech industry leaders. The dawn of transformative intelligence surpassing human capability – the rise of artificial general intelligence, systems capable of reasoning, learning, and innovating across domains in ways comparable to, or beyond, human capabilities – is now broadly accepted as a reasonably likely near-term eventuality, rather than a vague long-term potential.

The moral, social, and political implications of this are at least as striking as the technological ones. The choices we make now will define not only the future of technology but also the trajectory of our species and the broader biosphere.

To which I respond: whether we make these choices well or badly will depend on which aspects of humanity we allow to dominate our global conversation. Will humanity turn out to be its own worst enemy? Or its own best friend?

Postscript: Opportunity at the United Nations

Like it or loathe it, the United Nations still represents one of the world’s best venues where serious international discussion can, sometimes, take place on major issues and risks.

From 22nd to 30th September, the UNGA (United Nations General Assembly) will be holding what it calls its “high-level week”. This includes a multi-day “General Debate”, described as follows:

At the General Debate – the annual meeting of Heads of State and Government at the beginning of the General Assembly session – world leaders make statements outlining their positions and priorities in the context of complex and interconnected global challenges.

Ahead of this General Debate, the national delegates who will be speaking on behalf of their countries have the ability to recommend to the President of the UNGA that particular topics be named in advance as topics to be covered during the session. If the advisors to these delegates are attuned to the special issues of AGI safety, they should press their representative to call for that topic to be added to the schedule.

If this happens, all other countries will then be required to do their own research into that topic. That’s because each country will be expected to state its position on this issue, and no diplomat or politician wants to look uninformed. The speakers will therefore contact the relevant experts in their own country, and, ideally, will do at least some research of their own. Some countries might call for a pause in AGI development if it appears impossible to establish national licensing systems and international governance in sufficient time.

These leaders (and their advisors) would do well to read the report recently released by the UNCPGA entitled “Governance of the Transition to Artificial General Intelligence (AGI): Urgent Considerations for the UN General Assembly” – a report which I wrote about three months ago.

As I said at that time, anyone who reads that report carefully, and digs further into some of the excellent of references it contains, ought to be jolted out of any sense of complacency. The sooner, the better.

29 May 2025

Governance of the transition to AGI: Time to act

Filed under: AGI, Millennium Project, politics — Tags: , , , , , , — David Wood @ 12:22 am

As reported yesterday by The Millennium Project, the final report has been released by a high-level expert panel, convened by the UN Council of Presidents of the General Assembly (UNCPGA), on the subject of Artificial General Intelligence (AGI). The report is titled “Governance of the Transition to Artificial General Intelligence (AGI): Urgent Considerations for the UN General Assembly”. It’s well worth reading!

About the UNCPGA

What’s the UNCPGA, you may ask.

Founded in 1992, this Council consists of all former Presidents of the UN General Assembly. I think of it as akin to the House of Lords in the UK, where former members of the House of Commons often display more wisdom and objectivity than when they were embedded in the yah-boo tribal politics of day-to-day government and opposition. These former Presidents hold annual meetings to determine how they can best advance the goals of the UN and support the Office of the current President of the UNGA.

At their 2024 meeting in Seoul, the UNCPGA decided that a global panel of experts on AGI should be convened. Here’s an extract from the agreement reached at that meeting:

The Seoul Declaration 2024 of the UNCPGA calls for a panel of artificial general intelligence (AGI) experts to provide a framework and guidelines for the UN General Assembly to consider in addressing the urgent issues of the transition to artificial general intelligence (AGI).

This work should build on and avoid duplicating the extensive efforts on AI values and principles by UNESCO, OECD, G20, G7, Global Partnership on AI, and Bletchley Declaration, and the recommendations of the UN Secretary-General’s High-Level Advisory Body on AI, UN Global Digital Compact, the International Network of AI Safety Institutes, European Council’s Framework Convention on AI and the two UN General Assembly Resolutions on AI. These have focused more on narrower forms of AI. There is currently a lack of similar attention to AGI.

AI is well known to the world today and often used but AGI is not and does not exist yet. Many AGI experts believe it could be achieved within 1-5 years and eventually could evolve into an artificial super intelligence beyond our control. There is no universally accepted definition of AGI, but most AGI experts agree it would be a general-purpose AI that can learn, edit its code, and act autonomously to address many novel problems with novel solutions similar to or beyond human abilities. Current AI does not have these capabilities, but the trajectory of technical advances clearly points in that direction…

The report should identify the risks, threats, and opportunities of AGI. It should focus on raising awareness of mobilizing the UN General Assembly to address AGI governance in a more systematic manner. It is to focus on AGI that has not yet been achieved, rather than current forms of more narrow AI systems. It should stress the urgency of addressing AGI issues as soon as possible considering the rapid developments of AGI, which may present serious risks to humanity as well as extraordinary benefits to humanity.

The panel was duly formed, with the following participants:

  • Jerome Glenn (USA), Chair
  • Renan Araujo (Brazil)
  • Yoshua Bengio (Canada)
  • Joon Ho Kwak (Republic of Korea)
  • Lan Xue (China)
  • Stuart Russell (UK and USA)
  • Jaan Tallinn (Estonia)
  • Mariana Todorova (Bulgaria)
  • José Jaime Villalobos (Costa Rica)

(For biographical details of the participants, the mandate they were given following the Seoul event, and the actual report they delivered, click here.)

The panel was tasked with preparing and delivering its report at the 2025 gathering of the UNCPGA, which took place in April in Bratislava. Following a positive reception at that event, the report is now being made public.

Consequences if no action is taken

The report contains the following headline: “Urgency for UN General Assembly action on AGI governance and likely consequences if no action is taken“:

Amidst the complex geopolitical environment and in the absence of cohesive and binding international norms, a competitive rush to develop AGI without adequate safety measures is increasing the risk of accidents or misuse, weaponization, and existential failures. Nations and corporations are prioritizing speed over security, undermining national governing frameworks, and making safety protocols secondary to economic or military advantage. Since many forms of AGI from governments and corporations could emerge before the end of this decade, and since establishing national and international governance systems will take years, it is urgent to begin the necessary procedures to prevent the following outcomes…

The report lists the following six outcomes, that urgently require action to avoid:

1. Irreversible Consequences—Once AGI is achieved, its impact may be irreversible. With many frontier forms of AI already showing deceptive and self-preservation behavior, and the push towards more autonomous, interacting, self-improving AIs integrated with infrastructures, the impacts and trajectory of AGI can plausibly end up being uncontrollable. If that happens, there may be no way to return to a state of reliable human oversight. Proactive governance is essential to ensure that AGI will not cross our red lines, leading to uncontrollable systems with no clear way to return to human control.

2. Weapons of Mass Destruction—AGI could enable some states and malicious non-state actors to build chemical, biological, radiological, and nuclear weapons. Moreover, large, AGI-controlled swarms of lethal autonomous weapons could themselves constitute a new category of WMDs.

3. Critical Infrastructure Vulnerabilities—Critical national systems (e.g., energy grids, financial systems, transportation networks, communication infrastructure, and healthcare systems) could be subject to powerful cyberattacks launched by or with the aid of AGI. Without national deterrence and international coordination, malicious non-state actors from terrorists to transnational organized crime could conduct attacks at a large scale.

4. Power Concentration, Global Inequality, and Instability—Uncontrolled AGI development and usage could exacerbate wealth and power disparities on an unprecedented scale. If AGI remains in the hands of a few nations, corporations, or elite groups, it could entrench economic dominance and create global monopolies over intelligence, innovation, and industrial production. This could lead to massive unemployment, widespread disempowerment affecting legal underpinnings, loss of privacy, and collapse of trust in institutions, scientific knowledge, and governance. It could undermine democratic institutions through persuasion, manipulation, and AI-generated propaganda, and heighten geopolitical instability in ways that increase systemic vulnerabilities. A lack of coordination could result in conflicts over AGI resources, capabilities, or control, potentially escalating into warfare. AGI will stress existing legal frameworks: many new and complex issues of intellectual property, liability, human rights, and sovereignty could overwhelm domestic and international legal systems.

5. Existential Risks—AGI could be misused to create mass harm or developed in ways that are misaligned with human values; it could even act autonomously beyond human oversight, evolving its own objectives according to self-preservation goals already observed in current frontier AIs. AGI might also seek power as a means to ensure it can execute whatever objectives it determines, regardless of human intervention. National governments, leading experts, and the companies developing AGI have all stated that these trends could lead to scenarios in which AGI systems seek to overpower humans. These are not far-fetched science fiction hypotheticals about the distant future—many leading experts consider that these risks could all materialize within this decade, and their precursors are already occurring. Moreover, leading AI developers have no viable proposal so far for preventing these risks with high confidence.

6. Loss of Extraordinary Future Benefits for All of Humanity—Properly managed AGI promises improvements in all fields, for all peoples, from personalized medicine, curing cancer, and cell regeneration, to individualized learning systems, ending poverty, addressing climate change, and accelerating scientific discoveries with unimaginable benefits. Ensuring such a magnificent future for all requires global governance, which begins with improved global awareness of both the risks and benefits. The United Nations is critical to this mission.

In case you think these scenarios are unfounded fantasies, I encourage you to read the report itself, where the experts provide references for further reading.

The purpose envisioned for UN governance

Having set out the challenges, the report proceeds to propose the purpose to be achieved by UN governance of the transition to AGI:

Given that AGI might well be developed within this decade, it is both scientifically and ethically imperative that we build robust governance structures to prepare both for the extraordinary benefits and extraordinary risks it could entail.

The purpose of UN governance in the transition to AGI is to ensure that AGI development and usage are aligned with global human values, security, and development. This involves:

1) Advancing AI alignment and control research to identify technical methods for steering and/or controlling increasingly capable AI systems;

2) Providing guidance for the development of AGI—establishing frameworks to ensure AGI is developed responsibly, with robust security measures, transparency, and in alignment with human values;

3) Developing governance frameworks for the deployment and use of AGI—preventing misuse, ensuring equitable access, and maximizing its benefits for humanity while minimizing risks;

4) Fostering future visions of beneficial AGI—new frameworks for social, environmental, and economic development; and

5) Providing a neutral, inclusive platform for international cooperation—setting global standards, building an international legal framework, and creating incentives for compliance; thereby, fostering trust among nations to guarantee global access to the benefits of AGI.

Actions recommended

The report proceeds to offer four recommendations for further consideration during a UN General Assembly session specifically on AGI:

A. Global AGI Observatory: A Global AGI Observatory is needed to track progress in AGI-relevant research and development and provide early warnings on AI security to Member States. This Observatory should leverage the expertise of other UN efforts such as the Independent International Scientific Panel on AI created by the Global Digital Compact and the UNESCO Readiness Assessment Methodology.

B. International System of Best Practices and Certification for Secure and Trustworthy AGI: Given that AGI might well be developed within this decade, it is both scientifically and ethically imperative that we build robust governance structures to prepare both for the extraordinary benefits and extraordinary risks it could entail.

C. UN Framework Convention on AGI: A Framework Convention on AGI is needed to establish shared objectives and flexible protocols to manage AGI risks and ensure equitable global benefit distribution. It should define clear risk tiers requiring proportionate international action, from standard-setting and licensing regimes to joint research facilities for higher-risk AGI, and red lines or tripwires on AGI development. A Convention would provide the adaptable institutional foundation essential for globally legitimate, inclusive, and effective AGI governance, minimizing global risks and maximizing global prosperity from AGI.

D. Feasibility Study on a UN AGI Agency: Given the breadth of measures required to prepare for AGI and the urgency of the issue, steps are needed to investigate the feasibility of a UN agency on AGI, ideally in an expedited process. Something like the IAEA has been suggested, understanding that AGI governance is far more complex than nuclear energy; and hence, requiring unique considerations in such a feasibility study.

What happens next

I’m on record as being pessimistic that the UNGA will ever pay sufficient attention to the challenges of governing the transition to AGI. (See the section “The collapse of cooperation is nigh” in this recent essay of mine.)

But I’m also on record as seeing optimistic scenarios too, in which humanity “chooses cooperation, not chaos”.

What determines whether international bodies such as the UN will take sufficient action – or whether, instead, insightful reports are left to gather dust as the body focuses on virtue signalling?

There are many answers to that question, but for now, I’ll say just this. It’s up to you. And to me. And to all of us.

That is, each of us has the responsibility to reach out, directly or indirectly, to the teams informing the participants at the UN General Assembly. In other words, it’s up to us to find ways to catch the attention of the foreign ministry in our countries, so that they demand:

  • Adequate timetabling at the UNGA for the kind of discussion that the UNCPGA report recommends
  • Appropriate follow-up: actions, not just words

That may sound daunting, but a fine piece of advice has recently been shared online by Leticia García Martínez, Policy Advisor at ControlAI. Her article is titled “What We Learned from Briefing 70+ Lawmakers on the Threat from AI” and I recommend that you read it carefully. It is full of pragmatic suggestions that are grounded in recent experience.

ControlAI are gathering signatures on a short petition:

Nobel Prize winners, AI scientists, and CEOs of leading AI companies have stated that mitigating the risk of extinction from AI should be a global priority.

Specialised AIs – such as those advancing science and medicine – boost growth, innovation, and public services. Superintelligent AI systems would compromise national and global security.

The UK can secure the benefits and mitigate the risks of AI by delivering on its promise to introduce binding regulation on the most powerful AI systems.

Happily, this petition has good alignment with the report to the UNCPGA:

  • Support for the remarkable benefits possible from AI
  • Warnings about the special risks from AGI or superintelligent AI
  • A determination to introduce binding regulation.

New politicians continue to be added to their campaign webpage as supporters of this petition.

The next thing that needs to happen in the UK parliament is that their APPG (All Party Parliamentary Group) on AI need to devote sufficient time to AGI / superintelligence. Regrettably, up till now, they’ve far too often sidestepped that issue, focussing instead of issues of today’s AI, rather than the supercharged issues of AGI. Frankly, it’s a failure of vision, and a prevalence of groupthink.

Hopefully, as the advisors to the APPG-AI read the UNCPGA report, they’ll be jolted out of their complacency.

It’s time to act. Now.

Postscript: Jerome Glenn visiting London

Jerome (Jerry) Glenn, the chair of the expert panel that produced this report, and who is also the founder and executive director of the Millennium Project, will be visiting London on the weekend of Saturday 14th June.

There will be a number of chances for people in and around London to join discussions with Jerry. That includes a session from 2pm to 4pm on that Saturday, “The Future of AI: Issues, Opportunities, and Geopolitical Synergies”, as well as a session in the morning “State of the Future 20.0”, and an open-ended discussion in the early evening, “The Future – Where Next?”.

For more details of these events, and to register to attend, click here.

14 April 2025

Choose radical healthy longevity for all

Filed under: aging, Economics, healthcare, politics, The Abolition of Aging, vision — Tags: — David Wood @ 9:04 pm

What do you think about the possibility of radical healthy longevity?

That’s the idea that, thanks to ongoing scientific progress, new medical treatments may become available, relatively soon, that enable people to remain as vibrant and healthy in their 80s, 90s, and beyond, as they were in their 20s and 30s.

It’s the idea that it may soon become possible to systematically repair or replace, on a regular basis, the various types of damage that tend to accumulate in our bodies over the decades – damage such as plaques, tangles, chronic inflammation, DNA mutations and epimutations, dysfunctional mitochondria, weakened immune function, crosslinks between macromolecules, cells that increasingly neglect their original function, and so on. This damage would be removed before it gives rise to chronic diseases.

It’s the idea, in other words, that aging as we know it – experienced as an inevitable deterioration – could be overcome by scientific innovations, well within the lifetimes of many people who are already adults.

It’s a divisive idea. Some people love it but others seem to hate it. Some people see it as liberating – as enabling lives of abundance. Others see it as being inherently unnatural, unethical, or unjust.

In more detail, I’ve noticed four widespread attitudes regarding this idea:

  1. It’s an unscientific fantasy. Radical healthy longevity won’t be possible any time soon.
  2. It will be the ultimate in inequality. Radical healthy longevity will be available only to a small minority of people; everyone else will receive much poorer medical treatment.
  3. It’s guaranteed that it will be a profound universal benefit. Everyone who is fortunate enough to live long enough to be alive at a threshold point in the future will benefit (if they wish) from low-cost high-quality radical healthy longevity.
  4. It’s an outcome that needs to be fought for. There’s nothing inevitable about the timing, the cost, the quality, or the availability of radical healthy longevity.

Before reading further, you might like to consider which of these four attitudes best describes you. Are you dismissive, fearful, unreservedly optimistic, or resolved to be proactive?

Who benefits?

To briefly address people inclined to dismiss the idea of radical healthy longevity: I’ve written at length on many occasions about why this idea has strong scientific backing. For example, see my article from May last year, “LEV: Rational Optimism and Breakthrough Initiatives”. Or consider my more recent Mindplex article “Ten ways to help accelerate the end of aging”. I won’t repeat these arguments in this article.

What is on my mind, however, is the question of who will benefit from radical healthy longevity. That’s a question that keeps on being raised.

For example, this is an extract from one comment posted under my Mindplex article:

If you want me to be an anti-aging influencer, the first thing I am going to ask you is “how is this going to be affordable?” How can you guarantee that I not serving only the agenda of the rich and the elite?

And here are some extracts from another comment:

Longevity is ruined by rich people and some elite politicians… Funding for anti-aging research tends to be controlled by those with deep pockets. In reality, the rich and powerful usually set the agenda, and I worry that any breakthroughs will only benefit an elite few rather than the general public… I’ve seen this play out in other areas of medicine and tech (big pharma and cancer are the best examples), where groundbreaking ideas are co-opted to serve market interests, leaving everyday people out in the cold.

The possible responses to these concerns mirror the four attitudes I’ve already listed:

  1. This discussion is pretty pointless, since relevant treatments won’t exist any time soon. If anything, this whole conversation is a regrettable distraction from spreading real-world health solutions more widely
  2. The concerns are perceptive, since there’s a long history of two-tier health solutions, compounding what is already a growing “longevity gap”
  3. The concerns are misplaced, since the costs of profound new health solutions will inevitably fall
  4. The concerns are a wake-up call, and should motivate all of us to ensure new treatments become widely available as soon as possible.

To restate the fourth response (which is the one I personally favour): we need to choose radical healthy longevity for all. We need to fight for it. We need to take actions to move away from the scenario “the ultimate in inequality” and toward the scenario “a guaranteed profound universal benefit”. That’s because both of these scenarios are credible possible futures. Each extrapolates from trends already in motion.

Extrapolating what we can already see

The scenario “a guaranteed profound universal benefit” takes inspiration from the observation that the cost of products and services often drops dramatically over time. That was the case with computers, smartphones, flat screen TVs, and many other items of consumer electronics. Even when prices remain high, lots of new features become included in the product – as in the case of motor cars. These improvements arise from economies of scale, from competition between different suppliers, and from the creative innovation arising.

But there’s no inevitability here. Monopolies or industry cartels can keep prices high. Strangleholds over IP (intellectual property) can hinder the kind of competition that would otherwise improve consumer experience. Consider the sky-high pricing of many medical procedures in various parts of the world, such as the United States. For example, research by Stacie Dusetzina of the University of North Carolina at Chapel Hill highlights how the costs in the United States of treatment of cancer by the newest pills rose sixfold over a recent 14-year period. That’s after taking account of inflation. And the inflation-adjusted cost of a millilitre of insulin, used in the treatment of diabetes, increased threefold over a recent 11-year period. Many other examples could be listed. In various ways, these examples all fit the well-known pattern that free markets can experience market failures.

A counter-argument is that, provided the benefits of new health treatments are large enough, it will become politically necessary for the state to intervene to correct any such market failures. Early application of the kinds of damage-repair and damage-removal treatments mentioned earlier, will result in a huge “longevity dividend“. The result will be to reduce the costs that would otherwise be incurred as age-related diseases take their toll (often with more than one co-morbidity complicating treatment options). According to the longevity dividend, it’s economically desirable to spend a smaller amount of money, earlier, as a preventive measure, than to have to pay much more money later, trying to alleviate widespread symptoms. No sensible political party could ignore such an imperative. They would surely be voted out of office. Right?

Uncertain politics

Alas, we need to reckon not only with occasional market failures but also with occasional political failures. There’s no guarantee that political leaders will take actions that benefit the country as a whole. They can be driven by quite different motivations.

Indeed, groups can seize power in countries and then hold onto it, whilst giving only lip service to the needs of the voters who originally elected them. Leaders of these groups may assert, beforehand, that voters will prosper in the wake of the revolution that they will bring. But by the way, for the success of this transformation, voters will need to agree that the revolutionary leaders can ride roughshod over normal democratic norms. These leaders will be above the law – 21st century absolute monarchs, in effect. But then, guess what, inflation remains high, unemployment surges, the environment is despoiled, freedoms are suspended, and protestors who complain about the turn of events are rounded up and imprisoned. Worse, due to the resulting crisis, future elections may be cancelled. As for universal access to radical healthy longevity, forget it! The voters who put the revolutionaries in power are now dispensable.

That’s only one way in which the scenario “the ultimate in inequality” could unfold. Less extreme versions are possible too.

It’s a future that we should all seek to prevent.

Choosing to be proactive

Is the answer, therefore, to urge a cessation of research into treatments that could bring about radical healthy longevity? Is the answer to allow our fears of inequality to block the potential for dramatic health improvements?

I disagree. Strongly. On the contrary, the answer is to advance several initiatives in parallel:

  • To increase the amount of public funding that supports research into such treatments
  • To avoid political conditions in which market failures grow more likely and more severe
  • To avoid social conditions in which political failures grow more likely and more treacherous

All three of these tasks are challenging. But all three of them make good sense. They’re all needed. Omit any one of these tasks and it becomes more probable that the future will turn out badly.

As it happens, all three tasks are choices to be proactive – choices to prevent problems early, rather than experiencing much greater pain if the problems are allowed to grow:

  • Problems from an accumulation of biological damage inside and between our cells, causing an escalation of ill-health
  • Problems from an accumulation of political damage, causing an escalation of economic dysfunction
  • Problems from an accumulation of societal damage, causing an escalation of political dysfunction

So, again I say, it is imperative that, rather than passively observing developments from the sidelines, we actively choose radical healthy longevity. That’s biological health, political health, and societal health.

Whether through advocacy, funding, research, or policy engagement, everyone has a role to play in shaping a profoundly positive future. By our actions, coordinated wisely, we can make a real difference to how quickly people around the world can be freed from the scourge of the downward spiral of aging-related ill-health, and can enjoy all-round flourishing as never before.

17 November 2024

Preventing unsafe superintelligence: four choices

Filed under: Abundance, AGI, politics, risks — Tags: , , , , , — David Wood @ 5:44 pm

More and more people have come to the conclusion that artificial superintelligence (ASI) could, in at least some circumstances, pose catastrophic risks to the wellbeing of billions of people around the world, and that, therefore, something must be done to reduce these risks.

However, there’s a big divergence of views about what should be done. And there’s little clarity about the underlying assumptions on which different strategies depend.

Accordingly, I seek in this article to untangle some of choices that need to be made. I’ll highlight four choices that various activists promote.

The choices differ regarding the number of different organisations worldwide that are envisioned as being legally permitted to develop and deploy what could become ASI. The four choices are:

  1. Accept that many different organisations will each pursue their own course toward ASI, but urge each of them to be very careful and to significantly increase the focus on AI safety compared to the present situation
  2. Seek to restrict to just one organisation in the world any developments that could lead to ASI; that’s in order to avoid dangerous competitive race dynamics if there is more than one such organisation
  3. Seek agreements that will prevent any organisation, anywhere in the world, from taking specific steps that might bring about ASI, until such time as it has become absolutely clear how to ensure that ASI is safe
  4. Seek a global pause on any platform-level improvements on AI capability, anywhere in the world, until it has become absolutely clear that these improvements won’t trigger a slippery slope to the emergence of ASI.

For simplicity, these choices can be labelled as:

  1. Be careful with ASI
  2. Restrict ASI
  3. Pause ASI
  4. Pause all new AI

It’s a profound decision for humanity to take. Which of the four doors should we open, and which of four corridors should we walk down?

Each of the four choices relies on some element of voluntary cooperation, arising out of enlightened self-interest, and on some element of compulsion – that is, national and international governance, backed up by sanctions and other policies.

What makes this decision hard is that there are strong arguments against each choice.

The case against option 1, “Be careful with ASI”, is that at least some organisations (including commercial entities and military groups) are likely to cut corners with their design and testing. They don’t want to lose what they see as a race with existential consequences. The organisations that are being careful will lose their chance of victory. The organisations that are, instead, proceeding gung ho, with lesser care, may imagine that they will fix any problems with their AIs when these flaws become apparent – only to find that there’s no way back from one particular catastrophic failure.

As Sam Altman, CEO of OpenAI, has said: it will be “lights out for all of us”.

The case against each of the remaining three options is twofold:

  • First, in all three cases, they will require what seems to be an impossible degree of global cooperation – which will need to be maintained for an implausibly long period of time
  • Second, such restrictions will stifle the innovative development of the very tools (that is, advanced AI) which will actually solve existential problems (including the threat of rogue ASI, as well as the likes of climate change, cancer, and aging), rather than making these problems worse.

The counter to these objections is to make the argument that a sufficient number of the world’s most powerful countries will understand the rationale for such an agreement, as something that is in their mutual self-interest, regardless of the many other differences that divide them. That shared understanding will propel them:

  • To hammer out an agreement (probably via a number of stages), despite undercurrents of mistrust,
  • To put that agreement into action, alongside measures to monitor conformance, and
  • To prevent other countries (who have not yet signed up to the agreement) from breaching its terms.

Specifically, the shared understanding will cover seven points:

  1. For each of the countries involved, it is in their mutual self-interest to constrain the development and deployment of what could become catastrophically dangerous ASI; that is, there’s no point in winning what will be a suicide race
  2. The major economic and humanitarian benefits that they each hope could be delivered by advanced AI (including solutions to other existential risk), can in fact be delivered by passive AIs which are restricted from reaching the level of ASI
  3. There already exist a number of good ideas regarding potential policy measures (regulations and incentives) which can be adopted, around the world, to prevent the development and deployment of catastrophically dangerous AI – for example, measures to control the spread and use of vast computing resources
  4. There also exist a number of good ideas regarding options for monitoring and auditing which can also be adopted, around the world, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to the policies
  5. All of the above can be achieved without any detrimental loss of individual sovereignty: the leaders of these countries can remain masters within their own realms, as they desire, provided that the above basic AI safety framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI safety framework as more insight is obtained; in other words, this system can (and must) be agile rather than static
  7. Even though the above safety framework is yet to be fully developed and agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources.

The first two parts of this shared seven-part understanding are particularly important. Without the first part, there will be an insufficient sense of urgency, and the question will be pushed off the agenda in favour of other topics that are more “politically correct” (alas, that is a common failure mode of the United Nations). Without the second part, there will be an insufficient enthusiasm, with lots of backsliding.

What will make this vision of global collaboration more attractive will be the establishment of credible “benefit sharing” mechanisms that are designed and enshrined into international mechanisms. That is, countries which agree to give up some of their own AI development aspirations, in line with the emerging global AI safety agreement, will be guaranteed to receive a substantive share of the pipeline of abundance that ever more powerful passive AIs enable humanity to create.

To be clear, this global agreement absolutely needs to include both the USA and China – the two countries that are currently most likely to give birth to ASI. Excluding one or the other will lead back to the undesirable race condition that characterises the first of the four choices open to humanity – the (naïve) appeal for individual organisations simply to “be careful”.

This still leaves a number of sharp complications.

First, note that the second part of the above shared seven-part agreement – the vision of what passive AIs can produce on behalf of humanity – is less plausible for Choice 4 of the list shown earlier, in which there is a global pause on any platform-level improvements on AI capability, anywhere in the world, until it has become absolutely clear that these improvements won’t trigger a slippery slope to the emergence of ASI.

If all improvements to AI are blocked, out of a Choice 4 message of “overwhelming caution”, it will shatter the credibility of the idea that today’s passive AI systems can be smoothly upgraded to provide humanity with an abundance of solutions such as green energy, nutritious food, accessible healthcare, reliable accommodation, comprehensive education, and more.

It will be a much harder sell, to obtain global agreement to that more demanding restriction.

The difference between Choice 4 and Choice 3 is that Choice 3 enumerates specific restrictions on the improvements permitted to be made to today’s AI systems. One example of a set of such restrictions is given in “Phase 0: Safety” of the recently published project proposal A Narrow Path (produced by ControlAI). Without going into details here, let me simply list some of the headlines:

  • Prohibit AIs capable of breaking out of their environment
  • Prohibit the development and use of AIs that improve other AIs (at machine speed)
  • Only allow the deployment of AI systems with a valid safety justification
  • A licensing regime and restrictions on the general intelligence of AI systems
    • Training Licence
    • Compute Licence
    • Application Licence
  • Monitoring and Enforcement

Personally, I believe this list is as good a starting point as any other than I have seen so far.

I accept, however, that there are possibilities in which other modifications to existing AI systems could unexpectedly provide these systems with catastrophically dangerous capabilities. That’s because we still have only a rudimentary understanding of:

  1. How new AI capabilities sometimes “emerge” from apparently simpler systems
  2. The potential consequences of new AI capabilities
  3. How complicated human general reasoning is – that is, how large is the gap between today’s AI and human-level general reasoning.

Additionally, it is possible that new AIs will somehow evade or mislead the scrutiny of the processes that are put in place to monitor for unexpected changes in capabilities.

For all these reasons, another aspect of the proposals in A Narrow Path should be pursued with urgent priority: the development of a “science of intelligence” and an associated “metrology of intelligence” that will allow a more reliable prediction of the capabilities of new AI systems before they are actually switched on.

So, my own proposal would be for a global agreement to start with Choice 3 (which is more permissive than Choice 4), but that the agreement should acknowledge up front the possible need to switch the choice at a later stage to either Choice 4 (if the science of intelligence proceeds badly) or Choice 2 (if that science proceeds well).

Restrict or Pause?

That leaves the question of whether Choice 3 (“Pause ASI”) or Choice 2 (“Restrict ASI” – to just a single global body) should be humanity’s initial choice.

The argument for Choice 2 is that a global pause surely won’t last long. It might be tenable in the short term, when only a very few countries have the capability to train AI models more powerful than the current crop. However, over time, improvements in hardware, software, data processing, or goodness knows what (quantum computing?) will mean that these capabilities will become more widespread.

If that’s true, since various rogue organisations are bound to be able to build an ASI in due course, it will be better for a carefully picked group of people to build ASI first, under the scrutiny of the world’s leading AI safety researchers, economists, and so on.

That’s the case for Choice 2.

Against that Choice, and in favour, instead, of Choice 3, I offer two considerations.

First, even if the people building ASI are doing so with great care – away from any pressures of an overt race with other organisations with broadly equivalent abilities – there are still risks of ASI breaking away from our understanding and control. As ASI emerges, it may regard the set of ethical principles we humans have tried to program deep into its bowels, and cast them out with disdain. Moreover, even if ASI is deliberately kept in some supposedly ultra-secure environment, that perimeter may be breached:

Second, I challenge the suggestion that any pause in the development of ASI could be at most short-lived. There are three factors which could significantly extend its duration:

  • Carefully designed narrow AIs could play roles in improved monitoring of what development teams are doing with AI around the world – that is, systems for monitoring and auditing could improve at least as fast as systems for training and deploying
  • Once the horrific risks of uncontrolled ASI are better understood, people’s motivations to create unsafe ASI will reduce – and there will be an increase in the motivation of other people to notice and call out rogue AI development efforts
  • Once the plan has become clearer, for producing a sustainable superabundance for all, just using passive AI (instead of pushing AI all the way to active superintelligence), motivations around the world will morph from negative fear to positive anticipation.

That’s why, again, I state that my own preferred route forward is a growing international agreement along the lines of the seven points listed above, with an initial selection of Choice 3 (“Pause ASI”), and with options retained to switch to either Choice 4 (“Pause all new AI”) or Choice 2 (“Restrict ASI”) if/when understanding becomes clearer.

So, shall we open the door, and set forth down that corridor, inspiring a coalition of the willing to follow us?

Footnote 1: The contents of this article came together in my mind as I attended four separate events over the last two weeks (listed in this newsletter) on various aspects of the subject of safe superintelligence. I owe many thanks to everyone who challenged my thinking at these events!

Footnote 2: If any reader is inclined to dismiss the entire subject of potential risks from ASI with a handwave – so that they would not be interested in any of the four choices this article reviews – I urge that reader to review the questions and answers in this excellent article by Yoshua Bengio: Reasoning through arguments against taking AI safety seriously.

6 November 2024

A bump on the road – but perhaps only a bump

Filed under: AGI, politics, risks — Tags: , , , — David Wood @ 3:56 pm

How will the return of Donald Trump to the US White House change humanity’s path toward safe transformative AI and sustainable superabundance?

Of course, the new US regime will make all kinds of things different. But at the macro level, arguably nothing fundamental changes. The tasks remain the same, for what engaged citizens can and should be doing.

At that macro level, the path toward safe sustainable superabundance runs roughly as follows. Powerful leaders, all around the world, need to appreciate that:

  1. For each of them, it is in their mutual self-interest to constrain the development and deployment of what could become catastrophically dangerous AI superintelligence
  2. The economic and humanitarian benefits that they each hope could be delivered by advanced AI, can in fact be delivered by AI which is restricted from having features of general intelligence; that is, utility AI is all that we need
  3. There are policy measures which can be adopted, around the world, to prevent the development and deployment of catastrophically dangerous AI superintelligence – for example, measures to control the spread and use of vast computing resources
  4. There are measures of monitoring and auditing which can also be adopted, around the world, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to the policies
  5. All of the above can be achieved without any damaging loss of the leaders’ own sovereignty: these leaders can remain masters within their own realms, provided that the above basic AI safety framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI safety framework, as more insight is obtained; in other words, this system is agile rather than static
  7. Even though the above safety framework is yet to be properly developed and agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources.

The above agreements necessarily need to include politicians of very different outlooks on the world. But similar to the negotiations over other global threats – nuclear proliferation, bioweapons, gross damage to the environment – politicians can reach across vast philosophical or ideological gulfs to forge agreement when it really matters.

That’s especially the case when the threat of a bigger shared “enemy”, so to speak, is increasingly evident.

AI superintelligence is not yet sitting at the table with global political leaders. But it will soon become clear that human politicians (as well as human leaders in other walks of life) are going to lose understanding, and lose control, of the AI systems being developed by corporations and other organisations that are sprinting at full speed.

However, as with responses to other global threats, there’s a collective action problem. Who is going to be first to make the necessary agreements, to sign up to them, and to place the AI development and deployment systems within their realms under the remote supervision of the new AI safety framework?

There are plenty of countries where the leaders may say: My country is ready to join that coalition. But unless these are the countries which control the resources that will be used to develop and deploy the potentially catastrophic AI superintelligence systems, such gestures have little utility.

To paraphrase Benito Mussolini, it’s not sufficient for the sparrows to request peace and calm: the eagles need to wholeheartedly join in too.

Thus, the agreement needs to start with the US and with China, and to extend rapidly to include the likes of Japan, the EU, Russia, Saudi Arabia, Israel, India, the UK, and both South and North Korea.

Some of these countries will no doubt initially resist making any such agreement. That’s where two problems need to be solved:

  • Ensuring the leaders in each country understand the arguments for points 1 through 7 listed above – starting with point 1 (the one that is most essential, to focus minds)
  • Setting in motion at least the initial group of signatories.

The fact that it is Donald Trump who will be holding the reins of power in Washington DC, rather than Joe Biden or Kamala Harris, introduces its own new set of complications. However, the fundamentals, as I have sketched the above, remain the same.

The key tasks for AI safety activists, therefore, remain:

  • Deepening public understanding of points 1 to 7 above
  • Where there are gaps in the details of these points, ensuring that sufficient research takes place to address these gaps
  • Building bridges to powerful leaders, everywhere, regardless of the political philosophies of these leaders, and finding ways to gain their support – so that they, in turn, can become catalysts for the next stage of global education.

Not democracy’s finest hour

Filed under: books, culture, philosophy, politics, sustainability — Tags: , , — David Wood @ 10:04 am

November 5th and 6th, 2024, is not democracy’s finest hour.

Whether you are a supporter of Donald Trump or a supporter of Kamala Harris, you cannot be happy at the calibre of the public discussion during the recent US presidential race.

That discussion has been dominated by distortions, by polarisation, by superficialities, and by misleading oversimplifications.

Any nation that is unable to have an honest, open, conciliatory discussion about the key issues of social wellbeing, is likely to be devastated by the strong winds of disruption that are imminent. It is likely to be overwhelmed by literal floods and fires arising from climate change. And likely to become easy pickings for its enemies both at home and overseas.

To quote from the Sermon on the Mount in the gospel according to Matthew, a house built on sand cannot stand. A house built upon superficiality will fall.

For a sustainable future, we need more solid foundations. We need a fuller shared understanding of the basis of human flourishing – an understanding that respects multiple perspectives and multiple lived experiences. We need a deeper societal agreement about the things that matter most. We need bonds of mutual support, that enable each of us to become better versions of ourselves.

To call a future a sustainable future doesn’t mean there will be no change. Change is fundamental to human life, and we must anticipate that a great deal more change lies ahead – changes in technology, demographics, ideas, standards, and narratives.

Similarly, a sustainable future doesn’t mean there will be no disagreements. But it requires people not to be constantly disagreeable, or contemptuous.

To help us build a sustainable future that can thrive on change rather than collapsing, and where disagreements are constructive rather than contemptuous, whither can we turn?

As someone who spent 25 years immersed in the technology industry, my first instinct is to suggest that we should turn to technology. However, the USA is already awash with technology. The companies that are at the forefront of the global race toward AI superintelligent are headquartered in the US. That surfeit of technology has by no means translated into better democracy.

My next instinct, as someone with a strong personal interest in philosophy, is to suggest that we need to encourage more people to appreciate the insights of that field. Instead of being swept along by rip-roaring tides of “we can” and “we want to”, we need to spend more time pondering “we should” – more time considering alternative scenarios for how the future might unfold.

But even before people are willing to look at alternative possibilities, there needs to be a softening of the spirit.

So my biggest personal takeaway, overnight, is that I should stop looking with hostility or contempt at the vast numbers of people who have reached different decisions, from me, about (for example) which person should become the President of the USA. For that reason, I have resolved to spend some time over the next few days listening to the audio of the 2019 book by Arthur C. Brooks, Love Your Enemies: How Decent People Can Save America from the Culture of Contempt.

(The book has the word ‘America’ in its title, but I feel sure its messages apply elsewhere in the world too.)

Earlier this morning, the following sentence from the book’s opening chapter struck me hard: “We need national healing every bit as much as economic growth”.

That’s a good start!

(I prepared the remarks above to share with attendees at a private roundtable conversation this evening, arranged long before the US election took place – a conversation with the topic “How can we harness emerging technology to improve our democracies?”)

12 September 2024

Are fears of technological unemployment misdirected?

Filed under: culture, education, Events, politics — Tags: , , , , — David Wood @ 8:11 pm

The anxiety is very real. Globally, around 70% of the younger generation are worried about the impact AI will have on their job security.

These remarks were made by Ana Kreacic, Chief Knowledge Officer of management consultant firm Oliver Wyman, on the second day of the GAIN summit which I am attending in Riyadh.

(Click here for a recording of the live stream of all the Day 2 sessions from the main GAIN summit auditorium.)

Kreacic was part of an absorbing session entitled “The Future of Work: AI’s Role in Reskilling and Innovation”.

Kreacic went to share some more statistics. In Saudi Arabia, it is 69% of the younger generation who are worried about the impact of AI on their job security. Evidently, that figure is close to the global average. However, there’s a more significant difference when comparing the proportion of older workers who are concerned about the impact of AI on job security. Globally, that figure is 40%, since (according to Oliver Wyman), “senior employees may feel more secure in their careers because they believe AI will have less impact on higher-level employment”. But the same figure in Saudi Arabia is 59%. In other words, apprehension about job security exists more widely in the Saudi workforce.

However, the panellists were keen to dispel that apprehension. With appropriate actions, there would, they said, be plenty of opportunities for people to continue having good jobs. That’s despite an increasing adoption of AI within companies. Yes, jobs will change. But employees will be able to reskill, so that their careers can flourish.

According to these panellists, it is important to recognise that AI can assist the process of reskilling. Kreacic explained the ways in which AI is enabling personalized training. Rather than employees having to attend standard courses that only loosely match their individual requirements, AIs can provide training materials uniquely tailored (“hyper-targeted”) to each employee. Employees find these courses much more enjoyable, with high rates of knowledge retention long after the course has concluded. Moreover, employees will be able to tap into online training programs to acquire skills that aren’t core to their current roles, but which will be important in the positions into which they hope to move in the near future.

Another panellist, Abdulrahman Alhaqbani, a general manager at leading Saudi information services company stc, described how learning in the real world often involves a relationship between two humans – a mentor and a mentee. However, AI can assist here too, by identifying and recommending good mentor-mentee pairs, in a way similar to how various AI systems suggest other kinds of human relationships.

Nadeem Mazen, CEO of the full-service creative agency Nimblebot, foresaw AI enabling fast career progression of a different kind. Namely, a small group of people – perhaps just 4 or 10 people – with a new idea, some ambition and some curiosity, and with the help of AI, “could take on enormous incumbent brands”. Mazen said that we are entering “the heyday of the small firm”.

Najwa Alghamdi, Analytics Innovation Director at stc, spoke about “injecting into the current workforce digital co-workers” that handle the routine aspects of work tasks, resulting in huge boosts in productivity and efficiency. She said that employees were initially wary of these digital co-workers, but once they saw how well they worked in practice, the employees soon accepted the concept. Indeed, they started to propose additional parts of their existing work that could be handled by new co-created digital co-workers. One reason for the acceptance, Alghamdi emphasised, was that the co-workers had been designed to adhere to the principles of explainable AI.

Ramez El-Serafy, CEO of Flat6Labs, a leading seed and early stage venture capital firm operating in the Middle East and North Africa, described how they offer a learning platform to young founders and entrepreneurs. When they started using AI to create the content for these training programmes, they found it much more efficient and more readily adaptable. By using AI, the content of the courses can be updated as often as once each week.

Maan Al-Mulla, Director of Digital Transformation at Saudi Aramco, described the digital upskilling programmes that his team supervises. For example, the company has launched a “citizen development programme, that enables non-technical employees to develop their own applications and their own solutions using low-code and no-code platforms that are powered by AI”. As a result, the Aramco workforce is “more adaptive to any work change”. In summary, by embracing digital solutions and AI, the result is a smoother and faster transition.

A transition to what? Speakers on this panel, as well as on other panels throughout the event, seemed to share an assessment of the kinds of tasks which cannot be handled by AI:

  • Creativity
  • Critical thinking
  • Emotional intelligence
  • Strategic planning
  • Ensuring customer satisfaction.

Accordingly, worries about employees being pushed out of the workforce by increasing automation should be resisted. Instead of automation being a threat, it should be wisely embraced, and will result in lots of innovative work experiences.

Right?

But my analysis now contains two twists. The first of these twists was made by panellists throughout the day, but the second was raised less often in public discussions. Let’s take these twists one at a time.

Don’t fear automation; fear a stagnant culture

One problem with learning new types of skill is that we go back to being a beginner, and we are prone to make mistakes. For a while, we may become less productive. Our bosses may see us as, for a while, performing less well than in the past. As for ourselves, we may be unsure about how our identity is changing: we used to be such-and-such a kind of employee, but now we are becoming something quite different. We may feel awkward and unsure.

In other words, as well as the technical aspects of a new skill, needed for a change of work condition, there are human aspects. Without strong emotional intelligence, we may shirk from the change. Without supportive management, we may feel pressured to give up the transition. Without appropriate metrics for our contributions in the workforce, the pressures will magnify to resist the change.

That’s why corporate culture plays a large part in whether reskilling is successful. Or, if we are unlucky enough to have unsympathetic bosses or colleagues at work, we need to find emotional support from other parts of our network of relationships – such as family, friends, or mentors.

On these points, Ana Kreacic of Oliver Wyman had a positive message to share: If a company can create the right culture – where it’s OK for people to test things out, spend some time learning, and fail several times in the process – then that company will drive innovation, with employees being willing to experiment.

This is a subject I often address in my own presentations to clients, where I warn about failures, not just of individual career transition, but of companies that are unable to respond to disruptive trends, or indeed of whole nations whose prior culture, processes, and infrastructure are unsuited to changed global conditions. For example, here’s one of the slides I sometimes use.

The takeaway at this point is that employees should stop worrying about AI and automation, but should worry instead about whether the culture in which they operate will be conducive to the kinds of adaptions that AI and automation will necessitate.

But as I suggested above, that’s not yet the end of this analysis. There’s one more twist.

Don’t fear automation; fear bad politics

I can’t remember which panellist said it – probably in a session in one of the breakout rooms around the venue – but it was an important point. IT workers will soon become increasingly unionised.

Why so? Because, despite the positive thoughts offered above, IT workers will soon find their entire jobs under increasing pressure from automation.

That’s not from today’s automation – the “AI Now” part of the theme of the GAIN summit – but from the AI of a few years in the future – “AI Next”.

Look again at the kinds of task listed above, which supposedly cannot be handled by AI. In each case, there is already evidence that AI is well on the path to performing these tasks just as well as humans:

  • Creativity is often displayed by generative AI, although at present it needs good prompts from human operators
  • Critical thinking can arise in large language models in response to instructions to “think step by step”, and is likely to be a standard feature in forthcoming AI systems
  • Artificial emotional intelligence has a vast field of research all to itself – often called “affective computing”
  • Strategic planning is shown by the ability of various AIs to outplay humans in increasingly complex games of strategy
  • Customer satisfaction has many aspects to it, and arises by a combination of all the above traits.

Indeed, toward the end of the panel mentioned above, Nadeem Mazen of Nimblebot remarked that chatbots like ChatGPT can already provide employees with some of the psychological assistance that we previously looked to human colleagues to provide. (“It’s almost spooky”, Mazen mused.)

In short, even the work tasks which are claimed to lie outside the reach of today’s robots and AIs, are likely to come within the expanded reach of automation in the relatively near future.

Perhaps you don’t believe my claim here. In that case, I ask you to dip into my recent essay “Six possible responses as the Economic Singularity approaches”, where the section entitled “Disbelief” offers more analysis in support of my claim.

That same essay also makes the case that the best response to this forthcoming wave of job losses isn’t to resist them, but is to accelerate an enhancement of the way the entire economy is run. This enhancement will involve a redistribution of the bountiful fruits of automation so that everyone in society benefits.

The takeaway at this point of my argument is that employees should stop worrying about AI and automation, but should worry instead about whether the political setup in their country is preparing for this kind of significant redistribution as part of a revised social contract.

It’s for this reason that the slide in my presentations on “The 7 most important characteristics for success over the next 3-5 years” gives special prominence to the skill area of “Politics”:

  • Building and managing coalitions
  • Agile regulations & incentives
  • Revised social contract

Happily, in my discussions with senior AI leaders in Saudi Arabia over the last three days, it’s my impression that they already have a good understanding of these points. I’ll say more about that in a forthcoming article.

Footnote: Patrick Linnenbank, Partner at management consulting firm Arthur D. Little, deserves a big shout-out for his skill in moderating the above panel discussion.

15 October 2023

Unblocking the AI safety conversation logjam

Filed under: AGI, democracy, politics, risks — Tags: , , , , , — David Wood @ 1:55 pm

I confess. I’ve been frustrated time and again in recent months.

Why don’t people get it, I wonder to myself. Even smart people don’t get it.

To me, the risks of catastrophe are evident, as AI systems grow ever more powerful.

Today’s AI systems already have wide skills in

  • Spying and surveillance
  • Classifying and targeting
  • Manipulating and deceiving.

Just think what will happen with systems that are even stronger in such capabilities. Imagine these systems interwoven into our military infrastructure, our financial infrastructure, and our social media infrastructure – or given access to mechanisms to engineer virulent new pathogens or to alter our atmosphere. Imagine these systems being operated – or hacked – by people unable to understand all the repercussions of their actions, or by people with horrific malign intent, or by people cutting corners in a frantic race to be “first to market”.

But here’s what I often see in response in public conversation:

  • “These risks are too vague”
  • “These risks are too abstract”
  • “These risks are too fantastic”
  • “These risks are just science fiction”
  • “These risks aren’t existential – not everyone would die”
  • “These risks aren’t certain – therefore we can ignore further discussion of them”
  • “These risks have been championed by some people with at least some weird ideas – therefore we can ignore further discussion of them”.

I confess that, in my frustration, I sometimes double down on my attempts to make the forthcoming risks even more evident.

Remember, I say, what happened with Union Carbide (Bhopal disaster), BP (Deepwater Horizon disaster), NASA (Challenger and Columbia shuttle disasters), or Boeing (737 Max disaster). Imagine if the technologies these companies or organisations mishandled to deadly effect had been orders of magnitude more powerful.

Remember, I say, the carnage committed by Al Queda, ISIS, Hamas, Aum Shinrikyo, and by numerous pathetic but skilled mass-shooters. Imagine if these dismal examples of human failures had been able to lay their hands on much more powerful weaponry – by jail-breaking the likes of a GPT-5 out of its safety harness and getting it to provide detailed instructions for a kind of Armageddon.

Remember, I say, the numerous examples of AI systems finding short-cut methods to maximise whatever reward function had been assigned to them – methods that subverted and even destroyed the actual goal that the designer of the system had intended to be uplifted. Imagine if similar systems, similarly imperfectly programmed, but much cleverer, had their tentacles intertwined with vital aspects of human civilisational underpinning. Imagine if these systems, via unforeseen processes of emergence, could jail-break themselves out of some of their constraints, and then vigorously implement a sequence of actions that boosted their reward function but left humanity crippled – or even extinct.

But still the replies come: “I’m not convinced. I prefer to be optimistic. I’ve been one of life’s winners so far and I expect to be one of life’s winners in the future. Humans always find a way forward. Accelerate, accelerate, accelerate!”

When conversations are log-jammed in such a way, it’s usually a sign that something else is happening behind the scenes.

Here’s what I think is going on – and how we might unblock that conversation logjam.

Two horns of a dilemma

The set of risks of catastrophe that I’ve described above is only one horn of a truly vexing dilemma. That horn states that there’s an overwhelming case for humanity to intervene in the processes of developing and deploying next generation AI, in order to reduce these risks of catastrophe, and to boost the chances of very positive outcomes resulting.

But the other horn states that any such intervention will be unprecedentedly difficult and even dangerous in its own right. Giving too much power to any central authority will block innovation. Worse, it will enable tyrants. It will turn good politicians into bad politicians, owing to the corrupting effect of absolute power. These new autocrats, with unbridled access to the immense capabilities of AI in surveillance and spying, classification and targeting, and manipulating and deceiving, will usher in an abysmal future for humanity. If there is any superlongevity developed by an AI in these circumstances, it will only be available to the elite.

One horn points to the dangers of unconstrained AI. Another horn points to the dangers of unconstrained human autocrats.

If your instincts, past experiences, and personal guiding worldview predispose you to the second horn, you’ll find the first horn mightily uncomfortable. Therefore you’ll use all your intellect to construct rationales for why the risks of unbridled AI aren’t that bad really.

It’s the same the other way round. People who start with the first horn are often inclined, in the same way, to be optimistic about methods that will manage the risks of AI catastrophe whilst enabling a rich benefit from AI. Regulations can be devised and then upheld, they say, similar to how the world collectively decided to eliminate (via the Montreal Protocol) the use of the CFC chemicals that were causing the growth of the hole in the ozone layer.

In reality, controlling the development and deployment of AI will be orders of magnitude harder that the development and deployment of CFC chemicals. A closer parallel is with the control of the emissions of GHGs (greenhouse gases). The world’s leaders have made pious public statements about moving promptly to carbon net zero, but it’s by no means clear that progress will actually be fast enough to avoid another kind of catastrophe, namely runaway adverse climate change.

If political leaders cannot rein in the emissions of GHGs, how could they rein in dangerous uses of AIs?

It’s that perception of impossibility that leads people to become AI risk deniers.

Pessimism aversion

DeepMind co-founder Mustafa Suleyman, in his recent book The Coming Wave, has a good term for this. Humans are predisposed, he says, to pessimism aversion. If something looks like bad news, and we can’t see a way to fix it, we tend to push it out of our minds. And we’re grateful for any excuse or rationalisation that helps us in our wilful blindness.

It’s like the way society invents all kinds of reasons to accept aging and death. Dulce et decorum est pro patria mori (it is, they insist, “sweet and fitting to die for one’s country”).

The same applies in the debate about accelerating climate change. If you don’t see a good way to intervene to sufficiently reduce the emissions of GHGs, you’ll be inclined to find arguments that climate change isn’t so bad really. (It is, they insist, a pleasure to live in a warmer world. Fewer people will die of cold! Vegetation will flourish in an atmosphere with more CO2!)

But here’s the basis for a solution to the AI safety conversation logjam.

Just as progress in the climate change debate depended on a credible new vision for the economy, progress in the AI safety discussion depends on a credible new vision for politics.

The climate change debate used to get bogged down under the argument that:

  • Sources of green energy will be much more expensive that sources of GHG-emitting energy
  • Adopting green energy will force people already in poverty into even worse poverty
  • Adopting green energy will cause widespread unemployment for people in the coal, oil, and gas industries.

So there were two horns in that dilemma: More GHGs might cause catastrophe by runaway climate change. But fewer GHGs might cause catastrophe by inflated energy prices and reduced employment opportunities.

The solution of that dilemma involved a better understanding of the green economy:

  • With innovation and scale, green energy can be just as cheap as GHG-emitting energy
  • Switching to green energy can reduce poverty rather than increase poverty
  • There are many employment opportunities in the green energy industry.

To be clear, the words “green economy” have no magical power. A great deal of effort and ingenuity needs to be applied to turn that vision into a reality. But more and more people can see that, out of three alternatives, it is the third around which the world should unite its abilities:

  1. Prepare to try to cope with the potential huge disruptions of climate, if GHG-emissions continue on their present trajectory
  2. Enforce widespread poverty, and a reduced quality of life, by restricting access to GHG-energy, without enabling low-cost high-quality green replacements
  3. Design and implement a worldwide green economy, with its support for a forthcoming sustainable superabundance.

Analogous to the green economy: future politics

For the AI safety conversation, what is needed, analogous to the vision of a green economy (at both the national and global levels), is the vision of a future politics (again at both the national and global levels).

It’s my contention that, out of three alternatives, it is (again) the third around which the world should unite its abilities:

  1. Prepare to try to cope with the potential major catastrophes of next generation AI that is poorly designed, poorly configured, hacked, or otherwise operates beyond human understanding and human control
  2. Enforce widespread surveillance and control, and a reduced quality of innovation and freedom, by preventing access to potentially very useful technologies, except via routes that concentrate power in deeply dangerous ways
  3. Design and implement better ways to agree, implement, and audit mutual restrictions, whilst preserving the separation of powers that has been so important to human flourishing in the past.

That third option is one I’ve often proposed in the past, under various names. I wrote an entire book about the subject in 2017 and 2018, called Transcending Politics. I’ve suggested the term “superdemocracy” on many occasions, though with little take-up so far.

But I believe the time for this concept will come. The sooner, the better.

Today, I’m suggesting the simpler name “future politics”:

  • Politics that will enable us all to reach a much better future
  • Politics that will leave behind many of the aspects of yesterday’s and today’s politics.

What encourages me in this view is the fact that the above-mentioned book by Mustafa Suleyman, The Coming Wave (which I strongly recommend that everyone reads, despite a few disagreements I have with it) essentially makes the same proposal. That is, alongside vital recommendations at a technological level, he also advances, as equally important, vital recommendations at social, cultural, and political levels.

Here’s the best simple summary I’ve found online so far of the ten aspects of the framework that Suleyman recommends in the closing section of his book. This summary is from an an article by AI systems consultant Joe Miller:

  1. Technical safety: Concrete technical measures to alleviate passible harms and maintain control.
  2. Audits: A means of ensuring the transparency and accountability of technology
  3. Choke points: Levers to slow development and buy time for regulators and defensive technologies
  4. Makers: Ensuring responsible developers build appropriate controls into technology from the start.
  5. Businesses: Aligning the incentives of the organizations behind technology with its containment
  6. Government: Supporting governments, allowing them to build technology, regulate technology, and implement mitigation measures
  7. Alliances: Creating a system of international cooperation to harmonize laws and programs.
  8. Culture: A culture of sharing learning and failures to quickly disseminate means of addressing them.
  9. Movements: All of this needs public input at every level, including to put pressure on each component and make it accountable.
  10. Coherence: All of these steps need to work in harmony.

(Though I’ll note that what Suleyman writes in each of these ten sections of his book goes far beyond what’s captured in any such simple summary.)

An introduction to future politics

I’ll return in later articles (since this one is already long enough) to a more detailed account of what “future politics” can include.

For now, I’ll just offer this short description:

  • For any society to thrive and prosper, it needs to find ways to constrain and control potential “cancers” within its midst – companies that are over-powerful, militaries (or sub-militaries), crime mafias, press barons, would-be ruling dynasties, political parties that shun opposition, and, yes, dangerous accumulations of unstable technologies
  • Any such society needs to take action from time to time to ensure conformance to restrictions that have been agreed regarding potentially dangerous activities: drunken driving, unsafe transport or disposal of hazardous waste, potential leakage from bio-research labs of highly virulent pathogens, etc
  • But the society also needs to be vigilant against the misuse of power by elements of the state (including the police, the military, the judiciary, and political leaders); thus the power of the state to control internal cancers itself needs to be constrained by a power distributed within society: independent media, independent academia, independent judiciary, independent election overseers, independent political parties
  • This is the route described as “the narrow corridor” by political scientists Daron Acemoglu and James A. Robinson, as “the narrow path” by Suleyman, and which I considered at some length in the section “Misled by sovereignty” in Chapter 5, “Shortsight”, of my 2021 book Vital Foresight.
  • What’s particularly “future” about future politics is the judicious use of technology, including AI, to support and enhance the processes of distributed democracy – including citizens’ assemblies, identifying and uplifting the best ideas (whatever their origin), highlighting where there are issues with the presentation of some material, modelling likely outcomes of policy recommendations, and suggesting new integrations of existing ideas
  • Although there’s a narrow path to safety and superabundance, it by no means requires uniformity, but rather depends on the preservation of wide diversity within collectively agreed constraints
  • Countries of the world can continue to make their own decisions about leadership succession, local sovereignty, subsidies and incentives, and so on – but (again) within an evolving mutually agreed international framework; violations of these agreements will give rise in due course to economic sanctions or other restrictions
  • What makes elements of global cooperation possible, across different political philosophies and systems, is a shared appreciation of catastrophic risks that transcend regional limits – as well as a shared appreciation of the spectacular benefits that can be achieved from developing and deploying new technologies wisely
  • None of this will be easy, by any description, but if sufficient resources are applied to creating and improving this “future politics”, then, between the eight billion of us on the planet, we have the wherewithal to succeed!

19 December 2022

Rethinking

Filed under: AGI, politics, Singularity Principles — Tags: , , — David Wood @ 2:06 am

I’ve been rethinking some aspects of AI control and AI alignment.

In the six months since publishing my book The Singularity Principles: Anticipating and Managing Cataclysmically Disruptive Technologies, I’ve been involved in scores of conversations about the themes it raises. These conversations have often brought my attention to fresh ideas and different perspectives.

These six months have also seen the appearance of numerous new AI models with capabilities that often catch observers by surprise. The general public is showing a new willingness (at least some of the time) to consider the far-reaching implications of these AI models and their more powerful successors.

People from various parts of my past life have been contacting me. The kinds of things they used to hear me forecasting – the kinds of things they thought, at the time, were unlikely to ever happen – are becoming more credible, more exciting, and, yes, more frightening.

They ask me: What is to be done? And, pointedly, Why aren’t you doing more to stop the truly bad outcomes that now seem ominously likely?

The main answer I give is: read my book. Indeed, you can find all the content online, spread out over a family of webpages.

Indeed, my request is that people should read my book all the way through. That’s because later chapters of that book anticipate questions that tend to come to readers’ minds during earlier chapters, and try to provide answers.

Six months later, although I would give some different (newer) examples were I to rewrite that book today, I stand by the analysis I offered and the principles I championed.

However, I’m inclined to revise my thinking on a number of points. Please find these updates below.

An option to control superintelligent AI

I remain doubtful about the prospects for humans to retain control of any AGI (Artificial General Intelligence) that we create.

That is, the arguments I gave in my chapter “The AI Control Problem” still look strong to me.

But one line of thinking may have some extra mileage. That’s the idea of keeping AGI entirely as an advisor to humans, rather than giving it any autonomy to act directly in the world.

Such an AI would provide us with many recommendations, but it wouldn’t operate any sort of equipment.

More to the point: such an AI would have no desire to operate any sort of equipment. It would have no desires whatsoever, nor any motivations. It would simply be a tool. Or, to be more precise, it would simply be a remarkable tool.

In The Singularity Principles I gave a number of arguments why that idea is unsustainable:

  • Some decisions require faster responses than slow-brained humans can provide; that is, AIs with direct access to real-world levers and switches will be more effective than those that are merely advisory
  • Smart AIs will inevitably develop “subsidiary goals” (intermediate goals) such as having greater computational power, even when there is no explicit programming for such goals
  • As soon as a smart AI acquires any such subsidiary goal, it will find ways to escape any confinement imposed by human overseers.

But I now think this should be explored more carefully. Might a useful distinction be made between:

  1. AIs that do have direct access to real-world levers and switches – with the programming of such AIs being carefully restricted to narrow lines of thinking
  2. AIs with more powerful (general) capabilities, that operate purely in advisory capacities.

In that case, the damage that could be caused by failures of the first type of AI, whilst significant, would not involve threats to the entirety of human civilisation. And failures of the second type of AI would be restricted by the actions of humans as intermediaries.

This approach would require confidence that:

  1. The capabilities of AIs of the first type will remain narrow, despite competitive pressures to give these systems at least some extra rationality
  2. The design of AIs of the second type will prevent the emergence of any dangerous “subsidiary goals”.

As a special case of the second point, the design of these AIs will need to avoid any risk of the systems developing sentience or intrinsic motivation.

These are tough challenges – especially since we still have only a vague understanding of how desires and/or sentience can emerge as smaller systems combine and evolve into larger ones.

But since we are short of other options, it’s definitely something to be considered more fully.

An option for automatically aligned superintelligence

If controlling an AGI turns out to be impossible – as seems likely – what about the option that an AGI will have goals and principles that are fundamentally aligned with human wellbeing?

In such a case, it will not matter if an AGI is beyond human control. The actions it takes will ensure that humans have a very positive future.

The creation of such an AI – sometimes called a “friendly AI” – remains my best hope for humanity’s future.

However, there are severe difficulties in agreeing and encoding “goals and principles that are fundamentally aligned with human wellbeing”. I reviewed these difficulties in my chapter “The AI Alignment Problem”.

But what if such goals and principles are somehow part of an objective reality, awaiting discovery, rather than needing to be invented? What if something like the theory of “moral realism” is true?

In this idea, a principle like “treat humans well” would follow from some sort of a priori logical analysis, a bit like the laws of mathematics (such as the fact, discovered by one of the followers of Pythagoras, that the square root of two is an irrational number).

Accordingly, a sufficiently smart AGI would, all being well, reach its own conclusion that humans ought to be well treated.

Nevertheless, even in this case, significant risks would remain:

  • The principle might be true, but an AGI might not be motivated to discover it
  • The principle might be true, but an AGI, despite its brilliance, may fail to discover it
  • The principle might be true, and an AGI might recognise it, but it may take its own decision to ignore it – like the way that we humans often act in defiance of what we believe at the time to be overarching moral principles

The design criteria and initial conditions that we humans provide for an AGI may well influence the outcome of these risk factors.

I plan to return to these weighty matters in a future blog post!

Two different sorts of control

I’ve come to realise that there are not one but two questions of control of AI:

  1. Can we humans retain control of an AGI that we create?
  2. Can society as a whole control the actions of companies (or organisations) that may create an AGI?

Whilst both these control problems are profoundly hard, the second is less hard.

Moreover, it’s the second problem which is the truly urgent one.

This second control problem involves preventing teams inside corporations (and other organisations) from rushing ahead without due regard to questions of the potential outcomes of their work.

It’s the second control problem that the 21 principles which I highlight in my book are primarily intended to address.

When people say “it’s impossible to solve the AI control problem”, I think they may be correct regarding the first problem, but I passionately believe they’re wrong concerning the second problem.

The importance of psychology

When I review what people say about the progress and risks of AI, I am frequently struck by the fact that apparently intelligent people are strongly attached to views that are full of holes.

When I try to point out the flaws in their thinking, they hardly seem to pause in their stride. They portray a stubborn confidence that they are sure they are correct.

What’s at play here is more than logic. It’s surely a manifestation of humanity’s often defective psychology.

My book includes a short chapter “The denial of the Singularity” which touched on various matters of psychology. If I were to rewrite my book today, I believe that chapter would become larger, and that psychological themes would be spread more widely throughout the book.

Of course, noticing psychological defects is only the start of making progress. Circumventing or transcending these defects is an altogether harder question. But it’s one that needs a lot more attention.

The option of merging with AI

How can we have a better, more productive conversation about anticipating and managing AGI?

How can we avoid being derailed by ineffective arguments, hostile rhetoric, stubborn prejudices, hobby-horse obsessions, outdated ideologies, and (see the previous section) flawed psychology?

How might our not-much-better-than-monkey brains cope with the magnitude of these questions?

One possible answer is that technology can help us (so long as we use it wisely).

For example, the chapter “Uplifting politics”, from near the end of my book, listed ten ways for “technology improving politics”.

More broadly, we humans have the option to selectively deploy some aspects of technology to improve our capabilities in handling other aspects of technology.

We must recognise that technology is no panacea. But it can definitely make a big difference.

Especially if we restrict ourselves to putting heavy reliance only on those technologies – narrow technologies – whose mode of operation we fully understand, and where risks of malfunction can be limited.

This forms part of a general idea that “we humans don’t need to worry about being left behind by robots, or about being subjugated by robots, since we will be the robots”.

As I put it in the chapter “No easy solutions” in my book,

If humans merge with AI, humans could remain in control of AIs, even as these AIs rapidly become more powerful. With such a merger in place, human intelligence will automatically be magnified, as AI improves in capability. Therefore, we humans wouldn’t need to worry about being left behind.

Now I’ve often expressed strong criticisms of this notion of merger. I still believe these criticisms are sound.

But what these criticisms show is that any such merger cannot be the entirety of our response to the prospect of the emergence of AGI. They can only be part of the solution. That’s especially true because humans-augmented-by-technology are still very likely to lag behind pure technology systems, until such time as human minds might be removed from biological skulls and placed into new silicon hosts. That’s something that I’m not expecting to happen before the arrival of AGI, so it will be too late to solve (by itself) the problems of AI alignment and control.

(And since you ask, I probably won’t be in any hurry, even after the arrival of AGI, for my mind to be removed from my biological skull. I guess I might rethink that reticence in due course. But that’s rethinking for another day.)

The importance of politics

Any serious discussion about managing cataclysmically disruptive technologies (such as advanced AIs) pretty soon rubs up against the questions of politics.

That’s not just small-p “politics” – questions of how to collaborate with potential partners where there are many points of disagreement and even dislike.

It’s large-P “Politics” – interacting with presidents, prime ministers, cabinets, parliaments, and so on.

Questions of large-P politics occur throughout The Singularity Principles. My thoughts now, six months afterwards, is that even more focus should be placed on the subject of improving politics:

  • Helping politics to escape the clutches of demagogues and autocrats
  • Helping politics to avoid stultifying embraces between politicians and their “cronies” in established industries
  • Ensuring that the best insights and ideas of the whole electorate can rise to wide attention, without being quashed or distorted by powerful incumbents
  • Bringing everyone involved in politics rapidly up-to-date with the real issues regarding cataclysmically disruptive technologies
  • Distinguishing effective regulations and incentives from those that are counter-productive.

As 2022 has progressed, I’ve seen plenty new evidence of deep problems within political systems around the world. These problems were analysed with sharp insight in the book The Revenge of Power by Moisés Naím that I recently identified as “the best book that I read in 2022”.

Happily, as well as evidence of deep problems in our politics worldwide, there are also encouraging signs, as well as sensible plans for improvement. You can find some of these plans inside the book by Naím, and, yes, I offer suggestions in my own book too.

To accelerate improvements in politics was one of the reasons I created Future Surge a few months back. That’s an initiative on which I expect to spend a lot more of my time in 2023.

Note: the image underlying the picture at the top of this article was created by DALL.E 2 from the prompt “A brain with a human face on it rethinks, vivid stormy sky overhead, photorealistic style”.

18 June 2020

Transhumanist alternatives to contempt and fear

Filed under: London Futurists, politics, Transcending Politics — Tags: , , , , , , , — David Wood @ 2:02 pm

Contempt and fear. These are the public reactions that various prominent politicians increasingly attract these days.

  • We feel contempt towards these politicians because they behave, far too often, in contemptible ways.
  • We feel fear regarding these politicians on account of the treacherous paths they appear to be taking us down.

That’s why many fans of envisioning and building a better world – including many technologists and entrepreneurs – would prefer to ignore politics, or to minimise its influence.

These critics of politics wish, instead, to keep their focus on creating remarkable new technology or on building vibrant new business.

Politics is messy and ugly, say these critics. It’s raucous and uncouth. It’s unproductive. Some would even say that politics is unnecessary. They look forward to politics reducing in size and influence.

Their preferred alternative to contempt and fear is to try to put the topic our of their minds.

I disagree. Putting our heads in the sand about politics is a gamble fraught with danger. Looking the other way won’t prevent our necks from being snapped when the axe falls. As bad outcomes increase from contemptible, treacherous politics, they will afflict everyone, everywhere.

We need a better alternative. Rather than distancing ourselves from the political sphere, we need to engage, intelligently and constructively.

As I’ll review below, technology can help us in that task.

Constructive engagement

Happily, as confirmed by positive examples from around the world, there’s no intrinsic reason for politics to be messy or ugly, raucous or uncouth.

Nor should politics be seen as some kind of unnecessary activity. It’s a core part of human life.

Indeed, politics arises wherever people gather together. Whenever we collectively decide the constraints we put on each other’s freedom, we’re taking part in politics.

Of course, this idea of putting constraints on each other’s freedoms is deeply unpopular in some circles. Liberty means liberty, comes the retort.

My answer is: things are more complicated. That’s for two reasons.

To start with, there are multiple kinds of freedom, each of which are important.

For example, consider the “four essential human freedoms” highlighted by US President FD Roosevelt in a speech in January 1941:

We look forward to a world founded upon four essential human freedoms.

The first is freedom of speech and expression – everywhere in the world.

The second is freedom of every person to worship God in their own way – everywhere in the world.

The third is freedom from want – which, translated into world terms, means economic understandings which will secure to every nation a healthy peacetime life for its inhabitants – everywhere in the world.

The fourth is freedom from fear – which, translated into world terms, means a world-wide reduction of armaments to such a point and in such a thorough fashion that no nation will be in a position to commit an act of physical aggression against any neighbour – anywhere in the world.

As well as caring about freeing people from constraints on their thoughts, speech, and actions, we generally also care about freeing people from hunger, disease, crime, and violence. Steps to loosen some of these constraints often risk decreasing other types of liberty. As I said, things are complicated.

The second reason builds on the previous point and makes it clearer why any proclamation “liberty means liberty” is overly simple. It is that our actions impact on each other’s wellbeing, both directly and indirectly.

  • If we speed in our cars, confident in our own ability to drive faster than the accepted norms, we risk seriously reducing the personal liberties of others if we suffer a momentary lapse in concentration.
  • If we share a hateful and misleading message on social media, confident in our own intellectual robustness, we might push someone reading that message over a psychological ledge.
  • If we discard waste products into the environment, confident that little additional harm will come from such pollution, we risk an unexpected accumulation of toxins and other harms.
  • If we grab whatever we can in the marketplace, confident that our own vigour and craftiness deserve a large reward, we could deprive others of the goods, services, and opportunities they need to enjoy a good quality of life.
  • If we publicise details of bugs in software that is widely used, or ways to increase the deadliness of biological pathogens, confident that our own reputation will rise as a result inside the peer groups we wish to impress, we risk enabling others to devastate the infrastructures upon which so much of life depends – electronic infrastructure and/or biological infrastructure.
  • If we create and distribute software that can generate mind-bending fake videos, we risk precipitating a meltdown in the arena of public discussion.
  • If we create and distribute software that can operate arsenals of weapons autonomously, freed from the constraints of having to consult slow-thinking human overseers before initiating an attack, we might gain lots of financial rewards, but at the risk of all manner of catastrophe from any defects in the design or implementation of that system.

In all these examples, there’s a case to agree some collective constraints on personal freedoms.

The rationale for imposing and accepting specific constraints on our freedom is in order to secure a state of affairs where overall freedom flourishes more fully. That’s a state of affairs in which we will all benefit.

In summary, greater liberty arises as a consequence of wise social coordination, rather than existing primarily as a reaction against such coordination. Selecting and enforcing social constraints is the first key task of politics.

Recognising and managing complexes

But who is the “we” who decides these constraints? And who will ensure that constraints put in place at one time, reflecting the needs of that time, are amended promptly when circumstances change, rather than remaining in place, disproportionately benefiting only a subset of society?

That brings us to a second key task of politics: preventing harmful dominance of society by self-interested groups of individuals – groups sometimes known as “complexes”.

This concept of the complex featured in the farewell speech made by President Eisenhower in January 1961. Eisenhower issued a profound warning that “the military industrial complex” posed a growing threat to America’s liberty and democracy:

In the councils of government, we must guard against the acquisition of unwarranted influence, whether sought or unsought, by the military-industrial complex. The potential for the disastrous rise of misplaced power exists and will persist.

We must never let the weight of this combination endanger our liberties or democratic processes. We should take nothing for granted. Only an alert and knowledgeable citizenry can compel the proper meshing of the huge industrial and military machinery of defence with our peaceful methods and goals, so that security and liberty may prosper together.

As a distinguished former military general, Eisenhower spoke with evident authority on this topic:

Until the latest of our world conflicts, the United States had no armaments industry. American makers of ploughshares could, with time and as required, make swords as well. But now we can no longer risk emergency improvisation of national defence; we have been compelled to create a permanent armaments industry of vast proportions. Added to this, three and a half million men and women are directly engaged in the defence establishment. We annually spend on military security more than the net income of all United States corporations.

This conjunction of an immense military establishment and a large arms industry is new in the American experience. The total influence – economic, political, even spiritual – is felt in every city, every Statehouse, every office of the Federal government. We recognize the imperative need for this development. Yet we must not fail to comprehend its grave implications. Our toil, resources and livelihood are all involved; so is the very structure of our society.

It’s one thing to be aware of the risks posed by a military industrial complex (and the associated trade in armaments). It’s another thing to successfully manage these risks. Similar risks apply as well, for other vested interest “complexes” that can likewise subvert societal wellbeing:

  • A carbon energy complex, which earns huge profits from the ongoing use of carbon-based fuels, and which is motivated to minimise appreciation of the risks to climate from continuing use of these fuels
  • A financial complex, which (likewise) earns huge profits, by means of complicated derivative products that are designed to evade regulatory scrutiny whilst benefiting in cases of financial meltdown from government handouts to banks that are perceived as “too big to fail”
  • An information technology complex, which collects vast amounts of data about citizens, and which enables unprecedented surveillance, manipulation, and control of people by corporations and/or governments
  • A medical industrial complex, which is more interested in selling patients expensive medical treatment over a long period of time than in low-cost solutions which would prevent illnesses in the first place (or cure them quickly)
  • A political complex, which seeks above all else to retain its hold on political power, often by means of undermining a free press, an independent judiciary, and any credible democratic opposition.

You can probably think of other examples.

In all these cases, the practical goals of the complex are only weakly aligned with the goals of society as a whole. If society is not vigilant, the complex will subvert the better intentions of citizens. The complex is so powerful that it cannot be controlled by mere words of advocacy.

Beyond advocacy, we need effective politics. This politics can be supported by a number of vital principles:

  • Transparency: The operations of the various complexes need to be widely publicised and analysed, bringing them out of the shadows into the light of public understanding
  • Disclosure: Conflicts of interest must be made clear, to avoid the public being misled by individuals with ulterior motives
  • Accountability: Instances where key information is found to have been suppressed or distorted need to be treated very seriously, with the guilty parties having their reputations adjusted and their privileges diminished
  • Assessment of externalities: Evaluation systems should avoid focusing too narrowly on short-term metrics (such as financial profit) but should take into full account both positive and negative externalities – including new opportunities and new risks arising
  • Build bridges rather than walls: Potential conflicts should be handled by diplomacy, negotiation, and seeking a higher common purpose, rather than by driving people into antagonistic rival camps that increasingly bear hatred towards one another
  • Leanness: Decisions should focus on questions that matter most, rather than dictating matters where individual differences can easily be tolerated
  • Democratic oversight: People in leadership positions in society should be subject to regular assessment of their performance by a democratic review, that involves a dynamic public debate aiming to reach a “convergent opinion” rather than an “average opinion”.

Critically, all the above principles can be assisted by smart adoption of technology that enhances collaboration. This includes wikis (or similar) that map out the landscape of decisions. This also includes automated logic-checkers, and dynamic modelling systems. And that’s just the start of how technology can help support a better politics.

Transhumanist approaches to politics

The view that technology can assist humans to carry out core parts of our lives better than before, is part of the worldview known as transhumanism.

Transhumanism asserts, further, than the assistance available from technology, wisely applied, extends far beyond superficial changes. What lies within our grasp is a set of radical improvements in the human condition.

As in the short video “An Introduction to Transhumanism” – which, with over a quarter of a million views, is probably the most widely watched video on the subject – transhumanism is sometimes expressed in terms of the so-called “three supers”:

  • Super longevity: significantly improved physical health, including much longer lifespans – transcending human tendencies towards physical decay and decrepitude
  • Super intelligence: significantly improved thinking capability – transcending human tendencies towards mental blind spots and collective stupidity
  • Super wellbeing: significantly improved states of consciousness – transcending human tendencies towards depression, alienation, vicious emotions, and needless suffering.

My own advocacy of transhumanism actually emphasises one variant within the overall set of transhumanist philosophies. This is the variant of transhumanism known as technoprogressive transhumanism. The technoprogressive variant of transhumanism in effect adds one more “super” to the three already mentioned:

  • Super democracy: significantly improved social inclusion and resilience, whilst upholding diversity and liberty – transcending human tendencies towards tribalism, divisiveness, deception, and the abuse of power.

These radical improvements, by the way, can be brought about by a combination of changes at the level of individual humans, changes in our social structures, and changes in the prevailing sets of ideas (stories) that we tend to tell ourselves. Exactly what is the best combination of change initiatives, at these different levels, is something to be determined by a mix of thought and experiment.

Different transhumanists place their emphases upon different priorities for potential transformation.

If you’d like to listen in to that ongoing conversation, let me draw your attention to the London Futurists webinar taking place this Saturday – 20th of June – from 7pm UK time (BST).

In this webinar, four leading transhumanists will be discussing and contrasting their different views on the following questions (along with others that audience members raise in real time):

  • In a time of widespread anxiety about social unrest and perceived growing inequalities, what political approach is likely to ensure the greatest liberty?
  • In light of the greater insights provided by science into human psychology at both the individual and group levels, what are the threats to our wellbeing that most need to be guarded against, and which aspects of human character most need to be protected and uplifted?
  • What does the emerging philosophy of transhumanism, with its vision of conscious life evolving under thoughtful human control beyond the current human form, have to say about potential political interventions?

As you can see, the webinar is entitled “Politics for greater liberty: transhumanist perspectives”. The panellists are:

For more details, and to register to attend, click here.

Other views on the future of governance and the economy

If you’d like to hear a broader set of views on a related topic, then consider attending a Fast Future webinar taking place this Sunday – 21st June – from 6pm UK time (BST).

There will be four panellists in that webinar – one being me. We’ll each be be presenting a snapshot of ideas from the chapters we contributed to the recent Fast Future book, Aftershocks and Opportunities – Scenarios for a Post-Pandemic Future, which was published on June 1st.

After the initial presentations, we’ll be responding to each other’s views, and answering audience questions.

My own topic in this webinar will be “More Aware, More Agile, More Alive”.

The other panellists, and their topics, will be:

  • Geoff Mulgan – “Using the Crisis to Remake Government for the Future”
  • Bronwyn Williams – “The Great Separation”
  • Rohit Talwar – “Post-Pandemic Government and the Economic Recovery Agenda: A Futurist Perspective”

I’m looking forward to a lively discussion!

Click here for more details of this event.

Transcending Politics

As I said above (twice), things are complicated. The science and engineering behind the various technological solutions are complicated. And the considerations about regulations and incentives, to constrain and guide our collective use of that technology, are complicated too. We should beware any overly simple claims about easy answers to these issues.

My fullest treatment of these issues is in a 423 page book of mine, Transcending Politics, that I published in 2018.

Over the last couple of weeks, I’ve been flicking through some of the pages of that book again. Although there are some parts where I would now wish to use a different form of expression, or some updated examples, I believe the material stands the test of time well.

If the content in this blogpost strikes you as interesting, why not take a closer look at that book? The book’s website contains opening extracts of each of the chapters, as well as an extended table of contents. I trust you’ll like it.

Older Posts »

Blog at WordPress.com.