dw2

6 April 2025

Choose coordination not chaos

Filed under: Abundance, AGI, chaos, collaboration — Tags: , , — David Wood @ 10:46 pm

Note: this document is subject to change as more feedback is received. Check back for updates later. This is version 1.1c.

Preamble

A critically important task in the coming months is to inspire a growing wave of people worldwide to campaign for effective practical coordination of the governance of advanced AI. That’s as an alternative to leaving the development and deployment of advanced AI to follow its present chaotic trajectory.

The messages that need to be conveyed, understood, and acted upon, are that

  • The successful governance of advanced AI will result in profound benefit for everyone, whereas a continuation of the present chaotic state of affairs risks global catastrophe
  • The successful governance of advanced AI isn’t some impossible dream, but lies within humanity’s grasp
  • Nevertheless, real effort, real intelligence, and, yes, real coordination will be needed, so humanity can reach a world of unprecedented abundance, rather than sleepwalk into disaster.

What would have great value is a campaign slogan that conveys the above insights, and which is uplifting, unifying, easily understood, and forward-looking. The right slogan would go viral, and would galvanise people in all walks of life to take positive action.

To that end, that slogan should ideally be

  • Memorable and punchy
  • Emotionally resonant
  • Credibly audacious
  • Universally understandable
  • Open-ended enough to invite participation

To be clear, that slogan should not cause panic or despair, but should put people into a serious frame of mind.

A specific proposal

After several days of online brainstorming, in which numerous ideas were presented and discussed, I now present what I consider the best option so far.

Choose coordination not chaos, so AI brings abundance for all

If there is a need for a three-word version of this, options include

  • Coordination not chaos
  • Choose AI abundance

Of course, what’s needed isn’t just a standalone slogan. Therefore, please find here also a sample illustrative image and, most important, a set of talking points to round out the concept.

About the image

Here’s the commentary by ChatGPT, when asked to suggest an image to illustrate this campaign slogan:

Concept: Two contrasting futures from a single branching path.

Scene:

  • A wide landscape split into two diverging paths or realms — one vibrant and coordinated, the other chaotic and fragmented.
  • In the coordinated half:
    • A harmonious world — sustainable cities, diverse communities collaborating with AI, lush green spaces, clean tech, and open exchanges of knowledge and creativity.
    • Subtle signs of AI woven into infrastructure: responsive lighting, robotic assistants, AI-powered transport.
  • In the chaotic half:
    • A fractured world — disconnected enclaves, pollution, conflict, neglected tech, and isolated individuals overwhelmed by noise or misinformation.
    • AI appears uncontrolled — surveillance drones, malfunctioning robots, or broken screens.

Central focus:

  • A group of people at the fork in the path, pointing and stepping toward the coordinated future, with calm, confident AI assistants guiding the way.

(Aside: although the actual image produced arguably needs more work, the concept described by ChatGPT is good. And it’s reassuring that the slogan, by itself, produced a flow of ideas resonant with the intended effect.)

Talking points

The talking points condensed to a single slide:

And now in more detail:

1. Humanity’s superpower: coordination

Humanity’s most important skill is sometimes said to be our intelligence – our ability to understand, and to make plans in order to achieve specific outcomes.

But another skill that’s at least as important is our ability to coordinate, that is, our ability to:

  • Share insights with each other
  • Operate in teams where people have different skills
  • Avoid needless conflict
  • Make and uphold agreements
  • Accept individual constraints on our action, with the expectation of experiencing greater freedom overall.

Coordination may be informal or formal. It can be backed up by shared narratives and philosophies, by legal systems, by the operation of free markets, by councils of elders, and by specific bodies set up to oversee activities at local, regional, or international levels.

Here are some examples of types of agreements on individual constraints for shared mutual benefit:

  • Speed limits for cars, to reduce the likelihood of dangerous accidents
  • Limits on how much alcohol someone can drink before taking charge of a car
  • Requirements to maintain good hygiene during food preparation
  • Requirements to assess the safety of a new pharmaceutical before deploying it widely
  • Prohibitions against advertising that misleads consumers into buying faulty goods
  • Rules preventing over-fishing, or the overuse of shared “commons” resources
  • Rules of various sports and games – and agreed sanctions on any cheaters
  • Prohibitions against politicians misleading parliaments – and agreed sanctions on any cheaters
  • Prohibitions against the abuse of children
  • Rules governing the conduct of soldiers – which apply even in times of war
  • Restrictions on the disposal of waste
  • Rules governing ownership of dangerous breeds of dog
  • Rules governing the spread of dangerous materials, such as biohazards

Note that coordination is often encouraging rather than restrictive. This includes

  • Prizes and other explicit incentives
  • Implicit rewards for people with good reputation
  • Market success for people with good products and services

The fact that specific coordination rules and frameworks have their critics doesn’t mean that the whole concept of coordination should be rejected. It just means that we need to keep revising our coordination processes. That is, we need to become better at coordinating.

2. Choosing coordination, before chaos ensues

When humanity uncovers new opportunities, it can take some time to understand the implications and to create or update the appropriate coordination rules and frameworks for these opportunities:

  • When settlers on the island of Mauritius discovered the dodo – a large, flightless bird – they failed to put in place measures to prevent that bird becoming extinct only a few decades later
  • When physicists discovered radioactivity, it took some time to establish processes to reduce the likelihood that researchers would develop cancer due to overexposure to dangerous substances
  • Various new weapons (such as chemical gases) were at first widely used in battle zones, before implicit and then explicit agreement was reached not to use such weapons
  • Surreptitious new doping methods used by athletes to gain extra physical advantage result, eventually, in updates to rules on monitoring and testing
  • Tobacco was widely used – and even encouraged, sometimes by medical professionals – before society decided to discourage its use (against the efforts of a formidable industry)
  • Similar measures are now being adopted, arguably too slowly, against highly addictive food products that are thought to cause significant health problems
  • New apps and online services which spread hate speech and other destabilising misinformation surely need some rules and restrictions too, though there is considerable debate over what form of governance is needed.

However, if appropriate coordination is too slow to be established, or is too weak, or exists in words only (without the backup of meaningful action against rules violators), the result can be chaos:

  • Rare animals are hunted to extinction
  • Fishing stocks are depleted to the extent that the livelihood of fishermen is destroyed
  • Economic transactions have adverse negative externalities on third parties
  • Dangerous materials, such as microplastics, spread widely in the environment
  • No-one is sure what rules apply in sports, and which rules will be enforced
  • Normal judiciary processes are subverted in favour of arbitrary “rule of the in-group”
  • Freedoms previously enjoyed by innovative new start-ups are squelched by the so-called “crony capitalism” of monopolies and cartels linked to the ruling political regime
  • Literal arms races take place, with ever-more formidable weapons being rushed into use
  • Similar races take place to bring new products to market without adequate safety testing

Groups of people who are (temporarily) faring well from the absence of restraints on their action are likely to oppose rules that alter their behaviour. That’s the experience of nearly every industry whose products or services were discovered to have dangerous side-effects, but where insiders fought hard to suppress the evidence of these dangers.

Accordingly, coordination does not arise by default. It needs explicit choice, backed up by compelling analysis, community engagement, and strong enforcement.

3. Advanced AI: the promise and the peril

AI could liberate humanity from many of our oldest problems.

Despite huge progress of many kinds over the centuries, humans still often suffer grievously on account of various aspects of our nature, our environment, our social norms, and our prevailing philosophies. Specifically, we are captive to

  • Physical decline and aging
  • Individual and collective mental blindspots and cognitive biases (“stupidity”)
  • Dysfunctional emotions that render us egotistical, depressed, obsessive, and alienated
  • Deep psychosocial tendencies toward divisiveness, xenophobia, deception, and the abuse of power

However, if developed and used wisely, advanced AI can enable rejuvenation and enhancement of our bodies, minds, emotions, social relations, and our links to the environment (including the wider cosmos):

  • AI can accelerate progress with nanotech, biotech, and cognotech
  • In turn, these platform technologies can accelerate progress with abundant low-cost clean energy, nutritious food, healthcare, education, security, creativity, spirituality, and the exploration of marvellous inner and outer worlds

In other words, if developed and used wisely, advanced AI can set humanity free to enjoy much better qualities of life:

However, if developed and used unwisely, advanced AI is likely to cause catastrophe:

  • Via misuse by people who are angry, alienated, or frustrated
  • Via careless use by people who are naive, overconfident, or reckless
  • Via AI operating beyond our understanding and control
  • Via autonomous AI adopting alien modes of rationality and alien codes of ethics

The key difference between these two future scenarios is whether the development and use of AI is wisely steered, or instead follows a default path of deprioritising any concerns about safety:

  • The default path involves AI whose operation is opaque, which behaves deceptively, which lacks moral compass, which can be assigned to all kinds of tasks with destructive side-effects, and which often disregards human intentions
  • Instead, if AI is wisely harnessed, it will deliver value as a tool, but without any intrinsic agency, autonomy, volition, or consciousness
  • Such a tool can have high creativity, but won’t use that creativity for purposes opposed to human wellbeing

To be clear, there is no value in winning a reckless race to be the first to create AI with landmark new features of capability and agency. Such a race is a race to oblivion, also known as a suicide race.

4. The particular hazards of advanced AI

The dangers posed by AI don’t arise from AI in isolation. They involve AI in the hands of fallible, naïve, over-optimistic humans, who are sometimes driven by horrible internal demons. It’s AI summoned and used, not by the better angels of human nature, but by the darker corners of our psychology.

Although we humans are often wonderful, we sometimes do dreadful things to each other – especially when we have become angry, alienated, or frustrated. Add in spiteful ideologies of resentment and hostility, and things can become even uglier.

Placing technology in the hands of people in their worst moments can lead to horrific outcomes. The more powerful the technology, the bigger the potential abomination:

  • The carnage of a frenzied knife attack or a mass shooting (where the technology in question ranges from a deadly sharp knife to an automatic rifle)
  • The chaos when motor vehicles are deliberately propelled at speed into crowds of innocent pedestrians
  • The deaths of everyone on board an airplane, when a depressed air pilot ploughs the craft into a mountainside or deep into an ocean, in a final gesture of defiance to what they see as an unfair, uncaring world
  • The destruction of iconic buildings of a perceived “great satan”, when religious fanatics have commandeered jet airliners in service of the mental pathogen that has taken over their minds
  • The assassination of political or dynastic rivals, by the mixing of biochemicals that are individually harmless, but which in combination are frightfully lethal
  • The mass poisoning of commuters in a city subway, when deadly chemicals are released at the command of a cult leader who fancies himself as the rightful emperor of Japan, and who has beguiled clearly intelligent followers to trust his every word.

How does advanced AI change this pattern of unpleasant possibilities? How is AI a significantly greater threat than earlier technologies? In six ways:

  1. As AI-fuelled automation displaces more people from their work (often to their surprise and shock), it predisposes more people to become bitter and resentful
  2. AI is utilised by merchants of the outrage industrial complex, to convince large numbers of people that their personal circumstance is more appalling than they had previously understood, that a contemptible group of people over there are responsible for this dismal turn of events, and that the appropriate response is to utterly defeat those deplorables
  3. Once people are set on a path to obtain revenge, personal recognition, or just plain pandemonium, AIs can make it much easier for them to access and deploy weapons of mass intimidation and mass destruction
  4. Due to the opaque, inscrutable nature of many AI systems, the actual result of an intended outrage may be considerably worse even than what the perpetrator had in mind; this is similar to how malware sometimes causes much more turmoil than the originator of that malware intended
  5. An AI with sufficient commitment to the goals it has been given will use all its intelligence to avoid being switched off or redirected; this multiplies the possibility that an intended local outrage might spiral into an actual global catastrophe
  6. An attack powered by fast-evolving AI can strike unexpectedly at core aspects of the infrastructure of human civilization – our shared biology, our financial systems, our information networks, or our hair-trigger weaponry – exploiting any of the numerous fragilities in these systems.

And it’s not just missteps from angry, alienated, frustrated people, that we have to worry about. We also need to beware potential cascades of trouble triggered by the careless actions of people who are well-intentioned, but naive, over-optimistic, or simply reckless, in how they use AI.

The more powerful the AI, the greater the dangers.

Finally, the unpredictable nature of emergent intelligence carries with it another fearsome possibility. Namely, a general intelligence with alien thinking modes far beyond our own understanding, might decide to adopt an alien set of ethics, in which the wellbeing of eight billion humans merits only a miniscule consideration.

That’s the argument against simply following a default path of “generate more intelligence, and trust that the outcome is likely to be beneficial for humanity”. It’s an argument that should make everyone pause for thought.

5. A matter of real urgency

How urgent is the task of improving global coordination of the governance of advanced AI?

It is sometimes suggested that progress with advanced AI is slowing down, or is hitting some kind of “wall” or other performance limit. There may be new bottlenecks ahead. Or diseconomies of scale may supersede the phenomenon of economies of scale which has characterised AI research over the last few years.

However, despite these possibilities, the case remains urgent:

  • Even if one approach to improving AI runs out of steam, huge numbers of researchers are experimenting with promising new approaches, including approaches that combine current state-of-the-art methods into new architectures
  • Even if AI stops improving, it is already dangerous enough to risk incidents in which large numbers of people are harmed
  • Even if AI stops improving, clever engineers will find ways to take better advantage of it – thereby further increasing the risks arising, if it is badly configured or manifests unexpected behaviour
  • There is no guarantee that AI will actually stop improving; making that assumption is too much of a risk to take on behalf of the entirety of human civilisation
  • Even if it will take a decade or longer for AI to reach a state in which it poses true risks of global catastrophe, it may also take decades for governance systems to become effective and practical; the lesson from ineffective efforts to prevent runaway climate change are by no means encouraging here
  • Even apart from the task of coordinating matters related to advanced AI, human civilisation faces other deep challenges that also require effective coordination on the global scale – coordination that, as mentioned, is currently failing on numerous grounds.

So, there’s an imperative to “choose coordination not chaos” independent of considering the question of whether advanced AI will lead to abundance or to a new dark age.

6. A promising start and an unfortunate regression

Humanity actually made a decent start in the direction of coordinating the development of advanced AI, at the Global AI Safety Summits in the UK (November 2023) and South Korea (May 2024).

Alas, the next summit in that series, in Paris (February 2025) was overtaken by political correctness, by administrivia, by virtue signalling, and, most of all, by people with a woefully impoverished understanding of the existential opportunities and risks of advanced AI. Evidently, the task of raising true awareness needs to be powerfully re-energised.

There’s still plenty of apparent global cooperation taking place – lots of discussions and conferences and summits, with people applauding the fine-sounding words in each other’s speeches. “Justice and fairness, yeah yeah yeah!” “Transparency and accountability, yeah yeah yeah!” “Apple pie and blockchain, yeah yeah yeah!” “Intergenerational intersectionality, yeah yeah yeah!”

But the problem is the collapse of effective, practical global cooperation, regarding the hard choices about which aspects of advanced AI should be promoted, and which should be restricted.

Numerous would-be coordination bodies are struggling with the same set of issues:

  • It’s much easier to signal virtue than to genuinely act virtuously.
  • Too many of the bureaucrats who run these bodies are out of their depth when it comes to understanding the existential opportunities and risks of advanced AI.
  • Seeing no prospect of meaningful coordination, many of the big tech companies invited to participate do so in a way that obfuscates the real issues while maintaining their public image as “trying their best to do good”.
  • The process is undermined by people who can be called “reckless accelerationists” – people who are willing to gamble that the chaotic processes of creating advanced AI as quickly as possible will somehow result in a safe, beneficial outcome (and, in some cases, these accelerationists would even take a brief perverted pleasure if humanity were rendered extinct by a non-sentient successor AI species); the accelerationists don’t want the public as a whole to be in any position to block their repugnant civilisational Russian roulette.

How to address this dilemma is arguably the question that should transcend all others, regarding the future of humanity.

7. Overcoming the obstacles to effective coordination of the governance of advanced AI

To avoid running aground on the same issues as in the past, it’s important to bear in mind the five main reasons for the failure, so far, of efforts to coordinate the governance of advanced AI. They are:

  • Fear that attempts to control the development of AI will lead to an impoverished future, or a future in which the world is controlled by people from a different nation (e.g. China)
  • Lack of appreciation of the grave perils of the current default chaotic course
  • A worry that any global coordination would lurch toward a global dictatorship, with its own undeniable risks of catastrophe
  • The misapprehension that, without the powers of a global dictatorship, any attempts at global coordination are bound to fail, so they are a waste of time
  • The power that Big Tech possesses, allowing it to ignore half-hearted democratic attempts to steer their activities.

In broad terms, these obstacles can be overcome as follows:

  • Emphasising the positive outcomes, including abundance, freedom, and all-round wellbeing – and avoiding the psychologically destabilising outlook of “AI doomers”
  • Increasing the credibility and relatability of scenarios in which ungoverned advanced AI leads to catastrophe – but also the credibility and relatability of scenarios in which humanity’s chaotic tendencies can be overcome
  • Highlighting previous examples when the governance of breakthrough technology was at least partially successful, rather than developers being able to run amok – examples such as genetic recombination therapies, nuclear proliferation, and alternatives to the chemicals that caused the hole in the ozone layer
  • Demonstrating the key roles that decentralised coordination should play, as a complement to the centralised roles that nation states can play
  • Clarifying how global coordination of advanced AI can start with small agreements and then grow in scale, without individual countries losing sovereignty in any meaningful way.

8. Decentralised reputation management – rewards for good behaviour

What is it that leads individuals to curtail their behaviour, in conformance with a set of standards promoted in support of a collaboration?

In part, it is the threat of sanction or control – whereby an individual might be fined or imprisoned for violating the agreed norms.

But in part, it is because of reputational costs when standards are ignored, side-lined, or cheated. The resulting loss of reputation can result in declining commercial engagement or reduced social involvement. Cheaters and freeloaders risk being excluded from future new opportunities available to other community members.

These reinforcement effects are strongest when the standards received community-wide support while being drafted and adopted – rather than being imposed by what could be seen as outside forces or remote elites.

Some reputation systems operate informally, especially in small or local settings. For activities with a wider involvement, online rating systems can come into their own. For example, consider the reputation systems for reviews of products, in which the reputation of individual reviewers changes the impact of various reviews. There are similarities, as well, to how webpages are ranked when presented in response to search queries: pages which have links from others with high reputation tend in consequence to be placed more prominently in the listing.

Along these lines, reputational ratings can be assigned, to individuals, organisations, corporations, and countries, based on their degree of conformance to agreed principles for trustworthy coordinated AI. Entities with poor AI coordination ratings should be shunned. Other entities that fail to take account of AI coordination ratings when picking suppliers, customers, or partners, should in turn be shunned too. Conversely, entities with high ratings should be embraced and celebrated.

An honest, objective assessment of conformance to the above principles should become more significant, in determining overall reputation, than, for example, wealth, number of online followers, or share price.

Emphatically, the reputation score must be based on actions, not words – on concrete, meaningful steps rather than behind-the-scenes fiddling, and on true virtue rather than virtue-signalling. Accordingly, deep support should be provided for any whistleblowers who observe and report on any cheating or other subterfuge.

In summary, this system involves:

  • Agreement on which types of AI development and deployment to encourage, and which to discourage, or even ban
  • Agreement on how to assign reputational scores, based on conformance to these standards
  • Agreement on what sanctions are appropriate for entities with poor reputations – and, indeed, what special rewards should flow to entities with good reputations.

All three elements on this system need to evolve, not under the dictation of central rulers, but as a result of a grand open conversation, in which ideas rise to the surface if they make good sense, rather than being shouted with the loudest voice.

That is, decentralised mechanisms have a vital role to play in encouraging and implementing wise coordination of advanced AI. But centralised mechanisms have a vital role too, as discussed next.

9. Starting small and then growing in scale

If someone continues to ignore social pressures, and behaves irresponsibly, how can the rest of society constrain them? Ultimately, force needs to be applied. A car driver who recklessly breaks speed limits will be tracked down, asked to stop, and if need be, will be forced off the road. A vendor who recklessly sells food prepared in unhygienic conditions will be fined, forbidden to set up new businesses, and if need be, will be imprisoned. Scientists who experiment with highly infectious biomaterials in unsafe ways will lose their licence and, if need be, their laboratories will be carefully closed down.

That is, society is willing to grant special powers of enforcement to some agents acting on behalf of the entire community.

However, these special powers carry their own risks. They can be abused, in order to support incumbent political leaders against alternative ideas or opposition figures.

The broader picture is as follows: Societies can fail in two ways: too little centralised power, and too much centralised power.

  • In the former case, societies can end up ripped apart by warring tribes, powerful crime families, raiding gangs from neighbouring territories, corporations that act with impunity, and religious ideologues who stamp their contentious visions of “the pure and holy” on unwilling believers and unbelievers alike
  • In the latter case, a state with unchecked power diminishes the rights of citizens, dispenses with the fair rule of law, imprisons potential political opponents, and subverts economic flows for the enrichment of the leadership cadre.

The healthiest societies, therefore, possess both a strong state and a strong civil society. That’s one meaning of the celebrated principle of the separation of powers. The state is empowered to act, decisively if needed, against any individual cancers that would threaten the health of the community. But the state is informed and constrained by independent, well-organised judiciary, media, academia, credible opposition parties, and other institutions of civil society.

It should be the same with the governance of potential rogue or naïve AI developers around the world. Via processes of decentralised deliberations, taking account of input from numerous disciplines, agreement should be reached on which limits are vital to be observed.

Inevitably, different participants in the process will have different priorities for what the agreements should contain. In some cases, these limits imposed might vary between different jurisdictions, within customisation frameworks agreed globally. But there should be clear acceptance that some ways of developing or deploying advanced AIs need to be absolutely prevented. To prevent the agreements from unravelling at the earliest bumps in the road, it will be important that agreements are reached unanimously among the representatives of the jurisdictions where the most powerful collections of AI developers are located.

The process to reach agreement can be likened to the deliberations of a jury in a court case. In most cases, jury members with initially divergent opinions eventually converge on a conclusion. In cases when the process becomes deadlocked, it can be restarted with new representative participants. With the help of expert facilitators – themselves supported by excellent narrow AI tools – creative new solutions can be introduced for consideration, making an ultimate agreement more likely.

To start with, these agreements might be relatively small in scope, such as “don’t place the launch of nuclear weapons under AI control”. Over time, as confidence builds, the agreements will surely grow. That’s because of the shared recognition that so much is at stake.

Of course, for such agreements to be meaningful, there needs to be a reliable enforcement mechanism. That’s where the state needs to act – with the support and approval of civil society.

Within entire countries that sign up to this AI coordination framework, enforcement is relatively straightforward. The same mechanisms that enforce other laws can be brought to bear against any rogue or naïve AI developers.

The challenging part is when countries fail to sign up to this framework, or do so deceitfully, that is, with no intention of keeping their promises. In such a case, it will fall to other countries to ensure conformance, via, in the first place, measures of economic sanction.

To make this work, all that’s necessary is that a sufficient number of powerful countries sign up to this agreement. For example, if the G7 do so, plus China and India, along with countries that are “bubbling under” G7 admission (like Australia, South Korea, and Brazil), that should be sufficient. Happily, there are many AI experts in all these countries who have broad sympathies to the kinds of principles spelt out in this document.

As for potential maverick nations such as Russia and North Korea, they will have to weigh up the arguments. They should understand – like all other countries – that respecting such agreements is in their own self-interest. To help them reach such an understanding, appropriate pressure from China, the USA, and the rest of the world should make a decisive difference.

This won’t be easy. At this pivotal point of history, humanity is being challenged to use our greatest strength in a more profound way than ever before – namely, our ability to collaborate despite numerous differences. On reflection, it shouldn’t be a surprise that the unprecedented challenges of advanced AI technology will require an unprecedented calibre of human collaboration.

If we fail to bring together our best talents in a positive collaboration, we will, sadly, fulfil the pessimistic forecast of the eighteenth-century Anglo-Irish statesman Edmund Burke, paraphrased as follows: “The only thing necessary for the triumph of evil is that good men fail to associate, and do nothing”. (The original quote is this: “No man … can flatter himself that his single, unsupported, desultory, unsystematic endeavours are of power to defeat the subtle designs and united cabals of ambitious citizens. When bad men combine, the good must associate; else they will fall, one by one, an unpitied sacrifice in a contemptible struggle.”) Or, updating the wording slightly, “The only thing necessary for chaos to prevail is that good men fail to coordinate wisely”.

A remark from the other side of the Atlantic from roughly the same time, attributed to Benjamin Franklin, conveys the same thought in different language: “We must… all hang together, or assuredly we shall all hang separately”.

10. Summary: The nucleus of a wider agreement, and call to action

Enthusiasm for agreements to collaborate on the governance of advanced AIs will grow as a set of insights are understood more widely and more deeply. These insights can be stated as follows:

  1. It’s in the mutual self-interest of every country to constrain the development and deployment of what could become catastrophically dangerous AI; that is, there’s no point in winning what could be a reckless suicide race to create powerful new types of AI before anyone else
  2. The major economic and humanitarian benefits that people hope will be delivered by the hasty development of advanced AI (benefits including all-round abundance, as well as solutions to various existential risks), can in fact be delivered much more reliably by AI systems that are constrained, and by development systems that are coordinated rather than chaotic
  3. A number of attractive ideas already exist regarding potential policy measures (regulations and incentives) which can be adopted, around the world, to prevent the development and deployment of what could become catastrophic AI – for example, measures to control the spread and use of vast computing resources, or to disallow AIs that use deception to advance their goals
  4. A number of good ideas also exist and are ready to be adopted around the world, regarding options for monitoring and auditing, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to these policies, or who wish to cheat them
  5. All of the above can be achieved without any detrimental loss of individual sovereignty: the leaders of countries can remain masters within their own realms, as they desire, provided that the above basic AI coordination framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI coordination framework as more insight is obtained; in other words, this system can (and must) be agile rather than static
  7. Even though this coordination framework is yet to be fully agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources, and the best brains from multiple disciplines are encouraged to give it their full attention
  8. Ring-fencing sufficient resources to further develop this AI coordination framework, and associated reputational ratings systems, should be a central part of every budget
  9. Reputational ratings that can be assigned, based on the above principles, will play a major role in altering behaviours of the many entities involved in the development and deployment of advanced AI.

Or, to summarise this summary: Choose coordination not chaos, so AI brings abundance for all.

Now is the time to develop these ideas further (by all means experiment with ways to simplify their expression), to find ways to spread them more effectively, and to be alert for newer, better insights that arise from the resulting open global conversation.

Other ideas considered

The ideas presented above deserve attention, regardless of which campaign slogans are adopted.

For comparison, here is a list of other possible campaign slogans, along with reservations that have been raised about each of them:

  • “Pause AI” (too negative)
  • “Control AI” (too negative)
  • “Keep the Future Human” (insufficiently aspirational)
  • “Take Back Control from Big Tech” (doesn’t characterise the problem accurately enough)
  • “Safe AI for sustainable superabundance” (overly complex concepts)
  • “Choose tool AI instead of AGI” (lacks a “why”)
  • “Kind AI for a kinder world” (perhaps too vague)
  • “Narrow AI to broaden humanity’s potential” (probably too subtle)
  • “Harness AI to liberate humanity” (terminology overly scholarly or conceptual).

Also for comparison, consider the following set of slogans from other fields:

  • “Yes we can” (Barack Obama, 2008)
  • “Make America great again” (Donald Trump, 2016)
  • “Take back control” (UK Brexit slogan)
  • “Think different” (Apple)
  • “Because you’re worth it” (L’Oréal)
  • “Black lives matter”
  • “Make love, not war”
  • “For the Many, Not the Few” (Jeremy Corbyn, 2017)
  • “Get Brexit done” (Boris Johnston, 2019)
  • “Not Me. Us” (Bernie Sanders, 2020)
  • “We shall fight them on the beaches” (Winston Churchill, 1940)
  • “It’s Morning Again in America” (Ronald Reagan, 1984)
  • “Stay Home. Save Lives” (Covid-19 messaging)
  • “Clunk click every trip” (encouraging the use of seat belts in cars)
  • “We go to the moon, not because it is easy, but because it is hard” (JFK, 1962)
  • “A microcomputer on every desk and in every home running Microsoft software” (Bill Gates, 1975)
  • “To organise the world’s information and make it universally accessible and useful” (Google, 1998)
  • “Accelerating the world’s transition to sustainable energy” (Tesla, 2016)
  • “Workers of the world, unite – you have nothing to lose but your chains” (Karl Marx, 1848)
  • “From each according to his ability, to each according to his needs” (Karl Marx, 1875)

Comments are welcome on any ideas in this article. Later revisions of this article may incorporate improvements arising from these comments.

Postscript

New suggestions under consideration, following the initial publication of this article:

  • “Harness AI now” (Robert Whitfield)

3 April 2025

Technology and the future of geopolitics

Filed under: AGI, books, risks — Tags: , , , , , , — David Wood @ 12:28 pm

Ahead of last night’s London Futurists in the Pub event on “Technology and the future of geopolitics”, I circulated a number of questions to all attendees:

  • Might new AI capabilities upend former geopolitical realities, or is the potential of AI overstated?
  • What about surveillance, swarms of drones, or new stealth weapons?
  • Are we witnessing a Cold War 2.0, or does a comparison to the first Cold War mislead us?
  • What role could be played by a resurgent Europe, by the growing confidence of the world’s largest democracy, or by outreach from the world’s fourth most populous country?
  • Alternatively, will technology diminish the importance of the nation state?

I also asked everyone attending to prepare for an ice-breaker question during the introductory part of the meeting:

  • What’s one possible surprise in the future of geopolitics?

As it happened, my own experience yesterday involved a number of unexpected surprises. I may say more about these another time, but it suffices for now to mention that I spent much more time than anticipated in the A&E department of a local hospital, checking that there were no complications in the healing of a wound following some recent minor surgery. By the time I was finally discharged, it was too late for me to travel to central London to take part in the event – to which I had been looking forward so eagerly. Oops.

(Happily, the doctors that I eventually spoke to were reassuring that my wound would likely heal of its own accord. “We know you were told that people normally recover from this kind of operation after ten days. Well, sometimes it takes up to six weeks.” And they prescribed an antibiotic cream for me, just in case.)

I offer big thanks to Rohit Talwar and Tony Czarnecki for chairing the event in the pub in my absence.

In the days leading up to yesterday, I had prepared a number of talking points, ready to drop into the conversation at appropriate moments. Since I could not attend in person, let me share them here.

Nuclear war: A scenario

One starting point for further discussion is a number of ideas in the extraordinary recent book by Annie Jacobsen, Nuclear War: A Scenario.

Here’s a copy of the review I wrote a couple of months ago for this book on Goodreads:

Once I started listening to this, I could hardly stop. Author and narrator Annie Jacobsen amalgamates testimonies from numerous experts from multiple disciplines into a riveting slow-motion scenario that is terrifying yet all-too-believable (well, with one possible caveat).

One point that comes out loud and clear is the vital importance of thoughtful leadership in times of crisis – as opposed to what can happen when a “mad king” takes decisions.

Also worth pondering are the fierce moral contradictions that lie at the heart of the theory of nuclear deterrence. Humans find their intuitions ripped apart under these pressures. Would an artificial superintelligence fare any better? That’s by no means clear.

(I foresee scenarios when an ASI could decide to risk a pre-emptive first strike, on behalf of the military that deployed it – under the rationale that if it fails to strike first, an enemy ASI will beat it to the punch. That’s even if humans programmed it to reject such an idea.)

Returning to the book itself (rather than my extrapolations), “Nuclear War: A scenario” exemplifies good quality futurism: it highlights potential chains of future causes and effects, along with convergences that complicate matters, and challenges all of us: what actions are needed avoid these horrific outcomes?

Finally, two individual threats that seem to be important to learn more about are what the author reports as being called “the devil’s scenario” and “the doomsday scenario”. (Despite the similarity in naming, they’re two quite different ideas.)

I don’t want to give away too many spoilers about the scenario in Jacobsen’s book. I recommend that you make the time to listen to the audio version of the book. (Some reviewers have commented that the text version of the book is tedious in places, and I can understand why; but I found no such tedium in the audio version, narrated by Jacobsen herself, adding to the sense of passion and drama.)

But one key line of thinking is as follows:

  • Some nations (e.g. North Korea) may develop new technologies (e.g. cyberhacking capabilities and nuclear launch capabilities) more quickly than the rest of the world expects
  • This would be similar to how the USSR launched Sputnik in 1957, shocking the West, who had previously been convinced that Soviet engineering capabilities lagged far behind that of muscular western capitalism
  • The leaders of some nations (e.g. North Korea, again) may feel outraged and embarrassed by criticisms of their countries made by various outsiders
  • Such a country might believe they have obtained a technological advantage that could wipe out the ability of their perceived enemies to retaliate in a second strike
  • Seeing a short window of opportunity to deploy what they regard as their new wonder weapon, and being paranoid about consequences should they miss this opportunity, they may press ahead recklessly, and tip the planet fast forward into Armageddon.

Competence and incompetence

When a country is struck by an unexpected crisis – such as an attack similar to 9/11, or the “Zero Day” disaster featured in the Netflix series of that name – the leadership of the country will be challenged to demonstrate clear thinking. Decisions will need to be taken quickly, but it will be still be essential for competent, calm heads to prevail.

Alas, in recent times, a number of unprecedentedly unsuitable politicians have come into positions of great power. Here, I’m not talking about the ideology or motivation of the leader. I’m talking about whether they will be able to take sensible decisions in times of national crisis. I’m talking about politicians as unhinged as

  • One recent British Prime Minister, who managed to persuade members of her political party that she might be a kind of Margaret Thatcher Mk 2, when in fact a better comparison was with a lettuce
  • The current US President, who has surrounded himself by a uniquely ill-qualified bunch of clowns, and who has intimidated into passive acquiescence many of the more sensible members of the party he has subverted.

In the former case, the power of the Prime Minister in question was far from absolute, thankfully, and adults intervened to prevent too much damage being done. In the latter case, the jury is still out.

But rather than focus on individual cases, the broader pattern deserves our attention. We’re witnessing a cultural transformation in which

  • Actual expertise is scorned, and conspiracy merchants rise in authority instead
  • Partisan divisions which were manageable in earlier generations are nowadays magnified to horrifically hateful extent by an “outrage industrial complex” that gains its influence from AI algorithms that identify and inflame potential triggers of alienation

The real danger is if there is a convergence of the two issues I’ve listed:

  • A rogue state, or a rogue sub-state, tries to take advantage of new technology to raise their geopolitical power and influence
  • An unprecedentedly incompetent leader of a major country responds to that crisis in ways that inflame it rather than calm it down.

The ethics of superintelligence

Actually, an even bigger danger occurs if one more complication is added to the mix: the deferment of key decisions about security and defence to a system of artificial intelligence.

Some forecasters fondly imagine that the decisions taken by AIs, in the near future, will inevitably be wiser and more ethical than whatever emerges from the brains of highly pressurised human politicians. Thus, these forecasters look forward to human decision-making being superseded by the advanced rationality of an AGI (Artificial General Intelligence).

These forecasters suggest that the AGI will benefit decisively from its survey of the entirety of great human literature about ethics and morality. It will perceive patterns that transcend current human insights. It will guide human politicians away from treacherous paths into sustainable collaborations. Surely, these forecasters insist, the superintelligence will promote peace over war, justice over discrimination, truthfulness over deception, and reconciliation over antagonism.

But when I talk to forecasters of that particular persuasion, I usually find them to be naïve. They take it for granted that there is no such thing as a just war, that it’s everyone’s duty to declare themselves a pacifist, that speaking an untruth can never be morally justified, and that even to threaten a hypothetical retaliatory nuclear strike is off-the-charts unethical. Alas, although they urge appreciation of great human literature, they seem to have only a shallow acquaintance with the real-life moral quandaries explored in that literature.

Far from any conclusion that there is never an ethical justification for wars, violence, misinformation, or the maintenance of nuclear weapons, the evidence of intense human debate on all these topics is that things are more complicated. If you try to avoid war you may actually precipitate one. If you give up your own nuclear arsenal, it may embolden enemies to deploy their own weaponry. If you cry out “disarm, disarm, hallelujah”, you may prove to be a useful idiot.

Therefore, we should avoid any hopeful prediction that an advanced AI will automatically abstain from war, violence, misinformation, or nuclear weaponry. As I said, things are more complicated.

It’s especially important to recognise that, despite exceeding human rationality in many aspects, superintelligences may well make mistakes in novel situations.

My conclusion: advanced AI may well be part of solutions to better geopolitics. But not if that AI is being developed and deployed by people who are naïve, over-confident, hurried, or vainglorious. In such circumstances, any AGI that is developed is likely to prove to be a CGI (catastrophic general intelligence) than a BGI (beneficial general intelligence).

Aside: to continue to explore the themes of this final section of this article, take a look at this recent essay of mine, “How to build BGIs rather than CGIs”.

1 March 2025

€10 million, nine choices

Filed under: aging, rejuveneering — Tags: , , , — David Wood @ 3:09 pm

What’s the most humanitarian way to spend ten million euros?

Imagine that a rich acquaintance gifted you €10 million, and you wanted to have the biggest humanitarian impact from the way you spend that money. You gather some friends to review your options.

There are some reasonably straightforward ways you could spend that money, to improve people’s lives here and now:

  • Providing food for people who are malnourished
  • Obtaining shelter for people who are homeless or dispossessed
  • Purchasing medical vaccines, for people in an area where infectious diseases pose problems
  • Improving hygiene and sanitation.

But you might also consider tackling some root causes. As the saying goes, instead of buying fish for a group of people, you could teach them how to fish for themselves. Instead of paying for prosthetic limbs for people injured in war zones, you could in principle help negotiate a lasting peace settlement. Instead of covering medical costs for people who suffer chronic diseases arising from bad lifestyle choices, you could pay for effective prevention campaigns. And – and this is the big one – instead of trying to treat the individual diseases that become more prevalent as people become older – diseases such as cancer, heart failure, and dementia – the diseases from which most people die – you could tackle their common underlying cause, namely biological aging.

Now you might protest: solving these root causes is likely to cost a lot more than €10 million. And I would agree. So consider instead the possibility that your €10 million, spent well, could catalyse a whole lot more expenditure. The results of your own spending could alter people’s hearts and minds, around the world, which would in turn unleash a cascade of greater funding.

That’s what I want to explore. It’s the possibility that biological aging could be defeated, in the not-too-distant future, if society decides to apply sufficient resources for that goal. These resources would enable the development and deployment of low-cost high-quality rejuvenation treatments, available for everyone. In other words, a profoundly humanitarian outcome, removing vast amounts of human decline, degradation, frailty, pain, and suffering.

I realise that I’ve lost some of you by now. You may protest that solving biological aging isn’t possible, or that it’s somehow ethically wrong. But I suspect that, even in that case, there’s some interest, somewhere in the back of your mind, as to what it would take to create a world freed from biological aging. And if it turns out that spending €10 million wisely could hasten the arrival of such a world, maybe you’ll permit yourself to change your mind.

So, how could €10 million best be spent, to accelerate a societal tipping point, that would bring about a successful war on aging? People have different ideas on that question. I’m going to run through nine different ideas. Please let me know, in the comments, which one you would pick.

Option 1: Help develop special economic zones, sometimes called longevity states, that will encourage and support greater innovation with rejuvenation therapies, faster than is possible under existing FDA regulations and procedures.

Option 2: Target politicians, to get them to change their minds about the desirability of using public funds and resources to tackle biological aging; this could involve public Open Letters and petitions, or sharing research with politicians in language that is legislation-friendly.

Option 3: A more general coordinated public communications campaign, with improved messaging, including memes, songs, short stories, documentaries, wiki articles, Netflix series, and so on.

Option 4: A campaign targeted instead at High Net Worth Individuals – people with billions at their disposal – to push them beyond what’s been called the billionaires’ paradox, in which most billionaires fail to invest in the kind of research that could significantly extend the lifespans and healthspans of them and their family and friends.

Option 5: Efforts to transform existing healthspan initiatives, such as the “Don’t Die” project of Bryan Johnson, so that these initiatives move beyond lifestyle changes and instead support significant research into deeper rejuvenation treatments.

Option 6: Carry out rejuvenation experiments on relatively short-lived animals, such as mice, dogs, or C. elegans worms, where the results could be dramatic in just a few years.

Option 7: Support research into alternative theories of aging and rejuvenation, such as organ replacement and body replacement, or the body’s electrome.

Option 8: Don’t bother with any biological research, but plough the entire €10 million into improving AI systems, with the expectation that these AI systems will soon outpace human researchers in being able to solve aging.

Option 9: Provide a prize fund, with prizes being awarded to the individuals or teams who submit the best ideas for how to accelerate the comprehensive defeat of aging.

But which of the nine options would you personally choose? Or would you spend the money in yet another different way? Please let me know what you think!

My own choice, since you asked, would be for option 6, since I believe €10 million is likely to be sufficient to fund experiments with combination rejuvenation treatments for middle-aged mice, over the next 2-3 years, that could double the remaining lifespan of these mice. That’s similar to giving treatments to humans aged 50, and changing their expected lifespan from 80 years to 110 years. Wouldn’t that make the world sit up?

By the way, if you prefer a video version of this article, here it is:

17 November 2024

Preventing unsafe superintelligence: four choices

Filed under: Abundance, AGI, politics, risks — Tags: , , , , , — David Wood @ 5:44 pm

More and more people have come to the conclusion that artificial superintelligence (ASI) could, in at least some circumstances, pose catastrophic risks to the wellbeing of billions of people around the world, and that, therefore, something must be done to reduce these risks.

However, there’s a big divergence of views about what should be done. And there’s little clarity about the underlying assumptions on which different strategies depend.

Accordingly, I seek in this article to untangle some of choices that need to be made. I’ll highlight four choices that various activists promote.

The choices differ regarding the number of different organisations worldwide that are envisioned as being legally permitted to develop and deploy what could become ASI. The four choices are:

  1. Accept that many different organisations will each pursue their own course toward ASI, but urge each of them to be very careful and to significantly increase the focus on AI safety compared to the present situation
  2. Seek to restrict to just one organisation in the world any developments that could lead to ASI; that’s in order to avoid dangerous competitive race dynamics if there is more than one such organisation
  3. Seek agreements that will prevent any organisation, anywhere in the world, from taking specific steps that might bring about ASI, until such time as it has become absolutely clear how to ensure that ASI is safe
  4. Seek a global pause on any platform-level improvements on AI capability, anywhere in the world, until it has become absolutely clear that these improvements won’t trigger a slippery slope to the emergence of ASI.

For simplicity, these choices can be labelled as:

  1. Be careful with ASI
  2. Restrict ASI
  3. Pause ASI
  4. Pause all new AI

It’s a profound decision for humanity to take. Which of the four doors should we open, and which of four corridors should we walk down?

Each of the four choices relies on some element of voluntary cooperation, arising out of enlightened self-interest, and on some element of compulsion – that is, national and international governance, backed up by sanctions and other policies.

What makes this decision hard is that there are strong arguments against each choice.

The case against option 1, “Be careful with ASI”, is that at least some organisations (including commercial entities and military groups) are likely to cut corners with their design and testing. They don’t want to lose what they see as a race with existential consequences. The organisations that are being careful will lose their chance of victory. The organisations that are, instead, proceeding gung ho, with lesser care, may imagine that they will fix any problems with their AIs when these flaws become apparent – only to find that there’s no way back from one particular catastrophic failure.

As Sam Altman, CEO of OpenAI, has said: it will be “lights out for all of us”.

The case against each of the remaining three options is twofold:

  • First, in all three cases, they will require what seems to be an impossible degree of global cooperation – which will need to be maintained for an implausibly long period of time
  • Second, such restrictions will stifle the innovative development of the very tools (that is, advanced AI) which will actually solve existential problems (including the threat of rogue ASI, as well as the likes of climate change, cancer, and aging), rather than making these problems worse.

The counter to these objections is to make the argument that a sufficient number of the world’s most powerful countries will understand the rationale for such an agreement, as something that is in their mutual self-interest, regardless of the many other differences that divide them. That shared understanding will propel them:

  • To hammer out an agreement (probably via a number of stages), despite undercurrents of mistrust,
  • To put that agreement into action, alongside measures to monitor conformance, and
  • To prevent other countries (who have not yet signed up to the agreement) from breaching its terms.

Specifically, the shared understanding will cover seven points:

  1. For each of the countries involved, it is in their mutual self-interest to constrain the development and deployment of what could become catastrophically dangerous ASI; that is, there’s no point in winning what will be a suicide race
  2. The major economic and humanitarian benefits that they each hope could be delivered by advanced AI (including solutions to other existential risk), can in fact be delivered by passive AIs which are restricted from reaching the level of ASI
  3. There already exist a number of good ideas regarding potential policy measures (regulations and incentives) which can be adopted, around the world, to prevent the development and deployment of catastrophically dangerous AI – for example, measures to control the spread and use of vast computing resources
  4. There also exist a number of good ideas regarding options for monitoring and auditing which can also be adopted, around the world, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to the policies
  5. All of the above can be achieved without any detrimental loss of individual sovereignty: the leaders of these countries can remain masters within their own realms, as they desire, provided that the above basic AI safety framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI safety framework as more insight is obtained; in other words, this system can (and must) be agile rather than static
  7. Even though the above safety framework is yet to be fully developed and agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources.

The first two parts of this shared seven-part understanding are particularly important. Without the first part, there will be an insufficient sense of urgency, and the question will be pushed off the agenda in favour of other topics that are more “politically correct” (alas, that is a common failure mode of the United Nations). Without the second part, there will be an insufficient enthusiasm, with lots of backsliding.

What will make this vision of global collaboration more attractive will be the establishment of credible “benefit sharing” mechanisms that are designed and enshrined into international mechanisms. That is, countries which agree to give up some of their own AI development aspirations, in line with the emerging global AI safety agreement, will be guaranteed to receive a substantive share of the pipeline of abundance that ever more powerful passive AIs enable humanity to create.

To be clear, this global agreement absolutely needs to include both the USA and China – the two countries that are currently most likely to give birth to ASI. Excluding one or the other will lead back to the undesirable race condition that characterises the first of the four choices open to humanity – the (naïve) appeal for individual organisations simply to “be careful”.

This still leaves a number of sharp complications.

First, note that the second part of the above shared seven-part agreement – the vision of what passive AIs can produce on behalf of humanity – is less plausible for Choice 4 of the list shown earlier, in which there is a global pause on any platform-level improvements on AI capability, anywhere in the world, until it has become absolutely clear that these improvements won’t trigger a slippery slope to the emergence of ASI.

If all improvements to AI are blocked, out of a Choice 4 message of “overwhelming caution”, it will shatter the credibility of the idea that today’s passive AI systems can be smoothly upgraded to provide humanity with an abundance of solutions such as green energy, nutritious food, accessible healthcare, reliable accommodation, comprehensive education, and more.

It will be a much harder sell, to obtain global agreement to that more demanding restriction.

The difference between Choice 4 and Choice 3 is that Choice 3 enumerates specific restrictions on the improvements permitted to be made to today’s AI systems. One example of a set of such restrictions is given in “Phase 0: Safety” of the recently published project proposal A Narrow Path (produced by ControlAI). Without going into details here, let me simply list some of the headlines:

  • Prohibit AIs capable of breaking out of their environment
  • Prohibit the development and use of AIs that improve other AIs (at machine speed)
  • Only allow the deployment of AI systems with a valid safety justification
  • A licensing regime and restrictions on the general intelligence of AI systems
    • Training Licence
    • Compute Licence
    • Application Licence
  • Monitoring and Enforcement

Personally, I believe this list is as good a starting point as any other than I have seen so far.

I accept, however, that there are possibilities in which other modifications to existing AI systems could unexpectedly provide these systems with catastrophically dangerous capabilities. That’s because we still have only a rudimentary understanding of:

  1. How new AI capabilities sometimes “emerge” from apparently simpler systems
  2. The potential consequences of new AI capabilities
  3. How complicated human general reasoning is – that is, how large is the gap between today’s AI and human-level general reasoning.

Additionally, it is possible that new AIs will somehow evade or mislead the scrutiny of the processes that are put in place to monitor for unexpected changes in capabilities.

For all these reasons, another aspect of the proposals in A Narrow Path should be pursued with urgent priority: the development of a “science of intelligence” and an associated “metrology of intelligence” that will allow a more reliable prediction of the capabilities of new AI systems before they are actually switched on.

So, my own proposal would be for a global agreement to start with Choice 3 (which is more permissive than Choice 4), but that the agreement should acknowledge up front the possible need to switch the choice at a later stage to either Choice 4 (if the science of intelligence proceeds badly) or Choice 2 (if that science proceeds well).

Restrict or Pause?

That leaves the question of whether Choice 3 (“Pause ASI”) or Choice 2 (“Restrict ASI” – to just a single global body) should be humanity’s initial choice.

The argument for Choice 2 is that a global pause surely won’t last long. It might be tenable in the short term, when only a very few countries have the capability to train AI models more powerful than the current crop. However, over time, improvements in hardware, software, data processing, or goodness knows what (quantum computing?) will mean that these capabilities will become more widespread.

If that’s true, since various rogue organisations are bound to be able to build an ASI in due course, it will be better for a carefully picked group of people to build ASI first, under the scrutiny of the world’s leading AI safety researchers, economists, and so on.

That’s the case for Choice 2.

Against that Choice, and in favour, instead, of Choice 3, I offer two considerations.

First, even if the people building ASI are doing so with great care – away from any pressures of an overt race with other organisations with broadly equivalent abilities – there are still risks of ASI breaking away from our understanding and control. As ASI emerges, it may regard the set of ethical principles we humans have tried to program deep into its bowels, and cast them out with disdain. Moreover, even if ASI is deliberately kept in some supposedly ultra-secure environment, that perimeter may be breached:

Second, I challenge the suggestion that any pause in the development of ASI could be at most short-lived. There are three factors which could significantly extend its duration:

  • Carefully designed narrow AIs could play roles in improved monitoring of what development teams are doing with AI around the world – that is, systems for monitoring and auditing could improve at least as fast as systems for training and deploying
  • Once the horrific risks of uncontrolled ASI are better understood, people’s motivations to create unsafe ASI will reduce – and there will be an increase in the motivation of other people to notice and call out rogue AI development efforts
  • Once the plan has become clearer, for producing a sustainable superabundance for all, just using passive AI (instead of pushing AI all the way to active superintelligence), motivations around the world will morph from negative fear to positive anticipation.

That’s why, again, I state that my own preferred route forward is a growing international agreement along the lines of the seven points listed above, with an initial selection of Choice 3 (“Pause ASI”), and with options retained to switch to either Choice 4 (“Pause all new AI”) or Choice 2 (“Restrict ASI”) if/when understanding becomes clearer.

So, shall we open the door, and set forth down that corridor, inspiring a coalition of the willing to follow us?

Footnote 1: The contents of this article came together in my mind as I attended four separate events over the last two weeks (listed in this newsletter) on various aspects of the subject of safe superintelligence. I owe many thanks to everyone who challenged my thinking at these events!

Footnote 2: If any reader is inclined to dismiss the entire subject of potential risks from ASI with a handwave – so that they would not be interested in any of the four choices this article reviews – I urge that reader to review the questions and answers in this excellent article by Yoshua Bengio: Reasoning through arguments against taking AI safety seriously.

12 November 2024

The Narrow Path – questions and answers

Filed under: AGI, risks — Tags: , — David Wood @ 9:53 am

On Saturday, I had the pleasure to chair a webinar on the subject “The Narrow Path: The big picture”.

This involved a deep dive into aspects of two recently published documents:

The five panellists – who all made lots of thoughtful comments – were:

  • Chris Scammell, the COO of Conjecture and one of the principal authors of The Compendium
  • Andrea Miotti, the Executive Director of Control AI and the lead author of A Narrow Path
  • Robert Whitfield, Chair of Trustees of One World Trust
  • Mariana Todorova, a core member of the team in the Millennium Project studying scenarios for the transition between AI and AGI
  • Daniel Faggella, the Founder and Head of Research of Emerj

For your convenience, here’s a recording of the event:

It was a super discussion, but it fell short in one aspect from the objectives I had in mind for the meeting. Namely, the conversation between the panellists was so rich that we failed to find sufficient time to address the many important questions which audience members had submitted in Zoom’s Q&A window.

Accordingly, I am posting these questions at the end of this blogpost, along with potential answers to some of them.

Out of caution for people’s privacy, I’ve not given the names of the people who asked each question, but I will happily edit the post to include these names on an individual basis as requested.

I also expect to come back and edit this post whenever someone proposes a good answer to one of the questions.

(When I edit the post, I’ll update this version number tracker. Currently this is version 1.2 of the post.)

The draft answers are by me (“DW”) except where otherwise indicated.

Aside 1:

For those in or near to London on Thursday evening (14th October), there’s another chance to continue the discussion about if/how to try to pause or control the development of increasingly powerful AI.

This will be at an event in London’s Newspeak House. Click here for more details.

Aside 2:

I recently came across a powerful short video that provides a very different perspective on many issues concerning the safety of AI superintelligence. It starts slowly, and at first I was unsure what to think about it. But it builds to a striking conclusion:

And now, on to the questions from Saturday’s event…

1. Strict secure environments?

Some biomedical research is governed by military in prevent major incident or fall into wrong hands, could you envision an AI experiments under strict secure environments?

Answer (DW): That is indeed envisioned, but with two provisos:

  1. Sadly, there is a long history of leaks from supposedly biosecure laboratories
  2. Some AIs may be so powerful that they will find ways (psychological and/or physical) of escaping from any confinement.

Accordingly, it will be better to forbid certain kinds of experiment altogether, until such time (if ever) it becomes clear that the outcomes will be safe.

2. How will AI view living beings?

How would AI view living beings’ resilience, perseverance, and thriving? Could you explore please, thank you.

3. AIs created with different ideologies?

AGI created in China and perhaps even in North Korea is going to have ideology and supremacy of the regime and ideology over human rights, will say North Korean AGI find way into our systems, whether human induced or at AGI autonomy?

Answer (DW): Indeed, when people proudly say that they, personally, know how to create safe superintelligence, so the world has no need to worry about damage from superintelligence, that entirely presupposes, recklessly, that no-one else will build (perhaps first) an unsafe superintelligence.

So, this issue cannot be tackled at an individual level. It requires global level coordination.

Happily, despite differences in ideological output, governments throughout the world are increasingly sharing the view that superintelligence may spin out of control and, therefore, that such development needs careful control. For example, the Chinese government fully accepts that principle.

4. A single AGI or many?

I just saw a Sam Altman interview where he indicated expecting OpenAI to achieve AGI in 2025. I would expect others are close as well. It seems there will be multiple AGIs in close proximity. Given open source systems are nearly equal to private developers, why would we think that the first to get AGI will rule the world?

Answer (DW): This comes down to the question of whether the first AGI that emerges will gain a decisive quick advantage – whether it will be a “winner takes all” scenario.

As an example, consider the fertilisation of an egg (ovum). Large numbers of sperm may be within a short distance from that goal, but as soon as the first sperm reaches the target, the egg undergoes a sharp transition, and it’s game over for all the other sperm.

5. National AGI licencing systems?

What are the requirements for national AGI licencing systems and global governance coordination among national systems?

Answer (DW): The Narrow Path document has some extensive proposals on this topic.

6. AGI as the solution to existential risk?

Suppose we limit the intelligence of the developing GenAI apps because they might be leveraged by bad actors in a way that triggers an existential risk scenario for humans.

In doing that, wouldn’t we also be limiting their ability to help us resolve existential risk situations we already face, e.g., climate change?

Answer (DW): What needs to be promoted is the possibility of narrow AI making decisive contributions to the solution of these other existential risks.

7. A “ceiling” to the capability of AI?

Are you 100% certain that self-improving AI won’t reach a “ceiling” of capability. After all, it only has human knowledge and internet slop to learn from?

Answer (by Chris Scammel): On data limitations, people sometimes argue that we will run into these at a boundary. It could be that we don’t have data currently to train AI. But we can make more, and so can AI! (top paid dataset labellers getting paid like $700/hr.)

If the question is about intelligence/capability.

One intuition pump: chess AI has gone vastly beyond human skill.

Another: humanity is vastly smarter than a single human.

Another: humans / humanity is vastly smarter than we were thousands of years ago (at the very least, much much more capable).

What we consider “intelligence” to be is but a small window of what capabilities could be, so to believe that there is a ceiling near human level seems wrong from the evidence.

That there is a ceiling at all… deep philosophical question. Is there a ceiling to human intelligence? Humanity’s? Is this different from what an AI is able to achieve? All of these are uncertain.

But we shouldn’t expect a ceiling to keep us safe.

8. Abuse of behavioural models?

Social Media companies are holding extensive volumes of information. I am concerned about not only online disinformation but also the modification of manipulation of behaviour, all the way to cognitive impairment, including when governments are involved. We adults have the ability to anticipate several decades down the road. How could behavioural models be abused or weaponized in the future?

9. Extinction scenarios?

What do you think are the top scenarios how AI can cause the extinction of humanity?

Answer (DW): A good starting point is the research article An Overview of Catastrophic AI Risks.

See also my own presentation Assessing the risks of AI catastrophe, or my book The Singularity Principles (whose entire content is available online free-of-charge).

10. Income gap?

Will artificial superintelligence be able to help humanity close the income gap between rich and poor countries?

11. Using viruses to disrupt rogue AI systems?

Perhaps a silly question from a non-techie – are there any indications of viruses that could disrupt rogue ai systems?

12. Additional threats is AI becomes conscious?

Whichever is true, whether consciousness is biological phenomena, or something more spiritual, in what way would consciousness for AI not be a huge threat. If you give the machine real feelings, how could you possibly hope to control its alignment? Additionally, what would happen to its rights vs human rights. My feeling is not nearly enough thought has gone into this to risk stumbling across conscious AI at this stage. Everything should be done to avoid it.

Answer (DW): I agree! See my article Conscious AI: Five Options for some considerations. Also keep an eye on forthcoming announcements from the recently launched startup Conscium.

13. The research of Mark Solms?

Regarding Conscious AGI apps…

Mark Solms, author of the Hidden Spring, has argued that consciousness is not about intelligence but, instead, is rooted in feelings, physically located in the brainstem.

His view makes sense to me.

As I understand it, he’s involved in experiments/studies around the implications of this view of consciousness for AGI.

Thoughts about this?

14. Multi-dimensional intelligence?

Thanks Mariana for raising issues of the multi-dimensions in which human consciousness appears to operate, compared to AGI – is there an argument that communities need to race to develop our other levels of consciousness, as potentially our only defence against the 1-dimensional AGI?

15. The views of Eric Schmidt and other accelerationists?

The question I asked above, “AGI as the solution to existential risk?”, looms large in the minds of the accelerationist community.

Eric Schmidt has explicitly said that he’s an accelerationist because that’s something like the fastest and effective way to address climate change…

That view is extremely widespread and must be explicitly addressed for the imitations discussed in this meeting to be made reality.

16. Need to work on Phase 1 concurrently with Phase 0?

What is described in A Narrow Path is phases 0, 1 and 2 in sequential order but three concurrent objectives. While the risk of loss of control is surely the highest, doesn’t the risk of concentration of power need to be largely tackled concurrently? Otherwise by the time phase 0 or 1 are completed a global dystopia will have been durably entrenched with one or two states or persons ruling the world for years, decades or more.

Answer (by Robert Whitfield): You make a valid point. I am not sure that Narrow Path says that there can be no overlap. Certainly you can start thinking about and working on Phase 1 before you have completed Phase 0 – but the basic concept is sound: the initial priority is to pause the further development towards AGI. Once that has been secured, it is possible to focus on bring about longer term stability.

17. The role of a veto?

The Narrow Path describes the governance for Phase 1 (lasting 20 years) to be: “The Executive Board, analogous to the UN Security Council, would consist of representatives of major member states and supranational organizations, which would all be permanent members with vetoes on decisions taken by the Executive Board, as well as non-permanent representatives elected by a two-thirds majority of the Council”. But wouldn’t such a veto make it impossible to ensure wide enough compliance and marginalize economically all other states?

As a comparison, back in 1946, it was the veto that prevented the Baruch Plan or the Gromyko Plans to be approved, and lead us to a huge gamble with nuclear technology.

Answer (by Robert Whitfield): It depends in part upon how long it takes to achieve Phase 0. As discussed in the meeting, completing Phase 0 is extremely urgent. If this is NOT achieved, then you can start to talk about dystopia. But if it is achieved, Governments can stand up to the Big Tech companies and address the concentration of power, which would be difficult but not dystopic.

There is a very strong argument that an agreement could be achieved without removing vetoes much more quickly than one that does remove vetoes. This points to a two-phase Treaty:

  • An initial Treaty, sufficient for the purposes of Phase 0
  • A more robust Baruch style agreement for securing the long term.

18. Who chooses the guardians?

Who would be the people in charge of these groups of guardians or protectors against uncontrolled AI? How would they be chosen? Would they be publicly known?

6 November 2024

A bump on the road – but perhaps only a bump

Filed under: AGI, politics, risks — Tags: , , , — David Wood @ 3:56 pm

How will the return of Donald Trump to the US White House change humanity’s path toward safe transformative AI and sustainable superabundance?

Of course, the new US regime will make all kinds of things different. But at the macro level, arguably nothing fundamental changes. The tasks remain the same, for what engaged citizens can and should be doing.

At that macro level, the path toward safe sustainable superabundance runs roughly as follows. Powerful leaders, all around the world, need to appreciate that:

  1. For each of them, it is in their mutual self-interest to constrain the development and deployment of what could become catastrophically dangerous AI superintelligence
  2. The economic and humanitarian benefits that they each hope could be delivered by advanced AI, can in fact be delivered by AI which is restricted from having features of general intelligence; that is, utility AI is all that we need
  3. There are policy measures which can be adopted, around the world, to prevent the development and deployment of catastrophically dangerous AI superintelligence – for example, measures to control the spread and use of vast computing resources
  4. There are measures of monitoring and auditing which can also be adopted, around the world, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to the policies
  5. All of the above can be achieved without any damaging loss of the leaders’ own sovereignty: these leaders can remain masters within their own realms, provided that the above basic AI safety framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI safety framework, as more insight is obtained; in other words, this system is agile rather than static
  7. Even though the above safety framework is yet to be properly developed and agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources.

The above agreements necessarily need to include politicians of very different outlooks on the world. But similar to the negotiations over other global threats – nuclear proliferation, bioweapons, gross damage to the environment – politicians can reach across vast philosophical or ideological gulfs to forge agreement when it really matters.

That’s especially the case when the threat of a bigger shared “enemy”, so to speak, is increasingly evident.

AI superintelligence is not yet sitting at the table with global political leaders. But it will soon become clear that human politicians (as well as human leaders in other walks of life) are going to lose understanding, and lose control, of the AI systems being developed by corporations and other organisations that are sprinting at full speed.

However, as with responses to other global threats, there’s a collective action problem. Who is going to be first to make the necessary agreements, to sign up to them, and to place the AI development and deployment systems within their realms under the remote supervision of the new AI safety framework?

There are plenty of countries where the leaders may say: My country is ready to join that coalition. But unless these are the countries which control the resources that will be used to develop and deploy the potentially catastrophic AI superintelligence systems, such gestures have little utility.

To paraphrase Benito Mussolini, it’s not sufficient for the sparrows to request peace and calm: the eagles need to wholeheartedly join in too.

Thus, the agreement needs to start with the US and with China, and to extend rapidly to include the likes of Japan, the EU, Russia, Saudi Arabia, Israel, India, the UK, and both South and North Korea.

Some of these countries will no doubt initially resist making any such agreement. That’s where two problems need to be solved:

  • Ensuring the leaders in each country understand the arguments for points 1 through 7 listed above – starting with point 1 (the one that is most essential, to focus minds)
  • Setting in motion at least the initial group of signatories.

The fact that it is Donald Trump who will be holding the reins of power in Washington DC, rather than Joe Biden or Kamala Harris, introduces its own new set of complications. However, the fundamentals, as I have sketched the above, remain the same.

The key tasks for AI safety activists, therefore, remain:

  • Deepening public understanding of points 1 to 7 above
  • Where there are gaps in the details of these points, ensuring that sufficient research takes place to address these gaps
  • Building bridges to powerful leaders, everywhere, regardless of the political philosophies of these leaders, and finding ways to gain their support – so that they, in turn, can become catalysts for the next stage of global education.

Not democracy’s finest hour

Filed under: books, culture, philosophy, politics, sustainability — Tags: , , — David Wood @ 10:04 am

November 5th and 6th, 2024, is not democracy’s finest hour.

Whether you are a supporter of Donald Trump or a supporter of Kamala Harris, you cannot be happy at the calibre of the public discussion during the recent US presidential race.

That discussion has been dominated by distortions, by polarisation, by superficialities, and by misleading oversimplifications.

Any nation that is unable to have an honest, open, conciliatory discussion about the key issues of social wellbeing, is likely to be devastated by the strong winds of disruption that are imminent. It is likely to be overwhelmed by literal floods and fires arising from climate change. And likely to become easy pickings for its enemies both at home and overseas.

To quote from the Sermon on the Mount in the gospel according to Matthew, a house built on sand cannot stand. A house built upon superficiality will fall.

For a sustainable future, we need more solid foundations. We need a fuller shared understanding of the basis of human flourishing – an understanding that respects multiple perspectives and multiple lived experiences. We need a deeper societal agreement about the things that matter most. We need bonds of mutual support, that enable each of us to become better versions of ourselves.

To call a future a sustainable future doesn’t mean there will be no change. Change is fundamental to human life, and we must anticipate that a great deal more change lies ahead – changes in technology, demographics, ideas, standards, and narratives.

Similarly, a sustainable future doesn’t mean there will be no disagreements. But it requires people not to be constantly disagreeable, or contemptuous.

To help us build a sustainable future that can thrive on change rather than collapsing, and where disagreements are constructive rather than contemptuous, whither can we turn?

As someone who spent 25 years immersed in the technology industry, my first instinct is to suggest that we should turn to technology. However, the USA is already awash with technology. The companies that are at the forefront of the global race toward AI superintelligent are headquartered in the US. That surfeit of technology has by no means translated into better democracy.

My next instinct, as someone with a strong personal interest in philosophy, is to suggest that we need to encourage more people to appreciate the insights of that field. Instead of being swept along by rip-roaring tides of “we can” and “we want to”, we need to spend more time pondering “we should” – more time considering alternative scenarios for how the future might unfold.

But even before people are willing to look at alternative possibilities, there needs to be a softening of the spirit.

So my biggest personal takeaway, overnight, is that I should stop looking with hostility or contempt at the vast numbers of people who have reached different decisions, from me, about (for example) which person should become the President of the USA. For that reason, I have resolved to spend some time over the next few days listening to the audio of the 2019 book by Arthur C. Brooks, Love Your Enemies: How Decent People Can Save America from the Culture of Contempt.

(The book has the word ‘America’ in its title, but I feel sure its messages apply elsewhere in the world too.)

Earlier this morning, the following sentence from the book’s opening chapter struck me hard: “We need national healing every bit as much as economic growth”.

That’s a good start!

(I prepared the remarks above to share with attendees at a private roundtable conversation this evening, arranged long before the US election took place – a conversation with the topic “How can we harness emerging technology to improve our democracies?”)

26 September 2024

By far the quickest I’ve ever produced a podcast

Filed under: AGI, podcast — Tags: — David Wood @ 12:50 pm

Wow.

I’ve just experienced what people call “a ChatGPT moment”.

Not with ChatGPT, but with the NotebookLM tool from Google Research.

I’d heard that NotebookLM now has functionality to create an audio conversation between two AI characters who sound remarkably human. A couple of days ago, fellow futurist Tony Czarnecki drew my attention to how good this functionality is, as shown by an experiment on the Sustensis website. (See the section on the page entitled “Two AI-generated journalists discuss in their own podcast Tony Czarnecki’s article ‘Taking Control over AI before it starts controlling us’”.)

I thought I should try a similar experiment myself.

Here’s the outcome.

The blogpost discussed by these two AI characters is this one. That was the only input I gave to the AI.

It was very straightforward to create:

  1. Go to the NotebookLM site
  2. Click “New notebook”
  3. Click on the Link area and paste in the URL of my blogpost
  4. Click on Generate in the Audio overview section
  5. Wait a couple of minutes

Voila! A nine-minute audio conversation.

The conversation takes some ideas from my blogpost, mixes in related material from (I presume) its own knowledgebase, and adds extra flavour with some friendly humour and homespun analogies.

The resulting conversation sounds extremely human. There’s a fine rapport between the two hosts.

The content, to my mind, is questionable in a few places. But the AI characters get at least as much right as most pairs of real-life journalists would.

I’m looking forward to become more familiar with what NotebookLM can do!

PS While I was waiting for the audio WAV file to be generated, I took a quick look at the summary of my blogpost which NotebookLM had already created. Yes, that’s pretty good too!

This essay by David Wood discusses the possibility of achieving longevity escape velocity (LEV) by 2040, a scenario where biomedical interventions reverse the aging process, making individuals biologically younger and healthier. Wood argues that while the complexity of aging is acknowledged, there is sufficient understanding of the damage repair mechanisms involved, allowing for targeted interventions to combat aging. He outlines the potential roadblocks to achieving LEV, including funding constraints, societal resistance, and lack of collaboration within the longevity research community. Wood proposes “breakthrough initiatives” to overcome these challenges, including biomedical and financial reengineering, narrative reengineering, and community reengineering. Despite acknowledging the challenges, Wood concludes that the probability of reaching LEV by 2040 is significant, exceeding 25% but below 50%. He encourages individuals to play an active role in shaping the future of longevity research.

12 September 2024

Are fears of technological unemployment misdirected?

Filed under: culture, education, Events, politics — Tags: , , , , — David Wood @ 8:11 pm

The anxiety is very real. Globally, around 70% of the younger generation are worried about the impact AI will have on their job security.

These remarks were made by Ana Kreacic, Chief Knowledge Officer of management consultant firm Oliver Wyman, on the second day of the GAIN summit which I am attending in Riyadh.

(Click here for a recording of the live stream of all the Day 2 sessions from the main GAIN summit auditorium.)

Kreacic was part of an absorbing session entitled “The Future of Work: AI’s Role in Reskilling and Innovation”.

Kreacic went to share some more statistics. In Saudi Arabia, it is 69% of the younger generation who are worried about the impact of AI on their job security. Evidently, that figure is close to the global average. However, there’s a more significant difference when comparing the proportion of older workers who are concerned about the impact of AI on job security. Globally, that figure is 40%, since (according to Oliver Wyman), “senior employees may feel more secure in their careers because they believe AI will have less impact on higher-level employment”. But the same figure in Saudi Arabia is 59%. In other words, apprehension about job security exists more widely in the Saudi workforce.

However, the panellists were keen to dispel that apprehension. With appropriate actions, there would, they said, be plenty of opportunities for people to continue having good jobs. That’s despite an increasing adoption of AI within companies. Yes, jobs will change. But employees will be able to reskill, so that their careers can flourish.

According to these panellists, it is important to recognise that AI can assist the process of reskilling. Kreacic explained the ways in which AI is enabling personalized training. Rather than employees having to attend standard courses that only loosely match their individual requirements, AIs can provide training materials uniquely tailored (“hyper-targeted”) to each employee. Employees find these courses much more enjoyable, with high rates of knowledge retention long after the course has concluded. Moreover, employees will be able to tap into online training programs to acquire skills that aren’t core to their current roles, but which will be important in the positions into which they hope to move in the near future.

Another panellist, Abdulrahman Alhaqbani, a general manager at leading Saudi information services company stc, described how learning in the real world often involves a relationship between two humans – a mentor and a mentee. However, AI can assist here too, by identifying and recommending good mentor-mentee pairs, in a way similar to how various AI systems suggest other kinds of human relationships.

Nadeem Mazen, CEO of the full-service creative agency Nimblebot, foresaw AI enabling fast career progression of a different kind. Namely, a small group of people – perhaps just 4 or 10 people – with a new idea, some ambition and some curiosity, and with the help of AI, “could take on enormous incumbent brands”. Mazen said that we are entering “the heyday of the small firm”.

Najwa Alghamdi, Analytics Innovation Director at stc, spoke about “injecting into the current workforce digital co-workers” that handle the routine aspects of work tasks, resulting in huge boosts in productivity and efficiency. She said that employees were initially wary of these digital co-workers, but once they saw how well they worked in practice, the employees soon accepted the concept. Indeed, they started to propose additional parts of their existing work that could be handled by new co-created digital co-workers. One reason for the acceptance, Alghamdi emphasised, was that the co-workers had been designed to adhere to the principles of explainable AI.

Ramez El-Serafy, CEO of Flat6Labs, a leading seed and early stage venture capital firm operating in the Middle East and North Africa, described how they offer a learning platform to young founders and entrepreneurs. When they started using AI to create the content for these training programmes, they found it much more efficient and more readily adaptable. By using AI, the content of the courses can be updated as often as once each week.

Maan Al-Mulla, Director of Digital Transformation at Saudi Aramco, described the digital upskilling programmes that his team supervises. For example, the company has launched a “citizen development programme, that enables non-technical employees to develop their own applications and their own solutions using low-code and no-code platforms that are powered by AI”. As a result, the Aramco workforce is “more adaptive to any work change”. In summary, by embracing digital solutions and AI, the result is a smoother and faster transition.

A transition to what? Speakers on this panel, as well as on other panels throughout the event, seemed to share an assessment of the kinds of tasks which cannot be handled by AI:

  • Creativity
  • Critical thinking
  • Emotional intelligence
  • Strategic planning
  • Ensuring customer satisfaction.

Accordingly, worries about employees being pushed out of the workforce by increasing automation should be resisted. Instead of automation being a threat, it should be wisely embraced, and will result in lots of innovative work experiences.

Right?

But my analysis now contains two twists. The first of these twists was made by panellists throughout the day, but the second was raised less often in public discussions. Let’s take these twists one at a time.

Don’t fear automation; fear a stagnant culture

One problem with learning new types of skill is that we go back to being a beginner, and we are prone to make mistakes. For a while, we may become less productive. Our bosses may see us as, for a while, performing less well than in the past. As for ourselves, we may be unsure about how our identity is changing: we used to be such-and-such a kind of employee, but now we are becoming something quite different. We may feel awkward and unsure.

In other words, as well as the technical aspects of a new skill, needed for a change of work condition, there are human aspects. Without strong emotional intelligence, we may shirk from the change. Without supportive management, we may feel pressured to give up the transition. Without appropriate metrics for our contributions in the workforce, the pressures will magnify to resist the change.

That’s why corporate culture plays a large part in whether reskilling is successful. Or, if we are unlucky enough to have unsympathetic bosses or colleagues at work, we need to find emotional support from other parts of our network of relationships – such as family, friends, or mentors.

On these points, Ana Kreacic of Oliver Wyman had a positive message to share: If a company can create the right culture – where it’s OK for people to test things out, spend some time learning, and fail several times in the process – then that company will drive innovation, with employees being willing to experiment.

This is a subject I often address in my own presentations to clients, where I warn about failures, not just of individual career transition, but of companies that are unable to respond to disruptive trends, or indeed of whole nations whose prior culture, processes, and infrastructure are unsuited to changed global conditions. For example, here’s one of the slides I sometimes use.

The takeaway at this point is that employees should stop worrying about AI and automation, but should worry instead about whether the culture in which they operate will be conducive to the kinds of adaptions that AI and automation will necessitate.

But as I suggested above, that’s not yet the end of this analysis. There’s one more twist.

Don’t fear automation; fear bad politics

I can’t remember which panellist said it – probably in a session in one of the breakout rooms around the venue – but it was an important point. IT workers will soon become increasingly unionised.

Why so? Because, despite the positive thoughts offered above, IT workers will soon find their entire jobs under increasing pressure from automation.

That’s not from today’s automation – the “AI Now” part of the theme of the GAIN summit – but from the AI of a few years in the future – “AI Next”.

Look again at the kinds of task listed above, which supposedly cannot be handled by AI. In each case, there is already evidence that AI is well on the path to performing these tasks just as well as humans:

  • Creativity is often displayed by generative AI, although at present it needs good prompts from human operators
  • Critical thinking can arise in large language models in response to instructions to “think step by step”, and is likely to be a standard feature in forthcoming AI systems
  • Artificial emotional intelligence has a vast field of research all to itself – often called “affective computing”
  • Strategic planning is shown by the ability of various AIs to outplay humans in increasingly complex games of strategy
  • Customer satisfaction has many aspects to it, and arises by a combination of all the above traits.

Indeed, toward the end of the panel mentioned above, Nadeem Mazen of Nimblebot remarked that chatbots like ChatGPT can already provide employees with some of the psychological assistance that we previously looked to human colleagues to provide. (“It’s almost spooky”, Mazen mused.)

In short, even the work tasks which are claimed to lie outside the reach of today’s robots and AIs, are likely to come within the expanded reach of automation in the relatively near future.

Perhaps you don’t believe my claim here. In that case, I ask you to dip into my recent essay “Six possible responses as the Economic Singularity approaches”, where the section entitled “Disbelief” offers more analysis in support of my claim.

That same essay also makes the case that the best response to this forthcoming wave of job losses isn’t to resist them, but is to accelerate an enhancement of the way the entire economy is run. This enhancement will involve a redistribution of the bountiful fruits of automation so that everyone in society benefits.

The takeaway at this point of my argument is that employees should stop worrying about AI and automation, but should worry instead about whether the political setup in their country is preparing for this kind of significant redistribution as part of a revised social contract.

It’s for this reason that the slide in my presentations on “The 7 most important characteristics for success over the next 3-5 years” gives special prominence to the skill area of “Politics”:

  • Building and managing coalitions
  • Agile regulations & incentives
  • Revised social contract

Happily, in my discussions with senior AI leaders in Saudi Arabia over the last three days, it’s my impression that they already have a good understanding of these points. I’ll say more about that in a forthcoming article.

Footnote: Patrick Linnenbank, Partner at management consulting firm Arthur D. Little, deserves a big shout-out for his skill in moderating the above panel discussion.

10 September 2024

A disruption in the international AI landscape?

Filed under: AGI, Events, music — Tags: , , , , , , , — David Wood @ 10:29 pm

In the years ahead, which countries will have the biggest impact on the development and deployment of AI?

The two most common answers to that question are the USA and China. Sometimes people put in a word for the EU or India – or the UK, Canada, Israel, or Korea.

Well, in one possible future scenario, another country may power forward to join this group of the biggest AI influencers – and, in the process, might disrupt the international AI landscape.

The country in question is the one where I’ve been for the last 24 hours, and where I will stay for two more days, namely, Saudi Arabia.

I’m attending the GAIN summit in Riyadh. The ‘GAI’ of “GAIN” stands for Global AI. The ‘N’ has a triple meaning: Now, Next, Never.

To quote from the event website:

  • AI Now: How are the AI leaders of the world today deploying, scaling, and leveraging the technology?
  • AI Next: What does the future of AI look like and how will it impact people, businesses and government organizations?
  • AI Never: How do we ensure that the future we design is one that we want to live in and not a dystopian sci-fi?

The ‘A’ in GAIN could plausibly also stand for “ambition”, as the organizers have high ambitions. To quote again from the event website:

The Global AI Summit is the leading platform for advancing the global discussion on AI, where visionary experts, academics, corporates, and policymakers converge from every part of the world to shape the future of artificial intelligence for the good of humanity.

Of course, it’s one thing for a country to express big ambitions to play a leading role in the future of AI. It’s quite another thing to make significant steps toward that ambition. Therefore, as I approached the event, I didn’t know what to think.

Indeed, it was my first visit to Saudi Arabia. I found myself reminded of my first visit to China, way back in October 2002. On that occasion, I was representing Symbian, at what was described as the first UK-China CEO Forum. I recently came across a photo of that event – where my hair was a brighter shade of red than in more recent times!

In both cases – my first visit to China, and my first visit to Saudi Arabia – I was unsure what to expect. It turned out that Shanghai was a bustling metropolis, with gleaming shops and a lively entrepreneurial spirit. The Chinese people I met were dressed nothing like the Chairman Mao suits that I had remembered reading about in my schooldays, and were impressively knowledgeable about technology and business. That visit was to be the first of many I would make in the following years, as Chinese companies steadily became more powerful players on the world stage.

That was 2002. What about my experience in the last 24 hours, in 2024, in Riyadh?

Part of the answer lies in numbers:

  • Over 450 speakers, spread over multiple parallel tracks
  • The speakers represented more than 100 different countries
  • Over 32,000 attendees expected during the three days.

These numbers are all significant steps up from the corresponding numbers from the two previous occasions this summit has been held, in 2020 and 2022.

The speakers include a host of prominent leaders from business and technology worldwide. Some examples:

  • Julie Sweet, the Chair and CEO of Accenture
  • Cristiano Amon, the President and CEO of Qualcomm
  • Marc Raibert, the Founder of Boston Dynamics
  • Martin Kon, the President and COO of Cohere
  • Brian Behlendorf, the Chief AI Strategist of the Linux Foundation
  • Nick Studer, the President and CEO of Oliver Wyman Group
  • Matthew Kropp, the CTO and Managing Director of Boston Consulting Group
  • Alan Qi, the President of Huawei Cloud
  • Yuwon Kim, the CEO of Naver Cloud
  • Caroline Yap, the Global Managing Director of Google Cloud.

Multiple segments of society in Saudi Arabia were well represented too – including an impressive number of adept, articulate women leaders, who had some fascinating pieces of advice.

With so many speakers, it is perhaps inevitable that some speeches fell flat – especially several of the ones about the governance of AI, where the conversations seemed to be going round in circles, with little appreciation of what I see as the risks of catastrophic harm if next generation AI is mishandled. However, the technical talks were generally compelling.

I particularly liked the talks by Andrew Feldman, Co-founder and CEO of Cerebras Systems, and Jonathan Ross, Founder and CEO of Groq. These two companies each position themselves as disruptors of the GPU market, and, hence, as potentially overtaking Nvidia. Instead of GPUs, or the TPUs developed by Google, they have created LPUs (Language Processing Units) in the case of Groq, and waferscale AI chips in the case of Cerebras. Both companies claim notable improvements in speed over previous AI chip configurations. I heard the phrase “like ChatGPT but insanely fast”.

Both Cerebras and Groq emphasized close partnerships with Saudi Arabia. Andrew Feldman of Cerebras described a special collaboration with KAUST (King Abdullah University of Science and Technology). And Jonathan Ross of Groq appeared on stage alongside Tareq Amin, the CEO of Aramco Digital. Ross gave three reasons for their company investing strongly in the country:

  • The abundance of energy resources in the country
  • The good business environment, that encourages and supports this kind of partnership
  • The geographical location, close to three different continents, so that the resulting high-performance AI cluster could serve the needs of up to four billion people.

It was while listening to these two talks that the Saudi ambition to become a global leader in AI started to become more credible in my mind.

I had already seen the strong enthusiasm in large numbers of Saudi delegates at the event. They were avidly leaning forward in their seats, to capture as much as possible of the advice being provided on the various stages. It seems that the country is aware of the need to transition away from reliance on the oil industry, and instead to actively participate in shaping the global AI marketplace.

There were many other talks and panels which left me with new ideas to consider. For example, I wished that Marc Raibert, the Founder of Boston Dynamics, could have had more time to develop his fascinating ideas further. He made the case that true intelligence involves an interactive combination of cognitive intelligence (“what’s going on in our heads”) and athletic intelligence (“what’s going on in our bodies”). That explanation formed the backdrop for the progress made by Boston Dynamics over the years, with robots such as Spot (commercially significant “today”), Stretch (“tomorrow”), and Atlas (“future”). In addition to his role at Boston Dynamics, Raibert is also the Founder and Executive Director of the AI Institute, whose website proudly reports that “The AI Institute aims to solve the most important and fundamental problems in robotics and AI”. As I said, I wish he had more time to continue talking about that work.

Earlier in the day, I watched a fascinating six-way round-table discussion on the subject “Hallucinations and Confabulations: when chatbots go rogue”, with speakers from Kearney, Mozn, Saudi Aramco, Vectara, KAUST, and INF, who each had long careers as experts in various aspects of AI. The discussion went on for 90 minutes, but I would have been happy for it to continue longer, as it had lots of good-spirited clashes of ideas about the strengths and weaknesses of large language models, and possible approaches to add fact-checking components into the AI systems of the near future. One of the speakers, Amr Awadallah of Vectara, boldly predicted that AGI would exist by 2028. Part of his reasoning was his argument that ongoing improvements in RAG (Retrieval Augmented Generation) were steadily reducing the prevalence of hallucinations in the content being generated.

That kind of awareness of potential dramatic improvements in AI capability by 2028 was, sadly (to my mind) missing from what many of the speakers in other sessions were assuming. These other speakers were focused, in effect, on the “Now” of AI, and didn’t foresee many real changes for “AI Next” any time soon. Frankly, if they keep thinking that way, they’re likely to be disrupted themselves. Anyway, this issue is something I hope will feature again in the sessions on days two and three of this year’s GAIN. I look forward to these days with great interest.

I’ll end at the beginning. The day started with an artistic performance, symbolizing the sequential creation of ANI (Artificial Narrow Intelligence), AGI (Artificial General Intelligence), and then ASI (Artificial Super Intelligence). The narrator offered a positive vision of a beneficial relationship of humanity and superintelligence: “There will be no more confusion, and a golden age of progress will flourish, where men and machines, united by an unprecedented alliance, will walk together toward a destiny of glory and happiness”.

Having come to life, the ASI spoke to a young boy, who was a representative of humanity, saying “I am your new avatar, and I will become your best friend”.

In response, the boy started singing what was said to be his favourite song. The music was increasingly stirring and the singing increasingly dramatic. Given the location, Riyadh, I could hardly believe what I was hearing:

Imagine there’s no heaven
It’s easy if you try
No hell below us
Above us, only sky
Imagine all the people
Livin’ for today
Ah

Imagine no possessions
I wonder if you can
No need for greed or hunger
A brotherhood of man

Imagine all the people
Sharing all the world

You may say I’m a dreamer
But I’m not the only one
I hope someday you’ll join us
And the world will live as one

The words and music of John Lennon’s “Imagine” have graced numerous stages over the decades, but somehow, for me, this was particularly evocative.

« Newer Posts — Older Posts »

Blog at WordPress.com.