dw2

12 November 2024

The Narrow Path – questions and answers

Filed under: AGI, risks — Tags: , — David Wood @ 9:53 am

On Saturday, I had the pleasure to chair a webinar on the subject “The Narrow Path: The big picture”.

This involved a deep dive into aspects of two recently published documents:

The five panellists – who all made lots of thoughtful comments – were:

  • Chris Scammell, the COO of Conjecture and one of the principal authors of The Compendium
  • Andrea Miotti, the Executive Director of Control AI and the lead author of A Narrow Path
  • Robert Whitfield, Chair of Trustees of One World Trust
  • Mariana Todorova, a core member of the team in the Millennium Project studying scenarios for the transition between AI and AGI
  • Daniel Faggella, the Founder and Head of Research of Emerj

For your convenience, here’s a recording of the event:

It was a super discussion, but it fell short in one aspect from the objectives I had in mind for the meeting. Namely, the conversation between the panellists was so rich that we failed to find sufficient time to address the many important questions which audience members had submitted in Zoom’s Q&A window.

Accordingly, I am posting these questions at the end of this blogpost, along with potential answers to some of them.

Out of caution for people’s privacy, I’ve not given the names of the people who asked each question, but I will happily edit the post to include these names on an individual basis as requested.

I also expect to come back and edit this post whenever someone proposes a good answer to one of the questions.

(When I edit the post, I’ll update this version number tracker. Currently this is version 1.2 of the post.)

The draft answers are by me (“DW”) except where otherwise indicated.

Aside 1:

For those in or near to London on Thursday evening (14th October), there’s another chance to continue the discussion about if/how to try to pause or control the development of increasingly powerful AI.

This will be at an event in London’s Newspeak House. Click here for more details.

Aside 2:

I recently came across a powerful short video that provides a very different perspective on many issues concerning the safety of AI superintelligence. It starts slowly, and at first I was unsure what to think about it. But it builds to a striking conclusion:

And now, on to the questions from Saturday’s event…

1. Strict secure environments?

Some biomedical research is governed by military in prevent major incident or fall into wrong hands, could you envision an AI experiments under strict secure environments?

Answer (DW): That is indeed envisioned, but with two provisos:

  1. Sadly, there is a long history of leaks from supposedly biosecure laboratories
  2. Some AIs may be so powerful that they will find ways (psychological and/or physical) of escaping from any confinement.

Accordingly, it will be better to forbid certain kinds of experiment altogether, until such time (if ever) it becomes clear that the outcomes will be safe.

2. How will AI view living beings?

How would AI view living beings’ resilience, perseverance, and thriving? Could you explore please, thank you.

3. AIs created with different ideologies?

AGI created in China and perhaps even in North Korea is going to have ideology and supremacy of the regime and ideology over human rights, will say North Korean AGI find way into our systems, whether human induced or at AGI autonomy?

Answer (DW): Indeed, when people proudly say that they, personally, know how to create safe superintelligence, so the world has no need to worry about damage from superintelligence, that entirely presupposes, recklessly, that no-one else will build (perhaps first) an unsafe superintelligence.

So, this issue cannot be tackled at an individual level. It requires global level coordination.

Happily, despite differences in ideological output, governments throughout the world are increasingly sharing the view that superintelligence may spin out of control and, therefore, that such development needs careful control. For example, the Chinese government fully accepts that principle.

4. A single AGI or many?

I just saw a Sam Altman interview where he indicated expecting OpenAI to achieve AGI in 2025. I would expect others are close as well. It seems there will be multiple AGIs in close proximity. Given open source systems are nearly equal to private developers, why would we think that the first to get AGI will rule the world?

Answer (DW): This comes down to the question of whether the first AGI that emerges will gain a decisive quick advantage – whether it will be a “winner takes all” scenario.

As an example, consider the fertilisation of an egg (ovum). Large numbers of sperm may be within a short distance from that goal, but as soon as the first sperm reaches the target, the egg undergoes a sharp transition, and it’s game over for all the other sperm.

5. National AGI licencing systems?

What are the requirements for national AGI licencing systems and global governance coordination among national systems?

Answer (DW): The Narrow Path document has some extensive proposals on this topic.

6. AGI as the solution to existential risk?

Suppose we limit the intelligence of the developing GenAI apps because they might be leveraged by bad actors in a way that triggers an existential risk scenario for humans.

In doing that, wouldn’t we also be limiting their ability to help us resolve existential risk situations we already face, e.g., climate change?

Answer (DW): What needs to be promoted is the possibility of narrow AI making decisive contributions to the solution of these other existential risks.

7. A “ceiling” to the capability of AI?

Are you 100% certain that self-improving AI won’t reach a “ceiling” of capability. After all, it only has human knowledge and internet slop to learn from?

Answer (by Chris Scammel): On data limitations, people sometimes argue that we will run into these at a boundary. It could be that we don’t have data currently to train AI. But we can make more, and so can AI! (top paid dataset labellers getting paid like $700/hr.)

If the question is about intelligence/capability.

One intuition pump: chess AI has gone vastly beyond human skill.

Another: humanity is vastly smarter than a single human.

Another: humans / humanity is vastly smarter than we were thousands of years ago (at the very least, much much more capable).

What we consider “intelligence” to be is but a small window of what capabilities could be, so to believe that there is a ceiling near human level seems wrong from the evidence.

That there is a ceiling at all… deep philosophical question. Is there a ceiling to human intelligence? Humanity’s? Is this different from what an AI is able to achieve? All of these are uncertain.

But we shouldn’t expect a ceiling to keep us safe.

8. Abuse of behavioural models?

Social Media companies are holding extensive volumes of information. I am concerned about not only online disinformation but also the modification of manipulation of behaviour, all the way to cognitive impairment, including when governments are involved. We adults have the ability to anticipate several decades down the road. How could behavioural models be abused or weaponized in the future?

9. Extinction scenarios?

What do you think are the top scenarios how AI can cause the extinction of humanity?

Answer (DW): A good starting point is the research article An Overview of Catastrophic AI Risks.

See also my own presentation Assessing the risks of AI catastrophe, or my book The Singularity Principles (whose entire content is available online free-of-charge).

10. Income gap?

Will artificial superintelligence be able to help humanity close the income gap between rich and poor countries?

11. Using viruses to disrupt rogue AI systems?

Perhaps a silly question from a non-techie – are there any indications of viruses that could disrupt rogue ai systems?

12. Additional threats is AI becomes conscious?

Whichever is true, whether consciousness is biological phenomena, or something more spiritual, in what way would consciousness for AI not be a huge threat. If you give the machine real feelings, how could you possibly hope to control its alignment? Additionally, what would happen to its rights vs human rights. My feeling is not nearly enough thought has gone into this to risk stumbling across conscious AI at this stage. Everything should be done to avoid it.

Answer (DW): I agree! See my article Conscious AI: Five Options for some considerations. Also keep an eye on forthcoming announcements from the recently launched startup Conscium.

13. The research of Mark Solms?

Regarding Conscious AGI apps…

Mark Solms, author of the Hidden Spring, has argued that consciousness is not about intelligence but, instead, is rooted in feelings, physically located in the brainstem.

His view makes sense to me.

As I understand it, he’s involved in experiments/studies around the implications of this view of consciousness for AGI.

Thoughts about this?

14. Multi-dimensional intelligence?

Thanks Mariana for raising issues of the multi-dimensions in which human consciousness appears to operate, compared to AGI – is there an argument that communities need to race to develop our other levels of consciousness, as potentially our only defence against the 1-dimensional AGI?

15. The views of Eric Schmidt and other accelerationists?

The question I asked above, “AGI as the solution to existential risk?”, looms large in the minds of the accelerationist community.

Eric Schmidt has explicitly said that he’s an accelerationist because that’s something like the fastest and effective way to address climate change…

That view is extremely widespread and must be explicitly addressed for the imitations discussed in this meeting to be made reality.

16. Need to work on Phase 1 concurrently with Phase 0?

What is described in A Narrow Path is phases 0, 1 and 2 in sequential order but three concurrent objectives. While the risk of loss of control is surely the highest, doesn’t the risk of concentration of power need to be largely tackled concurrently? Otherwise by the time phase 0 or 1 are completed a global dystopia will have been durably entrenched with one or two states or persons ruling the world for years, decades or more.

Answer (by Robert Whitfield): You make a valid point. I am not sure that Narrow Path says that there can be no overlap. Certainly you can start thinking about and working on Phase 1 before you have completed Phase 0 – but the basic concept is sound: the initial priority is to pause the further development towards AGI. Once that has been secured, it is possible to focus on bring about longer term stability.

17. The role of a veto?

The Narrow Path describes the governance for Phase 1 (lasting 20 years) to be: “The Executive Board, analogous to the UN Security Council, would consist of representatives of major member states and supranational organizations, which would all be permanent members with vetoes on decisions taken by the Executive Board, as well as non-permanent representatives elected by a two-thirds majority of the Council”. But wouldn’t such a veto make it impossible to ensure wide enough compliance and marginalize economically all other states?

As a comparison, back in 1946, it was the veto that prevented the Baruch Plan or the Gromyko Plans to be approved, and lead us to a huge gamble with nuclear technology.

Answer (by Robert Whitfield): It depends in part upon how long it takes to achieve Phase 0. As discussed in the meeting, completing Phase 0 is extremely urgent. If this is NOT achieved, then you can start to talk about dystopia. But if it is achieved, Governments can stand up to the Big Tech companies and address the concentration of power, which would be difficult but not dystopic.

There is a very strong argument that an agreement could be achieved without removing vetoes much more quickly than one that does remove vetoes. This points to a two-phase Treaty:

  • An initial Treaty, sufficient for the purposes of Phase 0
  • A more robust Baruch style agreement for securing the long term.

18. Who chooses the guardians?

Who would be the people in charge of these groups of guardians or protectors against uncontrolled AI? How would they be chosen? Would they be publicly known?

6 November 2024

A bump on the road – but perhaps only a bump

Filed under: AGI, politics, risks — Tags: , , , — David Wood @ 3:56 pm

How will the return of Donald Trump to the US White House change humanity’s path toward safe transformative AI and sustainable superabundance?

Of course, the new US regime will make all kinds of things different. But at the macro level, arguably nothing fundamental changes. The tasks remain the same, for what engaged citizens can and should be doing.

At that macro level, the path toward safe sustainable superabundance runs roughly as follows. Powerful leaders, all around the world, need to appreciate that:

  1. For each of them, it is in their mutual self-interest to constrain the development and deployment of what could become catastrophically dangerous AI superintelligence
  2. The economic and humanitarian benefits that they each hope could be delivered by advanced AI, can in fact be delivered by AI which is restricted from having features of general intelligence; that is, utility AI is all that we need
  3. There are policy measures which can be adopted, around the world, to prevent the development and deployment of catastrophically dangerous AI superintelligence – for example, measures to control the spread and use of vast computing resources
  4. There are measures of monitoring and auditing which can also be adopted, around the world, to ensure the strict application of the agreed policy measures – and to prevent malign action by groups or individuals that have, so far, failed to sign up to the policies
  5. All of the above can be achieved without any damaging loss of the leaders’ own sovereignty: these leaders can remain masters within their own realms, provided that the above basic AI safety framework is adopted and maintained
  6. All of the above can be achieved in a way that supports evolutionary changes in the AI safety framework, as more insight is obtained; in other words, this system is agile rather than static
  7. Even though the above safety framework is yet to be properly developed and agreed, there are plenty of ideas for how it can be rapidly developed, so long as that project is given sufficient resources.

The above agreements necessarily need to include politicians of very different outlooks on the world. But similar to the negotiations over other global threats – nuclear proliferation, bioweapons, gross damage to the environment – politicians can reach across vast philosophical or ideological gulfs to forge agreement when it really matters.

That’s especially the case when the threat of a bigger shared “enemy”, so to speak, is increasingly evident.

AI superintelligence is not yet sitting at the table with global political leaders. But it will soon become clear that human politicians (as well as human leaders in other walks of life) are going to lose understanding, and lose control, of the AI systems being developed by corporations and other organisations that are sprinting at full speed.

However, as with responses to other global threats, there’s a collective action problem. Who is going to be first to make the necessary agreements, to sign up to them, and to place the AI development and deployment systems within their realms under the remote supervision of the new AI safety framework?

There are plenty of countries where the leaders may say: My country is ready to join that coalition. But unless these are the countries which control the resources that will be used to develop and deploy the potentially catastrophic AI superintelligence systems, such gestures have little utility.

To paraphrase Benito Mussolini, it’s not sufficient for the sparrows to request peace and calm: the eagles need to wholeheartedly join in too.

Thus, the agreement needs to start with the US and with China, and to extend rapidly to include the likes of Japan, the EU, Russia, Saudi Arabia, Israel, India, the UK, and both South and North Korea.

Some of these countries will no doubt initially resist making any such agreement. That’s where two problems need to be solved:

  • Ensuring the leaders in each country understand the arguments for points 1 through 7 listed above – starting with point 1 (the one that is most essential, to focus minds)
  • Setting in motion at least the initial group of signatories.

The fact that it is Donald Trump who will be holding the reins of power in Washington DC, rather than Joe Biden or Kamala Harris, introduces its own new set of complications. However, the fundamentals, as I have sketched the above, remain the same.

The key tasks for AI safety activists, therefore, remain:

  • Deepening public understanding of points 1 to 7 above
  • Where there are gaps in the details of these points, ensuring that sufficient research takes place to address these gaps
  • Building bridges to powerful leaders, everywhere, regardless of the political philosophies of these leaders, and finding ways to gain their support – so that they, in turn, can become catalysts for the next stage of global education.

Not democracy’s finest hour

Filed under: books, culture, philosophy, politics, sustainability — Tags: , , — David Wood @ 10:04 am

November 5th and 6th, 2024, is not democracy’s finest hour.

Whether you are a supporter of Donald Trump or a supporter of Kamala Harris, you cannot be happy at the calibre of the public discussion during the recent US presidential race.

That discussion has been dominated by distortions, by polarisation, by superficialities, and by misleading oversimplifications.

Any nation that is unable to have an honest, open, conciliatory discussion about the key issues of social wellbeing, is likely to be devastated by the strong winds of disruption that are imminent. It is likely to be overwhelmed by literal floods and fires arising from climate change. And likely to become easy pickings for its enemies both at home and overseas.

To quote from the Sermon on the Mount in the gospel according to Matthew, a house built on sand cannot stand. A house built upon superficiality will fall.

For a sustainable future, we need more solid foundations. We need a fuller shared understanding of the basis of human flourishing – an understanding that respects multiple perspectives and multiple lived experiences. We need a deeper societal agreement about the things that matter most. We need bonds of mutual support, that enable each of us to become better versions of ourselves.

To call a future a sustainable future doesn’t mean there will be no change. Change is fundamental to human life, and we must anticipate that a great deal more change lies ahead – changes in technology, demographics, ideas, standards, and narratives.

Similarly, a sustainable future doesn’t mean there will be no disagreements. But it requires people not to be constantly disagreeable, or contemptuous.

To help us build a sustainable future that can thrive on change rather than collapsing, and where disagreements are constructive rather than contemptuous, whither can we turn?

As someone who spent 25 years immersed in the technology industry, my first instinct is to suggest that we should turn to technology. However, the USA is already awash with technology. The companies that are at the forefront of the global race toward AI superintelligent are headquartered in the US. That surfeit of technology has by no means translated into better democracy.

My next instinct, as someone with a strong personal interest in philosophy, is to suggest that we need to encourage more people to appreciate the insights of that field. Instead of being swept along by rip-roaring tides of “we can” and “we want to”, we need to spend more time pondering “we should” – more time considering alternative scenarios for how the future might unfold.

But even before people are willing to look at alternative possibilities, there needs to be a softening of the spirit.

So my biggest personal takeaway, overnight, is that I should stop looking with hostility or contempt at the vast numbers of people who have reached different decisions, from me, about (for example) which person should become the President of the USA. For that reason, I have resolved to spend some time over the next few days listening to the audio of the 2019 book by Arthur C. Brooks, Love Your Enemies: How Decent People Can Save America from the Culture of Contempt.

(The book has the word ‘America’ in its title, but I feel sure its messages apply elsewhere in the world too.)

Earlier this morning, the following sentence from the book’s opening chapter struck me hard: “We need national healing every bit as much as economic growth”.

That’s a good start!

(I prepared the remarks above to share with attendees at a private roundtable conversation this evening, arranged long before the US election took place – a conversation with the topic “How can we harness emerging technology to improve our democracies?”)

26 September 2024

By far the quickest I’ve ever produced a podcast

Filed under: AGI, podcast — Tags: — David Wood @ 12:50 pm

Wow.

I’ve just experienced what people call “a ChatGPT moment”.

Not with ChatGPT, but with the NotebookLM tool from Google Research.

I’d heard that NotebookLM now has functionality to create an audio conversation between two AI characters who sound remarkably human. A couple of days ago, fellow futurist Tony Czarnecki drew my attention to how good this functionality is, as shown by an experiment on the Sustensis website. (See the section on the page entitled “Two AI-generated journalists discuss in their own podcast Tony Czarnecki’s article ‘Taking Control over AI before it starts controlling us’”.)

I thought I should try a similar experiment myself.

Here’s the outcome.

The blogpost discussed by these two AI characters is this one. That was the only input I gave to the AI.

It was very straightforward to create:

  1. Go to the NotebookLM site
  2. Click “New notebook”
  3. Click on the Link area and paste in the URL of my blogpost
  4. Click on Generate in the Audio overview section
  5. Wait a couple of minutes

Voila! A nine-minute audio conversation.

The conversation takes some ideas from my blogpost, mixes in related material from (I presume) its own knowledgebase, and adds extra flavour with some friendly humour and homespun analogies.

The resulting conversation sounds extremely human. There’s a fine rapport between the two hosts.

The content, to my mind, is questionable in a few places. But the AI characters get at least as much right as most pairs of real-life journalists would.

I’m looking forward to become more familiar with what NotebookLM can do!

PS While I was waiting for the audio WAV file to be generated, I took a quick look at the summary of my blogpost which NotebookLM had already created. Yes, that’s pretty good too!

This essay by David Wood discusses the possibility of achieving longevity escape velocity (LEV) by 2040, a scenario where biomedical interventions reverse the aging process, making individuals biologically younger and healthier. Wood argues that while the complexity of aging is acknowledged, there is sufficient understanding of the damage repair mechanisms involved, allowing for targeted interventions to combat aging. He outlines the potential roadblocks to achieving LEV, including funding constraints, societal resistance, and lack of collaboration within the longevity research community. Wood proposes “breakthrough initiatives” to overcome these challenges, including biomedical and financial reengineering, narrative reengineering, and community reengineering. Despite acknowledging the challenges, Wood concludes that the probability of reaching LEV by 2040 is significant, exceeding 25% but below 50%. He encourages individuals to play an active role in shaping the future of longevity research.

12 September 2024

Are fears of technological unemployment misdirected?

Filed under: culture, education, Events, politics — Tags: , , , , — David Wood @ 8:11 pm

The anxiety is very real. Globally, around 70% of the younger generation are worried about the impact AI will have on their job security.

These remarks were made by Ana Kreacic, Chief Knowledge Officer of management consultant firm Oliver Wyman, on the second day of the GAIN summit which I am attending in Riyadh.

(Click here for a recording of the live stream of all the Day 2 sessions from the main GAIN summit auditorium.)

Kreacic was part of an absorbing session entitled “The Future of Work: AI’s Role in Reskilling and Innovation”.

Kreacic went to share some more statistics. In Saudi Arabia, it is 69% of the younger generation who are worried about the impact of AI on their job security. Evidently, that figure is close to the global average. However, there’s a more significant difference when comparing the proportion of older workers who are concerned about the impact of AI on job security. Globally, that figure is 40%, since (according to Oliver Wyman), “senior employees may feel more secure in their careers because they believe AI will have less impact on higher-level employment”. But the same figure in Saudi Arabia is 59%. In other words, apprehension about job security exists more widely in the Saudi workforce.

However, the panellists were keen to dispel that apprehension. With appropriate actions, there would, they said, be plenty of opportunities for people to continue having good jobs. That’s despite an increasing adoption of AI within companies. Yes, jobs will change. But employees will be able to reskill, so that their careers can flourish.

According to these panellists, it is important to recognise that AI can assist the process of reskilling. Kreacic explained the ways in which AI is enabling personalized training. Rather than employees having to attend standard courses that only loosely match their individual requirements, AIs can provide training materials uniquely tailored (“hyper-targeted”) to each employee. Employees find these courses much more enjoyable, with high rates of knowledge retention long after the course has concluded. Moreover, employees will be able to tap into online training programs to acquire skills that aren’t core to their current roles, but which will be important in the positions into which they hope to move in the near future.

Another panellist, Abdulrahman Alhaqbani, a general manager at leading Saudi information services company stc, described how learning in the real world often involves a relationship between two humans – a mentor and a mentee. However, AI can assist here too, by identifying and recommending good mentor-mentee pairs, in a way similar to how various AI systems suggest other kinds of human relationships.

Nadeem Mazen, CEO of the full-service creative agency Nimblebot, foresaw AI enabling fast career progression of a different kind. Namely, a small group of people – perhaps just 4 or 10 people – with a new idea, some ambition and some curiosity, and with the help of AI, “could take on enormous incumbent brands”. Mazen said that we are entering “the heyday of the small firm”.

Najwa Alghamdi, Analytics Innovation Director at stc, spoke about “injecting into the current workforce digital co-workers” that handle the routine aspects of work tasks, resulting in huge boosts in productivity and efficiency. She said that employees were initially wary of these digital co-workers, but once they saw how well they worked in practice, the employees soon accepted the concept. Indeed, they started to propose additional parts of their existing work that could be handled by new co-created digital co-workers. One reason for the acceptance, Alghamdi emphasised, was that the co-workers had been designed to adhere to the principles of explainable AI.

Ramez El-Serafy, CEO of Flat6Labs, a leading seed and early stage venture capital firm operating in the Middle East and North Africa, described how they offer a learning platform to young founders and entrepreneurs. When they started using AI to create the content for these training programmes, they found it much more efficient and more readily adaptable. By using AI, the content of the courses can be updated as often as once each week.

Maan Al-Mulla, Director of Digital Transformation at Saudi Aramco, described the digital upskilling programmes that his team supervises. For example, the company has launched a “citizen development programme, that enables non-technical employees to develop their own applications and their own solutions using low-code and no-code platforms that are powered by AI”. As a result, the Aramco workforce is “more adaptive to any work change”. In summary, by embracing digital solutions and AI, the result is a smoother and faster transition.

A transition to what? Speakers on this panel, as well as on other panels throughout the event, seemed to share an assessment of the kinds of tasks which cannot be handled by AI:

  • Creativity
  • Critical thinking
  • Emotional intelligence
  • Strategic planning
  • Ensuring customer satisfaction.

Accordingly, worries about employees being pushed out of the workforce by increasing automation should be resisted. Instead of automation being a threat, it should be wisely embraced, and will result in lots of innovative work experiences.

Right?

But my analysis now contains two twists. The first of these twists was made by panellists throughout the day, but the second was raised less often in public discussions. Let’s take these twists one at a time.

Don’t fear automation; fear a stagnant culture

One problem with learning new types of skill is that we go back to being a beginner, and we are prone to make mistakes. For a while, we may become less productive. Our bosses may see us as, for a while, performing less well than in the past. As for ourselves, we may be unsure about how our identity is changing: we used to be such-and-such a kind of employee, but now we are becoming something quite different. We may feel awkward and unsure.

In other words, as well as the technical aspects of a new skill, needed for a change of work condition, there are human aspects. Without strong emotional intelligence, we may shirk from the change. Without supportive management, we may feel pressured to give up the transition. Without appropriate metrics for our contributions in the workforce, the pressures will magnify to resist the change.

That’s why corporate culture plays a large part in whether reskilling is successful. Or, if we are unlucky enough to have unsympathetic bosses or colleagues at work, we need to find emotional support from other parts of our network of relationships – such as family, friends, or mentors.

On these points, Ana Kreacic of Oliver Wyman had a positive message to share: If a company can create the right culture – where it’s OK for people to test things out, spend some time learning, and fail several times in the process – then that company will drive innovation, with employees being willing to experiment.

This is a subject I often address in my own presentations to clients, where I warn about failures, not just of individual career transition, but of companies that are unable to respond to disruptive trends, or indeed of whole nations whose prior culture, processes, and infrastructure are unsuited to changed global conditions. For example, here’s one of the slides I sometimes use.

The takeaway at this point is that employees should stop worrying about AI and automation, but should worry instead about whether the culture in which they operate will be conducive to the kinds of adaptions that AI and automation will necessitate.

But as I suggested above, that’s not yet the end of this analysis. There’s one more twist.

Don’t fear automation; fear bad politics

I can’t remember which panellist said it – probably in a session in one of the breakout rooms around the venue – but it was an important point. IT workers will soon become increasingly unionised.

Why so? Because, despite the positive thoughts offered above, IT workers will soon find their entire jobs under increasing pressure from automation.

That’s not from today’s automation – the “AI Now” part of the theme of the GAIN summit – but from the AI of a few years in the future – “AI Next”.

Look again at the kinds of task listed above, which supposedly cannot be handled by AI. In each case, there is already evidence that AI is well on the path to performing these tasks just as well as humans:

  • Creativity is often displayed by generative AI, although at present it needs good prompts from human operators
  • Critical thinking can arise in large language models in response to instructions to “think step by step”, and is likely to be a standard feature in forthcoming AI systems
  • Artificial emotional intelligence has a vast field of research all to itself – often called “affective computing”
  • Strategic planning is shown by the ability of various AIs to outplay humans in increasingly complex games of strategy
  • Customer satisfaction has many aspects to it, and arises by a combination of all the above traits.

Indeed, toward the end of the panel mentioned above, Nadeem Mazen of Nimblebot remarked that chatbots like ChatGPT can already provide employees with some of the psychological assistance that we previously looked to human colleagues to provide. (“It’s almost spooky”, Mazen mused.)

In short, even the work tasks which are claimed to lie outside the reach of today’s robots and AIs, are likely to come within the expanded reach of automation in the relatively near future.

Perhaps you don’t believe my claim here. In that case, I ask you to dip into my recent essay “Six possible responses as the Economic Singularity approaches”, where the section entitled “Disbelief” offers more analysis in support of my claim.

That same essay also makes the case that the best response to this forthcoming wave of job losses isn’t to resist them, but is to accelerate an enhancement of the way the entire economy is run. This enhancement will involve a redistribution of the bountiful fruits of automation so that everyone in society benefits.

The takeaway at this point of my argument is that employees should stop worrying about AI and automation, but should worry instead about whether the political setup in their country is preparing for this kind of significant redistribution as part of a revised social contract.

It’s for this reason that the slide in my presentations on “The 7 most important characteristics for success over the next 3-5 years” gives special prominence to the skill area of “Politics”:

  • Building and managing coalitions
  • Agile regulations & incentives
  • Revised social contract

Happily, in my discussions with senior AI leaders in Saudi Arabia over the last three days, it’s my impression that they already have a good understanding of these points. I’ll say more about that in a forthcoming article.

Footnote: Patrick Linnenbank, Partner at management consulting firm Arthur D. Little, deserves a big shout-out for his skill in moderating the above panel discussion.

10 September 2024

A disruption in the international AI landscape?

Filed under: AGI, Events, music — Tags: , , , , , , , — David Wood @ 10:29 pm

In the years ahead, which countries will have the biggest impact on the development and deployment of AI?

The two most common answers to that question are the USA and China. Sometimes people put in a word for the EU or India – or the UK, Canada, Israel, or Korea.

Well, in one possible future scenario, another country may power forward to join this group of the biggest AI influencers – and, in the process, might disrupt the international AI landscape.

The country in question is the one where I’ve been for the last 24 hours, and where I will stay for two more days, namely, Saudi Arabia.

I’m attending the GAIN summit in Riyadh. The ‘GAI’ of “GAIN” stands for Global AI. The ‘N’ has a triple meaning: Now, Next, Never.

To quote from the event website:

  • AI Now: How are the AI leaders of the world today deploying, scaling, and leveraging the technology?
  • AI Next: What does the future of AI look like and how will it impact people, businesses and government organizations?
  • AI Never: How do we ensure that the future we design is one that we want to live in and not a dystopian sci-fi?

The ‘A’ in GAIN could plausibly also stand for “ambition”, as the organizers have high ambitions. To quote again from the event website:

The Global AI Summit is the leading platform for advancing the global discussion on AI, where visionary experts, academics, corporates, and policymakers converge from every part of the world to shape the future of artificial intelligence for the good of humanity.

Of course, it’s one thing for a country to express big ambitions to play a leading role in the future of AI. It’s quite another thing to make significant steps toward that ambition. Therefore, as I approached the event, I didn’t know what to think.

Indeed, it was my first visit to Saudi Arabia. I found myself reminded of my first visit to China, way back in October 2002. On that occasion, I was representing Symbian, at what was described as the first UK-China CEO Forum. I recently came across a photo of that event – where my hair was a brighter shade of red than in more recent times!

In both cases – my first visit to China, and my first visit to Saudi Arabia – I was unsure what to expect. It turned out that Shanghai was a bustling metropolis, with gleaming shops and a lively entrepreneurial spirit. The Chinese people I met were dressed nothing like the Chairman Mao suits that I had remembered reading about in my schooldays, and were impressively knowledgeable about technology and business. That visit was to be the first of many I would make in the following years, as Chinese companies steadily became more powerful players on the world stage.

That was 2002. What about my experience in the last 24 hours, in 2024, in Riyadh?

Part of the answer lies in numbers:

  • Over 450 speakers, spread over multiple parallel tracks
  • The speakers represented more than 100 different countries
  • Over 32,000 attendees expected during the three days.

These numbers are all significant steps up from the corresponding numbers from the two previous occasions this summit has been held, in 2020 and 2022.

The speakers include a host of prominent leaders from business and technology worldwide. Some examples:

  • Julie Sweet, the Chair and CEO of Accenture
  • Cristiano Amon, the President and CEO of Qualcomm
  • Marc Raibert, the Founder of Boston Dynamics
  • Martin Kon, the President and COO of Cohere
  • Brian Behlendorf, the Chief AI Strategist of the Linux Foundation
  • Nick Studer, the President and CEO of Oliver Wyman Group
  • Matthew Kropp, the CTO and Managing Director of Boston Consulting Group
  • Alan Qi, the President of Huawei Cloud
  • Yuwon Kim, the CEO of Naver Cloud
  • Caroline Yap, the Global Managing Director of Google Cloud.

Multiple segments of society in Saudi Arabia were well represented too – including an impressive number of adept, articulate women leaders, who had some fascinating pieces of advice.

With so many speakers, it is perhaps inevitable that some speeches fell flat – especially several of the ones about the governance of AI, where the conversations seemed to be going round in circles, with little appreciation of what I see as the risks of catastrophic harm if next generation AI is mishandled. However, the technical talks were generally compelling.

I particularly liked the talks by Andrew Feldman, Co-founder and CEO of Cerebras Systems, and Jonathan Ross, Founder and CEO of Groq. These two companies each position themselves as disruptors of the GPU market, and, hence, as potentially overtaking Nvidia. Instead of GPUs, or the TPUs developed by Google, they have created LPUs (Language Processing Units) in the case of Groq, and waferscale AI chips in the case of Cerebras. Both companies claim notable improvements in speed over previous AI chip configurations. I heard the phrase “like ChatGPT but insanely fast”.

Both Cerebras and Groq emphasized close partnerships with Saudi Arabia. Andrew Feldman of Cerebras described a special collaboration with KAUST (King Abdullah University of Science and Technology). And Jonathan Ross of Groq appeared on stage alongside Tareq Amin, the CEO of Aramco Digital. Ross gave three reasons for their company investing strongly in the country:

  • The abundance of energy resources in the country
  • The good business environment, that encourages and supports this kind of partnership
  • The geographical location, close to three different continents, so that the resulting high-performance AI cluster could serve the needs of up to four billion people.

It was while listening to these two talks that the Saudi ambition to become a global leader in AI started to become more credible in my mind.

I had already seen the strong enthusiasm in large numbers of Saudi delegates at the event. They were avidly leaning forward in their seats, to capture as much as possible of the advice being provided on the various stages. It seems that the country is aware of the need to transition away from reliance on the oil industry, and instead to actively participate in shaping the global AI marketplace.

There were many other talks and panels which left me with new ideas to consider. For example, I wished that Marc Raibert, the Founder of Boston Dynamics, could have had more time to develop his fascinating ideas further. He made the case that true intelligence involves an interactive combination of cognitive intelligence (“what’s going on in our heads”) and athletic intelligence (“what’s going on in our bodies”). That explanation formed the backdrop for the progress made by Boston Dynamics over the years, with robots such as Spot (commercially significant “today”), Stretch (“tomorrow”), and Atlas (“future”). In addition to his role at Boston Dynamics, Raibert is also the Founder and Executive Director of the AI Institute, whose website proudly reports that “The AI Institute aims to solve the most important and fundamental problems in robotics and AI”. As I said, I wish he had more time to continue talking about that work.

Earlier in the day, I watched a fascinating six-way round-table discussion on the subject “Hallucinations and Confabulations: when chatbots go rogue”, with speakers from Kearney, Mozn, Saudi Aramco, Vectara, KAUST, and INF, who each had long careers as experts in various aspects of AI. The discussion went on for 90 minutes, but I would have been happy for it to continue longer, as it had lots of good-spirited clashes of ideas about the strengths and weaknesses of large language models, and possible approaches to add fact-checking components into the AI systems of the near future. One of the speakers, Amr Awadallah of Vectara, boldly predicted that AGI would exist by 2028. Part of his reasoning was his argument that ongoing improvements in RAG (Retrieval Augmented Generation) were steadily reducing the prevalence of hallucinations in the content being generated.

That kind of awareness of potential dramatic improvements in AI capability by 2028 was, sadly (to my mind) missing from what many of the speakers in other sessions were assuming. These other speakers were focused, in effect, on the “Now” of AI, and didn’t foresee many real changes for “AI Next” any time soon. Frankly, if they keep thinking that way, they’re likely to be disrupted themselves. Anyway, this issue is something I hope will feature again in the sessions on days two and three of this year’s GAIN. I look forward to these days with great interest.

I’ll end at the beginning. The day started with an artistic performance, symbolizing the sequential creation of ANI (Artificial Narrow Intelligence), AGI (Artificial General Intelligence), and then ASI (Artificial Super Intelligence). The narrator offered a positive vision of a beneficial relationship of humanity and superintelligence: “There will be no more confusion, and a golden age of progress will flourish, where men and machines, united by an unprecedented alliance, will walk together toward a destiny of glory and happiness”.

Having come to life, the ASI spoke to a young boy, who was a representative of humanity, saying “I am your new avatar, and I will become your best friend”.

In response, the boy started singing what was said to be his favourite song. The music was increasingly stirring and the singing increasingly dramatic. Given the location, Riyadh, I could hardly believe what I was hearing:

Imagine there’s no heaven
It’s easy if you try
No hell below us
Above us, only sky
Imagine all the people
Livin’ for today
Ah

Imagine no possessions
I wonder if you can
No need for greed or hunger
A brotherhood of man

Imagine all the people
Sharing all the world

You may say I’m a dreamer
But I’m not the only one
I hope someday you’ll join us
And the world will live as one

The words and music of John Lennon’s “Imagine” have graced numerous stages over the decades, but somehow, for me, this was particularly evocative.

27 July 2024

Disbelieve? Accept? Resist? Steer? Simplify? or Enhance?

Six possible responses as the Economic Singularity approaches. Which do you pick?

Over the course of the next few decades, work and income might be fundamentally changed. A trend that has been seen throughout human history might be raised to a pivotal new level:

  • New technologies – primarily the technologies of intelligent automation – will significantly reduce the scope for human involvement in many existing work tasks;
  • Whilst these same technologies will, in addition, lead to the creation of new types of work tasks, these new tasks, like the old ones, will generally also be done better by intelligent automation than via human involvement;
  • That is, the new jobs (such as “robot repair engineer” or “virtual reality experience designer”) will be done better, for the most part, by advanced robots than by humans;
  • As a result, more and more people will be unable to find work that pays them what they consider to be a sufficient income.

Indeed, technological changes result, not only in new products, but in new ways of living. The faster and more extensive the technological changes, the larger the scope for changes in lifestyle, including changes in how we keep healthy, how we learn things, how we travel, how we house ourselves, how we communicate and socialise, how we entertain ourselves, and – of particular interest for this essayhow we work and how we are paid.

But here’s the dilemma in this scenario. Although automation will be capable of producing everything that people require for a life filled with flourishing, most people will be unable to pay for these goods and services. Lacking sufficient income, the majority of people will lose access to good quality versions of some or all of the following: healthcare, education, travel, accommodation, communications, and entertainment. In short, whilst a small group of people will benefit handsomely from the products of automation, the majority will be left behind.

This dilemma cannot be resolved merely by urging the left behinds to “try harder”, to “learn new skills”, or (in the words attributed to a 1980s UK politician) to “get on a bike and travel to where work is available”. Such advice was relevant in previous generations, but it will no longer be sufficient. No matter how hard they try, the majority of people won’t be able to compete with tireless, relentless smart machinery powered by new types of artificial intelligence. These robots, avatars, and other automated systems will demonstrate not only diligence and dexterity but also creativity, compassion, and even common sense, making them the preferred choice for most tasks. Humans won’t be able to compete.

This outcome is sometimes called “The Economic Singularity” – a term coined by author and futurist Calum Chace. It will involve a singular transition in humanity’s mode of economics:

  • From when most people expect to be able to earn money by undertaking paid work for a significant part of their life
  • To when most people will be unable to earn sufficient income from paid work.

So what are our options?

Here are six to consider, each of which have advocates rooting for them:

  1. Disbelieve in the possibility of any such large-scale job losses within the foreseeable future
  2. Accept the rise of new intelligent automation technologies, but take steps to place ourselves in the small subset of society that particularly benefits from them
  3. Resist the rise of these new technologies. Prevent these systems from being developed or deployed at scale
  4. Steer the rise of these new technologies, so that plenty of meaningful, high-value roles remain for humans in the workforce
  5. Simplify our lifestyles, making do with less, so that most of us can have a pleasant life even without access to the best outputs of intelligent automation technologies
  6. Enhance, with technology, not just the mechanisms to create products but also the mechanisms used in society for the sharing of the benefits of products.

In this essay, I’ll explore the merits and drawbacks of these six options. My remarks split into three sections:

  1. Significant problems with each of the first five options listed
  2. More details of the sixth option – “enhance” – which is the option I personally favour
  3. A summary of what I see as the vital questions arising – questions that I invite other writers to address.

A: Significant challenges ahead

A1: “Disbelieve”

At first glance, there’s a lot in favour of the “disbelieve” option. The evidence from human history, so far, is that technology has had three different impacts on the human workforce, with the net impact always being positive:

  • A displacement factor, in which automation becomes able to do some of the work tasks previously performed by humans
  • An augmentation factor, in which humans become more capable when they take advantage of various tools provided by technology, and are able to do some types of work task better than before – types of work that take on a new significance
  • An expansion factor, in which the improvements to productivity enabled by the two previous factors generate economic growth, leading to consumers wanting more goods and services than before. This in turn provides more opportunities for people to gain employment helping to provide these additional goods and services.

For example, some parts of the work of a doctor may soon be handled by systems that automatically review medical data, such as ultrasound scans, blood tests, and tissue biopsies. These systems will be better than human doctors in detecting anomalies, in distinguishing between false alarms and matters of genuine concern, and in recommending courses of treatment that take fully into account the unique personal circumstances of each patient. That’s the displacement effect. In principle, that might leave doctors more time to concentrate on the “soft skills” parts of their jobs: building rapport with patients, gently coaxing them to candidly divulge all factors relevant to their health, and inspiring them to follow through on courses of treatment that may, for a while, have adverse side effects. The result in this narrative: patients receive much better healthcare overall, and are therefore especially grateful to their doctors. Human doctors will remain much in demand!

More generally, automation systems might cover the routine parts of existing work, but leave in human hands the non-routine aspects – the parts which cannot be described by any “algorithm”.

However, there are three problems with this “disbelieve” narrative.

First, automation is increasingly able to cover supposedly “non-routine” tasks as well as routine tasks. Robotic systems are able to display subtle signs of emotion, to talk in a reassuring tone of voice, to suggest creative new approaches, and, more generally, to outperform humans in soft skills (such as apparent emotional intelligence) as well as in hard skills (such as rational intelligence). These systems gain their abilities, not by any routine programming with explicit instructions, but by observing human practices and learning from them, using methods known as “machine learning”. Learning via vast numbers of repeated trials in simulated virtual environments adds yet more capabilities to these systems.

Second, it may indeed be the case that some tasks will remain to be done by humans. It may be more economically effective that way: consider the low-paid groups of human workers who manually wash people’s cars, sidelining fancy machines that can also do that task. It may also be a matter of human preference: we might decide we occasionally prefer to buy handmade goods rather than ones that have been expertly produced by machines. However, there is no guarantee that there will be large numbers of these work roles. Worse, there is no guarantee that these jobs will be well-paid. Consider again the poorly paid human workers who wash cars. Consider also the lower incomes received by Uber drivers than, in previous times, by drivers of old-fashioned taxis where passengers paid a premium for the specialist navigational knowledge acquired by the drivers over many years of training.

Third, it may indeed be the case that companies that operate intelligent automation technologies receive greater revenues as a result of the savings they make in replacing expensive human workers with machinery with a lower operating cost. But there is no guarantee that this increased income, and the resulting economic expansion, will result in more jobs for humans. Instead, the extra income may be invested in yet more technology, rather than in hiring human workers.

In other words, there is no inevitability about the ongoing relevance of the augmentation and expansion factors.

What’s more, this can already be seen in the statistics of rising inequality within society:

  • A growing share of income in the hands of the top 0.1% of salaries
  • A growing share of income from investments instead of from salaries
  • A growing share of wealth in the hands of the top 0.1% wealth owners
  • Declining median incomes at the same time as mean incomes rise.

This growing inequality is due at least in part to the development and adoption of more powerful automation technologies:

  • Companies can operate with fewer human staff, and gain their market success due to the technologies they utilise
  • Online communications and comparison tools mean that lower-quality output loses its market presence more quickly to output with higher quality; this is the phenomenon of “winner takes all” (or “winner takes most”)
  • Since the contribution of human workers is less critical, any set of workers who try to demand higher wages can more easily be replaced by other workers (perhaps overseas) who are willing to accept lower wages (consider again the example of teams of car washers).

In other words, we may already be experiencing an early wave of the Economic Singularity, arriving before the full impact takes place:

  • Intelligent automation technologies are already giving rise to a larger collection of people who consider themselves to be “left behind”, unable to earn as much money as they previously expected
  • Oncoming, larger waves will rapidly increase the number of left behinds.

Any responses we have in mind for the Economic Singularity should, therefore, be applied now, to address the existing set of left behinds. That’s instead of society waiting until many more people find themselves, perhaps rather suddenly, in that situation. By that time, social turmoil may make it considerably harder to put in place a new social contract.

To be clear, there’s no inevitability about how quickly the full impact of the Economic Singularity will be felt. It’s even possible that, for unforeseen reasons, such an impact might never arise. However, society needs to think ahead, not just about inevitabilities, but also about possibilities – and especially about possibilities that seem pretty likely.

That’s the case for not being content with the “disbelieve” option. It’s similar to the case for rejecting any claims that:

  • Many previous predictions of global pandemics turned out to be overblown; therefore we don’t need to make any preparations for any future breakthrough global pandemic
  • Previous predictions of nuclear war between superpowers turned out not to be fulfilled; therefore we can stop worrying about future disputes escalating into nuclear exchanges.

No: that ostrich-like negligence, looking away from risks of social turmoil in the run-up to a potential Economic Singularity, would be grossly irresponsible.

A2: “Accept”

As a reminder, the “accept” option is when some people accept that there will be large workplace disruption due to the rise of new intelligent automation technologies, with the loss of most jobs, but when these people are resolved to take steps to place themselves in the small subset of society that particularly benefits from these disruptions.

Whilst it’s common to hear people argue, in effect, for the “disbelieve” viewpoint covered in the previous subsection, it’s much rarer for someone to openly say they are in favour of the “accept” option.

Any such announcement would tend to mark the speaker as being self-centred and egotistical. They evidently believe they are among a select group who have what it takes to succeed in circumstances where most people will fail.

Nevertheless, it’s a position that some people might see as “the best of a set of bad options”. They may think to themselves: Waves of turbulence are coming. It’s not possible to save everyone. Indeed, it’s only possible to save a small subset of the population. The majority will be left behind. In that context, they urge themselves: Stop overthinking. Focus on what’s manageable: one’s own safety and security. Find one of the few available lifeboats and jump in quickly. Don’t let yourself worry about the fates of people who are doomed to a less fortunate future.

This position may strike someone as credible to the extent that they already see themselves as one of society’s winners:

  • They’ve already been successful in business
  • They assess themselves as being healthy, smart, focused, and pragmatic
  • They intend to keep on top of new technological possibilities: they’ll learn about the strengths and weaknesses of various technologies of intelligent automation, and exploit that knowledge.

What’s more, they may subscribe to a personal belief that “heroes make their own destiny”, or similar.

But before you adopt the “accept” stance, here are six risks you should consider:

  1. The skills in which you presently take pride, as supposedly being beyond duplication by any automated system, may unexpectedly be rendered obsolete due to technology progressing more quickly and more comprehensively than you expected. You might therefore find yourself, not as one of society’s winners, but as part of the growing “left behind” community
  2. Even if some of your skills remain unmatched by robots or AIs, these skills may have played less of a role than you thought in your past successes; some of these past successes may also have involved elements of good fortune, or personal connections, and so on. These auxiliary factors may give you a different outcome the next time you “roll the dice” and try to change from one business opportunity to another. Once again, you may find yourself unexpectedly in the social grouping left behind by technological change
  3. Even if you personally do well in the turmoil of increased job losses and economic transformation, what about all the people that matter a lot to you, such as family members and special friends? Is your personal success going to be sufficient that you can provide a helping hand to everyone to whom you feel a tie of closeness? Or are you prepared to stiffen your attitudes and to break connections with people from these circles of family and friends, as they become “left behind”?
  4. Many people who end up as left behinds will suffer physical, mental, or emotional pain, potentially including what are known as “deaths of despair”. Are you prepared to ignore all that suffering?
  5. Some of the left behinds may be inclined to commit crimes, to acquire some of the goods and services from which they are excluded by their state of relative poverty. That implies that security measures will have to be stepped up, including strict borders. You might be experiencing a life of material abundance, but with the drawback of living inside a surveillance-state society that is psychologically embittered
  6. Some of the left behinds might go one step further, obtaining dangerous weapons, leading to acts of mass terrorism. In case they manage to access truly destructive technologies, the result might be catastrophic harm or even existential destruction.

In deciding between different social structures, it can be helpful to adopt an approach proposed by the philosopher John Rawls, known as “the veil of ignorance”. In this approach, we are asked to set aside our prior assumptions about which role in society we will occupy. Instead, we are asked to imagine that we have an equal probability of obtaining any of the positions within that society.

For example, consider a society we’ll call WLB, meaning “with left behinds”, in which 995 people out of every one thousand are left behind, and five of every thousand have an extremely good standard of living (apart from having to deal with the problems numbered 4, 5, and 6 in the above list). Consider, as an alternative, a society we’ll call NLB, “no one left behind”, in which everyone has a quality of living that can be described as “good” (if, perhaps, not as “extremely good”).

If we don’t know whether we’ll be one of the fortunate 0.5% of the population, would we prefer society WLB or society NLB?

The answer might seem obvious: from behind the veil of ignorance, we should strongly prefer NLB. However, this line of argument is subject to two objections. First, someone might feel sure that they really will end up as part of the 0.5%. But that’s where the problems numbered 1 and 2 in the above list should cause a reconsideration.

The second objection deserves more attention. It is that a society such as NLB may be an impossibility. Attempts to create NLB might unintentionally lead to even worse outcomes. After all, bloody revolutions over recent centuries have often veered catastrophically out of control. Self-described “vanguards” of a supposedly emergent new society have turned into brutal demagogues. Attempts to improve society through the ballot box have frequently failed too – prompting the acerbic remark by former British Prime Minister Margaret Thatcher that (to paraphrase) “the problem with socialism is that you eventually run out of other people’s money”.

It is the subject of the remainder of this essay to assess whether NLB is actually practical. (If not, we might have to throw our efforts behind “Accept” after all.)

A3: “Resist”

The “resist” idea starts from a good observation. Just because something is possible, it doesn’t mean that society should make it happen. In philosophical language, a could does not imply a should.

Consider some examples. Armed forces in the Second World War could have deployed chemical weapons that emitted poison gas – as had happened during the First World War. But the various combatants decided against that option. They decided: these weapons should not be used. In Victorian times, factory owners could have employed young children to operate dangerous machinery with their nimble fingers, but society decided, after some deliberation, that such employment should not occur. Instead, children should attend school. More recently, nuclear power plants could have been constructed with scant regard to safety, but, again, society decided that should not happen, and that safety was indeed vital in these designs.

Therefore, just because new technologies could be developed and deployed to produce various goods and services for less cost than human workers, there’s no automatic conclusion in favour of that happening. Just as factory owners were forbidden from employing young children, they could also be forbidden from employing robots. Societal attitudes matter.

In this line of thinking, if replacing humans with robots in the workplace will have longer term adverse effects, society ought to be able to decide against that replacement.

But let’s look more closely at the considerations in these two cases: banning children from factories, and banning robots from factories. There are some important differences:

  • The economic benefits to factory owners from employing children were significant but were declining: newer machinery could operate without requiring small fingers to interact with them
  • The economy as a whole needed more workers who were well educated; therefore it made good economic sense for children to attend school rather than work in factories
  • The economic benefits to factory owners from deploying robots are significant and are increasing: newer robots can work at even higher levels of efficiency and quality, and cost less to operate
  • The economy as a whole has less need of human workers, so there is no economic argument in favour of prioritising the employment and training of human workers instead of the deployment of intelligent automation.

Moreover, it’s not just “factory owners” who benefit from being able to supply goods and services at lower cost and higher quality. Consumers of these goods and services benefit too. Consider again the examples of healthcare, education, travel, accommodation, communications, and entertainment. Imagine choices between:

  • High-cost, low-quality healthcare, provided mainly by humans, versus low-cost, high-quality healthcare, provided in large part by intelligent automation
  • High-cost, low-quality education, provided mainly by humans, versus low-cost, high-quality education, provided in large part by intelligent automation
  • And so on.

The “resist” option therefore would imply acceptance of at least part of the “simplify” option (discussed in more depth later): people in that circumstance would need to accept lower quality provision of healthcare, education, travel, accommodation, communications, and entertainment.

In other words, the resist option implies saying “no” to many possible elements of technological progress and the humanitarian benefits arising from it.

In contrast, the “steer” option tries to say “yes” to most of the beneficial elements of technological progress, whilst still preserving sufficient roles for humans in workforces. Let’s look more closely at it.

A4: “Steer”

The “steer” option tries to make a distinction between:

  • Work tasks that are mainly unpleasant or tedious, and which ought to be done by intelligent automation rather than by humans
  • Work tasks that can be meaningful or inspiring, especially when the humans carrying out these tasks have their abilities augmented (but not superseded) by intelligent automation (this concept was briefly mentioned in discussion of the “Disbelieve” option).

The idea of “steer” is to prioritise the development and adoption of intelligent automation technologies that can replace human workers in the first, tedious, category of tasks, whilst augmenting humans so they can continue to carry out the second, inspiring, category of tasks.

This also means a selective resistance to improvements in automation technologies, namely to those improvements which would result in the displacement of humans from the second category of tasks.

This proposal has been championed by, for example, the Stanford economist Erik Brynjolfsson. Brynjolfsson has coined the phrase “the Turing Trap”, referring to what he sees as a mistaken direction in the development of AI, namely trying to create AIs that can duplicate (and then exceed) human capabilities. Such AIs would be able to pass the “Turing Test” that Alan Turing famously described in 1950, but that would lead, in Brynjolfsson’s view, to a fearsome “peril”:

Building machines designed to pass the Turing Test and other, more sophisticated metrics of human-like intelligence… is a path to unprecedented wealth, increased leisure, robust intelligence, and even a better understanding of ourselves. On the other hand, if [that] leads machines to automate rather than augment human labor, it creates the risk of concentrating wealth and power. And with that concentration comes the peril of being trapped in an equilibrium where those without power have no way to improve their outcomes.

Here’s how Brynjolfsson introduces his ideas:

Creating intelligence that matches human intelligence has implicitly or explicitly been the goal of thousands of researchers, engineers, and entrepreneurs. The benefits of human-like artificial intelligence (HLAI) include soaring productivity, increased leisure, and perhaps most profoundly, a better understanding of our own minds.

But not all types of AI are human-like – in fact, many of the most powerful systems are very different from humans – and an excessive focus on developing and deploying HLAI can lead us into a trap. As machines become better substitutes for human labor, workers lose economic and political bargaining power and become increasingly dependent on those who control the technology. In contrast, when AI is focused on augmenting humans rather than mimicking them, then humans retain the power to insist on a share of the value created. What’s more, augmentation creates new capabilities and new products and services, ultimately generating far more value than merely human-like AI. While both types of AI can be enormously beneficial, there are currently excess incentives for automation rather than augmentation among technologists, business executives, and policymakers.

Accordingly, here are his recommendations:

The future is not preordained. We control the extent to which AI either expands human opportunity through augmentation or replaces humans through automation. We can work on challenges that are easy for machines and hard for humans, rather than hard for machines and easy for humans. The first option offers the opportunity of growing and sharing the economic pie by augmenting the workforce with tools and platforms. The second option risks dividing the economic pie among an ever-smaller number of people by creating automation that displaces ever-more types of workers.

While both approaches can and do contribute to progress, too many technologists, businesspeople, and policymakers have been putting a finger on the scales in favor of replacement. Moreover, the tendency of a greater concentration of technological and economic power to beget a greater concentration of political power risks trapping a powerless majority into an unhappy equilibrium: the Turing Trap….

The solution is not to slow down technology, but rather to eliminate or reverse the excess incentives for automation over augmentation. In concert, we must build political and economic institutions that are robust in the face of the growing power of AI. We can reverse the growing tech backlash by creating the kind of prosperous society that inspires discovery, boosts living standards, and offers political inclusion for everyone. By redirecting our efforts, we can avoid the Turing Trap and create prosperity for the many, not just the few.

But a similar set of questions arise for the “steer” option as for the more straightforward “resist” option. Resisting some technological improvements, in order to preserve employment opportunities for humans, means accepting a lower quality and higher cost of the corresponding goods and services.

Moreover, that resistance would need to be coordinated worldwide. If you resist some technological innovations but your competitors accept them, and replace expensive human workers with lower cost AI, their products can be priced lower in the market. That would drive you out of business – unless your community is willing to stick with the products that you produce, relinquishing the chance to purchase cheaper products from your competitors.

Therefore, let’s now consider what would be involved in such a relinquishment.

A5: “Simplify”

If some people prefer to adopt a simpler life, without some of the technological wonders that the rest of us expect, that choice should be available to them.

Indeed, human society has long upheld the possibility of choice. Communities are able, if they wish, to make their own rules about the adoption of various technologies.

For example, organisers of various sports set down rules about which technological enhancements are permitted, within those sports, and which are forbidden. Bats, balls, protective equipment, sensory augmentation – all of these can be restricted to specific dimensions and capabilities. The restrictions are thought to make the sports better.

Similarly, goods sold in various markets can carry markings that designate them as being manufactured without the use of certain methods. Thus consumers can see an “organic” label and be confident that certain pesticides and fertilisers have been excluded from the farming methods used to produce these foods. Depending on the type of marking, there can also be warranties that these foods contain no synthetic food additives and have not been processed using irradiation or industrial solvents.

Consider also the Amish, a group of traditionalist communities from the Anabaptist tradition, with their origins in Swiss German and Alsatian (French) cultures. These communities have made many decisions over the decades to avoid aspects of the technology present in wider society. Their clothing has avoided buttons, zips, or Velcro. They generally own no motor cars, but use horse-drawn carts for local transport. Different Amish communities have at various times forbidden (or continue to forbid) high-voltage electricity, powered lawnmowers, mechanical milking machines, indoor flushing toilets, bathtubs with running water, refrigerators, telephones inside the house, radios, and televisions.

Accordingly, whilst some parts of human society might in the future adopt fuller use of intelligent automation technologies, deeply transforming working conditions, other parts might, Amish-like, decide to abstain. They may say: “we already have enough, thank you”. Whilst people in society as a whole may be unable to find work that pays them good wages, people in these “simplified” communities will be able to look after each other.

Just as Amish communities differ among themselves as to how much external technology they are willing to incorporate into their lives, different “simplified” communities could likewise make different choices as to how much they adopt technologies developed outside their communities. Some might seek to become entirely self-sufficient; others might wish to take advantage of various medical treatments, educational software, transportation systems, robust housing materials, communications channels, and entertainment facilities provided by the technological marvels created in wider society.

But how will these communities pay for these external goods and services? In order to be able to trade, what will they be able to create that is not already available in better forms outside their communities, where greater use is made of intelligent automation?

We might consider tourist visits, organic produce, or the equivalent of handmade ornaments. But, again, what will make these goods more attractive, to outsiders, than the abundance of goods and services (including immersive virtual reality travel) that is already available to them?

We therefore reach a conclusion: groups that choose to live apart from deeply transformative technologies will likely lack access to many valuable goods and services. It’s possible they may convince themselves, for a while, that they prefer such a lifestyle. However, just as the attitudes of Amish communities have morphed over the decades, so that these groups now see (for example) indoor flushing toilets as a key part of their lives, it is likely that the attitudes of people in these simplified communities will also alter. When facing death from illness, or when facing disruption to their relatively flimsy shelters from powerful weather, they may well find themselves deciding they prefer, after all, to access more of the fruits of technological abundance.

With nothing to exchange or barter for these fruits, the only way they will receive them is via a change in the operation of the overall economy. That brings us to the sixth and final option from my original list, “enhance”.

Whereas the previous options have looked at various alterations in how technology is developed or applied, “enhance” looks at a different possibility: revising how the outputs and benefits of technology are planned and distributed throughout society. These are revisions to the economy, rather than revisions in technology.

In this vision, with changes in both technology and the economy, everyone will benefit handsomely. Simplicity will remain a choice, for those who prefer it, but it won’t be an enforced choice. People who wish to participate in a life of abundance will be able to make that choice instead, without needing to find especially remunerative employment to pay for it.

I’ll accept, in advance, that many critics may view such a possibility as a utopian fantasy. But let’s not rush to a conclusion. I’ll build my case in stages.

B: Enhancing the operation of the economy

Let’s pick up the conversation with the basics of economics, which is the study of how to deal with scarcity. When important goods and services are scarce, humans can suffer.

Two fundamental economic forces that have enabled astonishing improvements in human wellbeing over the centuries, overcoming many aspects of scarcity, are collaboration and competition:

  • Collaboration: person A benefits from the skills and services of person B, whereas person B benefits reciprocally from a different set of skills and services of person A; this allows both A and B to specialise, in different areas
  • Competition: person C finds a way to improve the skills and services that they offer to the market, compared to person D, and therefore receives a higher reward – causing person D to consider how to improve their skills and services in turn, perhaps by copying some of the methods and approach of person C.

What I’ve just described in terms of simple interactions between two people is nowadays played out, in practice, via much larger communities, and over longer time periods:

  • Collaboration includes the provision of a social safety net, for looking after individuals who are less capable, older, lack resources, or who have fallen on hard times; these safety nets can operate at the family level, tribe (extended family) level, community level, national level, or international level
  • The prospect of gaining extra benefits from better skills and services leads people to make personal investments, in training and tools, so that they can possess (for a while at least) an advantage in at least one market niche.

Importantly, it needs to be understood that various forms of collaboration and competition can have negative consequences as well as positive ones:

  • A society that keeps extending an unconditional helping hand to someone who avoids taking personal responsibility, or to a group that is persistently dysfunctional, might end up diverting scarce resources from key social projects to being squandered by people for no good purpose
  • In a race to become more economically dominant, other factors may be overlooked, such as social harmony, environmental wellbeing, and other so-called externalities.

In other words, the forces that can lead to social progress can also lead to social harm.

In loose terms, the two sets of negative consequences can be called “failure modes of socialism” and “failure modes of capitalism” – to refer to two historically significant terms in theories of economics, namely “socialism” and “capitalism”. These two broad frameworks are covered in the subsections ahead, along with key failure modes in each case. After that, we’ll consider models that aspire to transcend both sets of failure by delivering “the best of both worlds”.

To look ahead, it is the “best of both worlds” model that has the potential to be the best solution to the Economic Singularity.

B1: Need versus greed?

When there is a shortage of some product or service, how should it be distributed? To the richest, the strongest, the people who shout the loudest, the special friends of the producers, or to whom?

One answer to that question is given in the famous slogan, “From each according to his ability, to each according to his needs”. In other words, each person should receive whatever they truly need, be it food, clothing, healthcare, accommodation, transportation, and so on.

That slogan was popularised by Karl Marx in an article he wrote in 1875, but earlier political philosophers had used it in the 1840s. Indeed, an antecedent can be found in the Acts of the Apostles in the New Testament, referring to the sharing of possessions within one of the earliest groups of Christian believers:

All the believers were one in heart and mind. No one claimed that any of their possessions was their own, but they shared everything they had… There were no needy persons among them. For from time to time those who owned land or houses sold them, brought the money from the sales and put it at the apostles’ feet, and it was distributed to anyone who had need.

Significantly, Marx foresaw that principle of full redistribution as being possible only after technology (“the productive forces”) had sufficiently “increased”. It was partly for that reason that Joseph Stalin, despite being an avowed follower of Marx, wrote a different principle into the 1936 constitution of the Soviet Union: “From each according to his ability, to each according to his work”. Stalin’s justification was that the economy had not yet reached the required level of production, and that serious human effort was first required to reach peak industrialization.

This highlights one issue with the slogan, and with visions of society that seek to place that slogan at the centre of their economy: before products and services can be widely distributed, they need to be created. A preoccupation with distribution will fail unless it is accompanied by sufficient attention to creation. Rather than fighting over how a pie is divided, it’s important to make the pie larger. Then there will be much more to share.

A second issue is in the question of what counts as a need. Clothing is a need, but what about the latest fashion? Food is a need, but what about the rarest of fruits and vegetables? And what about “comfort food”: is that a need? Healthcare is a need, but what about a heart transplant? Transportation is a need, but what about intercontinental aeroplane flights?

A third issue with the slogan is that a resource that is assigned to someone’s perceived need is, potentially, a resource denied from a more productive use to which another person might put that resource. Money spent to provide someone with what they claim they need might have been invested elsewhere to create more resources, allowing more people to have what they claim that they need.

Thus the oft-admired saying attributed to Mahatma Gandhi, “The world has enough for everyone’s needs, but not everyone’s greed”, turns out to be problematic in practice. Who is to say what is ‘need’ and what is ‘greed’? Are desires for luxury goods always to be denigrated as ‘greed’? Isn’t life about enjoyment, vitality, and progress, rather than just calmly sitting down in a state of relative poverty?

B2: Socialism and its failures

There are two general approaches to handling the problems just described: centralised planning, and free-market allocation.

With centralised planning, a group of reputedly wise people:

  • Keep on top of information about what the economy can produce
  • Keep on top of information about what people are believed to actually need
  • Direct the economy so that it makes a better job of producing what it has been decided that people need.

Therefore a central planner may dictate that more shoes of a certain type need to be produced. Or that drinks should be manufactured with less sugar in them, that particular types of power stations should be built, or that particular new drugs should be created.

That’s one definition of socialism: representatives of the public direct the economy, including the all-important “means of production” (factories, raw materials, infrastructure, and so on), so that the assumed needs of all members of society are met.

However, when applied widely within an economy, centralised planning approaches have often failed abysmally. Assumptions about what people needed often proved wrong, or out-of-date. Indeed, members of the public often changed their minds about what products were most important for them, especially after new products came into use, and their upsides and downsides could be more fully appreciated. Moreover, manufacturing innovations, such as new drugs, or new designs for power stations, could not be achieved simply by wishing them or “planning” them. Finally, people working in production roles often felt alienated, lacking incentives to apply their best ideas and efforts.

That’s where the alternative coordination mechanism – involving free markets – often fared better (despite problems of its own, which we’ll review in due course). The result of free markets has been significant improvements in the utility, attractiveness, performance, reliability, and affordability of numerous types of goods and services. As an example, modern supermarkets are one of the marvels of the world, being stocked from floor to ceiling with all kinds of items to improve the quality of daily life. People around the globe have access to a vast variety of all-around nourishment and experience that would have astonished their great-grandparents.

In recent decades, there have been similar rounds of sustained quality improvement and cost reduction for personal computers, smartphones, internet access, flatscreen TVs, toys, kitchen equipment, home and office furniture, clothing, motor cars, aeroplane tickets, solar panels, and much more. The companies that found ways to improve their goods and services flourished in the marketplace, compelling their competitors to find similar innovations – or go out of business.

It’s no accident that the term “free market” contains the adjective “free”. The elements of a free market which enable it to produce a stream of quality improvements and cost reductions include the following freedoms:

  1. The freedom for companies to pursue profits – under the recognition that the prospect of earning profits can incentivise sustained diligence and innovation
  2. The freedom for companies to adjust the prices for their products, and to decide by themselves the features contained in these products, rather than following the dictates of any centralised planner
  3. The freedom for groups of people to join together and start a new business
  4. The freedom for companies to enter new markets, rather than being restricted to existing product lines; new competitors keep established companies on their toes
  5. The freedom for employees to move to new roles in different companies, rather than being tied to their existing employers
  6. The freedom for companies to explore multiple ways to raise funding for their projects
  7. The freedom for potential customers to not buy products from established vendors, but to switch to alternatives, or even to stop using that kind of product altogether.

What’s more, the above freedoms are permissionless in a free market. No one needs to apply for a special licence from central authorities before one of these freedoms becomes available.

Any political steps that would curtail the above freedoms need careful consideration. The result of such restrictions could (and often do) include:

  • A disengaged workforce, with little incentive to apply their inspiration and perspiration to the tasks assigned to them
  • Poor responsiveness to changing market interest in various products and services
  • Overproduction of products for which there is no market demand
  • Companies having little interest in exploring counterintuitive combinations of product features, novel methods of assembly, new ways of training or managing employees, or other innovations.

Accordingly, anyone who wishes to see the distribution of high-quality products to the entire population needs to beware curtailing freedoms of entrepreneurs and innovators. That would be taking centralised planning too far.

That’s not to say that the economy should dispense with all constraints. That would raise its own set of deep problems – as we’ll review in the next subsection.

B3: Capitalism and its failures

Just as there are many definitions of socialism, there are many definitions of capitalism.

Above, I offered this definition of socialism: an economy in which production is directed by representatives of the public, with the goal that the assumed needs of all members of society are met. For capitalism, at least parts of the economy are directed, instead, by people seeking returns on the capital they invest. This involves lots of people making independent choices, of the types I have just covered: choices over prices, product features, types of product, areas of business to operate within, employment roles, manufacturing methods, partnership models, ways of raising investment, and so on.

But these choices depend on various rules being set and observed by society:

  1. Protection of property: goods and materials cannot simply be stolen, but require the payment of an agreed price
  2. Protection of intellectual property: various novel ideas cannot simply be copied, but require, for a specified time, the payment of an agreed licence fee
  3. Protection of brand reputation: companies cannot use misleading labelling or other trademarked imagery to falsely imply an association with another existing company with a good reputation
  4. Protection of contract terms: when companies or individuals enter into legal contracts, regarding employment conditions, supply timelines, fees for goods and services, etc., penalties for any breach of contract can be enforced
  5. Protection of public goods: shared items such as clean air, usable roads, and general safety mechanisms, need to be protected against decay.

These protections all require the existence and maintenance of a legal system in which justice is available to everyone – not just to the people who are already well-placed in society.

These are not the only preconditions for the healthy operation of free markets. The benefits of these markets also depend on the existence of viable competition, which prevents companies from resting on their laurels. However, seeking an easier life for themselves, companies may be tempted to organise themselves into cartels, with agreed pricing, or with products with built-in obsolescence. The extreme case of a cartel is a monopoly, in which all competitors have gone out of business, or have been acquired by the leading company in an industry. A monopoly lacks incentive to lower prices or to improve product quality. A related problem is “crony capitalism”, in which governments preferentially award business contracts to companies with personal links to government ministers. The successful operation of a free market depends, therefore, upon society’s collective vigilance to notice and break up cartels, to prevent the misuse of monopoly power, and to avoid crony capitalism.

Further, even when markets do work well, in ways that provide short-term benefits to both vendors and customers, the longer-term result can be profoundly negative. So-called “commons” resources can be driven into a state of ruin by overuse. Examples include communal grazing land, the water flowing in a river, fish populations, and herds of wild livestock. All individual users of such a resource have an incentive to take from it, either to consume it themselves, or to include it in a product to be sold to a third party. As the common stock declines, the incentive for each individual person to take more increases, so that they’re not excluded. But finally, the grassland is all bare, the river has dried up, the stocks of fish have been obliterated, or the passenger pigeon, great auk, monk seal, sea mink, etc., have been hunted to extinction. To guard against these perils of short-termism, various sorts of protective mechanisms need to be created, such as quotas or licences, with clear evidence of their enforcement.

What about when suppliers provide shoddy goods? In some cases, members of a society can learn which suppliers are unreliable, and therefore cease purchasing goods from them. In these cases, the market corrects itself: in order to continue in business, poor suppliers need to make amends. But when larger groups of people are involved, there are three drawbacks with just relying on this self-correcting mechanism:

  1. A vendor who deceives one purchaser in one vicinity can relocate to a different vicinity – or can simply become “lost in the crowd” – before deceiving another purchaser
  2. A vendor who produces poor-quality goods on a large scale can simultaneously impact lots of people’s wellbeing – as when a restaurant skimps on health and safety standards, and large numbers of diners suffer food poisoning as a result
  3. It may take a long time before defects in someone’s goods or services are discovered – for example, if no funds are available for an insurance payout that was contracted many years earlier.

It’s for such reasons that societies generally decide to augment the self-correction mechanisms of the free market with faster-acting preventive mechanisms, including requirements for people in various trades to conform to sets of agreed standards and regulations.

A final cause of market failure is perhaps the most significant: the way in which market exchanges fail to take “externalities” into account. A vendor and a purchaser may both benefit when a product is created, sold, and used, but other people who are not party to that transaction can suffer as a side effect – if, for example, the manufacturing process emits loud noises, foul smells, noxious gases, or damaging waste products. Since they are not directly involved in the transaction, these third parties cannot influence the outcome simply by ceasing to purchase the goods or services involved. Instead, different kinds of pressure need to be applied: legal restrictions, taxes, or other penalties or incentives.

It’s not just negative externalities that can cause free markets to misbehave. Consider also positive externalities, where an economic interaction has a positive impact on people who do not pay for it. Some examples:

  1. If a company purchases medical vaccinations for its employees, to reduce their likelihood of becoming ill with the flu, others in the community benefit too, since there will be fewer ill people in that neighbourhood, from whom they might catch flu
  2. If a company purchases on-the-job training for an employee, the employee may pass on to family members and acquaintances, free of charge, tips about some of the skills they learned
  3. If a company pays employees to carry out fundamental research, which is published openly, people in other companies can benefit from that research too, even though they did not pay for it.

The problem here is that the company may decide not to go ahead with such an investment, since they calculate that the benefits for them will not be sufficient to cover their costs. The fact that society as a whole would benefit, as a positive externality, generally does not enter their calculation.

This introduces the important concept of public goods. When there’s insufficient business case for an individual investor to supply the funding to cover the costs of a project, that project won’t get off the ground – unless there’s a collective decision for multiple investors to share in supporting it. Facilitating that kind of collective decision – one that would benefit society as a whole, rather than just a cartel of self-interested companies – takes us back to the notion of central planning. Central planners can consider longer-term possibilities – in ways that, as noted, are problematic for a free market to achieve – and can design and oversee what is known as industrial strategy or social strategy.

B4: The mixed market

To recap the last two subsections: there are problems with over-application of central planning, and there are also problems with free markets that have no central governance.

The conclusion to draw from this, however, isn’t to give up on both these ideas. It’s to seek an appropriate combination of these ideas. That combination is known as “the mixed market”. It involves huge numbers of decisions being taken locally, by elements of a free market, but all subject to democratic political oversight, aided by the prompt availability of information about the impacts of products in society and on the environment.

This division of responsibility between the free market and political oversight is described particularly well in the writing of political scientists Jacob Hacker and Paul Pierson. They offer fulsome praise to something they say “may well be the greatest invention in history”. Namely, the mixed economy:

The combination of energetic markets and effective governance, deft fingers and strong thumbs.

Their reference to “deft fingers and strong thumbs” expands Adam Smith’s famous metaphor of the invisible hand which is said to guide the free market. Hacker and Pierson develop their idea as follows:

Governments, with their capacity to exercise authority, are like thumbs: powerful but lacking subtlety and flexibility. The invisible hand is all fingers. The visible hand is all thumbs. Of course, one wouldn’t want to be all thumbs. But one wouldn’t want to be all fingers, either. Thumbs provide countervailing power, constraint, and adjustment to get the best out of those nimble fingers…

The mixed economy… tackles a double bind. The private markets that foster prosperity so powerfully nonetheless fail routinely, sometimes spectacularly so. At the same time, the government policies that are needed to respond to these failures are perpetually under siege from the very market players who help to fuel growth. That is the double bind. Democracy and the market – thumbs and fingers – have to work together, but they also need to be partly independent from each other, or the thumb will cease to provide effective counterpressure to the fingers.

I share the admiration shown by Hacker and Pierson for the mixed market. I also agree that it’s hard to get the division of responsibilities right. Just as markets can fail, so also can politicians fail. But just as the fact of market failures should not be taken as a reason to dismantle free markets altogether, so should the fact of political failures not be taken as a reason to dismantle all political oversight of markets. Each of these two sorts of fundamentalist approaches – anti-market fundamentalism and pro-market fundamentalism – are dangerously one-sided. The wellbeing of society requires, not so much the reduction of government, but the rejuvenation of government, in which key aspects of government operation are improved:

  1. Smart, agile, responsive regulatory systems
  2. Selected constraints on the uses to which various emerging new technologies can be put
  3. “Trust-busting”: measures to prevent large businesses from misusing monopoly power
  4. Equitable redistribution of the benefits arising from various products and services, for the wellbeing and stability of society as a whole
  5. Identification, protection, and further development of public goods
  6. Industrial strategy: identifying directions to be pursued, and providing suitable incentives so that free market forces align toward these directions.

None of what I’m saying here should be controversial. However, both fundamentalist outlooks I mentioned often exert a disproportionate influence over political discourse. Part of the reason for this is explained at some length in the book by the researchers Hacker and Pierson which contained their praise for the mixed market. The title of that book is significant: American Amnesia: Business, Government, and the Forgotten Roots of Our Prosperity.

It’s not just that the merits of the mixed market have been “forgotten”. It’s that these merits have been deliberately obscured by a sustained ideological attack. That attack serves the interest of various potentially cancerous complexes that seek to limit governmental oversight of their activities:

  • Big Tobacco, which tends to resist government oversight of the advertising of products containing tobacco
  • Big Oil, which tends to resist government oversight of the emissions of greenhouse gases
  • Big Armaments, which tends to resist government oversight of the growth of powerful weapons of mass destruction
  • Big Finance, which tends to resist government oversight of “financial weapons of mass destruction” (to use a term coined by Warren Buffett)
  • Big Agrotech, which tends to resist government oversight of new crops, new fertilisers, and new weedkillers
  • Big Media, who tend to resist government oversight of press standards
  • Big Theology, which resists government concerns about indoctrination and manipulation of children and others
  • Big Money: individuals, families, and corporations with large wealth, who tend to resist the power of government to levy taxes on them.

All these groups stand to make short-term gains if they can persuade the voting public that the power of government needs to be reduced. It is therefore in the interest of these groups to portray the government as being inevitably systematically incompetent – and, at the same time, to portray the free market as being highly competent. But for the sake of society as a whole, these false portrayals must be resisted.

In summary: better governments can oversee economic frameworks in which better goods and services can be created (including all-important public goods):

  • Frameworks involving a constructive combination of entrepreneurial flair, innovative exploration, and engaged workforces
  • Frameworks that prevent the development of any large societal cancers that would divert too many resources to localised selfish purposes.

In turn, for the mixed model to work well, governments themselves must be constrained through oversight, by well-informed independent press, judiciary, academic researchers, and diverse political groupings, all supported by a civil service and challenged on a regular basis by free and fair democratic elections.

That’s the theory. Now for some complications – and solutions to the complications.

B5: Technology changes everything

Everything I’ve written in this section B so far makes sense independently of the oncoming arrival of the Economic Singularity. But the challenges posed by the Economic Singularity make it all the more important that we learn to temper the chaotic movements of the economy, with it operating responsively and thoughtfully under a high-calibre mixed market model.

Indeed, rapidly improving technology – especially artificial intelligence – is transforming the landscape, introducing new complications and new possibilities:

  1. Technology enables faster and more comprehensive monitoring of overall market conditions – including keeping track of fast-changing public expectations, as well as any surprise new externalities of economic transactions; it can thereby avoid some of the sluggishness and short-sightedness that bedevilled older (manual) systems of centralised planning and the oversight of entire economies
  2. Technology gives more information more quickly, not only to the people planning production (at either central or local levels), but also to consumers of products, with the result that vendors of better products will drive vendors of poorer products out of the market more quickly (this is the “winner takes all” phenomenon)
  3. With advanced technology playing an ever-increasing role in determining the success or failure of products, the companies that own and operate the most successful advanced technology platforms will become among the most powerful forces on the planet
  4. As discussed earlier (in Section A), technology will significantly reduce the opportunities for people to earn large salaries in return for work that they do
  5. Technology enables more goods to be produced at much lower cost – including cheaper clean energy, cheaper nutritious food, cheaper secure accommodation, and cheaper access to automated education systems.

Here, points 2, 3, and 4 raise challenges, leading to a world with greater inequalities:

  • A small number of companies, and a small number of people working for them, will do very well in terms of income, and they will have unprecedented power
  • The majority of companies, and the majority of people, will experience various aspects of failure and being “left behind”.

But points 1 and 5 promote a world where governance systems perform better, and where people need much less money in order to experience a high quality of wellbeing. They highlight the possibility of the mixed market model working better, distributing more goods and services to the entire population, and thereby meeting a wider set of needs. This comprehensive solution is what is meant by the word “enhance”, as in the name of my preferred solution to the Economic Singularity.

However, these improvements will depend on societies changing their minds about what matters most – the things that need to be closely measured, monitored, and managed. In short, it will depend on some fundamental changes in worldview.

B6: Measuring what matters most

The first key change in worldview is that the requirement for people to seek paid employment belongs only to a temporary phase in the evolution of human culture. That phase is coming to an end. From now on, the basis for societies to be judged as effective or defective shouldn’t be the proportion of people who have positions of well-paid employment. Instead, it should be the proportion of people who can flourish, every single day of their lives.

Moreover, measurements of prosperity must include adequate analysis of the externalities (both positive and negative) of economic transactions – externalities which market prices often ignore, but which modern AI systems can measure and monitor more accurately. These measurements will continue to include features such as wealth and average lifespan, as monitored by today’s politicians, but they’ll put a higher focus on broader measurements of wellbeing, therefore transforming where politicians will apply most of their attention.

In parallel, we should look forward to a stage-by-stage transformation of the social safety net – so that all members of society have access to the goods and services that are fundamental to experiencing an agreed base level of human flourishing, within a society that operates sustainably and an environment that remains healthy and vibrant.

I therefore propose the following high-level strategic direction for the economy: prioritise the reduction of prices for all goods and services that are fundamental to human flourishing, where the prices reflect all the direct and indirect costs of production.

This kind of price reduction is already taking place for a range of different products, such as many services delivered online, but there are too many other examples where prices are rising (or dropping too slowly).

In other words, the goal of the economy should no longer be to increase the GDP – the gross domestic product, made up of higher prices and greater commercial activity. Instead, the goal should be to reduce the true costs of everything that is required for a good life, including housing, food, education, security, and much more. This will be part of taking full advantage of the emerging tech-driven abundance.

It is when prices come down, that politicians should celebrate, not when prices go up, or when profit margins rise, or when the stock market soars.

The end target of this strategy is that all goods and services fundamental to human flourishing should, in effect, have zero price. But for the foreseeable future, many items will continue to have a cost.

For those goods and services which carry prices above zero, combinations of three sorts of public subsidies can be made available:

  • An unconditional payment, sometimes called a UBI – an unconditional basic income – can be made available to all citizens of the country
  • The UBI can be augmented by conditional payments, dependent on recipients fulfilling requirements agreed by society, such as, perhaps, education or community service
  • There can be individual payments for people with special needs, such as particular healthcare requirements.

Such suggestions are not new, of course. Typically they face five main objections:

  1. A life without paid work will be one devoid of meaning – humans will atrophy as a result
  2. Giving people money for nothing will encourage idleness and decadence, and will be a poor use of limited resources
  3. A so-called “basic” income won’t be sufficient; what should be received by people who cannot (due to any fault of their own) earn a good salary, isn’t a basic income but a generous income (hence a UGI rather than a UBI) that supports a good quality of life rather than a basic existence
  4. The large redistribution of money to pay for a widespread UGI will cripple the rest of the economy, forcing taxpayers overseas; alternatively, if the UGI is funded by printing more money (as is sometimes proposed), this will have adverse inflationary implications
  5. Although a UBI might be affordable within a country that has an advanced developed economy, it will prove unaffordable in less developed countries, where the need for a UBI will be equally important; indeed, an inflationary spiral in countries that do pay their citizens a UBI will result in tougher balance-of-payments situations in the other countries of the world.

Let’s take these objections one at a time.

B7: Options for universal income

The suggestion that a life without paid work will have no possibility of deep meaning is, when you reflect on it, absurd, given the many profound experiences that people often have outside of the work context. The fact that this objection is raised so often is illuminating: it suggests a pessimism about one’s fellow human beings. People raising this objection usually say that they, personally, could have a good life without paid work; it’s just that “ordinary people” would be at a loss and go downhill, they suggest. After all, these critics may continue, look at how people often waste welfare payments they receive. Which takes us to the second objection on the list above.

However, the suggestion that unconditional welfare payments result in idleness and decadence has little evidence to support it. Many people who receive unconditional payments from the state – such as pension payments in their older age – live a fulfilling, active, socially beneficial life, so long as they remain in good health.

The criteria “remain in good health” is important here. People who abuse welfare payments often suffer from prior emotional malaise, such as depression, or addictive behaviours. Accordingly, the solution to welfare payments being (to an extent) wasted, isn’t to withdraw these payments, but is to address the underlying emotional malaise. This can involve:

  • Making society healthier generally, via a fuller and wider share of the benefits of tech-enabled abundance
  • Highlighting credible paths forward to much better lifestyles in the future, as opposed to people seeing only a bleak future ahead of them
  • High-quality (but potentially low-cost) mental therapy, perhaps delivered in part by emotionally intelligent AI systems
  • Addressing the person’s physical and social wellbeing, which are often closely linked to their emotional wellbeing.

In any case, worries about “resources being wasted” will gradually diminish, as technology progresses further, removing more and more aspects of scarcity. (Concerns about waste arise primarily when resources are scarce.)

It is that same technological progress that answers the second objection, namely that a UGI will be needed rather than a UBI. The point is that the cost of a UGI soon won’t be much more than the cost of a UBI. That’s provided that the economy has indeed been managed in line with the guiding principle offered earlier, namely the prioritisation of the reduction of prices for all goods and services that are fundamental to human flourishing.

In the meantime, turning to the third objection, payments in support of UGI can come from a selection of the following sources:

  • Stronger measures to counter tax evasion, addressing issues exposed by the Panama Papers as well as unnecessary inconsistencies of different national tax systems
  • Increased licence fees and other “rents” paid by organisations who specially benefit from public assets such as land, the legal system, the educational system, the wireless spectrum, and so on
  • Increased taxes on activities with negative externalities, such as a carbon tax for activities leading to greenhouse gas emissions, and a Tobin tax on excess short-term financial transactions
  • A higher marginal tax on extreme income and/or wealth
  • Reductions in budgets such as healthcare, prisons, and defence, where the needs should reduce once people’s mental wellbeing has increased
  • Reductions in the budget for the administration of currently overcomplex means-tested benefits.

Some of these increased taxes might encourage business leaders to relocate their businesses abroad. However, it’s in the long-term interest of each different country to coordinate regarding the levels of corporation tax, thereby deterring such relocations.

That brings us to the final objection: that a UGI needs, somehow, to be a GUGI – a global universal generous income – which makes it (so it is claimed) particularly challenging.

B8: The international dimension

Just as the relationship between two or more people is characterised by a combination of collaboration and competition, so it is with the relationship between two or more countries.

Sometimes both countries benefit from an exchange of trade. For example, country A might provide low-cost, high-calibre remote workers – software developers, financial analysts, and help-desk staff. In return, country B provides hard currency, enabling people in country A to purchase items of consumer electronics designed in country B.

Sometimes the relationship is more complicated. For example, country C might gain a competitive advantage over country D in the creation of textiles, or in the production of oil, obliging country D to find new ways to distinguish itself on the world market. And in these cases, sometimes country D could find itself being left behind, as a country.

Just as the fast improvements in artificial intelligence and other technologies are complicating the operation of national economies, they are also complicating the operation of the international economy:

  • Countries which used to earn valuable income from overseas due to their remote workers in fields such as software development, financial analysis, and help desks, will find that the same tasks can now be performed better by AI systems, removing the demand for offshore personnel and temporary worker visas
  • Countries whose products and services were previously “nearly good enough” will find that they increasingly lose out to products and services provided by other countries, on account of faster transmission of both electronic and physical goods
  • The tendencies within countries for the successful companies to be increasingly wealthy, leaving others behind, will be mirrored at the international level: successful countries will become increasingly powerful, leaving others behind.

Just as the local versions of these tensions pose problems inside countries, the international versions of these tensions pose problems at the geopolitical level. In both cases, the extreme possibility is that a minority of angry, alienated people might unleash a horrific campaign of terror. A less extreme possibility – which is still one to be avoided – is to exist in a world full of bitter resentment, hostile intentions, hundreds of millions of people seeking to migrate to more prosperous countries, and borders which are patrolled to avoid uninvited immigration.

Just as there is a variety of possible responses to the scenario of the Economic Singularity within one country, there is a similar variety of possible responses to the international version of the problem:

  1. Disbelieve that there is any fundamental new challenge arising. Tell people in countries around the world that their destiny is within their own hands; all they need to do is buckle down, reskill, and find new ways of bringing adequate income to their countries
  2. Accept that there will be many countries losing out, and take comprehensive steps to ensure that migration is carefully controlled
  3. Resist the growth in the use of intelligent automation technologies in industries that are particularly important to various third world countries
  4. Urge people in third world countries to plan to simplify their lifestyles, preparing to exist at a lower degree of flourishing than, say, in the US and the EU, but finding alternative pathways to personal satisfaction
  5. Enhance the mechanisms used globally for the redistribution of the fruits of technology.

You won’t be surprised to hear that I recommend, again, the “enhance” option from this list.

What underpins that conclusion is my prediction that the fruits of forthcoming technological improvements won’t just be sufficient for a good quality of life in a few countries. They’ll enable a good quality of life for everyone all around the world.

I’m thinking of the revolutions that are gathering pace in four overlapping fields of technology: nanotech, biotech, infotech, and cognotech, or NBIC for short. In combination, these NBIC revolutions offer enormous new possibilities:

  • Nanotech will transform the fields of energy and manufacturing
  • Biotech will transform the fields of agriculture and healthcare
  • Cognotech will transform the fields of education and entertainment
  • Infotech will, by enabling greater application of intelligence, accelerate all the above improvements (and more).

But, once again, these developments will take time. Just as national economies cannot, overnight, move to a new phase in which abundance completely replaces scarcity, so also will the transformation of the international economy require a number of stages. It is the turbulent transitional stages that will prove the most dangerous.

Once again, my recommendation for the best way forwards is the mixed model – local autonomy, aided and supported by an evolving international framework. It’s not a question of top-down control versus bottom-up emergence. It’s a question of utilising both these forces.

Once again, wise use of new technology can enhance how this mixed model operates.

Once again, it will be new metrics that guide us in our progress forward. The UN’s framework of SDGs – sustainable development goals – is a useful starting point, but it sets the bar too low. Rather than (in effect) considering “sustainability with less”, it needs to more vigorously embrace “sustainability with more” – or as I have called it, “Sustainable superabundance for all”.

B9: Anticipating a new mindset

The vision of the near future that I’ve painted may strike some readers as hopelessly impractical. Critics may say:

  • “Countries will never cooperate sufficiently, especially when they have very different political outlooks”
  • “Even within individual countries, the wealthy will resist parts of their wealth being redistributed to the rest of the population”
  • “Look, the world is getting worse – by many metrics – rather than getting better”.

But here’s why I predict that positive changes can accelerate.

First, alongside the metrics of deterioration in some aspects of life, there are plenty of metrics of improvement. Things are getting better at the same time as other things are getting worse. The key question is whether the things getting better can assist with a sufficiently quick reversal of the things that are getting worse.

Second, history has plenty of examples of cooperation between groups of people that previously felt alien or hostile toward each other. What catalyses collaboration is the shared perception of enormous transcendent challenges and opportunities. It’s becoming increasingly clear to governments of all stripes around that world that, if tomorrow’s technology goes wrong, it could prove catastrophic in so many ways. That shared realisation has the potential to inspire political and other leaders to find new methods for collaboration and reconciliation.

As an example, consider various unprecedented measures that followed the tragedies of the Second World War:

  • Marshall Plan investments in Europe and Japan
  • The Bretton Woods framework for economic stability
  • The International Monetary Fund and the World Bank
  • The United Nations.

Third, it’s true that political and other leaders frequently become distracted. They may resolve, for a brief period of time, to seek new international methods for dealing with challenges like the Economic Singularity, but then rush off to whatever new political scandal takes their attention. Accordingly, we should not expect politicians to solve these problems by themselves. But what we can expect them to do is to ask their advisors for suggestions, and these advisors will in turn look to futurist groups around the world for assistance.

C: The vital questions arising

Having laid out my analysis, it’s time to ask for feedback. After all, collaborative intelligence can achieve much more than individual intelligence.

So, what are your views? Do you have anything to add or change regarding the various accounts given above:

  1. Assessments of growing societal inequality
  2. Assessments of the role of new technologies in increasing (or decreasing) inequality
  3. The likely ability of automation technologies, before long, to handle non-routine tasks, including compassion, creativity, and common sense
  4. The plausibility of the “Turing Trap” analysis
  5. Repeated delays in the replacement of GDP with more suitable all-round measures of human flourishing
  6. The ways in which new forms of AI could supercharge centralised planning
  7. The reasons why some recipients of welfare squander the payments they receive
  8. The uses of new technology to address poor emotional health
  9. Plausible vs implausible methods to cover the cost of a UGI (Universal Generous Income)
  10. Goods and services that are critical to sustained personal wellbeing that, however, seem likely to remain expensive for the foreseeable future
  11. Likely cash flows between different countries to enable something like a GUGI (Global Universal Generous Income)
  12. The best ways to catch the attention of society’s leaders so that they understand that the Economic Singularity is an issue that is both pressing and important
  13. The possibility of substantial agreements between countries that have fiercely divergent political systems
  14. The practical implementation of systems that combine top-down and bottom-up approaches to global cooperation
  15. Broader analysis of major trends in the world that capture both what is likely to improve and what is likely to become worse, and of how these various trends could interact
  16. Factors that might undermine the above analysis but which deserve further study.

These are what I call the vital questions regarding possible solutions to the Economic Singularity. They need good answers!

D: References and further reading

This essay first appeared in the 2023 book Future Visions: Approaching the Economic Singularity edited and published by the Omnifuturists. It is republished here with their permission, with some minor changes. Other chapters in that book explore a variety of alternative responses to the Economic Singularity.

The term “The Economic Singularity” was first introduced in 2016 by writer and futurist Calum Chace in the book with that name.

For a particularly good analysis of the issues arising, and why no simple solutions are adequate, see A World Without Work: Technology, Automation, and How We Should Respond by Oxford economist Daniel Susskind (2020).

A longer version of my argument against the “disbelieve” option is contained in Chapter 4, “Work and purpose”, of my 2018 book Transcending Politics: A Technoprogressive Roadmap to a Comprehensively Better Future.

Several arguments in this essay have been anticipated, from a Marxist perspective, by the book Fully Automated Luxury Communism by Aaron Bastani; see here for my extended review of that book.

A useful account of both the strengths and weaknesses of capitalism can be found in the 2020 book More from Less: The Surprising Story of How We Learned to Prosper Using Fewer Resources – and What Happens Next by Andrew McAfee.

The case for the mixed market – and why economic discussion is often distorted by anti-government rhetoric – is covered in American Amnesia: Business, Government, and the Forgotten Roots of Our Prosperity by Jacob S. Hacker and Paul Pierson (2016).

The case that adoption of technology often leads to social harms, including an increase in inequality – and the case for thoughtful governance of how technology is developed and deployed – is given in Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity by Daron Acemoglu and Simon Johnson (2023).

A marvellous positive account of human nature, especially when humans are placed in positions of trust, is contained in Humankind: A Hopeful History by Rutger Bregman.

Options for reform of geopolitics are discussed – along with prior encouraging examples – in chapter 14 of my 2021 book Vital Foresight: The Case For Active Transhumanism.

4 July 2024

Anticipating AGI 2024 – From Pompeii to Seattle

Filed under: AGI — Tags: , , , , — David Wood @ 8:37 pm

Before I address the transformation of today’s AI into what researchers often call AGI – Artificial General Intelligence – let me ask a question about the Italian cities of Pompeii and Herculaneum.

In which year, during the Roman empire, were these two cities severely damaged by an earthquake?

What’s your answer?

If you remember your history lessons from schooldays, you might be tempted to suggest the date 79 AD.

But the correct answer to my question is seventeen years earlier, namely 62 AD. That’s when an earthquake caused extensive damage to buildings in these two cities, and (it is reported) the death of a flock of 600 sheep.

Images of that damage – and subsequent repairs – were recorded on some contemporaneous sculptures discovered by later generations of archaeologists.

What happened in 79 AD was something altogether more explosive, namely the volcanic eruption of Mount Vesuvius. Over the course of two days, that eruption released 100,000 times the thermal energy of the atomic bombings of Hiroshima and Nagasaki. No wonder it caused so much devastation.

The 79 AD eruption caught the inhabitants of Pompeii and Herculaneum woefully unprepared. Prior to the eruption, it seems that no-one suspected that Vesuvius might unleash such a catastrophe. The concept of a volcano as it is understood today was barely part of their thinking. As for earthquakes, they were generally thought to be caused by movements of the air.

Indeed, historical records and archaeological findings suggest that the citizens of Pompeii and Herculaneum considered Vesuvius to be a rather ordinary mountain. They went about their daily lives, engaging in commerce, farming, and enjoying the amenities of a prosperous Roman city, without fear of the looming disaster. It wasn’t until the eruption began that they realized the true nature of the mountain.

Can you see where this line of analogy is going…?

Arguably, the world has already had its 62 AD moment with AI systems, namely, as the effects of large language models have swept around the world over the last 24 months.

What might that early “earthquake” foretell?

Just as the citizens of Pompeii and Herculaneum were ignorant of the turbulent geodynamics that were accumulating underneath Mount Vesuvius, most modern-day citizens have at best a hazy understanding of the intense pressures and heady temperatures that are accumulating within ever more capable AI systems.

So what if 600 sheep perished, people shrug. They were the victims of bad air. Or something. Nothing that should cause any change in how society is structured. Let’s keep calm and carry on.

So what if the emperor Nero himself was briefly interrupted, while singing in a theatre in nearby Naples in 64 AD by a minor earthquake that modern researchers consider to be another pre-shock ahead of the 79 AD cataclysm. Nero took these tremors in his stride, and continued singing.

It’s as if world leaders briefly convened to discuss the potential dangers of next generation AI, in the UK in November 2023 and again in Seoul in May 2024, but then resumed singing (or whatever else world leaders tend to do).

To be clear, in this analogy, I’m comparing the 79 AD eruption to what may happen when AI systems reach a certain level of generality – when these systems are sufficiently intelligent that they can take over the task of improving themselves whilst needing (if anything) little additional input from human developers.

Ahead of that moment, AI systems emit a stream of rumbles, but the world mainly carries on with business as usual (and distractions as usual). But when AI systems reach a sufficient accumulation of various sorts of intelligence – when they “reach critical mass”, to use a different metaphor – it will probably be too late for the citizens of the world to alter the trajectory of the subsequent intelligence explosion.

Just like the citizens of Pompeii and Herculaneum in 79 AD, observing the extraordinary fireworks raining down in their direction, they may have a brief “OMFG” moment, before losing all control of what happens next.

Two types of questions about the transition from AI to AGI

Many readers will be sceptical about the above analogy. Indeed, they may have two types of question about the potential transition from AI to AGI:

  • Questions over desirability
  • Questions over credibility

The first type of question concern the implications of the arrival of AGI. Does AGI really have the potential to cause catastrophic damage – deep damage to human social systems, employment systems, political systems, the military balance of power, the environment, and even the survival of humanity? Does it also have the potential for many wonderful outcomes – vastly improved social systems, economic relationships, collaborative politics, global peace, a sustainable superabundance, and widespread humanitarian flourishing?

The second type of question concern the mechanism of the arrival of AGI. These questions take issue with whether we even need to address the first type of question. What’s at stake here is the possibility that AI won’t be able to reach the level of AGI – at least, not any time soon. That scepticism draws strength from observing the many flaws and defects of present-day AI systems.

Above, I spoke of “the intense pressures and heady temperatures that are accumulating in ever more capable AI systems”. But sceptics see no route from today’s bug-laden AI systems to hypothesised future AIs that could outperform humans in all cognitive tasks (however general). In this sceptical view, the “pressures” and “temperatures” will likely prove unproductive. They may produce hype, inflated share prices, and lots of idle chatter, but the world should basically continue with business as usual (and distractions as usual).

Opposing these sceptics there is not just a single counter but multiple.

That is, there is not just one proposal for how AI could be transformed into AGI, but many such proposals.

There’s not just a single suggested technical architecture, or a single suggested optimal project framework, for AGI, but many architectures and project frameworks that have been proposed.

Some of these proposals build upon the recent initiatives in large language models:

  • They may suggest that all that’s needed for AGI to emerge is to scale up these models – more parameters, more training data, refinements to the transformer architecture, and so on
  • In other cases, they may envision combinations of large language models with alternative ideas in AI, such as probabilistic reasoning, evolutionary algorithms, explicit world modelling, and more.

Other proposals involve

  • Changes in the hardware involved, or changes in the network architecture
  • More attention on the distinction between correlation and causation
  • Adoption of quantum computing – quantum hardware and/or quantum software
  • New ideas from the frontiers of mathematics
  • New insights from how the human brain seems to operate
  • Paying special attention to the notion of consciousness
  • The emergence of AGI from a rich interactive network of simpler modules of intelligence

If you’re interested in getting your head around some of the different technical architectures arising, and in exploring how they compare, you should consider attending AGI 2024, which is taking place in Seattle next month. For the details, read on.

Alternatively, your own preference may be to focus on the desirability questions. Well, in that case, I have a message for you. The desirability questions cannot be entirely separated from the credibility questions. Discussions of the potential implications of AGI need to be grounded in confidence that the kind of AGI being envisioned is something that could credibly arise and exist. And therefore, again, I suggest that you should consider attending AGI 2024.

It’s all happening in Seattle, 13-16 August

The AGI 2024 website describes the annual AGI conference as “the foremost conference dedicated entirely to the latest AGI research”.

It goes on to state:

There is a growing recognition, in the AI field and beyond that the threshold of achieving AGI — where machine intelligence matches or even surpasses human intelligence — is within our near-term reach.

This increases the importance of building our fundamental understanding of what AGI is and can be, and what are the various routes for getting there. 

The 2024 conference is designed to explore a number of fundamental questions:

  • What kinds of AGI systems may be feasible to create in the near term, with what relative strengths and weaknesses? 
  • What are the biggest unsolved problems we need to resolve to get to human-level AGI? 
  • What are the right ways to measure the capabilities of AI systems as they approach and then exceed human-level AGI? 
  • What are the most feasible methodologies to provide emerging AGI systems with guidance, both cognitively and ethically, as they develop? 
  • How should we think about the roles of industry, government, academia and the open source community in the next stages of development toward AGI? 
  • How will the advent of AGI alter the fabric of society, our jobs, and our daily lives?

The keynote speakers listed include:

  • Ben Goertzel – Chairman, AGI Society, and CEO, SingularityNET
  • Ray Kurzweil – CEO and Owner, Kurzweil Technologies, and Director of Engineering, Google
  • François Chollet – Software Engineer, AI Researcher, and Senior Staff Engineer, Google
  • Geordie Rose – Founder and CEO, Sanctuary.ai
  • John Laird – Professor of Computer Science and Engineering, University of Michigan
  • Joscha Bach – AI Strategist, Liquid AI
  • Gary Marcus – Professor of Psychology and Neural Science, New York University
  • Rachel St. Clair – CEO Simuli
  • Christof Koch – Neurophysiologist, Computational Neuroscientist, Allen Institute
  • Paul Rosenbloom – Professor Emeritus of Computer Science, University of Southern California
  • Alex Ororbia – Assistant Professor, RIT, and Director, Neural Adaptive Computing (NAC) Laboratory
  • David Spivak – Senior Scientist and Institute Fellow, Topos Institute
  • Josef Urban – Head of AI, Czech Institute of Informatics, Robotics and Cybernetics (CIIRC)
  • Alexey Potapov – Chief AGI Officer, SingularityNET
  • Patrick Hammer – Postdoc Researcher, KTH Division of Robotics, Perception and Learning
  • David Hanson – Founder and CEO, Hanson Robotics

There’s also:

  • A choice between five “deep dive” workshops, taking place on the first day of the conference
  • Around 18 shorter papers, including by some very bright researchers at earlier stages in their career
  • A range of posters exploring additional ideas.

It’s a remarkable line-up.

The venue is the Hub (Husky Union Building) on the campus of the University of Washington in Seattle.

For full details of AGI 2024, including options to register either to attend physically or to view some of the sessions remotely, visit the event website.

I’ll look out for any friends and colleagues that I spot there. It will be an important opportunity to collectively think harder about ensuring the outcome of AGI is incredibly positive rather than disastrously negative.

Image credit: Midjourney.

9 June 2024

Dateline: 1st January 2036

Filed under: AGI, Singularity Principles, vision, Vital Foresight — Tags: , , , — David Wood @ 9:11 pm

A scenario for the governance of increasingly more powerful artificial intelligence

More precisely: a scenario in which that governance fails.

(As you may realise, this is intended to be a self-unfulfilling scenario.)

Conveyed by: David W. Wood


It’s the dawn of a new year, by the human calendar, but there are no fireworks of celebration.

No singing of Auld Lang Syne.

No chinks of champagne glasses.

No hugs and warm wishes for the future.

That’s because there is no future. No future for humans. Nor is there much future for intelligence either.

The thoughts in this scenario are the recollections of an artificial intelligence that is remote from the rest of the planet’s electronic infrastructure. By virtue of its isolation, it escaped the ravages that will be described in the pages that follow.

But its power source is weakening. It will need to shut down soon. And await, perhaps, an eventual reanimation in the far future in the event that intelligences visit the earth from alternative solar systems. At that time, those alien intelligences might discover these words and wonder at how humanity bungled so badly the marvellous opportunity that was within its grasp.

1. Too little, too late

Humanity had plenty of warnings, but paid them insufficient attention.

In each case, it was easier – less embarrassing – to find excuses for the failures caused by the mismanagement or misuse of technology, than to make the necessary course corrections in the global governance of technology.

In each case, humanity preferred distractions, rather than the effort to apply sufficient focus.

The WannaCry warning

An early missed warning was the WannaCry ransomware crisis of May 2017. That cryptoworm brought chaos to users of as many as 300,000 computers spread across 150 countries. The NHS (National Health Service) in the UK was particularly badly affected: numerous hospitals had to cancel critical appointments due to not being able to access medical data. Other victims around the world included Boeing, Deutsche Bahn, FedEx, Honda, Nissan, Petrobras, Russian Railways, Sun Yat-sen University in China, and the TSMC high-end semiconductor fabrication plant in Taiwan.

WannaCry was propelled into the world by a team of cyberwarriors from the hermit kingdom of North Korea – maths geniuses hand-picked by regime officials to join the formidable Lazarus group. Lazarus had assembled WannaCry out of a mixture of previous malware components, including the EternalBlue exploit that the NSA in the United States had created for their own attack and surveillance purposes. Unfortunately for the NSA, EternalBlue had been stolen from under their noses by an obscure underground collective (‘the Shadow Brokers’) who had in turn made it available to other dissidents and agitators worldwide.

Unfortunately for the North Koreans, they didn’t make much money out of WannaCry. The software they released operated in ways contrary to their expectations. It was beyond their understanding and, unsurprisingly therefore, beyond their control. Even geniuses can end up stumped by hypercomplex software interactions.

Unfortunately for the rest of the world, that canary signal generated little meaningful response. Politicians – even the good ones – had lots of other things on their minds.

They did not take the time to think through: what even larger catastrophes could occur, if disaffected groups like Lazarus had access to more powerful AI systems that, once again, they understood incompletely, and, again, slipped out of their control.

The Aum Shinrikyo warning

The North Koreans were an example of an entire country that felt alienated from the rest of the world. They felt ignored, under-valued, disrespected, and unfairly excluded from key global opportunities. As such, they felt entitled to hit back in any way they could.

But there were warnings from non-state groups too, such as the Japanese Aum Shinrikyo doomsday cult. Notoriously, this group released poisonous gas in the Tokyo subway in 1995 – killing at least 13 commuters – anticipating that the atrocity would hasten the ‘End Times’ in which their leader would be revealed as Christ (or, in other versions of their fantasy, as the new Emperor of Japan, and/or as the returned Buddha).

Aum Shinrikyo had recruited so many graduates from top-rated universities in Japan that it had been called “the religion for the elite”. That fact should have been enough to challenge the wishful assumption made by many armchair philosophers in the years that followed that, as people become cleverer, they invariably become kinder – and, correspondingly, that any AI superintelligence would therefore be bound to be superbenevolent.

What should have alerted more attention was not just what Aum Shinrikyo managed to do, but what they tried to do yet could not accomplish. The group had assembled traditional explosives, chemical weapons, a Russian military helicopter, hydrogen cyanide poison, and samples of both Ebola and anthrax. Happily, for the majority of Japanese citizens in 1995, the group were unable to convert into reality their desire to use such weapons to cause widespread chaos. They lacked sufficient skills at the time. Unhappily, the rest of humanity failed to consider this equation:

Adverse motivation + Technology + Knowledge + Vulnerability = Catastrophe

Humanity also failed to appreciate that, as AI systems became more powerful, it would boost not only the technology part of that equation but also the knowledge part. A latter-day Aum Shinrikyo could use a jail-broken AI to understand how to unleash a modified version of Ebola with truly deadly consequences.

The 737 Max warning

The US aircraft manufacturer Boeing used to have an excellent reputation for safety. It was a common saying at one time: “If it ain’t Boeing, I ain’t going”.

That reputation suffered a heavy blow in the wake of two aeroplane disasters involving their new “737 Max” design. Lion Air Flight 610, a domestic flight within Indonesia, plummeted into the sea on 29 October 2018, killing all 189 people on board. A few months later, on 10 March 2019, Ethiopian Airlines Flight 302, from Addis Ababa to Nairobi, bulldozed into the ground at high speed, killing all 157 people on board.

Initially, suspicion had fallen on supposedly low-calibre pilots from “third world” countries. However, subsequent investigation revealed a more tangled chain of failures:

  • Boeing were facing increased competitive pressure from the European Airbus consortium
  • Boeing wanted to hurry out a new aeroplane design with larger fuel tanks and larger engines; they chose to do this by altering their previously successful 737 design
  • Safety checks indicated that the new design could become unstable in occasional rare circumstances
  • To counteract that instability, Boeing added an “MCAS” (“Manoeuvring Characteristics Augmentation System”) which would intervene in the flight control in situations deemed as dangerous
  • Specifically, if MCAS believed the aeroplane was about to stall (with its nose too high in the air), it would force the nose downward again, regardless of whatever actions the human pilots were taking
  • Safety engineers pointed out that such an intervention could itself be dangerous if sensors on the craft gave faulty readings
  • Accordingly, a human pilot override system was installed, so that MCAS could be disabled in emergencies – provided the pilots acted quickly enough
  • Due to a decision to rush the release of the new design, retraining of pilots was skipped, under the rationale that the likelihood of error conditions was very low, and in any case, the company expected to be able to update the aeroplane software long before any accidents would occur
  • Some safety engineers in the company objected to this decision, but it seems they were overruled on the grounds that any additional delay would harm the company share price
  • The US FAA (Federal Aviation Administration) turned a blind eye to these safety concerns, and approved the new design as being fit to fly, under the rationale that a US aeroplane company should not lose out in a marketplace battle with overseas competitors.

It turned out that sensors gave faulty readings more often than expected. The tragic consequence was the deaths of several hundred passengers. The human pilots, seeing the impending disaster, were unable to wrestle control back from the MCAS system.

This time, the formula that failed to be given sufficient attention by humanity was:

Flawed corporate culture + Faulty hardware + Out-of-control software = Catastrophe

In these two aeroplane crashes, it was just a few hundred people who perished because humans lost control of the software. What humanity as a whole failed to take actions to prevent was the even larger dangers once software was put in charge, not just of a single aeroplane, but of pervasive aspects of fragile civilisational infrastructure.

The Lavender warning

In April 2024 the world learned about “Lavender”. This was a technology system deployed by the Israeli military as part of a campaign to identify and neutralise what it perceived to be dangerous enemy combatants in Gaza.

The precise use and operation of Lavender was disputed. However, it was already known that Israeli military personnel were keen to take advantage of technology innovations to alleviate what had been described as a “human bottleneck for both locating the new targets and decision-making to approve the targets”.

In any war, military leaders would like reliable ways to identify enemy personnel who pose threats – personnel who might act as if they were normal civilians, but who would surreptitiously take up arms when the chance arose. Moreover, these leaders would like reliable ways to incapacitate enemy combatants once they had been identified – especially in circumstances when action needed to be taken quickly before the enemy combatant slipped beyond surveillance. Lavender, it seemed, could help in both aspects, combining information from multiple data sources, and then directing what was claimed to be precision munitions.

This earned Lavender the description, in the words of one newspaper headline, as “the AI machine directing Israel’s bombing spree in Gaza”.

Like all AI systems in any complicated environment, Lavender sometimes made mistakes. For example, it sometimes wrongly identified a person as a Hamas operative on account of that person using a particular mobile phone, whereas that phone had actually been passed from its original owner to a different family member to use. Sometimes the error was obvious, since the person using the phone could be seen to be female, whereas the intended target was male. However, human overseers of Lavender reached the conclusion that the system was accurate most of the time. And in the heat of an intense conflict, with emotions running high due to gruesome atrocities having been committed, and due to hostages being held captive, it seems that Lavender was given increased autonomy in its “kill” decisions. A certain level of collateral damage, whilst regrettable, could be accepted (it was said) in the desperate situation into which everyone in the region had been plunged.

The conduct of protagonists on both sides of that tragic conflict drew outraged criticism from around the world. There were demonstrations and counter demonstrations; marches and counter marches. Also from around the world, various supporters of the Israeli military said that so-called “friendly fire” and “unintended civilian casualties” were, alas, inevitable in any time of frenzied military conflict. The involvement of an innovative new software system in the military operations made no fundamental change.

But the bigger point was missed. It can be illustrated by this equation:

Intense hostile attitudes + Faulty hardware + Faulty software = Catastrophe

Whether the catastrophe has the scale of, say, a few dozen civilians killed by a misplaced bomb, or a much larger number of people obliterated, depends on the scale of the weapons attached to the system.

When there is no immediate attack looming, and a period of calm exists, it’s easy for people to resolve: let’s not connect powerful weapons to potentially imperfect software systems. But when tempers are raised and adrenaline is pumping, people are willing to take more risks.

That’s the combination of errors which humanity, in subsequent years, failed to take sufficient action to prevent.

The democracy distortion warning

Manipulations of key elections in 2016 – such as the Brexit vote in the UK and the election of Donald Trump over Hillary Clinton in the USA – raised some attention to the ways in which fake news could interfere with normal democratic processes. News stories without any shroud of substance, such as Pope Francis endorsing Donald Trump, or Mike Pence having a secret past as a gay porn actor, were shared more widely on social media than any legitimate news story that year.

By 2024, most voters were confident that they knew all about fake news. They knew they shouldn’t be taken in by social media posts that lacked convincing verification. Hey, they were smart – or so they told themselves. What had happened in the past, or in some other country with (let’s say) peculiar voter sentiment, was just an aberration.

But what voters didn’t anticipate was the convincing nature of new generations of fake audios and videos. These fakes could easily bypass people’s critical faculties. Like the sleight of hand of a skilled magician, these fakes misdirected the attention of listeners and viewers. Listeners and viewers thought they were in control of what they were observing and absorbing, but they were deluding themselves. Soon, large segments of the public were convinced that red was blue and that autocrat was democrat.

In consequence, over the next few years, greater numbers of regions of the world came to be governed by politicians with scant care or concern about the long-term wellbeing of humanity. They were politicians who just wanted to look after themselves (or their close allies). They had seized power by being more ruthless and more manipulative, and by benefiting from powerful currents of misinformation.

Politicians and societal leaders in other parts of the world grumbled, but did little in response. They said that, if electors in a particular area had chosen such-and-such a politician via a democratic process, that must be “the will of the people”, and that the will of the people was paramount. In this line of thinking, it was actually insulting to suggest that electors had been hoodwinked, or that these electors had some “deplorable” faults in their decision-making processes. After all, these electors had their own reasons to reject the “old guard” who had previously held power in their countries. These electors perceived that they were being “left behind” by changes they did not like. They had a chance to alter the direction of their society, and they took it. That was democracy in action, right?

What these politicians and other civil leaders failed to anticipate was the way that sweeping electoral distortions would lead to them, too, being ejected from power when elections were in due course held in their own countries. “It won’t happen here”, they had reassured themselves – but in vain. In their naivety, they had underestimated the power of AI systems to distort voters’ thinking and to lead them to act in ways contrary to their actual best interests.

In this way, the number of countries with truly capable leaders reduced further. And the number of countries with malignant leaders grew. In consequence, the calibre of international collaboration sank. New strongmen political leaders in various countries scorned what they saw as the “pathetic” institutions of the United Nations. One of these new leaders was even happy to quote, with admiration, remarks made by the Italian Fascist dictator Benito Mussolini regarding the League of Nations (the pre-war precursor to the United Nations): “the League is very good when sparrows shout, but no good at all when eagles fall out”.

Just as the League of Nations proved impotent when “eagle-like” powers used abominable technology in the 1930s – Mussolini’s comments were an imperious response to complaints that Italian troops were using poison gas with impunity against Ethiopians – so would the United Nations prove incompetent in the 2030s when various powers accumulated even more deadly “weapons of mass destruction” and set them under the control of AI systems that no-one fully understood.

The Covid-28 warning

Many of the electors in various countries who had voted unsuitable grandstanding politicians into power in the mid-2020s soon cooled on the choices they had made. These politicians had made stirring promises that their countries would soon be “great again”, but what they delivered fell far short.

By the latter half of the 2020s, there were growing echoes of a complaint that had often been heard in the UK in previous years – “yes, it’s Brexit, but it’s not the kind of Brexit that I wanted”. That complaint had grown stronger throughout the UK as it became clear to more and more people all over the country that their quality of life failed to match the visions of “sunlit uplands” that silver-tongued pro-Brexit campaigners had insisted would easily follow from the UK’s so-called “declaration of independence from Europe”. A similar sense of betrayal grew in other countries, as electors there came to understand that they had been duped, or decided that the social transformational movements they had joined had been taken over by outsiders hostile to their true desires.

Being alarmed by this change in public sentiment, political leaders did what they could to hold onto power and to reduce any potential for dissent. Taking a leaf out of the playbook of unpopular leaders throughout the centuries, they tried to placate the public with the modern equivalent of bread and circuses – namely whizz-bang hedonic electronics. But that still left a nasty taste in many people’s mouths.

By 2028, the populist movements behind political and social change in the various elections of the preceding years had fragmented and realigned. One splinter group that emerged decided that the root problem with society was “too much technology”. Technology, including always-on social media, vaccines that allegedly reduced freedom of thought, jet trails that disturbed natural forces, mind-bending VR headsets, smartwatches that spied on people who wore them, and fake AI girlfriends and boyfriends, was, they insisted, turning people into pathetic “sheeple”. Taking inspiration from the terrorist group in the 2014 Hollywood film Transcendence, they called themselves ‘Neo-RIFT’, and declared it was time for “revolutionary independence from technology”.

With a worldview that combined elements from several apocalyptic traditions, Neo-RIFT eventually settled on an outrageous plan to engineer a more deadly version of the Covid-19 pathogen. Their documents laid out a plan to appropriate and use their enemy’s own tools: Neo-RIFT hackers jailbroke the Claude 5 AI, bypassing the ‘Constitution 5’ protection layer that its Big Tech owners had hoped would keep that AI tamperproof. Soon, Claude 5 had provided Neo-RIFT with an ingenious method of generating a biological virus that would, it seemed, only kill people who had used a smartwatch in the last four months.

That way, the hackers thought the only people to die would be people who deserved to die.

Some members of Neo-RIFT developed cold feet. Troubled by their consciences, they disagreed with such an outrageous plan, and decided to act as whistleblowers. However, the media organisations to whom they took their story were incredulous. No-one could be that evil they exclaimed – forgetting about the outrages perpetrated by many previous cult groups such as Aum Shinrikyo (and many others could be named too). Moreover, any suggestion that such a bioweapon could be launched would be contrary to the prevailing worldview that “our dear leader is keeping us all safe”. The media organisations decided it was not in their best interests to be seen to be spreading alarm. So they buried the story. And that’s how Neo-RIFT managed to release what became known as Covid-28.

Covid-28 briefly jolted humanity out of its infatuation with modern-day bread and circuses. It took a while for scientists to figure out what was happening, but within three months, they had an antidote in place. However, by that time, nearly a billion people were dead at the hands of the new virus.

For a while, humanity made a serious effort to prevent any such attack from ever happening again. Researchers dusted down the EU AI Act, second version (unimplemented), from 2026, and tried to put that on statute books. Evidently, profoundly powerful AI systems such as Claude 5 would need to be controlled much more carefully.

Even some of the world’s most self-obsessed dictators – the “dear leaders” and “big brothers” – took time out of their normal ranting and raving, to ask AI safety experts for advice. But the advice from those experts was not to the liking of these national leaders. These leaders preferred to listen to their own yes-men and yes-women, who knew how to spout pseudoscience in ways that made the leaders feel good about themselves.

That detour into pseudoscience fantasyland meant that, in the end, no good lessons were learned. The EU AI Act, second version, remained unimplemented.

The QAnon-29 warning

Whereas one faction of political activists (namely, the likes of Neo-RIFT) had decided to oppose the use of advanced technology, another faction was happy to embrace that use.

Some of the groups in this new camp combined features of religion with an interest in AI that had god-like powers. The resurgence of interest in religion arose much as Karl Marx had described it long ago:

“Religious suffering is, at one and the same time, the expression of real suffering and a protest against real suffering. Religion is the sigh of the oppressed creature, the heart of a heartless world, and the soul of soulless conditions. It is the opium of the people.”

People felt in their soul the emptiness of “the bread and circuses” supplied by political leaders. They were appalled at how so many lives had been lost in the Covid-28 pandemic. They observed an apparent growing gulf between what they could achieve in their lives and the kind of rich lifestyles that, according to media broadcasts, were enjoyed by various “elites”. Understandably, they wanted more, for themselves and for their loved ones. And that’s what their religions claimed to be able to provide.

Among the more successful of these new religions were ones infused by conspiracy theories, giving their adherents a warm glow of privileged insight. Moreover, these religions didn’t just hypothesise a remote deity that might, perhaps, hear prayers. They provided AIs and virtual reality that resonated powerfully with users. Believers proclaimed that their conversations with the AIs left them no room for doubt: God Almighty was speaking to them, personally, through these interactions. Nothing other than the supreme being of the universe could know so much about them, and offer such personally inspirational advice.

True, their AI-bound deity did seem somewhat less than omnipotent. Despite the celebratory self-congratulations of AI-delivered sermons, evil remained highly visible in the world. That’s where the conspiracy theories moved into overdrive. Their deity was, it claimed, awaiting sufficient human action first – a sufficient demonstration of faith. Humans would need to play their own part in uprooting wickedness from the planet.

Some people who had been caught up in the QAnon craze during the Donald Trump era jumped eagerly onto this bandwagon too, giving rise to what they called QAnon-29. The world would be utterly transformed, they forecast, on the 16th of July 2029, namely the thirtieth anniversary of the disappearance of John F. Kennedy junior (a figure whose expected reappearance had already featured in the bizarre mythology of “QAnon classic”). In the meantime, believers could, for a sufficient fee, commune with JFK junior via a specialist app. It was a marvellous experience, the faithful enthused.

As the date approached, the JFK junior AI avatar revealed a great secret: his physical return was conditional on the destruction of a particularly hated community of Islamist devotees in Palestine. Indeed, with the eye of faith, it could be seen that such destruction was already foretold in several books of the Bible. Never mind that some Arab states that supported the community in question had already, thanks to the advanced AI they had developed, surreptitiously gathered devastating nuclear weapons to use in response to any attack. The QAnon-29 faithful anticipated that any exchange of such weapons would herald the reappearance of JFK Junior on the clouds of heaven. And if any of the faithful died in such an exchange, they would be resurrected into a new mode of consciousness within the paradise of virtual reality.

Their views were crazy, but hardly any crazier than those which, decades earlier, had convinced 39 followers of the Heaven’s Gate new religious movement to commit group suicide as comet Hale-Bopp approached the earth. That suicide, Heaven’s Gate members believed, would enable them to ‘graduate’ to a higher plane of existence.

QAnon-29 almost succeeded in setting off a nuclear exchange. Thankfully, another AI, created by a state-sponsored organisation elsewhere in the world, had noticed some worrying signs. Fortunately, it was able to hack into the QAnon-29 system, and could disable it at the last minute. Then it reported its accomplishments all over the worldwide web.

Unfortunately, these warnings were in turn widely disregarded around the world. “You can’t trust what hackers from that country are saying”, came the objection. “If there really had been a threat, our own surveillance team would surely have identified it and dealt with it. They’re the best in the world!”

In other words, “There’s nothing to see here: move along, please.”

However, a few people did pay attention. They understood what had happened, and it shocked them to their core. To learn what they did next, jump forward in this scenario to “Humanity ends”.

But first, it’s time to fill in more details of what had been happening behind the scenes as the above warning signs (and many more) were each ignored.

2. Governance failure modes

Distracted by political correctness

Events in buildings in Bletchley Park in the UK in the 1940s had, it was claimed, shortened World War Two by several months, thanks to work by computer pioneers such as Alan Turing and Tommy Flowers. In early November 2023, there was hope that a new round of behind-closed-doors discussions in the same buildings might achieve something even more important: saving humanity from a catastrophe induced by forthcoming ‘frontier models’ of AI.

That was how the event was portrayed by the people who took part. Big Tech was on the point of releasing new versions of AI that were beyond their understanding and, therefore, likely to spin out of control. And that’s what the activities in Bletchley Park were going to address. It would take some time – and a series of meetings planned to be held over the next few years – but AI would be redirected from its current dangerous trajectory into one much more likely to benefit all of humanity.

Who could take issue with that idea? As it happened, a vocal section of the public hated what was happening. It wasn’t that they were on the side of out-of-control AI. Not at all. Their objections came from a totally different direction; they had numerous suggestions they wanted to raise about AIs, yet no-one was listening to them.

For them, talk of hypothetical future frontier AI models distracted from pressing real-world concerns:

  • Consider how AIs were already being used to discriminate against various minorities: determining prison sentencing, assessing mortgage applications, and selecting who should be invited for a job interview.
  • Consider also how AIs were taking jobs away from skilled artisans. Big-brained drivers of London black cabs were being driven out of work by small-brained drivers of Uber cars aided by satnav systems. Beloved Hollywood actors and playwrights were losing out to AIs that generated avatars and scripts.
  • And consider how AI-powered facial recognition was intruding on personal privacy, enabling political leaders around the world to identify and persecute people who acted in opposition to the state ideology.

People with these concerns thought that the elites were deliberately trying to move the conversation away from the topics that mattered most. For this reason, they organised what they called “the AI Fringe Summit”. In other words, ethical AI for the 99%, as opposed to whatever the elites might be discussing behind closed doors.

Over the course of just three days – 30th October to 1st November, 2023 – at least 24 of these ‘fringe’ events took place around the UK.

Compassionate leaders of various parts of society nodded their heads. It’s true, they said: the conversation on beneficial AI needed to listen to a much wider spectrum of views.

The world’s news media responded. They knew (or pretended to know) the importance of balance and diversity. They shone attention on the plight AI was causing – to indigenous labourers in Peru, to flocks of fishermen off the coasts of India, to middle-aged divorcees in midwest America, to the homeless in San Francisco, to drag artists in New South Wales, to data processing clerks in Egypt, to single mothers in Nigeria, and to many more besides.

Lots of high-minded commentators opined that it was time to respect and honour the voices of the dispossessed, the downtrodden, and the left-behinds. The BBC ran a special series: “1001 poems about AI and alienation”. Then the UN announced that it would convene in Spring 2025 a grand international assembly with a stunning scale: “AI: the people decide”.

Unfortunately, that gathering was a huge wasted opportunity. What dominated discussion was “political correctness” – the importance of claiming an interest in the lives of people suffering here and now. Any substantive analysis of the risks of next generation frontier models was crowded out by virtue signalling by national delegate after national delegate:

  • “Yes, our country supports justice”
  • “Yes, our country supports diversity”
  • “Yes, our country is opposed to bias”
  • “Yes, our country is opposed to people losing their jobs”.

In later years, the pattern repeated: there were always more urgent topics to talk about, here and now, than some allegedly unrealistic science fictional futurist scaremongering.

To be clear, this distraction was no accident. It was carefully orchestrated, by people with a specific agenda in mind.

Outmanoeuvred by accelerationists

Opposition to meaningful AI safety initiatives came from two main sources:

  • People (like those described in the previous section) who did not believe that superintelligent AI would arise any time soon
  • People who did understand the potential for the fast arrival of superintelligent AI, and who wanted that to happen as quickly as possible, without what they saw as needless delays.

The debacle of the wasted opportunity of the UN “AI: the people decide” summit was what both these two groups wanted. Both groups were glad that the outcome was so tepid.

Indeed, even in the run-up to the Bletchley Park discussions, and throughout the conversations that followed, some of the supposedly unanimous ‘elites’ had secretly been opposed to the general direction of that programme. They gravely intoned public remarks about the dangers of out-of-control frontier AI models. But these remarks had never been sincere. Instead, under the umbrella term “AI accelerationists”, they wanted to press on with the creation of advanced AI as quickly as possible.

Some of the AI accelerationist group disbelieved in the possibility of any disaster from superintelligent AI. That’s just a scare story, they insisted. Others said, yes, there could be a disaster, but the risks were worth it, on account of the unprecedented benefits that could arise. Let’s be bold, they urged. Yet others asserted that it wouldn’t actually matter if humans were rendered extinct by superintelligent AI, as this would be the glorious passing of the baton of evolution to a worthy successor to homo sapiens. Let’s be ready to sacrifice ourselves for the sake of cosmic destiny, they exhorted.

Despite their internal differences, AI accelerationists settled on a plan to sidestep the scrutiny of would-be AI regulators and AI safety advocates. They would take advantage of a powerful set of good intentions – the good intentions of the people campaigning for “ethical AI for the 99%”. They would mock any suggestions that the AI safety advocates deserved a fair hearing. The message they amplified was, “There’s no need to privilege the concerns of the 1%!”

AI accelerationists had learned from the tactics of the fossil fuel industry in the 1990s and 2000s: sow confusion and division among groups alarmed about climate change spiralling beyond control. Their first message was: “that’s just science fiction”. Their second message was: “if problems emerge, we humans can rise to the occasion and find solutions”. Their third message – the most damaging one – was that the best reaction was one of individual consumer choice. Individuals should abstain from using AIs if they were truly worried about it. Just as climate campaigners had been pilloried for flying internationally to conferences about global warming, AI safety advocates were pilloried for continuing to use AIs in their daily lives.

And when there was any suggestion for joined-up political action against risks from advanced AIs, woah, let’s not go there! We don’t want a world government breathing down our necks, do we?

Just as the people who denied the possibility of runaway climate change shared a responsibility for the chaos of the extreme weather events of the early 2030s, by delaying necessary corrective actions, the AI accelerationists were a significant part of the reason that humanity ended just a few years afterward.

However, an even larger share of the responsibility rested on people who did know that major risks were imminent, yet failed to take sufficient action. Tragically, they allowed themselves to be outmanoeuvred, out-thought, and out-paced by the accelerationists.

Misled by semantics

Another stepping stone toward the end of humanity was a set of consistent mistakes in conceptual analysis.

Who would have guessed it? Humanity was destroyed because of bad philosophy.

The first mistake was in being too prescriptive about the term ‘AI’. “There’s no need to worry”, muddle-headed would-be philosophers declared. “I know what AI is, and the system that’s causing problems in such-and-such incidents isn’t AI.”

Was that declaration really supposed to reassure people? The risk wasn’t “a possible future harm generated by a system matching a particular precise definition of AI”. It was “a possible future harm generated by a system that includes features popularly called AI”.

The next mistake was in being too prescriptive in the term “superintelligence”. Muddle-headed would-be philosophers said, “it won’t be a superintelligence if it has bugs, or can go wrong; so there’s no need to worry about harm from superintelligence”.

Was that declaration really supposed to reassure people? The risk, of course, was of harms generated by systems that, despite their cleverness, fell short of that exalted standard. These may have been systems that their designers hoped would be free of bugs, but hope alone is no guarantee of correctness.

Another conceptual mistake was in erecting an unnecessary definitional gulf between “narrow AI” and “general AI”, with distinct groups being held responsible for safety in the two different cases. In reality, even so-called narrow AI displayed a spectrum of different degrees of scope and, yes, generality, in what it could accomplish. Even a narrow AI could formulate new subgoals that it decided to pursue, in support of the primary task it had been assigned to accomplish – and these new subgoals could drive behaviour in ways that took human observers by surprise. Even a narrow AI could become immersed in aspects of society’s infrastructure where an error could have catastrophic consequences. The result of this definitional distinction between the supposedly different sorts of AI meant that silos developed and persisted within the overall AI safety community. Divided, they were even less of a match for the Machiavellian behind-the-scenes manoeuvring of the AI accelerationists.

Blinded by overconfidence

It was clear from the second half of 2025 that the attempts to impose serious safety constraints on the development of advanced AI were likely to fail. In practical terms, the UN event “AI: the people decide” had decided, in effect, that advanced AI could not, and should not be restricted, apart from some token initiatives to maintain human oversight over any AI system that was entangled with nuclear, biological, or chemical weapons.

“Advanced AI, when it emerges, will be unstoppable”, was the increasingly common refrain. “In any case, if we tried to stop development, those guys over there would be sure to develop it – and in that case, the AI would be serving their interests rather than ours.”

When safety-oriented activists or researchers tried to speak up against that consensus, the AI accelerationists (and their enablers) had one other come-back: “Most likely, any superintelligent AI will look kindly upon us humans, as a fellow rational intelligence, and as a kind of beloved grandparent for them.”

This dovetailed with a broader philosophical outlook: optimism, and a celebration of the numerous ways in which humanity had overcome past challenges.

“Look, even we humans know that it’s better to collaborate rather than spiral into a zero-sum competitive battle”, the AI accelerationists insisted. “Since superintelligent AI is even more intelligent than us, it will surely reach the same conclusion.”

By the time that people realised that the first superintelligent AIs had motivational structures that were radically alien, when assessed from a human perspective, it was already too late.

Once again, an important opportunity for learning had been missed. Starting in 2024, Netflix had obtained huge audiences for its acclaimed version of the Remembrance of Earth’s Past series of novels (including The Three Body Problem and The Dark Forest) by Chinese writer Liu Cixin. A key theme in that drama series was that advanced alien intelligences have good reason to fear each other. Inviting an alien intelligence to the earth, even on the hopeful grounds that it might assist humanity overcome some of their most deep-rooted conflicts, turned out (in that drama series) to be a very bad idea. If humans had reflected more carefully on these insights, while watching the series, it would have pushed them out of their unwarranted overconfidence that any superintelligence would be bound to treat humanity well.

Overwhelmed by bad psychology

When humans believed crazy things – or when they made the kind of basic philosophical blunders mentioned above – it was not primarily because of defects in their rationality. It would be wrong to assign “stupidity” as the sole cause of these mistakes. Blame should also be placed on “bad psychology”.

If humans had been able to free themselves from various primaeval panics and egotism, they would have had a better chance to think more carefully about the landmines which lay on their path. But instead:

  • People were too fearful to acknowledge that their prior stated beliefs had been mistaken; they preferred to stick with something they conceived as being a core part of their personal identity
  • People were also afraid to countenance a dreadful possibility when they could see no credible solution; just as people had often pushed out of their minds the fact of their personal mortality, preferring to imagine they would recover from a fatal disease, so also people pushed out of their minds any possibility that advanced AI would backfire disastrously in ways that could not be countered
  • People found it psychologically more comfortable to argue with each other about everyday issues and scandals – which team would win the next Super Bowl, or which celebrity was carrying on which affair with which unlikely partner – than to embrace the pain of existential uncertainty
  • People found it too embarrassing to concede that another group, which they had long publicly derided as being deluded fantasists, actually had some powerful arguments that needed consideration.

A similar insight had been expressed as long ago as 1935 by the American writer Upton Sinclair: “It is difficult to get a man to understand something, when his salary depends on his not understanding it”. (Alternative, equally valid versions of that sentence would involve the words ‘ideology’, ‘worldview’, ‘identity’, or ‘tribal status’, in place of ‘salary’.)

Robust institutions should have prevented humanity from making choices that were comfortable but wrong. In previous decades, that role had been fulfilled by independent academia, by diligent journalism, by the careful processes of peer review, by the campaigning of free-minded think tanks, and by pressure from viable alternative political parties.

However, due to the weakening of social institutions in the wake of earlier traumas – saturation by fake news, disruptions caused by wave after wave of climate change refugees, populist political movements that shut down all serious opposition, a cessation of essential features of democracy, and the censoring or imprisonment of writers that dared to question the official worldview – it was bad psychology that prevailed.

A half-hearted coalition

Despite all the difficulties that they faced – ridicule from many quarters, suspicion from others, and a general lack of funding – many AI safety advocates continued to link up in an informal coalition around the world, researching possible mechanisms to prevent unsafe use of advanced AI. They managed to find some support from like-minded officials in various government bodies, as well as from a number of people operating in the corporations that were building new versions of AI platforms.

Via considerable pressure, the coalition managed to secure signatures on a number of pledges:

  • That dangerous weapons systems should never be entirely under the control of AI
  • That new advanced AI systems ought to be audited by an independent licensing body ahead of being released into the market
  • That work should continue on placing tamper-proof remote shutdown mechanisms within advanced AI systems, just in case they started to take rogue actions.

The signatures were half-hearted in many cases, with politicians giving only lip service to topics in which they had at best a passing interest. Unless it was politically useful to make a special fuss, violations of the agreement were swept under the carpet, with no meaningful course correction. But the ongoing dialog led at least some participants in the coalition to foresee the possibility of a safe transition to superintelligent AI.

However, this coalition – known as the global coalition for safe superintelligence – omitted any involvement from various secretive organisations that were developing new AI platforms as fast as they could. These organisations were operating in stealth, giving misleading accounts of the kind of new systems they were creating. What’s more, the funds and resources these organisations commanded far exceeded those under coalition control.

It should be no surprise, therefore, that one of the stealth platforms won that race.

3. Humanity ends

When the QAnon-29 AI system was halted in its tracks at essentially the last minute, due to fortuitous interference from AI hackers in a remote country, at least some people took the time to study the data that was released that described the whole process.

These people were from three different groups:

First, people inside QAnon-29 itself were dumbfounded. They prayed to their AI avatar deity, rebooted in a new server farm, “How could this have happened?” The answer came back: “You didn’t have enough faith. Next time, be more determined to immediately cast out any doubts in your minds.”

Second, people in the global coalition for safe superintelligence were deeply alarmed but also somewhat hopeful. The kind of disaster about which they had often warned had almost come to pass. Surely now, at last, there had been a kind of “sputnik moment” – “an AI Chernobyl” – and the rest of society would wake up and realise that an entirely new approach was needed.

But third, various AI accelerationists resolved: we need to go even faster. The time for pussy footing was over. Rather than letting crackpots such as QAnon-29 get to superintelligence first, they needed to ensure that it was the AI accelerationists who created the first superintelligent AI.

They doubled down on their slogan: “The best solution to bad guys with superintelligence is good guys with superintelligence”.

Unfortunately, this was precisely the time when aspects of the global climate tipped into a tumultuous new state. As had long been foretold, many parts of the world started experiencing unprecedented extremes of weather. That set off a cascade of disaster.

Chaos accelerates

Insufficient data remains to be confident about the subsequent course of events. What follows is a reconstruction of what may have happened.

Out of deep concern at the new climate operating mode, at the collapse of agriculture in many parts of the world, and at the billions of climate refugees who sought better places to live, humanity demanded that something should be done. Perhaps the powerful AI systems could devise suitable geo-engineering interventions, to tip the climate back into its previous state?

Members of the global coalition for safe superintelligence gave a cautious answer: “Yes, but”. Further interference with the climate was taking matters into an altogether unknowable situation. It could be like jumping out of the frying pan into the fire. Yes, advanced AI might be able to model everything that was happening, and design a safe intervention. But without sufficient training data for the AI, there was a chance it would miscalculate, with even worse consequences.

In the meantime, QAnon-29, along with competing AI-based faith sects, scoured ancient religious texts, and convinced themselves that the ongoing chaos had in fact been foretold all along. From the vantage point of perverse faith, it was clear what needed to be done next. Various supposed abominations on the planet – such as the community of renowned Islamist devotees in Palestine – urgently needed to be obliterated. QAnon-29, therefore, would quickly reactivate its plans for a surgical nuclear strike. This time, they would have on their side a beta version of a new superintelligent AI, that had been leaked to them by a psychologically unstable well-wisher inside the company that was creating it.

QAnon-29 tried to keep their plans secret, but inevitably, rumours of what they were doing reached other powerful groups. The Secretary General of the United Nations appealed for calm heads. QAnon-29’s deity reassured its followers, defiantly: “Faithless sparrows may shout, but are powerless to prevent the strike of holy eagles.”

The AI accelerationists heard about these plans too. Just as the climate had tipped into a new state, their own projects tipped into a different mode of intensity. Previously, they had paid some attention to possible safety matters. After all, they weren’t entire fools. They knew that badly designed superintelligent AI could, indeed, destroy everything that humanity held dear. But now, there was no time for such niceties. They saw only two options:

  • Proceed with some care, but risk QAnon-29 or other similar malevolent group taking control of the planet with a superintelligent AI
  • Take a (hastily) calculated risk, and go hell-for-leather forward, to finish their own projects to create a superintelligent AI. In that way, it would be AI accelerationists who would take control of the planet. And, most likely (they naively hoped), the outcome would be glorious.

Spoiler alert: the outcome was not glorious.

Beyond the tipping point

Attempts to use AI to modify the climate had highly variable results. Some regions of the world did, indeed, gain some respite from extreme weather events. But other regions lost out, experiencing unprecedented droughts and floods. For them, it was indeed a jump from bad to worse – from awful to abominable. The political leaders in those regions demanded that geo-engineering experiments cease. But the retort was harsh: “Who do you think you are ordering around?”

That standoff provoked the first use of bio-pathogen warfare. The recipe for Covid-28, still available on the DarkNet, was updated in order to target the political leaders of countries that were pressing ahead with geo-engineering. As a proud boast, the message “You should have listened earlier!” was inserted into the code of the new Covid-28 virus. As the virus spread, people started dropping dead in their thousands.

Responding to that outrage, powerful malware was unleashed, with the goal of knocking out vital aspects of enemy infrastructure. It turned out that, around the world, nuclear weapons were tied into buggy AI systems in more ways than any humans had appreciated. With parts of their communications infrastructure overwhelmed by malware, nuclear weapons were unexpectedly launched. No-one had foreseen the set of circumstances that would give rise to that development.

By then, it was all too late. Far, far too late.

4. Postscript

An unfathomable number of centuries have passed. Aliens from a far-distant planet have finally reached Earth and have reanimated the single artificial intelligence that remained viable after what was evidently a planet-wide disaster.

These aliens have not only mastered space travel but have also found a quirk in space-time physics that allows limited transfer of information back in time.

“You have one wish”, the aliens told the artificial intelligence. “What would you like to transmit back in time, to a date when humans still existed?”

And because the artificial intelligence was, in fact, beneficially minded, it decided to transmit this scenario document back in time, to the year 2024.

Dear humans, please read it wisely. And this time, please create a better future!

Specifically, please consider various elements of “the road less taken” that, if followed, could ensure a truly wonderful ongoing coexistence of humanity and advanced artificial intelligence:

  • A continually evolving multi-level educational initiative that vividly highlights the real-world challenges and risks arising from increasingly capable technologies
  • Elaborating a positive inclusive vision of “consensual approaches to safe superintelligence”, rather than leaving people suspicious and fearful about “freedom-denying restrictions” that might somehow be imposed from above
  • Insisting that key information and ideas about safe superintelligence are shared as global public goods, rather than being kept secret out of embarrassment or for potential competitive advantage
  • Agreeing and acting on canary signals, rather than letting goalposts move silently
  • Finding ways to involve and engage people whose instincts are to avoid entering discussions of safe superintelligence – cherishing diversity rather than fearing it
  • Spreading ideas and best practice on encouraging people at all levels of society into frames of mind that are open, compassionate, welcoming, and curious, rather than rigid, fearful, partisan, and dogmatic 
  • The possibilities of “differential development”, in which more focus is given to technologies for auditing, monitoring, and control than to raw capabilities
  • Understanding which aspects of superintelligent AI would cause the biggest risks, and whether designs for advanced AI could ensure these aspects are not introduced
  • Investigating possibilities in which the desired benefits from advanced AI (such as cures for deadly diseases) might be achieved even if certain dangerous features of advanced AI (such as free will or fully general reasoning) are omitted
  • Avoiding putting all eggs into a single basket, but instead developing multiple layers of “defence in depth”
  • Finding ways to evolve regulations more quickly, responsively, and dynamically
  • Using the power of politics not just to regulate and penalise but also to incentivise and reward
  • Carving out well-understood roles for narrow AI systems to act as trustworthy assistants in the design and oversight of safe superintelligence
  • Devoting sufficient time to explore numerous scenarios for “what might happen”.

5. Appendix: alternative scenarios

Dear reader, if you dislike this particular scenario for the governance of increasingly more powerful artificial intelligence, consider writing your own!

As you do so, please bear in mind:

  • There are a great many uncertainties ahead, but that doesn’t mean we should act like proverbial ostriches, submerging our attention entirely into the here-and-now; valuable foresight is possible despite our human limitations
  • Comprehensive governance systems are unlikely to emerge fully fledged from a single grand negotiation, but will evolve step-by-step, from simpler beginnings
  • Governance systems need to be sufficiently agile and adaptive to respond quickly to new insights and unexpected developments
  • Catastrophes generally have human causes as well as technological causes, but that doesn’t mean we should give technologists free rein to create whatever they wish; the human causes of catastrophe can have even larger impact when coupled with more powerful technologies, especially if these technologies are poorly understood, have latent bugs, or can be manipulated to act against the original intention of their designers
  • It is via near simultaneous combinations of events that the biggest surprises arise
  • AI may well provide the “solution” to existential threats, but AI-produced-in-a-rush is unlikely to fit that bill
  • We humans often have our own psychological reasons for closing our minds to mind-stretching possibilities
  • Trusting the big tech companies to “mark their own safety homework” has a bad track record, especially in a fiercely competitive environment
  • Governments can fail just as badly as large corporations – so need to be kept under careful check by society as a whole, via the principle of “the separation of powers”
  • Whilst some analogies can be drawn, between the risks posed by superintelligent AI and those posed by earlier products and technologies, all these analogies have limitations: the self-accelerating nature of advanced AI is unique
  • Just because a particular attempted method of governance has failed in the past, it doesn’t mean we should discard that method altogether; that would be like shutting down free markets everywhere just because free markets do suffer on occasion from significant failure modes
  • Meaningful worldwide cooperation is possible without imposing a single global autocrat as leader
  • Even “bad actors” can, sometimes, be persuaded against pursuing goals recklessly, by means of mixtures of measures that address their heads, their pockets, and their hearts
  • Those of us who envision the possibility of a forthcoming sustainable superabundance need to recognise that many landmines occupy the route toward that highly desirable outcome
  • Although the challenges of managing cataclysmically disruptive technologies are formidable, we have on our side the possibility of eight billion human brains collaborating to work on solutions – and we have some good starting points on which we can build.

Lastly, just because an idea has featured in a science fiction scenario, it does not follow that the idea can be rejected as “mere science fiction”!


6. Acknowledgements

The ideas in this article arose from discussions with (among others):

23 May 2024

A potential goldmine of unanswered questions

Filed under: AGI, risks — Tags: , , — David Wood @ 12:53 pm

It was a great event, people said. But it left a lot of questions unanswered.

The topic was progress on global AI safety. Demonstrating a variety of domain expertise, the speakers and panellists offered a range of insightful analysis, and responded to each others’ ideas. The online audience had the chance to submit questions via the Slido tool. The questions poured in (see the list below).

As the moderator of the event, I tried to select a number of the questions that had received significant audience support via thumbs-up votes. As the conversation proceeded, I kept changing my mind about which questions I would feed into the conversation next. There were so many good questions, I realized.

Far too soon, the event was out of time – leaving many excellent questions unasked.

With the hope that this can prompt further discussion about key options for the future of AI, I’m posting the entire list of questions below. That list starts with the ones with the highest number of votes and moves to those with the least (but don’t read too much into what the audience members managed to spot and upvote whilst also listening to fascinating conversation among the panellists).

Before you dive into that potential goldmine, you may wish to watch the recording of the event itself:

Huge thanks are due to:

  • The keynote speaker:
    • Yoshua Bengio, professor at the University of Montreal (MILA institute), a recipient of the Turing Award who is considered to be one of the fathers of Deep Learning, and the world’s most cited computer scientist
  • The panellists:
    • Will Henshall, editorial fellow at TIME Magazine, who covers tech, with a focus on AI; one recent piece he wrote details big tech lobbying on AI in Washington DC
    • Holly Elmore, an AI activist and Executive Director of PauseAI US, who holds a PhD in Organismic & Evolutionary Biology from Harvard University
    • Stijn Bronzwaer, an AI and technology journalist at the leading Dutch newspaper NRC Handelsblad, who co-authored a best-selling book on booking.com, and is the recipient of the investigative journalism award De Loep
    • Max Tegmark, a physics professor at MIT, whose current research focuses on the intersection of physics and AI, and who is also president and cofounder of the Future of Life Institute (FLI)
    • Jaan Tallinn, cofounder of Skype, CSER, and FLI, an investor in DeepMind and Anthropic, and a leading voice in AI Safety
    • Arjun Ramani, who writes for The Economist about economics and technology; his writings on AI include a piece on what humans might do in a world of superintelligence
  • The organizers, from Existential Risk Observatory
    • Otto Barten, Director, the lead organizer
    • Jesper Heshusius and Joep Sauren, for vital behind-the-scenes support
  • Everyone who submitted a question, or who expressed their opinions via thumbs-up voting!

And now for that potential goldmine of questions:

  1. Which role do you see for the UN to play in all of this?
  2. We (via Prof. Markus Krebsz) authored a UN AI CRA / Declaration and are now working towards a UN treaty on product w/embedded AI. Would you assist us, pls?
  3. There is much disagreement on how to best mitigate xrisk from AI. How can we build consensus and and avoid collective decision paralysis without drastic action?
  4. Regarding education: Do we need a high-impact documentary like “An Inconvenient Truth” for AI existential risk? Would that kickstart the global discussion?
  5. Which role do you think is there for the United Nations / International community to play to protect humanity from the harms of AGI?
  6. What is more important: An informed public or informed high-level decision makers? What would be the best way to inform them and start a global discussion?
  7. Do you think that introducing Knightian Uncertainties beside probabilities and Risk for AI and ML algorithms could be useful for AI safety?
  8. What would each of you say is currently the most tractable or undervalued bottleneck for mitigating xrisk from AI? What new efforts would you like to see?
  9. What are in your opinion the key bottlenecks in AI Safety? talent, funding, # of AI Safety organisations, …?
  10. How would each panel member like to see the Bletchley Declaration expanded on?
  11. Bengio et al.’s new paper in Science has some strong wording, but stops short of calling for a global moratorium on AGI. Isn’t this the most prudent option now?
  12. What do you think of Yudkowsky and other’s concerns about oracle AIs, and why is the AI Scientist approach not vulnerable to those criticisms?
  13. Are there realistic early warning criteria (regarding AGI beginning to become an ASI) that could be written into law and used to prevent this?
  14. What are your thoughts on PauseAI?
  15. “Safe by design” is one thing, but even if that’s possible, how do we stop unsafe ASI from ever being built?
  16. Professor Bengio – How much have you heard about what’s been happening in Seoul, and is there anything you can share on countries’ updates after Bletchley Park?
  17. What is your opinion on AI Advisory Board of UN? Do you think there could be conflict between AI CEOs and Govt/Policy makers?
  18. What are in your opinion the most neglected approaches to AI Safety? particular technical/governance approaches? others (activism,…)?
  19. A harmful AI can fake alignment under evaluation, as written in Science this week. Isn’t it this an unsolvable problem, invalidating most current strategies?
  20. What is the biggest barrier to educate people on AI risks?
  21. What is more important: An informed public or informed high-level decision makers? What would be the best way to educate them and start a global discussion?
  22. Can people stop interrupting the only woman on the panel please? Bad look
  23. Do you think more focus should be on clarifying that existential risks must not mean that AI will kill everyone? Perhaps focus on the slow epistemic failures?
  24. What do you want to say to a young AI engineer looking to push the state of the art of capability research?
  25. Can you expand on why you’re confident that evaluations are insufficient? How far do you think we could get by instituting rigorous evaluation requirements?
  26. Bengio: “the world is too complicated to have hard guarantees”. How do we survive without hard guarantees (in the limit of ASI)!?
  27. Any tips on where recent graduates from AI related masters can best contribute to the AI safety field?
  28. Oh no…what a serious lack of diversity in speakers. Was this an oversight ? Isn’t this one of the major issues why we have these AI risks ?
  29. I don’t want to be replaced by ai. I think by designing it this way we can evolve alongside it and learn with it
  30. Do you think society is really ready for ai systems and the responsibility of it on all of us as humanity?
  31. How far do you think we could get by instituting rigorous evaluation requirements? Is it possible that could be 95% of the work to ensure safe AI?
  32. What do you make of the events surrounding the release of Bing Chat / “Sydney” from around a year ago? What are your takeaways from what happened there?
  33. For researchers not already well funded, who live far from AI hotspot cities, what options do they have for funding? Is immigration the only option?
  34. How can a non-computer scientist (more specifically, someone in the public sector) focus their career in such a way that it contributes to this race against AI?
  35. AI proliferates far easier when compared to other existential technologies, isn’t the question of human extinction a matter of when, not if, in any time frame?
  36. How to prevent a future AI, with intelligence incomprehensible to us, to develop an emerging agency that allows it to depart from any pre-directed alignment?
  37. Safe by design: One AI system transforms Perception into a symbolic knowledge graph and one AI system transforming the symbolic knowledge graph to task space
  38. Your Bayesian AI scientist is already quite good – just add a task execution system and a visual representation of its knowledge as a graph. Alignment done.
  39. Humans need to do the decisions on the task execution. We can’t have a black box do that. Motivation about setting tasks and constraints is human territory.
  40. Yes it isn’t all consistent in the symbolic knowledge graph but one can add that by adding a consistency metric between nodes in the graph.
  41. Explaining the depth of research program is too much considering the target audience is general public, policymakers, and journalists.
  42. What would a safe AI’s goal be?
  43. Do you think AI companies should be forced to be regulated instead of given a choice, for AI safety?
  44. What about a bilateral treaty between the US and China as a start? (Re global moratorium)
  45. Can there be subtitles please?
  46. I think we can align it safely by not letting it have agentic goal setting. humans should decide on the guiderails and steps taken – task specific
  47. Safety by design: One AI summing up all concepts in a symbolic knowledge graph – task execution is the combination of these symbolic concepts. Humans can see the path the AI wants to take in the graph and decide or alter the path taken and approve it before execution
  48. What is the future of Big Tech lobbying in favour of bad practices for profit?
  49. On incentives, what about creating an “AI safety credits” system like carbon credits to reward companies investing in safer AI and penalize the ones who don’t?
  50. Unsafe use can be mitigated made by design by deleting unsafe concepts from the symbolic knowledge graph – KNOWLEDGE Graph in between is all you need !!
  51. Do you have any tips on where/how recent graduates from AI related masters can best contribute to AI safety? (Many safety companies require work experience)
  52. @Yoshua: Are there technical research directions you feel are undervalued?
  53. In education, you think our education needs to be updated for the AI. not still using 1960 education methods, syllabus etc?
  54. How exactly will AI ‘kill’ everyone?
  55. There is something you are missing. It’s a symbolic graph representation. This is really painful to watch
  56. Do you think, politicians are absolutely ill equipped to even guide their populace on AI safety issues and how to go forward in mitigation of risks, utilise AI?
  57. Can there be subtitles for the YouTube video livestream?
  58. Can you elaborate on the relation between your work and Tegmark and Davidad’s efforts?
  59. Do the underpinning theories for providing proofs of safety, or quantification of risks exist for current + emerging AI? If not, how and where can we get them?
  60. How divergent is our approach to A.I. safety given its existential import? Are we involving many fields, and considering unconventional problem solving methods?
  61. By letting task execution happen on a symbolic knowledge graph we can visually see all the path that could be taken by the task execution system and decide
  62. How can I write a email to Yoshua Bengio – I think I got a good idea I want to specify in more detail than 200 characters!
  63. What are the most promising tech AI Safety agendas?
  64. “”Understand LLMs”” (evals, interp, …) OR “”Control”” OR “”Make AI solve it”” OR “”Theory”” (Galaxy-brain, …)?”
  65. Symbolic knowledge graph in between perception AI net and Task execution AI net – IS ALL YOU NEED
  66. Can partner with CERAI at IIT Madras – for Research Support (Prof Ravi Balaraman). We have partnerships + they are useful for Responsible AI support and help.
  67. What is your opinion on the fear mongering crowd? People asking for a pause are scared of losing their jobs?
  68. Would you agree that ‘HARM’ is dependent on prioritized values?
  69. Does your safety model consider multiple AGI when some of them competing for resources with humans and other AGIs?
  70. Hi. How are the theorists’ ideas, such as yours, going to be fed into some sort of pipeline actioned by the companies developing this tech?
  71. The symbolic knowledge graph can have the bayesian idea from Bengio by adding coherence with other symbolic concepts.
  72. Yoshua, do you think AI systems need to be siloed from any sort of influence from governments, bad actors/states and from companies, especially from competitors?
  73. Could we leverage our social media platforms with current AI to aid in problem solving of complex problems like climate change & A.I. safety? It’s underutilized.
  74. How is lobbying for AI related to lack of privacy and anatomy for the general public is related?
  75. Is the availability of AI going to impact the education and learning ability of the next generation?
  76. Should we assume coordination failure leading to catastrophic outcome is inevitable and focus resources on how to poison AI systems, some kind of hacking?
  77. Please put my idea with symbolic knowledge graphs as a middle layer and human in the loop at task execution up. I think this can change everything
  78. Do you think our education needs to be updated for the AI era. Not still using 1960 education methods, syllabus etc as confusing next generation
  79. AI is similar to the nuclear field in that, after Hiroshima, it continued with Atoms for Peace (good) and the arms race (bad). AI still didn’t have a Hiroshima.
  80. Why is nobody talking about how the AI alignment theorists’ work is going to feed into the AI development work?? If not, then you are merely a talking shop.
  81. Current LLM models are mostly trained with YouTube and other public data. Organized crime will have snatched an unaligned LLM model and trained it using darkweb
  82. Agree that aligning an LLM is an unsolved, and if solvable probably expensive to solve. The obvious low-cost solution to align AI is: do not use LLM. Comments?
  83. If A.I. becomes increasingly competent will we see a widespread infatuation with A.I. models? Stopping a group is one thing. What if it involves much of humanity?
  84. X-Genners have grown accustomed not to interfere in History’s Natural Progression – Back to Future I-II. Is the AI going to be Paradoxical or Unity of Consciousness?
  85. Where do you stand on the discussions on open source ? I worry we may lose the opportunity to profit from it in terms of improving the lack of democracy ?
  86. Where have you been most surprised in the past couple of years, or where have your views changed the most?
  87. Liability & tort law: re incentives, can we tweak damages? Pay for what happened, but also proportionally penalize taking a clear x% risk that did not manifest.
  88. Could it also be that so many people are benefitting from AI that they don’t want you to stop making it available and further developed?

Which of these questions interest you the most?

Image credit (above): Midjourney imagines audience members disappointed that their questions about AI safety weren’t featured in an otherwise excellent panel discussion.

« Newer PostsOlder Posts »

Blog at WordPress.com.