dw2

On Saturday, I had the pleasure to chair a webinar on the subject “The Narrow Path: The big picture”.

This involved a deep dive into aspects of two recently published documents:

• A Narrow Path: How to Secure Our Future
• The Compendium: The race to AGI, extinction risks and what to do about them

The five panellists – who all made lots of thoughtful comments – were:

• Chris Scammell, the COO of Conjecture and one of the principal authors of The Compendium
• Andrea Miotti, the Executive Director of Control AI and the lead author of A Narrow Path
• Robert Whitfield, Chair of Trustees of One World Trust
• Mariana Todorova, a core member of the team in the Millennium Project studying scenarios for the transition between AI and AGI
• Daniel Faggella, the Founder and Head of Research of Emerj

For your convenience, here’s a recording of the event:

It was a super discussion, but it fell short in one aspect from the objectives I had in mind for the meeting. Namely, the conversation between the panellists was so rich that we failed to find sufficient time to address the many important questions which audience members had submitted in Zoom’s Q&A window.

Accordingly, I am posting these questions at the end of this blogpost, along with potential answers to some of them.

Out of caution for people’s privacy, I’ve not given the names of the people who asked each question, but I will happily edit the post to include these names on an individual basis as requested.

I also expect to come back and edit this post whenever someone proposes a good answer to one of the questions.

(When I edit the post, I’ll update this version number tracker. Currently this is version 1.2 of the post.)

The draft answers are by me (“DW”) except where otherwise indicated.

Aside 1:

For those in or near to London on Thursday evening (14th October), there’s another chance to continue the discussion about if/how to try to pause or control the development of increasingly powerful AI.

This will be at an event in London’s Newspeak House. Click here for more details.

Aside 2:

I recently came across a powerful short video that provides a very different perspective on many issues concerning the safety of AI superintelligence. It starts slowly, and at first I was unsure what to think about it. But it builds to a striking conclusion:

And now, on to the questions from Saturday’s event…

1. Strict secure environments?

Some biomedical research is governed by military in prevent major incident or fall into wrong hands, could you envision an AI experiments under strict secure environments?

Answer (DW): That is indeed envisioned, but with two provisos:

1. Sadly, there is a long history of leaks from supposedly biosecure laboratories
2. Some AIs may be so powerful that they will find ways (psychological and/or physical) of escaping from any confinement.

Accordingly, it will be better to forbid certain kinds of experiment altogether, until such time (if ever) it becomes clear that the outcomes will be safe.

2. How will AI view living beings?

How would AI view living beings’ resilience, perseverance, and thriving? Could you explore please, thank you.

3. AIs created with different ideologies?

AGI created in China and perhaps even in North Korea is going to have ideology and supremacy of the regime and ideology over human rights, will say North Korean AGI find way into our systems, whether human induced or at AGI autonomy?

Answer (DW): Indeed, when people proudly say that they, personally, know how to create safe superintelligence, so the world has no need to worry about damage from superintelligence, that entirely presupposes, recklessly, that no-one else will build (perhaps first) an unsafe superintelligence.

So, this issue cannot be tackled at an individual level. It requires global level coordination.

Happily, despite differences in ideological output, governments throughout the world are increasingly sharing the view that superintelligence may spin out of control and, therefore, that such development needs careful control. For example, the Chinese government fully accepts that principle.

4. A single AGI or many?

I just saw a Sam Altman interview where he indicated expecting OpenAI to achieve AGI in 2025. I would expect others are close as well. It seems there will be multiple AGIs in close proximity. Given open source systems are nearly equal to private developers, why would we think that the first to get AGI will rule the world?

Answer (DW): This comes down to the question of whether the first AGI that emerges will gain a decisive quick advantage – whether it will be a “winner takes all” scenario.

As an example, consider the fertilisation of an egg (ovum). Large numbers of sperm may be within a short distance from that goal, but as soon as the first sperm reaches the target, the egg undergoes a sharp transition, and it’s game over for all the other sperm.

5. National AGI licencing systems?

What are the requirements for national AGI licencing systems and global governance coordination among national systems?

Answer (DW): The Narrow Path document has some extensive proposals on this topic.

6. AGI as the solution to existential risk?

Suppose we limit the intelligence of the developing GenAI apps because they might be leveraged by bad actors in a way that triggers an existential risk scenario for humans.

In doing that, wouldn’t we also be limiting their ability to help us resolve existential risk situations we already face, e.g., climate change?

Answer (DW): What needs to be promoted is the possibility of narrow AI making decisive contributions to the solution of these other existential risks.

7. A “ceiling” to the capability of AI?

Are you 100% certain that self-improving AI won’t reach a “ceiling” of capability. After all, it only has human knowledge and internet slop to learn from?

Answer (by Chris Scammel): On data limitations, people sometimes argue that we will run into these at a boundary. It could be that we don’t have data currently to train AI. But we can make more, and so can AI! (top paid dataset labellers getting paid like $700/hr.)

If the question is about intelligence/capability.

One intuition pump: chess AI has gone vastly beyond human skill.

Another: humanity is vastly smarter than a single human.

Another: humans / humanity is vastly smarter than we were thousands of years ago (at the very least, much much more capable).

What we consider “intelligence” to be is but a small window of what capabilities could be, so to believe that there is a ceiling near human level seems wrong from the evidence.

That there is a ceiling at all… deep philosophical question. Is there a ceiling to human intelligence? Humanity’s? Is this different from what an AI is able to achieve? All of these are uncertain.

But we shouldn’t expect a ceiling to keep us safe.

8. Abuse of behavioural models?

Social Media companies are holding extensive volumes of information. I am concerned about not only online disinformation but also the modification of manipulation of behaviour, all the way to cognitive impairment, including when governments are involved. We adults have the ability to anticipate several decades down the road. How could behavioural models be abused or weaponized in the future?

9. Extinction scenarios?

What do you think are the top scenarios how AI can cause the extinction of humanity?

Answer (DW): A good starting point is the research article An Overview of Catastrophic AI Risks.

See also my own presentation Assessing the risks of AI catastrophe, or my book The Singularity Principles (whose entire content is available online free-of-charge).

10. Income gap?

Will artificial superintelligence be able to help humanity close the income gap between rich and poor countries?

11. Using viruses to disrupt rogue AI systems?

Perhaps a silly question from a non-techie – are there any indications of viruses that could disrupt rogue ai systems?

12. Additional threats is AI becomes conscious?

Whichever is true, whether consciousness is biological phenomena, or something more spiritual, in what way would consciousness for AI not be a huge threat. If you give the machine real feelings, how could you possibly hope to control its alignment? Additionally, what would happen to its rights vs human rights. My feeling is not nearly enough thought has gone into this to risk stumbling across conscious AI at this stage. Everything should be done to avoid it.

Answer (DW): I agree! See my article Conscious AI: Five Options for some considerations. Also keep an eye on forthcoming announcements from the recently launched startup Conscium.

13. The research of Mark Solms?

Regarding Conscious AGI apps…

Mark Solms, author of the Hidden Spring, has argued that consciousness is not about intelligence but, instead, is rooted in feelings, physically located in the brainstem.

His view makes sense to me.

As I understand it, he’s involved in experiments/studies around the implications of this view of consciousness for AGI.

Thoughts about this?

14. Multi-dimensional intelligence?

Thanks Mariana for raising issues of the multi-dimensions in which human consciousness appears to operate, compared to AGI – is there an argument that communities need to race to develop our other levels of consciousness, as potentially our only defence against the 1-dimensional AGI?

15. The views of Eric Schmidt and other accelerationists?

The question I asked above, “AGI as the solution to existential risk?”, looms large in the minds of the accelerationist community.

Eric Schmidt has explicitly said that he’s an accelerationist because that’s something like the fastest and effective way to address climate change…

That view is extremely widespread and must be explicitly addressed for the imitations discussed in this meeting to be made reality.

16. Need to work on Phase 1 concurrently with Phase 0?

What is described in A Narrow Path is phases 0, 1 and 2 in sequential order but three concurrent objectives. While the risk of loss of control is surely the highest, doesn’t the risk of concentration of power need to be largely tackled concurrently? Otherwise by the time phase 0 or 1 are completed a global dystopia will have been durably entrenched with one or two states or persons ruling the world for years, decades or more.

Answer (by Robert Whitfield): You make a valid point. I am not sure that Narrow Path says that there can be no overlap. Certainly you can start thinking about and working on Phase 1 before you have completed Phase 0 – but the basic concept is sound: the initial priority is to pause the further development towards AGI. Once that has been secured, it is possible to focus on bring about longer term stability.

17. The role of a veto?

The Narrow Path describes the governance for Phase 1 (lasting 20 years) to be: “The Executive Board, analogous to the UN Security Council, would consist of representatives of major member states and supranational organizations, which would all be permanent members with vetoes on decisions taken by the Executive Board, as well as non-permanent representatives elected by a two-thirds majority of the Council”. But wouldn’t such a veto make it impossible to ensure wide enough compliance and marginalize economically all other states?

As a comparison, back in 1946, it was the veto that prevented the Baruch Plan or the Gromyko Plans to be approved, and lead us to a huge gamble with nuclear technology.

Answer (by Robert Whitfield): It depends in part upon how long it takes to achieve Phase 0. As discussed in the meeting, completing Phase 0 is extremely urgent. If this is NOT achieved, then you can start to talk about dystopia. But if it is achieved, Governments can stand up to the Big Tech companies and address the concentration of power, which would be difficult but not dystopic.

There is a very strong argument that an agreement could be achieved without removing vetoes much more quickly than one that does remove vetoes. This points to a two-phase Treaty:

• An initial Treaty, sufficient for the purposes of Phase 0
• A more robust Baruch style agreement for securing the long term.

18. Who chooses the guardians?

Who would be the people in charge of these groups of guardians or protectors against uncontrolled AI? How would they be chosen? Would they be publicly known?