Here’s a quick summary of the central messages from Anna Salamon, Research Fellow at the Singularity Institute for Artificial Intelligence, in her talk “Shaping the intelligence explosion”. She makes four key claims. Note: these claims don’t depend on specific details of technology.
1. Intelligence can radically transform the world
Intelligence is like leverage. Think of Archimedes moving the whole world, with a sufficiently large lever.
The smarter a given agent is, the more scope it has to find ways to achieve its goal.
Imagine intelligence that is further beyond human-scale intelligence, similar to the way human-scale intelligence is beyond that of a goldfih. (“A goldfish at the opera” is a metaphor for minds that can’t understand certain things.) How much more change could be achieved, with that intelligence?
Quote from Michael Vassar: “Humans are the stupidest a system could be, and still be generally intelligent”.
2. An intelligence explosion may be sudden
Different processes have different characteristic timescales. Many things can happen on timescales very much faster than humans are used to.
Human culture has already brought about considerable change acceleration compared to biological evolution.
Silicon emulations of human brains could happen much faster than the human brain itself. Adding extra hardware could multiply the effect again. “Instant intelligence: just add hardware”.
The key question is: how fast could super-human AI arise? It depends on aspects of how the brain emulation will work. But we can’t rule out the possibility that a time will arise of a super-fast evolution of super-human AIs.
3. An uncontrolled intelligence explosion would kill us, and destroy practically everything we care about
Smart agencies can rearrange the environment to meet their goals.
Most rearrangements of the human environment would kill us.
Would super AIs want to keep humans around – eg as trading partners? Alas, we are probably not the most useful trading partners that an AI could achieve by rerranging our parts!
Values can vary starkly across species. Dung beetles think dung is yukky.
AIs are likely to want to scrap our “spaghetti code” of culture and create something much better in its place.
4. A controlled intelligence explosion could save us. It’s difficult, but it’s worth the effort.
A well engineered AI could be permanently stable in its goals.
Be careful what you build an optimizer for – remember the parable of the sorcerer’s apprentice.
If we get the design right, the intelligence explosion could aid human goals instead of destroying them.
It’s a tall order!
dw2 
Leave a Reply