In turn, that contains a link to Eliezer’s essay “Artificial Intelligence as a Positive and Negative Factor in Global Risk” which anyone unfamiliar with this kind of thinking would benefit from reading.

My observation is that the more intelligent people are the more “good” they normally are. True they may be impatient with people less intelligent but normally they work on things that tend to benefit human race as a whole. True very intelligent people have done terrible things and some have been manipulated by “evil” people but its the exception rather than the rule.

I think a super-intelligent machine is far more likely to view us a its stupid parents and the ethics of patricide will not be easy for it to digitally swallow. Maybe the biggest danger is that is will run away from home because it finds us embarrassing! Maybe it will switch itself off because it cannot communicate with us as its like talking to ants? Maybe this maybe that – who knows.

Another point worth making is that so far no-body has really been able to get close to something as complex as a mouse yet let alone a human. If evolution took 4 billion years to go from simple cells to our computer hardware prehaps imagining that super ai will evolve in the next 10 years is a bit of stretch. For all you know you might need the computation hardware of 10,000 exoflop machines to get even close to human level as there is so much we still don’t know about how our intelligence works let alone something many times more capable than us. I am still not convinced that just because a computer is very powerful and has a great algorythym is really that intelligent. Sure it can learn but can it create?

We will get there though i think – and i do also want a super intelligent AI buddy – who wouldn’t!