Areas requiring more thought/investigation:
- Could a mouse perform calculus? - Is there something in the human brain that enables it to perform higher level thinking that a mouse brain lacks? It seems likely that we are only utilizing a small fraction of the neurons in our brain when we perform higher order thinking. This has relevance to neuromorphic computing. Does it just need to simulate something the size of a mouse brain, or does it need to simulate something the size of a human brain, to perform what we think of as human-only thinking?
Power - Does society only function because as individuals we have similar amounts of power, and so we can construct a society (law, courts, penal institutions) that keeps one another in check? Would an intellect with a thousand or a million times the power of an individual be able to dominate the equation?
- Exit strategy - Smarter-than-human AI is likely to result in a massive concentration of power. There are few cases in history where the concentration of power has been used to benefit the world globally. The lesson of Microsoft is interesting. Bill Gates global health initiative has done a lot of good, but Microsoft continues to march on extracting monopoly rents on Windows and Office. Anyone contemplating a for profit venture to produce AGI needs to also have a plan for ending the for profit nature of the venture should it prove too successful.