Does super-intelligence necessary lead to consciousness, self awareness, freewill or emotion After seeing this talk, the question popped in my mind. The idea is that as soon as a system is complex enough or intelligent enough, it able to act on its own. It seems to be a common belief. For example, you can find it in the movie Terminator, where skynet, an advanced computer, becomes self aware and decide to eradicate humanity. The same is true with the  HAL in 2001: A Space Odyssey, where the computer decides to act despite receiving the opposite orders from the astronaut.

I believe what is said in this ted talk, where Antonio Damasio explains that consciousnesses is not a great product of the neocortex, but rather a trick achieved by other older parts in the brain (essentially the brainstem). If, in addition, you believe that human intelligence and cognition takes place in the neocortex (as in the Memory-prediction framework model, introduced in On Intelligence by Jeff Hawkins), it doesn't make sense that intelligence is linked to consciousness, self awareness, freewill or emotion.

So my question is: is there any scientific evidence for this link, is there any paper about it ?
As the other response noted, a lot of the answers will depend on how you define the terms. Here, I will use Legg &amp; Hutter's (2007) definition of intelligence, where an agent is intelligent if it can achieve a large variety of goals in a large variety of environments. For instance, a chess-playing robot is great at achieving the goal of winning at chess when it is in the environment of a chess game where nobody cheats or does anything else odd, but it fails miserably at just about everything else. So it has low intelligence. In contrast, a human can achieve many different goals in many different environments, from hunting food in a rainforest, to repairing solar panels while wearing a space suit on Earth's orbit. So a human has a considerably higher intelligence. See the paper for a formalization of this intuitive definition.

Note that a system can be exceedingly complex, or good at one thing, without necessarily being intelligent in this sense. Complexity does not imply intelligence by itself. But let's suppose that a system were intelligent in this sense? Well, let's tackle each of your terms in turn.

Self-awareness. One widely-used definition for self-awareness is being "aware of themselves as a differentiated and unique entity in the world". For instance, Rochat (2003) discusses the development of self-awareness in babies and very young children:


  All parents also notice an important change at around 2 years of age when children
  manifest ‘‘self-consciousness,’’ the so-called secondary emotions such as embarrassment or pride
  in very specific situations such as mirror exposure or competitive games (Kagan, 1984; Lewis,
  1992). Prior to the second year, an infant placed in front of a mirror will typically smile, coo, and
  explore in apparent delight of the perfect contingency between acted and seen movements
  bouncing back at them from the polished surface of the mirror (Amsterdam, 1972). By 2 years, the
  specular image is associated with radically different behaviors. Toddlers become typically frozen
  and sometime behave as if they wanted to hide themselves by tucking their head in their shoulders
  or hiding their face behind their hands. They show embarrassment. This is a robust phenomenon
  and one is naturally tempted to ask what it means psychologically for children in their development. [...]
  
  It appears indeed that by 2–3
  years young children do start to have others in mind when they behave. The expression of embarrassment
  that children often begin to display in front of mirrors at around this age is the
  expression of such ‘‘self-consciousness.’’ They behave not unlike criminals hiding their face to the
  cameras. Their behavior indicates a drive to vanish from the public eyes, as if they came to grip via
  the experience of their own specular image of how they present themselves to the world. Not only
  do they discover in the mirror that it is themselves, they also realize that it is themselves as
  perceived by others. The malaise might come from the realization of a fundamental discrepancy
  between how the child represents herself from within, and how he or she is actually perceived by
  others as reflected in the mirror.


So self-awareness implies an understanding of the fact that there is such a thing as a "me", which is separate from the world as a whole, and which others will perceive differently than it will perceive itself. There can also be varying levels of self-awareness, ranging from the level 1 self-awareness that newborn infants have ("immediately after birth, infants are capable of demonstrating
already a sense of their own body as a differentiated entity: an entity among other entities in the
environment") to level 5 self-awareness involving a theory of mind and an understanding of the fact that others may have false beliefs about something while you yourself have a true belief of it.

Would a superintelligence necessarily need to be self-aware, in any of these senses? I'd say that it would. While there are lots of goals that could plausibly be solved without any kind of self-awareness (proving mathematical theorems, for instance), many seem impossible to solve without it. Pretty much any social activity requires you to have a model about others as distinct from yourself, as well as an ability to model their beliefs and attitudes towards yourself. So in order to really be capable of solving many goals, you need to be self-aware. On the other hand, if it was a narrow intelligence that had been built to only solve theorems of math, it probably wouldn't need self-awaraness.

Emotion. There are many theories of emotion. For instance, the Information Principle says that emotional feelings provide conscious information from unconscious appraisals of situations (Clore et al. 2001). Your brain is constantly appraising the situation you happen to be in. It notes things like a passerby having slightly threatening body language, or conversation with another person being easy and free of misunderstandings. There are countless of such evaluations going on all the time, and you aren't consciously aware of them, because you don't need to. Your subconscious mind can handle them just fine on its own. The end result of all those evaluations is packaged into a brief summary, which is the only thing that your conscious mind sees directly. That "executive summary" is what you experience as a particular emotional state. The passerby makes you feel slightly nervous and you avoid her, or your conversational partner feels pleasant to talk with and you begin to like him, even though you don't know why.

So this theory says that emotions are both a source of information about unconscious appraisals, and a way to bias your behavior on the basis of circumstances. For instance, in happy moods (when things are presumably going well), we tend to rely more on our pre-existing routines and heuristics, while in sad or frustrated moods (when things aren't going so well), our minds switch to a more analytical mode of thought where we're more likely to take apart our existing beliefs and routines to find whether there's something wrong with them (Schwarz, 2010).

Would a superintelligence have something that we would consider to be akin to "emotions"? I would think that it might or might not, depending on its architecture. Thinking for a moment about humans, maybe a human with a vastly larger working memory capacity wouldn't need to have the subconscious appraisals packaged into summaries in the form of emotions, but could just constantly review their un-summarized conclusions while also thinking about other things. Maybe such a human could also learn to consciously control the style of their information processing, and adapt either heuristic or analytical thought as seemed reasonable. Arguably, such a human wouldn't need emotions. So it seems plausible that a superintelligence wouldn't necessarily have them either.

Free will. Okay, now we are getting into tricky territory. There are lots of definitions about free will, so I'll merely say that the computational theory of mind seems to be the majority view in cognitive science. If you accept the computational theory of mind, then you also consider the human mind to be a kind of a computer executing certain algorithms. Then on most accounts, if you would consider a human to have free will, you would probably also consider a machine intelligence to have free will, and vice versa.

Consciousness. This is another vague term, and people mean different things by it. Copeland (1993) notes at least three possible meanings:


Baseline sense: an entity is conscious if it is capable of perceiving the world via sense organs of some sort, and if it is capable of performing inner activities such as reasoning, deliberating, judging, hypothesing, planning, and so on.
Consciousness as internal monitoring: an entity is conscious if it is capable of observing and reporting on itself. For instance, I am not simply thinking looking up Copeland's definitions and typing a summary of them: I also know that I am doing so, and can think about this fact.
Phenomenological consciousness: an entity is conscious if there's something that being that entity "feels like". Pain and pleasure are not just algorithmical responses that modify an entity's behavior: they feel painful or pleasant. This is the hardest category to really express in words, but hopefully everyone understands what's meant.


A superintelligence would most definitely need to be conscious in the baseline sense: if it couldn't collect information from the world and make plans on that basis, it wouldn't be much of an intelligence. It would probably also need be conscious in the sense of being capable of internal monitoring, for otherwise it might not really be self-aware, and we already concluded that self-awareness is necessary for (broad) intelligence.

What about phenomenological consciousness? Chalmers (1995) has a pretty convincing (to me) thought experiment, where he says that a human who was changed into a robot, replacing one neuron at a time with an artificial one, would remain phenomenally conscious. But that argument presumes that the human retains their mental architecture unchanged. If we built a mind that had an entirely different mental architecture as humans did, would it exhibit phenomenal consciousness? There are lots of arguments, but the essential gist is that nobody knows.

Indepedent goal-directed behavior. But there's one more thing that your question seems to be implying, which wasn't fully covered by any of the terms above. You seem to also be asking whether an intelligent system might start doing things which conflict with the desires of its creators.

The answer seems to be yes. Omohundro (2008) argues that even a chess-playing robot, if it were intelligent enough and if it were not designed carefully enough, "will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety". Suppose that the robot has "winning at chess" as its goal. Now, it will exhibit these behaviors because, if it is turned off it might not be able to play any more chess, if it can make copies of itself those copies can continue to play chess even if some were turned off, and extra resources will help it in making sure that it can always win at chess. Omohundro also mentions a number of other drives which will arise as natural subgoals of almost any other goal. 

Armstrong et al. (forthcoming) have an extensive discussion about "Oracle AI", an artificial intelligence which would be built to be safe, as it was built to only answer questions and not do anything else. But even this is harder to do than it might seem. Even trying to program the Oracle AI to want to stay within a specific physical box requires a number of definitions about what constitutes its physical location and what constitutes the Oracle AI itself (is it still staying within the box if it makes remote copies of itself?), and it might plausibly discover theories of physics in which those definitions became meaningless. Note that all of this presumes that we actually understand the OAI well enough to be able to specify its goals with such detail. If we don't, then that plan fails right away. Probably the best existing review article about this topic is Muehlhauser &amp; Helm (forthcoming).

References

Armstrong, S. &amp; Sandberg, A. &amp; Bostrom, N. (forthcoming) Thinking inside the box: using and
controlling an Oracle AI. http://www.aleph.se/papers/oracleAI.pdf

Chalmers, D.J. (1995) Absent Qualia, Fading Qualia, Dancing Qualia. In Metzinger, T. (Ed.), Conscious Experience. Imprint Academic. http://consc.net/papers/qualia.html

Clore, G.L. &amp; Gasper, K., &amp; Garvin, E. (2001). Affect as information. In Forgas, J.P. (Ed.), Handbook of affect and social cognition (pp. 121–144). Mahwah, NJ: Erlbaum.

Copeland, J. (1993) Artificial Intelligence: A Philosophical Introduction. Blackwell.

Legg, S. &amp; Hutter, M. (2007) Universal Intelligence: A Definition of Machine Intelligence. Minds &amp; Machines, 17:4 391-444. http://arxiv.org/abs/0712.3329

Muehlhauser, L. &amp; Helm, L. (forthcoming). The Singularity and Machine Ethics. http://commonsenseatheism.com/wp-content/uploads/2011/11/Muehlhauser-Helm-The-Singularity-and-Machine-Ethics-draft.pdf

Omohundro, S. (2008) "The basic AI drives," in Wang, P. Goertzel, P. and Franklin,
S. (eds.), Proceedings of the First AGI Conference. Frontiers in Articial Intelli-
gence and Applications, Volume 171 (IOS Press) 483-494. https://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf

Rochat, P. (2003) Five levels of self-awareness as they unfold early in life. Consciousness and Cognition, 12, 717-731. http://www.psychology.emory.edu/cognition/rochat/lab/5%20levels%20of%20self-awareness.pdf

Schwarz, N. (2010) Feelings as information. In Van Lange, P. &amp; Kruglanski, A. &amp; Higgins, E.T. (Eds.), Handbook of theories of social psychology, Sange. http://people.ict.usc.edu/~gratch/CSCI534/schwarz_feelings-as-information_7jan10.pdf