what is the reptile brain? As a Carl Sagan fan, I remember (in Cosmos TV shoe) "territoriality, aggression and ritual" are products of the reptile brain while higher-level stuff like loyalty, planning for future are mammalian, which implies different brain regions for each functions.

Even Sheldon Cooper has something to say about this:


  Penny’s emotional responses originate from the primitive portion of
  the brain known as the Amygdala, while speech is centred in the much
  more recently developed Neocortex. The former can easily overpower
  the latter giving scientific credence to the notion of being rendered
  speechless.


And said by on-screen and off-screen neuroscientist and science consultant Mayim Bialik:


  So says your prefrontal cortex. But meanwhile, the limbic system of
  your brain is calculating that if another woman is attracted to
  Leonard, it must be because he’s desirable.




It implies that the brain is composed of three parts, with vast differences in development, can overpower each other and process stimuli independently.

What exactly is the reptile brain and do modern theories prove such a concept?
The reptilian brain is the oldest part of the triune brain. And the triune brain is a unified account of brain function and brain evolution. 

The story goes like this. The brain can be divided into three sections:


The reptilian brain, named so because it encompasses structures that did not change much from reptile to man. It includes the basal ganglia and cerebellum, and controls pretty much everything that a reptile can do: breathing, walking, appetite, sensation, simple motor reactions to the environment, etc.
The mammalian brain (sometimes called paleomammalian), which encompasses structures that are unique to mammals but similar across mammalian species. These structures belong to the limbic system and are dedicated to processing emotion and memory. Anything your cat can do, should be either in the reptilian or the mammalian brain. Anything it can't do, is in the third part of the triune brain.
The neocortex, found in primates, which is involved in higher order functions such as language.


Now, this is an oversimplification both structurally and functionally. For example, birds don't fit this theory very well because they don't have much of a limbic system, but do have some impressive problem solving skills. And functionally, these functions are interrelated in man, who can, for example, experience moral disgust which bridges higher-order and emotional functions. This is why the triune brain is the kind of thing you would find in an undergraduate psychology textbook, but it's not the kind of thing neuroscientists talk about among themselves.