what is the reptile brain?
The reptilian brain is the oldest part of the triune brain. And the triune brain is a unified account of brain function and brain evolution. 

The story goes like this. The brain can be divided into three sections:


The reptilian brain, named so because it encompasses structures that did not change much from reptile to man. It includes the basal ganglia and cerebellum, and controls pretty much everything that a reptile can do: breathing, walking, appetite, sensation, simple motor reactions to the environment, etc.
The mammalian brain (sometimes called paleomammalian), which encompasses structures that are unique to mammals but similar across mammalian species. These structures belong to the limbic system and are dedicated to processing emotion and memory. Anything your cat can do, should be either in the reptilian or the mammalian brain. Anything it can't do, is in the third part of the triune brain.
The neocortex, found in primates, which is involved in higher order functions such as language.


Now, this is an oversimplification both structurally and functionally. For example, birds don't fit this theory very well because they don't have much of a limbic system, but do have some impressive problem solving skills. And functionally, these functions are interrelated in man, who can, for example, experience moral disgust which bridges higher-order and emotional functions. This is why the triune brain is the kind of thing you would find in an undergraduate psychology textbook, but it's not the kind of thing neuroscientists talk about among themselves.