How do people develop abstract thought processes? By abstract thinking, I am referring to not only notions of self and identity, but also aspects that can not be directly observed and 'thinking outside of the box'.  Essentially, from my own experience as a teacher, where a student gradually develops the ability to go beyond 'concrete' thinking (based on what they can sense), to be able to incorporate abstract ideas and knowledge (based on what can not be sensed).

An example of this is learning chemistry, it has been my experience in the classroom that junior science students don't always get the notion of atomic structure because they can not see or adequately visualise the details necessary; however, the can understand what happens in chemical reaction as it is clear to see.

What cognitive processes occur that allow a person to be able to think and problem solve abstractly?
A 2013 book tackles this subject : Surfaces and Essences, written by Douglas Hofstadter and Emmanuel Sander. They mostly cite anecdotal evidence (not only, Emmanuel Sander worked on this subject for years), but the theory is very convincing, and quite beautiful. The book received praise from some big names (including Steven Pinker and Elizabeth Loftus).

So, their theory in a nutshell : we naturally reason by analogy. Analogy is defined here as "recognizing something perceived as identical to something stored in memory" - a very large definition, that encompasses categorization. We are bad at purely abstract thinking (as the famous Wason (1966) task shows, or most tasks used in reasoning psychology), but we are very good at making abstract links between concrete things, without noticing it. 

The book is filled by loads of examples, a striking one is comptuer science vocabulary. Even though computers brought loads of new "things", few new names have been invented. For example : desktop, file, directory, keyboard, window, program, application... all those words existed before computers, they have been used by analogy. When one makes such an analogy between a real, concrete desktop and a computer one, one extends the meaning of the word "desktop" - before, desktop refered to the piece of furniture ; after the analogy, desktop acquires a new, more abstract sense that encompasses the 2 concrete (furniture and computer) : a desktop is a working space... where you can store files, etc.

So, for your question : it seems that we extract abstract structures as we encode concrete examples - we can't (or at least it's much harder) directly understand the abstract knowledge. Providing good analogies is probably a very good way to teach something - if there are no such analogies, the domain remains very hard to grasp (statistics, probabilites and quantum physics are probably good examples of that).