What is the motivation model behind StackExchange?
The short answer is that anyone who gives you a short answer should be treated with suspicion.  ;-p

The StackExchange was founded in 2008, and has seen a lot of changes since that time.  That has provided the science community with little time to study the phenomenon in much detail.  As such, there are many many theories, but little consensus, and no comprehensive model in place.

There are several related articles on Wikipedia - one thing I can recommend for now, is to keep an eye on them.  Wikipedia tends to be updated with new evidence-based findings over time, so as knowledge progresses, it will likely be featured there:


Motivation for creating user-generated content
Motivation for online participation
Motivation for contributing to Wikipedia
And in particular: Motivation for contributing to Q&amp;A sites


Speaking of Wikipedia, it is clear that points and badges are not necessary to motivate contributions, as many crowdsourcing systems succeed without them.  Presumably, social psychology (eg, recognition, identity) plays a big role, as does intrinsic motivation (eg, ideology, curiosity).

However, what would happen if such extrinsic rewards were removed is a question about a controversial phenomenon called the overjustification effect:


  ... gamification refers to the application of game design elements to
  non-game contexts ... by providing symbolic rewards such as points,
  badges, or virtual currency. However, a number of academics and other
  critics have expressed concern that these rewards may backfire via the
  overjustification effect. ... these critics of gamification express
  concerns that gamified contexts ... might provide expected rewards for
  activities ... and therefore reduce intrinsic interest in those
  activities.


That is, if this effect is real and applicable to the StackExchange, then there is a danger that initially intrinsically motivated contributors might become dependent on the extrinsic rewards over time.  Complicated.

In reality, like any product or service on the market, StackExchange is subject to market forces, corporate decisions may be influenced by motivational models as well as user feedback, but are often made for other reasons, and therefore its sustainability is not easy to predict.  It is likely the case that the underlying motivational model is multi-faceted, with each aspect of the system contributing to (or detracting from) its success, interacting in a multitude of different ways with individual user attributes, and changing over time.