What prevents predator overpopulation?
No, I don't think auto-regulation explain much in the population sizes of predators. Group selection may explain such auto-regulation but I don't think it is of any considerable importance for this discussion.

The short answer is, as @shigeta said


  [predators] tend to starve to death as they are too many!


To have a better understanding of what @shigeta said, you'll be interested in understanding various model of prey-predator or of consumer-resource interactions. For example the famous Lotka-Volterra equations describe the population dynamics of two co-existing species where one is the prey and the other is a predator. Let's first define some variables…


$x$ : Number of preys
$y$ : number of predators
$t$ : time
$\alpha$, $\beta$, $\xi$ and $\gamma$ are parameters describing how one species influence the population size of the other one.


The Lotka-Voltera equations are:

$$\frac{dx}{dt} = x(\alpha - \beta y)$$
$$\frac{dy}{dt} = -y(\gamma - \xi x)$$

You can show that for some parameters the matrix for these equations have a complex eigenvalue meaning that the long term behavior of this system is cyclic (periodic behavior). If you simulate such systems you'll see that the population sizes of the two species fluctuate like this:



where the blue line represents the predators and the red line represents the preys.

Representing the same data in phase space, meaning with the population size of the two species on axes $x$ and $y$ you get:



where the arrows shows the direction toward which the system moves. If the population size of the predators ($y$) reaches 0 (extinction), then $\frac{dx}{dt} = x(\alpha - \beta y)\space$ becomes $\frac{dx}{dt} = x\alpha \space$ (which general solution is $x_t = e^{\alpha t}x_0$) and therefore the populations of preys will grow exponentially. If the population size of preys ($x$) reaches 0 (extinction), then $\frac{dy}{dt} = -y(\gamma - \xi x)\space$ becomes $\frac{dy}{dt} = -y\gamma \space$, and therefore the population of predators will decrease exponentially.

Following this model, your question is actually: Why are the parameters $\alpha$, $\beta$, $\xi$ and $\gamma$ not "set" in a way that predators cause the extinction of preys (and therefore their own extinction)? One might equivalently ask the opposite question? Why don't preys evolve in order to escape predators so that the population of predators crushes?

As showed, you don't need a complex model to allow the co-existence of predators and preys. You could describe your model a bit more accurately in another post and ask why in your model the preys always get extinct. But there are tons of possibilities to render your model more realistic such as adding spatial heterogeneities (places to hide for example as suggested by @AudriusMeškauskas). One can also consider other trophic levels, stochastic effects, varying selection pressure through time (and other types of balancing selection), age, sex or health-specific mortality rate due to predation (e.g. predators may target preferentially young ones or diseased ones), several competing species, etc..



I would also like to talk about other things that might be of interest in your model (two of them need you to allow evolutionary processes in your model):

1) lineage selection: predators that eat too much end up disappearing because they caused their preys to get extinct. This hypothesis has nothing to do with some kind of auto-regulation for the good of species. Of course you'd need several species of predators and preys in your model. This kind of hypothesis are usually considered as very unlikely to have any explanatory power.

2) Life-dinner principle. While the wolf runs for its dinner, the rabbit runs for its life. Therefore, there is higher selection pressure on the rabbits which yield the rabbits to run in average slightly faster than wolves. This evolutionary process protects the rabbits from extinction.

3) You may consider..


more than one species of preys or predators
environmental heterogeneity
partial overlapping of distribution ranges between predators and preys
When one species is absent, the model behave just like an exponential model. You might want to make a model of logistic growth for each species by including $K_x$ and $K_y$ the carrying capacity for each species.
Adding a predator (or parasite) to the predator species of interest

... and you might get very different results.