Predator prey interaction
I disagree with GForce's explanation; the meaning is not that growth of prey populations causes instability in predator species.

The sentence is merely saying that without predation, prey population growth is more likely to be at a level which leads to ecosystem instability. The term "but for predation" means "if it wasn't for the effects of predation". In other words:


  "Ecosystem instability can occur when population growth of some species goes unchecked by predation."


See here for more explanation, where this example comes from in which it says that running a red light caused a crash:


  "but for running the red light, the collision would not have occurred"


Biologically this makes sense in the sentence you show; without predators a species is limited by its supply of resources, and it can use these resources at an unsustainable level, whereas if you add predators to the mix there is additional extrinsic effects on population size, not determined by ecosystem properties such as space or nutrients.