Why aren't gene drives extremely common?
I have a feeling that the OP is confusing naturally occurring so-called selfish, or parasitic genetic elements and the CRISPR/Cas9-based gene drives that have been making news as potential tools to eradicate disease vectors like the aedes mosquito. My answer is working from that assumption.

The big difference between naturally occurring "parasitic" genetic elements (including bacterial CRISPR elements and things like transposons) and the gene drives that are being tested as vector control revolves around what we put inside the latter class of gene drive. 

The OP hit on this difference in his question, when he mentioned the decrease in fitness caused by engineered drives. In the naturally occurring drive-like elements, there is rarely ever a decrease in fitness in the organism and when it does occur, it is unlikely to be passed on to future generations. 

Transposons, for instance, are largely active in somatic cells, whose DNA is not passed on to descendants. Semi-randomly occurring genetic changes are sometimes passed on, but this is relatively rare and evolution favors those that are either neutral or that increase fitness. A brief search doesn't turn up, and I can't think of, any cases of naturally occurring deleterious genetic alterations caused by something like a transposon having been found to remain in a population long enough to be studied. 

Engineered gene drives, on the other hand, particularly in the cases of those meant to combat mosquitos, are purposefully designed to reduce the fitness of their target host. 

The last question, of what keeps naturally occurring drives at low levels, requires either a rather vague, or a very long answer. I'll go with vague and provide some links to longer ones. Briefly, our genomes have evolved a lot of ways to maintain genomic integrity, in the forms of proteins responsible for proof-reading DNA, repairing breaks, and destroying any "free" pieces of DNA found outside of a chromosome. We've also evolved a lot of mechanisms to maintain genomic integrity through small non-coding RNAs. 

Here are some good papers (without paywall) on the subject: 


How p53 restrains mobile genetic elements
Proteins that function to restrain mobile genetic elements in rice
Good review of transposons in general
Finally, how people are engineering gene drives to combat malaria


Hope that helps!