Mixed-mating plants: is the number of mixed mating plants supposed to be trivial?
I think the main message of the quote is

"Many would have expected that mixed mating can only exist as a transitional state and thought that stable equilibriums are either pure selfing or pure mixing (or both). The observation of a non-trivial number of mixed mating populations suggest that our expectations were wrong."

The first sentence of the quote is a statistical detail of their paper and not a premise for the conclusion that there is a non-trivial number of populations that show mixed mating.

The evolution of selfing is a large topic in evolutionary biology and cannot really be summarized in a single post (even more so that many question are still open to debate). Goodwillie et al. 2005 is a review on the subject that will help you understand the work that has been done on the evolution of selfing. Very related is the evolution of hermaphrodism, dioecy and other mating systems in plants. Pannell 2002 is a very good paper on the evolution of androdioecy and the maintenance of these unexpected mating systems.