How to ensure proper peer-review for a self-published book?
In mathematics, traditionally, and I think currently, very few books are reviewed at all in any serious sense. One possible sense would be "critical appraisal", and this is rare: a handful of books is given serious reviews in the Amer. Math. Soc.'s "Bulletin". The other possible sense of "peer review" is in a sense adding something to one's CV for jobs, tenure, promotion, grants. Almost by definition, this never happens. That is, "refereeing" is done for the conventional journals, whether paper or electronic, and referees are solicited by the editors, for papers submitted to that journal. The "valuation" of books is typically done not via "reviews", but by the status of the publishing house, and the status gain from publishing a book is probably less than a small paper in a medium-status journal. Thus, in particular, self-publication confers none of that cachet. And I'd wager that self-publication would compromise "review" in the sense of appraisal, as well, because other publications would have higher priority for the reason of status.

(I do hope things improve, but I do think this is the current state, in mathematics.)

Edit: As examples of "self-publication", disregarding the "peer review" idea entirely, a certain number of relatively senior (and not only!) people put book-like items on-line at their web-sites, often at universities. I heartily endorse this (and have been doing it myself for quite a while!) However (?), this appears to confer even less status than "books" published in physical hard-cover, I presume because there has been no hurdle of sufficiently-impressing publishers (who do ask opinions before "consenting" to publish). 

The more legitimate issue of "(peer?) review" to ascertain correctness, or helpfulness, etc., I think has so few precedents that little is happening. First, Math Reviews has no procedure in place to review such things, and it doesn't happen. Second, there appears to be considerable reticence to cite such things, even in stable situations, for a variety of (not entirely sensible) reasons. Third, while one might imagine that on-line, thus, dynamic, documents could be more reliable, by virtue of being correctible (indefinitely!), this actually disturbs/perturbs many people... Further, third(b), disappointingly to me, very, very few people have ever given me any feeback/corrections about my on-line stuff, or even asked for clarification (in some cases leading to correction or, anyway, better writing). I can imagine that some of this is politeness, or respect, which is understandable.

But, I might claim, the real "problem" is lack of precedent. The "refereed journal" model is 150+ years old, and itself depended upon evolution beyond the "reading before the Academy" 200+ years ago when printing itself was a non-trivial matter, etc.

Thus, rethinking the action-oriented sense of the original question: since "self-published" ought mean nothing, really, in today's context, apart from the fact that it doesn't have a "prior" approval from status-conferring entities... to "get an expert opinion" one ("gosh, let's just try to think clearly for a sec!" :) would send th'thing to (web-obtained?) experts, asking very politely whether they'd be so kind as to offer critical remarks... and as a very polite secondary question, whether they'd be willing to be quoted in such remarks.

(This has led me to thinking that the difficulty in quantifying "civility" and "politness", especially between different generations (if only in convention and usage), potentially causes substantial difficulties in on-line forums, and/or "stack-exchanges", and or . Not that I think high-status entities have a "moral" superiority, which is the stereotypical exaggeration-to-disqualify, and which is a popular (mis-) interpretation of that elite, but that experience can be worth something, and that something is not easily acquired by any other means.)