How to evaluate a book, rather than a paper, on a scientific subject?
Obviously, this question asks about reliable predictors for the quality of a book. Otherwise the answer can only be: You evaluate a book by reading it. With this out of the way:


Ask around. In particular when you are interested in using a textbook for a course, ask people who have directed similar courses.
Publishers. There is an informal hierarchy of publishers that can be used to predict a book's quality, just like conference venues and journal ranks for papers. This differs between disciplines, but the edge-cases are vanity presses (low-end) and Famous University Presses™ (high-end).
Citations. Just like for journal articles you can count the citations of a book to estimate its impact (with all the known caveats).
Reviews in journals. Many journals publish a review section. You can search these for discussions of the book that you are interested in.
Peer review. Some publishers also send full book manuscripts or at least the book proposal with sample chapters out for peer-review. 
Professionals. Ask your trusted librarian for advice.