Pattern recognition and Machine Learning reference
Another good book is:
 - Pattern Recognition by Theodoridis and Koutroumbas

It's about classifiers, features and clustering in the context of pattern recognition.

It contains all the mathematical stuff you want, and backs everything up with references to papers, books, etc. It has no exercises, so for that you need to look somewhere else.

Also see this StackOverflow question on the matter.