Is there some book about how to design a 3D math lib?
Books: http://www.essentialmath.com/book.htm is the one.

Best algos on the real 3D topics (and some tricky 2D like image resize and premult-alpha ), explained and with source most of the time.

The website is very nice too, the "tutorial" full of gems

Adding http://www.geometrictools.com/index.html which is the most complete source code repository on the topic, with loads of very precise algos implemented.