Where to start learning economics as a mathematician?
McCloskey's Applied Theory of Price is somewhat legendary as a model of clarity. It is, sadly, no longer in print. However, a PDF is available here: http://www.deirdremccloskey.com/docs/price.pdf. The level of mathematical sophistication in this book (like in most good economics!) is not that high, but the book gives a deeper look at the important concepts than is to be found in most extant introductory undergraduate texts.

If you really want a more mathematically formalised treatment then you are going to have to turn to graduate-level econ books. For microeconomics, the classic text of choice is Microeconomic Theory by Mas-Colell et al.

If you do go down the more formal route, I would urge you not to give up on simpler treatments altogether; they're great for building intuition. More than a decade of experience in economics has taught me that good intuition built through simple models is often far more valuable than a less intuitive understanding of a more sophisticated model.