Can you study and contribute to physics if you are a mathematician?
Yes, technically speaking, you can cross the fence -- either direction -- between those two disciplines, but be aware of their different agendas.

Physics uses the language of mathematics to construct mathematical models of reality, but the physics is really contained in the following three non-mathematical tasks:


Establishing the association between mathematical symbols and reality (interpretation of the mathematical models).
Verifying the validity of the conceived mathematical models (physical theory verification). 
Establishing their limits of validity.


You can be a great mathematician, but if you fail in understanding the above not-so-easy tasks, you will be a bad physicist.