Finding more mathematicians who can write recommendation letter(s) for me
It is certainly ok for your letter writers to be in an area that is different from your focus for your PhD. Based on my experience, this is more the norm than the exception.

The letters should describe why you will be successful, not something about particular research fields. Having strong letters from your physics profs will help you far more than a weak letter from a math prof that barely knows you.