The importance of the title "Mathematical Physics" vs. "Physics" on one's degree for grad school/jobs
Switch to Math Phys. It won't hurt your job prospects at all, since you don't want to be an experimentalist or phenomenologist. Math Phys covers any decent theoretical physics ground anyway. You'll be doing yourself a big favor and it will even help you focus more on one area of research, and put you into a healthier social environment.