Switching into Math/Physics
Maybe you could head for something a little in between like chemical physics or biophysics or chem engineering where some of the courses to date are useful?
My advice is to push for getting your 4 year degree (assuming US) on time.  Just see bad things when people delay too much from the switching.  If you can switch majors while still getting done on time, fine.  Otherwise, do some masters or the like.  Perhaps after working.
You have the wrong idea about the math needed for physics. Analysis (as opposed to "calculus"), algebra (of the abstract sort), and advanced geometry are all really pretty marginal for someone working on undergrad 'zoics.  Read up on Feynman or just talk to physicists (not math types).  You need to be very strong at calculus, high school algebra, and ODEs/PDES.  That's probably your biggest gap right now--diffyQs.  NOT the more theoretical math courses that people talk about on the Internetz.  
Getting through a "math methods" book would be good.  If you learn this book from start to finish, that will put you in great stead:  https://www.amazon.com/gp/product/B005G14K86/ref=dbs_a_def_rwt_hsch_vapi_tkin_p1_i0 (get a used earlier editions, don't spend $100, I have the 5th.)  Many physicists like Mary Boas (little easier but similar).  The traditional math methods for physics is Arfken Weber, but while it has a few harder topics, it's really a miserable grabbag and doesn't teach well.  I like Kryezing better for self study.)
Finally, I would caution you on the income aspect of physics (or math).  Unless you are a real superstar (which can be rare), you will face huge competition for academic jobs.  These forums are littered with people struggling to find jobs or unhappy with their advisor situation (and the relative subservience required in Ph.D. relationship).  Getting a certified job like a physician is a great move financially and in terms of societal prestige.