Fundamentals of mathematics and physics for physical chemistry
There are three levels of scholarship. An analogy is that someone has looked at the ocean from far away and he feels that he has seen it all (standard undergrad mindset). Next stage, is when you walk on the beach and your feet become wet and dirty with sand. You have a feeling how ocean water and the beach feels like (fresh MS/PhD). The higher level of scholarship is when you are deep diving into the ocean and realize there is too much to learn and the knowledge you have is quite limited (a mature scientist who has spent his life in research).

Since you are saying that you have not taken mathematics and physics courses, it may not be the right time to begin a proper physical chemistry course. 

Couple of nice books:

Mathematics in Chemistry An Introduction to Modern Methods by Hecht

Mathematics for Physical Chemistry by Robert G. Mortimer  

Get hold of them from a library and get a feel for it (the first level of scholarship). You don't need to master all to open an undergraduate physical chemistry book. You rather learn it along the way. See if you like the subject in general.