Is it possible to pursue Physics PhD with poor mathematics background?
As the comments (so far) say, it is very hard to succeed in physics without quite a lot of mathematics. But you don't need everything in mathematics to do so. Some skills are more important than others. Statistics can be important in some fields. 

But, first you should figure out which aspect of physics you want to work in. It is an incredibly wide field. Then, look to see which parts of math are most important there. A trusted faculty member can help you with this part, perhaps. 

But then, focus on learning those parts of math and the background necessary to understand it. Lots of people do poorly in math because their early teaching was inadequate and they missed the foundations on which to build. If that is the case here, then your path may be long, but it is possible to follow it. 

There are several parts of mathematics that require insight, not just facts and skills. Those require work to achieve the insight. And the insights are hard to transmit from one person to another. But math is a problem you will likely need to solve to be a success in academia as a physicist.