When applying to a Phd in Physics or Math , do grades in unrelated courses I took have any effect on my admission?
If you are applying to programs in the US, more important than your grades in English classes is your performance on the TOEFL. Many universities have a minimum TOEFL score required before the department can even consider your application, and these rules are often difficult or impossible for the department to bend, even for an exceptionally attractive candidate.

There is more information in the answers to this question. (For example, someone there mentions that the minimums at "Yale and Stanford are about 100, Harvard is 109" -- out of 120 points total.)

Given the quality of the English in your original question (before it was improved by helpful editors), I don't think you can afford to blow off your English classes if you have any hope of studying in the US.