Does a low GPA in an unrelated (second) major affect math PhD admissions?
For a math PhD program, the GPA in math is by far the most important.  Your overall GPA still counts for something, but GPA in a different major may not count for anything more: if you applied to my math PhD program (UGA), I don't think your philosophy GPA would specifically be considered by us.  If your overall GPA were something like 3.5 and your math GPA is 4.0: well, then you look like an overall good student who is outstanding at math.  That's what math PhD programs want!

Having said that: I note that (i) you intend to go to a math PhD program, (ii) you have a perfect(?) math GPA and a (perfectly respectable but) significantly lower GPA in your second major, and (iii) you describe your second major "as a spare time interest in high school".  I don't quite get it: you're not in high school anymore, so....tell me again why you're majoring in philosophy?