Is having five letters of recommendation for STEM PhD programs (math, specifically) ok?
This can depend on the specific university department. Some may welcome it as supplemental material to your application whereas some may discourage excess letters and tell you to choose the ones that will best showcase your abilities as a student. Don't add things to your application that the department doesn't welcome.

I've generally heard and been told that an additional good recommendation letter will boost your chances of admission,
but an additional generic recommendation letter that does not add anything unique to the other (assuming) 4 letters will only hurt it.

So for a general answer, make sure each recommendation letter is able to showcase different areas of skill rather than them overlapping with each other. Also, make sure to contact each department to make sure that they are okay with having more than 3 recommendation letters.

Hope this helps, good luck!