Is it advisable to try and avoid attending a PhD program from a U.S. public university?
Should I avoid applying to public universities to do a math PhD and instead try to attend a privately funded school?


No, it doesn't make sense to decide anything at this level of granularity.  You are thinking about some of the right issues, but oversimplifying the situation too much when you boil it down to public vs. private universities.

Public vs. private is literally one bit of information, and your graduate school decisions should be based on far more information than that (if you change your mind based on this one bit, then you are either overreacting or suffering from a tremendous lack of information).

Before deciding where to apply, you should talk with people you know (faculty, other students) and look into different options online.  Once you have offers, you should investigate the programs that have admitted you much more thoroughly, including visiting in person as well as gathering whatever other information you can.

Facilities, faculty and student morale, and grad student funding are all factors that you should consider, but you should evaluate them based on the specifics of each program, rather than by jumping to conclusions based on whether it's a public university.  (Keep in mind that both public universities differ dramatically from each other, as do private universities.)  These factors may also be outweighed by others: Stanford is much better off financially than Berkeley is, but your mathematical interests could be a better fit for Berkeley.