Will taking no courses from physical sciences as an undergrad affect my chances of going to grad school for mathematics in the US?
If you're applying to a pure math program, then it's unlikely that graduate admissions boards will be expecting courses in a particular outside discipline. If you're doing applied math, however, it would be much more logical to expect to see some coursework in an outside discipline.

However, economics may be suitable as an "outside" discipline if you're taking advanced coursework with a mathematical bent to it.

But overall, demonstrating that you are a strong candidate will make a much bigger impact than the specific training you have. Research experience, a strong statement of purpose, and great letters of recommendation will increase your chances of admission much more than if your "off-subject" courses are in economics or physics.