Does rigor/thoroughness of undergraduate program matter (for graduate/PhD applications)?
This would be a great question to discuss with your advisor, or another faculty member at your school whose opinion you respect.

My opinion, for what it's worth, is that the "bare-minimum" 3-year program you describe would be woefully inadequate to prepare you for graduate study in mathematics, and I think most admissions committees would agree.  The fact that you did it in 3 years with a double major won't mitigate that.  A year of analysis is pretty essential, some topology would be very helpful, and taking a significant number of upper-division math electives will be a great way to build what people call "mathematical maturity".

Your 4-year program sounds better, but honestly still a little on the light side.  Keep in mind that many applicants to top grad programs will have taken practically every undergraduate course their department offers, and maybe a couple of graduate courses besides, in addition to often having undergraduate research experience.  You also want to make sure you get to know (and impress) your mathematics professors well enough that they can write you strong letters 
of recommendation, and taking more math courses is the biggest part of that.  

The physics degree on your diploma will be nice, especially if you are looking to do a PhD in mathematical physics, but it may not go a long way to making up for weakness on the math side.