Is admission standard for math PhD significantly higher than that for physics PhD?
This is an attempt to gather some data supporting or refuting your hypothesis (or rather a slightly different one).  Ideally, we would like a direct comparison of admission rates at top places, but I could only find limited data on admissions rates, so let me start elsewhere.  At any rate, some of this data may be of interest.

The annual number of bachelor's degrees in physics is about 8000.
From the AMS's annual survey, this number for math is about 28,000.  This suggests there may be a lot more PhD program applicants for math.  However, I don't have data separating out which math degrees are on a math ed track (or similarly for physics, though I guess the numbers are much greater for math ed), and these people are unlikely to pursue PhDs.

What about actual numbers of PhD students?  I didn't see 1st year PhD numbers in physics for recent years but this slightly dated data puts it around 3000 new grad students in physics/astronomy (with about 93% aiming for PhDs), whereas the AMS annual survey has around 3600, and around 5000 if you include masters programs.  (Stats and biostats is separate with around 2000, I guess including masters.)  These statistics also say the number of physics versus math phd's awarded in recent years are pretty similar (about 1500-1600 for physics compared to 1400 for math).  So there may be many more "potential" PhD applicants in math, but both math and physics students seem to compete for roughly the same number of slots in grad programs.  (I don't know about how many of the PhD enrollments were domestic BS/BA holders, but we might guess the numbers are comparable as about 54% of enrollments were US citizens.)

So the above data tenuously supports your hypothesis.  Can we check this with some actual admission rates?

For physics schools, this website has grad school admission rates.  For top schools, the admissions rate seems to be around 10-15% (though Penn State seems to be an anomaly).  Unfortunately, I don't know such a nice tool for math schools, but a few math departments mention their admission rates. Northwestern is around 17% (about the same as for their physics program, 16.4%).  Notre Dame's is around 20% (a little lower than their 26% for physics).  

These were all I could easily find and I'm afraid it's not enough to make any real conclusions, but I might speculate that top math phd programs are only somewhat more competitive than top physics ones if at all.  (And in terms of undergrad research experience, I would guess that's more common in physics than in math.)

Edit: One qualitative issue for why you're seeing what you're seeing could be that top schools in math get lots of applications and if an admissions committee isn't familiar with a department, it doesn't know how to evaluate a transcript or the letters of recommendation from there, so it will tend to play it safe and accept students from places it's more familiar with.  This is one reason why it's very helpful for students at small, relatively unknown schools to do programs like REUs (or a master's first) where a letter writer from there can compare you with a wide range of students.  That said, I know many people who have gone straight from small, relatively unknown schools directly to top math PhD programs.