Is the gap between lower- and higher-ranked graduate programs in pure math really that big?
Looking at the problem as someone who has been on a variety of hiring committees and grant panels,
I would say that in general, having someone with a good reputation in the field writing a strong letter of support is the most important factor, together with the journals where you have published.

In view of this, the school doesn't really matter: if you do your PhD in a small school but with a supervisor with a solid reputation who produces a multi-page letter highlighting well your contributions to the field, you will usually fare better than if you come from a very well known school with very little endorsement from your supervisor. Of course, some people tend to get a little bit star struck, but the dynamics of a hiring (or funding) committee usually smooths things out. Actually, contrary to what Dave Kaye was saying, I would argue that the school matters less in academia, more if you want to venture out into the real world, where name recognition is more important and might help you land your first job.

So why favor a good school? Well, there will be more visiting speakers of a high caliber. There might be a wider variety of grad courses to choose from. Overall, probably a more enriching experience. The difficulty level will not be very different, if at all, once you have been admitted.