How to choose between multiple math postdocs offers?
nagniemerg is absolutely right that at the end of the day, your results will be much more important than a name on your CV.  I think being at a prestigious institution is very useful for getting a first look at your application (which is a f***ing valuable thing, as Rod Blagojevich would say); if the rest of what's there doesn't stand up, getting that look won't help much.  

That said, I would think carefully about how sure you are that institution Y really will provide a better research environment.  Prestige tends to correlate (far from perfectly) with having an active department with a lot of seminars and visitors, which can often be more valuable than day-to-day contact with a single advisor.  Having higher-quality students and colleagues can make you a better researcher.

I also think there's a lot to be said for the uncertainty of life.  Maybe you'll go somewhere and whoever you were going to work with will get a job somewhere else, or have a baby, or go on sabbatical.  Maybe you'll start a collaboration with someone you don't even know exists yet.  It's all very hard to predict, and on some level you have to go with your gut.  You could do a lot worse than just going with the most prestigious option, but if other factors seem to point another direction, I don't think you should just ignore them.

EDIT:  I wrote this is a bit of hurry last night, and I realize it might be a bit unhelpful. However, it's honestly quite difficult to say anything general, and you indicate that senior mathematicians who know you and know what your options are cannot come to a consensus.  To me this indicates that probably you will not be able to conclusively figure out where is the better choice.  This is not to say it doesn't matter, but once the uncertainties are sufficiently large, one might as well flip a coin (or at least flip a coin until you get the answer you want).