Assessing quality of research groups within math departments as a prospective PhD student?
For both your questions, I will posit that the key factor is whether the group in question is doing "high impact" work. For example, are their papers interesting and often cited? Are they working in very "popular" areas? Do the respected researchers in your (sub)field know the names of the senior group members?

To your questions:


I am not a mathematician, but I do know that the rankings will be at the level of the "department average", where faculty hires will be based on the "group" or even the individual. So, a high-power group in a third-tier school is better than a lower-power group in a higher-ranked school. 
The factors listed above are perhaps the "true" criteria -- but, as an undergraduate, you are probably not in a position to judge your prospective advisors' work. This is why the rankings are so useful. If you have anyone at your current institution that you trust to be well-informed as to the state-of-the-art in your (sub)field, you may want to ask that person for some advice as well.