Importance of undergraduate institution in grad-school admissions (math)
I predict that you will be fine. The admission to a grad/doctoral program depends much (much) more on what you do than on the ranking of the institution you attend. Rankings can be deceiving and no one sorts candidates by the rankings of, for example, news magazines. 

If you do well and impress people so that you get good letters of recommendation then you will be in contention. 

That said, the competition for slots in top programs is very fierce. 

The effect of mere college rankings would be small. But the effect of good grades in hard courses with good letters is large. 

In particular, an A student from a state school (almost any of them) would be in better shape than a C student from a school in anyone's top five. 

Work hard. Take advantage of opportunities. Work closely with a few professors.