How good do I actually have to be to get into a top graduate program from an average undergraduate program?
From what you have said about yourself, I think you're very well prepared for graduate school at a very good institution. There are always a few incredibly bright, very young mathematicians that fit the image you have in your mind of students at top tier universities. I know of a few myself. However a very large number of math graduate students are much like you (and I).

I had a very strong affinity for math as a young child but grew up in one of the worst academic environments in the US. A highly nontrivial amount of students in my area didn't graduate high school, even fewer went to college and fewer then even went to universities that were not the crappy local universities. I lucked out because one of the best high schools in the nation was in my area yet I didn't feel very challenged and more or less coasted along and didn't try. Ended up with a GPA of about 3.0 at the time of graduation. I even spectacularly failed statistics with a 40 or so from lack of effort.

Undergrad rolled around. I flourished in the environment: took lots of math classes, took plenty of courses outside my degree to broaden my horizon, did lots of research, but like you I was a big fish in a small pond. I felt similar to you - that I only looked good in comparison to all of the others. I almost had a 4.0 in math and had a 3.8 overall with a couple of years of research under my belt. When it came time for grad school applications, I cast a bit of a wide net, but got rejected from a lot of universities. (This was largely my fault since I seemed to mess something up on every application. Having not really done undergraduate applications, I was a bit overwhelmed and scatterbrained.) I actually got into my top choice university which surprised me a lot.

When I started my master's, I did notice that there was a bit of a gulf between me and the other students once I got there. My undergraduate institution didn't have nearly the course selection or resources that the other students had. However in terms of math capability, I was at least middle of the pack; so while I had a bit of a learning curve and some growing pains, I think I ended up being one of the stronger master's students because I fully dedicated myself and didn't give up. I did spend a ridiculous amount of time in my office the first semester, though. I went to school around 9:30 in the morning and went home at midnight many days.

For all of it, I grew incredibly and am doing quite well in my PhD program. Coursework is a cinch now, research is going well and I'm writing up a couple of papers which I hope to submit in the not-too-distant future. With your background and love for mathematics, I think you'll be in a much better position than I was in terms of applications. Your GPA is better, you seem much more fluent in many areas of mathematics and are much more mathematically mature. The biggest factor in determining your success (outside of pure genius - which is incredibly uncommon, even amongst mathematicians!) is your unwillingness to give up. Even if you're slightly weaker than some of your future fellow graduate students right now, once you get there, all of those disparities will quickly melt away if you put in the time and effort. Where you came from does not have to dictate where you end up. You are more than your past if you allow yourself to be.



Here is some general off-topic advice regarding grad school since you seem to be lacking in advising: just because you want to go to a top tier university and get in to one does not mean you should necessarily go. For undergrad, this is not the case; if you get into MIT, Stanford, Harvard, etc. and have the financial means, you should definitely go as it is a great opportunity and can directly impact your future. At the graduate level, things are so much more nuanced. (For example, the reputation of the university doesn't directly dictate your future success provided that you put in 100%. In the internet age, it is much easier to do really meaningful research and network with top researchers at any respected university since you have access to all of the information you want.)

When you apply, you are not applying to the university more than you are applying to a professor (or professors) or at least this is the philosophy I think one should have. You should have an idea of what kind of things you want to pursue for research and you should look to those people who do research in that direction. I don't mean to say that you need to know exactly the project you want to do but have a rough idea of the field you are interested in, say commutative algebra or functional analysis or harmonic analysis. If you apply to a university in which no one really pursues what interests you, you're going to be without a future advisor or you'll have to settle for second best. Applying to a graduate program just because of prestige doesn't guarantee success or happiness. Granted, at the top universities, this tends not to be an issue as they have very broad reach but it is something to be wary about. Just because it is a great university or great department does not mean it is a great fit for you.

There are also other factors to consider when applying to schools: Could you deal with the super competitive atmosphere (or alternatively the extremely laid back atmosphere)? Could you stand to live there for four to six years, e.g. if you're from a very hot climate, could you survive the very harsh winters in upstate NY or if you're from upstate NY, could you handle a Texas summer? If you cannot see yourself being happy with (or tolerating) a lot of these extra-academic aspects of where you are, you might want to reconsider.

A PhD is very demanding and can be soul-crushing at times. You'll often run into really difficult road blocks in coursework or research and if everything else in your life makes you miserable as well, you're going to have a really bad time. Your mental health is very important. You will be pushed to the extreme at times and in many ways throughout your PhD and your environment shouldn't amplify this. If you're having a tough time in research, the weather is absolutely miserable and you cannot stand your fellow graduates for whatever reason, your mental wellness might take a turn for the worst. I have seen this happen first hand and it is really unfortunate. These are not things academic advisors often tell students who wish to do graduate school because they are easy to overlook, but it is something to keep in mind. Hopefully this has been helpful.