What's my chance of getting into a good math or statistics graduate program, if I don't have impressive grades the first three semesters of college
You major GPA of 3.98 is what counts most. "Full of B's" and overall GPA is 3.5 means "half A's, half B's," which is not at all bad. The one C and one W aren't great, but don't necessarily hurt you much. "Everyone" knows that when you take courses outside your major for "balance," you will likely get "worse" (and a few bad) grades. That's especially true the first three semesters. 

I'd expect your GPA to rise to about 3.7 as you take more courses closer to your major in the later years. As long as you "finish" on a strong note, that's (mostly) what counts.