College in high school and math graduate school admissions
You are getting worried about nothing.  You will not be judged negatively for having taken college classes before starting your BA, even a lot of them.  Graduate committees want to have students coming into their program who are going to be successful, and they don't care much whether it took 6 years or 2 years to get that preparation.  They are going to look at what you did with your time, not how long you took.  As long as you're happy and it's not too much of a financial hardship, there's no reason to graduate in 2 or 3 years.  Instead, take the time to concentrate on things that will strengthen your application:


Take the hardest courses available which you can perform well in.  
Build relationships with faculty.  Taking graduate courses can help with this, but you can also seek out research opportunities or get involve with things like math club.


Performing well in challenging classes and having stellar letters of recommendation are the basis of a strong application to graduate school.  Issues of timing make absolutely no difference in comparison to them.  Other considerations:


You could do a second major, if there's something else you really want to try.  I think this has probably taken on a totemic significance with students beyond its real import (math grad schools are mostly concerned with how you've done in math; being well-rounded is something like a tertiary consideration), but it sounds like you have the room in your schedule, and certainly no one will think you're lazy.
Find good ways to occupy yourself over the summers. REUs are one good possibility. It sounds like you'll have a decent shot of getting into one after your freshman year. Another is if you can find a counselor position at summer program like PROMYS.  
Consider spending a year or semester somewhere else. There are highly regarded study abroad programs specific to mathematics in Budapest and Moscow, and also programs like Penn State's MASS.  There's also Part III in Cambridge, which perhaps you could do after graduating in 3 years (or in 4).