Importance of grades as a graduate student in math
In many (US, mathematics) graduate programs, there is quite significant grade inflation.  A grade of B would raise significant concern about your progress (and maybe cause advisors to think twice about working with you), and a grade of C would create grave doubts about your ability to continue in the program.  You may have a better sense if this is true in your program, but if so, it would be a really bad idea to reduce your efforts to the extent that your grades drop.

On the flip side, it is entirely possible that you could work a little less hard and still get A's (the standards for an A may be lower than you think).  I hesitate to recommend this, though.  The better you know this material, the easier your qualifying exams will be, and failing exams definitely gets students kicked out.  Solving all the problems on all the problem sets is a really good idea in that direction - rather than worrying about how much time it takes you, you should perhaps rather be glad that you can do it at all (I bet many of your classmates cannot).

Mathematics is unlike some other fields in that (US, PhD) graduate programs place a strong emphasis on coursework for the first couple of years.  A transition to research comes afterwards.  In many cases you cannot meaningfully discuss a research topic with a potential advisor until you have mastered the material from a course.  So it probably is in your long-term interest to prioritize your coursework now, even if some of your deeper mathematical interests are temporarily pushed to the back burner.  (Don't worry, they'll still be there after you pass your exams.)

It is good to spend at least a little time on pursuits outside coursework, and you can probably find some combination giving maximum payoff for minimum commitment.  For instance, attending seminars is good exposure to current research areas, and may require only an hour or two of your time per week.  (Protip: try to go to dinner with the speaker.  It's good networking, and your department may well pay for your meal (free food)).