Not sure if grad school was the right choice
You're experiencing what a lot of graduate students go through—moving from being a big fish in a small pond to being a big fish among other big fish in what seems to be a vast, uncharted ocean. Feelings of frustration, confusion, and even bewilderment are normal. It's part of the adjustment process. The truth is that many of your classmates are likely going through the same difficulties.

The question to ask is not "Should you be in grad school?" The question to ask is "Do I want to be here?" If you do, then don't let imposter syndrome drag you down! Also, remember that research bears often little resemblance to classwork.

Now, on the other hand, if you've realized that you're not interested in doing a PhD anymore, then you'll be much better off figuring out what you'd like to do instead. There are a lot of career options for mathematicians, and wasting a bunch of years getting a PhD you're no longer interested in won't be of much use for you in the long run.