How do I know if I'm unprepared for my math PhD?
The fact that your adviser thinks you're doing fine, in particular, suggests to me that you may be dealing with some impostor syndrome.  

I think when most of us start out in PhD programs, we feel like everyone around us is doing much better.  Like they all belong here, but we don't.  This is usually accompanied with a feeling of "I had better keep my head down and stay quiet, so no one notices I don't belong here and kicks me out."

You don't say much about your conversation with your adviser.  Hopefully it was an honest conversation about why you're concerned.  As long as it was, consider trusting that they know what normal looks like.  You might also want to talk with some of your classmates more - you may be surprised to find that the ones who seem to be endlessly more successful than you are feeling identically.

PhD work is usually a whole new level of difficult, so it's not necessarily a warning sign that you feel like you have to study for hours to get anywhere.