I got accepted into a math PhD program but I don't feel adequate enough to attend
First off, congratulations on being accepted to the PhD. That means the faculty have carefully evaluated your application materials and decided that you are a promising young mathematician. They are experts and they think you have what it takes to finish a PhD.

No one gets admitted just because of their gender and/or race. It would be a waste of the department's resources to admit unqualified students and it would be a disservice to those students -- setting them up to fail.

It sounds like you have Impostor Syndrome. The truth is that almost everyone, from time to time, feels like they don't belong. I have felt that way at many points.

No, you should not "just quit". You have a great opportunity in front of you. It sounds like you have other things that you want to consider doing. Perhaps you will decide that those other opportunities are better for you. But, if you do turn down the PhD, you should not think of it as quitting. You should think of it as taking a positive step in another direction.

That said, academia is hard. And I would not advise anyone to do a PhD unless they really enjoy research.

You should think seriously about what you want to do, as it sounds like you are uncertain about where your best option lies. Talk to people you know personally, as they can best advise you. 

One thing to note is that a PhD should give you some flexibility to study other things or, better yet, to combine different research topics. My PhD program allowed — required, in fact — me to take graduate courses in totally different fields.