Level of difficulty of exams in math PhD and anxiety over being able to do well
This depends very much on your university and your professor. There is a whole range of opportunities spanning a wide range of difficulties. This issue alone shouldn't be the reason to drop the idea of the PhD degree itself.

If you are really interested in the field and are willing to work for it, then by all means, continue; don't let a little test put you off. PhD is not about exam scores, it is about research quality and how much you could contribute to your field. If you are able to hold your ground on that, stay strong and move on.

My best wishes!