Can I allow my personal life to influence where I do my PhD?
So I'm just some random stranger on the internet, but you asked for advice, so here it is.  Given only the information in your post, I think you have a difficult choice ahead of you:

Move or propose.

From a strictly professional standpoint, you make a very strong argument for moving.  Having a more competent and more supportive advisor, and working in a field that excites you more, will have a tremendous impact on your several-decades-long mathematics career.    As Dan C writes elsewhere, if you want to do math for a living, you should enroll in the strongest PhD program you can.  (I'm assuming here that you actually want a career in mathematics; otherwise, getting a PhD in mathematics is kind of stupid.)

From a personal standpoint, you make a much weaker case for staying put.  It's just another city; you'll be fine.  You can always find another apartment.  Do not listen to the Impostor Syndrome; everyone is afraid that they're not good enough.  Your girlfriend is OK with your moving.  There's some possibility that your current relationship won't survive the move, but the fact that you call her your girlfriend, and not your wife or even fiancée, suggests that you don't really think of the relationship as permanent anyway.

Yeah, I know, that's harsh.  And maybe I'm totally off-base.  Maybe your current relationship really is more important to you than the rest of your professional life.  If that's true, tell her so.

Who knows?  Maybe you can do both!

But whatever you decide, don't base your decision on fear.  Instead, ask yourself: What do you really want?  What really matters to you?  Fear of change and Impostor Syndrome are completely natural, and far more common than most people think, but they're just voices in your head.  Letting them control your decisions is not healthy.  You have people in your life who believe in you, who support you, who want you to succeed and be happy, and who know you far better than Some Random Stranger on the Internet—listen to them!