Graduating from a terrible department in an ok University vs. a fantastic department in an obscure university
Generally speaking, for your future academic career, prestige of University as a whole is less important than prestige of department, and that is less important than prestige of advisor. And all of that is tempered by the strenth of reference letters you will obtain, and by the directly observable quality of your work. (By the latter, I mean how good your dissertation / publications are, the specifics of this highly dependent on your field).
Also worth saying, however, is that if you leave academia, the rank order of those factors for getting a good nonacademic job is generally reversed. (Ph.D. students generally don't like seriously thinking about nonacademic career prospects since their goal is an academic career, but in many disciplines the proportion who will go on to nonacademic careers is shockingly high, so you should at least contemplate this.)
In your instance, I would add 2 factors to consideration.

It's clear you aren't happy where you are now, and feel you would be happy somewhere else. Life is too short to deliberately choose an unhappy road to a (good) Ph.D. if a happier one is or becomes available to you.

&quot;The grass is always greener on the other side of the fence.&quot; Don't overindex on the negatives of what you are experiencing now, and compare that with only expected positives of what your potential alternative is. In particular, look hard in the mirror: what have you learnt to do differently if you go to a new place, and do you have a tendency in life to frequently hit the restart button -- especially multiple times in a row? (Bluntness alert!) Everyone in academia develops pattern recognition for smart people who somehow serially fail to deploy, and are always restarting, but the restart only leads to another restart later on. We don't know you, so don't treat this as an implied accusation that this applies to you  -- but do look in the mirror to consider whether this is the case, and if so, what you need to do differently to not have to restart again.


With all that, best wishes, and do what (with self-reflection) feels right!