4th year PhD, considering leaving for different program
My dilemma was broadly similar: I was doing a PhD in a top-tier university I hated; my advisor was awful; and I was considering leaving for a different program. What stopped me from actually doing the latter was that even with explicit invitations and introductions from helpful professors who liked me, I didn't get anything more useful than a desultory suggestion to find the department's webpage and send an application there to start again as a first-year grad student. I was three rather than four years in at that point, but I thought that was enough time that I didn't want to restart from scratch. 

In retrospect, that was a bad decision. In my case, I finished the PhD program but didn't get the contacts or publications that I needed to get an actual academic career out of it, so the five years I wound up spending at the university were a waste of time. Grad school can be miserable, advisors can be worse than worthless, and the academic market is terrible. It's just about the only way of getting the opportunity to do certain things, though, most notably serious research. If you're trying to just tough it out and finish, that may not actually get you what you want.