Can a PhD student transfer to another graduate school after advancing to candidacy?
Unfortunately, it's probably not feasible to start over elsewhere in the near future.  You can certainly try applying, and it might work out, especially if you have a really compelling explanation.  However, I think the chances are slim.

The basic question is how much progress towards a thesis you have made during your six years in grad school.  If you're close to finishing, then nobody will think it makes sense to start over from scratch in the same subfield (or even a related one).  At that point, you're not genuinely starting over, but just extending your time in grad school beyond six years.  In particular, continuing in a similar area will come across as a request for another school to provide the additional funding your current university won't give you, and that won't sound like a good use of money.  Most schools wouldn't give their own students a seventh year of funding, so why would they give it to a student from another university?

On the other hand, if you aren't close to finishing after six years, then starting over might make sense, but your track record will look bad.  You'll need to present a powerful argument for why you'll do much better the second time around.  In particular, there are people who have strong backgrounds and excellent grades but somehow don't end up succeeding at research, and the admissions committee will worry that you may fall into that category.  Taking a chance on a fresh graduate student will seem like a much better bet than admitting someone who has already spent six years in graduate school but been unable to finish.

So basically you're stuck.  Either you are too far along for starting over to make sense, or you have a track record of failure.

Of course it's difficult to say without knowing more about your specific situation, but I'd guess your best chances lie in two possibilities:


Even if you don't get admitted now, you might fare better in a few years.  As your previous time in graduate school fades into the past, you can try to make a case that you have increased perspective and maturity and a renewed desire to get a Ph.D. in math.  I don't know how likely this is to work, but at least it gives you a chance to explain that you have changed since your first attempt at grad school.
You could try radically changing your research area.  For example, from algebraic geometry to bioinformatics.  Of course this depends on being able to make a good argument for your change of interests.  You have to really demonstrate that you've finally figured out what you want to do, in the face of skepticism (many people will assume you are just flailing about looking for any chance to try something new).  However, it gives you a ready-made explanation for what went wrong the first time: you were trying to do something that in the end just wasn't suited to your interests and talents, but you've discovered that this new topic is a much better fit.


Whatever approach you take, I expect it will be difficult to get admitted.  It can't hurt to try, but I wouldn't get your hopes up.  My gut feeling is that it would be easier to try to finish your Ph.D. and then salvage your career (but of course I don't know enough about your situation to say that with any confidence).