What happens if a math PhD student fails to find a proof that is the main objective of his/her thesis topic?
I wrote a failed dissertation myself.  No papers resulted from it.  Fortunately, I had a significant side project which was published, and 11 years later, the main thread of my research still comes from this side project.  (I suppose I could have written up the side project as my dissertation instead, but I didn't.)

First, no project is ever a complete failure.  At the very least, you manage to prove some (possibly almost trivial) special cases, you prove some helpful lemmas, you find that certain lemmas you might have hoped would be helpful have counterexamples, and you find certain methods that you might use don't apply because some hypotheses needed to use those methods aren't satisfied in your case.  It's possible to write all of these things up, ending up with a dissertation that's basically about "How not to solve Problem X (except in this very tiny special case)".  (This is basically my dissertation.)

Second, it's rare for mathematics graduate students, especially at top programs, to work on only one problem.  Your advisor might suggest a main problem for you to work on, but you go to seminars and hear about other problems, talk to other graduate students or postdocs and learn about other problems, and so on, and graduate students are generally encouraged to spend at least a little time thinking about these other problems.  If you get stuck on your main problem, you still have other problems to solve, and it's quite common that what you learned to work on your main problem ends up helping you in solving these other problems instead.  (This is basically what happened to me, except this other problem didn't end up in my dissertation.)

Third, especially in the early stages of working on a problem, advisors are usually fairly quick at pulling the hook if it looks like no progress is being made.  Most advisors know of lots of problems, and they know what complete lack of progress due to a problem being too hard looks like.  Even later on, advisors can sometimes suggest simpler problems that can be solved in a shorter time frame (given what a student has already learned).  In some cases, after a few rounds of failed problems, the student ends up with a dissertation that's about as weighty as a half-decent undergraduate research project (and results in zero papers or one paper in a "write-only" journal).

It's true that a dissertation written out of a failed or almost trivial project tends not to bode well in applications for jobs where research matters (unless there is a more substantial side project).  Sometimes an influential or convincing advisor can make a strong enough case in recommendation letters for the student to get a postdoc, but this is harder now then it was 10 years ago given how much more competitive the job market is.