Should a math paper use non-alphabetical ordering of authors' names for unequal contributions?
Edit: this answer is about pure math.
It’s not common. In 25 years of reading math papers I‘ve only ever seen one example of a paper that had a nonalphabetical author ordering. It was an English translation of a Russian paper from the 1970s written by a PhD student and his advisor. I don’t know what the story was behind this unconventional author order, but all I know is that the advisor’s name appears first even though alphabetically he comes second.

In the situations of extremely unequal contributions, is it reasonable to break from this tradition?

No. Since you’re going against the overwhelming norm in math publishing, no one will have a clear sense of what exactly you’re trying to say. Also, consider the fact that it could be that the author order that reflects the actual author contributions might coincide with the alphabetical order (for example in the paper I mentioned, it was competely random that the PhD advisor came second alphabetically and could easily have been the other way around). In that case the signal will be lost entirely.
My conclusion is that the idea of forcing this signaling mechanism on a math publication culture that isn’t adapted to it is illogical, ineffective, and will lead to inconsistent results that are only very weakly correlated with the effect you’re trying to achieve. If I were a journal editor and I received a submission that tries to use this mechanism, I would tell the authors to find a different way to say what they want to say about the author contributions.