How to proceed when a mathematical result contradicts existing literature?
Certainly the fact that there is a contradiction with previous literature must be prominently advertised; to do otherwise would be scientific misconduct. I would not be confident in publishing such a paper unless I found the mistake in the other paper, or could give a counterexample showing they are wrong.

After having carefully worked out the other paper and my own paper, and asked any experts with whom I have an established connection, I might contact the authors of the contradicting paper (if they are still active). After their response, or lack thereof, I would consider publishing a preprint and after that submitting to a journal. The journal is the slowest and most uncertain way of finding out where the problem is.