Chances of academically publishing a book without professional degree
As someone who has served as referee for math textbook proposals for reputed academic publishers, I can tell you that if your proposal were sent to me, I could not care less about whether you have a degree or what your title is. I would care about the content of your proposal and whether it convinced me that you can write a high quality, correct, and interesting book that serves a need for students and researchers of mathematics.
However. You should have no illusions that convincing me of such a thing is an easy standard to meet. It’s not. It wouldn’t be easy even for a professional mathematician with many publications under their belt, and I would expect that it would be doubly difficult for someone who doesn’t have formal training in mathematics at a level equivalent to at least a PhD, let alone an undergraduate degree.
But as I said, the lack of a degree by itself would not be a problem for me.