What are the challenges faced by using NLP to convert mathematical texts into formal logic?
I can see several challenges, and the list below is not exhaustive:

i. The main problem is how to model a problem of translating a language test into a formal language. It will probably be something like the automatic translators, but with some guarantees that the proof semantics will be preserved. If you are more interested in this path, I recommend researching what PAC, Information Theory, Computational Proof theory, Complexity theory can contribute to this modeling.

ii. Another problem is how to get the data reliable. You commented that as people used it they would generate this data. But the problem is not just collecting the data. How much you will trust the data and how you will measure the model's performance in translation.

iii. Another problem is more humane, how do you get mathematicians to use such a system? And how to make the model self-explainable.

I believe that this is one of the most difficult problems in machine learning. I once saw this video a while ago and I don't know if it can help anything. I also recommend the stack exchange of theoretical computer science, there you will probably have a more complete answer.