Why is a public key insufficient to decrypt a message?
But couldn't a third party ... reverse those steps?


Simply said - no, it is impossible to reverse those steps. (this is true under certain conditions, how the keys are generated, etc.)

Why not? One of reasons is that RSA uses modular arithmetic. All operations are done using some modulo. If you know that $X mod A = B$, can you find the X? There are unlimited solutions: X = B, X = A + B, X = 2A + B, X = 3A + B, etc. In the reality it is more complex, like $X^{C} mod A = B$. We can say that such operation is not (easily) reversible.

For further reading see the link suggested by @kelalaka. When you have then further questions, welcome.