Fitting a model as a significance testing alternative
It's probably a quite subjective matter, here is my opinion:


I would say that this method makes a lot of sense if the ML model uses a number of features, something which would be hard to factor properly in a standard statistical test (especially if some kind of feature selection is involved).
Also in favor of the method, statistical significance tests are usually interpreted in a binary way (typically p-value higher or lower than 0.05) even though the underlying p-value is continuous: there's very little different between a p-value of 0.049 and 0.51 but the two will be interpreted in a completely opposite way.
However I agree with you that statistical significance tests are much more grounded in statistical theory, with a clear interpretation especially about levels of confidence.


Overall I can see an interest of using such a method from a practical viewpoint, but it is indeed limited in terms of theoretical interpretation and reliability.

Disclaimer: I'm not a statistician, I might be biased towards practical ML methods ;)