Fitting a model as a significance testing alternative My colleagues use somewhat unusual approach to estimate how far away performance of some of our company related processes diverge from historical levels. Their nature isn't really relevant for the purpose of this discussion. We may just assume here, that the final result can be represented with some real number which fluctuates over time and has some seasonality components.

They fit a regression model (with XGBoost) on historical data excluding the most recent period (say, the last month) and than compare the most recent values with model's forecast.

I argue, that this is basically an overkill and some sort of a crooked statistical significance test. And rather than fitting a model and making a forecast, we should actually calculate some normalised historical indicators (stratified by relevant groups and taking into account seasonality, where appropriate) and then do a standard significance test using a significance level, that would make sense from a business perspective.


Am I right or not? Are there any better ways to approach a problem of estimating performance and comparing it to historical data?
Does approach with modelling/forecasting has any legs? How it can be justified from a statistical standpoint?
It's probably a quite subjective matter, here is my opinion:


I would say that this method makes a lot of sense if the ML model uses a number of features, something which would be hard to factor properly in a standard statistical test (especially if some kind of feature selection is involved).
Also in favor of the method, statistical significance tests are usually interpreted in a binary way (typically p-value higher or lower than 0.05) even though the underlying p-value is continuous: there's very little different between a p-value of 0.049 and 0.51 but the two will be interpreted in a completely opposite way.
However I agree with you that statistical significance tests are much more grounded in statistical theory, with a clear interpretation especially about levels of confidence.


Overall I can see an interest of using such a method from a practical viewpoint, but it is indeed limited in terms of theoretical interpretation and reliability.

Disclaimer: I'm not a statistician, I might be biased towards practical ML methods ;)