How to justify a predictor in influencing the outcome?
Cross Validation

You had a post where we discussed causality, but with ML models assumption is that data represents your problem entirely and has all the information in it. In other words every pattern that you pickup in your train data you can expect it to behave pretty similar in production, hence with this assumption what you want is to "evaluate the entire train" (thats what you can do with CV.) and if it score good on average over all folds you want to add this change (be it a new feature)