Predictive modeling when output affects future input Assume I have a model  which predicts the outcome of number of icecream sold in a store.
The model is trained on data for the last 5 years while keep the last year as a validation set and has produced very good results.
We now put the model into production such that the CFO can create an estimate for the upcomming year's budget. The CFO now look at the prediction for May, say 2000 ice creams, and thinks &quot;Ooh... I was hoping for some more sale in May. I'll go 4000&quot; thus he orders some more advertising, introduces new flavours etc. and reaches the 4000 sold ice cream at the end of May as he was hoping for.
The first of June we talk to the CFO to evalutate the model after the first 6 months, and we see that our prediction in May is off by 100%!
This spike can be explained with the increased advertising etc., and all the other days the model have done really well, but if the CFO starts tweaking the advertising, flavours etc. each day to hit the budget, how will we ever be able to test, if our model is indeed good in production/real-world? And how will we be able to re-train the model, since the first 5 years sale is without any &quot;human influence&quot; whereas after a year, the sale has been influenced by advertising etc., thus the spike in May is not &quot;natural&quot; but is due to some exogenous variable we are not able to incorporate (e.g we don't know the CFO's budget).
I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (open access, para 3.7):

3.7. Forecasts can result in adjustments that make forecasts less accurate
It is obvious that if forecasts lead to adjustments, and responses
that affect the studied phenomenon, then one can no longer judge these
forecasts on their subsequent accuracy.

So, other than  communicating this clearly beforehand and reaching an agreement on how the predictions will be assessed in the presence of such adjustments, there is not much else you can do from a modelling or methodology perspective. The advice suggested further in the same paragraph quoted above:

In that sense a forecast can be a warning of the style “if you do not act, these are the costs”.

can form the basis of such a communication and agreement.