Including biased data in training model
That's a wonderful question, and beautifully posed.  I love how you capture the essence of the issue in such a clean way.  Unfortunately, the answer is that your data set does not have enough information to help you decide which cats you should feed.  Without a properly controlled experiment, you don't have a way to infer causality; you can't rule out the possibility of confounding factors.

Suppose that 70% of the cats he feeds make it to the top, and 20% of the cats he doesn't feed make it to the top.  Let me make up four alternative explanations for why that might happen:


Hypothesis #1: There's only one kind of cat in the world.  The characteristics of the cat are irrelevant.  If you feed them, they will have a 70% chance of making it to the top; if you don't, they'll have only a 20% chance.  This is true for all cats, regardless of their attributes.
Hypothesis #2: There are two kinds of cats in the world: the strong, and the weak.  The strong have a 70% chance of making it to the top of the mountain, regardless of whether you feed them or not.  The weak have a 20% chance of making it to the top, regardless of whether you feed them.  You can tell the two kinds of cats apart by their characteristics, and your friend happens to like the strong ones and dislike the weak ones.
Hypothesis #3: There are two kinds of cats in the world: the athletes, and the champion nappers.  The former have a 70% chance of making it to the top of the mountain if you feed them, or a 50% chance if you don't.  The latter have a 40% chance of making it to the top if you feed them, or a 20% chance if you don't.  You can tell the two apart by their characteristics, and your friend happens to like the athletes and dislike the nappers.
Hypothesis #4: There are two kinds of cats in the world: the hungry, and the overweight.  The former have a 70% chance of making it to the top of the mountain if you feed them (they're pretty fit, but they already have all the food they need, so why bother exploring?), or a 100% chance if you don't (hunger drives them to greater heights).  The latter have a 21% chance of making it to the top, if you feed them, or a 20% chance, if you don't (either way, they probably won't make it, because they are so overweight and out of shape).  You can tell the two apart by their characteristics, and your friend happens to like the hungry ones and dislike the overweight ones.


All four hypotheses are equally consistent with the data.  The data set is useless at helping you distinguish between them.  Yet each hypothesis leads to a different prediction about which cats to feed: the first two say it doesn't matter, the third says you should feed the ones your friend likes, and the fourth says you should feed the ones your friend dislikes.  No amount of statistics or machine learning is going to enable you to make a decision based solely on the data.

To distinguish between these possibilities, you either need some kind of prior that will give you a basis for choosing among the models that are consistent with the data (perhaps based on your domain knowledge), or you need to conduct a controlled experiment (where the choice of whether to feed a cat is randomized, rather than being selected based on whether your friend likes the cat).



What if we have a prior that says that feeding a cat can only help its chances of making it to the top, and not hurt its chances?  Well, that's an example of the first of two options I suggest in the last paragraph.  However, it's still not informative enough to let us decide which cats to feed.

The problem is that we need to tell whether feeding the cats your friend likes improves their odds more than it improves the odds of feeding the cats your friend doesn't like.  We just don't have any information on that.

The chances could be 70% vs 69% for cats he likes and 90% vs 20% for cats he doesn't (i.e., cats he like have a 70% chance if fed or a 69% chance if not fed, etc.); or it could be 70% vs 0% for cats he likes and 21% vs 20% for cats he doesn't.  In the former case, you should feed the cats he doesn't like, as they see a 70 percentage point improvement from feeding, while the ones he likes see only a 1 p.p. improvement.  In the latter case, you should feed the cats he does like, because they see a 70 p.p. improvement, while the ones he dislikes see only a 1 p.p. improvement.  You just can't distinguish between these two possibilities from the dataset you have.  So your prior would need to be considerably more detailed than that, before you can make any headway.

If you'd like to learn more about the subject and under what circumstances we can draw conclusions about causality despite the lack of a controlled experiment, you might start by reading Judea Pearl's work.