Offline evaluation of recommender systems Let's say I want to compare whether one recommender system (A) is better than the other (B). 

One approach is to let people rate recommendations returned by both systems.

However, there situations when I would like to evaluate my recommender system offline. One approach I considered is to collect user ratings for recommendations from system A and turn them into a test dataset. However, if the system B returns recommendations that do not appear in test dataset, that does not mean they are bad. It just means I have no rating about them.

Some alternatives:


try to increase test dataset volume as much as possible to increase overlap between recommenders.
limit the size of items space, let recommender select items from 500 instead of 5*10^6 items during tests.


What other approaches would you recommend suggest?
There is always a bias in offline testing in recommender systems. Real evaluation happens in A/B testing. But, that should not discourage one from doing offline testing. 

There is ongoing research on this topic using multi-armed bandits. I recommend reading offline testing procedures using reco-gym.

Following workshop talks about removing these biases and developing efficient offline estimators for recommender systems. They also conducted an online course on efficiently doing an offline evaluation of recommender systems.

You can also follow a video tutorial on this website on counterfactual evaluation