recommender system: how to compare different scores when calculated individually? I am building a small recommender system which aims at recommending ~10 products to customers. Instead of using a multi-label classification model, I have opted to build a separate scoring model for each product allowing me to take the other targeted products as features (e.g, if a customer has product X, that information could help predict detention of product Y ; conversely, knowing that a customer has product Y could help predicting detention of product X).

So I end up with ~10 models (say model X, model Y, etc.) with almost the same dataset (only the target changes, and the feature corresponding to that target is removed). However, as each model is different, I am not sure how to compare the scores I obtain for all my products and how to make the best recommendation.

For example, suppose that for a given customer:

score for product X is 0.7 and score for product Y is 0.8 but the precision of model Y is not as good as model X. Should I really recommend product Y?

I was suggested to standardize the scores for each product across customers to end up on the same scale. However, if a product ends up with very higher scores than others, this step would in effect penalize this product.

Any ideas?

PS : the detention of each product is pretty low (between 0.5 % to 2 %). I thought a first step would be to re-balance the dataset for each model (say with under of over sampling) in order to have the same class imbalance for each model as I observed that re-balancing biases the scores upwards (unfortunately without really improving the performance of my models).
You can continue to use machine learning for this model-combination step too. You're describing a form of ensembling. How about:


set aside some purchases as your validation set V, and keep the remaining as training set TRAIN
for TRAIN, use k-fold cross-validation to create your separate models
use TRAIN again with k-fold cross-validation to ensemble the models together
report performance of the ensembled models on V


Ensemble techniques will take into account the predictive quality of each individual model.