Reduce serving time complexity for real-time recommender systems
You write ... the model needs to be evaluated against each product in real-time., which gets me thinking that you use a binary classification (sigmoid in the final layer) architecture with negative sampling for the user/item interactions when training your model.
Have you considered using multi-class classification instead? Thus, for the user only predict once for the entire product catalogue, and selecting the top-k candidates from the softmax-layer. This way, you only need to feed-forward once through your neural net during inference.