Negative Latent Factors in Factorized Machines
Found after some digging many concepts related to Non-Negative Matrix Factorization, which if properly setup constrain a FM algorith to come up with non negative factors (and therefore predictions.). Many useful material here:

https://en.wikipedia.org/wiki/Non-negative_matrix_factorization
https://towardsdatascience.com/parts-based-learning-by-non-negative-matrix-factorisation-1279ffd29618