Topic models for Relevance Prediction
Certainly. A part of topic modeling output is typically some vector for each document describing it as a distribution over the topics in your model. You can use these as continuous-valued predictors for a classifier trained with the labels you have. I've done this before, but, depending on the domain characteristics of your problem, it may not be the best approach, but it is certainly a testable one!