1. A computer implemented method for determining a content item relevance function, the method comprising: collecting user preference data at a search provider for storage in a user preference data store; collecting expert-judgment data at the search provider for storage in a expert sample data store; training a base model by a modeling module through the use of the expert-judgment data; tuning the base model by the modeling module through the use of the user preference data to learn a set of one or more tuned models, wherein tuning the base model comprises optimizing a loss function; applying a B measure to evaluate the balanced performance of the tuned models over the expert-judgment data and the user preference data; and selecting the tuned model with the best balanced-performance as the content item relevance function.