1. A method by a computing system including a processor and a memory for improving a ranking model, the method comprising: obtaining a training set, wherein the training set includes: training queries: a document set associated with each training query; and document labels, from which at least one perfect and imperfect document set permutations can be inferred for each training query; initializing active sets to include a perfect document set permutation and an imperfect document set permutation for each training query; optimizing a ranking model in accordance with an empirical loss function, wherein the empirical loss function utilizes the active sets; identifying, for each training query, the perfect document set permutation for which the ranking model scores lowest, and the imperfect document set permutation for which the ranking model scores highest; updating, by a processor, the active sets to include at least one identified document set permutation that was not initially present; and using the updated active sets to additionally optimize the ranking model.