Crowdsourcing has become a primary means for label collection in many real-world machine learning applications. A classical method for inferring the true labels from the noisy labels provided by crowdsourcing workers is Dawid-Skene estimator. In this paper, we prove convergence rates of global optimizers of Dawid-Skene estimator. The revealed exponent in the rate of convergence is shown to be optimal via a lower bound argument. A projected EM algorithm is analyzed and is shown to achieve nearly the same exponent as that of the global optimizers. Our work resolves the long standing issue of whether Dawid-Skene estimator has sound theoretical guarantees besides its good performance observed in practice. In addition, a comparative study with majority voting illustrates both advantages and pitfalls of Dawid-Skene estimator.
@cite_5: Abstract : Lately the statistical methods of psychometrics have been severely criticized in psychological quarters. Thus Skinner [1] maintains that if order is to be found in human and animal behavior, then it should be extracted from investigations into individuals, and that psychometric methods are inadequate for such purposes since they deal with groups of individuals. And as regards abnormal psychology Zubin [2] states: Recourse must be had to individual statistics, treating each patient as a separate universe. Unfortunately, present day statistical methods are entirely group-centered, so that there is a real need for developing individual-centered statistics. In a recently published book [3] I have developed three models for reactions to certain attainment tests and intelligence tests.
@cite_12: An important way to make large training sets is to gather noisy labels from crowds of nonexperts. We propose a minimax entropy principle to improve the quality of these labels. Our method assumes that labels are generated by a probability distribution over workers, items, and labels. By maximizing the entropy of this distribution, the method naturally infers item confusability and worker expertise. We infer the ground truth by minimizing the entropy of this distribution, which we show minimizes the Kullback-Leibler (KL) divergence between the probability distribution and the unknown truth. We show that a simple coordinate descent scheme can optimize minimax entropy. Empirically, our results are substantially better than previously published methods for the same problem.
In addition to showing that Dawid-Skene estimator achieves the minimax rate, we are also interested in studying whether its generalization also shares optimality. For example, what if we consider both worker confusion matrices and item confusion vectors @cite_5 @cite_12 . The technique used in this paper cannot be directly extended to that setting. We will consider this harder problem in our future work.