In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-imbalanced data. To correct this bias, the algorithm's classification error rates have to be estimated. In this estimation, two issues arise when applying existing bias correction methods. First, inaccuracies in estimating classification error rates have to be taken into account. Second, impermissible estimates, such as a negative estimate for a positive value, have to be dismissed. We show that both issues are relevant in applications where the true labels are known only for a small set of data points. We propose a novel bias correction method using Bayesian inference. The novelty of our method is that it imposes constraints on the model parameters. We show that our method solves the problem of biased classification-based aggregates as well as the two issues above, in the general setting of multi-class classification. In the empirical evaluation, using a binary classifier on a real-world dataset of company tax returns, we show that our method outperforms existing methods in terms of mean squared error.
@cite_5: 
@cite_13: This text provides the first-ever compilation of bias analysis methods for use with epidemiologic data. It guides the reader through the planning stages of bias analysis, including the design of validation studies and the collection of validity data from other sources. Three chapters present methods for corrections to address selection bias, uncontrolled confounding, and classification errors. Subsequent chapters extend these methods to multidimensional bias analysis, probabilistic bias analysis, and multiple bias analysis. The text concludes with a chapter on presentation and interpretation of bias analysis results. Although techniques for bias analysis have been available for decades, these methods are considered difficult to implement. This text not only gathers the methods into one cohesive and organized presentation, it also explains the methods in a consistent fashion and provides customizable spreadsheets to implement the solutions. By downloading the spreadsheets (available at links provided in the text), readers can follow the examples in the text and then modify the spreadsheet to complete their own bias analyses. Readers without experience using quantitative bias analysis will be able to design, implement, and understand bias analyses that address the major threats to the validity of epidemiologic research. More experienced analysts will value the compilation of bias analysis methods and links to software tools that facilitate their projects.
@cite_3: INTRODUCTION Examples of Mismeasurement The Mismeasurement Phenomenon What is Ahead? THE IMPACT OF MISMEASURED CONTINUOUS VARIABLES The Archetypical Scenario More General Impact Multiplicative Measurement Error Multiple Mismeasured Predictors What about Variability and Small Samples? Logistic Regression Beyond Nondifferential and Unbiased Measurement Error Summary Mathematical Details THE IMPACT OF MISMEASURED CATEGORICAL VARIABLES The Linear Model Case More General Impact Inferences on Odds-Ratios Logistic Regression Differential Misclassification Polychotomous Variables Summary Mathematical Details ADJUSTMENT FOR MISMEASURED CONTINUOUS VARIABLES Posterior Distributions A Simple Scenario Nonlinear Mixed Effects Model: Viral Dynamics Logistic Regression I: Smoking and Bladder Cancer Logistic Regression II: Framingham Heart Study Issues in Specifying the Exposure Model More Flexible Exposure Models Retrospective Analysis Comparison with Non-Bayesian Approaches Summary Mathematical Details ADJUSTMENT FOR MISMEASURED CATEGORICAL VARIABLES A Simple Scenario Partial Knowledge of Misclassification Probabilities Dual Exposure Assessment Models with Additional Explanatory Variables Summary Mathematical Details FURTHER TOPICS Dichotomization of Mismeasured Continuous Variables Mismeasurement Bias and Model Misspecification Bias Identifiability in Mismeasurement Models Further Remarks APPENDIX: BAYES-MCMC INFERENCE Bayes Theorem Point and Interval Estimates Markov Chain Monte Carlo Prior Selection MCMC and Unobserved Structure REFERENCES
The first field is , which studies the distribution of health and disease. There, it is well-known how a low base rate can lead to high bias even if sensitivity ( @math ) and specificity ( @math ) are high @cite_13 (cf. The Base Rate Example). As mostly binary classifiers are considered (sick or not), bias correction is straightforward [pp. 87-89] lash_applying_2009 . In addition, a standard Bayesian method of bias correction, predominantly using a uniform or Jeffreys prior without parameter constraints, is applied in epidemiology @cite_3 @cite_5 . We will generalize these Bayesian methods to the entire family of conjugate prior distributions and to the setting of multi-class classification. Moreover, we will improve empirical performance by imposing well-chosen parameter constraints.