In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-imbalanced data. To correct this bias, the algorithm's classification error rates have to be estimated. In this estimation, two issues arise when applying existing bias correction methods. First, inaccuracies in estimating classification error rates have to be taken into account. Second, impermissible estimates, such as a negative estimate for a positive value, have to be dismissed. We show that both issues are relevant in applications where the true labels are known only for a small set of data points. We propose a novel bias correction method using Bayesian inference. The novelty of our method is that it imposes constraints on the model parameters. We show that our method solves the problem of biased classification-based aggregates as well as the two issues above, in the general setting of multi-class classification. In the empirical evaluation, using a binary classifier on a real-world dataset of company tax returns, we show that our method outperforms existing methods in terms of mean squared error.
@cite_2: This paper describes a general framework alternative to the traditional surveys that are commonly performed to estimate, for statistical purposes, the areal extent of predefined land cover classes across Europe. The framework has been funded by Eurostat and relies on annual land cover mapping and updating from remotely sensed and national GIS-based data followed by area estimation. Map production follows a series of steps, namely data collection, change detection, supervised image classification, rule-based image classification, and map updating generalization. Land cover area estimation is based on mapping but compensated for mapping error as estimated through thematic accuracy assessment. This general structure was applied to continental Portugal, successively updating a map of 2010 for the following years until 2015. The estimated land cover change was smaller than expected but the proposed framework was proved as a potential for statistics production at the national and European levels. Contextual and structural methodological challenges and bottlenecks are discussed, especially regarding mapping, accuracy assessment, and area estimation.
Assume that the classes @math are not known, but instead predicted to be @math . The predicted class matrix based on the predictions @math is denoted by @math . Similarly, @math denotes the predicted counts vector. Assume that @math is known for all @math . The goal is to estimate the sum of @math over each of the classes @math . In other words, the main problem statement is how to find, using the predicted @math and the known @math , an accurate estimator of the In the sentiment analysis application from before, the set @math is the set of messages and @math is binary (a positive or negative message). Each @math equals 1 and hence @math is a 2-vector with @math the number of positive messages and @math the number of negative messages. In the land cover mapping application obtained from @cite_2 , each @math is a pixel and @math may attain @math different values. The value @math is the real land area corresponding to pixel @math . Hence, the 15-vector @math contains the total land area of each of the 15 land cover classes.