In this paper, we study the accuracy of values aggregated over classes predicted by a classification algorithm. The problem is that the resulting aggregates (e.g., sums of a variable) are known to be biased. The bias can be large even for highly accurate classification algorithms, in particular when dealing with class-imbalanced data. To correct this bias, the algorithm's classification error rates have to be estimated. In this estimation, two issues arise when applying existing bias correction methods. First, inaccuracies in estimating classification error rates have to be taken into account. Second, impermissible estimates, such as a negative estimate for a positive value, have to be dismissed. We show that both issues are relevant in applications where the true labels are known only for a small set of data points. We propose a novel bias correction method using Bayesian inference. The novelty of our method is that it imposes constraints on the model parameters. We show that our method solves the problem of biased classification-based aggregates as well as the two issues above, in the general setting of multi-class classification. In the empirical evaluation, using a binary classifier on a real-world dataset of company tax returns, we show that our method outperforms existing methods in terms of mean squared error.
@cite_16: Agricultural management increasingly uses crop maps based on classification of remotely sensed data. However, classification errors can translate to errors in model outputs, for instance agricultural production monitoring (yield, water demand) or crop acreage calculation. Hence, knowledge on the spatial variability of the classier performance is important information for the user. But this is not provided by traditional assessments of accuracy, which are based on the confusion matrix. In this study, classification uncertainty was analyzed, based on the support vector machines (SVM) algorithm. SVM was applied to multi-spectral time series data of RapidEye from different agricultural landscapes and years. Entropy was calculated as a measure of classification uncertainty, based on the per-object class membership estimations from the SVM algorithm. Permuting all possible combinations of available images allowed investigating the impact of the image acquisition frequency and timing, respectively, on the classification uncertainty. Results show that multi-temporal datasets decrease classification uncertainty for different crops compared to single data sets, but there was no “one-image-combination-fits-all” solution. The number and acquisition timing of the images, for which a decrease in uncertainty could be realized, proved to be specific to a given landscape, and for each crop they differed across different landscapes. For some crops, an increase of uncertainty was observed when increasing the quantity of images, even if classification accuracy was improved. Random forest regression was employed to investigate the impact of different explanatory variables on the observed spatial pattern of classification uncertainty. It was strongly influenced by factors related with the agricultural management and training sample density. Lower uncertainties were revealed for fields close to rivers or irrigation canals. This study demonstrates that classification uncertainty estimates by the SVM algorithm provide a valuable addition to traditional accuracy assessments. This allows analyzing spatial variations of the classifier performance in maps and also differences in classification uncertainty within the growing season and between crop types, respectively.
@cite_4: Land cover data are widely used in ecology as land cover change is a major component of changes affecting ecological systems. Landscape change estimates are characterized by classification errors. Researchers have used error matrices to adjust estimates of areal extent, but estimation of land cover change is more difficult and more challenging, with error in classification being confused with change. We modeled land cover dynamics for a discrete set of habitat states. The approach accounts for state uncertainty to produce unbiased estimates of habitat transition probabilities using ground information to inform error rates. We consider the case when true and observed habitat states are available for the same geographic unit (pixel) and when true and observed states are obtained at one level of resolution, but transition probabilities estimated at a different level of resolution (aggregations of pixels). Simulation results showed a strong bias when estimating transition probabilities if misclassification was not accounted for. Scaling-up does not necessarily decrease the bias and can even increase it. Analyses of land cover data in the Southeast region of the USA showed that land change patterns appeared distorted if misclassification was not accounted for: rate of habitat turnover was artificially increased and habitat composition appeared more homogeneous. Not properly accounting for land cover misclassification can produce misleading inferences about habitat state and dynamics and also misleading predictions about species distributions based on habitat. Our models that explicitly account for state uncertainty should be useful in obtaining more accurate inferences about change from data that include errors.
@cite_7: Abstract The area of land use or land cover change obtained directly from a map may differ greatly from the true area of change because of map classification error. An error-adjusted estimator of area can be easily produced once an accuracy assessment has been performed and an error matrix constructed. The estimator presented is a stratified estimator which is applicable to data acquired using popular sampling designs such as stratified random, simple random and systematic (the stratified estimator is often labeled a poststratified estimator for the latter two designs). A confidence interval for the area of land change should also be provided to quantify the uncertainty of the change area estimate. The uncertainty of the change area estimate, as expressed via the confidence interval, can then subsequently be incorporated into an uncertainty analysis for applications using land change area as an input (e.g., a carbon flux model). Accuracy assessments published for land change studies should report the information required to produce the stratified estimator of change area and to construct confidence intervals. However, an evaluation of land change articles published between 2005 and 2010 in two remote sensing journals revealed that accuracy assessments often fail to include this key information. We recommend that land change maps should be accompanied by an accuracy assessment that includes a clear description of the sampling design (including sample size and, if relevant, details of stratification), an error matrix, the area or proportion of area of each category according to the map, and descriptive accuracy measures such as user's, producer's and overall accuracy. Furthermore, mapped areas should be adjusted to eliminate bias attributable to map classification error and these error-adjusted area estimates should be accompanied by confidence intervals to quantify the sampling variability of the estimated area. Using data from the published literature, we illustrate how to produce error-adjusted point estimates and confidence intervals of land change areas. A simple analysis of uncertainty based on the confidence bounds for land change area is applied to a carbon flux model to illustrate numerically that variability in the land change area estimate can have a dramatic effect on model outputs.
@cite_2: This paper describes a general framework alternative to the traditional surveys that are commonly performed to estimate, for statistical purposes, the areal extent of predefined land cover classes across Europe. The framework has been funded by Eurostat and relies on annual land cover mapping and updating from remotely sensed and national GIS-based data followed by area estimation. Map production follows a series of steps, namely data collection, change detection, supervised image classification, rule-based image classification, and map updating generalization. Land cover area estimation is based on mapping but compensated for mapping error as estimated through thematic accuracy assessment. This general structure was applied to continental Portugal, successively updating a map of 2010 for the following years until 2015. The estimated land cover change was smaller than expected but the proposed framework was proved as a potential for statistics production at the national and European levels. Contextual and structural methodological challenges and bottlenecks are discussed, especially regarding mapping, accuracy assessment, and area estimation.
The second field is , which analyzes land use based on large volumes of remote sensing data, for example using SVM @cite_16 . As can be expected, multi-class classification is not uncommon in this subject area (see @cite_2 for a case study with @math classes). Since accurately estimating the total area of types of vegetation is highly relevant in monitoring ecological systems @cite_4 , there have been many efforts in the field to make better use of accuracy data @cite_7 . To the best of our knowledge, our bias correction method is a novel contribution to these efforts.