A Survey of Deep Long-Tail Classification Advancements
Charika de Alvis Suranga Seneviratne










  The University of Sydney
  Sydney
  Australia
charika.weerasiriwardhane@sydney.edu.au



  The University of Sydney
  Sydney
  Australia
suranga.seneviratne@sydney.edu.au


Many data distributions in the real world are hardly uniform. Instead, skewed and long-tailed distributions of various kinds
are commonly observed. This poses an interesting problem for machine learning, where most
algorithms assume or work well with uniformly distributed data. The problem is further exacerbated by current state-of-the-art deep
learning models requiring large volumes of training data. As such, learning from imbalanced data remains a
challenging research problem and a problem that must be solved as we move towards more real-world applications of deep learning. In the context of class imbalance, state-of-the-art (SOTA) accuracies on standard benchmark datasets for classification typically fall
less than 75%, even for less challenging datasets such as CIFAR100. Nonetheless, there has been progress in this niche
area of deep learning. To this end, in this survey, we provide a taxonomy of various methods proposed for addressing the problem of
long-tail classification, focusing on works that happened in the last few years under a single mathematical framework.
We also discuss standard performance metrics, convergence studies, feature distribution and classifier analysis. We also provide a quantitative comparison of the performance of different SOTA methods and conclude the survey by discussing the remaining challenges and future research direction.
 



<ccs2012>
   <concept>
       <concept_id>10010147.10010257.10010258.10010259.10010263</concept_id>
       <concept_desc>Computing methodologies Supervised learning by classification</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Computing methodologies Supervised learning by classification



22 April 2007

A Survey of Deep Long-Tail Classification Advancements
    Suranga Seneviratne
    
======================================================





§ INTRODUCTION

sections/Introduction



§ BACKGROUND

sections/s0


§ TAXONOMY

sections/s00


§ LOSS REWEIGHTING
 

 sections/s2


§ MARGIN BASED LOGIT ADJUSTMENT



sections/s1





§ OPTIMIZED REPRESENTATION LEARNING


sections/s3



§ BALANCED CLASSIFIER LEARNING

sections/s4



§ RESEARCH GAPS

sections/s5
    



§ CONCLUSION


 In this paper, we surveyed recent advances in deep long-tail classification focusing on algorithmic-level level solutions.  We proposed a taxonomy to categorise the works into four broad categories: 1) Loss reweighting, 2) Margin-based Loss Modification, 3) Optimized Representation Learning, and 4) Balanced Classifier Learning, followed by subsequent sub-categories. We presented these works under a common mathematical notation, making it easier to follow across works and compare their inner working conveniently. In addition, we discuss standard performance metrics such as balanced accuracy and proxy measures such as feature deviation analysis. Finally, we discussed existing research gaps and future research directions in long-tail classification. A notable observation in the literature is SCL loss implemented with data augmentation combined with logit-adjusted CE loss yields significant performance leaps compared to other SOTA methods. Furthermore, we discussed that some long-tail techniques could be redundant to use jointly i.e., using class-balanced sampling for training data along with loss reweighting based on the inverse class frequency and loss-based logit adjustment during training time applied with post hoc logit adjustment.
 Despite advances in machine learning and deep learning, long-tail classification remains a challenge that hasn't yet been resolved completely with performance. This phenomenon is visible regarding SOTA datasets, as an average-level class imbalance would significantly affect the classification accuracy. CIFAR10 shows a balanced accuracy of 99% in the class balanced case; when IF =100, the best accuracy stays in the range of 85%-86% with the best combination of multiple long-tail handling techniques, including both data-level and algorithmic level. Similarly, with CIFAR100, accuracy is 96%-97% for the class balanced case, and when IF=100, the best accuracy is in the range of 53%-54%. ImageNet has an accuracy of 92%-93%, and for ImageNet_Lt with IF of 256, this value drops to 56%-57%. iNaturalist is inherently long-tailed. In this dataset, from lowest IF to highest IF (500), the accuracy drops from 93%-94% to 71%-72%. These metrics show that although several advanced approaches are practised to handle longtail classification, there is still room for improvement to bring classification accuracy closer to a balanced class case.





§ ACKNOWLEDGMENTS

This research was conducted by the University of Sydney for National Intelligence Postdoctoral Grant (project number NIPG-2022-006) and funded by the Australian Government.









references/ACM-Reference-Format

