Balanced Activation for Long-tailed Visual Recognition

Deep classifiers have achieved great success in visual recognition. However, real-world data is long-tailed by nature, leading to the mismatch between training and testing distributions. In this report, we introduce Balanced Activation (Balanced Softmax and Balanced Sigmoid), an elegant unbiased, and simple extension of Sigmoid and Softmax activation function, to accommodate the label distribution shift between training and testing in object detection. We derive the generalization bound for multiclass Softmax regression and show our loss minimizes the bound. In our experiments, we demonstrate that Balanced Activation generally provides ~3% gain in terms of mAP on LVIS-1.0 and outperforms the current state-of-the-art methods without introducing any extra parameters.


Introduction
Most real-world data comes with a long-tailed nature: a few high-frequency classes (or head classes) contribute to most of the observations, while a large number of low-frequency classes (or tail classes) are under-represented in data. In practice, the number of samples per class generally decreases from head to tail classes exponentially. Under the power law, the tails can be undesirably heavy. A model that minimizes empirical risk on long-tailed training datasets often underperforms on a class-balanced test dataset. As datasets are scaling up nowadays, long-tailed nature poses critical difficulties in many vision tasks.
An intuitive solution to long-tailed task is to re-balance the data distribution. Most state-of-the-art (SOTA) methods use the class-balanced sampling or loss re-weighting to "simulate" a balanced training set [1,10]. However, they may under-represent the head class or have gradient issues during optimization.
In this report, we first show that the Softmax function is intrinsically biased in the long-tailed scenario. We derive Balanced Activation (Balanced Softmax and Balanced Sigmoid) from the probabilistic perspective that explicitly models the test-time label distribution shift. Theoretically, we found that optimizing for the Balanced Softmax cross-entropy loss is equivalent to minimizing the generalization error bound. We take the off-the-shelf Mask-RCNN as our baseline model and obtain ∼3% mAP improvements on LVIS-1.0 via Balanced Activation without adding any extra computational overhead. Label Distribution Shift We begin by revisiting the multi-class Softmax regression, where we are generally interested in estimating the conditional probability p(y|x), which can be modeled as a multinomial distribution φ: From the Bayesian inference's perspective, φ j can also be interpreted as: where p(y = j) is in particular interest under the class-imbalanced setting. Assuming that all instances in the training dataset and the test dataset are generated from the same process p(x|y = j), there could still be a discrepancy between training and testing given different label distribution p(y = j) and evidence p(x).
With a slight abuse of the notation, we re-define φ to be the conditional distribution on the balanced test set and defineφ to be the conditional probability on the imbalanced training set. As a result, standard Softmax provides a biased estimation for φ.

Balanced Activation
To eliminate the discrepancy between the posterior distributions of training and testing, we introduce the Balanced Activation (Balanced Softmax and Balanced Sigmoid). We reach the same conclusion from both the label distribution shift (Section 2) perspective and the generalization error bound perspective (Section 3.2) that gives rise to Balanced Softmax. The same strategy is used to deduct Balanced Sigmoid. We refer readers to Section 3.4 for a summarized version of our findings.

From Label Distribution Shift Perspective
We use the same model outputs η to parameterize two conditional probabilities: φ for testing andφ for training.
Theorem 1. Assume φ to be the desired conditional probability of the balanced data set, with the form φ j = p(y = j|x) = p(x|y=j) k , andφ to be the desired conditional probability of the imbalanced training set, with the formφ j =p(y = j|x) = p(x|y=j) ni . If φ is expressed by the standard Softmax function of model output η, thenφ can be expressed aŝ We use the exponential family parameterization to prove Theorem 1.
Proof. Observe that the conditional probability of a categorical distribution can be parameterized as an exponential family. It gives us a standard Softmax function as an inverse parameter mapping and also a canonical link function: We begin by adding a term − log(φ j /φ j ) to both sides of Eqn. 5, Subsequently, Substitute Eqn. 9 back to Eqn. 7, we havê Recall that Finally, bring Eqn. 12 back to Eqn. 10 Theorem 1 essentially shows that applying the following Balanced Softmax function can naturally accommodate the label distribution shifts between the training and test sets. We define the Balanced Softmax function aŝ

From Generalization Error Bound Perspective
Generalization error bound gives the upper bound of a model's test error, given its training error. With drastically fewer training samples, the tail classes have much higher generalization error bounds than the head classes, making high classification accuracy on tail classes unlikely. In this section, we show that optimizing Eqn. 14 is equivalent to minimizing the generalization upper bound.
Margin theory provides a bound based on the margins [5]. Margin bounds usually negatively correlate to the magnitude of the margin, i.e., a larger margin leads to lower generalization error. Consequently, given a constraint on the sum of margins of all classes, there would be a trade-off between minority classes and majority classes [2].
Locating such an optimal margin for multi-class classification is non-trivial. The bound investigated in [2] was established for binary classification using hinge loss. Here, we try to develop the margin bound for the multi-class Softmax regression. Given the previously defined φ andφ, we derivel(θ) by minimizing the margin bound. Margin bound commonly bounds the 0-1 error: However, directly using the 0-1 error as the loss function is not ideal for optimization. Instead, negative log likelihood (NLL) is generally considered more suitable. With continuous relaxation of Eqn. 15, we have where t ≥ 0 is any threshold, and l y (θ) is a standard negative log-likelihood with Softmax. This new error is still a counter, but describes how likely the test loss will be larger than a given threshold. Naturally, we define our margin for class j to be where S j is the set of all class j samples. If we force a large margin γ j during training, i.e., force the training loss to be much lower than t, then err(t) will be reduced.
Theorem 2. Let t ≥ 0 be any threshold, for all γ j > 0, with probability at least 1 − δ, we have where err bal (t) is the error on the balanced test set, is used to hide constant terms and C is some measure on complexity. With a constraint on k j=1 γ j = β, Cauchy-Schwarz inequality gives us the optimal γ * j .
Proof. Firstly, we define f as, where l(θ) and t is previously defined in submission. However, f does not have a specific semantic meaning as it is defined only to keep consistent with notations in [5].
Let err j (t) be the 0-1 loss on example from class j and err γ,j (t) be the 0-1 margin loss on example from class j Letê rr γ,j (t) denote the empirical variant of err γ,j (t).
For any δ > 0 and with probability at least 1 − δ, for all γ j > 0, and f ∈ F, Theorem 2 in [5] directly gives us where sup (x,y))∈S |l(θ) − t| ≤ B andR j (F) denotes the empirical Rademacher complexity of function family F. By applying [2]'s analysis on the empirical Rademacher complexity and union bound over all classes, we have the generalization error bound for the loss on a balanced test set where is a low-order term of n j . To minimize the generalization error bound Eqn. 23, we essentially need to minimize By constraining the sum of γ as k j=1 γ j = β, we can directly apply Cauchy-Schwarz inequality to solve the optimal γ The optimal γ * suggests that we need larger γ for the classes with fewer samples. In other words, to achieve the optimal generalization ability, we need to focus on minimizing the training loss of the tail classes. Therefore, for each class j, the desired training lossl * j (θ) iŝ The equation above contradicts with the definition that sum ofφ * should be exactly equal to 1. To solve the contradiction, we introduce a term γ base > 0, such that To justify the new term γ base , we recall the definition of error If we tweak the threshold t with the term γ base err γ,j (t + γ base ) = Pr As γ * is not a function of t, the value of γ * will not be affected by the tweak. Thus, instead of looking forl * j (θ) = l j (θ) + γ * j that minimizes the generalization bound for err bal (t), we are in fact looking forl * j (θ) = (l j (θ) − γ base ) + γ * j that minimizes generalization bound for err bal (t + γ base ) Compare the form we obtained from generalization bound Eqn. 28 to Eqn. 14, we have an additional constant 1/4 before log n j . We empirically find that setting 1/4 to 1 leads to the optimal results, which may suggest that Eqn. 23 is not necessarily tight. To this point, the label distribution shift and generalization bound of multi-class Softmax regression lead us to the same loss form: Eqn. 14.

Derivation of Balanced Sigmoid
For instance segmentation, a detector is normally required to predict a background class explicitly. However, the number of background class is uncountable and we need the number of training samples of each class in the training set for Balanced Softmax. Thus, we introduce Balanced Sigmoid to circumvent this problem. By virtue of Bayesian law, a similar strategy can be applied to the multiple binary logistic regressions. Definition. Multiple Binary Logisitic Regression uses k binary logistic regression to do multi-class classification. Same as Softmax regression, the predicted label is the class with the maximum model output.
The only difference is that φ j is expressed by a logistic function of η j φ j = e ηj 1 + e ηj (35) and the loss function sums up binary classification loss on all classes Setup. By the virtue of Bayes Rule, φ j and 1 − φ j can be decomposed as and forφ and 1 −φ, Derivation. Again, we introduce the exponential family parameterization and have the following link function for φ j Bring the decomposition Eqn. 38 and Eqn.39 into the link function above Simplify the above equation Substitute the n j in to the equation above Then

TL;DR
The label distribution shift and generalization bound of multi-class Softmax regression lead us to the same Balanced Softmax loss form, for certain class y, we have:l We also provide the Balanced Sigmoid variant for broader usage:

Experiments
We first validate the proposed method on the long-tailed version of CIFAR-10 and CIFAR-100. Compared to LVIS, these two datasets are relatively small. Hence, they are suitable for fast validation of our ideas. After that, we show that our method leads to a ∼3% improvement over the Repeat Factor Sampling baseline and outperforms current SOTAs by a clear margin on LVIS.  Table 1. Top 1 accuracy for CIFAR-10/100-LT. Softmax denotes the standard crossentropy loss with Softmax, CBW denotes class-balanced weighting, and CBS denotes class-balanced sampling. Balanced Activation generally outperforms SOTA methods, especially when the imbalance factor is high. Note that for other methods, we reproduce a higher accuracy than the reported value in the original papers.

CIFAR-10/100-LT
We show results under different Imbalance Factors in Table 1. The Imbalance Factor is defined as the number of training instances in the largest class divided by that of the smallest. We also visualize the marginal likelihood p(y) on CIFAR-100-LT to show that Balanced Softmax is more stable under a high imbalance factor. We follow [3] and use ResNet-32 as backbone in all experiments.   test-time augmentation is used. We train all the repeat factor sampling models for 25 epochs and apply decoupled training for 10 epochs using other methods (cRT, LWS, Equalization Loss and Balanced Sigmoid). Models are all trained on 8 GTX 1080Ti GPUs. We compare our method with four existing methods and show improvements: Repeat Factor Sampling [4]: A simple and effective sampling strategy that increases the rate at which tail categories are observed by oversampling the images that contain them. Different from class-balance sampling, it does not force the sampling probability to be equal for all the classes. Classifier Re-training (cRT) [6]: A straightforward approach, which re-train the classifier with class-balanced sampling. The detector is first trained under the instance balance scheme at the first stage. During the second stage, the feature extractor (backbone) is fixed and only the classifier (regression layer) is optimized during training. Learnable weight scaling (LWS) [6]: An efficient approach to re-balance the decision boundaries of classifiers. The training setup is similar to cRT. The only difference is that there is one more regression layer inserted before the classifier to rebalance the decision boundary. Equalization Loss [9]: The solution of the winner of LVIS Challenge 2019. It pointed out that randomly dropping some scores of tail classes in the Softmax function can effectively help, by balancing the positive gradients and negative gradients flowing through the scoring output.

Conclusion
We have introduced Balanced Activation (Balanced Sigmoid and Balanced Softmax) for long-tail visual recognition tasks. Balanced Activation tackles the distribution shift between training and testing, combining with generalization error bound theory: it optimizes a Balanced Activation which theoretically minimizes the generalization error bound. We show the method outperforms the current SOTA method without introducing extra parameters. Although Balanced Activation provides a more appropriate objective under the long-tailed scenarios, the optimization process could still be challenging due to the overwhelming gradients from the head classes. A future research direction could be improving the optimization process in complementary to Balanced Activation, for example, a learnable importance sampling method in [8].