---
abstract: 'Supervised deep learning methods are enjoying enormous success in many practical applications of computer vision and have the potential to revolutionize robotics. However, the marked performance degradation to biases and imbalanced data questions the reliability of these methods. In this work we address these questions from the perspective of dataset imbalance resulting out of severe under-representation of annotated training data for certain classes and its effect on both deep classification and generation methods. We introduce a joint dataset repairment strategy by combining a neural network classifier with Generative Adversarial Networks (GAN) that makes up for the deficit of training examples from the under-representated class by producing additional training examples. We show that the combined training helps to improve the robustness of both the classifier and the GAN against severe class imbalance. We show the effectiveness of our proposed approach on three very different datasets with different degrees of imbalance in them. The code is available at .'
author:
- 'Aadarsh Sahoo[^1]'
- 'Ankit Singh^\*^'
- Rameswar Panda
- Rogerio Feris
- Abir Das
bibliography:
- 'egbib.bib'
title:
- Mitigating Dataset Imbalance via Joint Generation and Classification
- Mitigating Dataset Imbalance via Joint Generation and Classification
---

Introduction {#sec:intro}
============

Deep neural networks (DNN) and large-scale datasets have been instrumental behind the remarkable progress in computer vision research. The data-driven feature and classifier learning enabled Convolutional Neural Networks (CNNs) to achieve superior performance than traditional machine learning methods. However, the over-reliance on data has brought with it new problems. One of them is the problem of becoming too adapted to the dataset by essentially memorizing all its idiosyncrasies @Ponce2006Dataset [@Torralba2011Unbiased]. Due to the presence of massive number of learnable parameters, most DNNs require huge amount of annotated examples for each class. This is resource consuming, expensive and often impractical too in cases where recognition involves rare classes. Such a situation is not rare where large number of annotated examples for one class and only a few for the other are available resulting in an imbalanced dataset.

![We propose a joint generation and classification network to tackle severe dataset imbalance in classification scenario. The framework employs a Cycle-consistent GAN to generate new training examples from existing ones and feeds both to the classifier. The GAN and the classifier are trained jointly in an alternating fashion. Our model improves the classification performance especially when the dataset is highly imbalanced and the performance of the GAN is either at par with or experiences minor deterioration compared to the case when the classifier is not trained jointly.<span data-label="fig:schematic"></span>](figs/GAN-bias_Less_Wide_Zebra2Horse.pdf){width="0.99\linewidth"}

-0.1in

-0.2in

Class imbalance is a classic problem in machine learning and is known to have detrimental effect @Buda2018Systematic [@Haixiang2017Learning; @Japkowicz2002Class]. Classifiers trained traditionally minimizes the average error across all training examples. Such a strategy often results in fitting to the majority class (*i.e.*, the class with large number of annotated training examples) only. Simply by virtue of their numbers, fitting to the majority class reduces the overall training error than fitting to the minority class.

Traditionally, dataset imbalance mitigation methods have tried to put all classes in the level playing field by focusing on to take away the number advantage enjoyed by the majority class. This includes oversampling minority population @Chawla2002Smote or undersampling the majority population @Drummond2003Class [@Haixiang2017Learning], providing more importance to errors from the minority class while training @Zhou2006Training or adjusting the predicted class probabilities during inference according to the prior class probabilities @Richard1991Neural *etc*. However, these approaches are limited to the data at hand in the sense that these can not generate additional unseen data that can help increase the much needed diversity for unbiased training. In this paper, we advocate for a generative approach to address dataset imbalance. A Generative Adversarial Network (GAN) is used to transform majority class examples to minority class and vice-versa. Starting from an imbalanced dataset, this generates training examples that are not only much broader and diverse in nature but also balanced in terms of the number of examples from the two different classes.

Our proposed approach explores a Cycle-consistent GAN or simply cycle-GAN @Zhu2017Unpaired that is recently proposed to translate image from one domain (source) to another domain (target) (see Fig. \[fig:schematic\]). In addition to traditional generator and discriminator losses during training, a cycle-GAN employs a *cycle-consistency loss* that encourages the learned translations to be "cycle consistent", *i.e.*, if an image is translated from a source to a target domain and then it is translated back to the target domain again, original image and the doubly translated image should be same. We learn a joint architecture by extending cycle-GAN that feeds a classifier with images translated from a source class to a target class where the target class is also the minority class in the imbalanced dataset. Instead of using the cycle-GAN as only a component of the classification pipeline, we employ a joint training strategy that first trains the classifier using images generated by the cycle-GAN and then trains the cycle-GAN by back-propagating the classifier loss through it. The loss from the classifier acts as a multi-task supervision for the cycle-GAN in addition to the cycle-consistency and the traditional GAN loss. After few such iterations of alternate classifier and GAN training, the GAN learns to generate appropriate images for training a balanced classifier. Our key insight is that one can use a GAN to alleviate dataset imbalance effect on classifier and at the same time the gradients from the classifier loss can be propagated back to the GAN, to mitigate the same for the GAN.

We evaluate our proposed approach on three different datasets. The first one is the CelebA dataset @Liu2015Deep which is a largescale face dataset created from face images of celebrities, the second one is the Horse2Zebra dataset @Zhu2017Unpaired which is a collection of natural images of horses and zebras downloaded from the internet and the third one is CUB-200-2011 dataset @WahCUB2011Caltech consisting of fine-grained images of 200 bird species. Specifically, we demonstrate that in highly imbalanced scenarios, both classifier and the GAN achieves superior performances than traditional imbalance mitigation approaches.

Related Works {#sec:rel_works}
=============

Class imbalance in machine learning has been studied long @Japkowicz2002Class. However, in the midst of present data revolution, the problem has become ever so important. While images corresponding to some concepts or classes are available in plenty and easy to annotate, examples corresponding to many concepts are either rare or might require expert annotators. As a result, many real world datasets are highly imbalanced. MS-COCO @Lin2014Microsoft, SUN database @Xiao2010Sun, DeepFashion @Liu2016Deepfashion, Places @Zhou2014Learning, iNaturalist @Horn2018The . are all examples of datasets where the number of images in the most common class and the number of images in the least common class are hugely different. The community has commonly adopted two different kinds of approaches to address class imbalance: i) dataset level methods and ii) classifier or algorithmic level methods. A multidimensional taxonomy and categorization of the class imbalance problems can be obtained in the review papers @Buda2018Systematic [@He2008Learning].

The dataset level approaches consist of modification of imbalanced data in order to provide a balanced distribution with the aim that no change at the algorithm level is necessary and standard training algorithms would work. These are based on sampling either the majority or the minority classes differently. Basic random oversampling @Chawla2002Smote [@Estabrooks2004Multiple; @Weiss2001Effect] involves replicating randomly selected training examples from the minority class such that the total number of minority and majority class examples that goes to train the classifier is same. However the same mechanism can be used to achieve varying degrees of class distribution balance by varying the amount of replication. Oversampling methods are simple and effective, however there are evidences that these can lead to overfitting @Chawla2002Smote [@Wang2014Hybrid]. To this end, various adaptive and informative oversampling methods have been proposed. SMOTE @Chawla2002Smote creates synthetic examples of the minority class as a convex combination of existing examples and their nearest neighbors. Borderline-SMOTE @Han2005Borderline extends SMOTE by choosing more training examples near the class boundaries. Authors in @Batista2004astudy used a modified condensed nearest neighbor rule @Tomek1976Two to sample examples such that all minimally distanced nearest neighbor pairs are of same class and learned a classifier on these samples only. In CNNs, Shen *et al.*  @Shen2016Relay used a class-aware sampling strategy where the authors first sample a class and then sample an image from that class to form minibatches that are uniform with respect to the classes.

Another variant of sampling is undersampling where training examples are removed randomly from the majority classes until the desired balance in the curated dataset is achieved. This approach, though counter-intuitive works better than oversampling in some situations @Drummond2003Class. One obvious disadvantage of undersampling is that discarding examples may cause the classifier to miss out on important variabilities in the dataset. EasyEnsemble and BalanceCascade @Liu2008Exploratory are two variants of informed undersampling that addresses this issue. EasyEnsemble creates an ensemble of classifiers by independently sampling several subsets from the minority class while BalanceCascade develops the ensemble classifier by systematically selecting the majority class samples. The one-sided selection @Kubat1997Addressing approach discards redundant examples especially close to the boundary to get a more informed reduced sample set than random undersampling of majority class. Though sampling based methods change the data distribution to be more balanced, they hardly change the data itself. Our proposed method on the other hand is not only able to change the data distribution to make it more balanced, but also adds variability as additional data is generated in the process.

Among the approaches that modify the learning algorithm without changing the dataset distribution, Zhou *et al.* @Zhou2006Training weigh misclassification of the minority class examples more than the majority class examples during training. A naive approach @Lawrence1998Neural is to weigh the predicted probabilities of different classes during inference according to the prior probability of occurrences of these classes in the training set. Recently Cui *et al.*, @Cui2019Class proposed a class balanced loss depending on the "effective" number of training examples per class based on the fact that the additional benefit diminishes with increasing number of examples. This generalizes the class specific loss balancing based on frequency of the samples @Zhou2006Training to their effective frequencies.

GAN @Goodfellow2014_Generative and its variants use a minimax game to model high dimensional distributions of visual data enabling them to be used for data augmentation @Ratner2017Learning as well as for generating new images @Antoniou2017Data. However, training with GAN generated minority class samples can lead to boundary distortion @Santurkar2018Classification resulting in performance drop for the majority class. Mullick *et al.*, @Mullick2019Generative proposed to generate minority class samples by convexly combining existing samples in an adversarial setting by fooling both the discriminator as well as the classifier. Our proposed approach, on the other hand, generates additional minority samples from images belonging to the majority class. In contrast to prior works that focus on only perturbing the whole image to generate a slightly different version of it, we use a translational generative model (Cycle-GAN) to perturb the *content* (object of interest) without changing the *context* much. Our conjecture is: the classifier gets two different objects in the same context which would likely help the classifier to focus more on the object of interest rather than on the context for the disrciminative task. Training classifiers with such image pairs where the two objects are in the same common context not only alleviates the dataset imbalance problem but also creates a more robust classifier by learning object features rather than focusing on the context.

Approach {#sec:approach}
========

An overview of the training procedure is presented in Fig. (\[fig:GAN\_block\_diag\]). The goal is to learn a generator to translate images from majority class to images of minority class so that the generated images make the dataset balanced for training. We denote the majority class as $A$ and minority class as $B$. In Fig. (\[fig:GAN\_block\_diag\]), horse images are assumed to be the majority class while the zebra images are assumed to be the minority class.

The proposed system uses a generator ($G_{A\rightarrow B}$) to translate a majority example ($a$) in the scene to an example ($b_{gen}$) belonging to the minority class. Following the cycle-GAN philosophy, another generator ($G_{B\rightarrow A}$) will translate $b_{gen}$ to a majority class example ($a_{cyc}$). Since $a_{cyc}$ is generated from the translated image $b_{gen}$ which, in turn, is generated from the original majority class example $a$, $a_{cyc}$ is known as the cyclic image of $a$. The generated minority class image ($G_{A\rightarrow B}(a)$) along with a real minority class image ($b$) constitute the GAN loss ($\mathcal{L}_{GAN}$) while the real majority class image ($a$) and the cyclic image ($a_{cyc}$) will constitute the cycle-consistency loss ($\mathcal{L}_{cyc}$) as,

$$\begin{gathered}
\mathcal{L}_{GAN}(G_{A\rightarrow B}, D_B, A, B) = -\mathbb{E}_{b\sim p_B(b)}\big[ \log D_B(b) \big]\\
\shoveright{- \mathbb{E}_{a\sim p_A(a)}\big[\log\big(1 - D_B\big( G_{A\rightarrow B}(a) \big)\big) \big]}\\
\shoveleft{\mathcal{L}_{cyc}(A) = \mathbb{E}_{a\sim p_A(a)}\big[ || G_{B\rightarrow A}\big( G_{A\rightarrow B} (a) \big) - a||_1 \big]}\\
\label{eq:a_to_b}\end{gathered}$$

\[!t\] ![image](figs/flow_diagram_with_losses_zebra_horse.pdf){width="98.00000%"}

where, $D_B$ denotes the discriminator. Minimizing the GAN loss without the cycle-consistency constraint, can encourage the generator to map source domain images to a random permutation of the target domain images @Zhu2017Unpaired as the mapped images still produce an identical target distribution. This is, especially, true with imbalanced dataset where the network capacity can be too large with regard to the size of the dataset, facilitating the generator to memorize the target domain. Cycle-consistency loss addresses this by putting further constraints on the generator by asking it to map back to the original domain if source-to-target and target-to-source generators are composed sequentially. Following the original cycle-GAN formulation @Zhu2017Unpaired, the cycle-consistency loss in this work, is measured as the $\ell_1$ distance between an image and its mapped back version.

A similar procedure provides the corresponding losses ($\mathcal{L}_{GAN}(G_{B\rightarrow A}, \allowbreak D_A, B, A)$ and $\mathcal{L}_{cyc}(B)$) for the scenario when we want to convert a visual scene containing majority class objects to a scene containing minority class objects.

$$\begin{gathered}
\mathcal{L}_{GAN}(G_{B\rightarrow A}, D_A, B, A) = -\mathbb{E}_{a\sim p_A(a)}\big[ \log D_A(a) \big]\\
\shoveright{- \mathbb{E}_{b\sim p_B(b)}\big[\log\big(1 - D_A\big( G_{B\rightarrow A}(b) \big)\big) \big]}\\
\shoveleft{\mathcal{L}_{cyc}(B) = \mathbb{E}_{b\sim p_B(b)}\big[ || G_{A\rightarrow B}\big( G_{B\rightarrow A} (b) \big) - b||_1 \big]}\\
\label{eq:b_to_a}\end{gathered}$$

where, $D_A$ denotes the discriminator.

In addition, following @Taigman2016Unsupervised, an identity loss ($\mathcal{L}_{ide}(G_{A\rightarrow B}, G_{B\rightarrow A})$) is used that encourages the generators to produce an identity image when a real sample of the target domain is passed through them. $$\begin{gathered}
\mathcal{L}_{ide}(G_{A\rightarrow B}, G_{B\rightarrow A}) = \mathbb{E}_{b\sim p_B(b)}\big[||G_{A\rightarrow B}(b) - b||_1\big] +\\
\mathbb{E}_{a\sim p_A(a)}\big[||G_{B\rightarrow A}(a) - a||_1\big]
\label{eq:identity_loss}\end{gathered}$$ Without this loss, translated images often come with additional tint. This is, especially, problematic for the proposed framework as systematically tinted images at training time can confuse the classifier to predict almost randomly during inference where images are without such systematic tints.

The real (but abundant) and the translated (but originally rare) images will constitute a more balanced training set for the classifier $z:A\cup B\rightarrow [0,1]$, which provides a high value (close to 1) for a minority class image. Unlike the traditional classifiers, our binary cross entropy loss function comprises of losses from four different types of images namely, - Real images from minority class ($b$), Generated samples of the minority class ($b_{gen}$) and the same corresponding to the majority class ($a$ and $a_{gen}$ respectively). As the classifier provides the probability of an image to belong to the minority class, the loss coming from the minority class images is given by, $$\begin{gathered}
\label{eq:cls_loss_min_class}
\hspace{-4mm}\mathcal{L}_{cls}^B\!=\!-\mathbb{E}_{b\sim p_B(b)}\big[\log z(b) \big]\!-\!\mathbb{E}_{a\sim p_A(a)}\big[\log z(G_{A\rightarrow B} (a))\big]\end{gathered}$$ Similarly the loss coming from the majority class images is given by, $$\begin{gathered}
\hspace{-4mm}\mathcal{L}_{cls}^A\!=\!-\mathbb{E}_{a\sim p_A(a)}\big[\log(1- z(a)) \big]\!-\!\mathbb{E}_{b\sim p_B(b)}\big[\log(1-z(G_{B\rightarrow A} (b))) \big]
\label{eq:cls_loss_maj_class}\end{gathered}$$

The combined loss function for the classifier thus becomes, $$\label{eq:classification_loss}
\mathcal{L}_{cls} = \mathcal{L}_{cls}^B + \frac{1}{\gamma}\mathcal{L}_{cls}^A$$ where, $\gamma$ is the imbalance ratio that determines the relative weight of the loss components coming from the minority and the majority classes. $\gamma$ is defined to be the ratio of the number of training examples in the majority class to the same in the minority class. For a highly imbalanced scenario ($\gamma\!\!>>\!\!1$), the loss formulation penalizes the classifier much more when it misclassifies a minority class real image than when it misclassifies the same from the majority class. This is to encourage the classifier to be free from the bias induced by the majority class. The loss also penalizes the misclassification of images generated from the majority class more than the same generated from the minority class to help the classifier learn from the more diverse style captured by the GAN from the originally majority class images.

We propose two modes of training the proposed system - 1) **Augmented (AUG) Mode** 2) **Alternate (ALT) Mode**. For both the modes, the cycle-GAN is first trained on the imbalanced dataset. In augmented mode, the trained cycle-GAN is used throughout and acts as additional data generator for the classifier. In this mode, only the classifier loss $\mathcal{L}_{cls}$ (ref. Eqn. (\[eq:cls\_loss\_min\_class\])) is minimized. In alternate mode, we alternately train either the cycle-GAN or the classifier for a few epochs keeping the other fixed. The GAN is warm-started similar to the augmented mode of training while the classifier is trained from scratch. During the training of the classifier only the classifier loss $\mathcal{L}_{cls}$ is backpropagated to change the classifier weights and the GAN acts as additional data generator similar to the augmented mode. However, when the GAN is trained, the classifier acts as an additional teacher. The full objective in this case is given by, $$\begin{gathered}
\mathcal{L} = \mathcal{L}_{GAN}(G_{A\rightarrow B}, D_B, A, B) + \beta\mathcal{L}_{cyc}(A) +\\
\mathcal{L}_{GAN}(G_{B\rightarrow A}, D_A, B, A) + \beta\mathcal{L}_{cyc}(B) +\\
\alpha\mathcal{L}_{ide}(G_{A\rightarrow B}, G_{B\rightarrow A}) + \mathcal{L}_{cls}^A + \mathcal{L}_{cls}^B
\label{eq:overall_objective}\end{gathered}$$ where $\alpha$ and $\beta$ are hyperparameters weighing the relative importance of the individual loss terms. We have also experimented with end-to-end training of the whole system which works good in a balanced data regime, but the performance is not satisfactory when there is high imbalance in data. This may be due to the fact that the naturally less stable GAN training @Arjovsky2017Towards [@Salimans2016Improved] gets aggravated in presence of high imbalance in data. For example, with very few zebra and large number of horses, the cycle-GAN produces mostly the same image as the source horse with only a few faint stripes of zebra at different places of the image. Getting good quality input at the beginning is critical for a convnet classifier as the deficits due to the "noisy" inputs at the beginning of the learning process of the classifier can not be overcome later @Achille2019Critical.

Experiments {#sec:experiments}
===========

The proposed approach is evaluated on three publicly available datasets namely CelebA @Liu2015Deep, CUB-200-2011 @WahCUB2011Caltech and Horse2Zebra @Zhu2017Unpaired. Sections \[susec:celeba\], \[susec:cub\] and \[susec:horse2zebra\] respectively provide the experimental details and evaluation results on them.

We evaluate the proposed approach in terms the improvement of performance of the classifier as well as that of the cycle-GAN in presence of high class imbalance in the training data. The performance of the classifier is measured in terms of the $F_1$ score of the two classes so that the evaluation is fair irrespective of the skewness, if any, in the test data. Following the common practice, we also report the classifier performance in terms of Average Class Specific Accuracy (ACSA) @Huang2016Learning [@Mullick2019Generative]. The best performing classifier on the validation set is used for reporting the test-set performances. Due to space constraints, we provide only the $F_1$ score corresponding to the minority class in the main paper while the others are included in the . Although perceptual studies may be the gold standard for assessing GAN generated visual scenes, it requires fair amount of expert human effort. In absence of it, inception score @Salimans2016Improved is a good proxy. However, for an image-to-image translation task as is done by a cycle-GAN, inception score may not be ideal @Barratt2018Note. This is especially true in low data regime as one of the requirements of evaluating using inception score is to evaluate using a large number of samples (*i.e.*, 50k) @Salimans2016Improved. Thus, we have used an inception accuracy which measures how good an inception-v3 @Szegedy2016Rethinking model trained on real images can predict the true label of the generated samples. This is given by the accuracy of the inception-v3 model on the images translated by the cycle-GANs. A cycle-GAN providing higher accuracy to the same set of test images after translation, is better than one giving a lower accuracy. For this purpose, the ImageNet pretrained inception-v3 model is taken and the classification layer is replaced with a layer with two neurons. This modified network is then finetuned on balanced training data that is disjoint from the test set for all the three datasets. For each dataset the evaluation procedure is repeated 5 times while keeping the data fixed but using independent random initialization during runs. The average result of these 5 runs is reported.

Experiments on CelebA {#susec:celeba}
---------------------

Many datasets have contributed to the development and evaluation of GANs. However, the CelebA @Liu2015Deep dataset has been the most popular and canonical in this journey. CelebA contains faces of over 10,000 celebrities each of which has 20 images. We used images from this dataset to study the effect of imbalance for a binary classifier predicting the gender from the face images only. Gender bias due to imbalance in datasets @Miltenburg2016Stereotyping [@Hendricks2018Women] has started to gain attention recently and face images make the problem more challenging as many attributes can be common between males and females or can only be very finely discriminated.

<span>**Experimental Setup**</span>: We took a subset of 900 female images and considered this to be the majority class. The male images are varied in number from 100 to 900 in regular intervals of 100. In addition, we experimented with 50 male images too which makes the highest imbalance ratio ($\gamma$) for this dataset to be 18. We follow the same architecture of the cycle-GAN as that of Zhu *et al.* @Zhu2017Unpaired where the generator consists of two stride-2 convolutions, 9 residual blocks, and 2 fractionally-strided convolutions. For discriminator, similarly, a 5 layer CNN is used for extracting features, following the PatchGAN @Isola2017Image architecture. To avoid over-fitting in presence of less data, our binary classifier is a simple CNN without any bells and whistles. We used a classifier with same feature extraction architecture as that of the discriminator used above. The initial convolution layers are followed by two fully-connected layers of 1024 and 256 neurons along with a output layer consisting of two neurons. To handle over-fitting further, we used dropout with probability $0.5$ of dropping out neurons from the last fully-connected layer. Following @Zhu2017Unpaired, we have used $\alpha = 5$ and $\beta = 10$ as the loss weights for the cycle-GAN throughout the experiments. The test set for this dataset comprises of 300 images of male and female each.

<span>**Results**</span>: Table \[tab:celeba\_cub\] shows a comparative evaluation of the classifier performance of our proposed model for both AUG and ALT mode. In this table, we have provided the results for upto 500 male images due to space constraint and the rest are provided in the . We compared with some of the classic methods as well as methods particularly used with deep learning based classifiers. Namely they are random oversampling of minority class (OS), random undersampling of majority class (US), class frequency based cost sensitive weighing of loss (CS) @Lawrence1998Neural, adjusting decision threshold by class frequency at prediction time (TS) and SMOTE @Chawla2002Smote are some of the classical methods with which we compare our proposed method. We have also studied the effect of combining a few of the above classical methods and are reporting the two combinations - undersampling with cost sensitive learning (US+CS) and oversampling with cost sensitive learning (OS+CS). We have also compared with recent state-of-the-art approach using a class balanced loss (CBL) @Cui2019Class for deep learning based classifiers.

We started with a highly imbalanced dataset of male and female faces where we randomly took 50 male and 900 female face images. Training the classifier without any data balancing strategy (we refer this classifier as the 'vanilla' classifier) misclassifies almost all of the minority class examples (male) as majority class examples (female) resulting in a accuracy of only 0.08 for males while the accuracy for the female images is perfect (*i.e.*, 1). The $F_1$ score of the minority class for this case is 0.15. Our AUG mode of training improves the accuracy more than 5 fold to 0.8060 while the ALT mode of training improves it further to 0.8240. The fact that the classification is more balanced after the proposed dataset repairment is corroborated by the simultaneous increase of the precision of the majority class (female) from 0.5200 to 0.7720 (AUG mode) and 0.8260 (ALT mode). The male classification accuracy in AUG and ALT mode reaches up to 0.736 and 0.820 respectively. Table \[tab:celeba\_cub\] shows the minority class $F_1$ score comparison with the state-of-the-arts. For 4 out of the 6 highly skewed regions (imbalance ratio ranging from 18 to 1.8) our proposed method significantly outperforms the rest. Out of the two cases, our augmentation mode is a close second (for 900:300 scenario). As the imbalance increases, our relative performance is better and better compared to other approaches. Performing better in highly skewed data distribution is very important as high imbalance implies more difficult case. This shows that our approach can deal with the hard cases in the challenging CelebA dataset.

\[!t\]

We have also provided t-SNE @Maaten2008Visualizing visualizations of the learned feature representation from the last fully-connected layer the classifier before and after the dataset repairment. Fig. \[fig:celeba\_50\_900\_bias\] and show the visualization of the test set features for the vanilla classifier and the same trained in augmented mode respectively. This is for the case when the number of minority training examples is 50 while the majority training example is 900. It can be seen that the separation between the two classes is better after the repairment even in presence of such high imbalance. The same can be seen for the imbalanced training with 100 male and 900 female faces as shown in Fig. \[fig:celeba\_100\_900\_bias\] (vanilla) and (AUG).

Though in the high imbalance region (towards left of Table \[tab:celeba\_cub\]), both the modes of our proposed approach perform very good, the augmentation mode of training is consistently better than the alternate mode of training. The significance of the alternate mode of training, though, lies in the fact that it allows the cycle-GAN to improve itself by helping the classifier to discriminate better in presence of highly skewed training data. The improvement of performance of the GAN in terms of inception accuracy is shown in Table \[tab:gan\_perf\]. Except for one scenario, the GAN performance improves for all the rest. Thus, the proposed approach provides a choice between going for bettering the GAN by trading off a little in the classifier performance (ALT mode) or going for improving the classifier without changing the GAN (AUG mode).

**Visualizing important attributes for classification**: We aim to visually confirm that the proposed approach helps the classifier to look at the right evidences. For this purpose, we rely on two visual explanation techniques: Grad-CAM @Selvaraju2017Gradcam and RISE @Petsiuk2018Rise. They provide explanation heatmaps pointing to locations in the input image that are most important for a decision. Fig. \[fig:saliency\] shows visualizations for some representative minority images when they are passed through the vanilla classifier and the same after training in AUG and ALT mode. The top row shows a representative zebra image from the Horse2Zebra dataset. The vanilla model tends to concentrate on the surroundings (context) to classify (incorrectly), while the proposed models accurately concentrate on the zebras (object) for correct classification. Similarly, the bottom row shows a flycatcher image from CUB dataset. When classifying the image, the vanilla model tends to scatter its attention while the proposed models base their prediction on right areas by learning to ignore inappropriate evidences.

![Explanation heatmaps of images from Horse2Zebra (first row) and CUB (second row). Leftmost column shows the images while col 2-4 and 5-7 show RISE and Grad-CAM heatmaps respectively (importance increases from blue to red). The column sub-headings denote the corresponding classifier. Note that the heatmaps are generated for the predicted output by the classifiers. For vanilla mode, the prediction is wrong (horse and sparrow respectively) while for both AUG and ALT mode they are right. An interesting observation is revealed by the RISE generated heatmap (first row $4^{th}$ col) showing that the evidence for ALT mode classifier is coming from a third zebra hidden in the grass. Best viewed in color.<span data-label="fig:saliency"></span>](figs/Saliency-fig.pdf){width="0.95\linewidth"}

-0.2in

Experiments on CUB {#susec:cub}
------------------

CUB-200-2011 birds dataset @WahCUB2011Caltech is a popular benchmark dataset for fine-grained image classification. It contains 11,788 images of 200 different classes of North American bird species. Fine-grained classification is inherently difficult task as the members of the classes have very similar attributes with small inter-class variation requiring the classifier to focus on finer aspects of the images that are both descriptive and adequately discriminative. The challenges of fine-grained classification requires a lot of parameters to learn @Gao2016Compact [@Lin2015Bilinear; @Zhang2019Learning] which in turn needs a lot of training examples for proper training. Thus fine-grained classification provides us with an interesting testcase for low-data high-bias regime of classification.

<span>**Experimental Setup**</span>: For the experimentations, we have chosen two similar looking bird species (Flycatcher and Sparrow). We took all the images (226) of flycatcher that the dataset has and considered it to be the minority class. The 226 flycatcher images are split into 125 train, 50 val and 51 test images. The number of majority class (sparrow) images were kept bowersphilip@example.com. We vary the number of flycatcher images from 25 to 125 in steps of 25 making the imbalance ratio vary from 10 to 5 in steps of 1. In addition, we also experimented with only 12 flycatcher images. The val and test set size of sparrow is kept same as those of the flycatcher class. The model architectures are kept same as those used for CelebA experiments (ref. Sec \[susec:celeba\]).

<span>**Results**</span>: Comparative evaluations of the proposed approach for the pair of birds in the CUB-200-2011 data set is shown in the rightmost part of Table \[tab:celeba\_cub\]. The table provides the $F_1$ score of the minority class for the different approaches. It can be observed that our proposed method in augmentation mode is outperforming other approaches for highly skewed training data. The alternate mode of training the classifier does not perform as good as it does for the face images. However, the cycle-GAN inception accuracy gets improved for 4 out of 6 cases as can be seen in the middle part of Table \[tab:gan\_perf\] where for highly skewed training data, the alternate model betters the vanilla cycle-GAN.

Experiments on Horse2Zebra {#susec:horse2zebra}
--------------------------

Cycle-GAN @Zhu2017Unpaired has been particularly successful in translating between images of horses and zebras. Unlike finegrained bird images, horses and zebras have distinctive features with marked difference in skin texture which allows a classifier to perform well even with very less data. To test with this end of the spectrum, our proposed approach was applied for a classification task between horse and zebra images in presence of high imbalance in training data.

<span>**Experimental Setup**</span>: The Horse2Zebra dataset @Zhu2017Unpaired contains 1187 images of horses and 1474 images of zebras. We took the horse class as majority and used a total of 450 Horse images. The number of minority class (zebra) examples were varied from 25 to 100 in steps of 25 allowing us to experiment with low data and low balance regime. The val and test set consists of 100 and 150 images respectively for each of the animal categories. For this case also, the model architectures are kept same as those used for CelebA experiments (ref. Sec \[susec:celeba\]).

<span>**Results**</span>: Table \[tab:horse2zebra\] shows comparative evaluations of the proposed approach for this dataset in terms of the $F_1$ score of the minority class (zebra). The proposed method in AUG mode of training comprehensively outperforms the other approaches in the high imbalance region with the difference in $F_1$ score with the second best approach reaching almost 2.5% for the highest imbalance ratio with the number of training images for zebra being only 25. The inception accuracies of the cycle-GAN before and after training with ALT mode is provided in the rightmost part of Table. \[tab:gan\_perf\]. For this dataset we got consistently worse GANs while training in this mode. This can be due to the distinctive texture between the two animals that aids classification but imposes difficulty in translation especially in presence of the classifier loss coming from noisy data.

Conclusion {#sec:conclusion}
==========

We propose a joint generation and classification network to handle severe data imbalance by transforming majority class examples to minority class and vice versa. Our approach explores a Cycle-consistent GANs that not only generates training samples that are much broader and diverse but also balanced in terms of number of examples. Experiments on three standard datasets demonstrate that both classifier and the GAN achieves superior performance than existing data imbalance mitigation approaches.

<span><span>**Acknowledgements:**</span> This work was partially supported by the IIT Kharagpur ISIRD program and the SERB Grant SRG/330.284.8476.</span>

Appendix {#appendix: Results .unnumbered}
========

Training Details
================

On CelebA Dataset {#appendix: celeb}
-----------------

-   The Cycle-GAN was trained from scratch on the imbalance data with a learning rate of 2e-4, and weights initialized from a Gaussian distribution $N(0, 0.02)$. We follow @Zhu2017Unpaired for the hyperparameters and choice of optimizers. The learning rate was kept constant through out the training. The image size used for training is 256x256.

-   We use the 200-Epoch trained models for generating the translated images and using them for training the classifier in **AUG-Mode**.

    -   In **AUG-Mode**, we train the Classifier for 20 Epochs and use a validation-set containing 300 images from each class to select the best model.

-   For **ALT-Mode** training, we use the weights of the 200-Epoch models of Vanilla Cycle-GAN to warm-start the Cycle-GAN part of the ALT-Mode Model and the Classifier part is initialized with weights taken from a Gaussian distribution of $N(0, 0.02)$.

    -   The interval for swapping the training in AUG-Mode was kept to be 5-Epochs.

    -   The Classifier is trained for the first 5-Epochs keeping the Cycle-GAN part frozen.

    -   In the next 5-Epochs, the Cycle-GAN is trained keeping the Classifier part frozen, but in this case the additional loss obtained from the Classifier part is also back-propagated through the Cycle-GAN part.

    -   The above mentioned training-swap procedure is followed till the models are trained for a total of 100-Epochs.

-   For evaluating the GANs we take the GAN models obtained after the 100-Epoch training in ALT-Mode and compare them with the corresponding models obtained after 300-Epoch training of Vanilla Cycle-GAN. The 300-Epoch models of Vanilla Cycle-GAN were chosen to provide a fair comparison of the GAN Performance as in the AUG-Mode, we warm-start the Cycle-GAN part with 200-Epoch Models.

On CUB Dataset {#appendix: CUB}
--------------

-   The Cycle-GANs were trained in the same fashion as used in case of the CelebA Dataset with some changes mentioned below. The learning rate was kept constant for the first 50 Epochs and is linearly decayed to zero over the next 150 Epochs of training.

-   For CUB, we use the 50-Epoch trained models for **AUG-Mode**.

    -   In **AUG-Mode**, all procedures are kept same as that of CelebA but we use a validation-set containing 50 images from each class to select the best model.

-   For **ALT-Mode** training, we use the weights of the 50-Epoch models of Vanilla Cycle-GAN to warm-start the Cycle-GAN part of the ALT-Mode Model and the rest are same as that in case of CelebA Dataset.

    -   The training-swap procedure is followed till the models are trained for a total of 50-Epochs.

-   For evaluating the GANs we take the GAN models obtained after the 50-Epoch training in ALT-Mode and compare them with the corresponding models obtained after 100-Epoch training of Vanilla Cycle-GAN.

On Horse2Zebra Dataset {#appendix: H2Z}
----------------------

-   All the training procedures and the Epoch-checkpoints for using the models are kept same as that in the case of the CUB Dataset, except we use a validation-set containing 100 images from each class in this case.

[^1]: The first two authors contributed equally. The work was done when Ankit Singh was at IIT Kharagpur.

{'source': 'arXiv_out/2008'}