{'fraction_non_alphanumeric': 0.036057846813069096, 'fraction_numerical': 0.016197107659346545, 'mean_word_length': 4.5719231228363375, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Feature Transfer Learning for Deep Face Recognition with Long-Tail Data

Real-world face recognition datasets exhibit long-tail characteristics, which results in biased classifiers in conventionally-trained deep neural networks, or insufficient data when long-tail classes are ignored. In this paper, we propose to handle long-tail classes in the training of a face recognition engine by augmenting their feature space under a center-based feature transfer framework. A Gaussian prior is assumed across all the head (regular) classes and the variance from regular classes are transferred to the long-tail class representation. This encourages the long-tail distribution to be closer to the regular distribution, while enriching and balancing the limited training data. Further, an alternating training regimen is proposed to simultaneously achieve less biased decision boundaries and a more discriminative feature representation. We conduct empirical studies that mimic long-tail datasets by limiting the number of samples and the proportion of long-tail classes on the MS-Celeb-1M dataset. We compare our method with baselines not designed to handle long-tail classes and also with state-of-the-art methods on face recognition benchmarks. State-of-the-art results on LFW, IJB-A and MS-Celeb-1M datasets demonstrate the effectiveness of our feature transfer approach and training strategy. Finally, our feature transfer allows smooth visual interpolation, which demonstrates disentanglement to preserve identity of a class while augmenting its feature space with non-identity variations.


Introduction
Face recognition is one of the ongoing success stories of the deep learning era, yielding very high accuracies on traditional datasets [1][2][3]. However, it remains undetermined how these results translate to practical applications, or how deep learning classifiers for fine-grained recognition must be trained to maximally exploit real-world data. While it has been established that recognition engines are data-hungry and keep improving with more volume [4], mechanisms to derive benefits from the vast diversity of real data are relatively unexplored. In particular, real-world data is long-tailed [5], with only a few samples available for most classes. In practice, effective handling of long-tail classes is also indispensable in surveillance applications where subjects may not cooperate during data collection. 1 A visualization demo can be found at https://youtu.be/k4JOT5BcbVc and more detail can be found at: http://cvlab.cse.msu.edu/project-feature-transfer. (d) Even after data re-sampling, the variance of ID 1008 is much larger than ID 10449, causing decision boundary to still be biased towards the tail class. We augment the feature space of the tail classes as the dashed ellipsoid and propose improved training strategies, leading to an improved classifier.
It is evident that classifiers that ignore this long-tail nature of data likely imbibe hidden biases. Consider the example of the CASIA-Webface dataset [6] in Fig. 1(a), where about 39% of the 10K subjects have less than 20 images. A simple solution is to simply ignore the long-tail classes, as common for traditional batch construction and weight update schemes [7]. Besides reduction in the volume of data, the inherently uneven sampling leads to biases in the weight norm distribution across head and tail classes ( Fig. 1(b,c)). Sampling tail classes at a higher frequency addresses the latter, but still leads to biased decision boundaries due to insufficient intra-class variance in tail classes ( Fig. 1(d)).
In this paper, we propose strategies for training more effective classifiers for face recognition by adapting the distribution of learned features from tail classes to mimic that of head (or regular) classes. We propose to handle long-tail classes during training by augmenting their feature space using a center-based transfer. In particular, we assume a Gaussian prior, whereby most of the variance of regular classes is captured by the top few components of a Principal Components Analysis (PCA) decomposition. By transferring the principal components from regular to long-tail classes, we encourage the variance of long-tail classes to mimic that of regular classes. Motivations for center-based transfer can also be found in recent works on the related problem of low-shot recognition [8], where the feature center is found to be a good proxy that preserves identity. Thus, restricting the transfer variance within the minimum inter-class distance limits the transfer error to be within the classifier error.
Our feature transfer overcomes the issues of imbalanced and limited training data. However, directly using the augmented data for training is sub-optimal, since the transfer might further skew the class distributions. Thus, we propose a training regimen that alternates between carefully designed choices to solve for the feature transfer (with the goal of obtaining a less biased decision boundary) and feature learning (with the goal of learning a more discriminative representation). Further, we propose a novel metric regularization that jointly regularizes softmax feature space and weight templates, leading to empirical benefits such as reduced problems with vanishing gradients.
An approach for such feature-level transfer has also been proposed by Hariharan and Girshick [9] for 1K-class ImageNet classification [10]. But the face recognition problem is geared towards at least two orders of magnitude more classes, which leads to significant differences due to more compact decision boundaries and different nature of within-class variances. In particular, we note that the intuition of [9] to transfer semantic aspects based on relative positions in feature space is valid for ImageNet categories that vary greatly in shape and appearance, but not for face recognition. Rather, we must transfer the overall variance in feature distributions from regular to long-tail classes.
To study the empirical properties of our method, we mimic a long-tail dataset by limiting the number of samples for various proportions of classes in the MS-Celeb-1M dataset [3], while evaluating on LFW [1], IJB-A [2] and the hold-out set from MS-Celeb-1M dataset. We observe that our feature transfer consistently improves upon a method that does not specifically handle long-tail classes. Moreover, we observe that adding more long-tail classes improves the overall performance of face recognition. We compare against the state-of-the-art on LFW and IJB-A benchmarks, to obtain highly competitive results that demonstrate improvement due to our feature transfer. Further, our method can be applied to challenging low-shot or one-shot scenarios, where we show competitive results on the one-shot MS-Celeb-1M challenge [7] without any tuning. Finally, we visualize our feature transfer through smooth interpolations, which demonstrate that a disentangled representation is learned that preserves identity while augmenting non-identity aspects of the feature space.
To summarize, we make the following contributions to face recognition: -A center-based feature-level transfer algorithm to enrich the distribution of longtailed classes, leading to diversity without sacrificing volume. It also leads to an effective disentanglement of identity and non-identity feature representation. -A simple but effective metric regularization to enhance performances for both our method and baselines, which is also applicable to other recognition tasks. -A two-stage alternating training scheme to achieve an unbiased classifier and retain discriminative power of the feature representation despite augmentation. -Empirical analysis through extensive ablation studies and demonstration of benefits for face recognition in both general and one-shot settings.

Related Work
Imbalanced data classification Classic works study data re-sampling methods [11,12], which learn unbiased classifiers by changing the sampling frequency.
With the use of deep networks [13,14], the frontier of face recognition research has been significantly advanced [15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30]. Despite this success, there are only few previous works that discuss about learning from long-tail classes. Huang et al. [31] propose quintuplet sampling based hinge loss to maintain both intercluster and inter-class margins. Zhang et al. [32] propose the range loss that simultaneously reduces intra-class variance and enlarges the inter-class variance. Guo and Zhang [7] propose tail class promotion loss that regularizes the norm of weight vectors of tail classes. Other than designing data sampling rules or regularization on tail classes, we augment tail classes by generating feature-level samples through transfer of intra-class variance from regular classes.
One-shot and low-shot learning Low-shot learning aims at recognizing an image for a specific class with very few or even one image available at training. Some efforts are made by enforcing strong regularization [7,9] or utilizing nonparametric classification methods based on distance metric learning [33][34][35].
Generative model based methods have also been studied in recent years. Dixit et. al [36] propose a data augmentation method using attribute-guided feature descriptor for generation. Hariharan et. al [9] propose non-parametric generation of features by transferring within class pair-wise variation from regular classes under object classification task. Compared to their task on ImageNet [10] with 1K categories, face recognition is a more fine-grained classification task incorporating at least two order of magnitude more categories. leading to low inter-class variance.
Feature transfer learning Transfer learning applies information from a known domain to an unknown domain. We refer to [37] for further discussion. Attributes are used in [36] to synthesize feature-level data. In [38], features are transferred from web images to video frames using a generative adversarial network (GAN) [39]. In our framework, we similarly conduct feature level transfer. But we do not use additional supervision, which may introduce new biases to the transfer. Different from GANs, we do not use an additional discriminator, but utilize the classifier that is trained internally to serve as the perception loss.

The Proposed Approach
In Sec. 3.1, we introduce the problems caused by long-tail classes on training, such as classifier weight norm bias or intra-class variance bias, and overview challenges and solutions that will be discussed with more details in later sections. Then, we demonstrate the overall framework in Sec. 3.2 with a novel regularization. In Sec. 3.3, a center-based feature transfer method is proposed to resolve the intra-class variance bias of long-tail classes. We finally present an alternating regimen for updating the classifier with the proposed feature transfer and the feature representation to effectively train the entire system in Sec. 3.4.

Issues for Training with Long-tail Classes in Face Recognition
It is known that training deep face representations using data with long-tail distribution results in degraded performance [40]. We have similar observations in our experiments, where we train CASIA-Net [6] on CASIA-Webface [6], whose data distribution indeed shows long-tail behavior as in Fig. 1 (a). We further observe two atypical classifier behaviors, such as significant variations on norms of classifier weights or intra-class variances between regular and long-tail classes. Stage 1: The proposed framework includes a feature extractor Enc, a decoder Dec, a feature filtering module R, and a fully connected layer as classifier F C. The proposed feature transfer module G generates new featureg from original feature g. The network is trained with an alternative bi-stage strategy. At stage 1, we fix Enc and apply feature transfer G to generate new features (green triangle) that are more diverse and likely to violate decision boundary. In stage 2, we fix the rectified classifier F C, and update all the other models. As a result, the samples that are originally on or across the boundary are pushed towards their center (blue arrows in bottom right). Best viewed in color.
Imbalance in Classifier Weight Norm: As shown in Fig. 1 (b), we observe the norm of classifier weight (i.e., weight matrix of last fully connected layer) of regular classes is much larger than that of tail classes, which causes the decision boundary biases towards the tail class. This is mainly due to the fact that the weights of regular classes are more frequently updated than those of tail classes. In this regards, there exist several well-known solutions, such as data re-sampling in proportion to the volume of each class or class weights normalization [7].
Imbalance in Intra-class Variance: Unfortunately, we still observe significant imbalance after weight norm regularization via data re-sampling. 2 As an illustrative example, we randomly pick two classes, one from a regular class (ID=1008) and the other from a tail class (ID=10449). We visualize the features from two classes projected onto 2D space using t-SNE [41] in Fig. 1(c) and those after weight norm regularization in Fig. 1(d). Although the weights are regularized to be similar, the low intra-class variance of the tail class is not fully resolved. This causes the decision boundary to be biased, which impacts recognition performance.
We build upon this observation to posit that enlarging the intra-class variance for tail classes is the key to alleviate the impact of long-tail classes. In particular, we propose a data augmentation approach at the feature-level that can be used as extra positive examples for tail classes to enlarge the intra-class variance. As illustrated in Fig. 1(d), the feature distribution augmented by these virtual positive examples helps rectify the classifier boundary, which in turn allows reshaping the feature representation.
The remainder of this section proposes specific mechanisms for regularization, feature augmentation and neural network training to realize the above intuitions.

Proposed Face Recognition Framework
Many recent successes in deep face recognition are attributable to the design of novel loss or regularization [18,22,34,[42][43][44][45], that reduces over-fitting to limited amount of labeled training data. In contrast, our method focuses on recovering the missing samples of tail classes by transferring knowledge from regular classes to enlarge intra-class variance. At first glance, our goal of diversifying features of tail classes appears to contradict the general premise of deep learning frameworks, which is to learn compact and discriminative features. However, we argue that it is more advantageous to learn intra-class variance of tail classes for generalization, that is, adapting to unseen examples. To achieve this, we enlarge the intra-class variance of tail classes at a lower layer, which we call a rich feature layer [46], while subsequent filtering layers learn a compact representation with the softmax loss. Next, we define the training objectives of our proposed framework.
As illustrated in Fig. 2, the proposed face recognition system is composed of several components, such as an encoder, decoder, feature transfer module followed by filtering and classifier layers, as well as multiple training losses, such as image reconstruction loss, or classification loss. An encoder Enc computes rich feature g = Enc(x) ∈ R 320 of an input image x ∈ R 100×100 and reconstruct an input image with a decoder Dec, i.e., x = Dec(g) = Dec(Enc(x)) ∈ R 100×100 . This pathway is trained with the following pixel-wise reconstruction loss: The reconstruction loss allows g to contain diverse attributes besides identity, such as pose and expression that are to be transferred from regular classes to tail classes. A feature transfer module G transfers the variance computed from regular classes and generates a new featureg = G(g) ∈ R 320 from tail classes, as described in the next section. Then, a filtering network R is applied to generate identity-related features f = R(g) ∈ R 320 that are fed to a classifier layer F C with weight matrix [w j ] ∈ R Nc×320 . This pathway optimizes the softmax loss: where i is the ground truth label of f . We note that the softmax loss is scale-dependent, that is, the loss can be made arbitrarily small by scaling the norm of the weights w j or feature f . Typical solutions to prevent the problem are to either regularize the norm of weights 3 or features [44], or to normalize them [7,27]. However, we argue that these are too stringent since they penalize norms of individual weights and feature without considering their compatibility. Instead, we propose to directly regularize the norm of exponent of softmax loss as follows: Features are extracted by VGGFace [19] and samples are from CASIA-WebFace [6].
We term our proposed regularization a metric L 2 or m-L 2 . As we will discuss in Sec. 4.2, joint regularization of weights and features through the magnitude of their inner product works better in practice than individual regularization.
Finally, we formulate the overall training loss as Eqn. 4, with the regularization coefficients set to α sf mx =α recon =1, and α reg =0.25 unless otherwise stated:

Long-tail Class Feature Transfer
Following previous face recognition approaches, such as joint Bayesian face models [17,47,48], we assume that rich features g ik from class i lies in Gaussian distribution with the class-specific mean c i and the covariance matrix Σ i . To transfer intra-class variance from regular to long-tail classes, we assume the covariance matrices are shared across all classes, Σ i = Σ. Under this assumption, the mean, or a class center, is simply estimated as an arithmetic average of all features from the same class. As shown in the left of Fig. 3, the center representation for regular classes is identity-specific while removing irrelevant factors of variation such as pose, expression or illumination. However, as in the right of Fig. 3, due to lack of training examples, the center estimate of long-tail classes is not accurate and often biased towards certain identity-irrelevant factors, such as pose, which we find dominant in practice. To improve the quality of center estimate for long-tail classes, we discard examples with extreme pose variations. Furthermore, we consider averaging features from both the original and horizontally flipped images. Withḡ ik ∈ R 320 a rich feature extracted from the flipped image, the feature center is estimated as follows: where p ik andp ik are the pose codes forḡ ik and g ik , respectively. Ω i includes indices for examples with yaw angle less than a threshold τ . Next, we transfer the variance estimated from the regular classes to long-tail classes. In theory, one can draw feature samples of long-tail classes by adding a noise vector ∼ N (0, Σ). However, the direction of noise vectors might be too random when sampled from the distribution and does not reflect the true factors of variation found in the regular classes. Instead, we transfer the intraclass variance evaluated from individual samples of regular classes. To further remove identity-related component in the variance, we filter them using PCA basis Q ∈ R 320×150 [49] achieved from intra-class variances of all regular classes. We take the top 150 eigen vectors as preserving 95% energy. Our center-based feature transfer is achieved using: where g jk and c j are a feature sample and center of a regular class j, c i is the feature center of a long-tail class i andg ik is the transferred feature for class i. Here,g ik preserves the same identity as c i , with similar intra-class variance as g jk . By sufficiently sampling g jk across different regular classes, we expect to obtain an enriched distribution of the long-tail class i, which consists of both the original observed features g ik and the transferred featuresg ik .

Alternating Training of Classifier and Feature Representation
Given a training set of regular and long-tail classes D = {D reg , D lt }, we first train all modules M = {Enc, Dec, R, F C} using Eqn. 4 without any feature transfer. Then, we alternatively train a classifier until convergence with decision boundary reshaping using our proposed feature transfer and a feature representation with boundary-corrected classifier. The overview of our two-stage alternating training process is illustrated in Algorithm 1. We describe in more details of each training stage below.
Stage 1: Decision boundary reshape. In this stage, we update R and F C while fixing other modules using variance transferred features from regular to longtail classes to enlarge the intra-class variance of long-tail classes, thus, reshape the decision boundary. We first update the statistics including the feature centers C, PCA basis Q and an index list h of hard samples that are with the distance from the center more than the average distance for each regular class. The PCA basis Q is achieved by decomposing the covariance matrix V computed with the samples from regular classes D reg . Three batches are used for training in each iteration: a regular batch sampled from hard index list h: {g r , y r }, a long-tail batch sampled from long-tail classes {g t , y t }, and a transferred batch {g t , y t } by transferring the variances from regular batch to long-tail batch.
Stage 2: Compact feature learning. In this stage, we train Enc, Dec as well as R using normal batches {x, y} from regular and long-tail classes using Eqn. 4 without transferred batch. We keep F C fixed since it is already trained well from the previous stage with decision boundary corrected using feature transfer. The gradient directly back-propagates to R and Enc for more compact representation, which decreases violation of class boundaries.

Experiments
We train our models on MS-Celeb-1M dataset [3], which consists of 10M images from around 100K celebrities. Due to label noise, we adopt a cleaned version from [50] and further remove the subjects overlapped with LFW [1] and IJB-A [2], which results in 4.8M images of 76.5K classes for training. We apply face detection [51] and alignment [52] to pre-process a face image and crop to 100 × 100. A class with no more than 20 images is considered as a long-tail class, following [32]. For implementation, we apply encoder-decoder structure similar to [23] and ResNet-54 for Enc in Sec. 4.5. Model R consists of a F C layer, two full convolution layers, two convolution layers and another F C layer to achieve f ∈ R 320×1 . More detail is referred to supplementary material. Adam solver with learning rate 2e −4 is used in stage 0. Learning rate 1e −5 is used in stage 1 and 2, which alternated for every 5K iterations.

Feature Center Estimation
Feature center estimation is a key step for feature transfer. To evaluate center estimation for tail classes, 1K regular classes are selected from MS-Celeb-1M and features are extracted using a pretrained recognition model. We randomly select a subset of 1, 5, 10, 20 images to mimic a long-tail class. Three methods are compared: (1) "PickOne", randomly pick one sample as center. (2) "AvgAll", average feature of all images. (3) "AvgFlip", proposed method in Eqn. 5. Error is the difference between the center of full set (ground truth) and the subset. Intraclass and inter-class variance are provided as reference. All errors are normalized by the inter-class variance.
Results in Fig. 4 show that our "AvgFlip" achieves clear smaller error. When compared to the intra-class variance, the error is fairly smaller, which convince that our center estimate is accurate to support the feature transfer.

Effects of m-L2 Regularization
To study the effects of the proposed m-L 2 regularization, we show a toy example on the MNIST dataset [53]. We use LeNet++ network [21] to learn a 2D feature for better visualization. Two models are trained: one with softmax loss only; the other with softmax loss and m-L 2 regularization (α reg = 0.001).
m-L 2 regularization has several advantages. (1) m-L 2 effectively avoids overfitting. In Fig. 9, sof tmax training shows over-fitting as training error goes to 0 whereas with m-L 2 training error stays small but not 0. (2) m-L 2 enforces a more balanced feature distribution. Fig. 9 (c) shows a more balanced angular distribution than Fig. 9 (b). The performance with m-L 2 improves sof tmax from 99.06% to 99.35%. We believe m-L 2 is a simple yet powerful general regularization that can be easily adapted to other recognition problems.

Ablation Study
We study two factors to analyze the long-tail training: (1) the ratio of the portion of regular classes vs. the portion of long-tail classes; (2) the number of images per long-tail class. We use discrete approximation to mimic the real regular vs. long-tail class distribution and the continuous distribution of number of samples per tail class. Our main focus is to analyze the long-tail distribution impact on recognition thus assume discrete setting for simplicity. Regular/long-tail class ratio: we use 60K regular classes with most number of images from MS-Celeb-1M. The top 10K classes are selected as regular classes which are shared among all training sets. We regard the 10K and 60K sets to serve as the lower and upper bounds. Among the rest 50K classes sorted by number of images, we select the first 10K, 30K and 50K and randomly pick 5 images per class. In this way, we form the training set of 10K10K, 10K30K, and 10K50K, of which the first 10K are regular and the last 10K or 30K or 50K are called faked long-tail classes. A hold-out testing set is formed by selecting 5 images from each of the shared 10K regular classes and 10K tail classes, resulting in 100K testing images.
The evaluation on the hold out test set from MS-Celeb-1M is to mimic the low-shot learning scenario, where we use the feature center from the training images as the gallery and nearest neighbor (NN) for face matching. The rank-1 accuracy for both regular and long-tail classes are reported. We also evaluate the general face recognition performance on LFW and IJB-A. The results are shown in Table 1 and we draw the following observations.
-The feature space g is less discriminative than the feature space f , which validates our assumption that g is rich in intra-class variance for feature transfer while f is more discriminative for face recognition. -The proposed m-L 2 regularization boosts the performance with a large margin over the baseline softmax loss. -The proposed transfer method consistently improves over sfmx and sfmx+m-L 2 with significant margins, and largely close the gap from 10K0K to 60K0K. -Our method is more beneficial when more long-tail classes are added to training as more long-tail classes lead to better face recognition performance.
Number of images per long-tail class: we vary n = 1, 5, 10, 20 under setting 10K30K. Table 2 reveals that more images in long-tail classes leads to better results, due to better center estimation. Consistent with Table 1, the proposed algorithm significantly improves performance on low-shot setting of MS-Celeb-1M and general face recognition on LFW and IJB-A. On 10K30K (n = 5) setting, we  look into the F C classifier performance, 93.59% and 2.04% for regular and longtail respectively. Whereas our method achieves 96.26% and 81.89% accordingly, which suggests our method's effectiveness in correcting classifier bias.

One-shot Face Recognition
While our method has only tangential relation to one-shot recognition, we evaluate on the MS1M one-shot challenge as an illustration [7]. In this setting, the training data consists of a base set with 20K classes each with 50 ∼ 100 images and a novel set of 1K classes each with only 1 image. The test set consists of 1 image for each base class and 5 images for each novel class. The main purpose is to evaluate the recognition performance on the novel classes while monitoring the performance on the base classes. As shown in Table 3, we achieve 95.48% rank-1 accuracy with a single model and single crop testing. We use the output from softmax layer as the confidence score and achieve 92.60% coverage at precision of 0.99. Note that the best models [57] and [55] use model ensembling with different crops for testing. Compared to similar setting methods [7,56], we achieve competitive performance on the base classes and much better results on the novel classes.

Large-scale Face Recognition
In this section, we train our model on the full MS-celeb-1M dataset and evaluate on LFW and IJB-A. The cleaned dataset includes 76.5K classes, of which 9.5K classes consist of less than 20 images. We use a ResNet-54 structure for Enc. As shown in Table 4, the deeper network structure with our proposed m-L 2 regularization already provides good results. Our feature transfer learning further improves the performance significantly. On LFW, our performance is among the  Table 4. Face recognition on LFW and IJB-A. "MP" represents media pooling and "TA" represents template adaptation. The best and second-best results are highlighted. state-of-the-art. On IJB-A, our method significantly outperforms most of the methods except "NAN". While "NAN" is designed with attention or aggregation model to specifically incorporate temporal information, our method is geared towards still image recognition with long-tail classes.

Qualitative Results
We apply decoder Dec in our framework for feature visualization. It is well known that the skip link between an encoder and decoder can improve visual quality [25]. However, we do not apply it in order to encourage the feature g to incorporate the intra-class variance other than from the skip link.
Center Visualization Given a class with multiple samples, we compute a feature center, on which the Dec is applied to generate a center face. From Fig. 6, we confirm the observation that the center is mostly an identity-preserved frontal neutral face. It also applies to portrait and cartoon figures.

Feature Transfer
The transferred feature is visualized using the Dec. Let x 1,2 , x 1,2 , g 1,2 , c 1,2 denote the input images, reconstructed images, encoded features and feature centers of two classes, respectively. We transfer feature from class 1 to class 2 by: g 12 = c 2 + QQ T (g 1 − c 1 ), and visualize the decoded images. We also transfer from class 2 to class 1 and visualize the decoded images. Fig. 11 shows the examples of feature transfer between two classes. The transferred images preserve the target class identity while retaining intra-class variance of the source in terms of pose, expression and lighting, which shows that our feature transfer is effective at enlarging the intra-class variance.
Transfer Fig. 7. Feature transfer visualization between two classes for every two columns. The first row are the input, in which odd column denotes class 1: x1 and the even column denotes class 2: x2. The second row are the reconstructed images x 1 and x 2 . In the third row, odd column image is the decoded image of the transferred feature from class 1 to class 2 and even column image is the decoded image of the transferred feature from class 2 to class 1. It is clear that the transferred features share the same identity as the target class while obtain the source image's non-identity variance including pose, expression, illumination, and etc. Feature Interpolation The interpolation between two facial representations shows the appearance transition from one to the other [23,62]. Let g 1,2 , c 1,2 denote the encoded features and the feature centers of two samples. Previous work generates a new representation as g = g 1 + α(g 2 − g 1 ) where identity and non-identity changes are mixed together. In our work, we can generate a smooth transition of non-identity change as g = c 1 + αQQ T (g 2 − c 2 ) and identity change as g = g 1 + α(c 2 − c 1 ). Fig. 12 shows an interpolation example of a female with left pose and a male with right pose, where the illumination also changes significantly. Traditional interpolation generates undesirable artifacts. However, our method shows smooth transitions, which verifies that the proposed model is effective at disentangling identity and non-identity information.

Conclusions
In this paper, we propose a novel feature transfer approach for deep face recognition that exploits the long-tailed nature of training data. We observe that generic approaches to deep face recognition encounter classifier bias problems due to imbalanced distribution of training data across identities. In particular, uniform sampling of both regular and long-tail classes leads to biased classifier weights, since the frequency of updating them for long-tail classes is much lower. By applying the proposed feature transfer approach, we enrich the feature space of the tail classes, while retaining identity. Utilizing the generated data, our alternating feature learning method rectifies the classifier and learns more compact feature representations. Our proposed m-L 2 regularization demonstrates consistent advantages which can boost performance across different recognition tasks. The disentangled nature of the augmented feature space is visualized through smooth interpolations. Experiments consistently show that our method can learn better representations to improve the performance on regular, long-tail and unseen classes. While this paper focuses on face recognition, our future work will also derive advantages from the proposed feature transfer for other recognition applications, such as long-tail natural species [63]. Table 5 shows the network structures of our framework, which consists of an encoder Enc, a decoder Dec, a feature filtering module R, and a classifier F C. The Enc takes an input image x ∈ R 100×100×3 and generates a feature vector g ∈ R 320×1 . The Dec takes g as input and reconstructs the original input image as x ∈ R 100×100×3 . Module R takes the features g as input to generate a more discriminative representation f ∈ R 320×1 . The F C takes f as input for classification with a linear layer, which is not shown in Table 5. Batch Normalization (BN) and Rectified Linear Units (ReLU) are applied after each convolution (Cv) and full convolution (FCv) layer except "Cv53".

A Network Structures and Parameter Settings
For initialization, the convolution layers are initialized with a uniform distribution [− 3/n, 3/n], n is the number of entries of that layer's weight. The fully connected layers are initialized with a Gaussian distribution N (0, 1e −5 ). The training process is as Algorithm 1 in the main submission. For stage 0, we apply Eqn. 4 in the main paper to pre-train an overall network. Adam solver with batch size of 128 is adopted. The learning rate is set as 2e −4 . The network converges after 20 to 30 epochs. Then, we apply the alternative feature transfer learning scheme on top of the pre-trained model. We use Adam solver with a batch size of 128 and a learning rate of 1e −5 . Stage 1 and 2 are alternated for every 5K iterations. It takes 10 to 15 alternations to converge.
Further, Eqn. 9 shows that the joint regularization is upper-bounded by general L 2 regularization on both W and f . Eqn. 10 shows that the proposed m-L 2 regularization has the same asymptotic error as L 2 regularization.
(10) There is assumption of w j = 1 in Eqn. 10, which is reasonable as one can always set up normalization for the weight. Even without such weight restriction, as shown in the main submission Fig. 4 (b), the weight norm actually varies around scale 1.
The first inequality in Eqn. 10 obviously holds as N c ≥ Nc j=1 |cosθ j | 2 . N c is the number of subjects. To prove the second inequality in Eqn. 10, we have proposition 1.

Proposition 1.
Nc j=1 |cosθ j | 2 ≥ 1 holds when N c ≥ V C(f ), V C(f ) is VCdimension of feature f , θ j is the angle between feature f and the j th weight vector w j . The weight vectors are assumed optimally distributed. As shown in Fig. 9 (a), assuming in 2D space, N c ≥ V C(R 2 ) = 3. An arbitrary feature vector f , denoted as red dash arrow in Fig. 9 (a), must lie in one of those three angular intervals spanned by w 1 , w 2 and w 3 . At least two of the three angles that f spans with those three weight vectors should be less than or equal to π/3. Further, we notice that Nc j=1 |cosθ j | 2 is a convex function when θ ∈ [0, π/2). This constraint is strengthened to [0, π/3] in 2D space with 3 classes. Nc j=1 |cosθ j | 2 = 1 when θ = 0 or θ = π/3. Still in 2D space, when N c increases from 3 to 4 or more, function Nc j=1 |cosθ j | 2 maintains convex as θ ∈ [0, π/2), thus Nc j=1 |cosθ j | 2 ≥ 1 holds. When the feature dimension increases from 2D to 3D as shown in Fig. 9 (b) or higher dimension d, the VC-dimension becomes d + 1. We always have the same condition that function Nc j=1 |cosθ j | 2 remains convex as θ ∈ [0, π/2). Thus, Nc j=1 |cosθ j | 2 ≥ 1 holds. We further empirically analyze the differences between m-L 2 and general L 2 regularization. Assume the training is close to optimal, which satisfies that w T i f is non-zero while w T j f are close to zero if i is the right class label. Independent regularization of f or W will still affect those j classes, which is over-regularization as no error occurs. In contrast, our m-L 2 regularization considers the overall regularization. As all other w T j f close to zero, the regularization will mostly only emphasize on w T i f , which selectively penalizes on the right class for classification error. Nevertheless, we do not explicitly push w i or f to be small, but we push the angle between w i and f to be small, which squeezes the angular distribution more strictly. We consider such angular squeezing as better-suited for the sof tmax setting where the feature distribution is angular.

C Feature Visualization
Center Visualization As a supplementary to our main submission, in Fig. 10, we exhibit further visual examples to show the reconstructed image-level face centers as well to illustrate the relation between feature center and neutral frontal faces. We extract features using Enc within one class and compute its feature center. We apply Dec on the feature center to reconstruct a image-level center face. Further, we find one sample that is closest to the center at the feature level. Each row shows the original image of the closest sample, the reconstructed image from this sample, and the reconstructed image of the center. We observe that the feature centers correspond to frontal neutral faces, which are visually similar The reconstructed image of (a). (c) The reconstructed image from the feature center. We observe that the feature center corresponds to a neutral frontal face, which also shares similarity to the reconstructed image of the closest sample (row (b)).
to the reconstructed images from the closest samples. In some cases, the feature center shows a smiling face, which happens when the majority of the images in this class is smiling. Feature Transfer We perform feature transfer in the feature space g. The transferred features can be visualized using Dec. Let x 1,2 , x 1,2 , g 1,2 , c 1,2 denote the input images, reconstructed images, encoded features, and feature centers of two classes, respectively. Let Q denote the PCA basis of the intra-class variance. We transfer features from class 1 to class 2: g 12 = c 2 + QQ T (g 1 − c 1 ), and visualize the decoded images as x 12 . We also transfer features from class 2 to class 1: g 21 = c 1 + QQ T (g 2 − c 2 ), and visualize the decoded images as x 21 . Fig. 11 shows several examples of feature transfer between every two classes. The proposed feature transfer succeeded in transferring the intra-class variance of the source class to the center of the target class. The visualizations of the transferred features consistently preserve the target class identity by incorporating the source image attributes, that is, pose, expression, lighting condition, hat and so on, which shows that our feature transfer is effective at enlarging the intra-class variance. PCA Basis In our feature transfer framework, we use PCA to capture the intraclass variance. Here we visualize what is being captured in each basis. Specifically, we add one basis to the feature center to generate a new feature representation: g i = c i + Q(:, k) · 0.1, where c i is the center of class i, Q(:, k) is the kth PCA basis, and 0.1 is the mean absolute coefficient of all images when projecting to the top 10 basis. Fig. 13 shows the results of several examples. It is clear that each PCA basis consistently captures a mixture of pose, expression, illumination variations. For example, adding the 1st basis improves the image quality with good lighting condition; adding the 6th basis turns the face to left and makes it smile; adding the 7th basis turns the face downward slightly and opens the mouth, etc. It is critical that the PCA basis captures the various intra-class variations so that the feature transfer is semantically meaningful. This visualization supports that the reconstruction task in our baseline framework encourages the feature space g to capture these variations.  Fig. 11. Feature transfer visualization between two classes for every two columns. The first row are the input, in which odd column denotes class 1: x1 and the even column denotes class 2: x2. The second row are the reconstructed images x 1 and x 2 . In the third row, odd column image is the decoded image of the transferred feature from class 1 to class 2 and even column image is the decoded image of the transferred feature from class 2 to class 1. It is clear that the transferred features share the same identity as the target class while obtain the source image's non-identity variance including pose, expression, illumination, etc.

Feature Interpolation
The interpolation between two face representations help to understand the disentanglement of the identity feature and the nonidentity feature. This visualization of interpolation is widely used in GAN-based frameworks. However, previous works visualize this transition with a mixed change of identity and non-identity variations. In our approach, we model the feature space as a linear combination of feature center and its intra-class variance. Therefore, we can separate the visualization into two parts. Let g 1,2 , c 1,2 denote the encoded features and the feature centers of two samples from different classes respectively. Previous works generate a new representation as g = g 1 + α(g 2 − g 1 ).
In our work, we can generate a smooth transition of non-identity change as g = c 1 + αQQ T (g 2 − c 2 ), which is the same as the proposed feature transfer when α = 1. On the other hand, we can also generate a smooth transition of identity change as g = g 1 + α(c 2 − c 1 ). We vary α from 0.1 to 1 to visualize the transition results.  Table 6. Additional results on the controlled experiments by varying the ratio between regular and long-tail classes in the training sets. Fig. 12 shows the above five transitions on four examples. Traditional method shows the change of identity and non-identity components simultaneously. In our approach, we can visualize two separate transitions for identity and non-identity changes. E.g., example (a) shows the transition from a female with smiling expression to a male with right pose. Second row shows the same identity while the face is turning to the right. Third row shows the same attributes as the left image (pose and expression) while the identity is gradually changed to that of the right image.

D Ablation Study
We propose a two-stage alternative training scheme to correct the classifier bias and learn more discriminative features. In the main manuscript, we have shown sufficient results to support that we have learnt a better feature representation.
Here we present evidence that we have corrected the classifier bias as well.
In the ablation study, we perform classification on the hold-out testing set from MS-celeb-1M. We compare two methods. (1) FC: using the trained F C as the classifier. (2) NN: using the training images to calculate the class feature center as the gallery and Nearest Neighbor for classification. It is observed in [7] that the L 2 norm of the weights for a long-tail class is smaller than that of a regular class, which suggests that the classifier is biased towards regular classes. Therefore, we compare the weight norm of regular and long-tail classes before and after feature transfer learning to quantify the classifier bias.
The results are shown in Table 6. When using F C on the baseline models, the accuracy on the long-tail classes is very low. However when NN is used, the accuracy on the regular classes drops slightly while the accuracy on the long-tail classes improves significantly. This suggests that a long-tail sample that is closer to its center can be mis-classified into a neighboring regular class when using the trained F C. This bias is consistent with the imbalance between the weight norm of regular and long-tail classes. After applying our proposed feature transfer, we observe a consistent improvement when using F C for classification. The imbalance between the weight norm is also reduced to a large extent.  Table 7. Dataset statistics of CASIA-Webface and MS-celeb-1M.

E Data Distribution
Dataset statistics Fig. 14 shows the distribution of two public face benchmarks, CASIA-Webface [16] and the cleaned version of MS-celeb-1M [3]. Considering a class with no more than 20 images as long-tail class, the specific statistics of regular and long-tail classes of these two datasets are shown in Table 7. There is a large portion of long-tail classes for both datasets, i.e., 37% classes is long-tail in CASIA-Webface and 12.4% in MS-celeb-1M. Training with such long-tail data leads to the classifier bias problem and further results in inferior feature representation.
Long-tail distribution The main purpose of the proposed feature transfer is to enrich the biased distribution of a long-tail class. To visualize this effect, we project the features g onto 2D space. Fig. 15 (a) shows one regular class with a balanced distribution. When we randomly select 20 images to form a long-tail class, the distribution is biased, as shown in Fig. 15 (b). After the proposed feature transfer, we observe that the distribution is enriched after transferring 100 and 300 samples from other regular classes, as shown in Fig. 15 (c) and (d) respectively.