Question: represent the text to find a very short summary (1-2 sentences max)!

Question: Knowledge graph embedding research has overlooked the problem of probability calibration. We show popular embedding models are indeed uncalibrated. That means probability estimates associated to predicted triples are unreliable. We present a novel method to calibrate a model when ground truth negatives are not available, which is the usual case in knowledge graphs. We propose to use Platt scaling and isotonic regression alongside our method. Experiments on three datasets with ground truth negatives show our contribution leads to well calibrated models when compared to the gold standard of using negatives. We get significantly better than the uncalibrated models from all calibration methods. We show isotonic regression offers the best the performance overall, not without trade-offs. We also show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds. Knowledge graph embedding models are neural architectures that learn vector representations (i.e. embeddings) of nodes and edges of a knowledge graph. Such knowledge graph embeddings have applications in knowledge graph completion, knowledge discovery, entity resolution, and link-based clustering, just to cite a few (a). Despite burgeoning research, the problem of calibrating such models has been overlooked, and existing knowledge graph embedding models do not offer any guarantee on the probability estimates they assign to predicted facts. Probability calibration is important whenever you need the predictions to make probabilistic sense, i.e., if the model predicts a fact is true with 80% confidence, it should to be correct 80% of the times. Prior art suggests to use a sigmoid layer to turn logits returned by models into probabilities (a) (also called the expit transform), but we show that this provides poor calibration. Figure 1 shows reliability diagrams for off-the-shelf TransE and ComplEx. The identity function represents perfect calibration. Both models are miscalibrated: all TransE combinations in Figure 1a under-forecast the probabilities (i.e. probabilities are too small), whereas ComplEx under-forecasts or over-forecasts according to which loss is used (Figure1b). Calibration is crucial in high-stakes scenarios such as drug-target discovery from biological networks, where end-users need trustworthy and interpretable decisions. Moreover, since probabilities are not calibrated, when classifying triples (i.e. facts) as true or false, users must define relationspecific thresholds, which can be awkward for graphs with a great number of relation types. To the best of our knowledge, this is the first work to focus on calibration for knowledge embeddings. Our contribution is two-fold: First, we use Platt Scaling and isotonic regression to calibrate knowledge graph embedding models on datasets that include ground truth negatives. One peculiar feature of knowledge graphs is that they usually rely on the open world assumption (facts not present are not necessarily false, they are simply unknown). This makes calibration troublesome because of the lack of ground truth negatives. For this reason, our second and main contribution is a calibration heuristics that combines Platt-scaling or isotonic regression with synthetically generated negatives. Experimental show that we obtain better-calibrated models and that it is possible to calibrate knowledge graph embedding models even when ground truth negatives are not present. We also experiment with triple classification, and we show that calibrated models reach state-of-the-art accuracy without the need to define relation-specific decision thresholds. A comprehensive survey of knowledge graph embedding models is out of the scope of this paper. Recent surveys such as (a) and present an overview or recent literature. TransE is the forerunner of distance-based methods, and spun a number of models commonly referred to as TransX. The intuition behind the symmetric bilinear-diagonal model DistMult paved the way for its asymmetric evolutions in the complex space, RotatE and ComplEx (a generalization of which uses hypercomplex representations ). HolE relies instead on circular correlation (b). The recent TorusE operates on a lie group and not in the Euclidean space. While the above models can be interpreted as multilayer perceptrons, others such as ConvE or ConvKB include convolutional layers. More recent works adopt capsule networks architectures . Adversarial learning is used by KBGAN , whereas attention mechanisms are instead used by . Some models such as RESCAL , TuckER (Balažević et al., 2019), and SimplE rely on tensor decomposition techniques. More recently, ANALOGY adopts a differentiable version of analogical reasoning . In this paper we limit our analysis to four popular models: TransE, DistMult, ComplEx and HolE. They do not address the problem of assessing the reliability of predictions, leave aside calibrating probabilities. Besides well-established techniques such as Platt scaling and isotonic regression , recent interest in neural architectures calibration show that modern neural architectures are poorly calibrated and that calibration can be improved with novel methods. For example, successfully proposes to use temperature scaling for calibrating modern neural networks in classification problems. On the same line, proposes a procedure based on Platt scaling to calibrate deep neural networks in regression problems. The Knowledge Vault pipeline in extracts triples from unstructured knowledge and is equipped with Platt scaling calibration, but this is not applied to knowledge graph embedding models. KG2E proposes to use normally-distributed embeddings to account for the uncertainty, but their model does not provide the probability of a triple being true, so KG2E would also benefit from the output calibration we propose here. To the best of our knowledge, the only work that adopts probability calibration to knowledge graph embedding models is Krompaß &. The authors propose to use ensembles in order to improve the of knowledge graph embedding tasks. For that, they propose to calibrate the models with Platt scaling, so they operate on the same scale. No further details on the calibration procedure are provided. Besides, there is no explanation on how to handle the lack of negatives. Knowledge Graph. Formally, a knowledge graph G = {(s, p, o)} ⊆ E × R × E is a set of triples t = (s, p, o), each including a subject s ∈ E, a predicate p ∈ R, and an object o ∈ E. E and R are the sets of all entities and relation types of G. Triple Classification. Binary classification task where G (which includes only positive triples) is used as training set, and T = {(s, p, o)} ⊆ E × R × E is a disjoint test set of labeled triples to classify. Note T includes positives and negatives. Since the learned models are not calibrated, multiple decision thresholds τ i must be picked, where 0 < i < |R|, i.e. one for each relation type. This is done using a validation set . Classification metrics apply (e.g. accuracy). Link Prediction. Given a training set G that includes only positive triples, the goal is assigning a score f (t) ∈ R proportional to the likelihood that each unlabeled triple t included in a held-out set S is true. Note S does not have ground truth positives or negatives. This task is cast as a learning to rank problem, and uses metrics such as mean rank (MR), mean reciprocal rank (MRR) or Hits@N. Knowledge Graph Embeddings. Knowledge graph embedding models are neural architectures that encode concepts from a knowledge graph G (i.e. entities E and relation types R) into lowdimensional, continuous vectors ∈ R k (i.e, the embeddings). Embeddings are learned by training a neural architecture over G. Although such architectures vary, the training phase always consists in minimizing a loss function L that includes a scoring function f m (t), i.e. a model-specific function that assigns a score to a triple t = (s, p, o) (more precisely, the input of f m are the embeddings of the subject e s, the predicate r p, and the object e o). The goal of the optimization procedure is learning optimal embeddings, such that the scoring function f m assigns high scores to positive triples t + and low scores to triples unlikely to be true t −. Existing models propose scoring functions that combine the embeddings e s, r p, e o ∈ R k using different intuitions. Table 1b lists the scoring functions of the most common models. For example, the scoring function of TransE computes a similarity between the embedding of the subject e s translated by the embedding of the predicate e p and the embedding of the object e o, using the L 1 or L 2 norm || · ||. Such scoring function is then used on positive and negative triples t + ∈ G, t − ∈ N in the loss function. This is usually a pairwise margin-based loss , negative log-likelihood, or multi-class log-likelihood . Since the training set usually includes positive statements, we generate synthetic negatives t − ∈ N required for training. We do so by corrupting one side of the triple at a time (i.e. either the subject or the object), following the protocol proposed by . Calibration. Given a knowledge graph embedding model identified by its scoring function f m, with f m (t) =p, wherep is the estimated confidence level that a triple t = (s, p, o) is true, we define f m to be calibrated ifp represents a true probability. For example, if f m (·) predicts 100 triples all with confidencep = 0.7, we expect exactly 70 to be actually true. Calibrating a model requires reliable metrics to detect miscalibration, and effective techniques to fix such distortion. Appendix A.1 includes definitions and on the calibration metrics adopted in the paper. We propose two scenario-dependent calibration techniques: we first address the case with ground truth negatives t − ∈ N. The second deals with the absence of ground truth negatives. Calibration with Ground Truth Negatives. We propose to use off-the-shelf Platt scaling and isotonic regression, techniques proved to be effective in literature. It is worth reiterating that to calibrate a model negative triples N are required from a held-out dataset (which could be the validation set). Such negatives are usually available in triple classification datasets (FB13, WN11, YAGO39K) Calibration with Synthetic Negatives. Our main contribution is for the case where no ground truth negatives are provided at all, which is in fact the usual scenario for link prediction tasks. We propose to adopt Platt scaling or isotonic regression and to synthetically generate corrupted triples as negatives, while using sample weights to guarantee that the frequencies adhere to the base rate of the population (which is problem-dependent and must be user-specified). It is worth noting that it is not possible to calibrate a model without implicit or explicit base rate. If it is not implicit on the dataset (the ratio of positives to totals), it must be explicitly provided. We generate synthetic negatives N following the standard protocol proposed by 1: for every positive triple t = (s, p, o), we corrupt one side of the triple at a time (i.e. either the subject s or the object o) by replacing it with other entities in E. The number of corruptions generated per positive is defined by the user-defined corruption rate η ∈ N. Since the number of negatives N = |N | can be much greater than the number of positive triples P = |G|, when dealing with calibration with synthetically generated corruptions, we weigh the positive and negative triples to make the calibrated model match the population base rate α = P/(P + N) ∈, otherwise the base rate would depend on the arbitrary choice of η. Given a positive base rate α, we propose the following weighting scheme: where ω + ∈ R is the weight associated to the positive triples and ω − ∈ R to the negatives. The ω + weight removes the imbalance determined by having a higher number of corruptions than positive triples in each batch. The ω − weight guarantees that the given positive base rate α is respected. The above can be verified as follows. For the unweighted problem, the positive base rate is simply the ratio of positive examples to the total number of examples: If we add uniform weights to each class, we have: By defining ω + = η, i.e. adopting the ratio of negatives to positives (corruption rate), we then have: Thus, the negative weights is: We compute the calibration quality of our heuristics, showing that we achieve calibrated predictions even when ground truth negative triples are not available. We then show the impact of calibrated predictions on the task of triple classification. Datasets. We run experiments on triple classification datasets that include ground truth negatives (Table 1). We train on the training set, calibrate on the validation set, and evaluate on the test set. • WN11 . A subset of Wordnet , it includes a large number of hyponym and hypernym relations thus including hierarchical structures. • FB13 . A subset of Freebase , it includes facts on famous people (place of birth and/or death, profession, nationality, etc). • YAGO39K . This recently released dataset has been carved out of YAGO3 , and includes a mixture of facts about famous people, events, places, and sports teams. We also use two standard link prediction benchmark datasets, WN18RR (a subset of Wordnet) and FB15K-237 (a subset of Freebase). Their test sets do not include ground truth negatives. Implementation Details. The knowledge graph embedding models are implemented with the AmpliGraph library version 1.1, using TensorFlow 1.13 and Python 3.6 on the backend 2. All experiments were run under Ubuntu 16.04 on an Intel Xeon Gold 6142, 64 GB, equipped with a Tesla V100 16GB. Hyperparameter Tuning. For each dataset in Table 1a, we train a TransE, DistMult, and a ComplEx knowledge graph embedding model. We rely on typical hyperparameter values: we train the embeddings with dimensionality k = 100, Adam optimizer, initial learning rate α 0 = 1e-4, negatives per positive ratio η = 20, epochs = 1000. We train all models on four different loss functions: Selfadversarial , pairwise , NLL, and Multiclass-NLL . Different losses are used in different experiments. Calibration Success. Table 2 reports Brier scores and log losses for all our calibration methods, grouped by the type of negative triples they deal with (ground truth or synthetic). All calibration methods show better-calibrated than the uncalibrated case, by a considerable margin and for all datasets. In particular, to put the of the synthetic strategy in perspective, if we suppose to predict the positive base rate as a baseline, for each of the cases in Table 2 (the three datasets share the same positive base rate α = 0.5), we would get Brier score B = 0.25 and log loss L log = 0.69, that are always worse than our methods. There is considerable variance of between models given a dataset, which also happens when varying losses given a particular combination of model and dataset (Table 3). TransE provides the best for WN11 and FB13, while DistMult works best for YAGO39K. We later propose that this variance comes from the quality of the embeddings themselves, that is, better embeddings allow for better calibration. In Figure 2, we also evaluate just the frequencies themselves, ignoring sharpness (i.e. whether probabilities are close to 0 or 1), using reliability diagrams for a single model-loss combination, for all datasets (ComplEx+NLL). Calibration plots show a remarkable difference between the uncalibrated baseline (s-shaped blue line on the left-hand side) and all calibrated models (curves closer to the identity function are better). A visual comparison of uncalibrated curves in Figure 1 with those in Figure 2 also gives a sense of the effectiveness of calibration. Ground Truth vs Synthetic. As expected, the ground truth method generally performs better than the synthetic calibration, since it has more data in both quantity (twice as much) and quality (two classes instead of one). Even so, the synthetic method is much closer to the ground truth than to the uncalibrated scores, as highlighted by the calibration plots in Figure 2. For WN11, it is actually as good as the calibration with the ground truth. This shows that our proposed method works as intended and could be used in situations where we do not have access to the ground truth, as is the case for most knowledge graph datasets. Isotonic vs Platt. Isotonic regression performs better than Platt scaling in general, but in practice Isotonic regression has the disadvantage of not being a convex or differentiable algorithm. This is particularly problematic for the synthetic calibration, as it requires the generation of the synthetic corruptions, which can only be made to scale via a mini-batch based optimization procedure. Platt scaling, given that it is a convex and differentiable loss, can be made Table 2: Calibration test (self-adversarial loss ). Low score = better. Best in bold for each combination of dataset and metric. part of a computational graph and optimized with mini-batches, thus it can rely on the modern computational infrastructure designed to train deep neural networks. Influence of Loss Function. We experiment with different losses, to assess how calibration affects each of them (Table 3). We choose to work with TransE, which is reported as a strong baseline in (tion methods, across all datasets. Experiments also show the choice of the loss has a big impact, greater than the choice of calibration method or embedding model. We assess whether such variability is determined by the quality of the embeddings. To verify whether better embeddings lead to sharper calibration, we report the mean reciprocal rank (MRR), which, for each true test triple, computes the (inverse) rank of the triple against synthetic corruptions, then averages the inverse rank (Table 3). In fact, we notice no correlation between calibration and MRR. In other words, embeddings that lead to the best predictive power are not necessary the best calibrated. Positive Base Rate. We apply our synthetic calibration method to two link prediction benchmark datasets, FB15K-237 and WN18RR. As they only provide positive examples, we apply our method with varying base rates α i, linearly spaced from 0.05 to 0.95. We evaluate relying on the closed-world assumption, i.e. triples not present in training, validation or test sets are considered negative. For each α i we calibrate the model using the synthetic method with both isotonic regression and Platt scaling. We sample negatives from the negative set under the implied negative rate, and calculate a baseline which is simply having all probability predictions equal to α i. Figure 3 shows that isotonic regression and Platt scaling perform similarly and always considerably below the baseline. As expected from the previous , the uncalibrated scores perform poorly, only reaching acceptable levels around some particular base rates. Triple Classification and Decision Threshold. To overcome the need to learn |R| decision thresholds τ i from the validation set, we propose to rely on calibrated probabilities, and use the natural threshold of τ = 0.5. Table 4 shows how calibration affects the triple classification task, comparing with the literature standard of per-relation thresholds (last column). For simplicity, note we use the same self-adversarial loss in Table 2 and Table 4. We learn thresholds τ i on validation sets, ing in 11, 7, and 33 thresholds for WN11, FB13 and YAGO39K respectively. Using a single τ = 0.5 and calibration provides competitive compared to multiple learned thresholds (note uncalibrated with τ = 0.5 are poor, as expected). It is worth mentioning that Figure 3: Synthetic calibration on FB15K-237 and WN18RR, with varying positive base rates. The baseline stands for using the positive base rate as the probability prediction. Results are evaluated under the closed-world assumption, using the same positive base rate used to calibrate the models. Ground Truth (τ = .5) Table 4: Effect of calibration on triple classification accuracy. Best in bold. For all calibration methods there is one single threshold, τ = 0.5. For the per-relation τ, we learned multiple thresholds from validation sets (Appendix A.5). We did not carry out additional model selection, and used Table 2 hyperparameters instead. Isotonic regression reaches state-of-the-art for WN11. Results of * from ; from ; † from . we are at par with state-of-the-art for WN11. Isotonic regression is again the best method, but there is more variance in the model choice. Our proposed calibration method with synthetic negatives performs well overall, even though calibration is performed only using half of the validation set (negatives examples are replaced by synthetic negatives). We propose a method to calibrate knowledge graph embedding models. We target datasets with and without ground truth negatives. We experiment on triple classification datasets and apply Platt scaling and isotonic regression with and without synthetic negatives controlled by our heuristics. All calibration methods perform significantly better than uncalibrated scores. We show that isotonic regression brings better calibration performance, but it is computationally more expensive. Additional experiments on triple classification shows that calibration allows to use a single decision threshold, reaching state-of-the-art without the need to learn per-relation thresholds. Future work will evaluate additional calibration algorithms, such as beta calibration or Bayesian binning . We will also experiment on ensembling of knowledge graph embedding models, inspired by (Krompaß &). The rationale is that different models operate on different scales, but calibrating brings them all to the same probability scale, so their output can be easily combined. Reliability Diagram . Also known as calibration plot, this diagram is a visual depiction of the calibration of a model (see Figure 1 for an example). It shows the expected sample accuracy as a function of the estimated confidence. A hypothetical perfectly calibrated model is represented by the diagonal line (i.e. the identity function). Divergence from such diagonal indicates calibration issues . Brier Score . It is a popular metric used to measure how well a binary classifier is calibrated. It is defined as the mean squared error between n probability estimatesp and the corresponding actual outcomes y ∈ 0, 1. The smaller the Brier score, the better calibrated is the model. Note that the Brier score B ∈. Log Loss is another effective and popular metric to measure the reliability of the probabilities returned by a classifier. The logarithmic loss measures the relative uncertainty between the probability estimates produced by the model and the corresponding true labels. Platt Scaling. Proposed by for support vector machines, Platt scaling is a popular parametric calibration techniques for binary classifiers. The method consists in fitting a logistic regression model to the scores returned by a binary classifier, such thatq = σ(ap + b), wherê p ∈ R is the uncalibrated score of the classifier, a, b ∈ R are trained scalar weights. andq is the calibrated probability returned as output. Such model can be trained be trained by optimizing the NLL loss with non-binary targets derived by the Bayes rule under an uninformative prior, ing in an Maximum a Posteriori estimate. Isotonic Regression . This popular non-parametric calibration techniques consists in fitting a non-decreasing piecewise constant function to the output of an uncalibrated classifier. As for Platt scaling, the goal is learning a functionq = g(p), such thatq is a calibrated probability. Isotonic regression learns g by minimizing the square loss n i=1 (q i − y i) 2 under the constraint that g must be piecewise constant . We present in Figure 4 the total count of instances for each bin used in the calibration plots included in Figure 2. As expected, calibration considerably helps spreading out instances across bins, whereas in uncalibrated scenarios instances are squeezed in the first or last bins. A.3 IMPACT OF MODEL HYPERPARAMETERS: η AND EMBEDDING DIMENSIONALITY In Figure 5 we report the impact of negative/positive ratio η and the embedding dimensionality k. Results show that the embedding size k has higher impact than the negative/positive ratio η. We observe that calibrated and uncalibrated low-dimensional embeddings have worse Brier score. Results also show that any k > 50 does not improve calibration anymore. The negative/positive ratio η follows a similar pattern: choosing η > 10 does not have any effect on the calibration score. In Table 5, we present the traditional knowledge graph embedding rank metrics: MRR (mean reciprocal rank), MR (mean rank) and Hits@10 (precision at the top-10 ). We report the for all datasets and models used in the main text, which appear in Table 2, Table 4 Table 5: Standard filtered metrics for knowledge graph embeddings models. The models are implemented in the same codebase and share the same evaluation protocol. Note that we do not include from reciprocal evaluation protocols. We report in Table 6 the per-relation decision thresholds τ used in Table 4, under the'Reproduced' column. Note that the thresholds reported here are not probabilities, as they have been applied to the raw scores returned by the model-dependent scoring function f m (t). Table 6: Relation-specific decision thresholds learned on uncalibrated raw scores (See also Table 4 for a report on triple classification .)

We propose a novel method to calibrate knowledge graph embedding models without the need of negative examples.

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the scientific article to find a short TLDR

Question: As an emerging topic in face recognition, designing margin-based loss functions can increase the feature margin between different classes for enhanced discriminability. More recently, absorbing the idea of mining-based strategies is adopted to emphasize the misclassified samples and achieve promising . However, during the entire training process, the prior methods either do not explicitly emphasize the sample based on its importance that renders the hard samples not fully exploited or explicitly emphasize the effects of semi-hard/hard samples even at the early training stage that may lead to convergence issues. In this work, we propose a novel Adaptive Curriculum Learning loss (CurricularFace) that embeds the idea of curriculum learning into the loss function to achieve a novel training strategy for deep face recognition, which mainly addresses easy samples in the early training stage and hard ones in the later stage. Specifically, our CurricularFace adaptively adjusts the relative importance of easy and hard samples during different training stages. In each stage, different samples are assigned with different importance according to their corresponding difficultness. Extensive experimental on popular benchmarks demonstrate the superiority of our CurricularFace over the state-of-the-art competitors. Code will be available upon publication. The success of Convolutional Neural Networks (CNNs) on face recognition can be mainly credited to: enormous training data, network architectures, and loss functions. Recently, designing appropriate loss functions that enhance discriminative power is pivotal for training deep face CNNs. Current state-of-the-art face recognition methods mainly adopt softmax-based classification loss. Since the learned features with the original softmax is not discriminative enough for the open-set face recognition problem, several margin-based variants have been proposed to enhance features' discriminative power. For example, explicit margin, i.e., CosFace (a), Sphereface , ArcFace , and implicit margin, i.e., Adacos (a), supplement the original softmax function to enforce greater intra-class compactness and inter-class discrepancy, which are shown to in more discriminate features. However, these margin-based loss functions do not explicitly emphasize each sample according to its importance. As demonstrated in , hard sample mining is also a critical step to further improve the final accuracy. Recently, Triplet loss and SV-Arc-Softmax (b) integrate the motivations of both margin and mining into one framework for deep face recognition. Triplet loss adopts a semi-hard mining strategy to obtain semi-hard triplets and enlarge the margin between triplet samples. SV-Arc-Softmax (b) clearly defines hard samples as misclassified samples and emphasizes them by increasing the weights of their negative cosine similarities with a preset constant. In a nutshell, mining-based loss functions explicitly emphasize the effects of semi-hard or hard samples. However, there are drawbacks in training strategies of both margin-and mining-based loss functions. For margin-based methods, mining strategy is ignored and thus the difficultness of each sample is not fully exploited, which may lead to convergence issues when using a large margin on small backbones, e.g., MobileFaceNet . As shown in Fig. 1, the modulation coefficient for the negative cosine similarities I(·) is fixed as a constant 1 in ArcFace for all samples during the entire training process. For mining-based methods, over-emphasizing hard samples in early training Figure 1: Different training strategies for modulating negative cosine similarities of hard samples (i.e., the mis-classified sample) in ArcFace, SV-Arc-Softmax and our CurricularFace. Left: The modulation coefficients I(t, cos θj) for negative cosine similarities of hard samples in different methods, where t is an adaptively estimated parameter and θj denotes the angle between the hard sample and the non-ground truth j-class center. Right: The corresponding hard samples' negative cosine similarities N (t, cos θj) = I(t, cos θj) cos θj + c after modulation, where c indicates a constant. On one hand, during early training stage (e.g., t is close to 0), hard sample's negative cosine similarities is usually reduced and thus leads to smaller hard sample loss than the original one. Therefore, easier samples are relatively emphasized; during later training stage (e.g., t is close to 1), the hard sample's negative cosine similarities are enhanced and thus leads to larger hard sample loss. On the other hand, in the same training stage, we modulate the hard samples' negative cosine similarities with cos θj. Specifically, the smaller the angle θj is, the larger the modulation coefficient should be. stage may hinder the model to converge. As SV-Arc-Softmax claimed, the manually defined constant t plays a key role in the model convergence property and a slight larger value (e.g., >1.4) may cause the model difficult to converge. Thus t needs to be carefully tuned. In this work, we propose a novel adaptive curriculum learning loss, termed CurricularFace, to achieve a novel training strategy for deep face recognition. Motivated by the nature of human learning that easy cases are learned first and then come the hard ones , our CurricularFace incorporates the idea of Curriculum Learning (CL) into face recognition in an adaptive manner, which differs from the traditional CL in two aspects. First, the curriculum construction is adaptive. In traditional CL, the samples are ordered by the corresponding difficultness, which are often defined by a prior and then fixed to establish the curriculum. In CurricularFace, the samples are randomly selected in each mini-batch, while the curriculum is established adaptively via mining the hard samples online, which shows the diversity in samples with different importance. Second, the importance of hard samples are adaptive. On one hand, the relative importance between easy and hard samples is dynamic and could be adjusted in different training stages. On the other hand, the importance of each hard sample in current mini-batch depends on its own difficultness. Specifically, the mis-classified samples in mini-batch are chosen as hard samples and weighted by adjusting the modulation coefficients I(t, cosθ j) of cosine similarities between the sample and the non-ground truth class center vectors, i.e., negative cosine similarity N (t, cosθ j). To achieve the goal of adaptive curricular learning in the entire training, we design a novel coefficient function I(·) that is determined by two factors: 1) the adaptively estimated parameter t that utilizes moving average of positive cosine similarities between samples and the corresponding ground-truth class center to unleash the burden of manually tuning; and 2) the angle θ j that defines the difficultness of hard samples to achieve adaptive assignment. To sum up, the contributions of this work are: • We propose an adaptive curriculum learning loss for face recognition, which automatically emphasizes easy samples first and hard samples later. To the best of our knowledge, it is the first work to introduce the idea of adaptive curriculum learning for face recognition. • We design a novel modulation coefficient function I(·) to achieve adaptive curriculum learning during training, which connects positive and negative cosine similarity simultaneously without the need of manually tuning any additional hyper-parameter. • We conduct extensive experiments on popular facial benchmarks, which demonstrate the superiority of our CurricularFace over the state-of-the-art competitors. Margin-based loss function Loss design is pivotal for large-scale face recognition. Current stateof-the-art deep face recognition methods mostly adopt softmax-based classification loss. Since the learned features with the original softmax loss are not guaranteed to be discriminative enough for open-set face recognition problem, margin-based losses (; ;) are proposed. Though the margin-based loss functions are verified to obtain good performance, they do not take the difficultness of each sample into consideration, while our CurricularFace emphasizes easy samples first and hard samples later, which is more reasonable and effectiveness. Mining-based loss function Though some mining-based loss function such as Focal loss, Online Hard Sample Mining (OHEM) are prevalent in the field of object detection, they are rarely used in face recognition. OHEM focuses on the large-loss samples in one mini-batch, in which the percentage of the hard samples is empirically determined and easy samples are completely discarded. Focal loss is a soft mining variant that rectifies the loss function to an elaborately designed form, where two hyper-parameters should be tuned with a lot of efforts to decide the weights of each samples and hard samples are emphasized by reducing the weight of easy samples. The recent work, SV-Arc-Softmax (b) fuses the motivations of both margin and mining into one framework for deep face recognition. They define hard samples as misclassified samples and enlarge the weight of hard samples with a preset constant. Our method differs from SV-Arc-Softmax in three aspects: 1) We do not always emphasize the hard samples, especially in the early training stages. 2) We assign different weights for hard samples according to their corresponding difficultness. 3) There's no need in our method to manually tune the additional hyper-parameter t, which is estimated adaptively. Curriculum Learning Learning from easier samples first and harder samples later is a common strategy in Curriculum Learning (CL) , . The key problem in CL is to define the difficultness of each sample. For example, takes the negative distance to the boundary as the indicator for easiness in classification. However, the ad-hoc curriculum design in CL turns out to be difficult to implement in different problems. To alleviate this issue, designs a new formulation, called Self-Paced Learning (SPL), where examples with lower losses are considered to be easier and emphasized during training. The key differences between our CurricularFace with SPL are: 1) Our method focuses on easier samples in the early training stage and emphasizes hard samples in the later training stage. 2) Our method proposes a novel modulation function N (·) for negative cosine similarities, which achieves not only adaptive assignment on modulation coefficients I(·) for different samples in the same training stage, but also adaptive curriculum learning strategy in different training stages. The original softmax loss is formulated as follows: where x i ∈ R d denotes the deep feature of i-th sample which belongs to the y i class, denotes the j-th column of the weight W ∈ R d×n and b j is the bias term. The class number and the embedding feature size are n and d, respectively. In practice, the bias is usually set to b j = 0 and the individual weight is set to ||W j || = 1 by l 2 normalization. The deep feature is also normalized and re-scaled to s. Thus, the original softmax can be modified as follows: Since the learned features with original softmax loss may not be discriminative enough for open-set face recognition problem, several variants are proposed and can be formulated in a general form: Under review as a conference paper at ICLR 2020 sT (cos θy i)+ n j=1,j =y i e sN (t,cos θ j) is the predicted ground truth probability and G(p(x i)) is an indicator function. T (cos θ yi) and N (t, cos θ j) = I(t, cos θ j) cos θ j + c are the functions to modulate the positive and negative cosine similarities, respectively, where c is a constant, and I(t, cos θ j) denotes the modulation coefficients of negative cosine similarities. In margin-based loss function, e.g, ArcFace, G(p(x i)) = 1, T (cos θ yi) = cos(θ yi + m), and N (t, cos θ j) = cos θ j. It only modifies the positive cosine similarity of each sample to enhance the feature discrimination. As shown in Fig. 1, the modulation coefficients of each sample' negative cosine similarity I(·) is fixed as 1. The recent work, SV-Arc-Softmax emphasizes hard samples by increasing I(t, cos θ j) for hard samples. That is, G(p(x i)) = 1 and N (t, cos θj) is formulated as follows: If a sample is defined to be easy, its negative cosine similarity is kept the same as the original one, cos θ j; if as a hard sample, its negative cosine similarity becomes t cos θ j + t − 1. That is, as shown in Fig. 1, I(·) is a constant and determined by a preset hyper-parameter t. Meanwhile, since t is always larger than 1, t cos θ j + t − 1 > cos θ j always holds true, which means the model always focuses on hard samples, even in the early training stage. However, the parameter t is sensitive that a large pre-defined value (e.g., > 1.4) may lead to convergence issue. Next, we present the details of our proposed adaptive curriculum learning loss, which is the first attempt to introduce adaptive curriculum learning into deep face recognition. The formulation of our loss function is also contained in the general form, where G(p(x i)) = 1, positive and negative cosine similarity functions are defined as follows: N (t, cos θ j) = cos θj, T (cos θy i) − cos θj ≥ 0 cos θj(t + cos θj), T (cos θy i) − cos θj < 0. It should be noted that the positive cosine similarity can adopt any margin-based loss functions and here we adopt ArcFace as the example. As shown in Fig. 1, the modulation coefficient of hard sample negative cosine similarity I(t, θ j) depends on both the value of t and θ j. In the early training stage, learning from easy samples is beneficial to model convergence. Thus, t should be close to zero and I(·) is smaller than 1. Therefore, the weights of hard samples are reduced and the easy samples are emphasized relatively. As training goes on, the model gradually focuses on the hard samples, i.e., the value of t shall increase and I(·) is larger than 1. Then, the weights of hard samples are enlarged, which are thus emphasized. Moreover, within the same training stage, I(·) is monotonically decreasing with θ j so that harder sample can be assigned with larger coefficient according to its difficultness. The value of the parameter t is automatically estimated in our CurricularFace, otherwise it would require a lot of efforts for manually tuning. Adaptive estimation of t It is critical to determine appropriate values of t in different training stages. Ideally the value of t can indicate the model training process. We empirically find the average of positive cosine similarities is a good indicator. However, mini-batch statistic-based methods usually face an issue: when many extreme data are sampled in one mini-batch, the statistics can be vastly noisy and the estimation will be unstable. Exponential Moving Average (EMA) is a common solution to address this issue. Specifically, let r (k) be the average of the positive cosine similarities of the k-th batch and be formulated as r (k) = i cos θ yi, we have: where t 0 = 0, α is the momentum parameter and set to 0.99. As shown in Fig. 2, the parameter t increases with the model training, thus the gradient modulation coefficients' range of hard sample, M (·) = 2 cos θ j + t, also increases. Therefore, hard samples are emphasized gradually. With the EMA, we avoid the hyper-parameter tuning and make the modulation coefficients of hard sample Input: The deep feature of i-th sample xi with its corresponding label yi, last fully-connected layer parameters W, cosine similarity cos θj between two vectors, embedding network parameters Θ, learning rate λ, number of iteration k, parameter t, and margin m k ← 0, t ← 0, m ← 0.5; while not converged do k ← k + 1; if cos(θy i + m) > cos θj then N (t, cos θj) = cos θj; else N (t, cos θj) = (t (k) + cos θj) cos θj; end T (cos θy i) = cos(θy i + m); Compute the loss L by Eq. 8; Compute the back-propagation error of xi and Wj by Eq. 9; Update the parameters W and Θ by: ∂Θ (k); Update the parameter t by Eq. 7; end Output: W, Θ negative cosine similarities I(·) adaptive to the current training stage. To sum up, the loss function of our CurricularFace is formulated as follows: where N (t (k), cos θ j ) is defined in Eq. 6. The entire training process is summarized in Algorithm 1. Figure 2: Illustrations on the adaptive parameter t (red line) and gradient modulation coefficients M (·) = 2 cos θj + t of hard samples (green area). Since the number of mined hard samples reduces with the model training, the green area M (·) is relatively smooth in early stage and there are some burrs in later stage. Fig. 3 illustrates how the loss changes from ArcFace to our CurricularFace during training. Here are some observations: 1) As we excepted, hard samples are suppressed in early training stage but emphasized later. 2) The ratio is monotonically increasing with cosθ j, since the larger cosθ j is, the harder the sample is. 3) The positive cosine similarity of a perceptualwell image is often large. However, during the early training stage, the negative cosine similarities of the perceptual-well image may also be large so that it could be classified as the hard one. Optimization Next, we show our CurricularFace can be easily optimized by the conventional stochastic gradient descent. Assuming x i denotes the deep feature of i-th sample which belongs to the y i class, the input of the proposed function is the logit f j, where j denotes the j-th class. In the forwarding process, when j = y i, it is the same as the ArcFace, i.e., f j = sT (cos θ yi), T (cos θ yi) = cos(θ yi + m). When j = y i, it has two cases, if x i is an easy sample, it is the the same as the original softmax, i.e., f j = s cos θ j. Otherwise, it will be modulated as f j = sN (t, cos θ j), where N (t, cos θ j) = (t + cos θ j) cos θ j. In the backward propagation process, the gradient of x i and W j can also be divided into three cases and formulated as follows: Based on the above formulations, we can find the gradient magnitude of the hard sample is determined by two parts, the negative cosine similarity N (·) and the value of t. Softmax cos θy i = cos θj SphereFace cos(mθy i) = cos θj CosFace cos θy i − m = cos θj ArcFace cos(θy i + m) = cos θj SV-Arc-Softmax cos(θy i + m) = cos θj (easy) cos(θy i + m) = t cos θj + t − 1 (hard) CurricularFace (Ours) cos(θy i + m) = cos θj (easy) cos(θy i + m) = (t + cos θj) cos θj (hard) Comparison with ArcFace and SV-Arc-Softmax We first discuss the difference between our CurricularFace and the two competitors, ArcFace and SV-Arc-Softmax, from the perspective of the decision boundary in Tab. 1. ArcFace introduces a margin function T (cos θ yi) = cos(θ yi + m) from the perspective of positive cosine similarity. As shown in Fig. 4, its decision condition changes from cos θ yi = cos θ j (i.e., blue line) to cos(θ yi + m) = cos θ j (i.e., red line) for each sample. SV-Arc-Softmax introduces additional margin from the perspective of negative cosine similarity for hard samples, and the decision boundary becomes cos(θ yi + m) = t cos θ j + t − 1 (i.e., green line). Conversely, we adaptively adjust the weights of hard samples in different training stages. The decision condition becomes cos(θ yi +m) = (t+cos θ j) cos θ j (i.e., purple line). During the training stage, the decision boundary for hard samples changes from one purple line (early stage) to another (later stage), which emphasizes easy samples first and hard samples later. Comparison with Focal loss Focal loss is a soft mining-based loss, which is formulated as: β, where α and β are modulating factors that need to be tuned manually. The definition of hard samples in Focal loss is ambiguous, since it always focuses on relatively hard samples by reducing the weight of easier samples during the entire training process. In contrast, the definition of hard samples in our CurricularFace is more clear, i.e., mis-classified samples. Meanwhile, the weights of hard samples are adaptively determined in different training stages. Datasets We separately employ CASIA-WebFace and refined MS1MV2 as our training data for fair comparisons with other methods. We extensively test our method on several popular benchmarks, including LFW , CFP-FP , CPLFW , AgeDB , CALFW , IJB-B , IJB-C , and MegaFace . Training Setting We follow to generate the normalised faces (112 × 112) with five landmarks. For the embedding network, we adopt ResNet50 and ResNet100 as in. Our framework is implemented in Pytorch . We train models on 4 NVIDIA Tesla P40 (24GB) GPU with batch size 512. The models are trained with SGD algorithm, with momentum 0.9 and weight decay 5e − 4. On CASIA-WebFace, the learning rate starts from 0.1 and is divided by 10 at 28, 38, 46 epochs. The training process is finished at 50 epochs. On MS1MV2, we divide the learning rate at 10, 18, 22 epochs and finish at 24 epochs. We follow the common setting as to set scale s = 64 and margin m = 0.5, respectively. Last but not least, since we only modify the loss function but use the same backbone as previous methods (e.g., ArcFace), NO additional time complexity is introduced for inference. Effects on Fixed vs. Adaptive Parameter t We first investigate the effect of adaptive estimation of t. We choose four fixed values between 0 and 1 for comparison. Specifically, 0 means the modulation coefficient I(·) of each hard sample's negative cosine similarity is always reduced based on its difficultness. In contrast, 1 means the hard samples are always emphasized. 0.3 and 0.7 are between the two cases. Tab. 2 shows that it is more effective to learn from easier samples first and hard samples later based on our adaptively estimated parameter t. Effects on Different Statistics for Estimating t We now investigate the effects of several other statistics, i.e., mode of positive cosine similarities in a mini-batch, or mean of the predicted ground truth probability for estimating t in our loss. As Tab. 3 shows, on one hand, the mean of positive cosine similarities is better than the mode. On the other hand, the positive cosine similarity is more accurate than the predicted ground truth probability to indicate the training stages. As claimed in , ArcFace exists divergence issue when using small backbones like MobileFaceNet. As the , softmax loss must be incorporated for pre-training. To illustrate the robustness of our loss function on convergence issue with small backbone, we use the MobileFaceNet as the network architecture and train it on CASIA-WebFace. As shown in Fig. 5, when the margin m is set to 0.5, the model trained with our loss achieves 99.25 accuracy on LFW, while the model trained with ArcFace does not converge and the loss is NAN at about 2, 400-th step. When the margin m is set to 0.45, both losses can converge, but our loss achieves better performance (99.20% vs. 99.10%). Comparing the yellow and red curves, since the losses of hard samples are reduced in early training stages, our loss converges much faster in the beginning, leading to lower loss than ArcFace. Later on, the value of our loss is slightly larger than ArcFace, because we emphasize the hard samples in later stages. The prove that learning from easy samples first and hard samples later is beneficial to model convergence. Results on LFW, CFP-FP, CPLFW, AgeDB and CALFW Next, we train our CurricularFace on dataset MS1MV2 with ResNet100, and compare with the SOTA competitors on various benchmarks, including LFW for unconstrained face verification, CFP-FP and CPLFW for large pose variations, AgeDB and CALFW for age variations. As reported in Tab. 4, our CurricularFace achieves comparable (i.e., 99.80%) with the competitors on LFW where the performance is near saturated. While for both CFP-FP and CPLFW, our method shows superiority over the baselines including general methods, e.g.,, (b), and cross-pose methods, e.g., , , (a) and. As a recent face recognition method, SV-Arc-Softmax achieves better performance than ArcFace, but still worse than Our CurricularFace. Finally, for AgeDB and CALFW, as Tab. 4 shows, our CurricularFace again achieves the best performance than all of the other state-of-the-art methods. Results on IJB-B and IJB-C The IJB-B dataset contains 1, 845 subjects with 21.8K still images and 55K frames from 7, 011 videos. In the 1:1 verification, there are 10, 270 positive matches and 8M negative matches. The IJB-C dataset is a further extension of IJB-B, which contains about 3, 500 identities with a total of 31, 334 images and 117, 542 unconstrained video frames. In the 1:1 verification, there are 19, 557 positive matches and 15, 638, 932 negative matches. On IJB-B and IJB-C datasets, we employ MS1MV2 and the ResNet100 for a fair comparison with recent methods. We follow the testing protocol in ArcFace and take the average of the image features as the corresponding template representation without bells and whistles. Tab. 5 exhibits the performance of different methods, e.g., Multicolumn, DCN, Adacos (a), P2SGrad (b), PFE and SV-Arc-Softmax (b) on IJB-B and IJB-C 1:1 verification, our method again achieves the best performance. Results on MegaFace Finally, we evaluate the performance on the MegaFace Challenge. The gallery set of MegaFace includes 1M images of 690K subjects, and the probe set includes 100K photos of 530 unique individuals from FaceScrub. We report the two testing under two protocols (large or small training set). Here, we use CASIA-WebFace and MS1MV2 under the small protocol and large protocol, respectively. In Tab. 6, our method achieves the best singlemodel identification and verification performance under both protocols, surpassing the recent strong competitors, e.g., CosFace, ArcFace, Adacos, P2SGrad and PFE. We also report the following the ArcFace testing protocol, which refines both the probe set and the gallery set. As shown from the figure in Tab. 6, our method still clearly outperforms the competitors and achieves the best performance on both verification and identification. In this paper, we propose a novel Adaptive Curriculum Learning Loss that embeds the idea of adaptive curriculum learning into deep face recognition. Our key idea is to address easy samples in the early training stage and hard ones in the later stage. Our method is easy to implement and robust to converge. Extensive experiments on popular facial benchmarks demonstrate the effectiveness of our method compared to the state-of-the-art competitors. Following the main idea of this work, future research can be expanded in various aspects, including designing a better function N (·) for negative cosine similarity that shares similar adaptive characteristic during training, and investigating the effects of noise samples that could be optimized as hard samples.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

A novel  Adaptive Curriculum Learning loss for deep face recognition

Pseudo-labeling has shown to be a weak alternative for semi-supervised learning. We, conversely, demonstrate that dealing with confirmation bias with several regularizations makes pseudo-labeling a suitable approach.