Towards Backward-Compatible Representation Learning
Yantao Shen Yuanjun Xiong Wei Xia Stefano Soatto
cs.CV









Towards Backward-Compatible Representation Learning

    
Yantao Shen[1]    
Yuanjun Xiong    
Wei Xia    
Stefano Soatto 


AWS/Amazon AI

ytshen@link.cuhk.edu.hk,   {yuanjx, wxia, soattos}@amazon.com

    December 30, 2023
=====================================================================================================================================================







[1]Currently at The Chinese University of Hong Kong. Work conducted while at AWS.

We propose a way to learn visual features that are compatible with previously computed ones even when they have different dimensions and are learned via different neural network architectures and loss functions. Compatible means that, if such features are used to compare images, then “new” features can be compared directly to “old” features, so they can be used interchangeably. This enables visual search systems to bypass computing new features for all previously seen images when updating the embedding models, a process known as backfilling. Backward compatibility is critical to quickly deploy new embedding models that leverage ever-growing large-scale training datasets and improvements in deep learning architectures and training methods. We propose a framework to train embedding models, called backward-compatible training (BCT), as a first step towards backward compatible representation learning. In experiments on learning embeddings for face recognition, models trained with BCT successfully achieve backward compatibility without sacrificing accuracy, thus enabling backfill-free model updates of visual embeddings.




§ INTRODUCTION



Visual classification in an “open universe” setting is often accomplished by mapping each image onto a vector space using a function (“model”) implemented by a deep neural network (DNN). The output of such a function in response to an image is often called its “embedding” <cit.>.
Dissimilarity between a pair of images can then be measured by some type of distance between their embedding vectors. A good embedding is expected to  cluster images belonging to the same class in the embedding space. 




As images of a new class become available, their embedding vectors are used to spawn a new cluster in the open universe, possibly modifying its metric to avoid crowding, in a form of “life-long learning.” This process is known as indexing.
It is common in modern applications to have millions, in some cases billions, of images indexed into hundreds of thousands to millions of clusters.
This collection of images is usually referred to as the . 
A common use for the indexed   is to identify the closest clusters to one or a set of input images, a process known as visual search or visual retrieval. The set of input images for this task is known as the .
Besides the gallery and the query set, there is usually a separate large repository of images used for training the embedding model <cit.>, called the .



As time goes by, the datasets grow and the quality of the embeddings improves with newly trained  models
<cit.>.
However, to harvest the benefits of new models, one has to use the new models to re-process all images in the  to generate their embedding and re-create the clusters, a process known as “backfilling” or “re-indexing.”[The reader may have experienced this process when upgrading photo collection software, whereby the search feature is unavailable until the software has re-indexed the entire collection. 
This is a minor inconvenience in a personal photo collection, but for large-scale galleries, the cost of time and computation may be prohibitive, thus hampering the continuous update potential of the system.]
In this paper, we aim to design a system that enables new models to be deployed without having to re-index existing image collections. We call such a system backfill-free, the resulting embedding backward-compatible representation, and the enabling process backward-compatible training (BCT). 

We summarize our contributions as follow: 1) We formalize the problem of backward compatible representation learning in the context of open-set classification, or visual retrieval. The goal is to enable new models to be deployed without having to re-process the previously indexed . The core of this problem is backward compatibility, which requires a new embedding's output to be usable for comparisons against the old embedding model without compromising recognition accuracy. 2) We propose a novel backward compatible training (BCT) approach by adding an influence loss, which uses the learned classifier of the old embedding model in training the new embedding model. 3) We achieve backward compatible representation learning with minimal loss of accuracy, enabling backfill-free updates of the models. We empirically verify that BCT is robust against multiple changing factors in training the embedding models, e.g., neural network architectures, loss functions, and data growth. Finally, 4) we show that compatibility between multiple models can be attained via chain-like pairwise BCT training.





 §.§ Related Work

Embedding learning and open set recognition.
Open-set visual recognition <cit.> is relevant to retrieval <cit.>, face recognition <cit.> and person re-identification <cit.>. Common approaches  involve extracting visual features to instantiate test-time classifiers <cit.>. 
Deep neural networks (DNNs) are widely applied to 1) learn embedding models using closed-world classification as a surrogate task <cit.>, using various forms of loss functions <cit.> and supervision methods <cit.>  to improve generalization 2) perform metric learning <cit.> 
enforcing affinity for pairs <cit.> or triplets <cit.> of representations in the embedding space.
Specifically, <cit.> learns a single metric that is compatible for all tasks in a multi-task learning setting.
Supervising representation learning with classifier weights from other versions of models was proposed in <cit.> for the task of unsupervised representation learning. 

Learning across domains and tasks.
In domain adaptation <cit.>, techniques such as MMD <cit.> and related methods <cit.>, can be used to align the (marginal) distribution of the new and old classes, included those trained adversarially <cit.>.
Knowledge distillation in <cit.> trains new models to learn from existing models, but, unlike backward compatible representation learning, knowledge distillation does not require the embedding of the new model and the existing one to be compatible in inference.
Continual learning <cit.>, transfer learning <cit.>, and life-long learning <cit.> all deal with the cases where an existing model evolves over time. In  <cit.>,  model distillation is used as a form of regularization when introducing new classes. In <cit.>, old class centers are used to regularize samples from the new classes. Hou  <cit.> proposed a framework for learning a unified classifier in the incremental setting. In <cit.>, the authors designed a re-training loss function. Methods addressing catastrophic forgetting <cit.> are most closely related to our work, as a common reason for forgetting is the changing of the visual embedding for the subsequent classifiers.
The problem we are addressing differs in that we aim to achieve backward compatibility between any pair of old model and new model. The new model is not required to be initialized by nor share a similar network architecture as the old model.


Compatible representations.
In <cit.>, the authors discuss the possible mapping between feature vectors from multiple models trained on the same dataset; <cit.> introduce a design where multiple models with different channel widths but the same architecture share common subsets of parameters and representation, which implicitly imposes compatibility among representations from different models. we propose an approach to solve the problem of backward-compatibility in deep learning, in the sense defined in the previous section. We focus on open-universe classification using metric discriminant functions. 





§ METHODOLOGY

We first formulate the problem of backward compatible representation learning, then describe a backward compatible training approach and its implementations.




 §.§ Problem Formulation

    
    As a prototypical application, we use the following case of a photo collection D, serving the role of the gallery.  D is grouped into a number of classes or identities Y = {y_1, …, y__N}. 
    We have an embedding model ϕ_ old that maps each image x onto an embedding vector z = ϕ_ old(x) ∈ℝ^K_ old with x ∈ D. 
    The embedding model ϕ_ old is trained on an , T_ old. The embedding of any image  produced by  D can then be assigned to a class 
    through some distance d:ℝ^K ×ℝ^K →ℝ_+. 
    In the simplest case, dropping the subscript “old,”, each class in Y is associated with a “prototype” or cluster center ϕ_i,  i ∈ Y.
    The vector ϕ_i for the class i can be obtained by a set function ϕ_i = S({ϕ(x)}_y(x) = i), where y(x) is the corresponding class label of an image x ∈ D. Common choices of the set function S include averaging and attention models <cit.>.
    A test sample x is assigned to the class y = min_i ∈ Y d(ϕ(x), ϕ_i) ∈ Y. 
    Later, a new model ϕ_ new with  K_ new-dimensional embedding vectors becomes available, for instance trained with additional data in the new , T_ new (T_ new can be a superset of T_ old), or using a different architecture. The new embedding ϕ_ new is potentially living in a different embedding space and it is possible that K_ new≠ K_ old. 
    
    To harvest the benefit of the new embedding model ϕ_ new, we wish to use ϕ_ new to process any new images that in the  , D, as well as images for the . Since the  could get additional images and clusters, we denote it as D_ new =  D∪{x | y = N+1, ..., N_ new}, where N_ new is the number of clusters in D_ new.
    Then, the question becomes how to deal with images in D.
    In order to make the system backfill-free, we wish to directly use the already computed embedding from ϕ_ old for these images and obtain {ϕ_i}_i ≤ N .
    Our goal, then, is to design a training process for the new embedding model ϕ_ new so that any test images can be assigned to classes, new or old, in D_ new, without the need to compute ϕ_ new( D), i.e., to backfill. 
    The resulting embedding ϕ_ new, is then backward-compatible with ϕ_ old.
    


 §.§ Criterion for Backward Compatibility


In a strict sense, a model ϕ_ new is backward compatible if

    d(ϕ_ new(x_i), ϕ_ old(x_j))    ≥    d(ϕ_ old(x_i), ϕ_ old(x_j)), 
          ∀ (i, j) ∈{(i, j) | y_i ≠ y_j }. 
    and,       
    
        d(ϕ_ new(x_i), ϕ_ old(x_j))    ≤    d(ϕ_ old(x_i), ϕ_ old(x_j)), 
          ∀ (i, j) ∈{(i, j) | y_i = y_j }.

where d(·,·) is a distance in the embedding space. 
These constraints formalize the fact that the new embedding, when used to compare against the old embedding, must be at least as good as the old one in separating images from different classes and grouping those from the same classes. Note that the solution ϕ_ new = ϕ_ old is backward compatible. This trivial solution is excluded if the architectures are different, which is usually the case when updating a model. Although, to simplify the discussion, we assume the embedding dimensions for the two models to be the same  (K_ new = K_ old), our method is more general and not bound by this assumption. 

The criterion introduced in Eq. <ref> entails testing the gallery exhaustively, which is intractable at large scale and in the open-set setting.
    On the other hand, suppose we have an evaluation metric, M (ϕ_q, ϕ_g;  Q,  D,) on some testing protocols, , true positive identification rates for face search, where  Q denotes the query set, D denotes the gallery set, and we use ϕ_q for extracting the   feature and ϕ_g for the .
    Then, the empirical compatibility criterion, for the application can be defined as
    
    M (ϕ_ new, ϕ_ old;  Q,  D) > M (ϕ_ old, ϕ_ old;  Q,  D).

    
This criterion can be interpreted as follows: In an open-set recognition task with a fixed query set and a fixed gallery set, when the accuracy using ϕ_ new for queries without backfilling gallery images surpasses that of using ϕ_ old, we consider backward compatibility achieved and backfill-free update feasible. 
Note that simply setting ϕ_ new to ϕ_ old will not satisfy this criterion.



 §.§ Baseline and paragon
 
A naive approach to train the model ϕ_ new to be compatible with ϕ_ old, assuming they have the same dimension,
is to minimize the ℓ^2 distance between their embeddings computed on the same images. This is enforced for every image in T_ old, which is used to train ϕ_ old. This criterion can be framed as an additive regularizer R for the empirical loss L(ϕ_ new) when training the new embedding as



    ϕ_ new = min_ϕ L(ϕ,  T_ new) + λ R(ϕ),   where


    R(ϕ) = ∑_x ∈T_ old
      1/2ϕ(x)
      - ϕ_ old(x)^2_2.



We label the solution of the problem above  ϕ_ new-ℓ^2. Note that ϕ_ old will be fixed during  training of ϕ_ new-ℓ^2.
As we show in Sect. <ref>,   ϕ_ new-ℓ^2 
does not satisfy Eq. (<ref>) and it will not converge to ϕ_ old, since the training set has been changed to T_ new.  So, this naive approach cannot be used to obtain a backward compatible representation. 

On the other hand, performing the backfill on D with the model ϕ_ new, trained without any regularization, can be taken as a paragon. Since the embedding for D is re-computed, we can fully enjoy the benefit of ϕ_ new 
albeit at the cost of reprocessing the gallery. This sets the upper bound of accuracy for the backfill-free update, and thus the upper bound of the update gain. 



 §.§ Backward Compatible Training


We now focus on backward compatible training for    classification using the cross-entropy loss.
Let Φ  be a model parametrized by two disjoint sets of weights, w_c and w_ϕ. The first parametrizes the classifier κ, or the “head” of the model, whereas the second parametrizes the embedding ϕ, so that Φ(x) = κ_w_c(ϕ_w_ϕ(x)). Now, the cross-entropy loss can be written as

    L(w_c, w_ϕ;  T) = ∑_(x_i, y_i) ∈ T -logκ_w_c(ϕ_w_ϕ(x_i))_y_i.

Note that the classifier κ_w_c can take many forms, from the simple SoftMax <cit.> to recently proposed alternatives <cit.>.
The old model ϕ_ old is thus obtained by solving

    w_c   old, w_ϕ  old = min_w L(w_c, w_ϕ;  T_ old).

As for the new model ϕ_ new , while ordinary training would yield

    w_c  new, w_ϕ  new = min_w L(w_c, w_ϕ;  T_ new),

to ensure backwards-compatibility, we add a second term to the loss that depends on the classifier of the old model:

    w_c  new, w_ϕ  new = min_w L_ BCT (w_c, w_ϕ;  T_ new,  T_ BCT),

where

    L_ BCT(w_c, w_ϕ;  T_ new,  T_ BCT) = L(w_c, w_ϕ;  T_ new) + 
     + λ L(w_c   old, w_ϕ;  T_ BCT).

We call the second term “influence loss” since it biases the solution towards one that can use the old classifier. Note that w_c   old in the influence loss will be fixed during training.
Here, T_ BCT is a design parameter, referring to the set of images we apply the influence loss to. 
It can be either T_ old or T_ new. The approach of using T_ new as T_ BCT will be introduced in Sect. <ref>.
Note that the classifiers κ of the new and old models can be different.
We call this method backward compatible training, and the result backward compatible representation or embedding, which we evaluate empirically in the next section.




 §.§ Learning with Backward Compatible Training

In the proposed backward compatible training framework, there are several design choices to make.

Form of the classifier
The classifiers κ of the new and old models
κ_ new and κ_ old can be of the same form, for instance  Softmax, angular SoftMax classifier <cit.>, or cosine margin <cit.>.
They can also be of different forms, which is common in the cases where better loss formulations are proposed and applied to training new embedding models. 

Backward compatibility training dataset.

The most straightforward choice for the dataset T_ BCT, on which we apply the influence loss, is T_ old, which was used to train the old embedding ϕ_ old. The intuition is that, since the old model ϕ_ old is optimized together with its classifier κ_ old on the original training set T_ old, a new embedding model having a low influence loss will work with the old model's classifier and thus with the embedding vectors from ϕ_ old.
The second choice of T_ BCT is T_ new; this means that we not only compute the influence loss on the old training data for ϕ_ old, but also on the new training data. However, this choice poses a challenge in the computation of the loss value  L(w_c, w_ϕ;  T_ new)
for the images in {x | x ∈ T_ new, x ∉ T_ old}, due to the unknown classifier parameters for the classes. We propose two rules for computing the loss value for these images: 

Synthesized classifier weights. For classes in T_ new which are not in the set of classes in T_ old, we create their “synthesized” classifier weights by computing the average feature vector of ϕ_ old on the images in each class. This approach is inspired by open-set recognition using the class vector ϕ_it as described in Sect. <ref>. We use averaging as the set function in this case. The synthesized classifier weights for the new classes are concatenated with the existing w_c
to form the classifier parameters for the influence loss term.

Knowledge distillation. We penalize the KL-divergence of the classifier output probabilities between using ϕ_ new and ϕ_ old with existing classifier parameters w_c.
This removes the requirement to add new classes in T_ new to the classifiers corresponding to ϕ_ old. 


Backward compatible training is not restricted to certain neural network architecture or loss function. 
It only requires that both the old and new embedding models be trained with classification-based losses, which is common in open-set recognition problems <cit.>. 
It also does not need modification of the architecture nor of the parameters of the old model ϕ_ old.






§ EXPERIMENTS


We assess the effectiveness of the proposed backward compatibility training in 
face recognition.
We start with several baselines,
then test the hypothesis that BCT leads to backward compatible representation learning on two face recognition tasks: face verification and face search. 
Finally, we demonstrate the potential of BCT by applying it to the cases of multi-factor model changes and showing it is able to construct multiple compatible models.



 §.§ Datasets and Face Recognition Metrics

We use the IMDB-Face dataset <cit.> for training face embedding models. The IMDB-Face dataset contains about 1.7M images of 59K celebrities. 
For the openset test, we use the widely adopted IJB-C face recognition benchmark dataset <cit.>. It has around 130k images from 3,531 identities.
The images in IJB-C contain both still images and video frames.
We adopt the two standard testing protocols for face recognition: 1:1 verification and 1:N search (open set). 
For 1:1 verification, a pair of templates (a template contains one or more face images from the same person) are presented and the algorithm is required to decide whether they belong to the same person or not.
The evaluation metrics for this protocol are true acceptance rate (TAR) at different false acceptance rates (FAR). 
We present the results of TAR at the FAR of 10^-4.

For 1:N search, a set of templates is first indexed as the . Then each template in the  is used to search against the indexed templates. 
The quality metrics for this protocols are true positive identification rates (TPIR) at different false positive identification rates (FPIR).
We present the results of TPIR at 10^-2 FPIR.




 §.§ Implementation details


We use 8 NVIDIA Tesla V-100 GPUs in training the embedding models. The input size of embedding models is set to 112 × 112 pixels <cit.>. We use face mis-alignment and color distortion for data augmentation. Weight decay is set to 5 × 10^-4 and standard stochastic gradient descent (SGD) is used to optimize the loss. The initial learning rate is set to 0.1 and decreased to 0.01, 0.001, and 0.0001 after 8, 12, and 14 epochs, respectively. The training stops after 16 epochs. The batchsize is set to 320. Unless stated otherwise, we use ResNet-101 <cit.> as the backbone, a linear transform after its global average pooling layer to emit 128-dimensional feature vectors , and Cosine Margin Loss <cit.> with margin=0.4 as the loss function in our experiments.



 §.§ Measuring Backward-Compatibility
 



Based on the accuracy on the individual tests on the face recognition dataset, we can  test whether a pair of  models satisfies the empirical backward compatibility criterion. For a pair of models (ϕ_ new, ϕ_ old), on each evaluation protocol we test whether they satisfy Eq. (<ref>). If so, we consider the new model backward compatible with the old model in the corresponding task. 
In testing using the IJB-C 1:N protocol <cit.>, we use the new model ϕ_ new to extract embeddings for the   and the old model ϕ_ old to compute embeddings for . For the IJB-C 1:1 verification protocol <cit.>, we use ϕ_ new to extract the embeddings for the first template in the pair and ϕ_ old for the second.


To evaluate relative improvement brought by the backfill-free update, we define the update gain as

    G(ϕ_ new, ϕ_ old;  Q,  D) =  M (ϕ_ new, ϕ_ old; Q,  D) - M (ϕ_ old, ϕ_ old; Q,  D)/M (ϕ_ new^*, ϕ_ new^*; Q,  D) - M (ϕ_ old, ϕ_ old; Q,  D).

Here, M(ϕ_ new^*, ϕ_ new^*;  Q,  D) stands for the best accuracy level 
we can achieve from any variants of the new model by backfilling. 
It indicates the proportional gain we can obtain the performing backfill-free update compared with the update which performs the backfill regardless of the cost and interruption of services.
Note that the update gain is only valid when Eq. (<ref>) is satisfied.













 §.§ Baseline comparisons

The first hypothesis to be tested is whether BCT is necessary at all: is it possible to achieve backward compatibility with a more straightforward approach? In this section, we experiment with several  baseline approaches and validate the necessity of BCT.

Independently trained ϕ_ new and ϕ_ old

The first sanity-check is to directly compare the embedding of two versions of models trained independently. A similar experiment is done in <cit.> for multiple close-set classification models trained on the same dataset. 
Here we present two models. The ϕ_ old is trained with the randomly sampled 50% IDs subset of the IMDBFace dataset <cit.>. The new model is trained on the full IMDBFace dataset <cit.>.
This emulates the case where a new embedding model becomes available when the sizes of embedding training dataset grows. 
We name the new model ϕ_ new^* according to the experiment in Sect. <ref>, showing that it currently achieves the best accuracy among all new models with the same setting.
We directly test the compatibility of this pair of models (ϕ_ new^*, ϕ_ old) following the procedure described in Sect. <ref>. The results are illustrated in Tab. <ref>.
In the backward test for both protocols, we observed almost 0% accuracy. Unsurprisingly, 
independently trained ϕ_ new and ϕ_ old does not
naturally satisfy our compatibility criterion.

Does the naive baseline with ℓ^2-distance work?

In Sect. <ref> we described the naive approach of adding the ℓ^2-distance between the new and old embeddings as a regularizer when training the new model. 
We train a new model using the same old model above and train the new model with the loss function (<ref>) on the whole IMDBFace dataset <cit.>. 
We name this model ϕ_ new-ℓ^2 to reflect that it is ℓ^2-distance regularized towards the old model.
The same backward compatibility test is conducted with this pair of models (ϕ_ new-ℓ^2, ϕ_ old) on the same two protocols described in the previous baseline. The results are shown in Tab. <ref>.
We can observe that this approach only leads to slightly above 0% backward test accuracy, which means the new model ϕ_ new-ℓ^2 is far from satisfying the compatibility criterion.
One possible reason is that imposing an ℓ_2 distance penalty creates a bias that is too local  and restrictive to allow the new model to satisfy the compatibility constraints. 













 §.§ Learning with BCT



We now experiment with the proposed BCT framework for backward compatible representation learning, starting from its basic form described in Sect. <ref>.
We use the same old model in the previous section. For the new model, 
we train it with the objective function described in Eq. (<ref>).  This model is called ϕ_ new-β.
As illustrated in Tab. <ref> and Tab. <ref>, the model pair, (ϕ_ new-β, ϕ_ old), satisfies the backward compatibility criterion in Eq. (<ref>). 
Additionally, we observe update gains of 26.26% and 44.98% on the 1:1 verification and 1:N search protocols respectively.

We also evaluate a baseline approach adapted from <cit.>. The model trained with this approach is denoted as ϕ_ new-LwF. It uses fixed ϕ_ old and its classifier w_c   old to output soft labels of newly added samples x ∈T_ new∖T_ old as pseudo labels for training ϕ_ new-LwF. 

From Tab.<ref> and <ref> we can see that model pair (ϕ_ new-LwF, ϕ_ old) does not satisfy the empirical backward compatibility criterion. Showing that directly adapting methods from the continual learning task does not work out of the box. However, it is able to improves to some extend the backward comparison accuracy, suggesting that the knowledge distillation used in continual learning could be useful in BCT.
We further investigate its application in the following experiments.


BCT with newly added training data.
In Sect. <ref> we described two instantiations of BCT which can work with the new classes in the growing . 
The first is using the synthesised classifier, we name the new model trained with this form of BCT as ϕ_ new-β-sys. 
The second applies the idea of knowledge distillation to bypass obtaining classifier parameters for the new classes in the . 
We name the new model trained with this form of BCT as ϕ_ new-β-kd. 
The backward compatibility test results are summarized in Tab. <ref>.
We can see that both new models can achieve backward compatibility. 
By fully utilizing the additional training data, they also lead to higher update gain (30.00% for ϕ_ new-β-sys and 27.25% for ϕ_ new-β-kd) compared with the basic form of BCT (26.26% for ϕ_ new-β).



Does BCT hurt the accuracy of new models?
One natural question is whether the influence loss is detrimental to the new model's recognition performance. 
We assess this by performing standard face recognition experiments on the 1:1 and 1:N protocols by extracting embedding only using the new models. 
This process can be considered as performing the backfill, or the paragon setting as described in Sect. <ref>.
The results are summarized in Tab. <ref>.
We can see that training without BCT still yields the best accuracy in this setting. So, we name the model trained without BCT as ϕ_ new^*, to indicate that it is the paragon that achieves best accuracy among all variants of new models.
Note that models trained with the basic form of BCT, ϕ_ new-β, only leads to less than 3% drop of accuracy in both tasks.
The new models ϕ_ new-β-sys and ϕ_ new-β-kd further reduce the gap.













 §.§ Extensions of BCT

In the following experiments we explore whether BCT can be applied to different types of model training and achieve multi-model compatibility.

Other changes in training ϕ_ new.
Besides increasing the size of the , the new model ϕ_ new could have a new model architecture (e.g., depth), a different loss function for supervision, or different embedding dimensions.  
We experiment the effect of these factors on BCT. 
For network architectures, we test a new model using ResNet-152 <cit.> instead of ResNet-101 <cit.> in previous experiments, denoted as ϕ_ new-β^ R152.
In terms of loss types, we test using Norm-Softmax Loss <cit.> for the old model, ϕ_ old^ NS, and Cosine Margin Loss <cit.> for the new one, ϕ_ new-β^ Cos.
In terms of embedding dimension, we test increasing the dimensions from 128 to 256 in the new model. Note that when the new model's feature dimension is changed, we will not be able to directly feed it to w_c   old. Here, we simply take the first 128 elements of the new model's feature to feed into w_c   old during backward-compatible training and testing. We tried another approach of adding a linear transformer in training to match the feature dimension but without success. 

We also test the cases of changing several factors together, denoted as ϕ_ new-β^ R152+256D.
The results are shown in Tab. <ref>. BCT can make most of new models backward compatible, even when several factors change simultaneously. 
This shows that BCT can be used as a general framework for achieving backward compatible representation learning. 

There are two failure cases where backward compatibility is not achieved in the pairs: 1)(ϕ_ new-β^ Cos-S, ϕ_ old^ S), which uses Softmax loss <cit.> for the old model and Cosine Margin Loss <cit.> for new model; this is possibly due to the drastic change of form of the loss functions. 2) (ϕ_ new-β^ ReLU, ϕ_ old), which adds ReLU activation on the embedding of the new model. The latter is possibly due to the distributional shift introduced by the ReLU activation in the new model, which makes it difficult for the new model with non-negative embedding vector elements to be compatible with the old model. This suggests that additional work is needed to expand the set of models that BCT can support. 















Towards multi-model and sequential compatibility.
Here we investigate a simple case of three model versions. 
The first version ϕ_1 is trained with T_1, which is a randomly sampled  25% subset of the IMDBFace dataset <cit.>. 
The second version ϕ_2 is trained  with T_2, a 50% subset. 
And the third version ϕ_3 is trained with T_3, which is the full IMDBFace dataset <cit.>. 
We train ϕ_2 using BCT with ϕ_1 and train ϕ_3 using BCT with ϕ_2. 
Thus, in this process ϕ_3 has no direct influence from ϕ_1.
The backward compatibility test results are shown in Tab. <ref> and Fig. <ref>. 
We observe that, by training with BCT, the last model ϕ_3 is transitively compatible with ϕ_1 even though ϕ_1 is not directly involved in training ϕ_3.
It shows that transitive compatibility between multiple models is indeed achievable through BCT, which could enable sequential update of the embedding models.

BCT in other open-set recognition tasks.
We validate the BCT method on the person re-identification task using the Market-1501 <cit.> benchmark. We train an old embedding model following <cit.> with 50% of training data and two new embedding models with 100% of the new training data. Search mean average precision (mean AP) is used as the accuracy metric. Table <ref> shows the results of backward compatibility test.
We observe that the ϕ_new-β trained with BCT achieves backward compatibility without sacrificing its own search accuracy. 
This suggests that BCT can be a general approach for open-set recognition problems.





§ DISCUSSION




We have presented a method for achieving backward-compatible representation learning, illustrated specific instances, and compared them with both baselines and paragons. Our approach has several limitations. The first is the accuracy gap of the new models trained with BCT relative to the new model oblivious of previous constraints.  Though the gap is reduced by slightly more sophisticated forms of BCT, there is still work wo to be done in characterizing and achieving the attainable accuracy limits. 





ieee_fullname














§ IMPLEMENTATION DETAILS


In Section 2.5, we describe how to use the influence loss on newly added training data for BCT to 1) compute synthesized classifier weights with the old model, and 2) use knowledge distillation.

When we have a newly added training example whose class is not in the old embedding training set, we feed the new image into the old model ϕ_ old and old classifier w_ c  old to obtain the classifier responses. Then, we can provide supervision signal to the new model ϕ_ new for this image by feeding the new model's embedding to the old classifier w_ c  old and compute the knowledge distillation loss (cross-entropy with temperature-modulated SoftMax) between the two response vectors as the influence loss.
Because we are using the cosine margin loss <cit.> in the experiments which also has a temprature parameter, we set the temperature parameter in knowledge distillation to the same as the one in the cosine margin loss, which is 32.



§ PARTIAL BACKFILLING

Partial backfilling happens when only a part of the gallery classes have been processed by the new model ϕ_new. We test whether queries from the backward compatible new model ϕ_new-β can work with partially backfilled gallery sets. In Fig. <ref>, we illustrate the search accuracy on gallery sets of different backfill ratios.  As higher percentages of the gallery set are backfilled, search accuracy grow proportionally  towards the accuracy of the fully backfilled case. This suggests that one can upgrade to a new model, immediately benefiting from the improved accuracy, and optionally backfill the old gallery gradually in the background until paragon performance is achieved.








§ DETAILED BENCHMARK RESULTS


Due to space limitations, in the main paper we only report one specific operating point for each metric in the evaluation, e.g., TAR@  FAR=10^-4 for face verification and TPIR@  FPIR=10^-2 for face identification. 
Here we report results at additional operating points on the IJB-C <cit.> benchmark. 
In Table  <ref>, we show the performance of different compared baselines and our proposed method. 
In Table  <ref> and Table  <ref>, we show the performance of extensions of our proposed backward-compatible trainting process. In Table  <ref>, we illustrate extensions of BCT to different model depths, feature dimensions and supervision losses. In Table  <ref>, we show extensions to multi-model compatibility towards sequential updating. 




















