title: Delving into Inter-Image Invariance for Unsupervised Visual Representations
authors: Jiahao Xie Xiaohang Zhan Ziwei Liu Yew Soon Ong Chen Change Loy
text: 
  
  
  
  
  
  
  
  
  
  
  Jiahao Xie Nanyang Technological University 
  
                jiahao003@ntu.edu.sg
             Xiaohang Zhan The Chinese University of Hong Kong 
  
                xiaohangzhan@outlook.com
             Ziwei Liu Nanyang Technological University 
  
                ziwei.liu@ntu.edu.sg
             Yew Soon Ong Nanyang Technological University 
  
                asysong@ntu.edu.sg
             Chen Change Loy Nanyang Technological University 
  
                ccloy@ntu.edu.sg
  
  
  
  
  Delving into Inter-Image Invariance for Unsupervised Visual Representations
      Jiahao Xie      Xiaohang Zhan   Ziwei Liu       Yew Soon Ong    Chen Change Loy
  
      Received: date / Accepted: date
  ====================================================================================
  
  
  
  
  Contrastive learning has recently shown immense potential in unsupervised visual representation learning. Existing studies in this track mainly focus on intra-image invariance learning. The learning typically uses rich intra-image transformations to construct positive pairs and then maximizes agreement using a contrastive loss.
  
  The merits of inter-image invariance, conversely, remain much less explored. One major obstacle to exploit inter-image invariance is that it is unclear how to reliably construct inter-image positive pairs, and further derive effective supervision from them since no pair annotations are available. In this work, we present a comprehensive empirical study to better understand the role of inter-image invariance learning from three main constituting components: pseudo-label maintenance, sampling strategy, and decision boundary design. To facilitate the study, we introduce a unified and generic framework that supports the integration of unsupervised intra- and inter-image invariance learning. Through carefully-designed comparisons and analysis, multiple valuable observations are revealed: 1) online labels converge faster and perform better than offline labels; 2) semi-hard negative samples are more reliable and unbiased than hard negative samples; 3) a less stringent decision boundary is more favorable for inter-image invariance learning. With all the obtained recipes, our final model, namely InterCLR, shows consistent improvements over state-of-the-art intra-image invariance learning methods on multiple standard benchmarks. We hope this work will provide useful experience for devising effective unsupervised inter-image invariance learning. Code: <https://github.com/open-mmlab/mmselfsup>.
  
  
  
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  
  
  Unsupervised representation learning witnesses substantial progress thanks to the emergence of self-supervised learning[Self-supervised learning is a form of unsupervised learning. While these terms are used interchangeably in the literature, we use the more classical term of “unsupervised learning”, to reflect the general sense of “not supervised by human-annotated labels”. A more detailed treatment is provided in <cit.>.], which can be broadly divided into four categories: recovery-based <cit.>, transformation prediction <cit.>, clustering-based <cit.>, and contrastive learning <cit.>.
  
  Among the various paradigms, contrastive learning shows great potential and even surpasses supervised learning <cit.>.
  
  A typical contrastive learning method applies rich transformations to an image and maximizes agreement between different transformed views of the same image via a contrastive loss in the latent feature space.
  
  This process encourages the network to learn “intra-image” invariance (, instance discrimination <cit.>). 
  
  
  
  Some typical “intra-image” transformations, including random cropping, resizing, flipping and color distortion, are shown in Fig. <ref>. 
  
  Clearly, it is challenging to design convincing transformations to faithfully cover all the natural variances existing in natural images.
  
  Hence, it remains an open question whether the existing form of transformations can sufficiently lead to our ideal representations, which should be invariant to viewpoints, occlusions, poses, instance-level or subclass-level differences.
  
  Such variances naturally exist between pairs of instances belonging to the same semantic class.  
  
  However, it is challenging to exploit such “inter-image” invariance in the context of unsupervised learning since no pair annotations are available.
  
  Clustering is a plausible solution to derive such pseudo-labels for contrastive learning. For instance, LA <cit.> adopted off-the-shelf clustering to obtain pseudo-labels to constitute “inter-image” candidates.
  
  Nevertheless, the performance still falls behind state-of-the-art intra-image invariance learning methods.
  
  We believe there exist details that might have been ignored, if resolved, shall make the usefulness of inter-image invariance learning more pronounced than it currently does.
  
  In this work, we go back to basics and systematically investigate the effects of inter-image invariance from three major aspects:
  
  
  1) Pseudo-label maintenance.
  
  Owing to expensive computational cost, global clustering adopted in prior works <cit.> can only be performed sparsely every several training epochs.
  
  Hence, it inevitably produces stale labels relative to the rapidly updated network.
  
  To re-assign pseudo-labels continuously and instantly, we consider mini-batch k-means in place of global k-means by integrating the label and centroid update steps into each training iteration.
  
  In this way, clustering and network update are simultaneously undertaken, yielding more reliable pseudo-labels.
  
  
  2) Sampling strategy.
  
  It is common for supervised learning to adopt hard negative mining <cit.>.
  
  However, in the scenario of unsupervised learning, hard negatives might well have wrong labels, , they may be actually positive pairs.
  
  On the other hand, if we choose easy negative pairs and push them apart, they will still be easy negatives next time, and might never be corrected, leading to a shortcut solution.
  
  Hence, the sampling strategy in unsupervised inter-image invariance learning is non-trivial, which has so far been neglected.
  
  
  3) Decision boundary design.
  
  Existing works <cit.> in supervised learning design large-margin loss functions to learn discriminative features.
  
  While in unsupervised learning, it is unsure whether pursuing discriminative features benefits since pseudo-labels are noisy.
  
  For example, if a positive pair of images are misclassified as a negative one, the large-margin optimization strategy will further push them apart. Then the situation will never be corrected.
  
  We explore decision margin designs for both the unsupervised intra- and inter-image branches.
  
  Contributions – This study aims at revealing key aspects that should be carefully considered when leveraging unsupervised inter-image information in contrastive learning. 
  
  Although some of these aspects were originally considered in supervised learning, the conclusions are quite different and unique in the unsupervised scenario. To the best of our knowledge, this is the first empirical study on the effects of these aspects for unsupervised inter-image contrastive representations.
  
  The merits of inter-image invariance learning are demonstrated through its consistent improvements over state-of-the-art intra-image invariance learning methods on multiple standard benchmarks.
  
  
  
  
  
  
  
  § RELATED WORK
   
  
  
  
  Contrastive-based representation learning.
  
  Contrastive-based methods learn invariant features by contrasting positive samples against negative ones. A positive pair is usually formed with two augmented views of the same image, while negative ones are formed with different images. Typically, the positive and negative samples can be obtained either within a batch or from a memory bank. In batch-wise methods <cit.>, positive and negative samples are drawn from the current mini-batch with the same encoder that is updated end-to-end with back-propagation.
  
  For methods based on memory bank <cit.>, positive and negative samples are drawn from a memory bank that stores features of all samples computed in previous steps. Recently, MoCo <cit.> builds large and consistent dictionaries for contrastive learning using a slowly progressing encoder. BYOL <cit.> and SimSiam <cit.> further learn invariant features without negative samples.
  
  As opposed to our work, the aforementioned approaches only explore intra-image statistics for contrastive learning.
  
  Although there are a few prior attempts <cit.> to leverage inter-image statistics for contrastive learning, they mainly focus on either designing sampling metric or comparing instance-group features while leaving other important aspects unexplored. NNCLR <cit.> also embraces inter-image samples for contrastive learning. As opposed to our work, they use nearest neighbors from a support set to define positive samples, whereas we use cluster assignments to sample contrastive pairs. Besides, NNCLR still relies on a large batch size (, 4096) to achieve good results, while using a smaller batch size (, 256) will significantly decrease its performance. In contrast, InterCLR can achieve competitive performance using a more affordable batch size of 256. More importantly, we empirically study inter-image invariance learning from different aspects and show that pseudo-label maintenance, sampling strategy and decision boundary design should be collectively considered for good results.
  
  
  Our study is more related to a strand of recent research <cit.> that focuses on developing theoretical or empirical understanding of contrastive representations from different aspects.
  
  As opposed to their works, we provide empirical understanding of contrastive learning from its inter-image invariance perspective.
  
  There are also another group of works that perform dense contrastive learning by extending existing image-level methods to the pixel level <cit.> or the region level <cit.>. Although better performance emerges on dense prediction downstream tasks, most of their classification downstream performance is largely sacrificed. Our work differs from this line of research in that we do not aim at developing a more advanced pretext task to learn spatially structured representations. Instead, we aim at better leveraging inter-image invariance for contrastive learning to pursue generic representations that improve both classification and dense prediction downstream tasks. We expect that our findings can be further applied on these dense contrastive learning variants.
  
  
  
  Clustering-based representation learning.
  
  Earlier attempts have shown great potential of joint clustering and feature learning, but the studies are limited to small datasets <cit.>. DeepCluster <cit.> (DC) scales up the learning to millions of images through alternating between deep feature clustering and CNN parameters update. Although DC uses clustering during representation learning, it differs from our work conceptually in two aspects. First, DC optimizes the cross-entropy loss between predictions and pseudo-labels obtained by cluster assignments. Such optimization requires an additional parametric classifier. Second, DC adopts offline global clustering that unavoidably permutes label assignments randomly in different epochs. As a result, the classifier has to be frequently reinitialized after each label reassignment, which leads to training instability. In contrast, we optimize a non-parametric classifier at instance level and integrate the label update procedure into each training iteration with online clustering. ODC <cit.> also performs online clustering. Our work differs from theirs in that ODC follows DC to optimize the cross-entropy loss between predicted and cluster labels, requiring the computationally expensive parametric classifier. In addition, ODC uses loss re-weighting to handle clustering distribution, while InterCLR directly uses online mini-batch k-means without resorting to other specific techniques. Experiments in Sect. <ref> show that InterCLR substantially outperforms ODC.
  
  Recently, SwAV <cit.> enforces consistent cluster-assignment prediction between multiple views of the same image. The cluster assignments are produced by Sinkhorn-Knopp transform <cit.> under an equipartition constraint similar in <cit.>. As opposed to our work, SwAV does not compare the features but the cluster assignments of the same instance, whereas we directly sample and compare the features of different instances using the cluster assignments.
  
  In conclusion, the aforementioned differences make InterCLR a simple yet effective alternative to existing clustering-based methods so that we can focus on the essence of inter-image invariance learning without the interference of other factors.
  
  Negative mining.
  
  Selection of hard negative samples has been proven effective in deep metric learning <cit.>. Some recent works <cit.> also reveal that contrastive representation learning benefits from hard negative samples. However, prior works study the effect of negatives either in supervised learning or unsupervised intra-image invariance learning. Some works <cit.> further study the sampling issues for cross-modal invariance learning. In contrast, we target at the case of unsupervised inter-image invariance learning within a single modality, which is equally important but largely neglected. The inaccurate nature of unsupervised inter-image invariance learning makes our conclusion quite unique and complementary to existing works.
  
  Loss functions.
  
  Loss functions play an important role in unsupervised representation learning. A loss function is defined based on the properties of pretext tasks. For instance, context auto-encoders <cit.> incorporate L2 loss to reconstruct input pixels, while patch orderings <cit.> use cross-entropy loss to classify input image patches into pre-defined positions or orderings. Adversarial losses <cit.> used for representation learning are also explored in <cit.>. The current state-of-the-art contrastive learning approaches adopt contrastive losses <cit.> that measure the similarity of sample pairs in embedding space at the instance level. Prior works <cit.> have shown that it is beneficial to learn discriminative features by designing large-margin loss functions in supervised learning. However, the effect of decision boundary on the unsupervised counterpart, especially for the unsupervised inter-image invariance learning scenario, is still unknown. Our work makes a first attempt decision boundary design in contrastive learning for unsupervised representations.
  
  
  
  
  
  
  § PRELIMINARIES
  
  
  
  
  Intra-image invariance learning.
  
  A contrastive representation learning method typically learns a neural encoder f_θ(*) that maps training images ={_1,_2,...,_N} to compact features ={_1,_2,...,_N} with _i=f_θ(_i) in a D-dimensional L2-normalized embedding space, where the samples of a positive pair are pulled together and those of negative pairs are pushed apart.
  
  For intra-image invariance learning, the positive pair is usually formed with two different augmented views of the same image while the negative pairs are obtained from different images.
  
  To achieve this objective, a contrastive loss function is optimized with similarity measured by dot product.
  
  Here we consider an effective form of contrastive loss function, called InfoNCE <cit.>, as follows:
  
      .93!_InfoNCE=∑_i=1^N-logexp(_i·_i^+/τ)/exp(_i·_i^+/τ)+∑__i^-∈_Kexp(_i·_i^-/τ),
  
  where τ is a temperature hyper-parameter, _i^+ is a positive sample for instance i, and _i^-∈_K⊆∖{_i} denotes a set of K negative samples randomly drawn from the training images excluding instance i.
  
  Memory bank.
  
  Contrastive learning requires a large number of negative samples to learn good representations <cit.>.
  
  However, the number of negatives is usually limited by the mini-batch size.
  
  While simply increasing the batch to a large size (, 4k-8k) can achieve good performance <cit.>, it requires huge computational resources.
  
  To address this issue, one can use a memory bank to store running average features of all samples in the dataset computed in previous steps.
  
  Formally, let ={_1,_2,...,_N} denote the stored features in the memory bank, these features are updated by:
  
      _i←(1-ω)_i+ω_i,
  
  where ω∈(0,1] is a momentum coefficient.
  
  With a set of features , we can then replace  with  in Eq. (<ref>) without having to recompute all the features every time.
  
  
  
  
  
  
  § METHODOLOGY
  
  
  
  Based on the aforementioned intra-image invariance learning, we describe how to extend the notion to leverage inter-image invariance for contrastive learning.
  
  As shown in Fig. <ref>, we introduce two invariance learning branches in our framework, one for intra-image and the other for inter-image.
  
  The intra-image branch draws contrastive pairs by indices following the conventional protocol. The inter-image counterpart constructs contrastive pairs with pseudo-labels obtained by clustering: a positive sample for an input image is selected within the same cluster while the negative samples are obtained from other clusters.
  
  We use variants of InfoNCE described in Sect. <ref> as our contrastive loss and perform back-propagation to update the networks.
  
  Within the inter-image branch, three components have non-trivial effects on learned representations and require specific designs, , 1) pseudo-label maintenance, 2) sampling strategy for contrastive pairs, and 3) decision boundary design for the loss function.
  
  
  
  
   §.§ Maintaining Pseudo-Labels
  
  
  
  To avoid stale labels from offline clustering, we adopt mini-batch k-means to integrate label update into the network update iterations, thus updating the pseudo-labels on-the-fly.
  
  Formally, we first initialize all the features, labels and centroids via a global clustering process, , k-means.
  
  Next, in a mini-batch stochastic gradient descent iteration, the forward batch features are used to update the corresponding stored features in the memory bank with Eq. (<ref>).
  
  Meanwhile, the label of each involved sample is updated by finding its current nearest centroid following:
  
      min__i∈{0,1}^k, s.t. _i^T=1_i - _i_2^2,
  
  where k is the number of clusters, ∈ℝ^d× k is a recorded centroid matrix with each column representing a temporary cluster centroid that evolves during training, _i is a k-dimensional one-hot vector indicating the label assignment for instance i.
  
  Finally, the recorded centroid matrix is updated by averaging all the features belonging to their current and respective clusters.
  
  In this way, labels are updated instantly along with the features.
  
  
  
  
  
  
  
  
   §.§ Sampling Contrastive Pairs
  
  
  
  As discussed in Sect. <ref>, sampling positive and negative pairs in unsupervised inter-image invariance learning is non-trivial. To investigate the effect of sampling, we design and compare four sampling strategies for negative samples: hard, semi-hard, random, and semi-easy.
  
  We define samples sharing the same label with the input image _i in the memory bank as positive sample candidates 𝒮_i^p, while others as negative sample candidates 𝒮_i^n.
  
  For positive sampling, we randomly draw one sample from 𝒮_i^p and use it to form a positive pair with _i.
  
  For negative sampling, we sample K negatives from 𝒮_i^n.
  
  As shown in a schematic illustration in Fig. <ref>, for “hard negative” sampling, we sample K nearest neighbors of _i from 𝒮_i^n using cosine distance criterion.
  
  For “semi-hard negative” sampling, we first create a relatively larger nearest neighbor pool, , top 10% nearest neighbors from 𝒮_i^n, then we randomly draw K samples from this pool.
  
  For “random negative” sampling, we simply draw K negative samples at random from 𝒮_i^n.
  
  For “semi-easy negative” sampling, similar to the “semi-hard negative” strategy, we first sample a pool with top 10% farthest neighbors from 𝒮_i^n, then we randomly draw K samples from this pool[We define samples drawn within top 50% nearest neighbors from 𝒮_i^n as “semi-hard” negatives, while top 50% farthest neighbors from 𝒮_i^n as “semi-easy” negatives.].
  
  We do not include an “easy negative” strategy that chooses the top K easiest negatives. As mentioned in Sect. <ref>, the easiest samples are prone to a shortcut solution.
  
  
  
   §.§ Designing Decision Boundary
  
  
  
  Designing decision boundary for unsupervised inter-image invariance learning needs special care as pseudo-labels are noisy. Here, we present a way that allows decision margins to be more stringent or looser to suit the variability required in our task.
  
  Considering the contrastive loss in Eq. (<ref>), since features in the embedding space are L2-normalized, we replace _i·_j with cos(θ__i,_j).
  
  For simplicity of analysis, we consider the case where there is only one negative sample, , a binary classification scenario.
  
  The contrastive loss thus results in a zero-margin decision boundary given by:
  
      cos(θ__i,_i^+) = cos(θ__i,_i^-).
  
  
  To allow the decision margins to be more stringent or looser, we first introduce a cosine decision margin m such that the decision boundary becomes:
  
  
      C_+: cos(θ__i,_i^+)-m ≥cos(θ__i,_i^-), 
      
          C_-: cos(θ__i,_i^-)-m ≥cos(θ__i,_i^+).
  
  
  As shown in Fig. <ref>, m>0 indicates a more stringent decision boundary that encourages the discriminative ability of the representations, while m<0 stands for a looser decision boundary.
  
  Then, we define a margin contrastive loss (MarginNCE) as:
  
      1.01!_MarginNCE=∑_i=1^N-logexp((cos(θ__i,_i^+)-m)/τ)/exp((cos(θ__i,_i^+)-m)/τ)+∑__i^-∈_Kexp(cos(θ__i,_i^-)/τ).
  
  
  We make a hypothesis that for the intra-image MarginNCE loss (_Intra-MarginNCE), the margin should be positive, since the labels derived from image indices are always correct; while for the inter-image MarginNCE loss (_Inter-MarginNCE), the margin should be negative, since the pseudo-labels are evolving during training and are not accurate enough.
  
  The final loss consists of these two MarginNCE loss functions:
  
      _Intra-Inter-MarginNCE = λ_Intra-MarginNCE
      
          +(1-λ)_Inter-MarginNCE,
  
  where λ is the weight to balance the two terms.
  
  We study the effects of both m and λ in Sect. <ref>.
  
  
  
  
  
  
  § EXPERIMENTS
  
  
  
  
  
  
   §.§ Implementation Details
  
  
  
  
  
    §.§.§ Baselines
  
  
  
  Our most essential intra-image invariance learning baseline is NPID <cit.>: it is the special case of InterCLR by setting λ=1 in Eq. (<ref>). We find it possible to further improve the implementation of NPID by adopting more advanced techniques in <cit.>: a 2-layer MLP head and stronger Gaussian blur augmentation. We denote our improved version of NPID as NPIDv2. Comparing InterCLR with NPIDv2 is critical to study the effect of inter-image invariance learning that InterCLR aims to achieve. In addition, to demonstrate that InterCLR is more generally applicable, we also experiment with most recent baselines, , MoCov2 <cit.> and BYOL <cit.>.
  
  We refer to InterCLR built upon the aforementioned baselines as NPIDv2-InterCLR, MoCov2-InterCLR and BYOL-InterCLR, respectively.
  
  
  
    §.§.§ Training Details
  
  
  
  We use ResNet-50 <cit.> as the default backbone and perform unsupervised pre-training on the 1.28M ImageNet <cit.> training set without labels.
  
  Prior works <cit.> have shown that using a larger batch size or training for longer epochs can improve the performance of unsupervised representations.
  
  However, they require huge computational resources that are inaccessible to many research labs, which is not the core of this paper.
  
  We instead focus on comparing all methods under a more commonly affordable pre-training budget, , a batch size of 256 for 200 epochs with 4 NVIDIA Tesla V100 GPUs.
  
  Besides, to demonstrate that InterCLR can also benefit from longer training epochs, we further train BYOL-InterCLR for 1000 epochs with batch size 256.
  
  To ensure fair and direct comparisons, we generally follow the same setting of each baseline we experiment with. The details are described next.
  
  NPIDv2-InterCLR.
  
  Our improved version of NPID <cit.> (, NPIDv2) extends the original data augmentation in <cit.> by including Gaussian blur in <cit.>.
  
  However, we do not use the same heavy color distortion as <cit.> since it has diminishing gains in our higher baseline.
  
  Instead, we only apply a color jittering with a saturation factor in [0, 2], and a hue factor in [-0.5, 0.5].
  
  We also add a 2-layer MLP projection head (with a 2,048-dimensional hidden layer and ReLU) to project high-dimensional features into a 128-D L2-normalized embedding space following <cit.>.
  
  We use SGD as the optimizer with a momentum of 0.9 and a weight decay of 10^-4.
  
  We adopt the cosine learning rate decay schedule <cit.> with an initial learning rate of 0.03 using a batch size of 256 for 200 epochs.
  
  We set the temperature parameter τ=0.1, the number of negative samples K=16,384, and the momentum coefficient ω=0.5.
  
  The aforementioned modifications make NPIDv2 a stronger baseline, upon which we build InterCLR. Yet, NPIDv2-InterCLR substantially outperforms NPIDv2 as shown in Sect. <ref>.
  
  For inter-image branch, we use online pseudo-label maintenance, semi-hard negative sampling, and cosine margin m=-0.5 for _Inter-MarginNCE.
  
  We do not use any cosine margin for intra-image branch to solely verify the effectiveness of inter-image branch.
  
  We find over-clustering to be beneficial and set the number of clusters as 10,000, which is 10 times of the annotated number of ImageNet classes.
  
  We set the final loss weight λ=0.75 in Eq. (<ref>).
  
  MoCov2-InterCLR.
  
  We maintain an additional memory bank to store all features from the momentum-updated encoder of MoCov2 for online clustering. We set the number of negative samples K=16,384, and the momentum coefficient ω=1 for the memory bank. We use online pseudo-label maintenance, semi-hard negative sampling, and cosine margin m=-0.5 only for _Inter-MarginNCE. We set the number of clusters as 10,000 and the final loss weight λ=0.75. Other hyper-parameters including MLP projection head, temperature parameter, training procedure and data augmentation setting exactly follow <cit.>.
  
  
  
  BYOL-InterCLR.
  
  We exactly follow <cit.> for the training hyper-parameters and augmentation recipes.
  
  Considering that many previous methods report their performance on 200 epochs, we also train for 200 epochs for fair comparisons, following the 300-epoch recipes in <cit.>: base learning rate is 0.3, weight decay is 10^-6, and the base target exponential moving average parameter is 0.99. The same 200-epoch recipes are also adopted in <cit.>.
  
  For 1000-epoch pre-training, we follow the same 1000-epoch recipes in <cit.>: base learning rate is 0.2, weight decay is 1.5×10^-6, and the base target exponential moving average parameter is 0.996.
  
  Similar to MoCov2-InterCLR, we use a memory bank to store all features from the target network of BYOL to facilitate online clustering.
  
  We accumulate gradients to simulate batch size 4096 for BYOL experiments due to resource constraints. Thus, our reproduced 1000-epoch results are relatively lower than the original reported performance. Nevertheless, all experiments are done under the same setting for fair comparisons.
  
  Since BYOL uses a mean squared error loss without negative samples, we do not use any negative sampling and cosine margin in this entry. Instead, we only sample the positive pair after clustering and adopt the same loss in place of the MarginNCE loss for inter-image invariance learning. Following NPIDv2 and MoCov2 experiments, we set the number of clusters as 10,000 and the final loss weight λ=0.75. Although no negative pairs are sampled for inter-image branch, we empirically observe no collapsing solutions due to the stop-gradient operation introduced in BYOL.
  
  
  
   §.§ Results on Standard Benchmarks
  
  
  
  Following common practice in unsupervised representation learning <cit.>, we evaluate the quality of learned representations by transferring them to multiple downstream tasks.
  
  We perform experiments on a variety of datasets, focusing on four kinds of downstream tasks:
  
  image classification with linear models (Sect. <ref>), low-shot image classification (Sect. <ref>), semi-supervised learning (Sect. <ref>), and object detection (Sect. <ref>).
  
  Our evaluations cover two learning setups where the pre-trained network is either frozen as a feature extractor (Sect. <ref> and Sect. <ref>), or fully fine-tuned as weight initialization (Sect. <ref> and Sect. <ref>).
  
  
  
    §.§.§ Image Classification with Linear Models
  
  
  
  Setup.
  
  Following <cit.>, we freeze all the backbone parameters and train classifiers on representations from different depths of the network on three datasets, including ImageNet <cit.>, Places205 <cit.> and VOC07 <cit.>.
  
  For ImageNet and Places205, we train a 1000-way and 205-way fully-connected classifier, respectively, on the frozen representations using SGD with a momentum of 0.9.
  
  For ImageNet, we train for 100 epochs, with a batch size of 256 and a weight decay of 10^-4.
  
  The learning rate is initialized as 0.01, decayed by a factor of 10 after every 30 epochs.
  
  For Places205, we train for 14 epochs, with a batch size of 256 and a weight decay of 10^-4.
  
  The learning rate is initialized as 0.01, dropped by a factor of 10 at three equally spaced intervals.
  
  We report top-1 center-crop accuracy on the official validation split of ImageNet and Places205.
  
  For VOC07, we follow the same setup in <cit.> and train linear SVMs on the frozen representations using LIBLINEAR package <cit.>.
  
  We train on the trainval split of VOC07 and report mAP on the test split.
  
  Results.
  
  Table <ref> shows the results for the best-performing layer of each method.
  
  Our improved NPIDv2 baseline already performs well on the three datasets.
  
  However, InterCLR substantially outperforms NPIDv2, demonstrating the benefits of introducing inter-image invariance.
  
  Similar improvements are also observed in MoCov2 and BYOL.
  
  By building upon a stronger baseline in intra-image invariance learning, InterCLR outperforms previous unsupervised learners that are pre-trained within 200 epochs using a feasible 256 batch size on a standard ResNet-50 backbone, setting a new state of the art in this fair competition on all three datasets.
  
  Note that our 200-epoch BYOL-InterCLR has already yielded better performance than 1000-epoch BYOL, indicating that the performance efficiency of InterCLR is at least 5× than BYOL below 1000 epochs.
  
  When pre-trained for 1000 epochs, InterCLR still consistently outperforms its corresponding intra-image invariance learning baseline by clear margins.
  
  Both the higher performance efficiency and the final performance demonstrate that InterCLR helps learn generalizable representations faster and better.
  
  
  
  
  
    §.§.§ Low-Shot Image Classification
  
  
  
  Setup.
  
  Next, we explore the quality of learned representations when there are few training examples per category by transferring to the low-shot VOC07 classification task.
  
  Specifically, we vary the number of labeled examples in each class and train linear SVMs on the frozen backbone following the same procedure in <cit.>.
  
  We train on the trainval split of VOC07 and report mAP across five independent samples for each low-shot value evaluated on the test split of VOC07.
  
  Results.
  
  Table <ref> shows the final mAP results of different methods obtained with the averages of all low-shot values and all independent runs.
  
  InterCLR substantially outperforms its intra-image invariance learning counterpart.
  
  Fig. <ref> also provides the per-shot results pre-trained within 200 epochs.
  
  InterCLR improves upon all baselines and gradually bridges the gap to supervised pre-training when the number of labeled examples per class is increasing.
  
  
  
  
  
  
  
    §.§.§ Semi-Supervised Learning
  
  
  
  Setup.
  
  We perform semi-supervised learning on ImageNet following <cit.>.
  
  We randomly select 1% and 10% subsets of the labeled ImageNet training data in a class-balanced way.
  
  Then, we fine-tune our models on these two subsets.
  
  More specifically, we use the 1% and 10% ImageNet subsets specified in the official code release of SimCLR <cit.>.
  
  For both 1% and 10% labeled data, we fine-tune using SGD with a momentum of 0.9 and a batch size of 256 for 20 epochs, with the initial learning rate of backbone set as 0.01 and that of linear classifier as 1.
  
  The learning rate is decayed by a factor of 5 at 12 and 16 epochs.
  
  We use a weight decay of 5×10^-4 for 1% fine-tuning and 10^-4 for 10% fine-tuning.
  
  We report top-5 center-crop accuracy on the official ImageNet validation split.
  
  Results.
  
  As shown in Table <ref>, InterCLR again boosts the performance of all tested intra-image invariance learners by clear margins, especially when only 1% labeled data is available.
  
  We also observe that BYOL-InterCLR trained for only 200 epochs can achieve better results than BYOL trained for 1000 epochs.
  
  The 200-epoch results are even comparable with prior arts trained with a much larger compute.
  
  With 1000-epoch pre-training, InterCLR further improves the performance, performing on par (1% labels) or even better (10% labels) than state-of-the-art results in this more challenging low-data regime.
  
  
  
    §.§.§ Object Detection
  
  
  
  Setup.
  
  Following <cit.>, we use Detectron2 <cit.> to train the Faster-RCNN <cit.> object detection model with a R50-C4 backbone <cit.>, with BatchNorm tuned.
  
  Specifically, we use a batch size of 2 images per GPU, a total of 8 GPUs and fine-tune ResNet-50 models for 24k iterations (∼23 epochs).
  
  The learning rate is initialized as 0.02 with a linear warmup for 100 iterations, and decayed by a factor of 10 at 18k and 22k iterations.
  
  The image scale is [480, 800] pixels during training and 800 at inference.
  
  Following <cit.>, we also search the fine-tuning learning rate for BYOL experiments.
  
  We fine-tune all layers on the trainval split of VOC07+12, and evaluate on the test split of VOC07.
  
  We use the same setup for both supervised and unsupervised models.
  
  Results.
  
  The results averaged across five runs are summarized in Table <ref>.
  
  InterCLR consistently outperforms all examined intra-image invariance learning baselines pre-trained for 200 epochs, with the largest improvement observed in BYOL (+1.3%).
  
  InterCLR also surpasses the supervised pre-training by 1.4%.
  
  It should be noted that our 200-epoch results (, MoCov2-InterCLR and BYOL-InterCLR) are even better than other results obtained with much longer pre-training epochs.
  
  
  
   §.§ Empirical Study
  
  
  
  
  
  We conduct a comprehensive empirical study for inter-image invariance learning in this subsection.
  
  To perform a large amount of experiments needed for the study, we adjust the experimental setting to train each model with fewer (4,096) negative samples or for fewer (100) epochs while keeping the other hyper-parameters in Sect. <ref> unchanged[In practice, we observe that adopting 4096 negative samples or 100 pre-training epochs does not affect our empirical observations.].
  
  Specifically, when studying the three main aspects (, pseudo-label maintenance, sampling strategy, and decision boundary design) in Sect. <ref>, we train with 4,096 negative samples for 100 epochs and perform a set of experiments progressively:
  
  for pseudo-label maintenance study, we use random negative sampling and zero-margin decision boundary;
  
  for sampling strategy study, we use online pseudo-label maintenance and zero-margin decision boundary;
  
  for decision boundary study, we use online pseudo-label maintenance and semi-hard negative sampling.
  
  When conducting the further analysis in Sect. <ref>, we adopt exactly the same setting as in Sect. <ref> (, 200-epoch pre-training with 16,384 negative samples), except for the ablation on loss weight λ in Eq. (<ref>), where we train with 4,096 negative samples and use online pseudo-label maintenance, random negative sampling as well as zero-margin decision boundary.
  
  As a result, we obtain relatively lower performances in most cases.
  
  However, these experiments aim at better understanding the properties of InterCLR and provide a useful guidance on how to design each component for inter-image invariance learning.
  
  Throughout the section, we take NPIDv2-InterCLR as the prototype for empirical study and use the standard benchmarks from Sect. <ref> to measure the quality of learned representations.
  
  
  
  
  
    §.§.§ Main Observations
  
  
  
  Observation 1: Online labels converge faster and perform better than offline labels.
  
  We compare the effect of two investigated pseudo-label maintenance mechanisms (, online mini-batch k-means vs. offline global k-means) on the learned representations.
  
  As shown in Fig. <ref>(a), we observe that online labels achieve faster convergence and better performance than offline labels during the training process.
  
  Due to the expensive computational cost, the sparsely updated pseudo-labels of offline global k-means are rather stale relative to the rapidly updated network. Thus, offline labels are less reliable than instantly updated online labels as the latter is simultaneously undertaken along with the network update.
  
  This suggests the superiority of maintaining pseudo-labels online for inter-image invariance learning.
  
  Observation 2: Semi-hard negative sampling is more reliable and unbiased.
  
  We then study the importance of negative sampling strategies.
  
  Fig. <ref>(b) compares four negative sampling strategies discussed in Sect. <ref>.
  
  Interestingly, we find that semi-hard negative sampling achieves the best performance, while hard negative sampling is even worse than the naïve random sampling strategy.
  
  In the context of unsupervised inter-image invariance learning, the hard negatives are likely to have false labels due to the noisy cluster assignments, , they may actually be positive pairs. Solely mining hard negatives will intensify the bias during training. In contrast, semi-hard negative sampling provides the chance to correct these false negatives while maintaining the hardness of sampled negatives at the same time.
  
  In conclusion, the observation reveals that different from what is commonly adopted in supervised learning, hard negative mining is not the best choice in the unsupervised scenario; on the contrary, randomly sampled negatives within a relatively larger nearest neighbor pool are more reliable and unbiased for unsupervised inter-image invariance learning.
  
  
  
  
  Observation 3: Positive decision margin for “Intra” and negative decision margin for “Inter”.
  
  We study the impact of decision boundary using different cosine margins.
  
  Specifically, we perform a set of margin experiments for each branch by setting the margin of the other branch as 0.
  
  For the intra-image branch shown in Fig. <ref>(a), using a positive margin improves the performance, while a negative one degrades the performance.
  
  Since the labels derived from image indices are always correct, it is beneficial to adopt a positive margin to further strengthen the decision boundary.
  
  Hence, it is necessary to pursue highly discriminative features for intra-image invariance learning, which is in accordance with the case of supervised learning.
  
  However, the opposite phenomenon is observed for the inter-image branch as shown in Fig. <ref>(b).
  
  Using a negative margin (the best performance is observed when m=-0.5) improves upon zero margin, while a positive one fluctuates and even degrades the performance.
  
  We attribute this phenomenon to the inaccurate nature of inter-image invariance learning. The pseudo-labels derived from the inter-image branch are evolving during training and are noisy at initial epochs, , they usually contain false positive/negative samples. Adopting a positive margin will intensify the faulty cases and then the situation will never be corrected. In contrast, a negative margin can mitigate the inaccuracy during training, leading to a more stable evolution of pseudo-labels.
  
  Therefore, rather than solely pursue discriminative features, it is beneficial to design a less stringent decision boundary for inter-image invariance learning.
  
  
  
    §.§.§ Further Analysis
  
  
  
  Effect of loss weight λ.
  
  The benefits of inter-image invariance brought by InterCLR have been fully demonstrated on various downstream tasks in Sect. <ref>.
  
  Here, we further analyze the effect of λ that controls the weight between two MarginNCE losses in Eq. (<ref>).
  
  For λ=1, our framework degenerates to a typical form of intra-image invariance learning in <cit.>.
  
  For λ=0, only the inter-image invariance learning branch is retained.
  
  InterCLR benefits from the combination of two kinds of image invariance learning as shown in Fig. <ref>, with the best trade-off obtained when λ=0.75.
  
  
  
  
  
  
  
  
  Effect of the number of clusters.
  
  We study the effect of the number of clusters on the quality of learned representations. The results are presented in Fig. <ref>(a). Over-clustering tends to be beneficial for inter-image invariance learning while the performance gradually saturates around 10,000, which is 10 times of the annotated number of original ImageNet classes.
  
  Effect of negative samples.
  
  Prior works <cit.> have shown that intra-image invariance learning benefits greatly from a larger number of negative samples. We examine whether this trend still holds for inter-image invariance learning. As shown in Fig. <ref>(b), increasing the number of negatives from 4,096 to 16,384 only has marginal benefits on the final performance. Using a larger number of negative samples (, 65,536) even degrades the performance. This demonstrates that inter-image invariance learning is more robust to the number of negative samples, , we can mitigate the reliance of large negative samples for intra-image invariance learning by incorporating the inter-image branch.
  
  Computational cost.
  
  In Table <ref>, we compare the memory and time cost of intra- and inter-image invariance learning. Compared with the intra-image invariance learning baseline, , Intra, incorporating inter-image invariance learning inevitably increases the training cost. Nevertheless, maintaining iteration-based online labels, , Intra+Inter (online), is around × 1.3 faster than commonly adopted epoch-based offline labels, , Intra+Inter (offline).
  
  Feature space visualization.
  
  Apart from quantitative results, we also visualize the learned representations in the t-SNE <cit.> embedding space.
  
  As shown in Fig. <ref>, the “Intra-image only” model merely groups the same image in different augmentations together; however, different images are separated even though they are in the same class.
  
  The “Inter-image only” model shortens the distance between images in the same class; however, outliers emerge.
  
  The “Intra- & inter-image” model well inherits the advantages from above two models, resulting in a more separable feature space.
  
  Negative sample visualization.
  
  Fig. <ref> visualizes some example negative samples with different negative sampling strategies (, hard, semi-hard, random, and semi-easy) using the learned instance features during training.
  
  For “hard negative” sampling, we observe many false negatives, , they actually have the same category as the input image. For “random negative” and “semi-easy negative” sampling, although no false negatives are observed, the sampled negatives are largely visually dissimilar to the input and are much easier to be distinguished. In contrast, “semi-hard negative” sampling reduces the false negative cases while maintaining the difficulty of sampled negatives at the same time.
  
  KNN visualization.
  
  To further understand the benefits of inter-image invariance learning, we visualize some top-10 nearest neighbors retrieved with cosine similarity in the embedding space using the features learned by InterCLR.
  
  As shown in Fig. <ref>, compared with its intra-image invariance learning baseline, , Baseline (Intra), InterCLR (Intra+Inter) retrieves more correct images.
  
  Besides, InterCLR (Intra+Inter) also achieves higher cosine similarity with the queries than its intra-image invariance learning counterpart.
  
  We observe this not only for the same correctly retrieved samples, but also for the whole 10 retrieved nearest neighbors: even the 10th nearest neighbor retrieved by InterCLR (Intra+Inter) obtains higher cosine similarity than the 1st nearest neighbor retrieved by the baseline.
  
  The aforementioned gap is due to the inherent limitation of intra-image invariance learning: only encouraging the similarity of different augmented views of the same image while discouraging the similarity of different images even though they should belong to the same semantic class.
  
  This further demonstrates the benefits of inter-image invariance brought by InterCLR.
  
  
  
  
  
  
  § CONCLUSION
  
  
  
  
  
  
  
  
  In this work, we have investigated inter-image invariance learning from different perspectives and shown the effect of different design choices, pseudo-label maintenance, sampling strategy, and decision boundary design.
  
  By combining our observations, we introduced a unified and generic framework, InterCLR, for unsupervised intra- and inter-image invariance learning.
  
  With this framework, we consistently improve over existing state-of-the-art intra-image invariance learning methods on multiple standard benchmarks.
  
  We hope our empirical study can provide useful baselines and experience for future research.
  
  Limitations.
  
  This study mainly targets at a low-resource pre-training regime, , a batch size of 256 for 200 epochs. We have shown that InterCLR can also benefit from longer pre-training epochs as many prior works do in the paper. Pre-training with a larger batch size may further improve the performance. However, it usually comes at the cost of huge computational resources that are inaccessible to many researchers, which is not the core of this paper. We wish to highlight that our work wins with its merit of comprehensively revealing how to deal with the inaccurate nature of inter-image invariance learning from different perspectives instead of a large compute. It should be well noted that our 200-epoch InterCLR has already performed on par or even better than many prior large-batch long-epoch results attained with a much larger compute (, SimCLR <cit.>, SwAV <cit.>, BYOL <cit.>, and Barlow Twins <cit.>) on various downstream tasks. This demonstrates the appealing performance efficiency of InterCLR, , one can get higher performance after training for fewer epochs, which is indeed important considering the high compute cost of existing unsupervised learning methods. Meanwhile, we choose representative intra-image invariance learning baselines and transfer learning benchmarks to examine the generality of our framework. More baselines and benchmarks can be further studied. We leave these explorations to future work.
  
  Data availability statements.
  
  The datasets that support the findings of this study are all publicly available for the research purpose.
  
  
  This study is supported under the RIE2020 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). The project is also supported by Singapore MOE AcRF Tier 2 (T2EP20120-0001), the Data Science and Artificial Intelligence Research Center at Nanyang Technological University.
  
  
  spbasic
  
  
  
  
  