Beyond Single Instance Multi-view Unsupervised Representation Learning

Recent unsupervised contrastive representation learning follows a Single Instance Multi-view (SIM) paradigm where positive pairs are usually constructed with intra-image data augmentation. In this paper, we propose an effective approach called Beyond Single Instance Multi-view (BSIM). Specifically, we impose more accurate instance discrimination capability by measuring the joint similarity between two randomly sampled instances and their mixture, namely spurious-positive pairs. We believe that learning joint similarity helps to improve the performance when encoded features are distributed more evenly in the latent space. We apply it as an orthogonal improvement for unsupervised contrastive representation learning, including current outstanding methods SimCLR [7], MoCo [22], and BYOL [21]. We evaluate our learned representations on many downstream benchmarks like linear classification on ImageNet1k and PASCAL VOC 2007, object detection on MS COCO 2017 and VOC, etc. We obtain substantial gains with a large margin almost on all these tasks compared with prior arts.


Introduction
Unsupervised representational learning is now on the very rim to take over supervised representation learning. It is supposed to be a perfect solver for real-world scenarios full of unlabeled data. Among them, self-supervised learning has drawn the most attention for its good data efficiency and generalizability.
Self-supervised learning typically involves a proxy task to learn discriminative representations from self-derived labels. Among all manners of these proxy tasks [32,17,35,28,12], instance discrimination [48,30], known as contrastive representation learning, has emerged as the most effective paradigm. Its subsequent methods [58,22,7,21,40] have greatly reduced the gap between unsupervised and su- Note xi, xj and x k are different instances. Green lines link the positive pairs while red for negative. Circles show decision boundaries (same color for the same class). Instance discrimination narrows the boundary and pushes away all instances. Clusteringbased methods might cluster wrong instances (e.g. yellow and blue) due to a shortcut defect. BSIM mixes instances (where hue indicates the ratio) to construct spurious positive pairs, e.g., (xij, x i ) and (xij, x j ). Hence, it encourages contrastive competition among instances, thus, better at learning representation for both inter and intra classes.
Since different instances are continuously driven apart, SIM prevents itself from characterizing the relations among different instances from the same class, as opposed to supervised classification. Meanwhile, clustering-based self-supervised methods [3,54] alternate feature clustering with learning to capture similarities among different instances. These methods avoid the intrinsic weakness of instance discrimination, but suffer from a so-called 'shortcut' problem, i.e., when two instances are occasionally grouped into a cluster, their similarity will be further enhanced. As a result, the training easily drifts into trivial solutions, e.g., merely grouping images in similar color or texture.
In view of the fact that instance discrimination pushes apart different instances indistinguishably as shown in Figure 1 (a), and clustering-based methods are easily trapped in shortcut issues as shown in Figure 1 (b), we are motivated to to explore a new paradigm to distinguish both intraclass and inter-class instances. In this work, we propose BSIM to learn better representations that capture high-level inter-image relations, which also potentially avoid the shortcut issues. To make the minimal modification from previous works, BSIM shares similar pipelines to SimCLR [7], MoCo [22], and BYOL [21], while focusing on a new way to construct positive pairs. Specifically, as shown in Figure 1 (c), BSIM first creates mixtures among instances by proportion, e.g., x ij that mixes λ of x i and (1 − λ) of x j , where λ ∈ (0, 1) obeys a Beta distribution. Different from instance discrimination that constructs positive pairs between two views from the same image, BSIM makes use of mixed views x ij to create what we call spurious-positive pairs (x ij , x i ) and x ij , x j . The optimization also proportionally takes λ in to account for computing the losses. The interaction of spuriouspositive pairs compete for an equilibrium state when grouping intra-instance and inter-instance views, modulated by the distribution of λ 1 . Meantime, negative pairs keep pushing away different instances. As a result, BSIM encourages contrastive competition among instances, leaning towards exploring higher-level inter-image relations. Since BSIM does not maintain dynamically changing pseudo labels as clustering-based methods, the shortcut issue is naturally avoided.
Our contribution is threefold.
• We propose a novel paradigm, called BSIM, to encourage contrastive competition among instances for higher-level representation learning.
• BSIM is a plug-in approach with minimal modification to existing methods based on instance discrimination.
• BSIM achieves state-of-the-art performance in a large body of standard benchmarks.

Related Work
Self-supervised learning based on contrastive loss. Early methods focus on devising proxy tasks to either reconstruct the image after transformations [28,35,57], or predict the configurations of applied transformation on a single image [11,13,32,17]. Till recently contrastive loss based approaches [22,7,21,4] emerge as the mainstream paradigm, which features two components: the selection of positive or negative examples and the contrastive loss design. This routine leverages different augmented views of an image to construct positive pairs, while deeming other images as negative samples. Particularly, SimCLR [7] produces positive and negative pairs within a mini-batch of training data and chooses In-foNCE [33] loss to train the feature extraction backbone. It requires a large batch-size to effectively balance the positive and negative ones. MoCo [22] makes use of a feature queue to store negative samples, which greatly reduces high memory cost in [7]. Moreover, it proposes a momentum network to boost the consistency of features. BYOL [21] challenges the indispensability of negative examples and achieves impressive performance by only using positive ones. A mean square error loss is applied to make sure that positive pairs can predict each other.
Besides, carefully designed augmentations to build positive pairs are proven to be critical for good performance [7,40,8,31,39,1,19,44], because appropriate augmentations modulate the distribution of positive examples in the feature space. SwAV [4] obtains the state-of-the-art unsupervised performance by using a mixture of views in different resolutions in place of two full-resolution ones. In the meantime, some researches study the role of hard negative examples [25,26,9,49,46]. However, all the above approaches try to push each image instance away from each other by regarding them as its negative samples. Is it possible to model the vicinity relation by measuring that distance quantitatively? To our best knowledge, this problem is rarely studied in the field of self-supervised learning and BSIM is aimed to bridge this gap. Mixture as a regularization technique in supervised learning. Mixture of training samples like Mixup [56], CutMix [51], and Manifold Mixup [43] has been proved to be a strong regularization for supervised learning, based on the principle of vicinal risk minimization [6]. It is designed to model vicinity relation across different classes other than vanilla data augmentation tricks that only considers the same class. CutMix [51] debates that Mixup introduces unnatural artifacts by mixing the whole image region while Cutout [10] might pay attention to less discriminative parts. CutMix claims to effectively localize the two classes instead. [41] argues that mixing images is akin to mixing sounds [42] for CNNs, although not easily perceptible for humans. Other than deeming it as vanilla data augmentation that adds data variation, they consider it as an enlargement of Fisher's criterion [16], i.e., the ratio of the between-class scatter to the within-class scatter, and it regularizes positional relationship among latent feature distributions. Furthermore, [38] notices that label smoothing during mixup training has a calibration effect which regularizes over-confident predictions. Mixture in contrastive learning Different from the wide use of mixture as a useful regularization in supervised learning, how to use it in unsupervised learning remains to be an open problem. It seems that this preferred regularization in supervised learning is inherent incompatible with the contrastive learning framework: drawing near multiple augmented views of an image and in the meantime pushing away from the others, which we call Single Instance Multiview (SIM) for simplicity. A recent work [37] utilizes mixture as a pure data augmentation and design a complex procedure of mixing different images to soften the output predict space. Therefore, it still falls in the existing paradigm (SIM) without changing loss designs. Feature-level mixture is utilized as a regularization to perform hard example mining [26], which can boost discrimination. Other than using it as an extra augmentation, we focus on image-level mixture to define the spurious-positive examples and quantify how close two images are. As a result, we develop a new paradigm for contrastive learning.

Methodology
The central principle of contrastive learning is to encode semantically similar views (positive pairs) into latent representations that are close to each other while driving dissimilar ones (negative pairs) apart. A major question revolves around how to effectively synthesize positive and negative pairs given a dataset of i.i.d samples as raised by [40]. Another question engages the design of contrastive loss. In the following, we discuss each in details for our approach BSIM.

Spurious-Positive Views From Multiple Images
Given a set of images D, two images x 1 , x 2 ∼ D sampled uniformly from D, and two image augmentation distributions of T and T (whether T and T are the same depends on different methods), We generate a new example x 1,2 by mixing T (x 1 ) and T (x 2 ). Specifically, x 1,2 borrows λ region from T (x 1 ) and remaining (1 − λ) part from T (x 2 ). Thus, x 1,2 has two spurious-positive examples, i.e., T (x 1 ) and T (x 2 ). These images are encoded by a neural network to extract high-level features, followed by a project head g that maps the representation to a space ready to apply contrastive loss. The projection head function is often sim sim implemented as a simple MLP network. The distance between the new example and its parents is controlled by λ. BSIM uses a popular and handy option to generation λ from Beta distributions, i.e., λ ∼ β(α, α), where α is a hyper-parameter. It is evident that it would degrade to the single-instance multi-view case if λ is always 0 or 1 when α → 0. That being said, BSIM is a generalization of SIM so that previous SIM methods reside within our larger framework.

Loss Functions of BSIM
It's intuitive to change loss functions since spuriouspositive examples are introduced. For example, it's unreasonable to assign a new instance generated by half mixing two images (a dog and a cat, λ = 0.5) to a dog or cat. Human would easily tell this image is half cat and half dog. This means its projected feature in the high-dimensional latent space should be nearby a dog, as well as a cat, but much far from an orangutan. This motivates us to design a particular loss for BSIM. Specifically, we proceed by studying three recent and popular frameworks SimCLR [7], MoCo [22] and BYOL [21] and provide adapted losses. We believe it can also be easily incorporated into other methods. In order to make our paper more readable, we roughly follow the same naming conventions.
Adapt SimCLR to SimCLR-BSIM. SimCLR uses a single augmentation distribution, i.e. T and T are identical herein. The encoder network f encodes x 1,2 as f (x 1,2 ). Note x 1,2 should show similarities with x 1 as well as x 2 , which is measured by the sim function in the projected z space. We follow the definition in [7] for the similarity function as sim(z i , z j ) = z i z j /( z i z j ). We use λ to regularize these similarities and the matching loss in this case can be formulated as, Similarly, we can formulate i if we use x 1,2 as the anchor.
In this way, the NT-Xent [7] loss is defined by the summation of each individual loss within the mini-batch data of size N as follows, (2) SimCLR [7] has 2N positive pairs and 2N (N − 1) negative ones in total at each iteration. Whereas, our method includes 4N spurious-positive pairs, i.e., ( The proposed method is summarized in the Algorithm in supplementary materials and depicted in Figure 2.
Adapt MoCo to MoCo-BSIM. MoCo allows smaller batch sizes to learn good representations. We produce the query q by forwarding the mixed image controlled by λ. We illustrate the procedure in Figure 3.
where k λ + and k 1−λ + represent the corresponding key of images that produced the mixture respectively, and k i are the keys in the current queue.
Adapt BYOL to BYOL-BSIM. BYOL-BSIM generates two augmented views x 1 t (x 1 ) and x 1 t (x 1 ) from x 1 by applying respectively image augmentations t ∼ T and t ∼ T . Following the same procedure, we produce x 2 and x 2 . Then we produce a new image x 1,2 by λ-based mixture x 1 and x 2 through cutmix. The online network outputs y θ f θ (x 1,2 ) and the projection z θ g θ (y ). The target network outputs two 2 -normalized projectionsz 1 ,z 2 from x 1 and x 2 . Since BYOL does not rely on negative samples, we can propose two contrastive loss functions.
The first approach is similar to MoCo-BSIM. We sum up the MSE loss between the projection of the mixed image Figure 3. Applying BSIM to MoCo. Given a batch of images, we mix them in a pair-wise manner to produce xq (only one mixture is shown for simplicity). We use CutMix in this example. We encode the mixed images as query q and the whole current batch as keys k. We thus generate spurious-positive pairs from q and k, and the current queue is used as negative samples. The encoder network is updated according to loss Lq in Equation 3. The key network follows a momentum update as in MoCo. and its parents by the mixture coefficient λ. Formally, the loss is formulated as: Note z i,ξ and z j,ξ mean the projection of the generated representation of x i and x j by the target network. The other approach is designing a linear combination of embedding as a target. The mixed target is z 1,2 = λz 1 + (1 − λ)z 2 . And the final target isz 1,2 z 1,2 / z 1,2 || 2 . The loss is reformulated as We obtain L θ,ξ by using x 1 and x 2 as the input of online network. Note that BYOL doesn't rely on negative samples. The normalized projection is on the sphere of a unit ball in the high dimensional space, shown in the supplementary materials.
We compare these two designs and neglect some indices to make it clear. z, z 1 and z 2 are normalized. The first approach has the form of xi xj x i x j . Applying BSIM to BYOL as in Eq. 5. The above blue region is the online network, the below red one is the target network. The weight ξ is the moving average of θ. After learning, only f θ will be used to generate y θ .
The second one can be formulated as Problem 6 and 7 has the same local minimum z = 2 . However, because we use the back propagation to solve the optimization, both methods have different behaviors. Compared with Eq. 6, the gradients in Eq. 7 have the same direction but are adaptively scaled by then two loss designs behave similarly. However, when z 1 and z 2 are not that close and λ is neither near 1 or 0, s helps to update z with a relative large step to catch the target with higher confidence. We make ablation study to compare these two approaches and find that the second one, i.e. Eq.7 performs better. Therefore, we recommend it as the default choice.

BSIM as a General Adds-on Approach
Apart from the discussed integration with SimCLR, MoCo and BYOL, we can also simply treat BSIM as an adds-on to SIM-based methods by a weighted summation of loss functions, where w 1 , w 2 ∈ (0, 1). We refer this approach as weighted-BSIM (WBSIM). When w 1 = 0, w 2 = 1, it is the conventional single instance multi-view approach. When w 1 = 1, w 2 = 0, it is BSIM. And when w 1 , w 2 = 0, it benefits from both SIM and BSIM.

Experiments
Setup For self-supervised pre-training, we generally follow the settings of compared methods. Please refer to the details in D. 2 Proof is given in the supplementary materials. Memory and training cost MoCo-BSIM adds no extra memory cost to MoCo where we simply replace query samples with mixed ones. Whereas the WBSIM version has to maintain the originally augmented query samples to compute MoCo's original contrastive loss. The detail is shown in Table 1.
The sampling process. For a mini-batch of samples with size N , theoretically, we can sample λ for N times to enrich the information. Consequently, we can construct the loss by using N different weighted items. However, this process can hardly be implemented efficiently in the PyTorch framework. Instead, we make use of the 8 GPU workers and set different seeds at the beginning of training. This approach is quite efficient and possesses rich mixtures.
All the experiments are done on Tesla V100 machines with 8 GPUs. We use apex 3 to allow a larger batch size of 2048 for the BYOL experiment and accumulate gradients to simulate a batch size of 4096.
Comparison with mixture-based approaches. Shen et al. [37] propose a somewhat complicated iterative mixture strategy exploiting Mixup [56] and CutMix [51] to generate a weighted mixture of samples. The mixture can be considered a weakened version of the original images which is harder to recognize, hence rendering flattened predictions. As suggested from the label-smoothing perspective, it is meant to suppress incorrect response on hard negative samples. However, image mixtures are used as-is, i.e., it learns the mixture-to-mixture similarity when combined with MoCo [22], while we learn the similarity between the mixture and its parents (forming spurious-positive pairs). This poses a fundamental difference as the loss has to be redesigned accordingly. Notice [37] also designs a too complex approach to strive for semantical harmony by decaying the 'context' image while not necessary in our case.

Evaluation on Linear Classification
We evaluate our method on the linear classification benchmark by freezing the weights of ResNet50. Unless  otherwise specified, all methods are compared based on 200 epochs trained on the ImageNet dataset. Linear SVM classification on VOC2007. Following [34,20,54], we use the res4 block (notation from [18]) of ResNet-50 as the fixed feature representations and train SVMs [2] for classification using LIBLINEAR package [15]. We train on the trainval split of the VOC2007 dataset [14] and report the mean Average Precision (mAP) on the test split by 3 independent experiments. All methods are evaluated using the same hyper-parameters as in [20]. The result are shown in Table 2.
In most cases, BSIM consistently boosts the baselines by about 1% mAP. Particularly, BYOL-BISM boosts BYOL by a clear margin: 1.4% mAP. BYOL-BSIM300 outperforms the supervised pre-trained baseline with 0.4% higher mAP. BYOL-BSIM trained for 200 epochs is comparable to BYOL300 trained for 300 epochs. Moreover, WBSIM further boosts the performance. MoCoV2 benefits 1% higher mAP from BSIM, and an extra 0.6% higher one from WB-SIM, indicating that BSIM is complementary to SIM. It is worth emphasizing that BYOL-BSIM300 makes the first time that self-supervised learning surpasses the supervised counterpart in classification tasks with only 3x number of pre-training epochs.
Low-shot classification on VOC2007. We further evaluate the low-shot performance when each category contains  Table 3. Linear classification on ImageNet (top-1 center-crop accuracy on the validation set) using ResNet50. † : reproduced results. Best gains are indicated within brackets. much fewer images. Following [45,20,54], we use seven settings (N=1, 2, 4, 8, 16, 32, 64 and 96 positive samples), train linear SVMs on the low-shot splits, and report the test results across 3 independent experiments, shown in Table 2. BSIM also helps all baselines to achieve better performance by substantial margins. It's interesting to see that BYOL-BSIM300 gradually bridge its gap from the supervised baseline. When the number of the training set is more than 64, it's comparable to the supervised version.
Linear classification on ImageNet. We follow the linear classification protocol in [22] where a linear classifier is appended to frozen features for supervised training. As for MoCo, we train 100 epochs using SGD with a batch size of 256. The learning rate is initialized as 30 and scheduled by 0.1 × at epoch 30 and 60.
BYOL [21] adopts a quite different setting. To reproduce the baseline results, we train it for 90 epochs using the SGD optimizer with 0.9 momentum. Besides, we use L2 regularization with 0.0001 and a batch size of 256. The initial learning rate is 0.01 and scheduled by the 0.1 × at epoch 30 and 60. We also study the level of feature representation from five stages (F1-F5) of ResNet50 and from the average pooling layer since different methods vary in their best-performing layers. The results are shown in Table 3, where the performance of the competing methods are extracted from [55].

Evaluation on Semi-supervised Classification
For semi-supervised training, we use the same split 1% and 10% amount of labeled ImageNet images as done in [52,7]. We follow [7,24,27,53] Table 4. BSIM improves the baselines by significant margins, especially when the amount of available labels is small. MoCoV2-BSIM obtains 44.3% top-1 accuracy, which is 5.2% higher than MoCoV2. Although BYOL-BSIM is only trained for 200 epochs, it achieves comparable results as BYOL1000. WBSIM300 can further boost the performance to the stateof-the-art 57.2%. Specifically, it obtains 57.2% top-1 accuracy using 1% labeled data, which is about 4% higher than BYOL1000. When we collect more data (10%), BYOL-WBSIM outperforms BYOL1000 with about 2%. We believe combining SIM and BSIM learns better representations.

Evaluation on PASCAL VOC Object Detection
Following the evaluation protocol by [22] where ResNet50-C4 (i.e., using extracted features of the 4-th stage) is used as the backbone and Faster-RCNN [36] as the detector, we benchmark our method for the object detection task on the VOC07 [14] test set. All models are finetuned on the trainval of VOC07+12 dataset for 24k iterations. We use Detectron2 [47] like MoCo did. Results are reported in Table 5, which are mean scores across five trials as [8] using the COCO suite of metrics [29]. Combined with BSIM, BYOL achieves 1.4% higher AP and 0.8% higher AP 50 .  Table 6. Object detection and instance segmentation fine-tuned results on COCO2017 dataset using 2× schedule. All methods are pre-trained for 200 epochs except that SwAV800 is trained for 800 epochs.

Evaluation on COCO Objection Detection and Instance Segmentation
We also follow the evaluation protocol by [22] for the object detection and instance segmentation task on COCO2017 [29]. Specifically, we use the ResNet50-C4 Mask R-CNN framework [23] and follow 2× schedule [18] as [22] since this setting can make fairer evaluations. All models are fine-tuned on the train2017 set and evaluated on val2017. We report the bounding box AP and mask AP on COCO in Table 6. Combined with BSIM, all methods benefit about 0.2 AP, which again verifies its effectiveness and generalization.

Different Mixture Strategies
Mixup [56] can be another option to generate mixtures at image level. Whereas we don't apply it because it is less natural, even humans cannot easily tell the mixture coeffi- We also compare their performance using VOC object detection under the same metrics in section 4.3. The result is shown in Table 8. Mixup fails to improve the performance of its baseline, which disables mixtures. In contrast, Cut-Mix mixture can improve the detection performance. This is consistent with the supervised counterpart [51].

Disable Loss Assignment
A question for loss design is regarding CutMix as an extra data augmentation approach under the single instance multi view framework. We test it using MoCo-V2 because it takes less time. Specifically, we use CutMix (α = 1.0) as an extra augmentation operation without our loss designs and train it using the same setting as above. The classification result on VOC2007 is given in Table 7. MoCoV2-BSIM easily outperforms this group, which emphasizes the importance of BSIM.

Sensitivity on α
We further analyze the performance sensitivity of α. Regarding the intensive resource cost, we report the SVM and low-shot SVM results in Table 9 using MoCo-V2. We keep the same pre-training setting. The distribution from group α = 0.75 performs best. The performance keeps stable when α > 0.5. However, it drops severely once α → 0 when it degenerates to MoCo-V2.  Table 9. Performance sensitivity on α using MoCoV2-BSIM. The classification result is averaged across 5 independent experiments. When α < 0.01, MoCoV2-BSIM can be regarded as MoCoV2 approximately which achieves 83.8% mAP on SVM.

Discussion
Why does BSIM improve discrimination? A mixture is close to the decision boundaries in instance discrimination task where neural networks are normally less certain about. In instance discrimination, the decision boundaries keep being separated, leaving some area crowded while others sparse, which is unfavorable for learning high-level inter-image relations. In BSIM, when learning λ-balanced similarities between competing spuriouspositive pairs, the network encourages contrastive competition among instances to occupy the area near decision boundaries. In this way, we hypothesize that the network has to encode the latent representations more accurately like a ruler. As a result, the features has to scatter more evenly, especially for between-class areas that are harder to predict. We hypothesize this is the major magic that BSIM offers stronger discrimination capability.
Why does mixture as data augmentation fail? As mentioned in Table 7, simply adopting mixture methods as a data augmentation option severely degrades the performance. Regarding the mixed image as the same instance as the original one forces the network to expand the decision boundaries blindly. Consequently, the network might be trapped in shortcut solutions to group images in different classes indiscriminately.

Conclusion
In this paper, we propose BSIM, a novel self-supervised representation learning approach beyond the current instance discrimination paradigm. It makes minimal modification to existing instance discrimination methods such as SimCLR, MoCo and BOYL, while significant improving the performance on many downstream tasks. We justify the superiority of BSIM via analyzing the optimization behaviors when combined with different paradigms, which provides a new perspective in the field of contrastive representation learning. Extensive evaluation of our approach BSIM is conducted across various benchmarks. Being a simple and light-weight tool, it turns out to substantially enhance the performance and mostly achieves state-of-the-art results.

A. Index of Symbols
To facilitate readability, we give a list of notations in Table 10.

Symbol
Definition beta distribution λ sampled variable from β τ softmax temperature q query in the MoCo framework k λ + , k 1−λ + key of the mixed samples k i key in the current queue f (·), f θ encoder network g(·), g θ projection head q θ predictor i contrastive loss with x i,j as an anchor L N T −Xent (λ) loss of SimCLR-BSIM L q loss of MoCo-BSIM L θ,ξ loss of BYOL-BSIM L W BSIM loss of weighted BSIM Table 10. List of symbols used throughout the paper.

B. Proof
In this section, we give a proof that the loss of the second approach of BYOL-BSIM has the same local minimum as the first one. To do so, we need to solve the minimum as follows, where z 1 , z 2 are known normalized constant. λ lies in [0, 1].
Proof. We apply method of Lagrange multipliers to solve this constrained problem.
nels and a τ of 0.07. As for MoCoV2, we use two FC layers (2048,2048,128) to perform projections and a temperature coefficient of 0.2. BYOL-BSIM Data augmentation is the same as [21]. We follow [21] for the default hyper-parameters. Note that [21] states they prefer 300 epochs to make comparisons, we also conform to it to be consistent. Since many methods report their performance on 200 epochs, we also add an extra setting of 200 epochs to make fair comparison. To differentiate these two versions, we use 200 by default and name the BYOL300 for the former. Specifically, we also optimize LARS [50] with weight decay 1.5 · 10 6 . We set the initial learning rate 3.2 and use a batch size of 4096. The target network has an exponential moving average parameter τ = 0.996 and increased to 1. We use Equation 5 as the default loss and Equation 4 for the ablation.

E. More Discussions
We extend the discussions about the working mechanism of BSIM. As mentioned in the Discussion section in the main text, BSIM's latent space is less crowded to facilitate discrimination. To better illustrate this benefit, we pick 32 images and construct their mixtures and map their latent representations via TSNE, see Figure 6. It turns out that BSIM works like a ruler that measures how far a mixed instance should be from its parents. For instance, 6 25 (see Figure 5) is distant from both 6 and 25 in SIM, while it is closer to 6 in BSIM. It is also evident that the latent space is evenly spaced in BSIM than SIM.
Since BSIM is more accurate in measuring the latent distance (i.e. across different instances other than the same instance), instance of the same class are drawn more nearer while those of different classes are pushed apart. This is shown in Figure 7. We sample three times to demonstrate that BSIM in general gives cleaner intra-class representation, while SIM has a more intertwined one. This proves that learning the distance to the mixture helps improve classification by generating a better latent representation.  Figure 8 shows the difference of mixed images between Mixup and CutMix, where the latter is perceptually more natural. Figure 9 illustrates a schematic view of latent sphere where the mixed representation is normalized on the surface. Figure 10 gives the second implementation of BYOL-BSIM. Figure 11 depicts the beta distribution given different α, where we choose α to have a uniform distribution. Figure 6. The representation of 2D embeddings using TSNE. We use 32 images and construct the mixture using i-th and 31 − i-th images. The mixed embedding is marked by i 31 − i. Without BSIM, the embedding cannot make good use of the space (i.e. more crowded). Left: SIM, Right: BSIM. Notice SIM pushes a simple mixture 6 25 (red) distant from others, leaving a narrow space and a large amount of unused space. Besides, 12 19 and 19 12 (green) should be close but are pushed too far. In contrast, in BSIM all instances are scattered quite evenly, while mixed instances are also better placed near its parents. . Schematic view of the unit ball in high-dimensional space. Based on Eq 7, the target embedding of the mixed image is z i,j , which is formed by normalizing z i,j (dashed in green) to the sphere of the unit ball. z i,j is obtained by the λ controlled linear interpolation between z i and z j .

F. List of Additional Figures
xi xj x i x j x i,j y θ z θ qθ(z θ ) x i y i,ξ z i,ξ x j y j,ξ z j,ξ L i,θ,ξ L j,θ,ξ