
\section*{Appendix}

\section{Additional visualization}

\noindent\textbf{Correspondence pairs.}
Figure~\ref{fig:correspondence_supp} visualizes more top-ranked region correspondence pairs discovered by ORL in Stage 2. The high-response region pairs are usually objects or object parts.

\begin{figure}[ht]
	\centering
	\includegraphics[width=\linewidth]{correspondence_supp}
	\caption{\textbf{Top-ranked region correspondence discovered by ORL in Stage 2.} We show one discovered object-instance pair per image pair for clarity.}
	\label{fig:correspondence_supp}
\end{figure}

\noindent\textbf{Attention maps.}
Figure~\ref{fig:attention_supp} visualizes more attention maps generated by BYOL and ORL. As analyzed in Section~\ref{subsec:visualization}, although image-level BYOL can generate relatively high-quality attention maps focusing on some of the objects in scene images, our object-level ORL can activate more regions that contain objects and produce more accurate object boundary in the heatmap than BYOL.

\begin{figure}[ht]
	\centering
	\includegraphics[width=\linewidth]{attention_supp_v2}
	\caption{\textbf{Attention maps generated by BYOL and ORL.} ORL can activate more object regions and produce more accurate object boundary in the heatmap than BYOL. Best viewed with zoom in.}
	\label{fig:attention_supp}
\end{figure}

\section{Training details for transfer learning experiments}

\noindent\textbf{Image classification with linear models.}
We follow the procedure in~\cite{goyal2019scaling,misra2020self} to train linear models on the frozen representations of ResNet-50. On VOC07, we train linear SVMs using the LIBLINEAR~\cite{fan2008liblinear} package on the \texttt{trainval} split and report mAP on the \texttt{test} split. For linear evaluation on other datasets (ImageNet, Places205, and iNaturalist18), we train linear models with SGD using a batch size of 256, a learning rate of 0.01 decayed by 0.1 at three equally spaced intervals, a momentum of 0.9, and a weight decay of 0.0001. We train the linear models for 28 epochs on ImageNet, 14 epochs on Places205 and 84 epochs on iNaturalist18 to keep the number of training iterations roughly constant across all three datasets. We report top-1 center-crop accuracy on the validation split of ImageNet, Places205 and iNaturalist18.

\noindent\textbf{Low-shot image classification.}
We train linear SVMs on the frozen representations of ResNet-50 following the same procedure in~\cite{goyal2019scaling}. We train on the \texttt{trainval} split of VOC07 and report mAP on the \texttt{test} split. The results are averaged across five independent runs.

\noindent\textbf{Semi-supervised learning.}
We use the 1\% and 10\% ImageNet subsets specified in the official code release of SimCLR~\cite{chen2020simple}. We fine-tune the pre-trained ResNet-50 models with SGD for 20 epochs, using a batch size of 256, and a momentum of 0.9. We set the initial learning rate of backbone as 0.01 and that of linear classifier as 1. The learning rate is decayed by 0.2 at 12 and 16 epochs. We use a weight decay of 0 for 1\% fine-tuning and 0.0001 for 10\% fine-tuning.

\noindent\textbf{Object detection and segmentation.}
We use a Mask R-CNN model~\cite{he2017mask} with R50-FPN backbone~\cite{lin2017feature} implemented in Detectron2~\cite{wu2019detectron2}, following the same fine-tuning setup in~\cite{tian2020makes}. Specifically, we use a batch size of 2 images per GPU (a total of 8 GPUs), and fine-tune the ResNet-50 models for 90k iterations (standard 1$\times$ schedule). The learning rate is initialized as 0.02 with a linear warmup for 1000 iterations, and decayed by 0.1 at 60k and 80k iterations. The image scale is $[640, 800]$ pixels during training and 800 at inference.

\section{Reproducing related methods}

We re-implement each image-level contrastive learning method on COCO as faithfully as possible. The implementation details of our most essential image-level baseline, \ie, BYOL~\cite{grill2020bootstrap}, are provided in Section~\ref{subsec:details} (the similar pre-training settings are also adopted in~\cite{liu2020self} while our reproduction achieves \emph{better} results as shown in Table~\ref{tab:reproduction}). For MoCo v2~\cite{chen2020improved}, we follow~\cite{wang2020dense} to set the initial learning rate as 0.3. Other hyper-parameters use the default values in the original paper. For SimCLR~\cite{chen2020simple}, we use the largest batch size (\ie, 512) that fits in 8-GPU memory and linearly scale the learning rate with the batch size (\ie, $lr=0.3\times\text{BatchSize}/256$). Other hyper-parameters exactly follow the original paper.

\begin{table}[ht]
\centering
\small
\addtolength{\tabcolsep}{-2pt}
\caption{\textbf{Our reproduction vs. existing results for BYOL.} All are based on 800-epoch pre-training on COCO with ResNet-50. The results are object detection fine-tuned on COCO using Mask R-CNN R50-FPN with default 1$\times$ schedule.}
\label{tab:reproduction}
\begin{tabular}{lcccc}
\toprule
Method & Pre-train data & AP$^{bb}$ & AP$^{bb}_{50}$ & AP$^{bb}_{75}$ \\ \midrule
BYOL repro. in~\cite{liu2020self}  & COCO           & 38.5   & 58.9   & 41.7   \\
BYOL repro. by ours  & COCO           & 39.5   & 59.3   & 43.2   \\ \bottomrule
\end{tabular}
\end{table}

\section{Limitation}

In this paper, we mainly perform pre-training experiments with ResNet-50 on COCO dataset, and further scale them up on COCO+ dataset. However, the promise of self-supervised learning is to harness massive unlabeled data by scaling up to ever-larger datasets. Some prior works~
\cite{goyal2019scaling,caron2019unsupervised,goyal2021self} have attempted to leverage larger models and datasets to explore the limit of current self-supervised learning methods. For instance, a recent representative work SEER~\cite{goyal2021self} performs billion-scale self-supervised pre-training on internet images using the RegNet architectures~\cite{radosavovic2020designing} with 700M parameters over 512 GPUs. Training at scale requires huge computational resources that are inaccessible to many researchers, which is not the core of our paper. We wish to highlight that our general-purpose ORL has yielded better performance than concurrent works~\cite{wang2020dense,liu2020self} tailored for dense prediction downstream tasks when pre-trained on COCO (Table~\ref{tab:dense}), even surpassing the supervised ImageNet pre-training on several downstream tasks (Table~\ref{tab:linear}-\ref{tab:dense}). We expect that scaling ORL with larger architectures and datasets can further unleash its potential. We leave these explorations to future work.

\section{Broader impact}

We present a more effective approach for learning unsupervised visual representations. Compared to supervised learning, it can liberate humans from expensive annotations as well as take advantages of rapidly growing real-world data. Like other learning algorithms, self-supervised learning should be applied with cautions when deployed in the real-world scenario. First, it is susceptible to biased learning if the algorithm is given with biased data. The exposure to unlabeled data may amplify such biases. Thus, debiasing measures have to be taken. Second, it remains non-trivial to dissect what is learned by self-supervised models. Similar concerns about the calibration, robustness, and interpretability of supervised models are equally applicable to the unsupervised counterpart. Our work is limited to the improvement of self-supervised learning within our scope. However, we acknowledge the importance of providing more transparent explanations for classification decisions, as well as the credibility of each prediction. Finally, our method still relies on the traditional regime of centralized learning. Privacy can be compromised if the method is applied on an unsaved platform. Federated learning can be a solution. How to scale self-supervised learning to the regime of decentralized learning will be an interesting research question to answer.

\section{Conclusion}
\label{sec:conclusion}
In this work, we have presented a new self-supervised learning framework, ORL, for object-level representation learning from scene images. We leverage the latent prior of image-level self-supervised pre-training for discovering object-based region correspondence across images. The generated object-instance correspondence enables us to perform pairwise contrastive learning at the object level. ORL significantly improves the performance of self-supervised learning from scene images in a variety of downstream tasks. We expect that our method can be applied to larger-scale unlabeled data in the wild to fully realize its potential, and hope that our study can attract the community's attention to more general-purpose unsupervised representation learning from scene images.
\section{Experiments}
\label{sec:exp}
\subsection{Transferring to downstream tasks}
\label{subsec:transfer}

We evaluate the quality of learned representations by transferring them to multiple downstream tasks. Following common protocol~\cite{goyal2019scaling,misra2020self}, we use two evaluation setups: (\romannumeral1) the pre-trained network is \emph{frozen} as a feature extractor, and (\romannumeral2) the network parameters are fully \emph{fine-tuned} as weight initialization. We provide more experimental details in the appendix.

\begin{table}[ht]
\centering
\small
\addtolength{\tabcolsep}{-2pt}
\begin{minipage}{0.69\textwidth}
\begin{tabular}{lccccc}
\toprule
\multirow{2}{*}{Method} & \multirow{2}{*}{\begin{tabular}[c]{c}Pre-train\\ data\end{tabular}} & VOC07 & ImageNet & Places205 & iNat. \\
           &          & mAP & Top-1 & Top-1 & Top-1 \\ \midrule
Random~\cite{goyal2019scaling}     & -        & 9.6     & 13.7      &  16.6     & 4.8   \\
Supervised~\cite{misra2020self} & ImageNet & 87.5   &  75.9     & 51.5      & 45.4     \\ \midrule
SimCLR~\cite{chen2020simple}     & COCO     & 78.1    & 50.9      & 48.0      & 22.7      \\
MoCo v2~\cite{chen2020improved}    & COCO     & 82.2    & 55.1      & 48.8      & 27.8     \\
BYOL~\cite{grill2020bootstrap}       & COCO     & 84.5    & 57.8      & 50.5     & 29.5      \\
ORL (ours)        & COCO     & \textbf{86.7}    & \textbf{59.0}      & \textbf{52.7}      & \textbf{31.8}      \\ \midrule
BYOL~\cite{grill2020bootstrap}       & COCO+    & 87.0    & 59.6      & 52.7      & 30.9      \\
ORL (ours)        & COCO+    & \textbf{88.6}    &  \textbf{60.7}     & \textbf{54.1}      &  \textbf{32.0}     \\ \bottomrule
\end{tabular}
\end{minipage}\hspace{0.01\textwidth}
\begin{minipage}{0.28\textwidth} 
\caption{\textbf{Image classification with linear models.} All unsupervised methods are based on 800-epoch pre-training on COCO(+) with ResNet-50. We report mAP on the VOC07 dataset and top-1 center-crop accuracy on all other datasets. Numbers for all other methods are reproduced by us.}
\label{tab:linear}
\end{minipage}
\vspace{-0.1cm}
\end{table}

\begin{table}[ht]
\centering
\small
\addtolength{\tabcolsep}{-2pt}
\begin{minipage}{0.74\textwidth}
\begin{tabular}{lccccccccc}
\toprule
\multirow{2}{*}{Method} & \multirow{2}{*}{\begin{tabular}[c]{c}Pre-train\\ data\end{tabular}} & \multicolumn{8}{c}{VOC07 low-shot (mAP)} \\
          &          & 1 & 2 & 4 & 8 & 16 & 32 & 64 & 96 \\ \midrule
Random    & -        & 9.2  & 9.4  & 11.1  & 12.3  & 14.3   & 17.4   & 21.3   & 23.8   \\
Supervied & ImageNet & 53.0  & 63.6  & 73.7  & 78.8  & 81.8   & 83.8   & 85.2   & 86.0   \\ \midrule
SimCLR~\cite{chen2020simple}    & COCO     & 33.3  & 43.5  & 52.5  & 61.1  & 66.7   & 70.5   & 73.7   &  75.0  \\
MoCo v2~\cite{chen2020improved}   & COCO     & 39.5  & 49.3  & 60.4  & 69.3  & 74.1   & 76.8   & 79.1   & 80.1   \\
BYOL~\cite{grill2020bootstrap}      & COCO     & 39.4  & 50.9  & 62.2  & 71.7  & 76.6   & 79.2   & 81.3    & 82.2   \\
ORL (ours)       & COCO     & \textbf{39.6}  & \textbf{51.2}  & \textbf{63.4}  & \textbf{72.6}  & \textbf{78.2}   & \textbf{81.3}   & \textbf{83.6}    & \textbf{84.7}   \\ \midrule
BYOL~\cite{grill2020bootstrap}      & COCO+    & 41.1  & 54.3  & 66.6  & 75.2  & 80.1   & 82.6   & 84.6    & 85.4   \\
ORL (ours)       & COCO+    & \textbf{42.1}  & \textbf{54.9}  & \textbf{67.4}  & \textbf{75.7}  & \textbf{81.3}   & \textbf{83.7}   & \textbf{85.8}    & \textbf{86.7}   \\ \bottomrule
\end{tabular}
\end{minipage}\hspace{0.01\textwidth}
\begin{minipage}{0.24\textwidth}
\caption{\textbf{Low-shot image classification on VOC07} using linear SVMs trained on the fixed representations. All unsupervised methods are pre-trained on COCO(+) for 800 epochs with ResNet-50. We report mAP for each case across five runs.}
\label{tab:lowshot}
\end{minipage}
\vspace{-0.1cm}
\end{table}

\begin{table}[ht]
\centering
\small
\addtolength{\tabcolsep}{-2pt}
\begin{minipage}{0.65\textwidth}
\begin{tabular}{lccccc}
\toprule
\multirow{2}{*}{Method} & \multirow{2}{*}{\begin{tabular}[c]{c}Pre-train\\ data\end{tabular}} & \multicolumn{2}{c}{1\% labels} & \multicolumn{2}{c}{10\% labels} \\
          &          & Top-1 & Top-5 & Top-1 & Top-5 \\ \midrule
Random     & -        & 1.6      & 5.0      & 21.8      & 44.2      \\
Supervised~\cite{zhai2019s4l} & ImageNet & 25.4      & 48.4      &  56.4     & 80.4      \\ \midrule
SimCLR~\cite{chen2020simple}     & COCO     & 23.4      & 46.4      & 52.2      & 77.4      \\
MoCo v2~\cite{chen2020improved}    & COCO     & 28.2      & 54.7      & 57.1      & 81.7      \\
BYOL~\cite{grill2020bootstrap}       & COCO     & 28.4      & 55.9      & 58.4      & 82.7      \\
ORL (ours)        & COCO     & \textbf{31.0}      & \textbf{58.9}      & \textbf{60.5}      & \textbf{84.2}      \\ \midrule
BYOL~\cite{grill2020bootstrap}       & COCO+    & 28.3      & 56.0      & 59.4      & 83.6      \\
ORL (ours)        & COCO+    & \textbf{31.8}      & \textbf{60.1}      &  \textbf{60.9}     & \textbf{84.4}      \\ \bottomrule
\end{tabular}
\end{minipage}\hspace{0.01\textwidth}
\begin{minipage}{0.33\textwidth}
\caption{\textbf{Semi-supervised learning on ImageNet.} All unsupervised methods are pre-tained on COCO(+) for 800 epochs with ResNet-50. We fine-tune all models with 1\% and 10\% ImageNet labels, and report both top-1 and top-5 center-crop accuracy on the ImageNet validation set.}
\label{tab:semi}
\end{minipage}
\vspace{-0.1cm}
\end{table}

\begin{table}[ht]
\centering
\small
\addtolength{\tabcolsep}{-2pt}
\begin{minipage}{0.72\textwidth}
\begin{tabular}{lccccccc}
\toprule
\multirow{2}{*}{Method} &
  \multirow{2}{*}{\begin{tabular}[c]{c}Pre-train\\ data\end{tabular}} &
  \multicolumn{3}{c}{COCO detection} &
  \multicolumn{3}{c}{COCO instance seg.} \\
           &          & AP$^{bb}$ & AP$^{bb}_{50}$ & AP$^{bb}_{75}$ & AP$^{mk}$ & AP$^{mk}_{50}$ & AP$^{mk}_{75}$ \\ \midrule
Random~\cite{tian2020makes}     & -        & 32.8   & 50.9   & 35.3   & 29.9   & 47.9   & 32.0   \\
Supervised~\cite{tian2020makes} & ImageNet & 39.7   & 59.5   & 43.3   & 35.9   & 56.6   & 38.6   \\ \midrule
SimCLR~\cite{chen2020simple}     & COCO     & 37.0   & 56.8   & 40.3   & 33.7   & 53.8   & 36.1   \\
MoCo v2~\cite{chen2020improved}    & COCO     & 38.5   & 58.1   & 42.1   & 34.8   & 55.3   & 37.3   \\
Self-EMD~\cite{liu2020self}   & COCO     & 39.3   & 60.1   &  42.8  & -   & -  & - \\
DenseCL~\cite{wang2020dense}    & COCO     & 39.6   & 59.3   & 43.3   & 35.7   & 56.5   & 38.4   \\
BYOL~\cite{grill2020bootstrap}       & COCO     & 39.5   & 59.3   & 43.2   & 35.6   & 56.5   & 38.2   \\
ORL (ours)        & COCO     & \textbf{40.3}   & \textbf{60.2}   & \textbf{44.4}   & \textbf{36.3}   & \textbf{57.3}   & \textbf{38.9}   \\ \midrule
BYOL~\cite{grill2020bootstrap}       & COCO+    & 40.0   & 60.1   & 44.0   & 36.2   & 57.1   & 39.0   \\
ORL (ours)        & COCO+    & \textbf{40.6}   & \textbf{60.8}   & \textbf{44.5}   & \textbf{36.7}   & \textbf{57.9}   & \textbf{39.3}   \\ \bottomrule
\end{tabular}
\end{minipage}\hspace{0.01\textwidth}
\begin{minipage}{0.26\textwidth}
\caption{\textbf{Object detection and instance segmentation fine-tuned on COCO.} All unsupervised methods are based on 800-epoch pre-training on COCO(+). We use Mask R-CNN R50-FPN (1$\times$ schedule), and report bounding-box AP (AP$^{bb}$) and mask AP (AP$^{mk}$). Numbers for MoCo v2 are adopted from~\cite{wang2020dense}.}
\label{tab:dense}
\end{minipage}
\vspace{-0.5cm}
\end{table}

\noindent\textbf{Image classification with linear models.}
Following~\cite{goyal2019scaling,misra2020self}, we assess the quality of features by training linear classifiers on top of the \emph{fixed} representations extracted from different depths of the network for four datasets: VOC07~\cite{everingham2010pascal}, ImageNet~\cite{deng2009imagenet}, Places205~\cite{zhou2014learning}, and iNaturalist18~\cite{van2018inaturalist}. These datasets involve diverse classification tasks ranging from object classification, scene recognition to fine-grained recognition. For VOC07, we train linear SVMs using LIBLINEAR package~\cite{fan2008liblinear} following the setup in~\cite{goyal2019scaling,misra2020self}. We train on \texttt{trainval} split of VOC07 and evaluate mAP on \texttt{test} split. For ImageNet, Places205 and iNaturalist18, we follow~\cite{zhang2017split,goyal2019scaling,misra2020self} and train a 1000-way, 205-way and 8142-way linear classifier, respectively. We train on \texttt{train} split of each dataset, and report top-1 center-crop accuracy on the respective \texttt{val} split. Table~\ref{tab:linear} reports the results for the best-performing layer of each method. ORL substantially outperforms the BYOL baseline on all four datasets. We also observe that the COCO pre-trained ORL surpasses the supervised ImageNet pre-trained counterpart on Places205 by 1.2\% in top-1 accuracy. This is the first time that a self-supervised learner outperforms the ImageNet pre-training using only $\sim$1/10 images compared with ImageNet. When pre-trained on a larger COCO+ dataset, ORL again outperforms BYOL. Note that apart from Places205 (2.6\% gains), ORL also surpasses the supervised ImageNet counterpart on VOC07 by 1.1\% mAP, using merely $\sim$1/5 images.

\noindent\textbf{Low-shot image classification.}
We perform low-shot image classification with few training examples per class on VOC07 dataset following the same setup in~\cite{goyal2019scaling}. We vary the number of labeled examples per category used to train linear SVMs on \texttt{train} split of VOC07 and report the average mAP across five independent samples for each low-shot case evaluated on \texttt{test} split. Table~\ref{tab:lowshot} provides the results. ORL shows consistent performance improvement over BYOL for each low-shot value, with larger gains achieved as the number of labeled examples per class is increasing. ORL also gradually bridges the gap to the supervised ImageNet pre-training under this scenario. We observe the consistent performance boost when pre-training on the COCO+ dataset. Note that the COCO+ pre-trained ORL again outperforms the supervised ImageNet pre-training when the low-shot values are 64 and 96.

\noindent\textbf{Semi-supervised learning.}
We perform semi-supervised learning on ImageNet following the protocol of~\cite{wu2018unsupervised,henaff2019data,misra2020self,chen2020simple,grill2020bootstrap}. Specifically, we first randomly select 1\% and 10\% labeled data from ImageNet \texttt{train} split. We then fine-tune our models on these two training subsets and report both top-1 and top-5 accuracy on the official \texttt{val} split of ImageNet in Table~\ref{tab:semi}. Again, ORL outperforms BYOL as well as the supervised ImageNet counterpart by large margins.

\noindent\textbf{Object detection and segmentation.}
We train a Mask R-CNN model~\cite{he2017mask} with R50-FPN backbone~\cite{lin2017feature} implemented in Detectron2~\cite{wu2019detectron2}. We \emph{fine-tune} all layers end-to-end on COCO \texttt{train2017} split with the standard 1$\times$ schedule and evaluate on COCO \texttt{val2017} split. We follow the same setup in~\cite{tian2020makes}, with batch normalization layers synchronized across GPUs~\cite{peng2018megdet}. As shown in Table~\ref{tab:dense}, ORL yields 0.8\% AP and 0.7\% AP improvements over BYOL for object detection and instance segmentation, respectively. The improvements are consistent over all evaluation metrics. When pre-trained on the COCO+ dataset, ORL again outperforms BYOL. It should be well noted that ORL even outperforms the most recent Self-EMD and DenseCL that are specifically designed for dense prediction downstream tasks. More importantly, either COCO or COCO+ pre-trained ORL can surpass the supervised ImageNet pre-training on all metrics. This further demonstrates the superiority of learning unsupervised representations at the object level.

\subsection{Ablation study}
\label{subsec:ablation}

In this subsection, we conduct extensive ablation experiments to examine the effect of each component that contributes to ORL. We pre-train our models on COCO and observe the downstream performance of all ablations on VOC07 SVM classification benchmark as introduced in Section~\ref{subsec:transfer}.


\begin{table}[t]
\centering
\small
\caption{\textbf{Ablations for ORL.} (a) Effect of intra-RoI and inter-RoI losses. (b) Effect of NNs and RoI pairs. (c) Comparison with multi-crop BYOL. (d) Comparison with ground truth bounding boxes. (e) Pre-trainig schedules. We report mAP of linear SVMs on VOC07 classification benchmark.}
\label{tab:ablation}
\addtolength{\tabcolsep}{-2pt}
\vspace{-0.2cm}
\begin{subtable}[htp]{0.49\textwidth}
\centering
\caption{}
\label{tab:ablation loss}
\vspace{-0.2cm}
\begin{tabular}{lcccc}
\toprule
pre-train & intra-RoI & inter-RoI & VOC07 \\ \midrule
BYOL                 &  &  & 84.5    \\ \midrule
\multirow{3}{*}{ORL} & \checkmark  &   & 85.7    \\
                     &   & \checkmark  & 85.9    \\
                     & \checkmark  & \checkmark  & 86.7   \\ \bottomrule
\end{tabular}
\end{subtable}
\hfill
\begin{subtable}[htp]{0.49\textwidth}
\centering
\caption{}
\label{tab:ablation nn}
\vspace{-0.2cm}
\begin{tabular}{lcccccc}
\toprule
         & \multicolumn{3}{c}{\# of NNs} & \multicolumn{3}{c}{\# of RoI pairs} \\ \cmidrule(l){2-4}\cmidrule(l){5-7}
         & 1     & 10     & 20   & 5\%       & 10\%       & 20\%       \\ \midrule
VOC07    & 84.7      & 86.7       &  87.0     &  86.1       &  86.7        &  86.4        \\ \bottomrule
\end{tabular}
\end{subtable}
\vfill
\vspace{-0.1cm}
\begin{subtable}[htp]{0.32\textwidth}
\centering
\caption{}
\label{tab:ablation view}
\vspace{-0.2cm}
\begin{tabular}{lccc}
\toprule
pre-train & input views & VOC07 \\ \midrule
BYOL    & $2\times224+4\times96$    & 84.0  \\
ORL     & $2\times224+4\times96$   & 86.7    \\ \bottomrule
\end{tabular}
\end{subtable}
\hfill
\vspace{-0.1cm}
\begin{subtable}[htp]{0.19\textwidth}
\centering
\caption{}
\label{tab:ablation box}
\vspace{-0.2cm}
\begin{tabular}{ccc}
\toprule
boxes & VOC07 \\ \midrule
ground truth    &  85.4         \\
selective search    &  86.7           \\ \bottomrule
\end{tabular}
\end{subtable}
\hfill
\vspace{-0.1cm}
\begin{subtable}[htp]{0.32\textwidth}
\centering
\caption{}
\label{tab:ablation epoch}
\vspace{-0.2cm}
\begin{tabular}{lcccc}
\toprule
pre-train & 100 & 200 & 400 & 800 \\ \midrule
BYOL      &  77.1   & 81.8    &  83.7   & 84.5    \\
ORL       & 83.5    & 85.2    & 86.3    & 86.7    \\ \bottomrule
\end{tabular}
\end{subtable}
\end{table}


\noindent\textbf{Effect of intra-RoI and inter-RoI losses.}
Table~\ref{tab:ablation loss} ablates the effect of our introduced $\mathcal{L}_\text{intra-RoI}$ and $\mathcal{L}_\text{inter-RoI}$ losses in Equation~\ref{eq:total loss}. Adding either $\mathcal{L}_\text{intra-RoI}$ or $\mathcal{L}_\text{inter-RoI}$ can improve the performance, with the best results obtained by adding both terms.

\noindent\textbf{Effect of nearest neighbors and RoI pairs.}
Table~\ref{tab:ablation nn} ablates the effect of the number of nearest neighbors $K$ and RoI pairs $N$ used for generating inter-RoI pairs in Stage 2 of ORL. We set $N=10\%$ when ablating $K$, and set $K=10$ when ablating $N$. We observe that retrieving more nearest neighbors leads to better performance since more nearest neighbors provide more diverse image-level pairs to the subsequent generation of inter-RoI pairs. Although setting $K=20$ produces a slightly better performance, we choose $K=10$ by default as a trade-off considering the tendency of the saturated performance. Our method is more robust to the number of retrieved top-ranked RoI pairs after image-level nearest-neighbor retrieval, with $N=10\%$ performing slightly better.

\noindent\textbf{Comparison with multi-crop BYOL.}
Prior work~\cite{caron2020unsupervised} has indicated that cropping multiple views of the same image can improve the performance of self-supervised learning methods pre-trained on ImageNet. To investigate whether our gains are due to more accurate object-instance comparison or simply more number of mixed views, we randomly crop four additional smaller views for BYOL to ensure the number and size of the input patches are equal to ORL (\ie, $2\times224+4\times96$). As shown in Table~\ref{tab:ablation view}, different from the observation on ImageNet, simply adding more low-resolution crops tends to hurt the performance since it will further intensify the inconsistent noise on scene images. In contrast, ORL substantially outperforms this multi-crop variant, validating that the gains are truly due to our object-level representation learning mechanism.

\noindent\textbf{Comparison with ground truth bounding boxes.}
In Stage 2, ORL requires an unsupervised region proposal algorithm to extract approximate object-based regions, which is inaccurate to some extent. We further investigate whether the performance can be improved when more accurate object regions are available, \ie, with bounding box annotations. To this end, we replace our selective-search generated object proposals with ground truth bounding boxes provided from COCO \texttt{train2017} set, while keeping other procedures unchanged. As shown in Table~\ref{tab:ablation box}, adopting ground truth bounding boxes performs inferior to selective search. This is mainly due to that although the ground truth bounding boxes can provide more accurate object location, their numbers are too scarce compared with a large amount of region proposals generated by selective search. The more diverse region proposals can potentially induce more unknown object or object-part discovery beyond the manually annotated objects. In this case, the diversity can make up for the inaccuracy. 


\noindent\textbf{Pre-training schedules.}
Table~\ref{tab:ablation epoch} shows the results with different pre-training schedules. The performance of both ORL and BYOL improves when pre-trained for longer epochs, while ORL consistently outperforms BYOL by at least 2.2\% mAP. Note that our 200-epoch ORL has already surpassed the 800-epoch BYOL, demonstrating the efficiency of ORL.

\subsection{Visualization}
\label{subsec:visualization}

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{correspondence}
	\caption{\textbf{Top-ranked region correspondence discovered by ORL in Stage 2.} We show a pair of discovered object-instance per image pair for clarity. More discovered correspondence pairs are provided in the appendix.}
	\label{fig:correspondence}
\end{figure}

\begin{figure}[t]
	\centering
	\includegraphics[width=\linewidth]{attention}
	\caption{\textbf{Attention maps generated by BYOL and ORL.} ORL can activate more object regions and produce more accurate object boundary in the heatmap than BYOL. We provide more attention maps in the appendix. Best viewed with zoom in.}
	\label{fig:attention}
\end{figure}

\noindent\textbf{Correspondence pairs.}
Figure~\ref{fig:correspondence} visualizes some top-ranked region correspondence discovered in Stage 2 of ORL. We observe that each generated inter-RoI pair largely correspond to the regions with similar visual concepts (\ie, objects or object parts) across images. These correspondence pairs can provide more natural and diverse object-level intra-class variances compared with the typical contrastive learning methods that only perform aggressive intra-image augmentations to simulate such variances.

\noindent\textbf{Attention maps.}
Figure~\ref{fig:attention} visualizes the attention maps generated by BYOL and ORL. We observe that both BYOL and ORL can produce relatively high-quality attention maps that focus on the foreground objects. This reflects from the side that current image-level contrastive learning methods have already induced a latent space with rich visual concepts. Nevertheless, ORL can activate more object regions and produce more accurate object boundary than BYOL in the generated attention maps. It is mainly due to introducing object-level similarity learning into ORL, which can minimize the inconsistent noise caused by image-level contrastive learning. In contrast, BYOL only uses the whole image to extract features, thus activating the most discriminative region.

\section{Introduction}
\label{sec:intro}

Unsupervised visual representation learning aims at obtaining transferable features with abundant unlabeled data. Recent self-supervised learning (SSL) methods based on contrastive learning~\cite{wu2018unsupervised,he2020momentum,misra2020self,chen2020simple,grill2020bootstrap,caron2020unsupervised,chen2020exploring} have largely narrowed the gap and even surpassed the supervised counterpart on a number of downstream tasks~\cite{krizhevsky2012imagenet,russakovsky2015imagenet,girshick2014rich,ren2015faster,long2015fully,he2017mask}. These methods build upon the instance discrimination task that maximizes the agreement between different data-augmented views of the same image. Despite their success, current SSL methods are primarily pre-trained on the unlabeled ImageNet~\cite{deng2009imagenet} dataset that contains iconic images with single object as shown in Figure~\ref{fig:teaser}(a). The underlying object-centric constraint of ImageNet makes it hard to be applied in real world scenarios where more complex scene images with multiple objects are available. Meanwhile, na\"ively adopting the off-the-shelf contrastive learning methods on scene images introduces inconsistent learning signals since random crops of the same image may correspond to different objects as shown in Figure~\ref{fig:teaser}(b). Indeed, it has been shown that current contrastive learning methods tend to struggle on more complex scene datasets~\cite{grill2020bootstrap,selvaraju2020casting,liu2020self,wang2020dense} like COCO~\cite{lin2014microsoft} or Places365~\cite{zhou2017places}. Therefore, it is imperative to design an effective object-level representation learning paradigm as illustrated in Figure~\ref{fig:teaser}(c) to harness massive unlabeled scene images in the wild.

In this work, we are interested in going beyond ImageNet to obtain better representations on non-iconic images. Apparently, it is challenging to learn representations from scene-level images since they are entangled with many concepts including structures, objects, backgrounds and relationships. It remains an open question how to take advantages of spatial information of multiple objects naturally residing in the scene images when no object annotations are available, let alone further deriving object-level correspondence to construct positive object-instance pairs.

To tackle these challenges, we introduce a novel object-level unsupervised representation learning framework tailored for scene images. Our framework is based on a key insight of the current contrastive learning methods: they can implicitly group different images with similar visual concepts together even though they are explicitly optimized to group different views of the same image.
This phenomenon reveals that image-level contrastive learning has already induced a latent space with rich visual concepts. Though the latent space usually entangles other scene concepts like structures, backgrounds and relationships, it will be useful for object discovery if appropriately deployed.
Through computing the similarity of sampled regions between $k$-nearest-neighbor (KNN) images, we conclude two observations: 1) image-level contrastive learning encodes priors for region correspondence discovery across images; 2) high-response regions are usually objects or object parts.

Based on the observation above, we propose a multi-stage framework for unsupervised object-level representation learning. Specifically, we first extract potential object-based regions in scene images using the unsupervised region proposal algorithms (e.g., selective search~\cite{uijlings2013selective}). We then propose a region correspondence generation scheme to leverage the off-the-shelf image-level contrastive learning pre-trained model to discover corresponding object-instance pairs for the proposed regions in the embedding space. Finally, we use the obtained object-instance pairs to construct positive sample pairs for object-level representation learning. Figure~\ref{fig:teaser}(d) shows several cross-image object-instance pairs discovered by our framework on COCO dataset using the latent prior of BYOL~\cite{grill2020bootstrap}, the state-of-the-art image-level contrastive learning method. The discovered inter-corresponding pairs substantially provide diverse intra-class variances at the object-instance level to aid object-level representation learning.

\begin{figure}[t]
	\centering
	\includegraphics[width=.84\linewidth]{teaser_v2}
	\vspace{-0.2cm}
	\caption{(a) Current image-level contrastive learning methods heavily rely on the object-centric bias of ImageNet, \ie, different crops correspond to the same object. Prior works use either the different views of the same image~\cite{wu2018unsupervised,he2020momentum,misra2020self,chen2020simple,grill2020bootstrap,chen2020exploring} (\ie, intra-image) or similar images~\cite{zhuang2019local,li2020prototypical,xie2020delving,azabou2021mine,dwibedi2021little} (\ie, inter-image) to form positive pairs. (b) Directly adopting image-level contrastive learning methods on scene images can cause inconsistent learning signals since different crops may correspond to different objects. (c) Object-level contrastive learning can overcome the limitation in (b) by enforcing object-level consistency. (d) We find that image-level contrastive learning encodes priors for region correspondence discovery across images, and high-response regions are usually objects or object parts (we show one discovered object-instance pair per image pair for clarity), which is useful for object-level representation learning.}
	\label{fig:teaser}
	\vspace{-0.3cm}
\end{figure}

Overall, our main contributions are summarized as follows:

\textbf{1)} We observe that existing image-level contrastive learning methods have priors to discover object-level correspondence across images. We leverage this prior for the first time for unsupervised cross-image object-level correspondence discovery.

\textbf{2)} With the obtained correspondence, we introduce a novel multi-stage self-supervised learning pipeline, termed as ORL, for object-level representation learning from scene images, going beyond object-centric ImageNet.

\textbf{3)} We contribute the first study for object-level SSL. ORL substantially outperforms image-level contrastive learning approaches pre-trained on COCO dataset ($\sim$118k images with labels discarded), setting a new state of the art on this challenging dataset that contains diverse scenes in the wild.
The COCO pre-trained ORL even surpasses supervised ImageNet pre-training on several considered downstream tasks.
When SSL is conducted on a larger ``COCO+'' dataset (COCO \texttt{train2017} set plus COCO \texttt{unlabeled2017} set, $\sim$241k images in total), ORL further improves the performance, demonstrating its potential to benefit from more unlabeled scene data.

\section{Methodology}
\label{sec:method}

\begin{figure}[t]
	\centering
	\includegraphics[width=.9\linewidth]{framework}
	\caption{\textbf{Overview of our three-stage pipeline.} In Stage 1, we pre-train an image-level contrastive learning model, \eg, BYOL. In Stage 2, we first use the pre-trained model to retrieve KNNs for each image in the embedding space to obtain image-level visually similar pairs. We then use unsupervised region proposal algorithms (\eg, selective search) to generate rough RoIs for each image pair. Afterwards, we reuse the pre-trained model to retrieve the top-ranked RoI pairs, \ie, correspondence. We find these pairs of RoIs are almost objects or object parts. In Stage 3, with the corresponding RoI pairs discovered across images, we finally perform object-level contrastive learning using the same architecture as Stage 1.}
	\label{fig:framework}
\end{figure}

\vspace{-0.2cm}
We propose a new multi-stage self-supervised learning framework, \ie, ORL, for object-level representation learning from scene images.
ORL extends the existing image-level contrastive learning framework to object level by leveraging priors from image-level instance discrimination.
The overall pipeline of ORL is illustrated in Figure~\ref{fig:framework}.
It contains three stages: image-level pre-training, correspondence discovery, and object-level pre-training. We detail each stage as follows.

\subsection{ORL pipeline}
\label{subsec:pipeline}


\noindent\textbf{Preliminary: Contrastive learning.}
Our pipeline contains several contrastive learning modules in Stage 1 and 3.
Without loss of generality, we consider BYOL~\cite{grill2020bootstrap} as our basic contrastive learning module.
BYOL uses two neural networks: the \emph{online} network $f_\theta(x)$ and the \emph{target} network $g_\xi(x)$.
The target network provides the regression target to train the online network while its weights $\xi$ are updated by an exponential moving average of the online parameters $\theta$ with a decay rate $\tau\in\left[0,1\right]$ following BYOL.
\if 0
\begin{equation}\label{eq:momentum update}
 \xi\leftarrow\tau\xi+\left(1-\tau\right)\theta.
\end{equation}
\fi
Given two input images $x_1$ and $x_2$, the loss function is defined as:
\begin{equation}\label{eq:mse loss}
 \mathcal{L}\left(x_1, x_2\right)\triangleq\left\|f_\theta\left(x_1\right)-g_\xi\left(x_2\right)\right\|_2^2,
\end{equation}
We name it an ``intra-'' version of BYOL if $x_1$ and $x_2$ are two augmented views from the same image, otherwise an ``inter-'' one.

\noindent\textbf{Stage 1: Image-level pre-training.}
The foremost stage is to obtain an unsupervised pre-trained model from image-level tasks.
As shown in Figure~\ref{fig:framework} Stage 1, given two augmented views $v$ and $v'$ from the same input image $x$, we pre-train the network following the loss function $\mathcal{L}_\text{image}=\mathcal{L}\left(v, v'\right)$, constituting a standard image-level BYOL pre-training.
This stage can be freely replaced with other image-level contrastive learning methods. We adopt BYOL here for its simplicity and effectiveness.

\if 0
The foremost stage is to obtain an unsupervised pre-trained model from image-level tasks.
We consider BYOL~\cite{grill2020bootstrap} as our image-level pre-training approach due to its state-of-the-art performance and simplicity, \ie, without using negative sample pairs.
As shown in Figure~\ref{fig:framework} Stage 1, BYOL uses two neural networks: the \emph{online} network and the \emph{target} network. 
The online network is defined by a set of parameters $\theta$ and consists of a backbone $f_\theta$, an MLP projector $g_\theta$ and an MLP predictor $q_\theta$. 
The target network is parameterized by $\xi$ and has the same architecture as the online network except for the predictor. 
The target network provides the regression target to train the online network while its weights $\xi$ are updated by an exponential moving average of the online parameters $\theta$ with a decay rate $\tau\in\left[0,1\right]$ following:
\begin{equation}\label{eq:momentum update}
 \xi\leftarrow\tau\xi+\left(1-\tau\right)\theta.
\end{equation}
Given two randomly augmented views $v$ and $v'$ from an input image $x$, the two networks extract feature vectors from two views separately and minimize their mean squared error defined as:
\begin{equation}\label{eq:mse loss}
 \mathcal{L}\triangleq\left\|p-z'\right\|_2^2=2-2\frac{\langle p,z'\rangle}{\left\|p\right\|_2\cdot\left\|z'\right\|_2},
\end{equation}
where $p$ and $z'$ are the features from the online network and target network respectively, $\left\|\cdot\right\|_2$ is $\ell_2$-norm, and $\langle\cdot\rangle$ is the inner product. The symmetric loss $\widetilde{\mathcal{L}}$ is also computed by feeding $v'$ to the online network and $v$ to the target network, respectively.\fi

\noindent\textbf{Stage 2: Correspondence discovery.}
We employ the pre-trained image-level contrastive learning model in Stage 1 to mine object-level correspondence for the whole dataset.
As shown in Figure~\ref{fig:framework} Stage 2, the overall discovery process is comprised of three steps.

(\romannumeral1) Image-level nearest-neighbor retrieval. Specifically, for each query image $x$ in the training set $\mathcal{D}$, we first retrieve its top $K$ nearest neighbors $\mathcal{N}_k$, $k=1,...,K$, by cosine distance in the embedding space using the features learned from the first stage to form image-level pairs that contain similar visual context.

(\romannumeral2) Region-of-interest (RoI) generation. To generate object-based RoIs, we apply unsupervised region proposal algorithms, \eg, selective search~\cite{uijlings2013selective}, for each image in the pair.
Considering the redundancy of generated proposals (each image can have thousands of proposals), we filter certain number of them with some pre-defined thresholds\footnote{In practice, we filter bounding boxes with the minimal scale as 96 pixels, the aspect ratio between $1/3$ and $3/1$, and the maximal IoU as 0.5.} including the minimal scale, the range of aspect ratio, and the maximal intersection-over-union (IoU) among the filtered boxes. After the filtering operation, we select the top 100 proposals ranked with objectiveness as the candidate RoI set for subsequent RoI pair retrieval. To extract features with the equally-sized input that is compatible with the backbone, we crop and resize each RoI to $224\times224$. Note that even top RoIs ranked with objectiveness are still very noisy, containing a large proportion of non-object regions.

(\romannumeral3) Top-ranked RoI pair retrieval. For each query RoI from $x$, we compute its cosine similarity in the embedding space with all RoIs from its nearest-neighbor image $\mathcal{N}_k$ using the features learned from Stage 1 again.
Within the calculated cosine similarity matrix $\mathbf{M}_k\in\mathbb{R}^{100\times100}$, we retrieve top-ranked $N$ RoI pairs to construct the set of object-level corresponding pairs $\{\mathcal{B}^n_k\}$, where $n=\{1,...,N\}$. These high-response corresponding regions are almost objects or object parts. Finally, We save the nearest-neighbor image id and bounding box coordinate information of each corresponding pair.

\noindent\textbf{Stage 3: Object-level pre-training.}
With the corresponding inter-image RoI (inter-RoI) pairs obtained in Stage 2, we perform object-level representation learning following the BYOL framework as shown in Figure~\ref{fig:framework} Stage 3.
Specifically, given an input image $x$, we first randomly select one nearest-neighbor image $\mathcal{N}_k$ to obtain the corresponding set of inter-RoI pairs $\{\mathcal{B}^n_k\}$. We then randomly select one inter-RoI pair $\mathcal{B}^n_k$ as a positive pair.
With the bounding box coordinate stored in $\mathcal{B}^n_k$, we crop the corresponding inter-RoIs from $x$ and $\mathcal{N}_k$, respectively, and resize each patch to 96$\times$96, constituting two patches $p_1$ and $p_2$.
We feed the two patches to the online network and target network separately to compute the loss $\mathcal{L}_\text{inter-RoI} = \mathcal{L}\left(p_1, p_2\right)$.

To make full use of discovered objects, we introduce the intra-RoI contrastive learning via augmenting object patches. Specifically, we randomly select one filtered bounding box from $x$ obtained in Stage 2, and spatially jitter the box around its original location with the following operations\footnote{Inspired by the anchor technique used in object detection, we adopt the similar operations for our bounding box augmentations but in a continuous fashion.}: (\romannumeral1) a random box center shifting within 50\% of its width and height, (\romannumeral2) a random area scaling between 50\% and 200\% of the original box, and (\romannumeral3) a random aspect ratio between $1/2$ and $2/1$. Similarly, we crop the two intra-RoIs $p$ and $p'$, and resize each patch to 96$\times$96 for forward propagation to compute the loss $\mathcal{L}_\text{intra-RoI}=\mathcal{L}\left(p, p'\right)$. The diverse spatial jittering of the bounding box encourages the network to preserve common object information and disregard the background, thus further improving the localization ability.

We keep the two original global views in BYOL as well since they preserve the global image-level information compared with the local patches. The final loss for our ORL can thus be formulated as:
\begin{equation}\label{eq:total loss}
 \mathcal{L}_\text{ORL}=\lambda_1\mathcal{L}_\text{image}+\lambda_2\mathcal{L}_\text{intra-RoI}+\lambda_3\mathcal{L}_\text{inter-RoI},
\end{equation}
where $\lambda_1$, $\lambda_2$, $\lambda_3$ are the loss weights to balance each term. We set all loss weights to 1 by default. We also compute the symmetric loss $\widetilde{\mathcal{L}}_\text{ORL}$ following BYOL.

\subsection{Implementation details}
\label{subsec:details}

\noindent\textbf{Dataset.} We pre-train our models on the COCO \texttt{train2017} set that contains $\sim$118k images without using labels. Compared with the heavily curated object-centric ImageNet dataset, COCO contains more natural and diverse scenes in the wild, which is closer to real-world scenarios. We also perform self-supervised learning on a larger ``COCO+'' dataset (COCO \texttt{train2017} set plus COCO \texttt{unlabeled2017} set) to verify whether our method can benefit from more unlabeled scene data.

\noindent\textbf{Image augmentations.} The global image augmentation setting is the same as BYOL~\cite{grill2020bootstrap}: a 224$\times$224-pixel random resized crop with a random horizontal flip, followed by a random color distortion, random grayscale conversion, random Gaussian blur and solarization. For the local patch augmentation, we directly crop the corresponding intra-RoI and inter-RoI on the input images, and resize each cropped patch to 96$\times$96 to take place of the random resized cropping. The subsequent augmentations exactly follow the global ones.

\noindent\textbf{Network architecture.} We adopt ResNet-50~\cite{he2016deep} as the default backbone. We use the same MLP projector $g_\theta$ and predictor $q_\theta$ as in BYOL: a linear layer with output size 4096 followed by batch normalization (BN)~\cite{ioffe2015batch}, rectified linear units (ReLU)~\cite{nair2010rectified}, and a final linear layer with output dimension 256. We share the backbone and projector weights among the global and two local branches while the weights of predictor are not shared.

\noindent\textbf{Optimization.} For pre-training in Stage 1 and Stage 3, we use the same training hyper-parameters. Specifically, we use the SGD optimizer with a weight decay of 0.0001 and a momentum of 0.9. We adopt the cosine learning rate decay schedule~\cite{loshchilov2016sgdr} with a base learning rate of 0.2, linearly scaled\cite{goyal2017accurate} with the batch size ($lr=0.2 \times\text{BatchSize}/256$). The batch size is set to 512 by default, which is friendly to typical 8-GPU implementations. To keep the training iterations comparable with the ImageNet supervised pre-training, we train our models for 800 epochs with a warm-up period of 4 epochs. The exponential moving average parameter $\tau$ starts from 0.99 and is increased to 1 during training, following~\cite{grill2020bootstrap}. For correspondence generation in Stage 2, we retrieve top $K=10$ nearest neighbors for each image and select top-ranked $N=10\%$ RoI pairs for each image-level nearest-neighbor pair.
\section{Related work}
\label{sec:work}

\vspace{-0.2cm}
\noindent\textbf{Self-supervised learning.}
Self-supervised learning builds unsupervised representations by exploiting the internal priors or structures of data in the form of a pretext task. A wide range of pretext tasks have been proposed in the past few years. Examples include patch context prediction~\cite{doersch2015unsupervised}, jigsaw puzzles~\cite{noroozi2016unsupervised}, inpainting~\cite{pathak2016context}, colorization~\cite{larsson2016learning,zhang2016colorful}, cross-channel prediction~\cite{zhang2017split}, visual primitive counting~\cite{noroozi2017representation}, and rotation prediction~\cite{gidaris2018unsupervised}. Although good representations emerge with these pretext tasks, they are prone to lose generality due to their hand-crafted nature.

Recently, contrastive learning~\cite{hadsell2006dimensionality} that performs instance discrimination~\cite{wu2018unsupervised,he2020momentum,misra2020self,chen2020simple,grill2020bootstrap,caron2020unsupervised,chen2020exploring} has shown great potential in this filed, largely narrowing the gap to fully supervised learning. The core idea of contrastive learning is to gather positive pairs and separate negative pairs in the embedding space. A positive pair is usually formed with two transformed views of the same image while the negative pairs are formed with different images. Typically, contrastive learning methods require a large number of negative samples to avoid mode collapse. These samples can be maintained within a mini-batch~\cite{oord2018representation,hjelm2019learning,ye2019unsupervised,henaff2019data,bachman2019learning,chen2020simple}, a memory bank~\cite{wu2018unsupervised,tian2019contrastive,zhuang2019local,misra2020self} or a queue~\cite{he2020momentum,chen2020improved}. BYOL~\cite{grill2020bootstrap} and SwAV~\cite{caron2020unsupervised} further remove the necessity of involving negative pairs. BYOL directly predicts the features of one view from another view, while SwAV predicts the cluster assignments between multiple views of the same image. Despite their improved performance, the existing image-level contrastive learning methods are largely confined to the underlying object-centric bias of ImageNet.

More recently, a group of works that perform pixel-level~\cite{pinheiro2020unsupervised,wang2020dense,xie2020propagate,selvaraju2020casting,liu2020self,henaff2021efficient} or region-level~\cite{roh2021spatially,yang2021instance,xiao2021region,xie2021detco,ding2021unsupervised} representation learning have emerged. Our work is more related to region-level representation learning but substantially different from this line of research in the following aspects: 1) they still largely pre-train on object-centric ImageNet while we pre-train on non-iconic scene images, 2) they align pre-training specifically for dense prediction downstream tasks while we target at more general-purpose representation learning, 3) their randomly cropped local regions do not contain the explicit \emph{object} notion as ours, and 4) they only rely on intra-image transformations (\eg, random cropping) to construct corresponding positive pairs from the same image while we leverage the discovered high-level semantic correspondence to construct positive pairs across images.

There are also a few prior attempts~\cite{goyal2019scaling,caron2019unsupervised,goyal2021self} for self-supervised learning on non-curated scene images. As opposed to our work, most of them consider larger models and datasets to explore the limit of current self-supervised learning methods without further considering the object-level information residing in scene images. 

\noindent\textbf{Visual correspondence.}
Visual correspondence aims at finding pairwise pixels or regions across images that result from the same scene~\cite{zabih1994non}, which can be regarded as similarity learning of visual descriptors among matched points or patches. While early efforts learn dense correspondence with labeled data~\cite{han2015matchnet,zagoruyko2015learning,kanazawa2016warpnet,ufer2017deep,ono2018lf}, some recent works learn the similarity between the parts or landmarks of the data in an unsupervised manner~\cite{thewlis2017unsupervised,thewlis2019unsupervised}.
Our work substantially differs from this line of research from original intention.
Previous works aim at accurately detect all correspondence given two images, whereas our work focuses on retrieving high-quality correspondence to improve representation learning.