Towards Human-Machine Cooperation: Self-supervised Sample Mining for Object Detection
Keze Wang Xiaopeng Yan Dongyu Zhang Lei Zhang Liang Lin







Towards Human-Machine Cooperation: 
 Self-supervised Sample Mining for Object Detection
    Keze Wang^1,2     Xiaopeng Yan^1     Dongyu Zhang^1Corresponding author is Dongyu Zhang.     Lei Zhang^2     Liang Lin^1,3

^1Sun Yat-sen University    ^2The Hong Kong Polytechnic University    ^3SenseTime Research

kezewang@gmail.com; yanxp3@mail2.sysu.edu.cn; zhangdy27@mail.sysu.edu.cn

cslzhang@comp.polyu.edu.hk; linliang@ieee.org











    Received date / Accepted date
==============================================================================================================================================================================================================================================================================================================================================================

empty





Though quite challenging, leveraging large-scale unlabeled or partially labeled images in a cost-effective way has increasingly attracted interests for its great importance to computer vision. To tackle this problem, many Active Learning (AL) methods have been developed. However, these methods mainly define their sample selection criteria within a single image context, leading to the suboptimal robustness and impractical solution for large-scale object detection. In this paper, aiming to remedy the drawbacks of existing AL methods, we present a principled Self-supervised Sample Mining (SSM) process accounting for the real challenges in object detection. Specifically, our SSM process concentrates on automatically discovering and pseudo-labeling reliable region proposals for enhancing the object detector via the introduced cross image validation, i.e., pasting these proposals into different labeled images to comprehensively measure their values under different image contexts. By resorting to the SSM process, we propose a new AL framework for gradually incorporating unlabeled or partially labeled data into the model learning while minimizing the annotating effort of users. Extensive experiments on two public benchmarks clearly demonstrate our proposed framework can achieve the comparable performance to the state-of-the-art methods with significantly fewer annotations.





 

§ INTRODUCTION

In the past decade, object detection has gained incredible improvements both in accuracy and efficiency, benefiting from the remarkable success of deep Convolutional Neural Nets (CNNs) <cit.><cit.><cit.>. Through producing candidate object regions of input images, object detection is converted into the region classification task, e.g., R-CNN <cit.>. Recently, more powerful neural network architectures such as ResNet <cit.> have further pushed the object detection performance into new records. Behind these successes, massive data collection and annotation such as MS-COCO <cit.> are indispensable yet quite expensive. Under such a circumstance, there is an increasing demand of leveraging large-scale unlabeled data to promote the detection performance in an incremental learning manner. However, to achieve this goal, there remain two technical issues: i) Object annotation for training is usually labor-intensive. Compared with other visual recognition task (e.g., image classification), annotating object requests  to provide both the category label and bounding box of an object. In order to reduce the burden of active users, it is highly required to develop human-machine cooperation based approaches to self-annotate most of the unlabeled data; ii) Picking out the training samples that are advantageous to boost the detection performance is a non-trivial task. As figured out in <cit.>, existing detection benchmarks usually contain an overwhelming number of “easy” examples and a small number of “hard” ones (i.e., informative samples with various illuminations, deformations, occlusions and other intra-class variations). Utilizing these “hard” samples is a key to train the model more effectively and efficiently. However, as pointed out in <cit.>, due to following a long-tail distribution, these examples are quite uncommon. Hence, it is a sophisticated task to find “hard” yet informative samples.








To address the aforementioned issues, we investigate sample mining techniques to incrementally improve object detectors with minimal user effort. Recently, Active Learning (AL) <cit.> proposed to progressively select and annotate most informative unlabeled samples for user annotation to boost the model. Hence, the sample selection criteria play a crucial role in AL, and are typically defined according to the prediction certainty (confidence) or other informative criteria like diversity of samples. Recently, many AL methods <cit.> have been developed for training deep convolutional neural networks (CNNs). However, their sample selection criteria are dominantly performed under a single sample context. This make them sensitive to the bias of classifiers and the unbalance of samples.


Attempting to overcome the above-mentioned drawback of the existing AL approaches, this paper develops a Self-supervised Sample Mining (SSM) process for automatically mining valuable samples in terms of boosting performance of object detectors. Our developed SSM process is motivated by the recently popular self-supervised learning <cit.> technique. In stead of designing to learn a optimal visual representation as <cit.>, we concentrate on developing a rational pipeline of using the self-supervision to significantly decrease the amount of user annotations for improving detection performance. In the proposed SSM process, given the region proposals from unlabeled or partially labeled images, we evaluate their estimation consistency by performing the cross image validation, i.e., pasting them into different annotated images to validate its prediction consistency by the up-to-date object detector. Note that, to avoid ambiguity, the images for validation are randomly picked out from the labeled samples that do not belong to the estimated category of the under-processing proposal. Through a simple ranking mechanism, those incorrectly pseudo-labeled proposals have a large chance to be filtered due to the challenges inside various image contexts. In this way, the bias of classifiers and unbalance of samples can be effectively alleviated. Then, we propose to provisionally assign disposable pseudo-labels to the ones with high estimation consistency, and retrain the detectors within each mini-batch iteration. Since the pseudo-annotations may still contain errors, small amount of user interactions is necessary to keep our SSM under well control. 

By resorting to our SSM, we further propose a novel incremental learning framework to gradually incorporate unlabeled samples to enhance object detectors, as illustrated in Fig. <ref>.



In our framework, inspired by the recently proposed techniques: Curriculum Learning (CL) <cit.> and Self-paced Learning (SPL) <cit.><cit.>, we formulate the combining of the SSM and the AL as a concise optimization problem. Specifically, the SSM or AL process in our framework can jointly collaborate with each other. This is done by imposing a set of latent variables to progressively include samples into training. These variables determine whether a sample should be selected for pseudo-labeling or annotating. Meanwhile, the misleading of pseudo-labeled errors can be suppressed since the sample selection criterion is progressively optimized together with the batch-based incremental learning. In fact, the ambiguity of incorrectly annotated samples by users can also be eliminated, thanks to the correction of the majority pseudo-labeled samples. Hence, our SSM can further improve the robustness of classifiers against noisy samples/outliers in the pursuit of detection accuracy.

The main contributions of this work are two-fold. First, we propose a novel self-supervised process for automatically discovering and pseudo-labeling reliable region proposals via the cross image validation, which is compatible with the mini-batch based training for large-scale practical scenarios. Second, through fusing the proposed SSM process with the AL, we propose a principled framework with a concise optimization formulation and an alternative optimization algorithm.  Extensive experiments demonstrate that our proposed framework can not only achieve a clear performance gain by mining additional unlabeled data, but also outperform the dominantly state-of-the-art methods with significantly fewer annotations. 






§ RELATED WORK


Active Learning. 
This branch of works mainly focuses on the sample selection strategy, i.e., how to pick out the most informative unlabeled samples for annotation. One of the most common strategies is the certainty-based selection <cit.>, in which the certainties are measured according to the prediction confidence on new unlabeled samples. The diversity of the selected instance over the unlabeled data has been also considered in <cit.>. Recently, Elhamifar et al. <cit.> proposed to consider both the uncertainty and diversity measure via convex programming. Freytag et al. <cit.> presented a concept that generalizes previous methods based on the expected model change and incorporates the underlying data distribution. Vijayanarasimhan et al. <cit.> proposed a novel active learning approach in crowd-sourcing settings for live learning of object detectors, in which the system autonomously identifies the most uncertain instances via a hashing based solution. Rhee et al. <cit.> proposed to improve object detection performance by leveraging a collaborative sampling strategy, which integrates the uncertainty and diversity criteria from the AL and the feature similarity measurement of semi-supervised learning philosophy. However, these mentioned AL approaches usually emphasize those low-confidence samples (e.g., uncertain or diverse samples) while ignoring the rest majority of high-confidence samples. More recently, attempting to leverage these ignored samples, several works <cit.> have been proposed to progressively select the minority of most informative samples and pseudo-label the majority of high prediction confidence samples for network fine-tuning. Though these approaches have achieved promising performances, they have limitations due to that their defined hyper-parameters for pseudo-labeling is empirically set and updated within a single image context. Furthermore, these methods do not support mini-batch based training. Therefore, none of them has successfully proved their capability on handling large-scale object detection task. 

Self-paced Learning. Inspired by the cognitive principle of humans/animals, Bengio et al. <cit.> initialized the concept of curriculum learning (CL), in which a model is learned by gradually including samples into training from easy to complex. To make it more implementable, Kumar et al. <cit.> substantially prompted this learning philosophy by formulating the CL principle as a concise optimization model named self-paced learning (SPL). Recently, several works <cit.> provided more comprehensive understanding of the learning insight underlying CL/SPL, and formulated the learning model as a general optimization problem. 

Based on this framework, multiple SPL variants <cit.> have been proposed for object detection. Lee et al. <cit.> introduced a self-paced approach to focus on the easiest instances first, and progressively expands its repertoire to include more complex objects. Sangineto et al. <cit.> presented a self-paced learning protocol for object detection that iteratively selects the most reliable images and boxes according to class-specific confidence levels and inter-classifier competitions. Dong et al. <cit.> proposed an object detection framework that uses only a few bounding box labels per category by consistently alternating between detector amelioration and reliable sample selection. 


Self-supervised Learning. Aiming at training the feature representation without additional manual labeling, self-supervised learning (SSL) has first been introduced in <cit.> for vowel class recognition, and further extended for object extraction in  <cit.>. Recently, plenty of SSL methods<cit.> have been proposed, e.g., Wang et al. <cit.> proposed to employ visual tracking to provide the supervision for learning visual representations among thousands of unlabeled videos. Doersch et al. <cit.> investigated multiple self-supervised methods to encourage the network to factorize the information in its representation. Different from these methods that focus on learning an optimal visual representation, our SSM intends to use the self-supervision to mine valuable information from unlabeled and partially labeled data.





§ SELF-SUPERVISED SAMPLE MINING




 §.§ Formulation

In the context of object detection, suppose that we have n region proposals from m object categories. Denote the training sample set as 𝒟={x_i}_i=1^n⊂ R^d, where x_i is the i-th region proposal generated from the training images. We have m detectors/classifiers (including the background) for recognizing each proposal by the one-vs-rest strategy. Correspondingly, we denote the label set of x_i as 𝐲_i = {y_i}_j=1^m, where y_i^(j) corresponds to the label of x_i for the j-th object category. i.e., if y_i^(j)=1, this means that x_i is categorized as an instance of the j-th object category. We should give two necessary remarks on our problem setting. One is that most of sample labels 𝐘 = {𝐲_i}_i=1^n are unknown and need to be completed in the learning process. The initially annotated images are denoted by 𝐈. The other remark is that the data {x_i}_i=1^n might possibly be fed into the system in an incremental way. This means that the data scale might be consistently growing. 
The whole loss function for our proposed framework with SSM process is formulated as follows:


    Loss = ℒ_loc(𝐖) + ℒ_cls^AL(𝐖) + ℒ_cls^SSM(𝐖, 𝐕),
where ℒ_loc(𝐖) denotes the bounding box regression loss defined in <cit.>. ℒ_cls^AL(𝐖) and ℒ_cls^SSM(𝐖, 𝐕) imply the classification loss for the AL and SSM processes, respectively. We define the AL process as ℒ_cls^AL(𝐖) = 1/|Ω_I|∑_i ∈Ω_I∑_j=1^m ℓ_j(x_i, 𝐖), where Ω_I denotes the labeled proposals from the currently annotated image I (I ∈𝐈). ℓ_j(x_i, 𝐖) means the softmax loss of the proposal x_i in the j-th classifier:


    ℓ_j(x_i, 𝐖)=-(1 + y_i^(j)/2logϕ_j(x_i; 𝐖) + 
    1 - y_i^(j)/2log(1-ϕ_j(x_i; 𝐖))),
 
where 𝐖 represents the parameter of the CNN for all m categories (including background), ϕ_j(x_i; 𝐖) denotes the probability of belonging to the j-th category for each region proposal x_i. 

To adaptively select x_i for pseudo-labeling to update its 𝐲_i, our SSM process introduces a set of latent weight variables, i.e., 𝐕={𝐯^(j)}_j=1^m={[v_1^(j),⋯,v_n^(j)]^T}_j=1^m, and is formulated as:


    ℒ_cls^SSM(𝐖, 𝐕) = 
    1/|Ω_I|∑_i ∈Ω_I∑_j=1^mv_i^(j)ℓ_j(x_i, 𝐖) + R(x_i, v_i^(j), 𝐖) 
    s.t.  ∑_j=1^m | y_i^(j) + 1 |≤ 2, y_i^(j)∈{-1, 1},

where Ω_I denotes the unlabeled proposals from the unlabeled or partially labeled image I. The regularization function R(·) penalizes the sample weights linearly in terms of the loss. In this paper, we utilize the hard weighting regularizer due to its well adaptability to complex scenarios. The hard weighting regularizer is defined as:


    R(x_i, v_i^(j), 𝐖) = -f(x_i, 𝐖) v_i^(j).
 

Finally, we can directly calculate v_i^(j) as:


    v_i^(j) = {   1,    ℓ_j(x_i, 𝐖) ≤ f(x_i, 𝐖), 
       0,    otherwise. 
    .



 
In contrast to the existing works <cit.> that rely on only an empirical hyper-parameter for each category to control the loss tolerance, we exploit a sample-dependent manner, i.e., the cross image validation f(·), to include samples into training. Therefore, our model can be considered as a self-supervised self-paced learning framework. As illustrated in Fig. <ref>, the cross image validation is regarded as the estimation consistency of reliable region proposals. Specifically, f(·) is defined as:


    f(x_i, 𝐖) = 
    λ/|Ω_I^j|∑_p ∈Ω_I^j1(IoU(B_I(x_i), B_I(x_p)) ≥γ)  ϕ_j(x_p; 𝐖),
 
where Ω_I^j represents the labeled region proposals from the annotated image I without j-th category objects for consistency evaluation. λ denotes the pace parameter.
B_I(x_i) denotes the bounding box of the proposal x_i in the selected image I, while IoU(B_I(x_i), B_I(x_p)) implies the intersection of union between two bounding boxes B_I(x_i) and B_I(x_p). Note that, γ represents the threshold parameter to identify whether these two bounding boxes correspond to the same object, and it is set to 0.5 in all experiments according to the evaluation protocol of object detection. 1(·) is the indicator function. If the proposal x_p generated by the up-to-date detector from the newly pasted image includes the same object as x_i, i.e., IoU(B_I(x_i), B_I(x_p)) ≥γ, then we calculate its estimation consistency value ϕ_j(x_p; 𝐖), which denotes the possibility of x_p being the j-th category during the cross image validation. 







 §.§ Alternative Optimization Strategy

The alternative minimization is readily employed to solve this optimization. Specifically, the algorithm is designed by alternatively updating the sample importance weight set 𝐕 via the SSM process, the label set 𝐘 via pseudo-labeling and the AL process, and the network parameters 𝐖 via standard backpropagation. In the following, we introduce the details of these optimization steps. The convergence of this algorithm to the practical implementation of our framework will also be discussed.


Updating 𝐕: Fixing the {𝐘, 𝐗, 𝐖}, we can directly calculate f(x_i, 𝐖) via Eqn. (<ref>), and further obtain 𝐕 via Eqn. (<ref>). 

Updating 𝐘: After obtaining 𝐕, we calculate the consistency score s_i for sample selection as:


    j^* = max_j∈ [m]ϕ_j(x_p; 𝐖), 
    
    s_i = 1/|𝐈|∑_I ∈𝐈1/|Ω_I^j^*|∑_p ∈Ω_I^j^*ϕ_j^*(x_p; 𝐖).

where j^* represents the predicted category by the current detector with highest confidence.







Note that, to reduce the computation cost, we only randomly pick out at most N annotated images for evaluating the consistency of the proposal x_i. In all the experiments, we empirically set N = 5 for the trade-off between the accuracy and efficiency. Given 𝐒 = {s_i}_i=1^n, we rank all unlabeled samples in a descending order for each classifier as <cit.>, and pick out top-k non-zero ones at most for each object category. ℋ is regarded as high-consistency samples with assigned pseudo-labels. Specifically, these important samples for m categories are defined as ℋ=[H_1, ...,H_j,...,H_m] (|H_j| ≤ k), where k is an empirical parameter to control the number of selected samples for each category. 


Fixing {𝐖, 𝐕, {x_i}_i=1^ℋ}, we optimize 𝐲_i of Eqn. (<ref>) which corresponds to solve the following problem for each high-consistency sample i ∈ℋ with its important weight vector 𝐯_i ≠0:


    min_𝐲_i∈{-1, 1}^m, i ∈ℋ∑_j=1^m v_i^(j)ℓ_j(x_i, W),   s.t.∑_j=1^m| y_i^(j) + 1 |≤ 2,

where 𝐯_i is fixed, and can be treated as constant.
The constraint ∑_j=1^m| y_i^(j) + 1 |≤ 2 largely excludes all samples for pseudo-labeling except under the following two conditions: i) when y_i^(j) is predicted to be positive by one classifier but all 
other classifiers produce negative predictions, or ii) when all classifiers predict y_i^(j)
to be negative, i.e., x_i 
is rejected by all classifiers and identified as belonging to an undefined object category. These are the rational cases for practical object detection in large-scale scenarios. Note that we optimize 𝐘 by exhaustively attempting to assign -1 or 1 to each sample for all m categories to minimize Eqn. (<ref>). The computational cost of this process is acceptable because we only need to take  m+1 attempts. Through this fashion,  Eqn. (<ref>) always has a clear solution by enforcing pseudo-labels on those top-ranked high-consistency sample set ℋ. This is exactly the mechanism underlying a re-ranking technique <cit.>. Compared with the previous methods <cit.>, our framework can effectively suppress the error accumulation during the incremental pseudo-labeling via the following two advantages: (i) The cross image validation can provide more accurate and robust estimations under various challenging image contexts; (ii) All the pseudo-labels are disposable. They will be discarded after each mini-batch iteration. These advantages are beneficial for the detector to avoid being misled by the accumulate errors.












Low-consistency Sample Annotating: After pseudo-labeling high-consistency samples in such a self-supervised manner, we employ the AL process to update the annotated image set 𝐈 by providing more informative guidance based on human knowledge. The AL process aims to select most informative unlabeled samples and to label them as either positive or negative by requesting user annotation. Our selection criteria are based on the classical uncertainty-based strategy <cit.>. Specifically, we collect those samples with quite small s_i after performing the cross image validation. Then, we utilize the current classifiers to predict their labels. Those predicted as more than two positive labels (i.e., predicted as the corresponding object category) actually represent these samples making the current classifiers ambiguous. We thus adopt them as so called “low-consistency” samples and randomly add z of them into low-consistency sample set 𝒰 for further manually annotation by active users. Actually, other similar criterion can be utilized in this step. We employed this simple strategy just due to its intuitive rationality and efficiency.  








Updating 𝐖: Fixing {𝒟,𝐕, 𝐘}, the original model in Eqn. (<ref>) can be approximated as:


    min_𝐖1/|ℋ∪{Ω_I}_I ∈𝐈|∑_i ∈ℋ∪{Ω_I}_I ∈𝐈∑_j=1^mℓ_j (x_i, 𝐖) + ℒ_loc(𝐖).

Thus, we can update the network parameters 𝐖 by standard backpropagation. Note that, we do not consider the regularization function R(·), which is introduced to regularize the latent variable set 𝐕. 





The entire algorithm can then be summarized into Algorithm <ref>. It is easy to see that this algorithm finely accords with the pipeline of our proposed framework in Fig. <ref>. 


Convergence Analysis: Our framework can guarantee the convergence to a local optimum based on its implementation. The reason is three-fold: (i) the objective function Eqn. (<ref>) w.r.t 𝐕 is convex; (ii) network fine-tuning in Eqn. (<ref>) via backpropagation can converge to a local optimal; (iii) as the model becomes mature, the AL process stops when no low-consistency samples are found.







§ EXPERIMENTS



To justify the effectiveness of our proposed SSM process and framework, we have conducted a number of experiments on the public VOC 2007/2012 benchmarks <cit.>, whose data are usually divided into two-fold: trainval and test. To evaluate the model performance on the VOC 2007 benchmark, we regard the VOC 2007 trainval as the initial annotated samples, and consider the VOC 2012 trainval data as unlabeled data that need to be mined. Therefore, the active user annotating process equals to fetch the VOC 2012 annotations. As for the VOC 2012 benchmark, we employ the VOC 2007 trainval and test set for initialization. Moreover, we regard the large-scale object detection dataset COCO <cit.> as the `secondary' unlabeled data. In other words, we will perform sample mining on it only when all the VOC 2012 trainval annotations have been used. As for the annotation key `annotated' and `pseudo', the first one represents the proportion of the user annotations appended/fetched during the training over the pre-given annotations (i.e., the VOC 2007 trainval), which are used for initializing the object detectors. `pseudo' implies the percentage of pseudo-labeled object proposals from unlabeled data over the pre-given annotations. The lower `annotated' value is, the less user efforts for annotating are required. The higher `pseudo' value is, the more pseudo-labeled samples are obtained. Hence, when achieving the same performance, a superior method should have lower `annotated' but higher `pseudo' values. 

We adopt the PASCAL Challenge protocol, i.e., a correct detection should has more than 0.5 IoU with the ground truth bounding-box, and exploit the mean Average Precision (mAP) as for the evaluation metric. In all experiments, we set parameters {λ, k, N, γ, z} = {0.9, 500, 5, 0.5, 100}.  The fine-tuning manner for the RFCN pipeline is the same as <cit.>, except that we treat the COCO trainval set as unlabeled data for mining rather than pre-train the network. In the testing phase, we use a single scale of 600 pixels as input except for the VOC 2012 test set, which we use a multi-scale test as  <cit.>. All the experiments are conducted on a desktop with Intel CPU 3.4GHz and four Titan Xp GPUs. The testing time of our framework is 120 millisecond/image, and our training time is 620 millisecond/image. As for the base line method (i.e., RFCN), its testing time is 120 millisecond/image and training time is 150 millisecond/image. 





In order to demonstrate that our proposed framework is general to different network architecture and object recognition framework, we have incorporated our framework into the Fast R-CNN (FRCN) pipeline with AlexNet <cit.> and the new state-of-the-art RFCN <cit.> with ResNet-101 <cit.>. We denote these variants as “FRCN+Ours” and “RFCN+Ours”, respectively. To validate the superior performance of the proposed framework, we have compared it with the CEAL <cit.> and K-EM <cit.> approaches. Note that, since the CEAL is designed for image classification and is not mini-batch friendly, we extended it for object detection by alternatively performing sample selection and fine-tuning the CNN. Since the source code of K-EM <cit.> is not publicly available, we have obtained the results from their paper <cit.> directly. We denote these methods as “FRCN+CEAL,” and “FRCN+K-EM”, respectively. Moreover, we have also included five baseline methods that ignore the pseudo-labeling of unlabeled samples: “FRCN+RAND”, which randomly selects region proposals for user annotations,  FRCN+EMC (Expected-Model-Change) <cit.>, FRCN+Entropy <cit.>, FRCN+ELC (Expected-Labeling-Change) <cit.> and FRCN+Density <cit.>. For a fair comparison, we initialize all the methods by providing only 5% annotations, and allow FRCN+Ours and FRCN+CEAL to mine unlabeled samples only from the VOC 2007 train/val set. Moreover, we repeat the testing by five trails to report the average mAP with standard variance to present a comprehensive evaluation. 






 



















































































 §.§ Results and Comparisons

The Fig. <ref> demonstrates the comparison of detection performance using the Fast-RCNN (FRCN) pipeline with AlexNet <cit.> on the VOC 2007 test set. As illustrated in Fig. <ref>, our proposed framework FRCN+Ours consistently outperforms all the compared methods by clear margins under all annotation settings. Specifically, FRCN+Ours can achieve the equivalent of a fully supervised performance (i.e., FRCN with 100% user annotations) when fetching only approximately 30% user annotations, while most of the compared methods require nearly 60%. These results indicate the superior performance of our framework.


To demonstrate the feasibility and great potential of our framework, we have conducted amount of experiments to fine-tune RFCN with ResNet-101 (well pre-trained on ImageNet) on the VOC 2007/2012 trainval set by using our framework and the compared baseline RFCN+RAND. The compared results on the VOC 2007 and 2012 benchmarks are illustrated in Tab. <ref> (a)(b), respectively. By controlling the number of training iterations, RFCN+Ours and RFCN+RAND can fetch different amounts of annotations from the VOC 2012 trainval set. 


As one can see from Tab. <ref> (a)(b), both RFCN+RAND and RFCN+Ours gradually obtain increased detection accuracy when the number of annotations increases. Our framework consistently outperforms the compared baseline RFCN+RAND under all appending conditions on both the VOC 2007 and 2012 benchmarks by a clear margin. Moreover, our framework surpasses RFCN+RAND by nearly 1.5% (80.6% vs 79.1%) on the VOC 2007 benchmark and 1.3% (78.1% vs 76.8%) on the VOC 2012 benchmark with significantly small variations when sufficient pseudo-labeled object proposals are provided. This validates the effectiveness of our framework. Some examples of the selected high-consistency and low-consistency region proposals via the cross image validation are depicted in Fig. <ref>.








 §.§ Ablation Study

To validate the contribution of each component inside our framework, we have also conducted sufficient experiments for empirical analysis. The variant of our framework that discarding the active learning process is denoted as “FRCN+SSM” / “RFCN+SSM”. Similarly, these pipelines with Active Learning (AL), Self-paced Learning (SPL) are denoted by “RFCN+AL”, “RFCN+SPL”, respectively. RFCN+AL adaptively collects low-confidence proposals to request the annotations and stops when no low-confidence samples are found. RFCN+SPL is implemented according to <cit.>.



As illustrated in Tab. <ref> (c)(d), given the same amount of annotations during initialization, RFCN+SSM performs significantly better than RFCN+SPL on both VOC 2007 and the VOC 2012 test set. Specifically, RFCN+SSM achieves a nearly 2% performance improvement (76.7% vs 74.7%) with small variations over RFCN+SPL by pseudo-labeling the same amount of high-consistency region proposals for training on the VOC 2007 benchmark. A consistent performance gain of approximately 1.2% (72.1% vs 70.9%) is obtained on the VOC 2012 benchmark by RFCN+SSM. These results validate the significant contribution of the proposed SSM process on mining reliable region proposals for improving object detection.


We have also compared our self-supervised sample mining (SSM) process with three baseline methods under the AL process disabled setting. RFCN+Flip implies horizontally flipping images for validation, while RFCN+Rescale represents randomly rescaling an image from 50% to 200% of its original size. RFCN+Flip&Rescale means the fusion of them. As shown in Fig. <ref>, RFCN+SSM consistently outperforms all the competing methods by clear margins at all pseudo-labeling proportions. Moreover, compared to these baselines, RFCN+SSM obtains a slighter performance drop (caused by the accumulated pseudo-labeling errors) after reaching its peak performance. This also proves the effectiveness of our SSM for suppressing the error accumulation.

Tab. <ref> (c)(d) also demonstrates that RFCN+AL consistently outperforms the baseline RFCN and RFCN+SSM. Though the improvements are minor compared to the best performance of the RFCN+SSM, our proposed AL stage is still beneficial for promoting object detection. This slight improvement occurs because the informative samples with great potential for improving performance follow a long-tail distribution, as reported in <cit.>. Therefore, it is necessary to employ abundant training samples by asking active users to provide labels or finding other assistance. Fortunately, our proposed high-consistency sample pseudo-labeling via the SSM process is an effective way to address this issue.




The results of weaker initialization (5% and 10% annotations from the VOC 2007 train/val set) are listed in Tab. <ref>. As shown, our RFCN+SSM achieves a consistent performance gain of about 2% over the original RFCN. Compared to RFCN+RAND, RFCN+Ours obtains about 1.5% higher average mAP with much lower variances. However, compared to RFCN+Ours, FRCN+Ours in Fig. <ref> obtains a higher mAP gain. The reason is FRCN and RFCN use different algorithms (Selective Search <cit.> vs. Region Proposal Network <cit.>) to generate object proposals. Since our objective is to mine samples from these proposals, our model obtains various performance boosts based on the quality of proposals. This shows the generality and effectiveness of our model over different proposals.






§ ACKNOWLEDGMENT

This work was supported in part by the Hong Kong Polytechnic University’s Joint Supervision Scheme with the Chinese Mainland, Taiwan and Macao Universities (Grant no. G-SB20). This work was also supported by HK RGC General Research Fund (PolyU 152135/16E), in part by Guangdong “Climbing Program” Special Funds under Grant pdjhb0010, in part by National Natural Science Foundation of China (NSFC) under Grant U1611461 and Grant 61702565, and in part by Science and Technology Planning Project of Guangdong Province of No.2017B010116001.





§ CONCLUSION


In this paper, we have introduced a principled Self-supervised Sample Mining (SSM) process, and justified its effectiveness in mining valuable information from unlabeled or partially labeled data to boost object detection. We further involve this process in the AL pipeline with a concise formulation, which is developed for retraining object detectors via faithfully pseudo-labeled high-consistency object proposals after our proposed cross image validation. The proposed SSM process contributes to effectively improve the detection accuracy and the robustness against noisy samples. Meanwhile, the rest samples, being low consistency (high uncertainty) by the current detectors, can be handled by the AL, which benefits to generate reliable and diverse samples gradually. In the future, we will apply our SSM to improve other specific visual detection task with unlabeled web images/videos.


ieee



