Weakly Supervised Dataset Collection for Robust Person Detection
Munetaka Minoguchi Ken Okayama Yutaka Satoh Hirokatsu Kataoka
cs.CV







Weakly Supervised Dataset Collection
for Robust Person Detection
    Munetaka Minoguchi, Ken Okayama, Yutaka Satoh, Hirokatsu Kataoka

National Institute of Advanced Industrial Science and Technology (AIST)

Tsukuba, Ibaraki, Japan













    
===================================================================================================================================================================================





   To construct an algorithm that can provide robust person detection, we present a dataset with over 8 million images that was produced in a weakly supervised manner. Through labor-intensive human annotation, the person detection research community has produced relatively small datasets containing on the order of 100,000 images, such as the EuroCity Persons dataset, which includes 240,000 bounding boxes. Therefore, we have collected 8.7 million images of persons based on a two-step collection process, namely person detection with an existing detector and data refinement for false positive suppression. According to the experimental results, the Weakly Supervised Person Dataset (WSPD)  is simple yet effective for person detection pre-training. In the context of pre-trained person detection algorithms, our WSPD pre-trained model has 13.38 and 6.38% better accuracy than the same model trained on the fully supervised ImageNet and EuroCity Persons datasets, respectively, when verified with the Caltech Pedestrian. The dataset and pre-trained models used in the paper are publicly available on the GitHub[<https://github.com/cvpaperchallenge/FashionCultureDataBase_DLoader>]. The paper is under consideration at Pattern Recognition Letters.





§ INTRODUCTION














In the context of navigation and service robots, an appropriate human-centered operating environment usually starts with person detection. Therefore, we require robust and highly accurate person detection for collision avoidance for the realization of self-driving cars, mobile robots, and unmanned aerial vehicles.

To construct a learning-based object detector

(In this paper, the meaning of “object” detection includes person detection; in other words, “object” is taken as having a broader meaning)
, a large-scale and well-labeled dataset is required, as is a suitable model architecture. Towards this end, large-scale multiple-object datasets, such as MS COCO <cit.> and OpenImages <cit.>, have been produced to conduct highly accurate object detection. However, the datasets for person detection are currently relatively small. For example, the Caltech Pedestrian <cit.>, CityPersons <cit.>, and EuroCity Persons datasets <cit.> contain 350,000, 35,000, and 240,000 bounding boxes (bboxes), respectively. Compared to million-image multiple-object datasets like OpenImages, pedestrian datasets could still be greatly expanded to improve the performance of models that are trained with these datasets. Thus, there is great motivation to create a large-scale person dataset with a bboxes quantity on the order of millions.

Inspired by weakly supervised image labeling with Social Network Service (SNS) hashtags, as in the so-called “Instagram-3.5B” study <cit.>,  we conducted semi-automatic large-scale dataset collection in the context of person detection. Unlike the related work in Instagram-3.5B, our proposed dataset contains a large number of bboxes in addition to the captured images. We here consider a different method for constructing a pre-trained person dataset based on an existing detector and simple refinement.

 

This paper proposes a million-image person dataset based on a weakly supervised method for robust person detection, namely Weakly Supervised Person Detection (WSPD). Our large-scale person dataset is constructed by semi-automatic image collection and data refinement with SNS images. This dataset collection method allows us to significantly improve the performance of person detection with human annotation in a few hours. When used as a pre-trained person dataset, the model trained with the WSPD method outperforms the detection rate of models trained with fully supervised pre-trained datasets, such as EuroCity Persons (+6.38%) and ImageNet (+13.36%) on the Caltech Pedestrian with a Single-Shot multibox Detector (SSD) <cit.>. 

This paper makes the following contributions to person detection (see also Figure <ref>). (i) We propose a weakly supervised dataset collection method for building training datasets for person detection. We then use this method to construct a dataset containing millions of images with bboxes through an existing detector (e.g., Faster R-CNN <cit.>) and false positive suppression. (ii) The WSPD pre-trained model is demonstrated to perform well at person detection. The fine-tuned detector achieved  an accuracy 6.38 and 13.36%  higher than that of the baseline models (EuroCity Persons and ImageNet, respectively) on the Caltech Pedestrian. We provide examples of the detection results and performance comparisons in Figure <ref>.


 






§ RELATED WORK


In this section we briefly review some of the key concepts related to this paper, such as object detection, annotation, and dataset collection, to help highlight how the proposed method differs from existing methods.


Object detection. Object detection algorithms have progressed from hand-crafted detection using local features (e.g., Haar-like features <cit.>, HOG <cit.>, and ICF <cit.>) and well-organized classifiers (e.g., Deformable Parts Model <cit.> and aggregated detectors <cit.>), and currently we are in the era of deep neural networks (DNNs). In the literature, a two-step region identifier and DNN-based classifier has been proposed <cit.>. The basic technique, called R-CNN, has been adapted for use with any-size feature maps <cit.>, and it includes an end-to-end two-step method <cit.>. Current research is widely divided between one-shot detectors, such as you only look once (YOLO) <cit.> and SSD <cit.>. Recent studies have also focused on highly accurate detectors, such as RetinaNet <cit.> and M2Det <cit.>, and instance segmentation with Mask R-CNN <cit.>. Here, we use SSD, which is a balanced detector with a relatively short training time. Further, it can easily be optimized for use with baseline models to compare the dataset prepared with our collection method and the ImageNet and EuroCity Persons datasets in the context of a pre-trained detector. Moreover, we implemented M2Det to identify which person detector is more accurate.

Person detection. According to a comprehensive survey <cit.>, the performance rate of person detection algorithms has increased over the last decade as person detectors have evolved to use more sophisticated architectures. A recent study has proposed several configurations to improve recognition and localization with DNNs <cit.>, semantic meaning <cit.>, combined methods <cit.>, and analysis of small images or crowds <cit.>.

However, a large-scale person dataset must be prepared for training these models (e.g., SSD or M2Det) and fine-tuning their architecture.

Annotation for object bboxes. In recent machine learning research, annotation treatment has been shown to be important for successful network training. For example, Su et al. introduced a method for repeatedly checking bbox annotations in an image in three steps: drawing, quality checking, and coverage verification <cit.>. Papadopoulos et al. proposed a method that combines an existing detector and human annotation <cit.>. The combined method iterates between three annotation and quality control steps: model retraining, bbox relocalization, and human verification. Their annotation method results in an object detection dataset without human-drawn bboxes. Compared with these annotation methods, our proposed dataset collection method requires only a minimum of human-based annotation checks to improve the performance of person detection.

Person dataset collection. In addition to changes in the models used for person detection, person datasets have also evolved over the last decade. The first generation of person detection datasets consisted of small training and testing datasets (up to 10,000 images, including INRIA <cit.>, Daimler <cit.>, and ETHZ <cit.>), followed by a second generation of medium-size datasets (10,000–100,000 images, including Caltech Pedestrian <cit.>, CityPersons <cit.>, and EuroCity Persons <cit.>) that include occlusion and cluttered backgrounds. However, to the best of our knowledge, a large-scale person dataset (over 1 million images) is not currently freely available. In <cit.>, it was claimed that more high-quality person annotations are required for improving the accuracy of person detection algorithms. Data collection with weak supervision is one area of ongoing work in image classification <cit.>. As discussed in related work <cit.>, the increasing scale of datasets is also important for improving the accuracy of existing detection algorithms. To help produce large-scale datasets, we here present a weakly supervised pre-training dataset annotation method for person detection.


 




§ WEAKLY SUPERVISED DATASET COLLECTION




 §.§ Overview


To obtain a better representation of persons for detection during pre-training, a large-scale dataset with bboxes should be used, such as a combination of our WSPD pre-trained model and the Caltech Pedestrian <cit.> fine-tuned detector. The image localization labeling depends on the efforts of human annotators; therefore, an automatic dataset creation method would be useful for the person detection research community. 



Figure <ref> illustrates the concept of weakly supervised dataset collection. After collecting a large number of SNS images, we apply a two-step algorithm for weakly supervised dataset construction: person detection with an existing object detector and erasing false positives using a binary classifier. Here, we describe the problem setting to conduct our weakly supervised dataset collection. The setting of weakly supervised learning is simple yet effective for pre-training a person detector. At the beginning, we assign an object detector D to recognize bboxes and their labels from an input image x.


    (y^', b_box^')    =    D(x; θ),


where y^' and b^' denote predictions of object category and bboxes, respectively, and θ represents trained parameters in the detector. In case of person detection, the category is limited to the “person” label. The equation is simplified as follows:


    b_box^'   =    D(x; θ).


We used a support vector machine (SVM) to refine the detected bboxes with D. We used weakly supervised dataset collection to classify the detected bboxes as person ground truth (y_gt) or background (y_bg). The following equation shows the binary classifier with SVM:


    f(x)    =    w^TI(D(x; θ)) + b,


where I(*) represents a cropped image with detected bbox. The cropped image is divided depending on the f(x).


    g(x) = 
        y_gt    (f(x) ≥ 0) 
    
        y_bg    (f(x) < 0)
      ,


where y_gt is assigned to add a person label in the WSPD method. We use the Faster R-CNN as a person detector <cit.> and a binary classifier for determining whether the target's whole body is contained within the bbox. Our framework is simple yet effective for generating a large-scale dataset. We use the Yahoo! Creative Commons 100M Database (YFCC100M) <cit.>, which contains 100 million Flickr images. Our person dataset (WSPD) contains images of people from around the world but is limited to specific major cities <cit.>. The dataset consists of 2,822,421 original images and 8,716,461 person images (in bboxes). To the best of our knowledge, this is the largest person dataset for bbox-based detection currently available (see Table <ref>).











 §.§ Data collection, refinement, and configuration


Collection. We downloaded images from 21 global cities based on <cit.>; however, we excluded cities having fewer than 100,000 collected images. Consequently, 16 of the 21 cities were selected for the WSPD: London, New York, Boston, Paris, Toronto, Barcelona, Tokyo, San Francisco, Hong Kong, Zurich, Seoul, Beijing, Bangkok, Singapore, Kuala Lumpur, and New Delhi (listed from most images to fewest images).  These metropolitan areas do not overlap, as they are at least 200 km apart.  To create the images with bboxes, we applied the Pascal VOC pre-trained VGG16 model for the Faster R-CNN.
We initially set the threshold value as 0.8 and used only the person label.
A dataset consisting of a geo-tag and a time-stamp was replicated from the YFCC100M dataset. 
In the first step, we collected 76,532,519 images using automatic image collection and bbox annotation.



Refinement. We now consider how to exclude noisy images from our dataset.
The refinement strategy is to scan all images with a simple classifier based on a combination of StyleNet <cit.> and a SVM.
To create a sophisticated fashion-oriented database, we treat the WSPD refinement as a binary classification problem to distinguish between street-fashion-snapshot whole-body images and other cropped images, such as partial bodies or backgrounds without a person.
We trained and refined the database with 1,443 carefully annotated objective images and a large number of randomly cropped negative images.


Configuration. 
Our WSPD has the three following features:

    
  * Images captured from the YFCC100M dataset. After data refinement, the number of images decreased from 8,504037 to 2,822,421 original images.
    
  * Cropped person images with bboxes (treated as clothing images). As a result of data refinement, the number of person bboxes was reduced from 76,532,519 to 8,716,461.
    
    
  * Geo-location and time-stamp information. This relates to the 16 cities listed above.









Details of datasets. We give details on the self-collected dataset in Table <ref>. We also compare the proposed dataset (WSPD) with existing datasets for object and person detection. It is clear that our dataset contains the largest number of images and bboxes among the currently available person datasets. Also, our dataset contains a diversity of person images from different locations worldwide. From the semi-automatic dataset collection, we obtained millions of person bboxes that can be useful for training and testing a pre-trained detector.




Dataset quality analysis. We manually analyzed the results of our method using 1,000 randomly selected person bboxes from the WSPD dataset. Figure <ref> shows the four classifications for the randomly selected images, and Table <ref> indicates the corresponding frequency of occurrence of each class. The four bbox classifications were (i) high-quality annotation, (ii) low-quality annotation (partial image of a person), (iii) multiple persons in a bbox, and (iv) misclassification (not a person). Based on our random sample of 1,000 images, we expect the collected dataset to consist of 93% person images (i, ii, and iii combined), regardless of whether the images are perfectly annotated. Non-person images account for only 70 out of 1,000 bboxes. According to the Instagram-3.5B paper <cit.>, 10 and 25% noise reduced the performance rate by only 1.0 and 2.0%, respectively.


The effectiveness of the proposed weakly supervised dataset collection is shown in Section 5. We have considered multiple pre-trained datasets. 

 













§ CONFIGURATION FOR DETECTORS


In this section, we describe a suitable base model and training configuration. 



 §.§ Representative architecture in person detection

To assess the performance achieved when using the proposed WSPD collection method, we consider different types of representative detectors, namely the SSD <cit.> and M2Det <cit.>. In our explorative analysis, we utilized the WSPD dataset containing over 8.7 million bboxes to optimize the network parameters for bbox regression and person classification. The hyperparameter settings used here were the same  as those in <cit.>.






















 §.§ Training method for each dataset


We conducted pre-training with our WSPD and fine-tuning for each person dataset. Throughout the experiment, we evaluated the pre-trained models; therefore, fine-tuning was conducted for all pre-trained models on the pedestrian dataset. Our WSPD pre-trained model is compared with three different models: ImageNet, Pascal VOC, and EuroCity Persons pre-trained detector. The ImageNet pre-trained model is trained with a large number of images, but no bboxes are used during the pre-training. In contrast, the EuroCity Person pre-trained detector uses 240,000 person bboxes in the pre-training step. Moreover, the Pascal VOC pre-trained detector is not limited to person bboxes, and it has 20 object annotations. We show the procedures used for pre-training and fine-tuning in Table <ref>. We used the Caltech Pedestrian in the fine-tuning step.












§ EXPERIMENTAL RESULTS AND DISCUSSION




This section clarifies how the use of the weakly supervised dataset collection influences the accuracy of a pre-trained person detector. The weak but numerous annotations enable us to improve the performance in the fine-tuning task. The resulting accuracy is higher than that of other fully supervised pre-trained models (Table <ref>). We also present the results of the exploratory analysis and compare the performance when using different detection architectures.




 §.§ Exploratory study







The purpose of the exploratory study was to optimize the  hyperparameters for each architecture using the self-assembled dataset. Although there are numerous hyperparameters that must be selected in the detection architecture and learning strategy, we examined the effects of batch size {128, 256} and #epoch {25, 50, 100} with the WSPD method, as they seem to be the most important for model training. Therefore, we here calculate six different pre-trained SSD models in the pre-training phase. In addition to using the pre-trained detectors, we conducted further fine-tuning on a target dataset. To simplify the parameter tuning step, we employed the SSD (detection architecture), WSPD (pre-trained dataset), and Caltech Pedestrian (fine-tuned dataset).


The exploration results for the Caltech Pedestrian are shown in Table <ref>. The table shows the change in the miss rates (lower is better) for different numbers of batches and pre-training epochs. According to the results, we can confirm that 128 batches and 100 pre-training epochs provide the best performance.  Additionally, we found that the number of pre-training epochs tends to perform better when the pre-training time is longer. However, a smaller  number of pre-training epochs must be considered because the training time with 8.7 million bboxes is high. Especially during pre-training with the WSPD, the average training time is roughly 39 hours per epoch on four NVIDIA Tesla V100 GPUs. Therefore, pre-training with 100 epochs requires approximately 3,969 hours (165 days). Undoubtedly, the longer training will result in better pre-training results, but we must consider a more reasonable training time on larger detection architectures like M2Det. 







 §.§ Comparison with baseline models








We consider the detection results in detail for each architecture, backbone network, pre-trained dataset, and miss rate in Table <ref>. We focus on the validation of the architectures (SSD/M2Det) and pre-trained datasets (ImageNet, Pascal VOC, EuroCity Persons, and WSPD). We discuss the results for both the SSD and M2Det architectures.


SSD. Figure <ref> shows the results for the proposed method and three baselines with the SSD architecture. The difference between our proposed method and the baselines for the pre-training tasks is shown in Table <ref>. Basically, the pre-training results with the WSPD are significantly different because the dataset was collected in a weakly supervised manner. We confirmed that our WSPD pre-trained model achieved the highest score of 20.54%, which is a 6.38% and 13.36% better miss rate than the models pre-trained on EuroCity Persons and ImageNet, respectively. Note that the weakly supervised dataset collection for person detection was processed by a two-step algorithm using an existing detector and binary classification. Despite the presence of noise in the dataset, our method outperformed the fully supervised bboxes developed by human annotators. This result suggests that we can automatically generate a ground truth dataset in a simple way. The performance rate is higher than for the Pascal VOC pre-trained model (our method has an 8.74% better miss rate), which assigns multiple object detection labels in the pre-trained phase.


M2Det. In addition to the SSD model, we considered the M2Det (320/512) model. The M2Det detector represents the current state of the art in terms of detector accuracy. As described above, we compared the self-collected WSPD with ImageNet, Pascal VOC, and EuroCity Persons in the context of the pre-trained dataset with M2Det512 (see Figure <ref>). According to Table <ref>, the best miss rate was 16.44% with M2Det320. The miss rate is 4.01% better than that for the WSPD pre-trained SSD. The ImageNet pre-trained M2Det512 had a 32.46% miss rate on the Caltech Pedestrian. 


Moreover, we list the detection comparisons and results in Figure <ref> and Figure <ref>, respectively. 





 






























 §.§ Noise label analysis




Additionally, we investigated the effect of label noise.

In addition to the manual data quality analysis (see Table <ref>), we analyzed the relationship between the amount of noise in the dataset and the detection accuracy. We deliberately added a bbox translation with horizontal and vertical movement in the (x,y) coordinates. The procedure of making noise is shown at the bottom of Figure <ref>. We translated a bbox in the image and prevented it from overlapping a ground truth bbox.  We simultaneously list the relationship between noise rate and miss rate in the top of Figure <ref> and in Table <ref>. In the figure and table, 0% noise (miss rate of 23.86%) corresponds to normal training  and 100% noise (miss rate of 38.62%) corresponds to translating all bboxes. In the experiment, note that the data was obtained as 1 million bboxes randomly selected from the WSPD dataset; therefore, the miss rate is different from the 20.55% with 8.7M bboxes shown in Table <ref>.

According to the results, 30% noise produced only a small increase in the miss rate (from 23.86 to 24.81%, a difference of 0.95%). This confirmed that a small amount of noise does not greatly affect the performance rate of person detection. At noise rates greater than 30%, the miss rate continued to increase at 40% (miss rate of 26.82%, difference of 2.96%) to 80% noise (miss rate of 28.98%, difference of 5.12%). The 90 and 100% noise rates produced the worst results, with differences of 9.17 and 14.76% from the normal training rate. In particular, the results with the 100% noise rate are worse than those with ImageNet pre-training with the Caltech Pedestrian.






















































































§ CONCLUSION


This paper proposes a weakly supervised dataset collection method for improving pre-trained person detection models. In a comparison with the baseline detectors (e.g., ImageNet pre-trained model and person dataset fine-tuned model), our proposed method achieved a 6.38 and 13.36% better miss rate than the EuroCity Persons and ImageNet pre-trained models, respectively. The semi-automatic image and bbox collection can be performed by downloading images from SNS (e.g., Flickr), and using an existing detector (e.g., Faster R-CNN) for binary classification to determine whether an image contains a person's whole body. The weakly supervised dataset collection approach is simple yet highly effective for producing a pre-trained detector. We confirmed that using a large number of bboxes (8.7 million boxes in the WSPD dataset) in the pre-training task results in performance much better performance than that of the baseline detectors.

ieee_fullname



