ClickBAIT-v2: Training an Object Detector in Real-Time
Ervin Teng Rui Huang Bob Iannucci








ClickBAIT-v2: Training an Object Detector in Real-Time
    Ervin Teng, Rui Huang, and Bob Iannucci
Department of Electrical and Computer Engineering 

Carnegie Mellon University

NASA Ames Research Park, Building 23 (MS 23-11), Moffett Field, CA 94035

{ervin.teng, rui.huang, bob}@sv.cmu.edu
    December 30, 2023
=============================================================================================================================================================================================================================================

 

Modern deep convolutional neural networks (CNNs) for image classification and object detection are often trained offline on large static datasets. Some applications, however, will require training in real-time on live video streams with a human-in-the-loop. We refer to this class of problem as time-ordered online training (ToOT). These problems will require a consideration of not only the quantity of incoming training data, but the human effort required to annotate and use it. We demonstrate and evaluate a system tailored to training an object detector on a live video stream with minimal input from a human operator. We show that we can obtain bounding box annotation from weakly-supervised single-point clicks through interactive segmentation. Furthermore, by exploiting the time-ordered nature of the video stream through object tracking, we can increase the average training benefit of human interactions by 3-4 times. 













§ INTRODUCTION


Today's workflow for creating a convolutional neural network (CNN) application, such as an object detector, is to (a) obtain training dataset(s), perhaps from the Web or gathered in the field; (b) upload these to a training server; (c) run the training algorithm; (d) obtain resulting weights from the training server and (e) deploy to the end device. This static, offline approach, while effective, is a limiting factor in many applications. Consider, for instance, CNNs onboard small autonomous systems, such as unmanned aerial systems (UAS). CNNs can dramatically reduce the mental burden of human operators by automating much of the video processing. However, statically-trained CNNs will leave the UAS ineffective and vulnerable when it faces conditions and changes in the environment unexpected during the model creation process. 

Imagine a UAS is trained to find and track a particular person during a search-and-rescue or surveillance operation. Perhaps it is discovered early into the mission that the CNN model deployed to the UAS is performing poorly at human detection from an overhead angle. Or, perhaps during the mission the target person changes clothes—and immediately, the UAS that was following him or her loses track. In a statically-trained model workflow, re-gathering imagery, re-training, and re-deploying the model is a cumbersome process, and would result in the termination of the mission. 

People, however, learn very differently. They are not only trained statically beforehand, but are also field-trained,  they practice in situations resembling the usage scenario, with a trainer providing positive or negative reinforcement. Training does not terminate after the training phase—new information and experiences gathered “on the job” can also better the trainee. By analogy, we can imagine training a CNN-equipped UAS by showing it examples of what it's looking for (and what it's not looking for), over time building a better and better model—just like showing a child flash cards. Even during use, if the CNN behaves incorrectly, the user can correct its behavior. This type of training will require ingestion and annotation of data streams, such as video, in real-time. To do so, we must make special consideration of the human effort required to create image annotations. We call this class of problem time-ordered online training (ToOT).












In <cit.>, the authors present ClickBAIT, a system to perform online, real-time training of an image classifier onboard a small UAS. In this paper, we present ClickBAIT-v2, extending the system to object detection without increasing the annotation difficulty. Figure <ref> shows how ClickBAIT-v2 assists and automates part of the annotation process so that time-ordered online training is feasible in real-time. The main contributions are:





  * A weakly-supervised method for using user clicks to obtain bounding box annotations, eliminating the need for drawing them on every image. 


  * A combined approach to greatly increase the effectiveness of human input during time-ordered online training, combining click-based supervision with object tracking. 


This paper is structured as follows. Section <ref> provides an overview of the state-of-the-art. Section <ref> describes the design of ClickBAIT-v2 and its subsystem. Section <ref> describes an evaluation of the weakly supervised bounding box generation, and Section <ref> evaluates the effectiveness of ClickBAIT-v2 in reducing user interaction while training.
















§ BACKGROUND AND PRIOR WORK






Training CNNs has traditionally been done offline, whether on large datasets such as ImageNet <cit.>, on synthetic datasets <cit.>, or on collected datasets <cit.>. Active learning has recently been applied to human-in-the-loop deep image classification <cit.>, but requires the network to choose its images—something not possible with the continuous progress of time in a video sequence. Weakly-supervised training can train detection via class labels <cit.>, and unsupervised training of foreground detection can be achieved using non-learning methods <cit.>, but these methods have an inherent class-level or instance-level ambiguity. We use user interaction via clicks to remove this ambiguity.

Fine-tuning or transfer learning is the process of taking pre-trained weights and refining them to fit a particular application, using the fact that many of the features within a trained network are common across applications <cit.>. In <cit.> investigate continuous learning as a sub-problem of fine-tuning, and conclude that continuous, online learning can be achieved by directly fine-tuning the network in small steps. We will examine our particular problem, time-ordered online training, as a subset of continuous learning.

Weakening the supervision, from bounding boxes to clicks, required to train a model, is one way to make datasets easier to annotate. A prior approach to click supervision for object detection <cit.> does show significant success in reducing annotation time–however, their MIL-based approach is unsuited for online execution. Similarly, we found that multiple-interaction <cit.> and proposal-mining <cit.> methods for weakening supervision unsuited for real-time use, opting for a single-click approach. 



 §.§ Measuring the Effectiveness of a User Interaction


In <cit.>, the authors introduce time-ordered online training (ToOT) as the real-time training of a classifier on a live video stream, using human-in-the-loop annotations. They present a metric, training benefit, that describes the effectiveness of each user interaction, defined in relation to the trained model's accuracy, with the premise that human annotation will quickly become the bottleneck during real-time training. 

In this paper, we extend this definition to include models-under-training other than classifiers, and define cumulative and incremental training benefit <cit.> in terms of a scalar performance value P_i. Performance can indicate a number of metrics depending on the task at hand; for object detection, average precision (AP) is a suitable metric. We reserve P_0 as the performance of the untrained network.



For a vector of n video frames V = (v_1, v_2, ..., v_n) and a corresponding vector of user interactions 

    U = (u_1, u_2, ..., u_n),   u_i =  
          1    user int. on frame v_i
    
          0    otherwise

we define the cumulative training benefit of frames up to and including frame v_i as


    CTB_i =  
          P_i-P_0    i>0, i<=n
    undefined    otherwise


Subsequently, the CTB of the particular user interaction u_i is given by


    CTB_u_i =  
          P_k-1-P_0    u_i = 1
    undefined    otherwise

where k is the index of the next non-zero u_k, k > i. 

Likewise, the incremental training benefit of a user interaction taken during training round i can be defined by

    ITB_u_i =  
          P_k-1-P_i-1   u_i = 1
    undefined    otherwise


We can then compute the mean ITB for training rounds i...j as follows:


    ITB_(i,j) = ∑_x=i^j I_x/∑_x=i^j u_x


 
    I_x =  
          ITB_u_x   u_i = 1
    
          0     otherwise

As in  <cit.>, we make the stipulation that we terminate training once the model reaches some suitable final performance P_f. If we fix P_f, the mean ITB for the sub-vector of user interaction (u_1, ..., u_f) becomes a measure of the particular efficiency of a training algorithm, i.e. how well each user interaction is being used. 




§ TRAINING SYSTEM ARCHITECTURE


We consider the challenge of training a UAS to detect objects of interest within its field of view, while in-flight. While the system is operational, we perform online training for detection of a specific object instance. This object does not need to belong to a specific class, and the training process itself is class-agnostic. Furthermore, we make the constraint that any user interaction while the system is running must be a single click—and not time-consuming as dragging a bounding box. This section describes the three main components of  our system: the online-trained object detector; the segmenter, which allows bounding box supervision from a single click; and the object tracker, which multiplies each user interaction.



 §.§ Online Training of Object Detection


Pre-training: Due to hardware and real-time constraints, we chose the single shot object detector (SSD) <cit.> for its high framerate as compared to region proposal methods. We used the original SSD300 formulation and architecture with VGG-16, re-implemented in TensorFlow <cit.>. In order to convert the original 20-class model trained on PASCAL VOC2007+2012 <cit.> to a 1-class model, we re-initialized and fine-tuned the final class prediction layers on the VOC2007+2012 trainval set with all objects given the same label. Because our targeted UAS use case often involves detecting small objects from altitude on the ground, we also randomly shrank, with probability 0.5, the images and bounding boxes to 1/4 of the dimensions, and placed them on a blank canvas as part of the data augmentation strategy. 
We conduct further data augmentation by applying random crop where the minimum intersection over union (IoU) of the cropped image to the bounding box is 0.25, mirror with probability 0.5 and perturbations in hue, contrast, saturation, and brightness. 

Online Training: We then began each online training sequence with the all-class fine-tuned model above. During this process, the model is further fine-tuned using the Adam <cit.> optimizer, and learns instance-specific traits, including those outside of the original 20 classes. To imitate the shrinking image augmentation strategy described above during online training, we inserted into every batch one raw video frame, and one rescaled one. Since objects may be already very small in the raw frame, we rescaled the frames by factor S in a dynamic way as given by Equation <ref>. 


    S = 10max(h_b, w_b)/w_i

where w_b, h_b is the width and height of the initial bounding box, and w_i is the width of the (square) initial image. The image is then placed randomly on a canvas of dimensions (h_i, w_i) of the mean values of the VOC training set. We then perform the same data augmentation steps as in pre-training to a duplicate pair of the normal and shrunken images, bringing each online batch to a total size of four. 



 §.§ Bounding Box Supervision from Clicks





Conducting ToOT for object detection would require drawing bounding boxes for every frame in the video stream in near real-time, which is not realistic or safe for a human operating a UAS. In order to alleviate this difficulty, we adopt a customized interactive segmentation technique to automatically generate a high-quality class-agnostic object bounding box annotation from a single user click, described in Figure <ref>. Specifically, we modified the architecture from Xu <cit.> to only include positive clicks. In contrast to saliency detection, the segmenter produces a mask containing only the single object instance specified by the clicks, even if it is not the most salient within the image. With the binary mask output, we can derive a bounding box suitable for training detectors. We present the performance of the interactive segmentation and the weakly-supervised training of a detector in Section <ref>. 




During online training, we take three steps to reduce noise in the bounding box generated from the segmenter. First, the segmenter is provided input at full video resolution (720x720 pixels), rather than the input resolution of SSD300. Second, if the segmenter produces multiple regions, we select only the region within which the click landed, or the closest region to the click. The smallest bounding box that encompasses the region is created. If no region is produced by the segmenter, a small bounding box of 20x20 pixels is created around the click. This last case is a very rare occurrence. Finally, the segmenter has issues with long, thin objects, often cutting off part of the object. Therefore, if the produced bounding box's smallest side is less than one tenth of the image dimension, we increase both dimensions of the bounding box by 20%.



 §.§ Object Tracking for Training Assistance


To further reduce required user interactions, we exploited a property particular to ToOT sequences: the target object is unlikely to have moved far between frames. We can thus assist the user with object tracking, as long as the target item is in the scene. This tracked information can generate training events beyond what was produced by the user. Figure <ref> shows the method by which object tracking informs online training. When the user clicks on the target in the frame, the tracker is initialized on the target with a bounding box from segmentation. This initiates a training event. After training, a subsequent frame is retrieved from the camera, and the tracker is updated using this frame. If the tracking update succeeds, another training event is initiated with the center of the updated bounding box as the new click, and the cycle continues. If it fails, the user will have to click on the target again.



In our experiments, we use the Median Flow <cit.>, because of its robust tracking failure detection. 








§ EXPERIMENTS: WEAKLY SUPERVISED DETECTION


In this section, we validate our approach to generating weak supervision through clicks. In the following Section <ref>, we will evaluate the combined system on online training tasks.

We first compare our customized segmenter with Xu <cit.> on interactive segmentation. We fine-tune our FCN-8s segmenter on the PASCAL VOC 2012 <cit.> training dataset. The IoU accuracy of our model using 1 to 5 clicks per object on the validation set is 0.77, where <cit.> achieves 0.70 with a mean of 2.5 clicks. The result shows that our method achieves comparable performance without negative clicks.

We then use the segmenter as the only weak supervision in training a SSD300 model. To convert the fully supervised VOC dataset to a click-based one, we choose the center of each ground truth bounding box as the location of the click. For each object, we generate a click channel and input it into our segmenter (Figure <ref>). The output mask is used to derive a bounding box—first, by removing regions not located where the click is—which is then used to supervise our SSD300 model. Although we only have simulated bounding boxes from a weak supervision of clicks, our model obtained more than half of mean average precision (mAP) of a fully supervised model and is 6% higher than the state-of-the-art weakly supervised model that uses only class labels, as shown in Table <ref>. Figure <ref> shows some sample bounding boxes generated from this method. Note on the right side the most common failure mode: ambiguity of clicks. The center location of two of the sheep in the image both land on the same sheep, and cause the segmenter to produce two bounding boxes around the same object. We note that this is a limitation purely of how we generate clicks from the existing bounding boxes, and not an issue in human-driven online training; the user would not click on overlapped portions.













§ EXPERIMENTS: ONLINE TRAINING


We now validate our approach for accelerating online training. In our experiments, we want to (a) show that using weakly-supervised segmentation to produce bounding boxes do not break the training process, and (b) quantify the benefit of using optical-flow assisted training for time-ordered online training of an object detector. 



 §.§ Benchmark Datasets for ToOT


Existing datasets used in detection, such as PASCAL VOC <cit.> provide a rich corpus of annotated images of disparate scenes, but do not adequately represent a single training session in the field, where each image in the set directly follows the last. The KITTI suite <cit.> provides annotated video sequences, but they do not track any one particular training target, instead annotating classes of items ( cars). 

In order to validate our approach and compare online training methodologies systematically, we constructed two scenarios based on realistic use cases, each with train and test phases. Each phase consists of a time-ordered series of human-labeled images taken from the UAS in flight, once per second. In real-world terms, this is enough time for the UAS' embedded GPU platform to complete one training round. Each frame is annotated with three pieces of information: whether or not the target is in the frame, the coordinates of the center of the target, and the bounding box. Using these pieces of information in a frame-by-frame order, we can simulate, in a repeatable fashion, a user supervising the learning process.

We will refer to our scenarios as CarChaser (878 train and 317 test images) and PersonFinder (422 train and 156 test images), where the target in question is a car or a person, respectively. Images are 720x720 pixels in size. Figure <ref> shows examples from the dataset. The test scenarios are intentionally less controlled, and re-enact a sequence where a UAS is attempting to find a hidden or moving target. 




































 §.§ Training Strategies


We evaluated the following training strategies on our two ToOT scenarios. 



Online Training w/ Bounding Boxes: In this strategy, every positive example is used in training. In other words, u_i = 1, where i=1,...,n. Augmentation is performed as described in Section <ref>. In this case, we use manually-labeled bounding boxes as the ground truth for the SSD detector. This strategy forms a baseline for comparison of the two others. 

Online Training w/ Click-based Supervision: In this strategy, we performed training in the same manner as in the prior one, except the bounding boxes are generated purely from single-point clicks (Section <ref>) near the centroid of the target. In real-world terms, this means the user only has to click on the object during the training sequence, rather than draw a bounding box around it. 

Online Training with Object Tracking: In this strategy, we assisted our standardized user using Median Flow tracking, as described in Section <ref>. We made the following assumptions about how the user uses the tracking-assisted system:



  * When no tracker is initialized, and the target is in the scene, the user clicks on the target and initializes tracker. 

  * When tracker updates are successful, the user allows the system to perform training without interruption.

  * When the tracker loses the target ( the IoU of the tracked bounding box with the manually-labeled bounding box is below 0.5), the user again clicks on the target in the next frame, re-initializing the tracking. 


This means that on many occasions, v_i is added to the mini-batch even if the corresponding u_i = 0. This produces additional training benefit without additional user input. 




 §.§ Results


We examined two metrics—first, the maximum average precision reached by each click-supervised training strategy in comparison to using manually-labeled bounding boxes, and then the training benefit of training with or without object tracking. All tests used as a base a 1-class SSD model pre-trained on the PASCAL VOC2007+2012 training set. Adam with lr=0.0005, β_1=0.9, β_2=0.999, ϵ=1*10^-8 is used as the gradient descent algorithm. To mitigate the effects of stochasticity in gradient descent, all metrics were computed from a mean of 5 runs at each timestep. 



  §.§.§ Maximum Average Precision


We used average precision, as computed for PASCAL VOC 2007 <cit.>, of the single class as our performance metric P, as seen in Equations <ref>–<ref>.

In real-world usage, we are less concerned with finishing the entire training sequence as we are with achieving the best possible average precision, P_max, during training, where 


    P_max = max_i=0^n P_i


In other words, P_max is the highest achievable point when all frames to n are used. We began by taking P_max of using manually-labeled bounding boxes as ground truth, and used this as a baseline for evaluating the other measurements. We then compared this P_max to purely click-based supervision as well as click-based supervision accelerated with object tracking.





Figure <ref> shows this comparison for the CarChaser scenario, and Figure <ref> for the PersonFinder scenario. We see that using the click-based supervision does result in an expected but not unreasonable deviation in average precision. In PersonFinder, where the segmentation task is significantly more difficult (target is smaller, more complex), there is a noticeable degradation when using click-based supervision. On the other hand, using click-based supervision with and without object tracking produces comparable performance. This is promising as object tracking dramatically reduces the number of required clicks, as we will show. 











  §.§.§ Training Benefit


In Section <ref>, we defined CTB and ITB as metrics particular to ToOT that are a measure of the effectiveness of each user interaction. In particular, ITB_(i,j) can be used to compare training strategies, given all achieve a fixed final average precision P_f. We assigned P_f to be the minimum P_max of the various training strategies. Runs were terminated after P_f was reached. We plotted CTB_u_i and ITB_u_i over v_i, as well as computed ITB_(1,f), where f is the step when P_f is reached for that particular scenario. 

CarChaser: For CarChaser, the final average precision was P_f =  76.5%. Training with manually-labeled bounding boxes is also shown for reference. In Figure <ref>, we plot the CTB of both training strategies up to P_f. The top plot shows representative frames in the training scenario at that period of time. Each trace follows the accuracy at each training round i, regardless of if a user interaction happens at that step (u_i = 1). Each dot on the trace represents the rounds in which a user interaction did happen. 

Flat regions in the plots represent periods of no action,   places where the target is not in the scene and the frames are not being trained on. Noise in these flat regions are a result of non-deterministic GPU execution. Note that the lines on the “w/ object tracking” plot closely follow those of the first plot—but the number of dots (user interactions) is much less dense. This means that, as expected, object tracking generates user interactions that are not triggered by the user but still effective in training the model.

We also notice dips in CTB; these correspond to parts of the training sequence where the training actually worsens the accuracy. The largest trough corresponds to a part in the training sequence where the target car is very small relative to the frame. This suggests that certain properties of the frame may be predictors of their effectiveness in training the model—we leave this question for future study. 

Figure <ref> plots the incremental training benefit (ITB) for each user interaction in Figure <ref>. As expected, the tracking-enabled methods produce far less dense plots as they have far fewer user interactions (Table <ref>). We note that while some of the user interactions produce a very large ITB, the vast majority produce very small changes. To emphasize the smaller values, the y-axis is plotted on a logarithmic scale. With the object tracking-enabled methods, these low-impact training rounds can be triggered many at a time.



The ITB_(1,f) for each training technique is shown in Figure <ref>—the average of every point in Figure <ref>. We see that the object tracking-enabled strategy enables user interactions that are 4.50 times more effective ( produce on average 4.50 times more ITB per interaction) than without.





PersonFinder: We repeated the same experiments on the PersonFinder dataset. The PersonFinder test scenario contained far more clutter, and thus the initial model performed very poorly. The trained model rose from P_0 = 6.2% to P_f = 58.1%.

In Figure <ref>, enabling object tracking again reduces the density of clicks without reducing the upwards trend of performance. Interestingly, the CTB_i did not climb significantly until around i=150, suggesting those training examples were not useful in ruling out clutter from the test set. We see the same trend reflected in the ITB_u_i plot, with the user interactions that happened earlier having a smaller ITB_u_i than the ones that happen later. This implies that those early training examples were less “useful” than those that happen later, suggesting yet again that certain frames—in this case frames with very sparse backgrounds—are less valuable than others. 

Figure <ref> again plots ITB_(1,f) for the various training strategies. From the perspective of reducing user interactions, object tracking again dramatically increased ITB_(1,f), by 3.07 times. Table <ref> shows both the total user interactions required to reach P_f, as well as the mean ITB_u_i value for these interactions. 













§ CONCLUSION






In this work, we recognize that training CNN object detectors in real-time via human input encompasses a unique class of problem, in which we are not only concerned with the performance of the model produced, but also the human effort,  the number of user interactions, required to train the model. We use training benefit as a metric for measuring the impact of user interactions, and compare two training strategies in their effectiveness per user interaction. We show that we can obtain bounding box annotations from click-based supervision using interactive segmentation, eliminating the need to draw them manually during online training. Finally, by exploiting the time-ordered nature of video streams through object tracking, we can multiply mean incremental training benefit by about 3-4 times, depending on the scenario, over the one-frame-one-click approach.

We believe this work sets the stage for further investigation into real-time training on ToOT sequences. From Figures <ref> and <ref>, we can observe that not all training frames produce a strong positive benefit. Further work in informing frame selection could mitigate this problem. 




























ieee


