Adaptive ROI Generation for Video Object Segmentation Using Reinforcement Learning
Mingjie Sun Jimin Xiao Eng Gee Lim Yanchu Xie Jiashi Feng

1
.001

Mingjie Sun et al.

mode = title]Adaptive ROI Generation for Video Object Segmentation Using Reinforcement Learning                      

1]Mingjie Sun[orcid=0000-0002-3697-7927]

mingjie.sun18@xjtlu.edu.com

1]Jimin Xiao[]
[1]

jimin.xiao@xjtlu.edu.com

1]Eng Gee Lim

1]Yanchu Xie

2]Jiashi Feng



[1]Xi'an Jiaotong-Liverpool University, Suzhou, China
[2]National University of Singapore, Singapore


[cor1]Corresponding author


In this paper, we aim to tackle the task of semi-supervised video object segmentation across a sequence of frames where only the ground-truth segmentation of the first frame is provided. The challenges lie in how to online update the segmentation model initialized from the first frame adaptively and accurately, even in presence of multiple confusing instances or large object motion. The existing approaches rely on selecting the region of interest for model update, which however, is rough and inflexible, leading to performance degradation. To overcome this limitation, we propose a novel approach which utilizes reinforcement learning to select optimal adaptation areas for each frame, based on the historical segmentation information. The RL model learns to take optimal actions to adjust the region of interest inferred from the previous frame for online model updating. To speed up the model adaption, we further design a novel multi-branch tree based exploration method to fast select the best state action pairs. Our experiments show that our work improves the state-of-the-art of the mean region similarity on DAVIS 2016 dataset to 87.1%.



model adaptation video object segmentation reinforcement learning training accelearate



[
    [
     
=====





§ INTRODUCTION




Video object segmentation (VOS) <cit.> is a fundamental problem in the computer vision field with many applications including video editing  <cit.>, video surveillance <cit.>, and scene understanding <cit.>. The objective of single target video object segmentation is to label each pixel as foreground or background in a given frame of a video sequence. Labeling these pixels, however, is difficult due to background clutter, illumination change, motion blur, deformation, and so on.

There are many kinds of segmentation methods including semi-supervised video object segmentation <cit.>, unsupervised video object segmentation <cit.>, interactive video segmentation <cit.> and so on. 
For semi-supervised segmentation, the ground-truth annotation for each pixel of the first frame is provided. The segmentation model identifies pixels of foreground and background for the following frames. 

Most recent approaches addressing semi-supervised vid-
eo object segmentation are based on convolutional neural networks. In particular, the one-shot video object segmentation (OSVOS), introduced by Cacells et al.  <cit.> has achieved great success. The core idea is to fine-tune an ImageNet <cit.> pre-trained ConvNet on the video object segmentation data-set, such as DAVIS  <cit.>, to allow the segmentation model to find general objects in a frame at first. Then, the first frame of the inference video sequence will be used to fine-tune the segmentation model such that it can rapidly focus on the specified target object instance in the first frame. Despite the promising result obtained using OSVOS, such a method is challenged by the cases where the appearance of the target object changes dramatically in the video sequence and there may exist multiple confusing instances of similar appearance. OSVOS only learns the target object appearance from the first frame of a video sequence. It cannot adapt to the target object appearance variation when deformation occurs or the camera rotates. 

Inspired by the fact that the online adaption has achieved great progress on video object tracking at bounding box level
 <cit.>, some works start to deploy the online adaption to video object segmentation. For instance, online adaptation video object segmentation (OnAVOS) <cit.> proposes an online adaptive video object segmentation meth-
od which enables to update the segmentation model during inference time. However, model drift may interrupt the model online adaptation and lead to performance drop.

We observe that, for online VOS model update, how to identify to the region of interest (ROI) for model adaptation is critical. In particular, as can be observed in Figure <ref>, if we select the adaptation area in a rough way, the segmentation model after adaptation performs very poor when there are multiple similar objects in the frame, especially, the distraction object is very close to the target. As such selecting the optimal adaptation area in a more sophisticated and flexible way is very significant for video object segmentation. 

To tackle this problem, we formalize VOS as a conditional decision-making process where two reinforcement lea-
rning (RL) agents are employed to infer and adjust the ROI for adaption for both foreground and background in a flexible way. At each frame, the RL agent outputs actions to tune the ROI. Provided with such regions, the VOS model can be updated to be more specific and discriminating for the instance in the current frame. 

To select the optimal adaptation area for video object segmentation, a set of features of different adaptation areas of the current frames will be fed into the RL model. Then, the RL model will select the best action, and to choose the most suitable adaptation area for the current frame.  As a result, the segmentation model can obtain an accurate segmentation result. Though RL method is promising for region identification, it is notoriously slow to optimize the agent. In this work, to speed up the RL agent training, we propose a multi-branch tree based policy search method where possible action state pairs are organized in a tree structure. 

To sum up, this paper has three main contributions:

	
  * We observe the importance of identifying the ROI for VOS model adaptation and make the attempt to mine optimal adaptation areas for online adaptation in video object segmentation. Specifically, we deploy the “actor-critic” RL framework to train the agent for generating the adaptation areas. 
	
	
  * Both VOS with online adaptation and the RL model training are computational demanding processes. To speed up the RL training, we further design a novel
	multi-branch tree based exploration method to fast select
	the best state action pairs.
	
	
  * The proposed approach has been validated on DAVIS 2016, SegTrack V2 and Youtube-Object dataset. New state-of-the-art result of mean region similarity is obtained for the DAVIS 2016 dataset, which is 87.1%.
	
	




§ RELATED WORK





 §.§ Video Object Segmentation


Recently, with the popularity of deep neural network, various deep learning based video object segmentation models have been proposed. These existing approaches can be classified into three different categories, including unsupervised methods, weakly supervised methods and semi-super-
vised methods. Unsupervised methods and weakly supervised methods are more difficult than semi-supervised methods because no pixel-level annotations are available. Utilizing motion information and detecting the primary object in a frame are two common ways to address
this problem. In <cit.>, Pavel et al. combine the object appearance information and the motion information together successfully and achieve a good performance. In <cit.>, Song et al. propose to use concatenated pyramid dilated convolution features and dramatically improve the final accuracy. 

Semi-supervised video object segmentation task, where the pixel-level annotation of the first video frame is available, is also an extensively studied task. The most common approach for this task is to pre-train a general segmentation network. Then, the network is fine-tuned using the first frame annotation to enable the network focusing on the particular object in the frame <cit.>. In order to adapt to object appearance variation, a novel approach to update the segmentation network during test time is proposed in <cit.>. In  <cit.>, another method is proposed where proposals will be generated first, and then they will be merged into accurate and temporally consistent pixel-wise object tracks. In <cit.>, this task is viewed as a spatio-temporal Markov random filed problem, and ConvNet is utilized to encode the dependencies among pixels. To overcome the shortage of training data, it is proposed to use static images to generate additional training samples in <cit.>. In <cit.>, Everingham et al. attempt to utilize the part-based tracking method to generate bounding box and segmentation for each part. In <cit.>, Xiao et al. apply meta leaning to video segmentation and dramatically speeds up the segmentation process.   

Different from the existing works, we formulate the selection of ROI for online segmentation adaptation as a Markov decision process and utilize the RL to address this problem. 



 §.§ Deep Reinforcement Learning

RL algorithm learns to achieve a complex objective from past experience. “actor-critic” <cit.> is a popular RL framework that inherits several previous RL frameworks including deep Q-learning <cit.> and policy gradient <cit.> which are valued-based and policy-based strategies, respectively. 

RL has been applied to many areas of computer vision, in particular for visual object tracking at bounding box level. In <cit.>, Yun et al. use RL to choose sequential actions to move the bounding box step by step from the original object location in the previous frame to correct location. In <cit.>, Huang et al. propose a novel RL based model which can choose the most suitable number of deep convolutional layers according to the current frame complexity, which dramatically reduces the running time. In <cit.>, Dong et al. propose to deploy the RL to choose the optimal hyper-parameter for correlation filter. 

In pixel-level video object segmentation, to the best of our knowledge, however, there is only one attempt to apply RL to this task. Han et al. establish a novel RL framework which can choose the optimal object box and the context box <cit.>. This work is motivated by the observation that, for an identical segmentation model, different object boxes and context boxes generate different segmentation masks. Thus, it is natural to utilize RL to choose an optimal object-context box pair to achieve the best segmentation result.

Different from the existing work <cit.>, our work aims to utilize the RL to choose the optimal ROI to update the segmentation network during test time.







§ OUR APPROACH







 §.§ Overview


In general, the main objective of our work is to utilize RL to improve the performance of video object segmentation with online adaptation. Different from the existing  approaches to select the adaptation ROI in a rough way, our work utilizes RL to choose the optimal adaptation ROI for each individual frame to avoid model drift. In other words, different frames will own its particular standard to choose adaptation ROI. The overview diagram of our proposed met-
hod is described in Figure <ref>. 



  §.§.§ VOS with Online Adaptation


Our work is built on the top of VOS model with online adaptation. Before finally segmenting the current frame F_t  of a video sequence, a part of pixels in F_t will be used to update the current segmentation model to adapt to the change of the foreground and background. These pixels consist of positive ROI regarded as foreground, denoted as S_p, and the negative ROI regarded as background, denoted as S_n. In OnAVOS <cit.>, S_p and S_n are chosen according to two fixed thresholds including T_p and T_n. In order to determine S_p, F_t will be fed into the segmentation model and a temporary (before online adaptation) probability map M_f, whose size is the same as the F_t, will be generated. In M_f, for a pixel i, M_f(i) refers to the probability that pixel i belongs to the foreground of F_t. Then, S_p will be chosen according to T_p, as follows:

    S_p =
    {i|i∈ F_t, M_f(i)>T_p}.

Negative ROI S_n is decided by the distance to the positive ROI S_p. S_n are the pixels far away from S_p, as follows:

    S_n =
    {i|i∈ F_t, distance(i)>T_n},

where distance(i) refers to the distance of pixel i to S_p.



  §.§.§ RL Based Online Adaptation


Existing VOS methods with online adaptation, i.e., OnAVOS <cit.>, adopt a fixed standard to select the adaptation area, including T_p and T_n, where different characteristics of each frame are not considered. To address this problem, we use flexible thresholds, t_p and t_n, to choose the adaptation ROI. We build two RL models to choose the most suitable t_p and t_n for each frame. 

Our RL framework includes state s∈ S, threshold selection action a^p∈ A^p to determine the value of t_p and threshold selection action a^n∈ A^n to determine the value of t_n, state transition function s'=T (s,a^p,a^n) and the reward function g(s,a^p,a^n). Given a frame F_t of one sequence, first of all, F_t will be fed into the segmentation network to obtain a temporary probability map M_f. A set of ROIs can be obtained according to the candidate thresholds. In our work, we need 5 ROIs where the probability value is greater than α_large, α_small, β_large, β_medium and β_small, respectively. Note that β_micro is ignored to diminish the complexity of the state. The first two ROIs are used for the RL model to choose t_p while the last three ROIs are used for RL model to choose t_n. In other words, the possible values of t_p can be α_large or α_small, and the possible values of t_n can be β_large, β_medium, β_small or β_micro. After t_p and t_n are determined, pixels with probability values greater than t_p will be viewed as the adaptation ROI for foreground, as follows:

    S_p =
    {i|i∈ F_t, M_f(i)>t_p},

while these pixels with probability value less than t_n, combined with the negative pixels chosen by distance using (<ref>), will be regarded as the adaptation ROI for background, as follows:

    S_n = {i|i∈ F_t, distance(i)>T_n}
    ∪{i|i∈ F_t, M_f(i)<t_n}.

S_p and S_n are used to update the current segmentation model while other pixels are ignored. After the segmentation model has been updated, F_t will be fed into the new segmentation network and the final segmentation result is obtained. 

The pseudo-code of our algorithm is described in Algorithm <ref>.







 §.§ Agent Action


The framework of our work consists of two RL models, including one to choose t_p and another to choose t_n, as shown in Figure <ref>. The action set A^p used for the first RL model to choose S_p contains 2 candidate thresholds: α_large and α_small. As the difference of the ROI areas with different
t_p is not very big, we only set two candidate thresholds
for the action set A^p. The action set A^n used for the second RL model to choose S_n contains 4 candidate thresholds: β_large, β_medium, β_small and β_micro. As the size of S_n is much larger than that of S_p, we set more candidate thresholds for t_n.

In terms of the design of the candidate values of t_p, in OnAVOS <cit.>, T_p is 0.97 and it is fixed for all frames. This threshold is very safe and conservative because it should work in any situation, especially for some frames with very bad segmentation result. In fact, for some frames with good segmentation result, the value of t_p ought to be much lower, so more correct pixels of the object can be used to update the model and improve the final segmentation result. As the difference of the size of the adaptation ROIs with different t_p is not very huge, we only set two candidate thresholds for the action set A^p, which can lower the difficulty of training the RL model. 

In terms of the design of the candidate values of t_n, in OnAVOS <cit.>, S_n are chosen according to the distance to the target object rather than the value of the M_f. If a pixel is far away from the target, it will be viewed as S_n . Similarly, the selection of S_n in OnAVOS <cit.> is very conservative as it works for almost all frames. In our work, we still keep the adaptation ROI which are chosen by the distance, and then try to add more areas to include more correct adaptation ROI for background. The additional area is chosen by the value of M_f. Finally, we combine these two parts of areas together as the final area of S_n using (<ref>).
As the size of the area for S_n is much larger than the area of S_p, we set more candidate thresholds for t_n. 

In this way, after training, these two RL models can cho-
ose the best action and achieve the optimal thresholds t_p and t_n.






 §.§ State and Reward




The state s is the input of the RL model. As we have two RL models, we need two sets of states for two models. In general, the state s is a feature map combined by the feature maps of different candidate adaptation ROIs where the probability value of all pixels in a certain adaptation ROI is less than a certain threshold.

First, given a certain frame F_t, we feed F_t into the segmentation model before test time adaptation and generate the temporary probability map M_f. Then, we generate 5 ROIs with 5 different candidate thresholds according to the value of M_f. The ROIs where the probability values of all pixels in this area are greater than α_large and α_small, receptively, are combined as the state of the RL model to choose t_p, as follows,

    state_p = feature({i|i∈ F_t, M_f(i)>α_large}) 
    + feature({i|i∈ F_t, M_f(i)>α_small}).

The ROIs where the probability values of all pixels in this area are greater than β_large,  β_medium and  β_small, receptively, are combined as the state of the RL model to choose t_n, as follows, 

    state_n = feature({i|i∈ F_t, M_f(i)>β_large}) 
    + feature({i|i∈ F_t, M_f(i)>β_medium}) 
    + feature({i|i∈ F_t, M_f(i)>β_small}).


Note that β_micro is ignored in (<ref>) to diminish the complexity of state_n, as feature({i|i∈ F_t, M_f(i)>β_small}) already provides sufficient information to help the RL model make the decision. We use the VGG model <cit.>, pre-trained on the ImageNet classification dataset <cit.>, to extract the features of these ROIs first. Then, we concatenate these features together as the state of the RL model. We use the first 5 convolutional blocks of the VGG19 model which results in a feature size of  ℝ^7×7×512 for one ROI. For the RL model to choose t_p, the features of two ROIs will be concatenated to generate the final state s_t_p∈ℝ^7×7×1024. For the RL model to choose t_n, the features of three ROIs will be concatenated to generate the final state s_t_n∈ℝ^7×7×1536.
Finally, states s_t_p and s_t_n will be fed into the corresponding RL model and result in the actions to choose the optimal thresholds t_p and t_n.

The reward function is defined as r_t=g(s_t,a_p,a_n) which reflects the performance of the final segmentation result of each frame in the video sequence:


    g(s_t,a_t,a_n)={
    IOU+1     IOU>0.1 
     
    -1     IOU<=0.1 
    .
    ,

where IOU indicates the intersection-over-union (IOU) between the prediction and the ground-truth, which reflect the quality of the predicted segmentation. 



 §.§ Training in Actor-Critic Framework


In our work, we adopt the “actor-critic” framework <cit.> for RL training. In general, one “actor-critic” framework consists of two roles including an “actor” role to generate an action and a “critic” role to measure how good this action is. In this work, we need to select the optimal adaptation ROIs for both foreground and background separately. Therefore, we need two “actor-critic” model pairs, including  one “actor-critic” pair for foreground, and another pair for background. Four individual RL models are deployed in total. 

In our “actor-critic” framework, given a current frame F_t, the first step is to feed the state into the “actor” network and generate an action a, which is to choose the optimal adaptation ROIs. The corresponding reward r_t will also be obtained after conducting this action. r_t is decided by the IOU of the segmentation result according to (<ref>). 

In the training process, after the forward process, the “critic” network will be updated first in the valued-based way, as follows:

    w = w' + α*δ_t∇_w' V_w'(s_t),

where

    δ_t =  r_t+ γ*V_w'(s_t+1)-V_w'(s_t).

In (<ref>) and (<ref>), w and w' indicate the weight of the “critic” model after and before update.  α is the learning rate of the “critic” model. δ_t is the TD error which indicates the difference of the actual score and the predicted score. V_w'(s_t) refers to the accumulated reward of state s_t which is predicted by the “critic” model before update. γ refers to the discount factor.

After the “critic” model has been updated, the “actor” model will be updated in a policy-based way, as follows:

    θ  = θ' + β * ∇ (logπ_θ' (s_t,a_t)) * A(s_t,a_t),

where A(s,a) refers to the advantage function, and A(s_t,a_t)=δ_t according to (<ref>), θ and θ' indicate the weight of the “actor” model after and before update. β is the learning rate of the “actor” model. Policy function π(s,a) is a network whose input is the state s and a certain action a, and output is the probability of selection action a in state s. 

In this way, when training the RL models, our “actor-critic” framework can avoid the shortage of value-based and policy-based methods. Instead of waiting until the end of the episode, our RL models can be updated at each step, which dramatically reduces the training time but maintains the RL training stability. 



§ IMPLEMENTATION DETAILS





 §.§ Train the Segmentation Network


The proposed method trains the segmentation network follows the strategy of <cit.> and <cit.>. The first step is to train a ImageNet pre-trained network on a pixel-level annotated dataset such as PASCAL VOC <cit.>. In the second step, we use the DAVIS video dataset <cit.> to train the network so that the network is able to adapt to this dataset. We also fine-tune the network on the first frame of DAVIS test videos whose ground-truth annotation is provided. In this way, the segmentation network is well trained and can achieve a good result before test time adaptation.



 §.§ Train the RL Model

Before training the RL model, the related data need to be stored in advance to accelerate the training process, which will be described in section 4.3. In terms of the training of the RL models, specifically, we divide all video sequences in DAVIS 2016 training set into video clips with the fixed number of frames. A video clip includes 10/5 consecutive frames is used as a sample for the RL model to select foreground/background ROI. Using stored clips for training dramatically reduces the training time of the RL model.

We randomly select 20 clips as a batch for training the RL models. At the beginning of the training, the learning rate α for “actor” model is 1e-5, and the leaning rate β for “critic” model is 5e-5. The learning rate decreases gradually during the training, and it decreases by 1% for each 200 iterations. The discount rate γ for the reward is 0.9. In terms of values for candidate threshold t_p and t_n, we found α_large=0.97, α_small=0.7, β_large=0.4, β_medium=0.2, β_small=0.1 and β_micro=
0.01 works well through cross validation. The training of our RL models takes about 3 days on a NVIDIA GTX 1080 Ti GPU and a 12 Core Intel i7-8700K CPU@3.7GHz. 



 §.§ Accelerating RL Training

Segmentation with online adaptation is slow because the segmentation model should be updated for each frame. Meanwhile, RL training itself is also slow. Training the RL model heavily relies on a large number of attempts for different actions. Normally, to generate a well trained RL model, the model should be trained with more than one million iterations. If the running time for each training process is too long, the total time is unbearable. Thus, it is impossible to train the RL model in a regular way.

To address this issue, inspired by the idea of sacrificing space to improve efficiency, we propose a novel multi-branch tree structure, as shown in Figure <ref>, to store all possible segmentation results using different adaptation ROIs into a repository in advance. Training an “actor-critic” framework needs 4 types of information including the action, the reward, the states before and after the action. For each node in the multi-branch tree, a corresponding directory is generated. The image of the frame, the temporary probability map and the IOU of the segmentation result after executing a certain action are stored as files. All possible actions are stored as a links to next layer of nodes. In this way, the image and the probability map are used to generate the state. The IOU value is used to generate the reward. Finally, this repository will be organized in a multi-branch tree structure, whose stored data will be used to update the RL model during the process of training using the method described in section 3.4.
















§ EXPERIMENTS






 §.§ Experiment Setup


Our method is evaluated on three widely-used datasets including the DAVIS 2016 dataset <cit.>, Youtube-Object data-set <cit.> and Segtrack V2 dataset <cit.>. DAVIS 2016 dataset consists of 50 high quality video sequences and 3,455 frames, spanning multiple occurrences of common video object segmentation challenges such as occlusions, motion-blur and appearance changes. 30 video sequences of DAVIS 2016 are used for training, and 20 video sequences are used for testing. In DAVIS 2016, in each video sequence, only a single object instance is annotated. DAVIS 2017 dataset <cit.> extends the DAVIS 2016 dataset where multiple objects, rather than only one object, are annotated in each frame. As our method targets for single instance segmentation, we only do the experiment on DAVIS 2016. In Youtube-Object, there are 155 video sequences and a total of 570,000 frames. These video sequences are divided into 10 classes. Training set and testing set are not separated in Youtube-Object dataset so it is only used for testing. In SegTrack V2 dataset, there are 14 video sequences with more occlusion than appearance changes compared with Youtube-Object dataset. 

We evaluate our method following the approach proposed in <cit.>. The adopted evaluation metrics include region similarity J and contour accuracy F. The region similarity is calculated as J= | m∩ gt/m∪ gt | by the intersection-over-union between the predicted segmentation m and the ground-truth gt. The contour accuracy is defined as F=2P_cR_c/P_c+R_c, which indicates the trade-off between counter-based precision P_c and recall R_c.





 §.§ Comparison with State-of-the-arts

In this section, we will compare our proposed work with other state-of-the-art semi-supervised video object segmentation methods, including  PReMVOS <cit.>, OnAVOS <cit.>, CINM <cit.>, LucidTracker <cit.>, MSK <cit.>, OSVOS <cit.>, STV <cit.>, ObjectFlow <cit.>. Note that OSVOS-S <cit.> is not included in the comparison list as it utilizes additional dataset for training. 

Table <ref> summarizes the quantitative results of recent methods on the DAVIS 2016 validation set consisting of 20 videos. The top 3 performing methods have been highlighted with different colors. It can be observed that our work has achieved outstanding result under both mean region similarity J_m and the mean contour accuracy F_m. Especially on mean region similarity J_m, our method achieves the best result which outperforms any existing state-of-the-art methods. Compared with the most competitive and related method OnAVOS <cit.>, our method improves the mean region similarity J_m to 87.1%. It should be noted that the gain over  <cit.> is solely due to the fact that better ROIs are obtained for online adaptation using RL. Note that, same to <cit.>, according to the randomness of the segmentation network, the final accuracy may fluctuate around 0.4%. The proposed mean region similarity J_m is the average value from experiments of 10 times. 

Figure <ref> shows the qualitative segmentation masks for different methods. As can be observed, our method performs better on videos with significant appearance change for the target object, for instance, the camel and breakdance video sequences. Especially when multiple similar objects are close to each other, e.g., the camel sequence, our method has the ability to distinguish the target object from other similar objects successfully. 

On SegTrack V2 and Youtube-Object datasets, as training set and evaluation set are not split, all video sequences are used for evaluation. From Table <ref>, we can observe that our method also performs well on both datasets. Compared with OnAVOS <cit.>, our method improve the mean region similarity J_m by 10.8% on SegTrack V2 dataset, which demonstrates the effectiveness of our RL models to choose the online adaptation ROIs. In addtion, our approach also improves the mean region similarity J_m by 2.1% on Youtube-Object dataset. This result can show the robustness of our method on different evaluation datasets.

In addition, we compare our run time with other state-of-the-art methods, and the result is also reported in Table <ref>. Despite the fact that we improve the mean region similarity J_m of our baseline method with online adaptation <cit.> by 10.8% on SegTrack V2 dataset and 1.4% on Davis 2016 dataset, the run time of our method is only about 1 second longer than <cit.>, which demonstrates the efficiency of our method. 





 §.§ Ablation studies


In this section, we conduct four ablation studies on our method using the testing video sequences of DAVIS 2016 dataset and SegTrack V2 dataset. 

We conduct the first ablation studies on the DAVIS 2016 and SegTrack V2 datasets, where parts of our method are disabled to investigate the impact of each component. In this study, we also explore the contribution of each individual RL model in our method, one of these two RL models is disabled during this ablation study respectively, and the results will be compared with the result generated by the method with full RL models. Table <ref> shows the result of this ablation study on the DAVIS 2016 dataset. On the DAVIS 2016 dataset, when using both foreground and background adaptation, we obtain the mean region similarity J_m of 86.5% without CRF <cit.>. J_m of our method with full adaptation is greater than the method without any adaptation by 6.2%, which demonstrates the effectiveness of our online adaptation approach. In addition, compared with the rough approach to choose the adaptation ROI adopted by OnAVOS <cit.>, before CRF, our method is also 1.9% greater than it. After executing CRF, the gain degrades a little to 1.4%, which demonstrates that our flexible way to choose the optimal adaptation ROI for each frame is significant for the final segmentation result. To study the individual influence of each RL model, we remove the RL model to choose the optimal adaptation ROI for background, obtaining the method foreground adaptation, and remove the model to choose the adaptation ROI for foreground, obtaining the method background adaptation. As can be observed from the Table <ref>, using foreground adaptation method and background adaptation method obtains J_m of 82.1% and 85.3%  on DAVIS 2016 dataset, respectively, which indicates that both RL models improve the segmentation result while the RL to choose the optimal adaptation ROI for background makes larger contribution to the final segmentation result. This observation also explains the reason why we have more threshold candidates for background than foreground. We also conduct the same ablation studies on the Segtrack V2 dataset and obtain the similar results as can be observed from the Table <ref>. When using both foreground and background adaptation, we obtain J_m of 61.4% without CRF. J_m of our method with full adaptation is greater than the method without any adaptation by 15.2%. The comparison between the rough approach to choose the adaptation ROI adopted by OnAVOS <cit.> and the proposed method is conducted on Segtrack V2 dataset as well, before CRF. The result is that J_m of the proposed method is 11.7% greater than the baseline method, which is a dramatic improvement of the mean region similarity. After the process of CRF, the gain degrades a little to 10.8%, which is still a huge improvement. To study the individual influence of each RL model on Segtrack V2 dataset, we also adopt the method foreground adaptation and the method background adaptation, as on the DAVIS 2016 dataset. As can be observed from the Table <ref>, using foreground adaptation method and background adaptation method obtains 66.2% and 73.2% J_m on Segtrack V2 dataset, respectively, which also indicates that both of the two RL models contribute to the improvement of segmentation result. This experiment also demonstrates the importance of the optimal adaptation ROI mining. 



The purpose of the second ablation experiment is to dem-
onstrate that it is important to select different t_n and t_p for each specific frame, rather than adopting a particular fixed set of thresholds. In other words, no matter which set of t_n and t_p are chosen, as long as these thresholds are fixed for all frames, the final segmentation result will be worse than the result generated by adopting the optimal thresholds selected by the RL models for each specific frame. More specifically, the RL model to select the adaptation ROI for background has 4 candidate thresholds including {0.4, 0.2 ,0.1, 0.01}. The RL model to select the adaptation ROI for foreground has 2 candidate thresholds including {0.97, 0.7}. In this way, there are totally 8 possible combinations of t_p and t_n, which are listed in Table <ref>. In this experiment, each combination of t_n and t_p is adopted as the fixed thresholds for the segmentation model adaptation and its corresponding result is evaluated and compared with the result generated by the proposed method. As can be observed from Table <ref>, among these combinations, the most “conservative” one, i.e. t_p=0.97 and t_n=0.01 performs best. Note that these values are different from the final result of OnAVOS<cit.> because OnAVOS selects the background adaptation ROI according to the distance, rather than the value of the probability map. The performance decreases dramatically when the value of the fixed threshold gets close to 0.5 because the error of segmentation will propagate quickly when adopting these "greedy" thresholds for all frames. The highest obtained J_m (83.6 %) of the method adopting fixed thresholds, however, is still much lower than the result generated the proposed method (87.1 %). According to the observation of this experiment, it is obvious that the improvement does result from the RL models that choose the optimal t_n and t_p for each individual frame, rather than the contribution of a certain fixed combination of t_n and t_p.

The purpose of the third ablation experiment is to demonstrate that the proposed method is able to find a better adaptation ROI, compared with the baseline method. The metric to evaluate the selected adaptation ROI is the IOU between the adaptation ROI and the ground-truth. In this way, we firstly calculate the IOU between the adaptation ROI selected by the RL model and ground-truth. Then, we calculate the IOU between the adaptation ROI selected by the baseline method and the ground-truth. Finally, we compare these two IOUs and their contributions to the final final segmentation result. As can be observed from Table <ref>, on average, the IOU of the proposed method is 40.9 % and 19.4 % higher than the baseline method, for background and foreground respectively. This result also coincides with the facts reported in Table <ref> and Table <ref> that background ROI adaptation using RL  brings more performance gain than foreground ROI adaptation. Finally, as more correct pixels are adopted to update the segmentation model in the proposed method, the segmentation model is able to properly adjust itself to the change of the target in the video. It is also the reason why the proposed method can achieve improvement on the final segmentation result. 



The fourth ablation experiment is to study the influence of different states for the RL model. The adopted state for the RL model also greatly affects the convergence difficulty of the training process as well as the final segmentation performance. We design three different states to study this factor. The first state is to feed the initial image (3 channels) and the temporary probability map (1 channel) into a train-from-scratch ConvNet without using pre-trained VGG model. This is because the input of the VGG model should be a 3-channel image. The second one is to feed the initial image (3 channels) concatenated with several mask channels to indicate different adaptation area, similarly, without the feature extracting of VGG model. The third one is the proposed one where we generate a set of images with different masks to indicate different adaptation areas. Feature of each image is extracted by the VGG model. Then, the combined feature will be fed into the RL model. When using the first two types of states, after full training, the RL model still cannot choose the optimal adaptation area for each frame, which demonstrates these states are not suitable for this task. We believe the main reason is that the pre-trained VGG model is able to extract a better feature which contains more information of the original image. Also, the first two ways to indicate the different adaptation areas are not sufficiently explicit and discriminative.



§ CONCLUSION


In this paper, we have proposed a novel method that can select the best adaptation ROI for each frame to take full advantage of the test time adaptation for video object segmentation. Two RL models are applied to choose the optimal adaptation area for foreground and background individually. Comprehensive experiments on three common benchmark datasets demonstrate the great performance of our method compared with other state-of-the-art methods. In future, we plan to replace the discrete-value actions with the continue-value actions to enable the RL model to find a more flexible way to choose the optimal adaptation ROI. 



cas-model2-names




