DMM-Net: Differentiable Mask-Matching Network for Video Object Segmentation
Xiaohui Zeng Renjie Liao Li Gu Yuwen Xiong Sanja Fidler Raquel Urtasun







DMM-Net: Differentiable Mask-Matching Network 
 for Video Object Segmentation
    Xiaohui Zeng1, 2Equal contribution.     Renjie Liao1, 2, 3*     Li Gu1     Yuwen Xiong1, 2, 3 
     Sanja Fidler1, 2, 4     Raquel Urtasun1, 2, 3, 5

University of Toronto1   Vector Institute2   Uber ATG Toronto3 
 
NVIDIA4   Canadian Institute for Advanced Research5

{xiaohui, rjliao, yuwen, fidler}@cs.toronto.edu   li.gu@mail.utoronto.ca   urtasun@uber.com

    Received: date / Accepted: date
=============================================================================================================================================================================================================================================================================================================================================================================







In this paper, we propose the differentiable mask-matching network (DMM-Net) for solving the video object segmentation problem where the initial object masks are provided.
Relying on the Mask R-CNN backbone, we extract mask proposals per frame and formulate the matching between object templates and proposals at one time step as a linear assignment problem where the 

cost matrix is predicted by a CNN.
We propose a differentiable matching layer by unrolling a projected gradient descent algorithm in which the projection exploits the Dykstra's algorithm.
We prove that under mild conditions, the matching is guaranteed to converge to the optimum.
In practice, it performs similarly to the Hungarian algorithm during inference. 
Meanwhile, we can back-propagate through it to learn the cost matrix.
After matching, a refinement head is leveraged to improve the quality of the matched mask. 
Our DMM-Net achieves competitive results on the largest video object segmentation dataset YouTube-VOS.  
On DAVIS 2017, DMM-Net achieves the best performance without online learning on the first frames.
Without any fine-tuning, DMM-Net performs comparably to state-of-the-art methods on SegTrack v2 dataset.
At last, our matching layer is very simple to implement;
we attach the PyTorch code (<50 lines) in the supplementary material. Our code is released at <https://github.com/ZENGXH/DMM_Net>.








§ INTRODUCTION


Video object and instance segmentation problems have received significant attention <cit.> attributed to the recent availability of high-quality datasets, , YouTube-VOS <cit.>, DAVIS <cit.>.
Given an input video, the aim is to separate the objects or instances from the background at the pixel-level.
This is a fundamental computer vision task due to its wide range of applications including autonomous driving, video surveillance, and video editing.



Two main setups of this problem are unsupervised and semi-supervised which differ from each other in whether the ground-truth annotated masks of the object  instances in the first frame of the video are provided <cit.> during inference.
In this paper, we focus on the semi-supervised setting, , the instance masks are provided for the first frames of the test videos. 
However, even with some annotated information at test time, this task is still very challenging. 
For example, the algorithm needs to deal with not only the dramatic appearance changes, occlusion and deformation but also potentially with large camera and object motion.
Furthermore, the expectation from a good video instance segmentation model is to produce temporally cohesive and stable predictions, which presents an additional challenge.


Existing algorithms typically leverage pretrained deep neural networks to predict object instance masks.
Some of them, , <cit.>, directly predict the masks in a frame-independent way and achieve surprisingly good performance which verifies the great transfer ability of deep neural networks.
Many algorithms leverage the previously predicted masks in various ways thus enabling the mask propagation over time.
This strategy is demonstrated to improve temporal coherence and segmentation quality.
Moreover, template matching between the reference and current frames is often exploited at pixel or mask level to deal with the object disappearance-reappearance phenomenon, occlusion, and fast motion.
However, to the best of our knowledge, none of the existing work integrates the optimal matching algorithm into their framework, which may partly be due to the non-differentiable nature of the problem.

In this paper, we propose the differentiable mask-matching network ().
We first extract mask proposals via a pretrained Mask R-CNN <cit.> in a frame-independent manner.
For each time step, we then match proposals with the templates in the reference frame such that at most one mask proposal is assigned to one template instance.
The matching cost between a pair of the template and proposal masks is determined based on the intersection-over-union (IoU) of masks and the cosine similarity between their feature maps predicted by a deep convolutional neural network (CNN). 
The key contribution of our paper is that we introduce a differentiable matching layer which solves the linear assignment problem.

Specifically, we unroll the projected gradient descent algorithm in which the challenging projection step is achieved by an efficient cyclic projection method known as Dykstra's algorithm. 
The proposed double-loop matching algorithm is very simple to implement, guaranteed to converge, and achieves similar performance to the optimal matching obtained by the Hungarian algorithm <cit.>.
More importantly, it is fully differentiable which enables learning of the matching cost, thus having a better chance of handling large deformation and appearance changes. 
After matching, we adopt a refine head to refine the matched mask.
Note that our main contribution is somewhat orthogonal to many existing work in a way that our differentiable matching can be integrated with other networks to potentially boost their performance.

On DAVIS 2017 <cit.>, SegTrack v2 <cit.> and YouTube-VOS <cit.> datasets, our model achieves either state-of-the-art or comparable performance.






§ RELATED WORK


The problem of video object/instance segmentation has been extensively studied in the past <cit.>.
Many algorithms in this field rely on techniques like template matching which are popular in the object tracking and image matching literature <cit.>.
However, video object/instance segmentation is more challenging than tracking as it requires pixel-level object/instance mask as output rather than the bounding box.
Meanwhile, it is also very different from matching in that it requires semantic understanding of the image rather than the similarities of low-level cues like color, motion and so on.
Related literature can be classified based on the problem setup, , unsupervised vs. semi-supervised.
Methods under the unsupervised category <cit.> typically exploit the dense optical flow and appearance feature to group the pixels within the spatio-temporal neighborhood.











In this paper, we focus on the semi-supervised setting. 

Based on whether an explicit matching between template and proposal mask is performed and at which level the matching is performed, we can further divide the related work into three sub-categories.




  
Pixel-level Matching
Pixel-level matching network (PLM) <cit.> first exploits a Siamese type of CNN to extract features of the current frame and the masked reference frame. 
Based on the features, it computes the pixel-level similarity scores and the instance masks.
VideoMatch <cit.> applies a CNN to extract features from the reference and the current frames, respectively.
The feature of the reference frame is further split into the foreground and background ones which are used to compute the similarity with the feature of the current frame via a specially designed soft-matching module.
The similarity-weighted combination of feature is used to predict the final mask.
A fully convolutional Siamese network based approach (SiamMask) is proposed in <cit.>.
It computes the depth-wise cross correlation between features of templates in the reference and the current frames.
It also consists of mask, box, and score prediction heads similar to Mask R-CNN.
Although these methods do not solve the exact matching problem, the pixel-level similarity scores output by a learnable CNN are still helpful for the task of mask prediction.
However, since cross-correlation between different pixels from the template and the current frame is required, they tend to be intensive on computation and memory. 




  
Mask-level Matching
Instead of operating on the pixel-level, some methods including our  resort to the mask-level matching.
Based on pre-computed feature maps, DyeNet <cit.> iteratively uses the re-identification and the recurrent mask propagation modules to retrieve disappearing-reappearing objects and handle temporal variations of pose and scale separately. 
Authors in <cit.> propose to track the object parts in the video and also compute the similarity scores between the proposal and template parts in the reference frame in order to deal with the missing of tracking and background noises.
In these work, matching is computationally light due to the number of masks/parts is significantly smaller than the number of pixels.
However, they all exploit the greedy solution for matching, , for each template, it returns the maximum-scored assignment if the score is above some threshold otherwise returns no assignment.
In contrast, we solve the matching problem via an iterative solver which is better than the greedy solution in most cases as verified by the experiments.




  
No Explicit Matching
Some of the recent works directly exploit deep neural networks to predict the masks.
In <cit.>, a pretrained CNN is first fine-tuned to predict both the segmentation mask and contour per frame and then a boundary snapping step is applied to combine both results. 
Authors in <cit.> later extended this work by introducing an online adaptation step to bootstrap the foreground object segmentation.
Video propagation network (VPN) <cit.> proposes a bilateral network along with a CNN to propagate the previously predicted masks and images.
MaskRNN <cit.> exploits the optical flow, images and mask proposals in a recurrent fashion to predict the masks per frame.
MaskTrack <cit.> uses a CNN which takes the last predicted instance mask and the current frame as input and outputs the refined mask. 
Spatial propagation network (SPN) <cit.> performs foreground segmentation and instance recognition simultaneously and then refines the instance masks using a spatial propagation module.


Pixel-wise metric learning (PML) <cit.> formulates the video object segmentation as a pixel-wise retrieval problem where the embedding space is predicted by a CNN and then learned via triplet-constrained metric learning.
Based on optical flow and a spatial CNN, a pixel-level spatio-temporal Markov random field (MRF) is built in <cit.> where approximate inference is achieved by a CNN.
Authors in <cit.> propose two sub-networks to compute the visual feature of the templates and spatial attention map from the last frame respectively to guide the mask prediction.
Relying on a U-Net, authors in <cit.> combine the concatenation of the current frame with last predicted mask and the concatenation of the reference frame with the template mask to predict the current mask.
These works are orthogonal to ours in the sense that we can use some of their networks as our feature extractor, while our matching layer could also potentially improve their models.



§ MODEL


In this section, we introduce our approach which consists of two key components: differentiable mask matching and mask refinement.
Our model assumes we have access to mask proposals in each frame. 
We first explain how we obtain the mask proposals. 
Then we describe our differentiable mask matching approach and discuss the mask refinement. 
Overview of our approach is illustrated in Fig. <ref>.



We assume a video has T frames. 
The mask templates in the first frame are denoted as R = { r_i| i = 1, …, n } where n is the total number of instances throughout the video. 







  
Mask Proposal Generation

We first extract mask proposals independently per frame with a COCO-pretrained Mask R-CNN <cit.> (details in Sec. <ref>).  
We only keep the top-50 proposals based on their scores, ensuring that recall is sufficiently high. This step is performed off-line, , our method will run on top of these fixed proposals. 
We denote mask proposals in frame t as P^t = { p_j^t| j = 1, …, m^t} where m^t is the total number of proposals at time t. 







  
Differentiable Mask Matching




The main motivation for performing object-level matching is to deal with the cases where large deformation, motion and dramatic appearance change are present. 
As aforementioned, proposal based matching is typically superior to optical flow based methods when motion is large. 
Moreover, we design a learnable matching cost which could potentially handle the dramatic appearance change and deformation. 

In particular, at time step t, we use a CNN, denoted as f_θ, to extract features for the mask proposals P^t and the templates R in the first frame. Here θ denotes the learnable parameters.
The details of the feature extractor is explained in Sec. <ref>. 
For the i-th mask template r_i (ground-truth mask in first frame) and the j-th mask proposal p_j^t, we compute their features as f_θ(r_i) and f_θ(p_j^t), respectively.
The matching cost matrix C^t consists of the cosine similarity between features and IoU between masks as below,


    C^t_i, j = (λ - 1) cos( f_θ(p_j^t), f_θ(r_i) )  - λIoU (p_j^t, r_i),


where λ is a hyperparameter and 0 < λ < 1.
The overall cost matrix C^t is of size n × m^t where each row and column correspond to a template and a mask proposal, respectively. From here on out, we drop the superscript t for simplicity.



We now introduce how we solve the bipartite matching problem.
In particular, we first formulate the minimum-cost bipartite matching as the following integer linear programming (ILP) problem,


    min_X       ( CX^⊤)
    
    	s.t.         X 1_m = 1_n
        X^⊤1_n≤1_m
        X ≥ 0 
        X_i, j∈{0, 1}    ∀ (i, j)


where X ∈ℝ^n × m is the boolean assignment matrix. 
1_n and 1_m are all one vectors with size n and m, respectively.
Here we slightly abuse the notation such that subscript i,j denotes the (i, j)-th entry of the matrix.
We add the constraint X ≥ 0 which will be helpful in understanding the relaxed version introduced later.
Note that the problem formulated in Eq. (<ref>) and the standard linear assignment problem (LAP) are slightly different in that we replace X^⊤1 = 1 with X^⊤1≤1.
This is due to the fact that the number of proposals m is much larger than the number of templates n in our case, , X is a wide matrix.

To solve this ILP problem, one can introduce dummy variables to make X a squared matrix and then use the Hungarian method to optimize  the standard LAP.
However, this naive extension increases the time complexity to O(m^3) and is not easy to back-propagate through.
Also, we may not necessarily require the exact matching, , real-valued approximated assignment matrix X may be sufficient for the later stage. 
Therefore, we resort to the following linear programming (LP) relaxation,


    min_X       ( CX^⊤)
    
    	s.t.         X 1_m = 1_n
        X^⊤1_n≤1_m
        X ≥ 0.


Although there are many standard solvers for LP, , the simplex method and the interior point methods, we here introduce a differentiable and easy-to-implement projected gradient descent algorithm.
The algorithm is presented in Alg. <ref> where N_grad, N_proj are the number of gradient decent (outer-loop) steps and the number of projection (inner-loop) steps, respectively.

At each iteration, we update X following the negative gradient direction.
The major challenge lies in projecting the updated X onto the constraint set.
It is not an easy task since the constraint set in Eq. (<ref>) is the intersection between three closed convex sets.

To compute the projection, we adopt a cyclic constraint projection method, known as Dykstra's algorithm <cit.> which is proved to be convergent for projection onto the non-empty intersection of finite closed convex sets.
The key idea is to break the whole constraint set into multiple simple subsets such that we can easily find the projection operator.
In particular, we can split the constraint set 𝒞 into individual constraints, , 𝒞 = 𝒞_1⋂𝒞_2⋂𝒞_3 where


    𝒞_1    = { X | X 1_m = 1_n}
    𝒞_2    = { X | X^⊤1_n≤1_m}
    𝒞_3    = { X | X ≥ 0 }.


It is straightforward to derive the projection operators each constraint as follows,


    𝒫_1(X)     = X - 1/m (X 1_m - 1_n) 1_m^⊤


    𝒫_2(X) = {[                            X             if    X^⊤1_n≤1_m; X - 1/n1_n (1_n^⊤ X - 1_m^⊤)                    otherwise ].


    𝒫_3(X)     = X^+


Note that 𝒫_3 is just the ReLU operator.
All these projection operators are differentiable and simple.
Dykstra's algorithm works by iteratively projecting the corrected point onto individual constraint set in a cyclic order and then updating the correction by the difference between pre-projection and post-projection. 
The final solution is obtained by averaging the intermediate projected assignment matrices.

The convergence results of the Dykstra's algorithm are established in <cit.>.
Relying on the convergence analysis under the framework of inexact projection primal first-order methods for convex optimization in <cit.>, we derive the following convergence result of our projected gradient descent algorithm for matching.

Let r_0 = ‖ X^0 - X^∗‖_F where X^0 and X^∗ are the initial and optimal assignment matrices, respectively. 
Let the learning rate 0 < α < min (15 r_0, r_0/‖ C ‖_F).
There exists some constants 0 ≤ c < 1 and ρ > 0 such that at any iteration of outer loop i in Alg. <ref>, the error of projection ‖ X^i - 𝒫_𝒞(X^i) ‖_F≤δ = ρ c^N_proj where X^i and 𝒫_𝒞(X^i) are the assignment matrix and its correct projection onto 𝒞, respectively.
Moreover, for any 0 < ϵ < 1, there exists a N_proj≥log_1/c( ρ√(15K/αϵ)) such that,

    δ≤αϵ/15 r_0

and after at most K iterations where

    K = ⌈6r_0^2/αϵ⌉,

the output of Alg. <ref> X̂ is ϵ-optimal, , ‖X̂ - 𝒫_𝒞(X̂) ‖_F≤ϵ and |( C X̂^⊤) - ( CX^∗⊤)|≤ϵ.

We leave the proof to the supplementary material.
In practice, the convergence is typically observed with moderately large N_proj and reasonable learning rate. 

The implementation of the overall algorithm is very simple.
Please see an example implementation using PyTorch (less than 50 lines) and empirical convergence analysis with different hyperparameters in the supplementary material.







  
Mask Refinement

After matching, for each template, we need to output one mask which is fed to the refinement stage.
Recall that we obtain the optimal assignment X̂ (approximately optimal if the previous algorithm does not run till convergence), we can compute a weighted combination of the proposal masks P.
Specifically, we first resize mask proposals such that they have the same resolution as the input image.
We then paste them into void images to get the full masks which have the same size as the input images, denoted as P̃.
We obtain the matched mask P̂ as:


    P̂ = X̂⊗P̃


where X̂∈ℝ^n × m, P̃∈ℝ^m × H × W, P̂∈ℝ^n × H × W, and ⊗ indicates the tensor contraction operator along the last and the first dimensions of X̂ and P̃, respectively.
Here H and W denote the height and width of the input image.
Each spatial slice of P̂ denotes the matched mask corresponding to a particular template.
This process is shown in the yellow box of Fig. <ref>.
The matched mask for the template will be used to compute the IoU score, shown in Eq. (<ref>), at the next time step. 
Therefore, we propagate the latest mask information over time.




Given the output mask from matching, we then refine it using the template of the same instance.









In particular, we construct the input by stacking multi-scale image features from the backbone, the matched mask and template mask. The multi-scale features are extracted from the last layer of the conv 2-5 blocks in the feature extractor backbone, respectively.   

Inspired by RVOS, we adopt a decoder containing four ConvLSTM <cit.> layers as the refinement module to predict the masks of all objects at each time step and carry over the memory and hidden states to the next time step. 















§ EXPERIMENTS


In this section, we compare our  with a wide range of recent competitors on YouTube-VOS, DAVIS 2017 and SegTrack v2 datasets. 
YouTube-VOS has 3,471 and 474 videos for training and validation, respectively. Among the 91 object categories in the validation set, 65 are seen in the training set while the other 26 are unseen. 

DAVIS 2017 has 60 and 30 video sequences for training and validation respectively and the average video length is around 70.
The average number of instances per sequence are 2.30 and 1.97 for training and validation, respectively.
For the SegTrack v2 dataset, there are 14 low resolution videos (947 frames in total) with 24 generic foreground objects.

All of our experiments are conducted on NVIDIA Titan XP GPUs.



































 §.§ Implementation Details
 

We first introduce implementation details of our model.





  
Mask Proposal Generation
In the stage of mask proposal generation, we use Mask R-CNN with ResNeXt-101-FPN as the backbone which is pretrained on COCO dataset <cit.>.
Score threshold of the ROI head is set to 0.
We resize the input image such that its short-side is no larger than 800. 

We first train the class-agnostic binary mask proposal network on COCO.
Following the strategy used in <cit.>, we then finetune the proposal network on the combination of COCO and YouTube-VOS with learning rate 0.02, batch size 8 and number of training iteration 200,000.



  
Differentiable Mask Matching
For the feature extractor f_θ, we use a COCO-pretrained Mask R-CNN with a ResNet-50-FPN backbone. 
We also try a ResNet-101 backbone of which the weights are initialized from the released model of RVOS <cit.>.
We denote this model as DMM-Net+. 

Note that it is possible to share the backbone between the proposal network and the feature extractor of matching such that the overall model is more compact. 
However, sharing backbone leads to worse results in our experiments which may suggest that generating good proposals and learning good feature for matching require different representations.

After we obtain the proposals for each frame, we perform ROI pooling for each proposal to extract multi-scale feature 
from the backbone and then average the feature spatially to obtain a single feature vector. Similar to the feature fed to the refinement layer, we obtain the proposals feature from the last layer of conv2-5 block in the backbone. 
Input image of the feature extractor is resized such that the short-side is no larger than 480 and 800 for DAVIS 2017 and SegTrack v2, respectively. 
For YouTube-VOS, we resize image to 255×448 in order to have a fair comparison with S2S <cit.> and RVOS <cit.>.
The score weight λ used to compute the matching cost in Eq. (<ref>) is set to 0.3 for DAVIS 2017 and YouTube-VOS, and 0.9 for SegTrack v2.
For the mask matching, we set N_grad = 40, N_proj = 5 and learning rate α=0.1. Ablation study on the hyperparameters of the matching is left in the supplementary material. 
After the matching assignment matrix X̂ is obtained, we also found it helpful to remove the non-confident matching by applying a differentiable masking  X̂ = X̂·1 [X̂ = max(X̂)].




















  
Refinement Network








A light version of refinement network containing only four ConvLSTM layers is used for experiments on YouTube-VOS to reduce the computational cost. 
The weights are randomly initialized and trained with the matching layer in an end-to-end manner. 
Input images are resized to be 255 × 448 for both training and inference. 
The refinement network outputs our final predicted masks. 

A heavier version of refinement network is used for model trained on DAVIS 2017. We follow the U-Net style architecture as RGMP <cit.> and initialize the weights from their released model. The refinement network is also trained together with the matching layer. 



  
Fine-tune

Since our  is end-to-end differentiable, we fine-tune it on the training sets of YouTube-VOS and DAVIS 2017.
We use the Adam <cit.> optimizer with the learning rate 1.0e^-4 for the weights initialized from pre-trained model and 1.0e^-3 for the weights from random initialization. 
We set batch-size to 24 and train on YouTube-VOS dataset for 10 epochs in total. 
Data augmentation such as random affine transformation is applied during training. 
On DAVIS 2017, we use the same optimizer with learning rate 1.0e^-7 and batch size 1. We fine-tune the model for 8 epochs in total.










  
Online Learning
For online learning, we train both the proposal generator and the DMM-Net with the refinement module on the first frame of the validation set. We use the same learning rate and batch size as  Section <ref> except that the number of epochs is 100 for online learning. 



  
Evaluation
For YouTube-VOS and DAVIS 2017, we follow <cit.> and use the region (𝒥), boundary (ℱ) and their average (𝒢) score as the metrics.
For SegTrack v2, there are two types of mean IoU adopted in the existing literature. 
Specifically, one can compute the IoU averaging over all instances per frame and then average over all frames as in <cit.>, denoted as mIoU^∗.
One can also compute the IoU per instance, average over the frames where the instance has appeared, and then average over all instances as in <cit.>, denoted as mIoU^†. 
We report both metrics on this dataset.



 §.§ Main Results








  
YouTube-VOS
We fine-tune both our DMM-Net and proposal net on YouTube-VOS. We first split the official training set into train-train, train-val and train-test splits. Our train-val split consists of 200 videos while the train-train split consists of 3000 videos. Our training is performed on the train-train split, and the best model is selected based on the performance on the train-val split. The final performance on the official validation set is reported in Table <ref>. 
Compared to the state-of-the-art method S2S, our model achieves competitive segmentation metrics with double speed.
In general, we obtain a good trade off between the performance and running time. 
Moreover, we found that using a stronger backbone, , DMM-Net+, can further boost the performance.



  
DAVIS 2017
We compare with a wide range of recent competitors on the validation set of DAVIS 2017. For experiments on DAVIS, we only fine-tune our DMM-Net on the training set of DAVIS 2017 and use the proposal generator pretrained on COCO. 
The models without online learning are listed in Table <ref>. 
From the table, it is clear that without online learning, our method achieves the state-of-the-art performance.





In Fig. <ref>, we show the qualitative results of our  at different time steps (uniformly sampled percentage the whole video length) of each video sequence.
From the figure, we can see that our model consistently keeps a very good segmentation quality as time goes on.
We also show a visual comparison at the last frame of the soapbox sequence with other strong competitors in Fig. <ref>.
It is clear that our model does a better job in segmenting the details of the persons and the soapbox.
However, some failure cases still exist. 
For example, in the 100% column and 4th row of Fig. <ref>, the segmentation of the goldfish in the bottom-right corner is unsatisfying.



  
SegTrack v2
We test our  model (fine-tuned on DAVIS 2017 training set) directly on the full SegTrack v2 dataset.
We do not perform any fine-tuning on this dataset. 
Moreover, for simplicity, we again do not adopt any online learning such that we could fully test the generalization ability of our model.
The quantitative results are listed in Table <ref>.
From the table, we can see that without any fine-tuning and online learning, our  achieves comparable performance to the state-of-the-art methods.
We show some visual examples in the bottom two rows of Fig. <ref>.
We can see that our model again has a consistently high segmentation quality across different time steps.





 §.§ Ablation Study


In this section, we conduct thorough ablation study to justify the design choice and hyperparameters of our model.



































































































  
Greedy vs. Hungarian vs. Our Matching Layer
We first test the matching layer against the optimal matching using the Hungarian method and the popular greedy approximation during inference.
For a fair comparison, we use the same set of mask proposals, the same feature extractor network pretrained on COCO dataset.
We show the mean of 𝒥 and ℱ scores on YouTube-VOS dataset in Table <ref>. All the ablation results are obtained by training on our train-train split and evaluating on train-val split. 
From the table, we can see that our matching layer has similar performance compared to the optimal matching and is superior to the greedy one during inference.








































  
End-to-End Fine-tuning
We now study the effect of fine-tuning the whole model on the train-train split of YouTube-VOS.
As shown in Table <ref>, the performance is improved significantly, which verifies the benefits of the end-to-end training and the differentiability of our matching layer.



  
Mask Proposal Network
We also try different backbones for the mask proposal network and fine-tune the network on YouTube-VOS dataset. As shown in Table <ref>, the performance gain is pretty high by fine-tuning the proposal net on large scale video dataset such as YouTube-VOS.




  
Number of Unrolled Steps 
At last, we investigate the number of unrolled steps of the refinement network during training.
As shown in Table <ref>, unrolling more than 2 step seems not helpful and increases the memory cost significantly.
Note that during test we unroll from the beginning to the end of the video sequence. 






























§ CONCLUSION

In this paper, we propose the  () for solving the problem of video object segmentation.
Relying on the pre-computed masks proposals,  first conducts the mask matching between proposals and templates via a projected gradient descent method which is guaranteed to converge and fully differentiable.
It enables the learning of the cost matrix of matching.
Based on the template mask, we then refine the current matched mask to further improve the segmentation quality.
We demonstrate that our model achieves the state-of-the-art or comparable performances under different settings of several challenging benchmarks.
In the future, we would like to apply our differentiable matching layer to other backbone networks for the purpose of further boosting the performance.
Moreover, exploring the mask matching in a longer temporal window, , a multi-partite matching problem similar to tracking, would be very interesting.



  §.§.§ Acknowledgments

We gratefully acknowledge support from Vector Institute.
RL was supported by Connaught International Scholarships, RBC Fellowship and NSERC. 
SF acknowledges the Canada CIFAR AI Chair award at Vector Institute. Part of this work was also supported by NSERC Cohesa grant, and Samsung. 
We are grateful to Relu Patrascu for infrastructure support.




ieee_fullname




