Region Refinement Network for Salient Object Detection
Zhuotao Tian Hengshuang Zhao Michelle Shu Jiaze Wang Ruiyu Li Xiaoyong Shen Jiaya Jia








 

The infrared or ultraviolet scale for a gauge theory 
    

Zhuotao Tian 1 Hengshuang Zhao 1 Michelle Shu 2 Jiaze Wang 1 




Ruiyu Li 3 Xiaoyong Shen 3Jiaya Jia 1,3




 1The Chinese University of Hong Kong  2John Hopkins University



 3Youtu Lab, Tencent 



{zttian,hszhao,jzwang,leojia}@cse.cuhk.edu.hk shu.michelle97@gmail.com   {royryli,dylanshen}@tencent.com 

    December 30, 2023
==========================================================================================================================================================================================================================================================================================================================




Albeit intensively studied, false prediction and unclear boundaries are still major issues of salient object detection. In this paper, we propose a Region Refinement Network (RRN), which recurrently filters redundant information and explicitly models boundary information for saliency detection.
Different from existing refinement methods, we propose a Region Refinement Module (RRM) that optimizes salient region prediction by incorporating supervised attention masks in the intermediate refinement stages. The module only brings a minor increase in model size and yet significantly reduces false predictions from the background. To further refine boundary areas, we propose a Boundary Refinement Loss (BRL) that adds extra supervision for better distinguishing foreground from background. 
BRL is parameter free and easy to train. We further observe that BRL helps retain the integrity in prediction by refining the boundary.
Extensive experiments on saliency detection datasets show that our refinement module and loss bring significant improvement to the baseline and can be easily applied to different frameworks.
We also demonstrate that our proposed model generalizes well to portrait segmentation and shadow detection tasks.










§ INTRODUCTION

Salient objects are one primary information source in the human perception of natural scenes. Salient Object Detection (SOD), as an essential part of computer vision, aims at identifying the salient objects in wild scenes. Previous SOD methods, as fundamental tools, have benefited a wide range of applications. Topics that are closely or remotely related to visual
saliency include semantic segmentation <cit.>, instance segmentation <cit.>, object detection<cit.>, to name a few. 

Early SOD methods utilize low-level, hand-crafted features that are vulnerable to complex scenarios. Recently, deep learning based methods <cit.> greatly improved performance on benchmark datasets. However, this task is still challenging in natural scenes due to the varying attributes such as tiny and thin structures, various background and multiple objects. 

To alleviate these difficulties, the work of <cit.> used recurrent structures to refine results. These methods do not control the information flow during the recurrent refinement process and allow redundant information pass, possibly resulting in performance reduction, which will be discussed more in Section <ref>. To selectively pass useful information for better results, we design a cascaded region refinement module with a gating mechanism that can be viewed as an attention map. It is guided by the ground truth to control information flow and helps the network focus on refining only salient regions. Our extensive experiments in Section <ref> show that this design is general for applying to other frameworks and improves their respective performance.

Common boundary area errors on salient object predictions are also addressed. As shown later in Figure <ref>(d)&(f), absolute errors between ground truth and predicted saliency maps mainly stem from the boundary area. To obtain a better prediction, we propose Boundary Refinement Loss (BRL) to help refine boundary areas. Compared with previous boundary strategies <cit.>, BRL is more efficient because it does not require additional parameters and computation. Although the method of <cit.> also considers boundary enhancement, our ablation study in Section <ref> show that our strategy achieves better performance because both foreground and background near the boundary are supervised and optimized.

Our contribution in this paper is threefold.



    
  * We propose a new effective region refinement module.
    
  * We propose an efficient enhancement strategy to refine prediction on the boundary.
    
  * Our model achieves new state-of-the-art results on five representative benchmark datasets. Also, our designs can be easily applied to other frameworks.










§ RELATED WORK




In recent years, salient object detection has attracted much attention due its wide range of applications <cit.>. Methods designed for salient object detection can be divided into two categories of deep learning based methods and those without learning. Non-deep-learning methods are primarily based on low-level features like regional contrast <cit.> and priors <cit.>. More details can be found in <cit.>. In this section, we mainly discuss deep learning based solutions.

Early deep learning methods use features generated by deep neural networks (DNNs) to calculate saliency of image units, like super-pixels and proposals. For example, in <cit.>, with the features extracted from DNN, the score of each super-pixel of an image can be easily calculated. In <cit.>, two networks are utilized (one for local superpixel estimation and the other for global proposal search). The final saliency map is generated by a weighted sum of salient object regions. 

Later, end-to-end frameworks <cit.> became the mainstream and many effective structures were proposed. In <cit.>, multi-level features are aggregated from a backbone network for salient object detection. <cit.> proposes a deep contrast network with a pixel-level fully convolutional stream and a segment-wise spatial pooling module to deal with blurry regions. <cit.> introduces short connections to the skip-layer structures within the holistically-nested edge detector (HED) <cit.> to boost performance. <cit.> designs a bi-directional structure that enables information flow between lower and higher layers, and <cit.> proposes an attention mechanism that helps capture contextual information for each pixel in both local and global forms. <cit.> proposes to hierarchically and progressively refine the saliency map with local context information. Our work is different in that we use recurrent structures and exploit boundary enhancement.


Recurrent structures have been proved useful <cit.>. In R3Net <cit.>, Residue Refinement Blocks (RRBs) are recurrently applied to the extracted deep feature to refine the output. <cit.> combines spatial transformer and recurrent network units to iteratively perform saliency refinement on sub-regions of images. <cit.> leverages a stage-wise refinement network to refine saliency prediction in a coarse-to-fine manner. Our method is inherently different from the above methods. With a novel supervised mask, our proposed cascaded refinement module involves the attention-like mechanism. It significantly improves performance by keeping meaningful information. 

On the other hand, boundary (or contour) is essential to consider <cit.>. C2S-Net <cit.> has two branches: one predicts contour and the other generates saliency maps with a contour-to-saliency transferring method to utilize contour learning in saliency map prediction. <cit.> proposes a local Boundary Refinement Network to adaptively learn the local contextual information for each pixel, which helps capture relationships in neighbors. <cit.> uses a separate branch to generate boundary prediction and fuses it with a region saliency map to refine results. Both methods involve additional parameters to form a new branch for boundary prediction. Differently, <cit.> proposes an additional IoU Boundary Loss generated by gradients of ground truth and predicted salient region map. Likewise, our boundary refinement strategy only involves an additional loss, but it directly optimizes the boundary region rather than gradients, resulting in even better performance.












§ OUR METHOD



In this section, we motivate our solution and detail the new Region Refinement Module (RRM) and Boundary Refinement Loss (BRL), vastly useful in saliency detection. Then we propose the Region Refinement Network (RRN).






 §.§ Region Refinement Module




Recently proposed refinement modules refine input features without any selection process. We note input features always bring redundant information that may cause error or ambiguity. For example, for features extracted from multiple layers <cit.>, pixels belonging to tiny structures, such as animal hair and tail, need additional spatial information, while pixels for large objects like elephant body and airplane need more semantic clues. Without carefully handling this issue, the final prediction may contain hard samples for false positives with high probability and false negatives with low probability. To alleviate this difficulty, we propose Region Refinement Module (RRM) in the cascaded structure that helps accumulate useful information to the final feature. In addition, different from previous work in Section <ref>, we utilize the output salient map in each intermediate refinement stage as an attention map to guide the information flow. Our experiments with/without using the mask in Section <ref> show that this gating mechanism can refine features in a particular way for better overall performance.








  §.§.§ Module Structure



As shown in Figure <ref>, RRM is composed of several Region Refinement Layers (RRL). N is the number of RRL, and M is the number of 3×3 convolutions (equals to the number of 1×1 convolutions) in each RRL.
Each RRL receives coarse features from the input and previous layers and passes the refined features to the next layer. At the end of each RRL, the output is processed by a Sigmoid function. It is then used as an attention mask to gate the refined feature to the next layer, making the model focus more on refining salient pixels.


    RRL_i,j={[                              𝒢(X)                                                             i=1,j=1;         𝒢(ℱ(RRL_i-1,j⊙Y_i-1) + X)                                                             i>1,j=1;                      𝒢(RRL_i,j-1)                                                             i=1,j>1; 𝒢(ℱ(RRL_i-1,j⊙Y_i-1) + RRL_i,j-1)                                                           Otherwise ].



    Y_i = ℱ_ρ(RRL_i,M)    i=1,2,......,N


The formulation of RRM is shown in Eqs. (<ref>) and (<ref>) where X is the input feature,  Y_i is the output from RRL_i, N is the number of layers, M is the number of 3×3 convolutions, ⊙ denotes pixel-wise multiplication, ℱ represents 1×1 convolution with ReLU function, 𝒢 represents 3×3 convolution with ReLU function and ℱ_ρ represents 1×1 convolution with Sigmoid function. 

As Eq. (<ref>) shows, output Y_i from current layer RRL_i is used as a gate to control the information passed to next layer RRL_i+1 by a pixel-wise multiplication, which helps filter out redundant feature with low probability and retain the information of interest with high probability in the previous stage.



    RRL_f = ℱ(U(RRL_N,M)) +  𝒢(U(X))


    Y_f = ℱ_ρ(RRL_f)

At last layer RRL_N, as Eqs. (<ref>) and (<ref>) show (U is the upsampling function), RRM generates final output Y_f by upsampling RRL_N,M and input feature twice larger then applying 1x1 convolution to both. 


In RRM we calculate cross entropy loss for all outputs. Since RRM only takes Y_f as the final output, the final loss combines Y_f  and Y_i  as

    L = L_f + λ/N∑_i=1^NL_i = L_f + σ∑_i=1^NL_i

where L_f is the cross entropy loss of Y_f, L_i is the cross entropy loss of Y_i. σ is used for balancing the loss of previous outputs. σ equals λ/N where N is the number of RRL and λ is a balancing factor for Y_i. The effect of σ is discussed in Section <ref>.






 §.§ Boundary Refinement Loss





  §.§.§ Motivation




As boundary causes errors on saliency prediction as Figure <ref> shows, recent work such as <cit.> and <cit.> added a boundary (or contour) detection branch to the framework to enhance foreground mask generator. However, these structures bring additional parameters and computation and make the system more complex. In addition, because of the fragility of boundary prediction, recent SE2Net<cit.> applied fully connected conditional random field (CRF) <cit.> on both salient region prediction and boundary prediction to refine results before post-processing. Performing CRF twice is quite time-consuming. 

Differently, we propose a simple and yet effective refining strategy called Boundary Refinement Loss (BRL) on the existing segmentation branch. BRL only needs generating a loss on predictions, which involves negligible computation and no additional parameter. An efficient boundary refinement strategy is also proposed in <cit.> that utilizes gradients (denoted as `Grad') to generate the IoU Loss. We note that the boundary mask generated by gradients is not stable since the boundary is composed of a lot of values between 0 and 1 while this method only covers a small portion of the boundary area. The loss generating area of BRL covers both foreground and background, which is more robust than that of <cit.>. Therefore, such a strategy of <cit.> performs less well than BRL, as shown in Table <ref>. 

As shown in Figure <ref>(g)-(h), results refined by the proposed BRL have a much sharper boundary than baseline in (c)-(d) and `Grad'<cit.> in (e)-(f). Even the paws can be clearly detected in (g). Additionally, as shown in the last row of Figure <ref>, the internal integrity of the salient region is better maintained if the boundary is refined well.









  §.§.§ Loss Formulation



 Before training, we generate a boundary area for each image as shown in Figure <ref>. Since the width of the gradient boundary generated by Sobel operator is very small, we extract the boundary mask as shown in Figure <ref>(c). Note that each pixel belonging to the original boundary expands on its surroundings to form the mask in (c). Therefore the new boundary area covers both background and foreground. We denote the expanding distance of each pixel as the Width of the boundary area. We find Width=5 yields best results and set it as the default without specification. 

During training, BRL simply adds an extra boundary loss L_b onto segmentation loss in the form of

    L_b = 𝒯(P ⊙ B, G ⊙ B)

where ⊙ denotes pixel-wise multiplication, P is the foreground prediction, G is the ground truth, B is the boundary area generated before training as Figure <ref>(c) and 𝒯 is the boundary loss function. In Eq. (<ref>), G and P are masked by B, which filters out other areas and lets the loss function focus only on the foreground and background segmentation in the new boundary area. Similar to previous work <cit.>, we choose the IoU Loss as 𝒯.



    L =L_f + σ∑_i=1^NL_i + η (L_bf + σ∑_i=1^NL_bi)

If BRL is used, the total loss L is shown in Eq. (<ref>) where L_bf and L_bi are boundary losses of Y_f and Y_i (i = 1,2,......,N), and η is used for weighing the sum of L_bf and L_bi. Following <cit.>, we set η to 1.0 in all experiments. As shown later in experiments, BRL improves the results by a large margin by refining the boundary area on segmentation prediction.





 §.§ Region Refinement Network




To prove the effectiveness of RRM and BRL, we proposed the Region Refinement Network (RRN) as shown in Figure <ref>. The network contains a base network (b) and a Region Refinement Module (RRM). The base network sends features extracted from input images to RRM for further refinement, and BRL is then applied to the outputs of RRM.

Our baseline model (Figure <ref>(b)) is built upon ResNet-50<cit.> (Figure <ref>(a)). In <cit.>, it is observed that features in multiple levels together are helpful for accurate prediction. We accordingly modify the network in the FPN<cit.> style, as shown in Figure <ref>(b). By fusing features from four intermediate layers from FPN to form a multilevel feature before sending to the pyramid pooling module, the new base network outperforms the original PSPNet<cit.> by a large margin, as demonstrated in Table <ref>. The fusion of feature maps in different sizes is done by bilinear interpolation and concatenation.

For a fair comparison, we select VGG16 <cit.>, ResNet-50 <cit.> and ResNet-101 <cit.> as our backbones.  
As for the VGG16 model, we extract features from Conv2_2, Conv3_3, Conv4_3 and Conv5_3 and send them to the FPN part to form a merged feature with 1/4 of the resolution. For models based on ResNet-50 and 101, we use the same backbone configuration as PSPNet <cit.>, and extract the last feature maps of the first MaxPool, Conv2, Conv3, Conv4 and Conv5 to the fusion stage. Features extracted from the backbone network are processed by 1×1 convolutions with 128 output channels.







§ EXPERIMENTS











 §.§ Settings



Our framework is based on PyTorch. The backbone network is initialized on VGG16, ResNet-50 and ResNet-101 trained on ImageNet. Other layers are initialized with PyTorch default setting. We adopt SGD with momentum as the optimizer where momentum and weight decay are set to 0.9 and 0.0001 respectively. Following <cit.>, we use the `poly' learning rate decay policy where current learning rate equals to the base learning rate multiplying (1 - current_iter/max_iter)^power. We set the base rate to 0.01 and power to 0.9. As for hyper-parameters, we set N, M, Width, σ and η to 4, 3, 5, 2.0 and 1.0 respectively. Data augmentation is important for over-fitting prevention. In our experiments, we first perform mirroring and re-scaling from 0.75 and 1.25, and then add random rotation from -10 to 10 degrees on training images. Finally, we randomly crop 401×401 patches from the processed images as training samples. The batch size is 16, and we run 50 epochs for training in our setting. We output the saliency map from our network without additional refinement like the fully connected conditional random field (CRF) <cit.> used in <cit.>, <cit.>, <cit.> and <cit.>.
Our experiments are conducted on an NVIDIA Tesla P40 GPU and Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz.




  
Datasets
We use the training data of DUTS <cit.> (10,553 images) for training as previous work <cit.>,<cit.>,<cit.>,<cit.>. Then we evaluate our model on HKU-IS <cit.> (4,447 images), 
ECSSD <cit.> (1,000 images),
DUTS <cit.> (5,018 images), 
DUT-OMRON <cit.> (5,168 images) 
and PASCAL-S <cit.> (850 images).
For a fair comparison, we also train our model (VGG16) on MSRA-10K<cit.>(10000 images) and MSRA-B<cit.>(2500 images).





  
Evaluation Metrics
Our evaluation metrics are Mean Absolute Error (MAE) and maximum F-measure (F_β). MAE is the average pixel-wise error between prediction and ground truth as 

    MAE = 1/H W∑_i=1^H∑_j=1^W|P(i,j) - G(i,j)|

where P is prediction, G is ground truth, H and W are height and width of the testing image. 
F_β is used for measuring the overall performance <cit.> with a balancing factor β as

    F_β = (1+β^2)(Recall + Precision)/β^2Precision + Recall

where β^2 is set to 0.3 as suggested in <cit.>. A good result usually yields a large F_β and a small MAE.







 §.§ Comparison with the State of the Art









In this section, we compare our proposed method with several state-of-the-art deep learning based algorithms shown in Table <ref>. For a fair comparison, we calculate the results on the salient maps; or the models provided by authors with the same evaluation code. It shows that our method achieves the new state-of-the-art on all backbone methods. The speeds of our final models with VGG16, ResNet50 and ResNet101 are 54.8, 52.1 and 32.8 FPS respectively on 352×352 input.

For visual comparison, we compare the salient maps with state-of-the-art methods (CPD-R <cit.>,  PiCANet-R <cit.>, R3-Net <cit.>, DGRL <cit.>, C2S-Net <cit.>, SRM <cit.>) shown in Figure <ref>. In Figure <ref>, our final model (c) is composed of both BRL and RRM. It is more robust to large objects, tiny structures like limbs and ears, complex background, and multiple objects, which prove its effectiveness. Additionally, our method yields much better boundaries in salient maps than other methods.









 §.§ Ablation Study and Analysis





  §.§.§ Effectiveness of BRL




In order to demonstrate the effectiveness of our proposed Boundary Refinement Loss (BRL), we compare BRL with the enhancement strategy (denoted as `Grad') proposed in <cit.> and the baseline structure in Table <ref>. `Grad' yields slight improvements over the baseline model except for PASCAL-S in terms of F_β and MAE. Our proposed BRL makes the baseline model generate higher quality results on all benchmark datasets. The results are even comparable with previous state-of-the-art ones. The qualitative comparison is shown in Figure <ref> where BRL in (g)-(h) refines boundary better than Grad <cit.> in (e)-(f). We put experiments on different boundary Width in the supplementary file.













  §.§.§ Effectiveness of RRM




As shown in Figure <ref>(c), RRM further refines the results by  reducing false prediction existing in Figure <ref>(d), and the model with RRM (N=4,M=3) achieves best performance in Table <ref>. In Eq. (<ref>), we follow <cit.> to set η to 1.0. There are three parameters (N, M and σ) that are not decided. 

In this section, we first conduct experiments on different values of σ to find the best balancing factor. Then, we show the effectiveness of the mask used as attention for controlling information flow. Examples illustrating the refinement process in each stage of RRM and experiments on different values of N (layer number) and M (convolution number) are shown in the supplementary material. We find RRM with (N=4, M=3) brings the best performance.






  
Balancing Factor (σ)
The side outputs from intermediate layers of RRM generate auxiliary losses σ∑_i=1^NL_i that may influence the learning of final output. In Table <ref>, to find the best σ for balancing learning between side and final output, we select five values for σ as [0, 0.25, 0.5, 0.75, 1.0] and find that 0.5 yields the best performance. When σ = 0, there is no supervision on side outputs, which causes self-learned attention maps. However, as shown in our experiments, the supervision of ground truth on side outputs produces better attention maps to control the information flow in RRM. Also, when σ is close to one, the loss information back-propagated from the final output is dominated by previous layers' outputs; a large σ yields sub-optimal final results. Therefore σ should be close to 0.5 for optimal results shown in Table <ref>.





  
Mask in RRM
In Figure <ref>, we show the output of the previous layer is used as an attention mask applied to the next layer to control the information flow. We compare models with and without masks in RRM to evaluate the usefulness of the mask and show results in Table <ref>. It is clear that when N and σ are set reasonably, without the mask to control the useful information flow in the module, redundant information becomes the bottleneck.












 §.§ Applications



Our method (RRN) can be applied to other tasks to further prove its effectiveness. For Portrait segmentation based on the portrait dataset <cit.>, we apply RRN and make a comparison with the results of <cit.>. As listed in Table <ref>, RRN (VGG-16 backbone) achieves the best mIoU performance.

We also apply RRN (VGG-16 backbone) to shadow detection and evaluate it on the SBU <cit.>, ISTD <cit.> and UCF <cit.> datasets. We first train our model on SBU dataset, which is the largest publicly available annotated shadow dataset with 4,089 training images and test the model on SBU testing images, ISTD and UCF. We use the Balanced Error Rate (BER) as our evaluation metrics and the same evaluation code as <cit.>. As shown in Table <ref>, our method outperforms all previous state-of-the-art methods on the benchmark.









  §.§.§ Other Frameworks




Region Refinement Module and Boundary Refinement Loss, as two independent refinement strategies, can be applied to other frameworks generally. Besides the baseline in Table <ref>, we also use it in Cascaded Partial Decoder (CPD) <cit.> (ResNet-50) and R3Net <cit.> (ResNeXt-101), two state-of-the-art SOD frameworks, to verify our designs. In our experiments, we use the default training setting and use fixed initial random seeds for fair comparisons. We simply insert RRM (without the final upsampling stage RRL_f) into the two models without any further change.





  
CPD We apply two 4-layers RRMs in two aggregation modules of CPD before the final convolution of each aggregation and use the pre-generated boundary masks for BRL during training. As listed in Table <ref>, though our reproduced baseline is less effective as claimed in the original paper, RRM and BRL both improve performance of the model on most of the benchmark datasets. They still help the final model (CPD+BRL+RRM) outperform the original one. It is worth noting that each RRM only brings about 3MB increase in terms of the model size. The speeds (on 352×352 input) of CPD and CPD with two RRMs are 47.3 and 36.2 FPS respectively.







  
R3Net On R3Net, we apply a single 4-layers RRM after the high-level integrated
features and before the following Residual Refinement Blocks. R3Net in the original paper is trained on MSRA10K <cit.> with results refined by CRF<cit.>. We also train the improved models on DUTS-TR/MSRA10K and test them with/without CRF to verify the performance as shown in Table <ref>. The speeds (on 352×352 input) of R3Net and R3Net+RRM and R3Net+RRM+CRF are 21.8, 19.2 and 2.4 FPS.







§ CONCLUSION



We have presented a novel Region Refinement Module (RRM) to optimize saliency prediction with a gating mechanism. Our boundary refinement loss (BRL) refines the boundary directly on the prediction without introducing new parameters and extra computation. These two designs not only help our Region Refinement Network (RRN) achieve new state-of-the-art results on five popular benchmark datasets but also can be easily applied to other frameworks. Possible future work includes extending these designs to video saliency detection and further improving the performance.













§ APPENDIX



This section is the supplementary material for Region Refinement Network for Salient Object
Detection. In Section <ref>, we illustrate the mean, variance and error bars of our model. More experiments on our proposed Region Refinement Module (RRM) and Boundary Refinement Loss (BRL) can be found in Section <ref>. In Section <ref> and Section <ref>, we show more examples for visual comparison and links for publicly available datasets used in our experiments.





 §.§ Mean, Variance and Error Bar




We train extra 10 models to show the mean, variance and error bars. Due to the limited computing resources, extra experiments are based on VGG16 backbone and trained 50 epochs on MSRA-B (2500 images) dataset to have a quick analysis. Finally, we evaluate these models on HKU-IS <cit.> (4447 images), 
ECSSD <cit.> (1000 images) 
DUTS <cit.> (5018 images), 
DUT-OMRON <cit.> (5168 images) 
and PASCAL-S <cit.> (850 images). The results are shown in Table <ref> and Figure <ref>.










 §.§ More Details and Ablation Studies





  §.§.§ Region Refinement Module (RRM)

In the proposed Regional Refinement Network (RRN) and applications on CPD<cit.> and R3Net<cit.>, the input features have 64, 96 and 256 output channels respectively (we process the input feature in RRN by a 1×1 convolution with 64 output channels), therefore all intermediate 3×3 and 1×1 convolutions in RRM have same input and output channels, except for the last convolution in each RRL (Region Refinement Layer) outputs a salient map with 1 output channel. 



  
Layer Number (N)
The results of different N are shown in Table <ref>. Note that when N is set to zero, the model degrades to Baseline+BRL in Table <ref>. Our results from RRM_0 to RRM_7 are gradually refined, decent in terms of both F_β and MAE. We observe that the model with RRM_4 is optimal in most cases even if it contains fewer parameters than RRM_N (N > 4). We thus choose to apply RRM_4 in our final model.



  
Number of 3×3 and 1×1 Convolution (M)
Then, apart from the vertical analysis on layer number N, we further perform horizontal analysis on the number of convolutions. Since the numbers of 1×1 convolutions and 3×3 convolutions are equal in each Region Refinement Layer (RRL), we use M to denote the numbers of two convolutions in each RRL. The results of M on RRN are as Table <ref> shows. M = 3 yields the best performance.





  
RRM Refinement Process
We take outputs from RRL_1, RRL_2, RRL_3 and RRL_4 to better illustrate the refining process in RRM. As shown in Figure <ref>, false predictions are gradually eliminated from RRL_1 to RRL_4.






  §.§.§ Boundary Refinement Learning (BRL)



  
Boundary Mask Generation
In BRL, we use boundary mask for extra supervision on boundary area. In this section, we show the impacts of different boundary widths. As shown in Table <ref>, Width=5 yields the best performance.








  §.§.§ More Results of R3Net



We provide the detailed results of our application on R3Net <cit.> in Table <ref> where we add the performance of DUTS+CRF and MSRA10K+Non-CRF to the original table.







 §.§ More Examples of Visual Comparisons




More visual comparisons are shown in Figure <ref>.










 §.§ Links for Datasets




  §.§.§ Salient Object Detection

HKU-IS <cit.>: https://i.cs.hku.hk/~gbli/deep_saliency.html

ECSSD <cit.>: http://www.cse.cuhk.edu.hk/leojia/projects/hsaliency/dataset.html 

DUTS <cit.>: http://saliencydetection.net/duts/ 

DUT-OMRON <cit.>: http://saliencydetection.net/dut-omron/ 

PASCAL-S <cit.>: http://www.cbi.gatech.edu/salobj/ 

MSRA-B <cit.> & MSRA-10K <cit.>: https://mmcheng.net/zh/msra10k/







  §.§.§ Shadow Detection

UCF <cit.>: http://aqua.cs.uiuc.edu/site/projects/shadow.html 

SBU <cit.>: https://www3.cs.stonybrook.edu/~cvl/dataset.html 

ISTD <cit.>: https://drive.google.com/file/d/1I0qw-65KBA6np8vIZzO6oeiOvcDBttAY/view 





  §.§.§ Portrait Segmentation

Flickr Portrait Dataset<cit.>: http://xiaoyongshen.me/webpage_portrait/index.html


ieee
    




