Hard Pixel Mining for Depth Privileged Semantic Segmentation
Zhangxuan Gu Li Niu Haohua Zhao Liqing Zhang





















































































Hard Pixel Mining for Depth Privileged Semantic Segmentation
    Zhangxuan Gu,
Li Niu*,
Haohua Zhao, and
Liqing Zhang



Zhangxuan Gu is with MOE Key Lab of Artificial Intelligence, Department of
Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai,
China (email: zhangxgu@126.com).


Li Niu is with MOE Key Lab of Artificial Intelligence, Department of
Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai,
China (email: ustcnewly@sjtu.edu.cn).


Haohua Zhao is with MOE Key Lab of Artificial Intelligence, Department of
Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai,
China (email: haoh.zhao@sjtu.edu.cn).


Liqing Zhang is with MOE Key Lab of Artificial Intelligence, Department of
Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai,
China (email: zhang-lq@cs.sjtu.edu.cn).
    ...
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================







Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depth-aware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on two benchmark datasets, and even outperforms the methods which need depth input during testing.


Semantic segmentation, hard sample mining, privileged information, RGBD semantic segmentation.

































§ INTRODUCTION

























Semantic segmentation is a fundamental problem with the goal to classify each pixel in an image, which has a wide range of real-world applications including autonomous driving, visual scene understanding, and image editing. Recently, abundant RGB based semantic segmentation methods <cit.> have been developed and achieved excellent performances in given datasets. However, these methods still exhibit clear limitations due to the hard pixels induced by complicated scenes, poor lighting condition, confusing object appearances, and so on.




To address the issues in RGB semantic segmentation, many attempts <cit.> have been made to exploit depth information for semantic segmentation, because the depth map can provide complementary 3D information as shown in Fig. <ref>(a), which may be helpful for the segmentation task. For example, <cit.> presents a segmentation network with depth-adaptive multiscale (DaM) convolution layers, while other methods like LS-DeconvNet <cit.>, CFN <cit.>, ACNet <cit.>, RDFNet <cit.> proposed to fuse RGB features and depth features in different ways. More recently, <cit.> utilizes depth to segment and track objects for crowds. These methods use extra depth information for both training and test images, which falls into the scope of multi-view learning. Specifically, each training or test sample consists of two views, i.e., RGB and depth. The experimental results showed that these methods benefit the segmentation tasks from using depth information.








However, the above methods require depth information of test images during testing, which is often inaccessible in the real-world applications. Therefore,
some other semantic segmentation works attempted to use depth information only in the training stage, in accordance with Learning Using Privileged Information (LUPI) <cit.>, in which privileged information means the information only available for training samples but unavailable for test samples. It has been proved in <cit.> that only using privileged information of training samples can still help to learn a more robust model. We refer to the semantic segmentation task using depth as privileged information as depth privileged semantic segmentation. We compare three settings: RGB semantic segmentation, RGBD semantic segmentation and depth privileged semantic segmentation in Fig. <ref>(b). Several multi-task methods like LW-RefineNet <cit.>, TRL <cit.>, and PAP <cit.>, which simultaneously handle segmentation task and depth prediction task could be used for depth privileged segmentation. 

One severe drawback of the above depth-privileged segmentation methods is that they treat all pixels in an image equally and exhibit clear limitations due to the hard pixels induced by complicated scenes, confusing object appearances, and etc. In fact, one research line of LUPI for image classification is using privileged information to mine hard samples and assigning larger weights on the training losses of these samples <cit.>. Hard sample mining has also been proved effective in previous research works <cit.> not limited to LUPI. However, no previous works attempt to fuse RGB and depth information to mine hard pixels for the semantic segmentation task.

In this work, we propose a novel network module, i.e., Loss Weight Module (LWM), to mine the hard pixels in semantic segmentation. Measurement of hard pixels is an open and challenging problem. To this end, our proposed LWM employs two hard pixel measurements: Depth Prediction Error (DPE) and Depth-aware Segmentation Error (DSE). The motivation for hard pixel measurements can be explained as follows. By considering segmentation and depth prediction as two joint tasks, we conjecture that segmentation error and depth prediction error are positively correlated. For example, if two neighboring objects from two different categories have a huge depth gap, mispredicted depth may lead to the failure of delineating the depth boundary between them, resulting in the segmentation error of these two objects. In other words, inaccurate depth prediction of these two regions, which leads to the failure of detecting depth boundary, might be highly correlated with the segmentation error. As illustrated in the first row in Fig. <ref>, a chair is placed next to a glass table with a large depth gap. However, the chair is misclassified as the table, probably because they are visually similar. Therefore, we conjecture that Depth Prediction Error (DPE) could be used as a measurement of hard pixels, which is also quantitatively verified in Section <ref>.



Nevertheless, in some circumstances, DPE may be less effective in identifying hard pixels. For example, in a depth-aware local region (a local region with similar depth), if the categories of different subregions are confused with each other due to similar visual appearance, this region becomes a hard region. In this case, accurate depth prediction cannot help separate different subregions, and thus DPE may not be correlated with the difficulty of segmenting this hard region.
As illustrated in the third row in Fig. <ref>, neighboring pillow and cushion with similar depth form a hard region, in which the cushion is misclassified as a pillow although the depth prediction is correct.

Therefore, we introduce another measurement of hard pixels, that is, Depth-aware Segmentation Error (DSE), to pay more attention to those hard depth-aware local regions. Concretely, we divide the holistic image into depth-aware local regions and compute the segmentation error rate within each region to identify those hard regions.

After identifying the hard pixels, our Loss Weight Module (LWM) outputs a loss weight map by fusing the DPE and DSE maps, in which we investigate several fusion strategies. Then, the loss weight map is used to weight pixel-wise training losses to learn a more robust model. Moreover, our LWM can be easily applied to multi-scale outputs in different network layers for mining the hard pixels on different scales, which can fully exploit multi-scale information. Except for training with easy and hard pixels at the same time, we also explore another training strategy with the proposed LWM, i.e., starting with easy pixels and gradually including hard ones. This training strategy is related to curriculum learning <cit.> and capable of further improving the segmentation performance. Our main contributions are summarized as:


  * This is the first work to mine hard pixels in depth privileged semantic segmentation.

  * We propose a novel Loss Weight Module (LWM), which fuses RGB and depth information to mine hard pixels based on Depth Prediction Error (DPE) and Depth-aware Segmentation Error (DSE).

  * We apply our LWM to multi-scale outputs for mining hard pixels on different scales. Based on LWM, we also explore a training strategy similar to curriculum learning to gradually include hard pixels in the training stage.

  * Extensive experiments demonstrate that our method can achieve the state-of-the-art performances on SUNRGBD and NYUDv2 datasets, and even surpass the methods which require input depth information during testing.










§ RELATED WORKS








In this section, we will conduct a comprehensive review of existing RGB semantic segmentation methods, RGBD semantic segmentation methods, and depth privileged semantic segmentation methods. We will also discuss other related works on learning using privileged information and hard sample mining.



 §.§ RGB semantic segmentation

Deep learning methods <cit.> have shown impressive results in RGB semantic segmentation. Most of them are based on the Fully Convolutional Networks (FCN) <cit.>. The extension based on FCN can be grouped into the following two directions: capturing the contextual information at multiple scales and designing more sophisticated decoder. In the first direction, some works combined feature maps generated by different dilated convolutions and pooling operations. For example, PSPNet <cit.> adopts Spatial Pyramid Pooling which pools the feature maps into different sizes for detecting objects of different scales. Deeplabv3 <cit.> proposed an Atrous Spatial Pyramid Pooling by using dilated convolutions to keep the large receptive field.
In the second direction, some works <cit.> proposed to construct better decoder modules to fuse mid-level and high-level features. For example, RefineNet <cit.> is a multi-path refinement network which fuses features at multiple levels of the encoder and decoder. 
However, all the above methods are based on RGB images, while our approach tends to use a depth map as privileged information to mine hard pixels to boost the performance of semantic segmentation.










 §.§ RGBD semantic segmentation



Different from RGB segmentation, recent RGBD segmentation methods exploit depth information under the framework of multi-view or multi-task learning.
By using multi-view learning, many works <cit.> combine depth information and RGB information in various ways. 
For example, LS-DeconvNet <cit.> proposed deconvolution decoder and gated fusion module for RGBD segmentation. Another interesting method ACNet <cit.> used Attention Complementary Module to extract and fuse weighted features from RGB and depth branches. More recently, CFN <cit.> is a neural network with multiple branches of the context-aware receptive field to learn better contextual information, while RDFNet <cit.> captures multi-level RGBD features by using the proposed multi-modal feature fusion blocks, which combine residual RGB and depth feature to fully exploit the depth information.



Alternatively, some works <cit.> adopt a multi-task learning framework, which predicts depth map and segmentation mask in one unified network.
Although our method is also under the multi-task learning framework, our focus is using depth to mine hard pixels, which contributes more to the segmentation task than merely multi-task learning (see Section <ref>).






 §.§ Learning Using Privileged Information

The concept of Learning Using Privileged Information (LUPI) was first introduced in <cit.> for classification, in which privileged information stands for the information which is only available in the training stage but not available in the testing stage.


Besides for classification task <cit.>, privileged information has also been used for clustering <cit.>, verification <cit.>, hashing <cit.>, random forest <cit.>, and etc.


Recently, privileged information has also been utilized in deep learning methods <cit.>, which aim to distill knowledge from privileged information or use privileged information to control the training process.


More recently, SPIGAN <cit.> and DADA <cit.> proposed to use depth as privileged information in the semantic segmentation task. However, their main contribution is exploiting depth information to assist domain adaptation, which adapts the synthetic image domain to the real image domain. So the motivation and solution of their methods are intrinsically different from ours. Some RGBD segmentation methods <cit.> under the multi-task learning framework are able to utilize depth as privileged information. 
In contrast with previous depth privileged segmentation methods, this paper is the first work to use depth as privileged information to mine hard pixels for the semantic segmentation task.




 §.§ Hard samples mining

Abundant previous works have shown that assigning higher weights to the losses of hard samples could help learn a better classifier. Most of them only use one data modality and mine hard samples based on classification score/loss, such as OHEM <cit.>, Loss Max-Pooling (LMP) <cit.>, and Focal Loss (FL) <cit.>. In the field of LUPI, a few methods like <cit.> use extra modality (e.g., depth) as privileged information to mine hard samples for classification task based on classification loss. However, no previous works have attempted to fuse two modalities to mine hard pixels for the semantic segmentation task.



§ METHODOLOGY

In this section, we first introduce a multi-task learning framework and then propose our Loss Weight Module (LWM). For ease of implementation, given a training image 𝐈, we denote its predicted segmentation mask and ground-truth mask as 𝐒 and 𝐒^* respectively. Similarly, we use 𝐃 and 𝐃^* to denote its predicted depth map and ground-truth depth map, respectively.




 §.§ Multi-Task Learning


The multi-task learning framework is based on RefineNet <cit.>, which has achieved compelling results in the semantic segmentation task recently. As illustrated in Fig. <ref>, the encoder consists of four pretrained ResNet <cit.> blocks, and the decoder consists of four RefineNet blocks. We then use bilinear interpolation to upsample different-scale decoder feature maps to the same size as 640×480 and concatenate them in the segmentation decoders. Finally, one 3×3 convolutional layer is applied to the concatenated feature map to predict the output segmentation mask. For traditional RGB segmentation, we adopt cross-entropy segmentation loss as follows:

    L_s=-1/n∑_i=1^n∑_j=1^cy_ijlog(p_ij),

where n is the number of pixels in image 𝐈, c is the number of categories, y_ij is the ground-truth binary label to indicate whether the i-th pixel belongs to the j-th category, and p_ij is the predicted probability of the i-th pixel for the j-th category.

Inspired by <cit.>, we extend another decoder branch to predict depth map, since depth estimation and semantic segmentation are two highly related tasks. To encourage the information to be shared between two decoder branches, we propagate the output feature maps of depth decoder blocks to the output feature maps of segmentation decoder blocks by pixel-wise summation.





For the training loss of depth prediction, inspired by <cit.>, we compute the mean square error (MSE) of the differences between predicted and ground-truth logarithm of depths. Concretely, given the predicted log-depth map log𝐃 and the ground-truth log-depth map log𝐃^*, we use logD_i and logD_i^* to denote the value indexed by pixel position i in log𝐃 and log𝐃^* respectively. D_i and D_i^* are in the range of [1, 256], so log𝐃 and log𝐃^* are in the range of [0, log(256)]. By denoting R_i = logD_i-logD_i^*, the depth prediction loss can be written as





    L_d= 1/n∑_i=1^n(logD_i-logD_i^*)^2= 1/n∑_i=1^nR_i^2.
















By combining segmentation and depth prediction loss, the total loss of multi-task learning is L_s+L_d similar to that in <cit.>. The multi-task learning framework of jointly predicting segmentation mask and depth map is not new for semantic segmentation, but this is not the focus of this paper. Our major contribution is a novel Loss Weight Module (LWM) based on two measurements of hard pixels, which will be introduced in Section <ref>. We will also explore applying LWM to multi-scale side outputs in Section <ref> and a progressive training strategy based on mined hard pixels in Section <ref>.




 §.§ Loss Weight Module (LWM)

We tend to identify hard pixels with privileged depth information and assign larger weights to the segmentation losses of hard pixels. The weight map applied to segmentation loss can be deemed as an attention map, which pays more attention to hard pixels to learn a more robust model. Therefore, we refer to our module as Loss Weight Module (LWM), which utilizes two depth-related measurements of hard pixels, i.e., Depth Prediction Error (DPE) and Depth-aware Segmentation Error (DSE), to produce the loss weight map as shown in Fig. <ref>. Moreover, this module can be easily integrated into any segmentation network.




  §.§.§ Depth Prediction Error (DPE)
 
As mentioned in Section <ref> and shown in Fig. <ref>, accurate detection of the depth boundary between two regions from different categories might be highly correlated with successful segmentation of these two regions. In other words, inaccurate depth prediction of these two regions could cause the failure of detecting depth boundary, which may imply the difficulty of segmenting these two regions.

Thus, we use Depth Prediction Error (DPE) map: 
    𝐑 = |log𝐃-log𝐃^*|,
 as a measurement of hard pixels. We normalize DPE within each image to [0,1] by dividing the max value. The correlation between depth prediction error and segmentation error is justified by experiments in Section <ref>. Instead of directly using segmentation error as the measurement of hard pixels, the depth prediction error could offer extra guidance after employing additional depth information. Besides, we will introduce another measurement based on segmentation error in the following subsection.


























  §.§.§ Depth-aware Segmentation Error (DSE)
 
As discussed in Section <ref>, when a local region contains different confusing categories within the same ground-truth depth bin, DPE may be less effective in measuring the segmentation difficulty. Thus, we propose another measurement named Depth-aware Segmentation Error (DSE).

For a better explanation, we define a local region in the same depth bin as a Depth-aware Local Region (DLR). In some DLRs, the categories of different subregions could be easily confused with each other due to their visual resemblance. We refer to such DLRs with confusing subregions as hard regions. One example is provided in Fig. <ref>. The cushion (subregion A) and the pillow (subregion B) have similar depth, and the region AB forms a DLR. The cushion is misclassified as a pillow by RefineNet because the cushion looks very similar to the pillow. Thus, we claim region AB as a hard region. In some other cases, one subregion in a hard region could also be misclassified into an arbitrary wrong category instead of the category of neighboring subregions.
Without loss of generality, we calculate the segmentation error rate in the DLRs as our second measurement DSE. Note that for DSE, we only use the ground-truth depth map 𝐃^* to obtain DLRs without using the predicted depth map 𝐃.

To calculate DSE, we need to locate depth-aware local regions (DLRs) first. For ease of implementation, we divide an image into two sets of regions based on two criteria. On one hand, we uniformly divide image 𝐈 into w̃×h̃ cells, leading to one set of regions ℙ = ⋃_k=1^m {P_k} with m =w̃×h̃. On the other hand, we divide the range of ground-truth depth values 𝐃^* (i.e., [1, 256]) into ⌈ 256/d ⌉ depth bins with the size of each bin being d (we empirically set d=10 in our experiments, see Section <ref>), leading to another set of regions 𝔹 = ⋃_t=1^⌈ 256/d ⌉{B_t}, in which each region contains the pixels with depth values 𝐃^* belonging to each depth bin. With two sets of regions ℙ and 𝔹, we calculate the intersection between P_k and B_t, yielding a DLR P_k ∩ B_t. Then, if P_k ∩ B_t ≠∅, we can simply calculate the segmentation error rate within this DLR as

    Z_P_k,B_t=(∑_i∈ P_k ∩ B_tΔ_i)/|P_k ∩ B_t|,

where Δ_i=1 if pixel i is misclassified and 0 otherwise. We use Z_P_k,B_t to measure the segmentation difficulty of the region P_k ∩ B_t and the regions with large 𝐙 are regarded as hard regions. Taking a further step, we use 𝐙 to denote DSE map with the entries in region P_k ∩ B_t defined in (<ref>).

Instead of using segmentation error rate, an alternative choice is calculating weights based on the averaged segmentation loss within each DLR, but that may require a calibration function with additional hyper-parameter like <cit.>. So for simplicity, we use segmentation error rates as weights, and the obtained weights are naturally within the range [0,1], which is uniform with our DPE map.

It is worth mentioning that for a hard depth-aware local region (DLR), we assign the same weight to all pixels instead of only misclassified pixels, which can be explained as follows.
Recall that in Fig. <ref>, subregion A from category C_a is misclassified as the category C_b . If we only assign higher weights on subregion A, the classifier might be biased towards category C_a, which means that a subregion from category C_b bearing visual resemblance with subregion A may be prone to be misclassified as category C_a. Therefore, we assume both subregions A and B as the hard region to better distinguish between category C_a and category C_b.

One remaining problem is how to determine the number of cells m. Two special cases are m=1 and m=w× h, in which w and h are the width and height of image 𝐈. When m=1, the image is divided into depth-aware global regions, in which multiple distant disjoint subregions in the same depth bin could be grouped into one region.
Considering that two distant disjoint subregions in the same depth-aware global region, the segmentation difficulty of one subregion is less affected by the other distant region, so it may be unhelpful to consider them as one hard region. When m=w× h, each cell only contains one pixel. In this case, DSE is equivalent to identifying misclassified pixels instead of hard regions. So depth information is not used, and DSE becomes similar to previous hard pixel mining methods <cit.> merely based on segmentation error.
In our experiments, we set m=w̅×h̅=8× 8 by cross validation, which lies between the above two extreme cases (see Section <ref>). 






























  §.§.§ Fusion of DPE and DSE
 
Since DPE map 𝐑 and DSE map 𝐙 are in the same scale [0,1], we can use a function F(·) to fuse two maps, yielding the integrated loss weight map 𝐌=F(𝐑,𝐙). When applying the obtained loss weight map 𝐌 to segmentation loss map, we can arrive at our weighted segmentation loss:

    L_h=-1/n∑_i=1^n∑_j=1^cM_iy_ijlog(p_ij),

in which M_i is the entry for pixel i in 𝐌.
Intuitively, L_h aims to place more emphasis on hard pixels to learn a better segmentation network. For the fusion operator F(·), we experiment with three different choices: pixel-wise summation, pixel-wise product, and pixel-wise max, among which pixel-wise summation achieves the best performance (see Section <ref>).







 §.§ LWM on Multi-Scale Side Outputs
 
It is commonly acknowledged that the output feature maps of different decoder blocks contain the information of different granularities, which make different contributions to the semantic segmentation task. To take full advantage of our LWM, we apply LWM to multiple decoder blocks to mine hard pixels based on the information of different granularities, as shown in Fig. <ref>.

Specifically, we add a 1× 1 convolution layer to each of the four decoder blocks for predicting multi-scale side outputs (i.e., segmentation masks). Then, we apply LWM loss on four side outputs to better mine the hard pixels and provide better supervision. We denote the weighted segmentation losses on multi-scale side outputs as {L_hk|_k=1^4}. We totally produce four side outputs (see Fig. <ref>) and L_hk means employing L_h in (<ref>) on the k-th side output.


So far, our total training loss for Hard Pixels Mining can be written as

    L_HPM =  L_s + L_d + α L_h+β∑_k=1^4L_hk ,

where α and β are two trade-off parameters. Note that we only use the losses based on side outputs during training. In the testing stage, we simply use the final predicted segmentation mask for evaluation.



 §.§ Curriculum Learning Based on LWM

Based on the loss function in (<ref>), with identified hard pixels, one intuitive training strategy is employing L_h and L_hk on all pixels. An alternative strategy is that L_h (resp., L_hk) is first employed on easy pixels and then extended to hard pixels, similar to curriculum learning <cit.>.

Curriculum learning has achieved great success in a variety of applications <cit.>, in which the training process starts with easy training samples and gradually includes hard ones. The key problem in curriculum learning is how to define easy and hard training samples, which is solved by our LWM. One popular curriculum learning approach is Self-Paced Learning (SPL) <cit.>, in which the training samples with small training losses are regarded as easy training samples. Specifically, SPL aims to learn the model parameters θ and the binary indicator vector 𝐯 = [v_1,…, v_N]^T indicating easy training samples, in which N is the total number of training samples. The objective function of SPL can be written as

    min_θ, 𝐯∈[0,1]^N∑_i=1^N v_iL(𝐱_i;θ)-η∑_i=1^N v_i,

in which L(𝐱_i;θ) is the training loss of the i-th training sample 𝐱_i and η is the learning pace. The problem in (<ref>) can be solved by updating θ and 𝐯 alternatingly.
When fixing θ, we use L_i to store L(𝐱_i;θ) and v_i can be updated as δ(L_i<η), which is equal to 1 if L_i<η and 0 otherwise. Then, model parameters θ can be updated by solving

    min_θ∑_i=1^N δ(L_i<η) L(𝐱_i;θ),

which only preserves the losses of easy samples with L_i<η.
Analogous to (<ref>), we extend L_h (resp., L_hk) to L̂_h (resp., L̂_hk) by filtering out the hard pixels. Next, we take L̂_h as an example and L̂_hk can be obtained in a similar way. In the current epoch e, L̂_h is formulated as

    L̂_h = -1/n∑_i=1^nδ(M_i<η_e) (∑_j=1^cM_iy_ijlog(p_ij)),

where the learning pace η_e is defined as u^1+e×(u^2-u^1)/E with E being the total number of epochs, u^1 (resp., u^2) is the median (resp., maximum) of {M_1,…,M_n}, δ(M_i<η_e)=1 if M_i<η_e and 0 otherwise. Intuitively, we first choose the easy pixels (M_i<η_e) for calculating the weighted segmentation loss, and then gradually add hard pixels as the number of epochs increases. Thus, the loss function based on curriculum learning can be written as

    L_T =  L_s + L_d + αL̂_h+β∑_k=1^4L̂_hk.


The training strategy based on (<ref>) can be explained as follows. At first, only easy pixels are selected for training with L̂_h (resp., L̂_hk). Then, more and more hard pixels will contribute to training with L̂_h (resp., L̂_hk). Note that when setting η_e as a sufficiently large constant, L_T will reduce to L_HPM.




§ EXPERIMENTS


In this section, we compare our method with our special cases and state-of-the-art baselines on two benchmark datasets NYUDv2 and SUNRGBD. We also provide comprehensive ablation studies, hyper-parameter analyses, and in-depth qualitative analyses.







 §.§ Datasets and Implementation Details


We evaluate our method on two benchmark datasets: the popular NYUDv2 <cit.> dataset and a large-scale SUNRGBD <cit.> dataset. The NYUDv2 dataset contains 1449 densely labeled pairs of RGB and depth images. Following the standard train/test split, we use 795 training images and 654 test images. We evaluate our network for 40 categories by using the labels provided by <cit.>. The SUNRGBD dataset consists of 10335 pairs of RGB and depth images. We use the standard split of 5285 for training and 5050 for testing with 37 categories.


Our method is built upon RefineNet <cit.> with ResNet152 as backbone, unless specified otherwise. In the training stage, we set the momentum as 0.9 and weight decay as 5× 10^-4. Following previous paper <cit.>, we train RGB and depth branches together with the same learning rate 10^-3. The learning rate is divided by 10 when the loss stops decreasing. The number of training epochs for NYUDv2 and SUNRGBD are 120 and 60 respectively. The size of the training batch is 8, and the input image is resized to 640× 480.

We use random horizontal flipping, cropping, and image color augmentation. We report results based on three evaluation metrics, i.e., pixel accuracy, mean accuracy and mean Intersection over Union (IoU). We set α=1,β=0.1 in (<ref>) according to cross validation.



 §.§ Correlation Analysis


As claimed in Section <ref>, depth prediction error and segmentation error might be positively correlated. We provide a quantitative analysis of this hypothesis for both multi-task learning baseline (i.e., L_s+L_d) and our method based on Depth Prediction Error (DPE) (i.e., L_s+L_d+L_h(𝐑)), in which “L_h(𝐑)" means we only use DPE map in L_h. Specifically, for test images in NYUDv2, we divide the logarithm of depth prediction error into 10 bins and calculate the segmentation error rate of the pixels in each bin. The segmentation errors in all bins for two methods are plotted in Fig. <ref>.

One observation is that the segmentation error rate grows higher as the depth prediction error increases, which proves that segmentation error and depth prediction error are positively correlated. Another observation is that our method beats multi-task learning in each bin, especially when the depth prediction error is large, which demonstrates the effectiveness of DPE.

We also compare our method (L_s+L_d+L_h (𝐑)) with hard pixel mining baselines based on segmentation error in Section <ref>, which shows that L_h (𝐑) performs more favorably probably because additional depth information provides extra valuable supervision.


















 §.§ Comparison with the State-of-the-art


We divide all baselines into three groups: 1) S group: RGB segmentation baselines without using depth information; 2) D group: RGBD segmentation baselines using depth information for both training and test images; 3) P group: depth privileged segmentation baselines using depth information only for training images. For a fair comparison, we also list the backbones of all methods. We report two versions of our method with backbone ResNet50 and ResNet152 in Table <ref>.

For the S group, we compare with RefineNet <cit.>, PSPNet <cit.>, and DeepLabv3 <cit.>.
As our backbone network RefineNet only releases MATLAB code, we implement it using python and obtain better results than those reported in<cit.>.

For the D group, we compare with FCN <cit.>, LS-DeconvNet <cit.>, Wang et al. <cit.>, ACNet <cit.>, CFN <cit.>, and RDFNet <cit.>. Among them, RDFNet is also built upon RefineNet and closely related to our method, so we additionally report our reproduced results.

For the P group, depth information is not available in the testing stage. RDFNet is originally an RGBD segmentation method (D group), and we adapt it to use only RGB images during testing. In particular, we remove its depth input and retrain the remaining network for several epochs. We also try feeding both RGB images and pseudo depth maps predicted by our method to RDFNet during testing, which is marked with “PD". LW-RefineNet <cit.>, TRL <cit.>, and PAP <cit.> are multi-task learning method which can naturally use depth as privileged information. Among them, PAP <cit.> used extra Surface Norm (SN) annotations as the third branch of multi-task learning during training, and thus achieves the best pixel-wise accuracy on SUNRGBD. For other methods from the D group, some cannot be easily adapted to depth privileged method while some adapted ones are much worse than their original version, so we omit their results here.


Experimental results are summarized in Table <ref>. We observe that RefineNet <cit.> achieves very competitive results within the S group. We also observe that recent methods CFN <cit.>, ACNet <cit.>, and RDFNet <cit.> from D group are generally better than the methods in S group, which proves that depth information is beneficial for semantic segmentation. Our method achieves significant improvements over our backbone network RefineNet, i.e., 4.17% and 4.92% IoU improvement on SUNRGBD and NYUDv2 datasets respectively. Besides, our method generally beats the baselines in the P group. Although PAP <cit.>(+SN) uses extra Surface Norm (SN) annotations during training, our method is still better than <cit.> except pixel accuracy on SUNRGBD. Moreover, our method also generally outperforms the baselines in the D group, which use more information than our method during testing.
















































 §.§ Ablation Studies



To validate the effectiveness of each component in our method, by taking NYUDv2 as an example, we report the results of our special cases in Table <ref>. The first four rows are baselines without using our LWM. Specifically, L_s is RefineNet152 in Table <ref> which only uses segmentation loss. L_s+L_d is a multi-task learning baseline. We futher try using multi-scale (MS) training (calculate losses on multi-scale side outputs) or curriculum learning (CL) based on L_s+L_d. In the next three rows, we report the results of L_s+L_d+L_h using only DPE (𝐑), only DSE (𝐙), or both in L_h. Finally, we report the results of L_HPM and L_T by adding MS and CL to L_s+L_d+L_h step by step.




















From Table <ref>, we observe that the improvements of adding multi-scale (MS) and curriculum learning (CL) to multi-task baseline L_s+L_d is minor. Moreover, the largest performance gain is brought by our LWM (L_s+L_d+L_h v.s. L_s+L_d), about 3.06%. Our proposed method is more effective compared with multi-task learning (L_s+L_d v.s. L_s), which only lead to 0.66% IoU improvements. It can also be seen that DPE (𝐑) and DSE (𝐙) are both useful, while 𝐙 is more effective than 𝐑. Besides, using both DPE and DSE is better than using DPE or DSE alone, which reflects that our proposed two measurements are complementary with each other and can jointly contribute to the final performance. Finally, multi-scale L_hk and curriculum learning can also help achieve better results, which demonstrates the effectiveness of applying LWM to multi-scale side outputs and our designed training strategy.







 §.§ Comparison with hard pixel mining methods
 

By taking NYUDv2 as an example, we further list the results of applying OHEM <cit.>, focal loss (FL) <cit.>, and LMP <cit.> to multi-task learning baseline L_s+L_d in Table <ref> for comparison because they all use only RGB images or segmentation losses to mine the hard training pixels. Specifically, LMP assigns larger weights on pixels with larger losses by proposed max-pooling function, while OHEM only calculates the losses of pixels which are larger than a threshold and ignore the remaining ones. Recently, focal loss down-weights the loss assigned to well-segmented pixels. For our method, we copy three results from Table <ref>: L_s+L_d+L_h(𝐑), L_s+L_d+L_h(𝐙), and L_s+L_d+L_h.


From Table <ref>, it can be seen that our method with single-scale LWM is more effective than merely mining hard pixels based on RGB information (L_s+L_d+L_h v.s. L_s+L_d+OHEM/FL/LMP), which indicates the advantage of fusing RGB and depth to mine hard pixels. Moreover, when only using 𝐑 or 𝐙 for L_h, our method is still better than other hard pixel mining baselines.



 §.§ Hyper-parameters Analyses

In this section, we investigate the impact of different hyper-parameters in our method by taking NYUDv2 dataset as an example. We conduct experiments using our method (L_HPM) without curriculum learning, considering that L_HPM is our main contribution.
Recall that we have two hyper-parameters α and β in (<ref>).
We plot the performance variance when changing the hyper-parameters α,β in Fig. <ref>. When varying α (resp., β) within the range [0.01,100], and the obtained IoU (%) of our method (L_HPM) is in the range of [50.23, 50.93] (resp., [49.78, 50.93]) as shown in Fig. <ref>, which demonstrates the effectiveness of our method when setting α and β in a reasonable range.














In Section <ref>, we mentioned that different fusion functions F(·) can be used to fuse DPE map and DSE map. Here, we explore three simple strategies: pixel-wise max, pixel-wise product, and pixel-wise summation. The results are summarized in Table <ref>, which shows that pixel-wise summation achieves the best result.

In Section <ref>, we have bin size d when grouping depth values into bins and the number of cells m=w̅×h̅ when dividing the whole image into cells. The results using different d are reported in Table <ref>, which indicates that our method is insensitive to the bin size d.
Finally, we report the results using different m in Table <ref>. It can be seen that the results with m=8× 8 are better than those with m=1×1 or m=w× h, which are two extreme cases as discussed in Section <ref>. The results verify that it would be more beneficial to assign higher weights on hard local regions instead of hard global regions (8× 8 v.s. 1× 1) or only misclassified pixels (8× 8 v.s. w× h).




 §.§ Base Network Analyses

Note that our method is built upon RefineNet as the base network for segmentation.
However, our hard pixel mining method only requires additional depth prediction branch and thus can be adapted to other base networks with minor modifications. In this section, we attempt to build our method on other RGB segmentation networks like Deeplabv3 and PSPNet by adding a 1× 1 conv layer after their last pooling layer to predict depth. By using three different base networks with backbone ResNet152, we report the results of multi-task learning baseline L_s+L_d, our method (L_s+L_d+L_h) with single-scale LWM (see Table <ref>), and a competitive hard pixel mining baseline L_s+L_d+FL (see Table <ref>) on NYUDv2.

The performances on NYUDv2 are shown in Table <ref>, from which we can observe the effectiveness of our method (L_s+L_d+L_h) compared with the multi-task baseline and the hard pixel mining baseline when using different base networks. These results indicate the good generalization ability of our method.







 §.§ Visualization of Loss Weight Map


By taking two images from NYUDv2 dataset as examples, we visualize the loss weight maps obtained by our method in Fig. <ref>, in which the brighter pixels correspond to larger weights and indicate harder pixels. As shown in Fig. <ref> (a), we list the input images, Depth Prediction Error (DPE) maps (𝐑), Depth-aware Segmentation Error (DSE) maps (𝐙), and the fused loss weight maps 𝐌 using pixel-wise summation. It can be seen that DPE map and DSE map have certain overlap because depth prediction error is positively correlated with segmentation error as shown in Section <ref>. However, DPE focus on the hard pixels with huge depth gap while DSE could identify some hard regions not specified by DPE, which proves that DPE and DSE are complementary with each other and jointly lead to the performance gain.

Moreover, to visualize the procedure of curriculum learning, we exhibit the loss weight maps δ(M_i<η_e)M_i (see (<ref>)) in different epochs (e.g., 30, 60, 90, 120) from a total of 120 training epochs in Fig. <ref> (b). As the training epoch increases, more bright pixels appear in the loss weight map, which means that more hard pixels participate in the training procedure and contribute to learning a more robust model.







 §.§ Qualitative Analyses


In this section, we provide some visualization results of the baselines and our method on the NYUDv2 test set. To fully corroborate the effectiveness of our method, we visualize both the comparison with state-of-the-art methods in Fig. <ref> and the comparison with our special cases in Fig. <ref>.

In Fig. <ref>, we visualize the segmentation results of our base network RefineNet (L_s), the results after using DPE (L_s+L_d+L_h(𝐑)), the results after using DSE (L_s+L_d+L_h(𝐙)), and the results of our full method. In the “DPE" column, it can be seen that our method is adept at segmenting some objects with sharp depth boundary (a,b). For example, our method can successfully segment the upper part of the chair back in (a), the table under the TV set in (b), which is attributed to our DPE map. In the “DSE" column, our method also demonstrates excellent ability in classifying confusing objects with similar visual appearances in the same depth bin (c,d,e), which are misclassified as incorrect categories by baseline RefineNet. For example, our method can successfully segment the sink in (c), the curtain in (d), and the cabinet in (e), which is credited to our DSE map. In the “Ours" column, we can also observe that by combining DPE and DSE to mine the hard pixels, our model achieves significantly better results.

In Fig. <ref>, we visualize the segmentation results of different state-of-the-art methods. The results in Fig. <ref> further show that our method significantly outperforms other RGB and even RGBD segmentation methods, especially on depth-related regions such as mirror, armoire, and desk.






























§ CONCLUSIONS

In this work, we have proposed to mine hard pixels by fusing RGB information and depth information in depth privileged semantic segmentation task. In particular, we have designed a novel Loss Weight Module (LWM), which fuses two measurements of hard pixels (depth prediction error and depth-aware segmentation error) and outputs a loss weight map for weighting pixel-wise training losses.
Moreover, we have applied LWM to multi-scale side outputs and explored curriculum learning strategy based on LWM. Comprehensive quantitative and qualitative results on two benchmark datasets have demonstrated the superiority of our method.






















































































































































IEEEtran
































































