abstract: Deep convolutional neural networks have become a key element in the recent breakthrough of salient object detection. However, existing CNN-based methods are based on either patch-wise (region-wise) training and inference or fully convolutional networks. Methods in the former category are generally time-consuming due to severe storage and computational redundancies among overlapping patches. To overcome this deficiency, methods in the second category attempt to directly map a raw input image to a predicted dense saliency map in a single network forward pass. Though being very efficient, it is arduous for these methods to detect salient objects of different scales or salient regions with weak semantic information. In this paper, we develop hybrid contrast-oriented deep neural networks to overcome the aforementioned limitations. Each of our deep networks is composed of two complementary components, including a fully convolutional stream for dense prediction and a segment-level spatial pooling stream for sparse saliency inference. We further propose an attentional module that learns weight maps for fusing the two saliency predictions from these two streams. A tailored alternate scheme is designed to train these deep networks by fine-tuning pre-trained baseline models. Finally, a customized fully connected CRF model incorporating a salient contour feature embedding can be optionally applied as a post-processing step to improve spatial coherence and contour positioning in the fused result from these two streams. Extensive experiments on six benchmark datasets demonstrate that our proposed model can significantly outperform the state of the art in terms of all popular evaluation metrics.
author: Guanbin Li and Yizhou Yu
[^1] [^2] [^3]
[^4]
bibliography: dcl_saliency.bib
title: Contrast-Oriented Deep Neural Networks for Salient Object Detection

Li and Yu: Contrast-Oriented Deep Neural Networks for Salient Object Detection

# Introduction

Visual saliency detection aims to locate the most conspicuous regions in images according to the human visual system and has recently received increasing research interest. Image saliency detection is traditionally approached in the form of either eye-fixation prediction or salient object detection. The former focuses on the natural mechanism of visual attention and aims at accurately predicting human eye attended image locations. However, previous research has pointed out that salient object detection, which is more concerned with the integrity of the predicted object regions, is more conducive to a series of computer vision tasks including semantic segmentation , object localization and detection , content-aware image editing , visual tracking  and person re-identification . Although numerous valuable models have been proposed, salient object detection remains challenging due to a variety of complex factors in real-world scenarios.

Perceptual studies  have shown that visual contrast is the key factor that affects visual saliency. A series of conventional salient object detection algorithms based on local or global contrast modeling  have been successfully proposed. In previous research efforts, visual contrast modeling is generally focused on the differences among various handcrafted low-level features and coupled with heuristic saliency priors. Although handcrafted features tend to perform well in simple cases, they are not robust enough for more challenging scenarios. For example, it is hard for local contrast models to accurately segment out large homogeneous regions inside salient objects while global contrast information may fail to handle images with cluttered background. Although there exist machine learning based algorithms for salient object detection , they are basically focused on integrating various handcrafted features  or merging multiple saliency maps computed by different methods .

Recently, deep convolutional neural networks have been widely used in salient object detection  because of their powerful feature representations and have achieved substantially better performance than traditional methods. Methods based on deep convolutional neural networks can be roughly divided into two categories. Methods in the first category generally perform patch-wise (or region-wise) training and inference. Specifically, an image is first divided into a set of regions or patches, and deep CNN based regression or classification models are then trained to independently map each image patch or region to a saliency score or a binary class label (salient or non-salient). However, this results in serious storage and computational redundancies, making training and testing very time-consuming. For example, training a patch-oriented CNN model takes over two GPU days while requiring hundreds of megabytes of storage to save deep features extracted from one single image. Inspired by the latest trends of developing fully convolutional neural networks for pixel-level image understanding problems , methods in the second category train end-to-end models that directly map an input image of arbitrary size to a saliency map with the same size, performing dense feedforward computation and backpropagation over the entire image. This type of methods have rapidly become the cornerstone of this field as they not only achieve very favorable performance but also are very efficient. However, it is still arduous for these methods to detect salient objects of different scales or salient regions with weak semantic information. Moreover, pixel-level correlation is typically not considered in such fully convolutional networks (FCNs), which usually give rise to incomplete salient regions with blurry contours.

In this work, we develop hybrid contrast-oriented deep neural networks to overcome the aforementioned limitations of two types of contemporary CNN-based salient object detection methods. Our deep networks are composed of a fully convolutional stream for dense prediction and a segment-level spatial pooling stream for sparse saliency inference. We devise a multi-scale fully convolutional network (MS-FCN) in the first stream, which receives an entire image as input and directly learns to map it to a dense saliency prediction with pixel-level accuracy. Our MS-FCN can not only learn multi-scale feature representations, but also accurately judge the saliency of every pixel by mining visual contrast information hidden in multi-scale receptive fields. The segment-level spatial pooling stream computes another sparse saliency map over superpixels by modeling the contrast between every superpixel and its spatially adjacent regions. It extracts multi-scale regional features very efficiently by performing feature masking in the feature map of an intermediate layer of MS-FCN. At the end, we produce our final saliency map by merging the saliency maps from both streams with weight maps generated from a proposed attentional module in our deep network. Our MS-FCN can also be re-trained to generate a contour map for salient objects. This contour map can be used to improve contour localization in the fused saliency map via a fully connected CRF.

In summary, this paper has the following contributions:

-   We propose end-to-end contrast-oriented deep neural networks for localizing salient objects using multi-scale contextual information. They incorporate a fully convolutional stream for dense prediction and a segment-wise spatial pooling stream for sparse inference. A tailored alternate scheme is designed to train these deep networks by fine-tuning pre-trained baseline models.

-   A multi-scale VGG-16 or ResNet-101 network pre-trained for image classification is re-purposed as the fully convolutional stream to infer a dense saliency prediction directly from the raw input image in a single forward pass. This fully convolutional network can also be re-trained to infer a salient object contour map, which can be represented as a feature embedding and incorporated in a fully connected CRF model to further improve contour localization in the final result.

-   We have also devised a segment-wise spatial pooling stream complementary to the fully convolutional stream in our deep network. This stream efficiently masks out segment-wise features from one designated feature map of MS-FCN, and accurately models visual contrast among superpixels and well captures saliency discontinuities along region boundaries.

The rest of this paper is organized as follows. Section  reviews related work on salient object detection. In Section , we introduce our proposed contrast-oriented deep neural networks. The complete algorithm is presented in Section . Section  provides extensive performance evaluation as well as comparisons against state-of-the-art models. Finally, we conclude this paper in Section .

# Related Work

Traditional salient object detection can be categorized into bottom-up approaches with handcrafted low-level features   and top-down approaches incorporating high-level knowledge . Bottom-up methods are usually based on the center bias or background priors and infer saliency maps from global or local contrast represented as a combination of handcrafted low-level features (e.g. color, texture and image gradient). Bottom-up computational models are primarily based on a center-surround scheme and compute saliency maps using a linear or non-linear combination of low-level features such as color, intensity, texture and orientation of edges . Top-down methods are in general task-dependent and require a machine learning scheme to incorporate high-level knowledge into a process which was originally limited to specified objects or assumptions . Graph based methods have also been widely used to enhance spatial consistency and refine detected saliency maps . Recently, deep learning based methods have been widely used for salient object detection and have promoted its research into a new phase. Since the focus of this paper is deep learning based salient object detection, we highlight the most relevant previous work in the following discussion.

In recent years, the successful application of deep convolutional neural networks has triggered a revolution in machine learning and artificial intelligence, and has yielded significant improvement in a variety of visual comprehension tasks, including image classification , object detection  and semantic segmentation , closing the gap to human-level performance. Motivated by this, several attempts have also been made to apply deep neural network models to salient object detection . Han *et al.*  first attempted to develop stacked denoising autoencoders to learn powerful representations for salient object detection in an unsupervised and bottom-up manner. In , a weighted sparse coding framework is proposed for image saliency detection. Recently, with the widespread application of convolutional neural networks in image analysis and comprehension tasks, it is not surprising to see a surging number of research papers where very good results have been achieved on salient object detection via the application of CNNs. Li *et al.*  trained a multi-layer fully connected network for deriving the saliency value of every superpixel from its contextual CNN features. Wang *et al.*  proposed two deep neural networks, which take into account both low-level features and high-level objectness, for salient object detection at the patch level. A multi-context deep CNN framework incorporating both global and local contexts is presented in . However, all these methods include fully connected layers and infer saliency maps in an isolated patch-wise manner, the crucial spatial information in the input image is ignored. However, since all the image patches are treated as independent samples during network training and inference, there is no shared computation among overlapping image segments, which results in significant redundancies and excessive computational cost during training and testing.

To address these issues, inspired by the seminal work of developing end-to-end deep networks for semantic image segmentation , variant of fully convolutional neural networks have been introduced to solve the problem of salient object detection since the publication of our earlier conference version . Li *et al.*  proposed to explore the correlations between saliency detection and semantic image segmentation using a multi-task fully convolutional neural network. Liu *et al.*  propose a hierarchical recurrent CNN to progressively refine the details of saliency maps from a coarse prediction result generated from the forward pass of a fully convolutional VGG-16 network. Kuen *et al.*  proposed a recurrent attentional convolution-deconvolution network (RACDNN), which consists of a recurrent neural network and a spatial transform module, to recurrently attend to selected image sub-regions for saliency refinement. In , Wang *et al.* introduced a recurrent fully convolutional network (RFCN) to iteratively refine the saliency map with incorporated prior knowledge. These FCN based models have greatly improved both accuracy and efficiency in saliency detection, there are still three aspects of the flaws. First of all, these models are mostly based on the topmost feature map of the network for saliency inference, the over-reliance on the regional semantic feature may result in the pool detection performance on the salient region with weak semantic information. Second, all of these methods consider feature modeling at a single scale and may not accurately detect salient objects of very different sizes. And finally, as the value at each position of a saliency map generated from FCN-based models is derived from a context with a fixed size (receptive field), the contours of salient objects can hardly be well detected, and the generated saliency maps usually have inadequate spatial consistency. Our proposed method instead delves into the nature of saliency prediction, capturing the key aspect in this problem, which is contrast learning. The proposed method is not only able to infer a saliency probability map from the contrast information in a multiscale deep CNN but also from edge-preserving region-wise contrast information. In addition, it has been proven that fully connected CRFs can be formulated as recurrent neural networks (RNNs). However, experimental results show that RNNs can hardly be trained to achieve comparable results as CRFs. Our proposed method therefore exploits the effectiveness of a contour-aware CRF. Our experimental results demonstrate the superiority of our proposed method in comparison to all existing FCN based salient object detection techniques.

Note that the initial deep contrast network reported in CVPR 2016  can be viewed as the first piece of work that aims at designing an end-to-end fully convolutional network for visual contrast modeling. To a certain extent, it inspired the subsequent development of FCN-based models in this field. Our updated contrast-oriented deep neural network for salient object detection has several improvements over its initial version. First, we adapt the state-of-the-art ResNet-101 network  for image classification to a fully convolutional network and use it to replace the VGG-16 network in the original fully convolutional stream, achieving better performance. Second, the fully convolutional stream is run on multiple scaled versions of the original input image while the segment-wise spatial pooling stream is trained using segments from multi-level image segmentation. These strategies make our deep model more accurately detect salient objects at different scales. Third, we propose to add an attentional module which learns pixel-wise soft weights for fusing the two saliency maps respectively generated from the two streams. Fourth, we discover that the proposed multi-scale fully convolutional stream in our deep network can be re-trained to detect salient region contours, which can be integrated into a fully connected CRF model to further improve contour localization in the final saliency map. Finally, we present a more comprehensive experimental comparison among multiple model variants and report improved results on all benchmarks using all evaluation metrics.

# Deep Contrast Network

As illustrated in Fig. , our proposed contrast-oriented deep neural network is composed of two complementary components, a fully convolutional stream for dense saliency prediction and a segment-wise spatial pooling stream for sparse saliency inference. Specifically, the first component is a multi-scale fully convolutional network (MS-FCN), which receives an entire image as input and is trained to map the input to a dense saliency map $S_1$ in an end-to-end mode by exploiting visual contrast across multiple levels of feature maps. The segment-wise spatial pooling stream is trained to infer the saliency map $S_2$ at the segment level by discovering the contrast among spatially adjacent regions on the basis of features masked out from one designated feature map of the first stream and a multi-layer perceptron. At the end, these two intermediate saliency predictions from the above two network streams are merged according to weight maps prescribed by a trained attention module. The merged map becomes our final saliency map $S$.

## Multi-Scale Fully Convolutional Network

Inspired by the groundbreaking application of fully convolutional networks in pixel-level image comprehension, we focus on constructing an end-to-end pixelwise regression network, which can directly map a raw input image to a dense saliency map. Considering the centrality of contrast modeling for saliency detection, we have the following considerations when designing the structure of this end-to-end network. First, the network should be deep enough to accommodate features from multiple levels since visual saliency relies on modeling the contrast among both low-level appearance features as well as high-level semantic features. Second, the network needs to be able to explore the visual contrast across multiple feature maps and detect salient objects of various scales. Finally, due to the lack of training images with pixel-wise labeling, it is much desired to fine-tune an existing pre-trained network instead of training from scratch.

As VGG  and ResNet  are the two most representative and widely used deep classification networks with publicly available pre-trained models, we choose them as our pre-trained networks and adapt for our requirements. Here we describe in detail the transformation of the VGG-16 network, and ResNet-101 can be similarly transformed to satisfy the requirements. To re-purpose the VGG-16 network for dense saliency map generation, we first convert the two fully connected layers of VGG-16 into $1\times 1$ convolutional ones as described in . Moreover, as the original VGG-16 network consists of 5 max pooling layers and each with stride 2, the resulting network can only yield low-resolution prediction maps with $1/32$ the input resolution. To make the resulting saliency map have a higher resolution, we remove the downsampling operation in the last two max-pooling layers by simply setting their "stride" to 1, which results in downsampling by a factor of 8 instead of 32. At the same time, to maintain the same size of the receptive fields of the convolutional layers that follow, we refer to  and apply the dilation operation to the corresponding filter kernels. The dilation algorithm (also called à trous algorithm), which was originally proposed to improve the computational efficiency of undecimated wavelet transforms , has recently been incorporated into the Caffe framework  as "dilated convolution" to efficiently control the resolution of feature maps within deep CNNs without the need to learn extra parameters. It works by inserting zeros between filter weights. Specifically, consider applying the dilated version of a convolutional filter $w$ to an input feature map $x$, and generating an output feature map $y$. The output value at position $i$ is calculated as $$y[i]=\sum_{k}{x[i+r\cdot k]w[k]},$$ where the dilation rate $r$ corresponds to the stride with which we sample the input feature map. This is equivalent to applying convolution to the input feature map $x$ with filters up-sampled by inserting $r-1$ zeros between any two originally adjacent filter elements along each dimension. This dilated convolution allows us to explicitly control the density of feature responses in our customized fully convolutional networks. In our implementation, after setting the stride of the last two pooling layers to $1$, we replace all subsequent convolutional layers with dilated convolutional layers with dilation rate $r=2$ or $r=4$ ($r=2$ for the three consecutive convolutional layers after the penultimate max-pooling layer and $r=4$ for the last two newly converted $1\times 1$ convolutional layers).

VGG-16 has five max pooling layers performing downsampling operations. If we start from the pooling layer closest to the input image, these pooling layers have an increasingly larger receptive field containing contextual information. To design a deep convolutional network that is capable of mining visual contrast information crucial in saliency inference, we further develop a multiscale network from the above fully convolutional version of VGG-16. As shown in the left part of Fig. , we connect three extra convolution layers to each of the first four max-pooling layers. The first extra layer uses $3\times 3$ convolution kernels and has 128 channels while the second one uses $1\times 1$ convolution kernels and also has 128 channels. And the third extra layer has one $1\times 1$ kernel and a single channel, which is used to produce the output saliency map. To make the output feature maps of the four sets of extra convolutional layers have the same size (8$\times$ downsampling resolution), the stride of the first layer in these four sets are set to 4, 2, 1, and 1, respectively. Although the four resulted feature maps are of the same size, they are computed using receptive fields with different sizes and hence represent contextual features at 4 different scales. We further stack these four feature maps with the last output feature map of the above customized fully convolutional conversion. The stacked feature maps (5 channels) are fed into a final convolution layer with a $1\times 1$ kernel and a single output channel, which is modulated by the sigmoid activation function to produce the saliency probability map. Though the resulting saliency map of this network stream has a downsampling factor of $8$ in comparison to the input image, it is smooth enough and allows us to use simple bilinear interpolation to restore the resolution of the original input at a negligible computational cost. We call this resized saliency map $S_1$.

Note that the ResNet-101 network has no hidden fully connected layers. To adapt ResNet-101 for dense saliency prediction, we simply replace its 1000-way linear classification layer with a linear convolutional layer with a $1\times 1$ kernel and a single output channel. Similar to VGG-16, the resolution of the feature maps before the linear convolutional layer is only $1/32$ of that of the original input image because the original ResNet-101 consists of one pooling layer and $4$ convolutional layers, each of which has stride $2$. We call these five layers "down-sampling layers". As described in , the $101$ layers in ResNet-101 can be divided into five groups. Feature maps computed by different layers in each group share the same resolution. To increase the resolution of the final saliency map, we replace the last two down-sampling layers with dilated convolution layers, skip subsampling by setting their stride to $1$, and correspondingly increase the dilation rate of subsequent convolution kernels to enlarge their receptive fields. Therefore, all the features maps in the last three groups have the same resolution, $1/8$ original resolution, after network transformation. To develop a multiscale version of the above end-to-end extension of ResNet-101, as shown in the right of Fig. , we connect an extra sub-network with three convolutional layers to each of the final layers in the first four groups. These additional layers have the same structure as those added to VGG-16. Similar to the multiscale extension of VGG-16, the four output feature maps from these four sub-networks are stacked together with the final output feature map of the transformed ResNet-101, and fed into a final convolutional layer with a $1\times 1$ kernel and a single output channel for final saliency map inference.

## Segment-Level Saliency Inference

Salient objects in images are usually presented in a variety of irregular shapes and the corresponding saliency map often exhibits discontinuities along the object boundaries. Our multiscale fully convolutional network operates at a subsampled pixel level and equally treats each pixel in the input image without explicitly taking into account such saliency discontinuities. To better model visual contrast between regions and visual saliency along region boundaries, we design a segment-wise spatial pooling stream in our network.

We first divide an input image into a set of superpixels, and call each superpixel a segment. A mask is computed for every segment in the feature map generated from one selected convolutional layer of MS-FCN, which is named the feature masking layer. We choose the convolutional layer Conv5_3 as the feature masking layer in the MS-FCN based on VGG-16, and the last convolutional layer in the fourth layer group as the feature masking layer in the MS-FCN based on ResNet-101 as suggested in . Since the activations at each location in the feature masking layer is controlled by a receptive field in the input image, we first project every location in the feature masking layer to the center of its receptive field as in . For each segment in the input image, we first generate a binary mask within its bounding box. In this mask, pixels inside the segment are labeled '$1$' while others are labeled '$0$'. Each pixel labeled as '$1$' in the binary mask is first assigned to the closest receptive field center and then backprojected onto the feature masking layer. Thus, each location in the feature masking layer collects multiple '$1$' labels backprojected from its receptive field. The ratio between the number of collected '$1$' labels at the location and the number of pixels in the input image closest to its receptive field center is recorded. To yield a binary mask for the segment on the feature masking layer, the previously computed ratio at every location is thresholded at 0.5, and the set of locations with nonzero values after thresholding form the segment mask. In the event that the ratio at all locations is below 0.5, the set of locations with nonzero ratios before thresholding form the segment mask. The resulting segment mask is then applied to the output feature map of the feature masking layer by simply multiplying this binary mask with each channel of the feature map. We call the resulting features segment-masking features in our method. Note that the feature map generated from the feature masking layer has a downsampling factor of 8 instead of 32 in the original VGG-16 network or 16 in the original ResNet-101 network since subsampling has been skipped in the last two downsampling layers as described in Section . Therefore, the resolution of the feature map generated from the feature masking layer is sufficient for segment masking.

Since segments have irregular shapes and variable sizes when projected onto the feature masking layer, we further perform a spatial pooling (SP) operation to produce a feature vector of fixed length for each segment. It is a simplified version of spatial pyramid pooling described in . Specifically, we divide the bounding box of a projected segment into $h\times w$ cells and perform Max- or mean-pooling over valid positions (with mask label '$1$') in each grid cell. This results in $h\times w$ feature vectors of size $C$, which is the number of convolutional filters in the feature masking layer. Afterwards, we concatenate the feature vectors extracted from all grid cells of the same segment to obtain the final feature vector with $h\times w\times C$ dimensions for that segment.

To discover segment-level visual contrast, we represent each segment with a concatenation of three feature vectors respectively for three nested and increasingly larger regions masked out from the designated feature map. These three regions include the bounding box of the considered segment, the bounding box of the immediate neighboring segments as well as the entire feature map from the feature masking layer (with the considered segment excluded to indicate the position of the segment). The above-mentioned feature representation of each segment is further fed into two fully connected layers. The output of the second fully connected layer is fed into a "Sigmoid" layer which employs the sigmoid function to perform logistic regression and produces a distribution over binary saliency labels. We call the saliency map generated in this way $S_2$.

In fact, this segment-wise spatial pooling stream of our network is an accelerated version of our previous work proposed in . Although they share the identical idea of inferring saliency from contrast among multiscale contextual regions, feature extraction and processing in the current method is much more efficient as hundreds of segmental features for the same image are instantaneously masked out from the feature map generated by MS-FCN in a single forward pass. Moreover, our segment-wise spatial pooling stream also achieves better results as segment features are extracted from our multiscale fully convolutional network, which has been fine-tuned for salient object detection, instead of from the original VGG-16 model for image classification.

## Attentional Module for Saliency Map Fusion

To merge predicted saliency scores from the two different streams, there are three straightforward options: average pooling, max-pooling and $1 \times 1$ convolution. However, all these strategies are image content independent. As our two network streams have complementary strengths in saliency map prediction, inspired by , we design a trainable attentional module to generate content-dependent weight maps for fusing the results from the two streams.

Let $S_1$ and $S_2$ be the probabilistic saliency maps from the two network streams, respectively. The final saliency map from our deep contrast network is calculated as a weighted sum of these two maps. The spatially varying weights are adaptively learned. Therefore, they are called weight maps. Let $S$ be the fused saliency map, $W_1$ be the weight map for the saliency map generated from the MS-FCN stream and $W_2$ be the weight map for the saliency map generated from the second stream. The merged saliency map is calculated by summing the element-wise product between each probability map (resized to $1/8$ the input image resolution) and its corresponding weight map: $$S = W_1\odot S_1 + W_2 \odot S_2.$$

We refer to  and call $W_1$ and $W_2$ attention weights as they reflect how much attention should be paid to individual network streams as well as saliency scores at different spatial locations. These two attention weights can also be considered as feature maps that have the same size as the predicted saliency maps, and thus can be jointly trained in a fully convolutional network. In this work, we employ a differentiable attention module to our deep network to infer these attention weights. As illustrated in Fig. , the proposed attention module receives as input the output feature map from the feature masking layer, and it contains two convolutional layers. The first layer has 512 filters with kernel size $3\times 3$ while the second layer has two convolutional filters with kernel size $1\times 1$. The output feature map has two channels, further fed into a SoftMax layer, which generates two score maps corresponding to the aforementioned two attention weights.

## Deep Contrast Network Training

We propose an alternate training scheme to train our network. Specifically, in the initialization phase, we pre-compute the segments of all training images and train the segment-wise spatial pooling stream alone until convergence to obtain its initial network parameters. Segment-wise saliency labeling is performed by thresholding the average pixel-wise labeling inside each segment, and the segment features are extracted using the VGG-16 or ResNet-101 image classification model pre-trained on the ImageNet dataset . After initialization, we alternately update the weights in the two network streams. First, we fix the weights of the second stream and train the MS-FCN as well as the attention module for one epoch. Note that the weights in the attention module for adaptively merging the predicted saliency maps from the two streams are trained simultaneously with the MS-FCN stream in an end-to-end mode. Next, we fix the weights in the MS-FCN as well as the attention module, and fine-tune the parameters in the second stream for one epoch using segment features extracted with an updated VGG-16 or ResNet-101 network embedded in the MS-FCN stream. We alternately train the two streams 8 times (16 epochs in total) until the whole training process converges. We define the following class-balanced cross-entropy as the loss function for training the multi-scale fully convolutional steam and the attention module of our network, $$\begin{aligned}
L=&-\beta_i\sum_{i=1}^{|I|}G_i\log P\left ( S_i=1|I_i,W \right ) \\& -  \left ( 1-\beta_i \right )\sum_{i=1}^{|I|}\left ( 1-G_i \right )\log P\left ( S_i=0|I_i,W \right ),
\end{aligned}$$ where $\beta_i$ represents the class balancing weight, denoted as $\beta_i=\frac{|I|\_}{|I|}$ and $1-\beta_i=\frac{|I|_+}{|I|}$, where $|I|$, $|I|_+$ and $|I|\_$ respectively indicate the total number of pixels, salient pixels and non-salient ones in image $I$. $G$ represents the groundtruth annotation and $W$ represents the collection of all network weights in the MS-FCN stream and the attention module. When fine-tuning the segment-wise spatial pooling stream, we use a batch of images as a unit and update parameters by minimizing the summed squared errors accumulated over all segments from the same batch of training images.

# The Complete Algorithm

## Superpixel Segmentation

The segment-wise spatial pooling stream of our network requires the input image to be decomposed into non-overlapping segments. In order to better avoid artificial boundaries in the generated saliency map, each segment should be a perceptually homogeneous region while at the same time, strong contours and edges should still be well preserved. In our earlier version , we use a geodesic distance  based SLIC algorithm for superpixel generation. In this work, we discover that graph based image segmentation  produces segments with better edge preservation than the SLIC algorithm, and using segments generated from multiple levels of image segmentation can further improve the performance. Therefore, we refer to  and employ the graph based image segmentation algorithm therein to generate three levels of segments with different parameter settings. We train a single segment-wise spatial pooling stream for all the segments across three levels of segmentation instead of learning different model parameters for segments from different levels of segmentation. When generating a saliency map from the segment-wise spatial pooling stream, we apply the same stream to infer a saliency map for each level of segmentation and then simply average the three resulting saliency maps.

## Salient Contour Detection

While in most cases our proposed deep contrast network works well, it sometimes produces saliency maps where salient region boundaries are not accurately localized, particularly for images containing small salient regions. Meanwhile, we find that our multi-scale fully convolutional network described in Section , when re-trained using annotated salient region contours, is also capable of detecting the contours of salient regions. The detected contours can be further encoded as feature vectors and embedded into a CRF framework to enhance spatial coherence and the preservation of salient region contours in saliency maps. To prepare training data for salient region contour detection, boundary pixels of salient regions in the groundtruth saliency maps are labeled '$1$', and all other pixels are labeled '$0$'. Such salient region contour maps are taken as the groundtruth annotations when MS-FCN is trained for salient region contour detection, and the class-balancing weight is updated according to the fraction of pixels on salient region contours.

Given a detected salient region contour map $M$, we apply the normalized cut  algorithm to generate per-pixel feature vectors, which are used in a fully connected CRF to improve boundary localization in our final saliency map. First, we construct a sparse graph where every pixel is connected to other pixels in its $11\times 11$ neighborhood. The affinity matrix $W$ of this graph is defined as follows, $$W_{ij}=\exp \left( -\max_{p\in \overline{ij}} \left\{ \frac{M(p)^2}{\rho} \right\} \right),$$ where $W_{ij}$ denotes the affinity between pixels $i$ and $j$, $p$ represents pixels along the line segment ($\overline{ij}$) connecting pixels $i$ and $j$, $M(p)$ indicates the probability of pixel $p$ being on a salient region contour, and $\rho$ is a constant scaling factor, which is set to 0.1 in our experiments. The idea is that two pixels should have a similar saliency value if there is no salient region contour crossing the line segment connecting these two pixels. Given an affinity matrix $W$, we further define $D_{ii} = \Sigma_{i\neq j}W_{ij}$, and solve for generalized eigenvectors of the following system, $\left(D-W\right)v= \lambda Dv$. We use these eigenvectors as additional features to improve spatial coherence. In our experiments, we use eigenvectors corresponding to the 16 smallest eigenvalues.

## Spatial Coherence

Since both streams of our deep contrast network independently infer the saliency score of each individual pixel or segment without considering the impact of the correlation among pixels and segments on saliency prediction, the resulting saliency maps contain more or less incomplete or false positive salient objects. To mitigate this issue, we adopt a fully connected conditional random field (CRF)  in a post-processing step to enhance spatial coherence. The energy function of the CRF model is formulated as $$E\left( L \right) = -\sum_{i}\log P\left(l_i\right)+\sum_{i,j}\theta_{ij}\left(l_i, l_j\right),$$ where $L$ is the binary label prediction for all pixels (salient or not salient). $P(l_i)$ indicates the probability of pixel $x_i$ being labeled $l_i$. As an initialization, $P(l_i=1)=S_i$ and $P(l_i=0)=1-S_i$, where $S_i$ refers to the predicted probabilistic saliency value at pixel $x_i$ of the saliency map $S$ generated from our deep contrast network. The pairwise potential $\theta_{ij}\left(l_i,l_j\right)$ is defined as $$\begin{split}
\theta_{ij}=\mu\left(l_i,l_j\right)\Bigg[ \omega_1\exp\Bigg(-\frac{\left \|p_i-p_j  \right \|^2}{2\sigma_\alpha^2}-\frac{\left \|I_i-I_j \right \|^2}{2\sigma_\beta^2}\\ - \frac{\left \|v_i-v_j \right \|^2}{2\sigma_\gamma ^2}\Bigg) + \omega_2\exp\left(-\frac{\left \|p_i-p_j \right\|^2}{2\sigma_\epsilon^2}\right)\Bigg],
\end{split}$$ where $\mu\left(l_i,l_j\right) = 1$ if $l_i \neq l_j$, and zero, otherwise. It involves a summation of two Gaussian kernels. The first kernel is based on the observation that neighboring pixels should be assigned similar saliency scores if they have similar colors but do not have intervening salient region contours. It therefore depends on pixel positions ($p$), pixel intensities ($I$) and the contour feature embedding ($v$) discussed in Section . The importance of color similarity, spatial closeness and salient region contours are controlled by three parameters ($\sigma_\alpha$, $\sigma_\beta$ and $\sigma_\gamma$), respectively. The second kernel is only dependent on pixel positions with hyperparameter $\sigma_\epsilon$ controlling the scale of the Gaussian function. As pointed out in , it helps to enhance label smoothness and remove small isolated regions.

As it has been proved in , this energy minimization process can be modeled as efficient approximate probabilistic inference by adopting a mean-field approximation to the original CRF. High-dimensional filtering can be employed to speed up the computation. We adapt the publicly available implementation of  to minimize the above energy function. The optimization process takes less than 0.5 second for an image with $300\times 400$ pixels. After CRF model optimization, a saliency map $S_{crf}$ can be generated from the pixelwise posterior probabilities of saliency labels. We visualize the effectiveness of our CRF in Fig. . As can be seen, the original saliency maps from the proposed method without CRF are rather coarse and the integrity (spatial coherence) of detected salient regions can hardly be maintained. Though saliency maps generated with a traditional CRF (without the contour feature embedding) can enhance the spatial coherence of detected salient regions to some extent, salient region contours still may not be well positioned and there may be false detections in the smooth background (e.g. the third row). The fourth column of the figure demonstrates salient region contours detected by our proposed method. As can be seen, it is usually possible to accurately capture the boundaries of salient regions and its corresponding embedded features can further enhance the consistency of saliency prediction across salient region contours and correct prediction errors. A quantitative analysis of our CRF based saliency refinement will be provided in Section .

```latex
\begin{table*}[]\centering
\resizebox{0.95\textwidth}{!}
{
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
Data Set                    & Metric & SF    & GC    & HS    & DRFI  & PISA  & BSCA                         & LEGS                         & MC                                    & MDF                          & DS                                    & RFCN                                  & DHSNet                                & DCL                                   & DCL+                                  \\ \hline
                            & maxF   & 0.700 & 0.719 & 0.813 & 0.845 & 0.837 & 0.830                        & {\color[HTML]{333333} 0.870} & {\color[HTML]{333333} 0.894}          & 0.885                        & {\color[HTML]{32CB00} \textbf{0.898}} & ---                                   & ---                                   & {\color[HTML]{3166FF} \textbf{0.929}} & {\color[HTML]{FE0000} \textbf{0.931}} \\ \cline{2-16}
\multirow{-2}{*}{MSRA-B}    & MAE    & 0.166 & 0.159 & 0.161 & 0.112 & 0.102 & 0.130                        & {\color[HTML]{333333} 0.081} & {\color[HTML]{32CB00} \textbf{0.054}} & 0.066                        & 0.067                                 & ---                                   & ---                                   & {\color[HTML]{3166FF} \textbf{0.046}} & {\color[HTML]{FE0000} \textbf{0.042}} \\ \hline
                            & maxF   & 0.548 & 0.597 & 0.727 & 0.782 & 0.764 & 0.758                        & {\color[HTML]{333333} 0.827} & 0.837                                 & 0.832                        & {\color[HTML]{333333} 0.900}          & 0.899                                 & {\color[HTML]{32CB00} \textbf{0.907}} & {\color[HTML]{3166FF} \textbf{0.921}} & {\color[HTML]{FE0000} \textbf{0.925}} \\ \cline{2-16}
\multirow{-2}{*}{ECSSD}     & MAE    & 0.219 & 0.233 & 0.228 & 0.170 & 0.150 & 0.183                        & {\color[HTML]{333333} 0.118} & {\color[HTML]{333333} 0.100}          & 0.105                        & 0.079                                 & 0.091                                 & {\color[HTML]{3166FF} \textbf{0.059}} & {\color[HTML]{32CB00} \textbf{0.061}} & {\color[HTML]{FE0000} \textbf{0.058}} \\ \hline
                            & maxF   & 0.590 & 0.588 & 0.710 & 0.776 & 0.753 & 0.723                        & {\color[HTML]{333333} 0.770} & 0.798                                 & {\color[HTML]{333333} 0.861} & {\color[HTML]{333333} 0.866}          & {\color[HTML]{32CB00} \textbf{0.896}} & 0.892                                 & {\color[HTML]{3166FF} \textbf{0.909}} & {\color[HTML]{FE0000} \textbf{0.913}} \\ \cline{2-16}
\multirow{-2}{*}{HKU-IS}    & MAE    & 0.173 & 0.211 & 0.213 & 0.167 & 0.127 & 0.174                        & {\color[HTML]{333333} 0.118} & 0.102                                 & {\color[HTML]{333333} 0.076} & 0.079                                 & 0.073                                 & {\color[HTML]{32CB00} \textbf{0.052}} & {\color[HTML]{3166FF} \textbf{0.050}} & {\color[HTML]{FE0000} \textbf{0.041}} \\ \hline
                            & maxF   & 0.495 & 0.495 & 0.616 & 0.664 & 0.630 & 0.617                        & {\color[HTML]{333333} 0.669} & {\color[HTML]{333333} 0.703}          & {\color[HTML]{333333} 0.694} & {\color[HTML]{32CB00} \textbf{0.773}} & 0.747                                 & ---                                   & {\color[HTML]{3166FF} \textbf{0.799}} & {\color[HTML]{FE0000} \textbf{0.811}} \\ \cline{2-16}
\multirow{-2}{*}{DUT-OMRON} & MAE    & 0.147 & 0.218 & 0.227 & 0.150 & 0.141 & 0.191                        & {\color[HTML]{333333} 0.133} & {\color[HTML]{333333} 0.088}          & {\color[HTML]{333333} 0.092} & {\color[HTML]{32CB00} \textbf{0.084}} & 0.095                                 & ---                                   & {\color[HTML]{3166FF} \textbf{0.070}} & {\color[HTML]{FE0000} \textbf{0.064}} \\ \hline
                            & maxF   & 0.493 & 0.539 & 0.641 & 0.690 & 0.660 & {\color[HTML]{333333} 0.666} & {\color[HTML]{333333} 0.752} & 0.740                                 & 0.764                        & {\color[HTML]{32CB00} \textbf{0.834}} & 0.832                                 & 0.824                                 & {\color[HTML]{3166FF} \textbf{0.851}} & {\color[HTML]{FE0000} \textbf{0.857}} \\ \cline{2-16}
\multirow{-2}{*}{PASCAL-S}  & MAE    & 0.240 & 0.266 & 0.264 & 0.210 & 0.196 & 0.224                        & {\color[HTML]{333333} 0.157} & 0.145                                 & 0.145                        & {\color[HTML]{333333} 0.108}          & 0.118                                 & {\color[HTML]{3166FF} \textbf{0.094}} & {\color[HTML]{32CB00} \textbf{0.098}} & {\color[HTML]{FE0000} \textbf{0.092}} \\ \hline
                            & maxF   & 0.516 & 0.526 & 0.646 & 0.699 & 0.660 & {\color[HTML]{333333} 0.654} & {\color[HTML]{333333} 0.732} & 0.727                                 & 0.785                        & {\color[HTML]{32CB00} \textbf{0.829}} & 0.805                                 & 0.823                                 & {\color[HTML]{3166FF} \textbf{0.848}} & {\color[HTML]{FE0000} \textbf{0.857}} \\ \cline{2-16}
\multirow{-2}{*}{SOD}       & MAE    & 0.267 & 0.284 & 0.283 & 0.223 & 0.223 & 0.251                        & {\color[HTML]{333333} 0.195} & 0.179                                 & {\color[HTML]{333333} 0.155} & {\color[HTML]{32CB00} \textbf{0.127}} & 0.161                                 & {\color[HTML]{32CB00} \textbf{0.127}} & {\color[HTML]{3166FF} \textbf{0.122}} & {\color[HTML]{FE0000} \textbf{0.120}} \\ \hline
\end{tabular}
}
\caption{Quantitative comparison in terms of maximum F-measure (larger is better) and MAE (smaller is better). The three best performing algorithms are marked in \color[HTML]{FE0000}\textbf{red}\color{black}, \color[HTML]{3166FF}\textbf{blue}\color{black}, and \color[HTML]{32CB00}\textbf{green}\color{black}, respectively. As the testing set of the MSRA-B dataset is used as part of the training set in the released model of DHSNet~\cite{liu2016dhsnet} and RFCN~\cite{wang2016saliency}, and the part of the DUT-OMRON dataset is also used in training the DHSNet model, we exclude the corresponding results here.
}
\label{tab:comp_quantity}

\end{table*}
```

# Experimental Results

## Experimental Setup

### Datasets

We evaluate our proposed method on 6 widely used saliency detection benchmarks, including MSRA-B , HKU-IS , PASCAL-S , DUT-OMRON , ECSSD  and SOD . MSRA-B includes 5,000 images, most of which holds a single salient object. HKU-IS is proposed in our previous work , which has 4,447 images and most of the images include multiple separate salient objects. PASCAL-S is based on the validation set of PASCAL VOC2010 segmentation challenge  and contains 850 natural images. DUT-OMRON has 5,168 challenging images, which have relatively complex and diversified contents. SOD has 300 images and was originally designed for image segmentation. It is very challenging as most of the images contain multiple objects and have low contrast or cluttered background. We train the proposed contrast-oriented deep neural networks based on the combination of both the training sets of the MSRA-B (2500 images) and the HKU-IS (2500 images). The two validation sets are also combined as our final validation, which contains a total of 1,000 images. We test the model trained on this combined training set over all other datasets to verity the model's adaptability.

### Evaluation Criteria

We employ precision-recall (PR) curves, F-measure and mean absolute error (MAE) to quantitatively evaluate the performance of our method as well as other salient object detection methods. Given a saliency map with continuous values normalized to the range of 0 and 255, we compute the binary masks by using every possible fixed integer threshold. A pair of precision/recall values can be computed by comparing each binary mask against the ground truth. The precision is defined as the ratio between detected groundtruth salient pixels and all predicted salient pixels in the binary mask while the recall being the ratio between detected groundtruth salient pixels and all groundtruth salient pixels. Once the precision/recall pairs of all binary maps have been computed, the PR curve can be plotted by averaging all pairs of precision and recall values over all saliency maps of a given dataset. F-measure is defined as the harmonic mean of the average precision and the average recall, which can be calculated as $$F_{\beta} = \frac{(1+\beta^2)\cdot Precision \cdot Recall}{\beta^2\cdot Precision + Recall},$$ where $\beta^2$ is set to 0.3 to place more emphasis on precision than recall, as suggested in . During evaluation, we report the maximum F-measure (maxF) among all F-measure scores computed from precision/recall pairs on the PR curve. We also use twice the mean value of every saliency map as the threshold to generate the corresponding binary map and report the average precision, recall and F-measure of all binary maps. As a complement, we also calculate the mean absolute error (MAE)  as follows to quantitatively measure the average absolute per-pixel difference between an estimated saliency map $S$ and the corresponding groundtruth saliency map $G$. $$MAE = \frac{1}{W\times H}\sum_{x=1}^{W}\sum_{y=1}^{H}|S(x,y) - G(x,y)|.$$

### Implementation

Our proposed model has been implemented on top of the open source code of DeepLab , which is based on the Caffe platform . It was trained with a GTX Titan X GPU and Intel-i7 3.6GHz CPU.

During training, we resize all the images and their corresponding groundtruth saliency maps to $321\times 321$, and perform data augmentation by horizontal flipping. While training the MS-FCN stream, we set the learning rate for all newly added layers to $10^{-3}$ and the learning rate for the rest of the layers to $10^{-4}$. We employ a "poly" learning rate updating policy  (the learning rate is scaled by $\left(1-\frac{iter}{max\_iter}\right)^{power}$ after each iteration, and $power = 0.9$). We set the weight decay to 0.0005 and the momentum parameter to 0.9 during training. For the segment-wise spatial pooling stream, we refer to  and obtain 300 segments for each image from 3 levels of image segmentation achieved with different parameter settings. We set the grid size to $2\times 2$ while performing spatial pooling over each segment and the aggregated feature is of 6144 dimensions in the VGG-16 based MS-FCN and 12288 dimensions in the ResNet-101 based MS-FCN. This feature is further fed into a sub-network consisting of two fully connected layers, each of which contains 300 neurons. As in , we determine the parameters of the fully connected CRF by performing cross validation on the validation set. Finally, the actual value of $w_1$, $w_2$, $\sigma_\alpha$, $\sigma_\beta$, $\sigma_\gamma$ and $\sigma_\epsilon$ are respectively set to $3.0$, $5.0$, $3.0$, $50.0$, $3.0$ and $9.0$ during evaluation.

We use DCL$^+$ and DCL to respectively represent our best saliency detectors with and without CRF-based refinement. While it takes approximately 25 hours to train our model, it only costs around 0.7 second for DCL to process an image of size $400\times300$ on a PC with NVIDIA Titan X GPU and Intel-i7 3.6GHz CPU. Note that this is far more efficient than region-wise deep saliency detectors which independently treat all image patches or superpixels during saliency estimation. However, CRF-based post-processing is more expensive, and requires additional 8 seconds since we need to compute generalized eigenvectors used in the CRF model. Experimental results reported in the following section show that DCL alone without CRF refinement already performs better than most of the existing state-of-the-art methods. A specific comparison of the computational cost of different methods is summarized in Table .

```latex
\begin{table*}[]\centering
\resizebox{0.85\textwidth}{!}
{
\begin{tabular}{c|c|c|c|c|c|c|c|c|c|c|c|c|c}
\hline
        & SF    & GC   & HS   & DRFI  & PISA & BSCA & LEGS  & MC    & MDF   & DS    & RFCN  & DHSNet & DCL   \\ \hline
Time(s) & 0.115 & 0.25 & 0.43 & 47.08 & 0.65 & 2.03 & 2.00* & 2.38$^*$ & 8.00$^*$ & 0.25$^*$ & 4.60$^*$ & 0.24$^*$  & 0.68$^*$ \\ \hline
\end{tabular}
}
\caption{Comparison of running time. *: GPU time.}
\label{tab:runtime_cmp}
\end{table*}
```
```latex
\begin{table*}[]\centering

\resizebox{0.95\textwidth}{!}
{
\begin{tabular}{|l|c|c|c|c|l|l|l|l|l|}
\hline
\multicolumn{4}{|c|}{MSFCN}                                                                                & \multicolumn{2}{c|}{Segment-Level}                                                                                & \multicolumn{2}{c|}{CRF}                                          & \multicolumn{2}{c|}{Metric} \\ \hline
VGG16                            & \multicolumn{1}{l|}{ResNet-101} & \begin{tabular}[c]{@{}c@{}}Attentional\\ Module\end{tabular} & \begin{tabular}[c]{@{}c@{}}Multi-Scale\\ Input\end{tabular} & \begin{tabular}[c]{@{}c@{}}SLIC\\ Superpixel\end{tabular} & \multicolumn{1}{c|}{\begin{tabular}[c]{@{}c@{}}Multi-Scale\\ Segmentation\end{tabular}} & w/o contour                     & \multicolumn{1}{c|}{w/ contour} & maxF         & MAE          \\ \hline
\multicolumn{1}{|c|}{\textbf{${\surd}$}} & \multicolumn{1}{l|}{\textbf{}}  & \multicolumn{1}{l|}{\textbf{}}                               & \multicolumn{1}{l|}{\textbf{}}                              & \textbf{${\surd}$}                                                & \textbf{}                                                                               & \textbf{}                       & \textbf{}                       & 0.733        & 0.084        \\ \hline
\multicolumn{1}{|c|}{\textbf{${\surd}$}} & \multicolumn{1}{l|}{\textbf{}}  & \multicolumn{1}{l|}{\textbf{}}                               & \multicolumn{1}{l|}{\textbf{}}                              & \textbf{${\surd}$}                                                & \textbf{}                                                                               & \multicolumn{1}{c|}{\textbf{${\surd}$}} & \textbf{}                       & 0.757        & 0.080        \\ \hline
\multicolumn{1}{|c|}{\textbf{${\surd}$}} & \multicolumn{1}{l|}{\textbf{}}  & \textbf{${\surd}$}                                                   & \multicolumn{1}{l|}{\textbf{}}                              & \textbf{${\surd}$}                                                & \textbf{}                                                                               & \textbf{}                       & \textbf{}                       & 0.746        & 0.082        \\ \hline
\textbf{}                        & \textbf{${\surd}$}                      & \textbf{${\surd}$}                                                   & \multicolumn{1}{l|}{\textbf{}}                              & \textbf{${\surd}$}                                                & \textbf{}                                                                               & \textbf{}                       & \textbf{}                       & 0.773        & 0.076        \\ \hline
\textbf{}                        & \textbf{${\surd}$}                      & \textbf{${\surd}$}                                                   & \textbf{${\surd}$}                                                  & \textbf{${\surd}$}                                                & \textbf{}                                                                               & \textbf{}                       & \textbf{}                       & 0.792        & 0.071        \\ \hline
\textbf{}                        & \textbf{${\surd}$}                      & \textbf{${\surd}$}                                                   & \textbf{${\surd}$}                                                  & \multicolumn{1}{l|}{\textbf{}}                            & \multicolumn{1}{c|}{\textbf{${\surd}$}}                                                         & \textbf{}                       & \textbf{}                       & 0.799        & 0.070        \\ \hline
\textbf{}                        & \textbf{${\surd}$}                      & \textbf{${\surd}$}                                                   & \textbf{${\surd}$}                                                  & \textbf{}                                                 & \multicolumn{1}{c|}{\textbf{${\surd}$}}                                                         & \multicolumn{1}{c|}{\textbf{${\surd}$}} & \multicolumn{1}{c|}{\textbf{}}  & 0.804        & 0.068        \\ \hline
\textbf{}                        & \textbf{${\surd}$}                      & \textbf{${\surd}$}                                                   & \textbf{${\surd}$}                                                  & \textbf{}                                                 & \multicolumn{1}{c|}{\textbf{${\surd}$}}                                                         & \multicolumn{1}{c|}{\textbf{}}  & \multicolumn{1}{c|}{\textbf{${\surd}$}} & 0.811        & 0.064        \\ \hline
\end{tabular}
}
\caption{Performance evaluation of different model factors on DUT-OMRON Dataset.}
\label{tab:model_factor_analysis}
\end{table*}
```

## Comparison with the State of the Art

We compare our models (DCL and DCL$^+$) with $9$ other state-of-the-art algorithms, including SF , GC , HS , DRFI , PISA , BSCA , LEGS , MC , MDF , DS , RFCN , DHSNet . The last three are fully convolutional neural network based methods, which were published after the publication of our earlier conference version .

For qualitative evaluation, Figure  provides a visual comparison of saliency detection results, and the results from our proposed method achieve much improvement over those from other state-of-the-art algorithms. Specifically, our method is capable of highlighting salient regions missed by other methods in various challenging cases, e.g., salient regions touching the image boundary (the first and fifth rows), low contrast between salient objects and the background (the third and sixth rows) and images with multiple separate salient objects (the last three rows).

Our method significantly outperforms all other methods, including those fully convolutional network based deep models published after our earlier conference version , by a large margin on all public datasets in terms of the PR curve (Fig. ) as well as average precision, recall and F-measure (Fig. ). Moreover, for the purpose of quantitative evaluation, we report a comparison of maximum F-measure and MAE in Table . Our complete model (DCL$^+$) clearly outperforms the previous best-performing method by 3.67%, 1.98%, 1.90%, 10.64%, 2.76% and 3.38% in terms of maximum F-measure on MSRA-B (skipping RFCN and DHSNet on this dataset), ECSSD, HKU-IS, DUT-OMRON (skipping DHSNet), PASCAL-S and SOD, respectively. And at the same time, it respectively lowers the MAE by 22.22%, 1.69%, 21.15%, 23.81%, 2.13% and 5.51%. It can also be observed that the proposed method (DCL) without CRF-based post-processing already outperforms all evaluated methods on all considered datasets. We also compare run-time efficiency among the considered algorithms. As shown in Table , our DCL model needs around 0.68 second to generate a saliency map in the testing phase, which is comparable to other fully convolutional methods (DS , RFCN  and DHSNet ), and is much more efficient than other region-based CNN models (LEGS , MC , MDF ).

## Ablation Studies

### Component-wise Effectiveness of Deep Contrast Network

To validate the necessity and effectiveness of the two components contained in our deep contrast network, we take the VGG-16 based version as a representative and compare the saliency maps $S_1$ inferred from the first stream (MS-FCN), the saliency maps $S_2$ from the second stream as well as the fused ones based on $S_1$ and $S_2$. As shown in Fig. , the fused saliency map consistently performs best under all evaluation metrics on the testing set of the MSRA-B dataset, and the fully convolutional stream contributes to the merged prediction far more than the segment-wise spatial pooling stream. The two streams of our deep contrast network are complementary and are capable of discovering global and local contrast collaboratively through multiscale feature aggregation in both streams. To validate the effectiveness of MS-FCN, we have also generated saliency maps from the last scale of MS-FCN for comparison. As illustrated in Fig. , a single scale of MS-FCN (SC_MSFCN) may lead to significantly inferior performance when compared to the full version of MS-FCN in terms of the PR curve as well as average precision, recall and F-measure. Fig.  shows sample visualizations to demonstrate the complementary nature of the two streams inside the DCL network. As shown in the figure, although the fully convolutional stream and the segment-wise spatial pooling stream can produce promising saliency maps, they are far from perfect. MS-FCN tends to generate very smooth saliency maps but cannot well maintain the integrity of salient regions while the segment-wise stream predicts saliency maps in the unit of superpixels, it can hardly capture the global contrast and cannot well handle images with a complex background. However, the fused DCL model exploits the advantages of both and produces more accurate saliency predictions, which confirms the complementarity of these two sub-networks. In particular, there are examples (e.g. the second image in Fig. ) where the two streams have different mistakenly predicted regions, but our proposed network still preferentially integrate respectively predicted salient pixels and produce more accurate results. This further demonstrates the robustness of our network and the strong complementarity of the two network streams.

### Effectiveness of Contour Guided CRF

As described in Section , we incorporate a fully connected CRF with embedded contour features to further improve spatial coherence and contour positioning in the saliency maps generated from our deep contrast network. We compare the performance of the generated saliency maps with and without CRF as post-processing. As shown in Fig. , CRF significantly increases the accuracy of the saliency maps generated for the testing images of the MSRA-B dataset. We also show a visual comparison in Figure  to illustrate the effectiveness of conventional CRF post-processing and CRF incorporating salient region contours. As shown in this figure, conventional CRF improves the spatial consistency of predicted results to a certain extent while incorporating salient region contours enhances the confidence of saliency predictions especially for pixels near detected salient region boundaries.

## Improvements after Conference Version

After the conference version of this work, we have made the following five major modifications to our method: (1) adding an attention module to infer spatially varying weights for saliency map fusion, (2) employing the ResNet-101 network in the fully convolutional stream, (3) running the fully convolutional stream on multiple scaled versions of the original input image and fusing the results using max-pooling, (4) training and testing the segment-wise spatial pooling stream using segments from multi-level image segmentation, and (5) performing salient region contour detection and incorporating detected contours in the fully connected CRF during post-processing. In Table , we evaluate how each of these factors affects the maximum F-measure and MAE on the DUT-OMRON dataset. As shown in the table, these five factors together contribute a 7.13% improvement in the maximum F-measure and a 20.0% decline in MAE in comparison to the best reported results in the earlier conference version of this paper.

### Effectiveness of Attention Module

As described in Section , instead of simply adding a $1\times 1$ convolutional layer on top of the saliency maps from the two network streams, we design an attention module to infer spatially varying weight maps. To validate its effectiveness, we conduct a performance comparison between a deep contrast network with a trained attention module and another deep contrast network with a simple $1\times 1$ convolutional layer. As shown in Table , adopting the attention module for saliency map fusion improves the maximum F-measure on the DUT-OMRON dataset by 1.77% while lowering the MAE by 2.38%. Because of the effectiveness of this mechanism, we always integrate this module in our network in subsequent experiments.

### Effectiveness of ResNet-101 in MS-FCN

As described in Section , we have attempted to replace the VGG-16 network with a transformed ResNet-101 network in the fully convolutional stream of our deep network. To demonstrate its effectiveness, we have trained a new deep contrast network model for comparison. This new model is trained using the same setting as Section  except that the transformed VGG-16 network is replaced with a pre-trained and transformed ResNet-101. As shown in Table , adopting ResNet-101 instead of VGG-16 significantly improves the maximum F-measure on the DUT-OMRON dataset by 3.62% while lowering the MAE by 7.32%. We have also reached the same conclusion as the VGG based DCL network that ResNet-101 in the single scale setting generates over-smoothed saliency maps with prediction errors and performs much worse than the multi-scale version with side branches. As shown in the second and third columns of Fig , our proposed DCL network with multi-scale ResNet-101 generates much more confident and cleaner results than DCL with the original single-scale ResNet-101.

### Effectiveness of Multiple Scaled Inputs

Inspired by , we adopt a multi-scale input strategy when generating a saliency map from the fully convolutional stream. Specifically, we obtain three scaled versions of the original input image with the scaling factor respectively set to $1$, $0.75$, and $0.5$, and independently feed these scaled images to the fully convolutional stream. The three resulting saliency maps are fused by taking the maximum response across scales for each position (i.e. max pooling). As shown in Table , multi-scale input brings an extra 2.46% improvement in the maximum F-measure while lowering the MAE by 6.58%. Sample visualizations are shown in the fourth column of Fig , where fusing saliency predictions from multi-scale inputs gives rise to more accurate saliency maps especially when there exists multiple salient objects of different scales in the testing image.

### Effectiveness of Multi-Level Image Segmentation

As described in Section , the final saliency map from the revised segment-wise spatial pooling stream is the average of three saliency maps, each of which is computed using all superpixels from one of $3$ levels of image segmentation. As shown in Table , multi-level image segmentation further improves the maximum F-measure by 0.88% and lowers the MAE by 1.40%.

### Effectiveness of Salient Region Contours

As described in Section , we revise the CRF-based post-processing step in this version by integrating an additional feature vector computed from detected salient region contours. Salient region contours are detected using a separately trained contour detection model, which has the same network structure as the MS-FCN stream. We compare saliency maps computed without CRF, with CRF but without contour saliency features, and with contour guided CRF, respectively. As shown in Table , post-processing our saliency maps with a dense CRF always yields performance improvement. For the VGG16 based deep contrast network, running CRF as a post-processing step boosts the maximum F-measure by 3.27% and lowers the MAE by 4.76%. For the ResNet-101 based deep contrast network, which already achieves a much better performance itself, adding a dense CRF still brings a 0.63% improvement in the maximum F-measure and a 2.86% decrease in MAE. It is worth noting that contour guided CRF results in more accurate saliency maps with a 1.50% improvement in the maximum F-measure and a 8.57% decrease in MAE.

# Conclusions

In this work, we have proposed end-to-end contrast-oriented deep neural networks for salient object detection. Our deep networks contain two complementary sub-networks and are capable of extracting a wide variety of visual contrast information. The first sub-network is based on a multiscale fully convolutional network, and is intended to infer pixel-wise saliency by looking into contexts (receptive field) of multiple scales around each pixel. The second sub-network is designed to capture the contrast information among adjacent regions, which can not only maintain the consistency of saliency prediction within homogeneous regions but also better detect discontinuities along salient region boundaries. An attentional module with learnable weights is introduced to adaptively fuse the two saliency maps from the two sub-networks. Finally, to produce more accurate saliency predictions, we incorporate a CRF with a contour feature embedding to further enhance the spatial coherence and contour localization of the produced saliency map. Experimental results show that the proposed model achieves state-of-the-art performance on six public benchmark datasets under various evaluation metrics.

[^1]: This work was supported in part by the National Natural Science Foundation of China under Grant 61702565 and was also sponsored by CCF-Tencent Open Research Fund.

[^2]: G. Li is with Sun Yat-sen University, Guangzhou 510006, China (e-mail: firstname.lastname@example.com).

[^3]: Y. Yu is with the Department of Computer Science, The University of Hong Kong (e-mail: firstname.lastname@example.com).

[^4]: A preliminary version of this paper appeared in CVPR 2016 .


abstract: The question of whether information is lost in black holes is investigated using Euclidean path integrals. The formation and evaporation of black holes is regarded as a scattering problem with all measurements being made at infinity. This seems to be well formulated only in asymptotically AdS spacetimes. The path integral over metrics with trivial topology is unitary and information preserving. On the other hand, the path integral over metrics with non-trivial topologies leads to correlation functions that decay to zero. Thus at late times only the unitary information preserving path integrals over trivial topologies will contribute. Elementary quantum gravity interactions do not lose information or quantum coherence.
author: S.W.Hawking
title: Information Loss in Black Holes

# Introduction

The black hole information paradox started in 1967 when Werner Israel showed that the Schwarzschild metric was the only static vacuum black hole solution . This was then generalized to the no hair theorem, the only stationary rotating black hole solutions of the Einstein Maxwell equations are the Kerr Newman metrics . The no hair theorem implied that all information about the collapsing body was lost from the outside region apart from three conserved quantities: the mass, the angular momentum, and the electric charge.

This loss of information wasn't a problem in the classical theory. A classical black hole would last for ever and the information could be thought of as preserved inside it, but just not very accessible. However, the situation changed when I discovered that quantum effects would cause a black hole to radiate at a steady rate . At least in the approximation I was using the radiation from the black hole would be completely thermal and would carry no information. So what would happen to all that information locked inside a black hole that evaporated away and disappeared completely? It seemed the only way the information could come out would be if the radiation was not exactly thermal but had subtle correlations. No one has found a mechanism to produce correlations but most physicists believe one must exist. If information were lost in black holes, pure quantum states would decay into mixed states and quantum gravity wouldn't be unitary.

I first raised the question of information loss in 75 and the argument continued for years without any resolution either way. Finally, it was claimed that the issue was settled in favor of conservation of information by ADS-CFT. ADS-CFT is a conjectured duality between string theory in anti de Sitter space and a conformal field theory on the boundary of anti de Sitter space at infinity . Since the conformal field theory is manifestly unitary the argument is that string theory must be information preserving. Any information that falls in a black hole in anti de Sitter space must come out again. But it still wasn't clear how information could get out of a black hole. It is this question, I will address in this paper.

# Euclidean Quantum Gravity

Black hole formation and evaporation can be thought of as a scattering process. One sends in particles and radiation from infinity and measures what comes back out to infinity. All measurements are made at infinity, where fields are weak and one never probes the strong field region in the middle. So one can't be sure a black hole forms, no matter how certain it might be in classical theory. I shall show that this possibility allows information to be preserved and to be returned to infinity.

I adopt the Euclidean approach , the only sane way to do quantum gravity nonperturbatively. One might think one should calculate the time evolution of the initial state by doing a path integral over all positive definite metrics that go between two surfaces that are a distance $T$ apart at infinity. One would then Wick rotate the time interval $T$ to the Lorentzian.

The trouble with this is that the quantum state for the gravitational field on an initial or final space-like surface is described by a wave function which is a functional of the geometries of space-like surfaces and the matter fields

$$\Psi[h_{ij}, \phi, t]$$ where $h_{ij}$ is the three metric of the surface, $\phi$ stands for the matter fields and $t$ is the time at infinity. However there is no gauge invariant way in which one can specify the time position of the surface in the interior. This means one can not give the initial wave function without already knowing the entire time evolution.

One can measure the weak gravitational fields on a time like tube around the system but not on the caps at top and bottom which go through the interior of the system where the fields may be strong. One way of getting rid of the difficulties of caps would be to join the final surface back to the initial surface and integrate over all spatial geometries of the join. If this was an identification under a Lorentzian time interval $T$ at infinity, it would introduce closed time like curves. But if the interval at infinity is the Euclidean distance $\beta$ the path integral gives the partition function for gravity at temperature $\Theta=\beta^{-1}$.

$$\begin{aligned}
Z(\beta)&=&\int DgD\phi e^{-I[g, \phi]}\nonumber \\
&=&\text{Tr}(e^{-\beta H})
\end{aligned}$$

There is an infrared problem with this idea for asymptotically flat space. The partition function is infinite because the volume of space is infinite. This problem can be solved by adding a small negative cosmological constant $\Lambda$ which makes the effective volume of the space the order of $\Lambda^{-3/2}$. It will not affect the evaporation of a small black hole but it will change infinity to anti-de Sitter space and make the thermal partition function finite.

It seems that asymptotically anti-de Sitter space is the only arena in which particle scattering in quantum gravity is well formulated. Particle scattering in asymptotically flat space would involve null infinity and Lorentzian metrics, but there are problems with non-zero mass fields, horizons and singularities. Because measurements can be made only at spatial infinity, one can never be sure if a black hole is present or not.

#  The Path Integral

The boundary at infinity has topology $S^1 \times S^2$. The path integral that gives the partition function is taken over metrics of all topologies that fit inside this boundary. The simplest topology is the trivial topology $S^1 \times D^3$ where $D^3$ is the three disk. The next simplest topology and the first non-trivial topology is $S^2 \times D^2$. This is the topology of the Schwarzschild anti-de Sitter metric. There are other possible topologies that fit inside the boundary but these two are the important cases, topologically trivial metrics and the black hole. The black hole is eternal: it can not become topologically trivial at late times.

The trivial topology can be foliated by a family of surfaces of constant time. The path integral over all metrics with trivial topology can be treated canonically by time slicing. The argument is the same as for the path integral for ordinary quantum fields in flat space. One divides the time interval $T$ into time steps $\Delta t$. In each time step one makes a linear interpolation of the fields $q_i$ and their conjugate momenta between their values on succesive time steps. This method applies equally well to topologically trivial quantum gravity and shows that the time evolution (including gravity) will be generated by a Hamiltonian. This will give a unitary mapping between quantum states on surfaces separated by a time interval $T$ at infinity.

This argument can not be applied to the non-trivial black hole topologies. They can not be foliated by a family of surfaces of constant time because they don't have any spatial cross-sections that are a three cycle, modulo the boundary at infinity. Any global symmetry would lead to conserved global charges on such a three cycle. These would prevent correlation functions from decaying in topologically trivial metrics. Indeed, one can regard the unitary Hamiltonian evolution of a topologically trivial metric as a global conservation of information flowing through a three cycle under a global time translation. On the other hand, non-trivial black hole topologies won't have any conserved quantity that will prevent correlation functions from decaying. It is therefore very plausible that the path integral over a topologically non trivial metric gives correlation functions that decay to zero at late Lorentzian times. This is born out by explicit calculations. The correlation functions decay as more and more of the wave falls through the horizon into the black hole.

# Giant Black Holes

In a thought provoking paper , Maldacena considered how the loss of information into black holes in AdS could be reconciled with the unitarity of the CFT on the boundary of AdS. He studied the canonical ensemble for AdS at temperature $\beta^{-1}$. This is given by the path integral over all metrics that fit inside the boundary $S^1\otimes S^2$ where the radius of the $S^1$ is $\beta$ times the radius of the $S^2$. For $\beta <<  \Lambda$ there are three classical solutions that fit inside the boundary: periodically identified AdS, a small black hole and a giant black hole. If one normalizes AdS to have zero action, small black holes have positive action and giant black holes have very large negative action. They therefore dominate the canonical ensemble but the other solutions are important.

Maldacena considered two point correlation functions in the CFT on the boundary of AdS. The vacuum expectation value $<O(x) O (y)>$ can be thought of as the response at y to disturbances at x corresponding to the insertion of the operator $O$. It would be difficult to compute in a strongly coupled CFT but by AdS-CFT it is given by boundary to boundary Green functions on the AdS side which can be computed easily.

The Green functions in the dominant giant black hole solution have the standard form for small separation between $x$ and $y$ but decay exponentially as $y$ goes to late times and most of the effect of the disturbance at $x$ falls through the horizon of the black hole. This looks very like information loss into the black hole. On the CFT side it corresponds to screening of the correlation function whereby the memory of the disturbance at $x$ is washed out by repeated scattering.

However the CFT is unitary, so theoretically it must be possible to compute its evolution exactly and detect the disturbance at late times from the many point correlation function. All Green functions in the black hole metrics will decay exponentially to zero but Maldacena realized that the Green functions in periodically identified AdS don't decay and have the right order of magnitude to be compatible with unitarity. In this paper I have gone further and shown that the path integral over topologically trivial metrics like periodically identified AdS is unitary.

So in the end everyone was right in a way. Information is lost in topologically non-trivial metrics like black holes. This corresponds to dissipation in which one loses sight of the exact state. On the other hand, information about the exact state is preserved in topologically trivial metrics. The confusion and paradox arose because people thought classically in terms of a single topology for spacetime. It was either $R^4$ or a black hole. But the Feynman sum over histories allows it to be both at once. One can not tell which topology contributed to the observation, any more than one can tell which slit the electron went through in the two slits experiment. All that observation at infinity can determine is that there is a unitary mapping from initial states to final and that information is not lost.

# Small black holes

Giant black holes are stable and won't evaporate away. However, small black holes are unstable and behave like black holes in asymptotically flat space if $M << \Lambda^{-\frac{1}{2}}$ . However, in the approach I am using, one can not just set up a small black hole, and watch it evaporate. All one can do, is to consider correlation functions of operators at infinity. One can apply a large number of operators at infinity, weighted with time functions, that in the classical limit would create a spherical ingoing wave from infinity, that in the classical theory would form a small black hole. This would presumably then evaporate away.

For years, I tried to think of a Euclidean geometry that could represent the formation and evaporation of a single black hole, but without success. I now realize there is no such geometry, only the eternal black hole, and pair creation of black holes, followed by their annihilation. The pair creation case is instructive. The Euclidean geometry can be regarded as a black hole moving on a closed loop, as one would expect. However, the corresponding Lorentzian geometry, represents two black holes that come in from infinity in the infinite past, and accelerate away from each other for ever. The moral of this is that one should not take the Lorentzian analytic continuation of a Euclidean geometry literally as a guide to what an observer would see. Similarly, the formation and evaporation of a small black hole, and the subsequent formation of small black holes from the thermal radiation, should be represented by a superposition of trivial metrics and eternal black holes. The probability of observing a small black hole, at a given time, is given by the difference between the actions. A similar discussion of correlation functions on the boundary shows that the topologically trivial metrics make black hole formation and evaporation unitary and information preserving. One can restrict to small black holes by integrating the path integral over $\beta$ along a contour parallel to the imaginary axis with the factor $e^{\beta E_0}$. This projects out the states with energy $E_0$.

$$Z(E_0)=\int_{-i\infty}^{+i\infty}d\beta Z(\beta)e^{\beta E_0}$$

For $E_0 << \Lambda^{-\frac{1}{2}}$ most of these states will correspond to thermal radiation in AdS which acts like a confining box of volume $\Lambda^{-\frac{3}{2}}$. However, there will be thermal fluctuations which occasionally will be large enough to cause gravitational collapse to form a small black hole. This black hole will evaporate back to thermal AdS. If one now considers correlation functions on the boundary of AdS, one again finds that there is apparent information loss in the small black hole solution but in fact information is preserved by topologically trivial geometries. Another way of seeing that information is preserved in the formation and evaporation of small black holes is that the entropy in the box does not increase steadily with time as it would if information were lost each time a small black hole formed and evaporated.

# Conclusions

In this paper, I have argued that quantum gravity is unitary and information is preserved in black hole formation and evaporation. I assume the evolution is given by a Euclidean path integral over metrics of all topologies. The integral over topologically trivial metrics can be done by dividing the time interval into thin slices and using a linear interpolation to the metric in each slice. The integral over each slice will be unitary and so the whole path integral will be unitary.

On the other hand, the path integral over topologically non trivial metrics will lose information and will be asymptotically independent of its initial conditions. Thus the total path integral will be unitary and quantum mechanics is safe.

How does information get out of a black hole? My work with Hartle showed the radiation could be thought of as tunnelling out from inside the black hole. It was therefore not unreasonable to suppose that it could carry information out of the black hole. This explains how a black hole can form and then give out the information about what is inside it while remaining topologically trivial. There is no baby universe branching off, as I once thought. The information remains firmly in our universe. I'm sorry to disappoint science fiction fans, but if information is preserved, there is no possibility of using black holes to travel to other universes. If you jump into a black hole, your mass energy will be returned to our universe but in a mangled form which contains the information about what you were like but in a state where it can not be easily recognized. It is like burning an encyclopedia. Information is not lost, if one keeps the smoke and the ashes. But it is difficult to read. In practice, it would be too difficult to re-build a macroscopic object like an encyclopedia that fell inside a black hole from information in the radiation, but the information preserving result is important for microscopic processes involving virtual black holes. If these had not been unitary, there would have been observable effects, like the decay of baryons.

There is a problem describing what happens because strictly speaking, the only observables in quantum gravity are the values of the field at infinity. One can not define the field at some point in the middle because there is quantum uncertainty in where the measurement is done. What is often done is to adopt the semi-classical approximation in which one assumes that there are a large number N of light matter fields coupled to gravity and that one can neglect the gravitational fluctuations because they are only one among N quantum loops. However, in ignoring quantum loops, one throws away unitarity. A semi-classical metric is in a mixed state already. The information loss corresponds to the classical relaxation of black holes according to the no hair theorem. One can not ask when the information gets out of a black hole because that would require the use of a semi-classical metric which has already lost the information.

In 1997, Kip Thorne and I, bet John Preskill that information was lost in black holes. The loser or losers of the bet were to provide the winner or winners with an encyclopedia of their own choice, from which information can be recovered with ease. I gave John an encyclopedia of baseball, but maybe I should just have given him the ashes.

**Acknowledgments**

I am very grateful to my student, Christophe Galfard[^1], for help and discussions. He is working on a proof that correlation functions decay in topologically non trivial metrics.

[^1]: email@example.com