Understanding and interpreting the decisions made by deep learning models is valuable in many domains. In computer vision, computing heatmaps from a deep network is a popular approach for visualizing and understanding deep networks. However, heatmaps that do not correlate with the network may mislead human, hence the performance of heatmaps in providing a faithful explanation to the underlying deep network is crucial. In this paper, we propose I-GOS, which optimizes for a heatmap so that the classification scores on the masked image would maximally decrease. The main novelty of the approach is to compute descent directions based on the integrated gradients instead of the normal gradient, which avoids local optima and speeds up convergence. Compared with previous approaches, our method can flexibly compute heatmaps at any resolution for different user needs. Extensive experiments on several benchmark datasets show that the heatmaps produced by our approach are more correlated with the decision of the underlying deep network, in comparison with other state-of-the-art approaches.
@cite_0: Human eye fixations often correlate with locations of salient objects in the scene. However, only a handful of approaches have attempted to simultaneously address the related aspects of eye fixations and object saliency. In this work, we propose a deep convolutional neural network (CNN) capable of predicting eye fixations and segmenting salient objects in a unified framework. We design the initial network layers, shared between both the tasks, such that they capture the object level semantics and the global contextual aspects of saliency, while the deeper layers of the network address task specific aspects. In addition, our network captures saliency at multiple scales via inceptionstyle convolution blocks. Our network shows a significant improvement over the current state-of-the-art for both eye fixation prediction and salient object segmentation across a number of challenging datasets.
@cite_12: Abstract Based on human fixation data on natural images, there is an increasing demand in studying mappings from features to saliency maps. These models have achieved eclipse results than practically unsupervised saliency models. However, they constantly use more set of features annoying to explain all usable saliency-related factors, which leaves the truly effective features unknown and increases time cost. Through the supervised feature selection, it discloses that the features handled in existing models are highly redundant. On each of three touchstone datasets proposed in this paper is to predict the fixations of human eye with certain fixations method, some few numbers of features are found to be valuable enough. By using this model, the output is achieved in well manner with the help of the state-of-the-art. Both the features engaged and the model trained on any of the dataset exhibit good performance on the other two datasets indicating robustness of the occupied features and models across different datasets. At last, after training on a dataset for two different tasks, the prediction of eye fixation and detection of salient object is occupied and selected features prove robustness across the two tasks. So these findings suggest that a few set of features would account for visual saliency.
@cite_11: Understanding where people look in images is an important problem in computer vision. Despite significant research, it remains unclear to what extent human fixations can be predicted by low-level (contrast) compared to highlevel (presence of objects) image features. Here we address this problem by introducing two novel models that use different feature spaces but the same readout architecture. The first model predicts human fixations based on deep neural network features trained on object recognition. This model sets a new state-of-the art in fixation prediction by achieving top performance in area under the curve metrics on the MIT300 hold-out benchmark (AUC = 88 , sAUC = 77 , NSS = 2.34). The second model uses purely low-level (isotropic contrast) features. This model achieves better performance than all models not using features pretrained on object recognition, making it a strong baseline to assess the utility of high-level features. We then evaluate and visualize which fixations are better explained by lowlevel compared to high-level image features. Surprisingly we find that a substantial proportion of fixations are better explained by the simple low-level model than the stateof- the-art model. Comparing different features within the same powerful readout architecture allows us to better understand the relevance of low- versus high-level features in predicting fixation locations, while simultaneously achieving state-of-the-art saliency prediction.
Another seemingly related but different domain is the saliency map from human fixation @cite_12 . Fixation Prediction @cite_0 @cite_11 aims to identify the fixation points that human viewers would focus on at first glance of a given image. When predicting eye fixation, the algorithm is ”guessing” the regions humans are looking at, while our goal is to explain what the deep models focus on for a given image to make decisions. Deep models may use completely different mechanisms to classify than humans, hence human fixations should not be used to train or evaluate heatmap models.