Abstract Despite the tremendous achievements of deep convolutional neural networks (CNNs) in many computer vision tasks, understanding how they actually work remains a significant challenge. In this paper, we propose a novel two-step understanding method, namely Salient Relevance (SR) map, which aims to shed light on how deep CNNs recognize images and learn features from areas, referred to as attention areas, therein. Our proposed method starts out with a layer-wise relevance propagation (LRP) step which estimates a pixel-wise relevance map over the input image. Following, we construct a context-aware saliency map, SR map, from the LRP-generated map which predicts areas close to the foci of attention instead of isolated pixels that LRP reveals. In human visual system, information of regions is more important than of pixels in recognition. Consequently, our proposed approach closely simulates human recognition. Experimental results using the ILSVRC2012 validation dataset in conjunction with two well-established deep CNN models, AlexNet and VGG-16, clearly demonstrate that our proposed approach concisely identifies not only key pixels but also attention areas that contribute to the underlying neural network's comprehension of the given images. As such, our proposed SR map constitutes a convenient visual interface which unveils the visual attention of the network and reveals which type of objects the model has learned to recognize after training. The source code is available at https: github.com Hey1Li Salient-Relevance-Propagation .
@cite_35: In this paper, a novel method for the detection of multiple salient regions that is based on the integration of estimated foreground clues is proposed. Although this subject has been very well studied for the detection of salient objects, many technical challenges still exist regarding the multiple-object-detection task; in particular, unlike a single-object-detection problem, a high inter-object dissimilarity causes new difficulties. By analyzing the limitations of the existing models, the following two main frameworks that are based on a multi-level foreground-segmentation strategy are proposed: non-parametric cluster-based saliency (NS) and parametric cluster-based saliency (PS). Each framework consists of a vector classification, a foreground estimation, an energy generation, and an integration process. In contrast to previous models, the proposed method is not dependent upon the contrast features, and is unaffected by the size, thickness, and shape of the objects. In the experiment results, a superior detection accuracy for the SED2 benchmark was achieved with the use of the proposed scheme; furthermore, the corresponding precision and recall are superior to those of the state-of-the-art approaches, and more effective performances were also achieved on the MSRA-ASD, SED2 and CSSD benchmarks. Display Omitted We address the problem of multiple salient region detection.The method consists of the parametric and non-parametric cluster based streams.The limitations of the existing models that are based on contrast are addressed.A spatial objectness is only considered for the computation of saliency scores.
@cite_18: A visual attention system, inspired by the behavior and the neuronal architecture of the early primate visual system, is presented. Multiscale image features are combined into a single topographical saliency map. A dynamical neural network then selects attended locations in order of decreasing saliency. The system breaks down the complex problem of scene understanding by rapidly selecting, in a computationally efficient manner, conspicuous locations to be analyzed in detail.
@cite_4: The human visual system (HSV) is quite adept at swiftly detecting objects of interest in complex visual scene. Simulating human visual system to detect visually salient regions of an image has been one of the active topics in computer vision. Inspired by random sampling based bagging ensemble learning method, an ensemble dictionary learning (EDL) framework for saliency detection is proposed in this paper. Instead of learning a universal dictionary requiring a large number of training samples to be collected from natural images, multiple over-complete dictionaries are independently learned with a small portion of randomly selected samples from the input image itself, resulting in more flexible multiple sparse representations for each of the image patches. To boost the distinctness of salient patch from background region, we present a reconstruction residual based method for dictionary atom reduction. Meanwhile, with the obtained multiple probabilistic saliency responses for each of the patches, the combination of them is finally carried out from the probabilistic perspective to achieve better predictive performance on saliency region. Experimental results on several open test datasets and some natural images demonstrate that the proposed EDL for saliency detection is much more competitive compared with some existing state-of-the-art algorithms.
@cite_33: Object detection, including objectness detection (OD), salient object detection (SOD), and category-specific object detection (COD), is one of the most fundamental yet challenging problems in the computer vision community. Over the last several decades, great efforts have been made by researchers to tackle this problem, due to its broad range of applications for other computer vision tasks such as activity or event recognition, content-based image retrieval and scene understanding, etc. While numerous methods have been presented in recent years, a comprehensive review for the proposed high-quality object detection techniques, especially for those based on advanced deep-learning techniques, is still lacking. To this end, this article delves into the recent progress in this research field, including 1) definitions, motivations, and tasks of each subdirection; 2) modern techniques and essential research trends; 3) benchmark data sets and evaluation metrics; and 4) comparisons and analysis of the experimental results. More importantly, we will reveal the underlying relationship among OD, SOD, and COD and discuss in detail some open questions as well as point out several unsolved challenges and promising future works.
@cite_41: Although texture is important for many vision-related tasks, it is not used in most salience models. As a consequence, there are images where all existing salience algorithms fail. We introduce a n ...
@cite_21: In this paper, we propose an unsupervised salient object segmentation approach using saliency and object features. In the proposed method, we utilize occlusion boundaries to construct a region-prior map which is then enhanced using object properties. To reject the non-salient regions, a region rejection strategy is employed based on the amount of detail (saliency information) and density of KAZE keypoints contained in them. Using the region rejection scheme, we obtain a threshold for binarizing the saliency map. The binarized saliency map is used to form a salient superpixel cluster. Finally, an iterative grabcut segmentation is applied with salient texture keypoints (SIFT keypoints on the Gabor convolved texture map) supplemented with salient KAZE keypoints (keypoints inside saliency cluster) as the foreground seeds and the binarized saliency map (obtained using the region rejection strategy) as a probably foreground region. We perform experiments on several datasets and show that the proposed segmentation framework outperforms the state of the art unsupervised salient object segmentation approaches on various performance metrics. Display Omitted Effective object segmentation using saliency and KAZE features is proposed.Region rejection strategy utilizing saliency and density of KAZE keypoints.KAZE keypoints are most suited for characterization of boundaryness.Objectness level information is enhanced with the help of salient keypoints.Outperform state of the art unsupervised salient object segmentation techniques.
@cite_43: 
@cite_5: ▪ Abstract Over the past decade, the study of attention in infancy has seen dramatic progress. This review delineates four attentional functions (alertness, spatial orienting, attention to object features, and endogenous attention) that are relevant to infancy and uses these functions as a framework for summarizing the developmental course of attention in infancy. Rudimentary forms of various attentional functions are present at birth, but each of the functions exhibits different and apparently dissociable periods of postnatal change during the first years of life. The role of attention in development should therefore be considered in the context of interaction among different systems at different levels of maturity during the first years of life.
@cite_31: A number of psychophysical studies concerning the detection, localization and recognition of objects in the visual field have suggested a two-stage theory of human visual perception. The first stage is the “preattentive” mode, in which simple features are processed rapidly and in parallel over the entire visual field. In the second, “attentive” mode, a specialized processing focus, usually called the focus of attention, is directed to particular locations in the visual field. The analysis of complex forms and the recognition of objects are associated with this second stage.1 The computational justification for such a hypothesis comes from the realization that while it is possible to imagine specific algorithms performing tasks such as shape analysis and recognition at specific locations, it is difficult to imagine these algorithms operating in parallel over the whole visual scene, since such an approach will quickly lead to a combinatorial explosion in terms of required computational resources.2 This is essentially the major critique of Minsky and Papert to a universal application of perceptrons in visual perception.3 Taken together, these empirical and theoretical studies suggest that beyond a certain preprocessing stage, the analysis of visual information proceeds in a sequence of operations, each one applied to a selected location (or locations).
@cite_34: This review focuses on covert attention and how it alters early vision. I explain why attention is considered a selective process, the constructs of covert attention, spatial endogenous and exogenous attention, and feature-based attention. I explain how in the last 25 years research on attention has characterized the effects of covert attention on spatial filters and how attention influences the selection of stimuli of interest. This review includes the effects of spatial attention on discriminability and appearance in tasks mediated by contrast sensitivity and spatial resolution; the effects of feature-based attention on basic visual processes, and a comparison of the effects of spatial and feature-based attention. The emphasis of this review is on psychophysical studies, but relevant electrophysiological and neuroimaging studies and models regarding how and where neuronal responses are modulated are also discussed.
Visual saliency is a biologically inspired model of measuring which information stands out relative to its neighbors and so attracts human attention @cite_34 . It was originally promoted by psychologists in the study of attention in infancy @cite_5 . Itti @cite_18 presented a computational architecture to introduce the basic Koch and Ullman model @cite_31 to the field of computer vision. There is an extensive body of literature on various applications of visual saliency, such as @cite_4 @cite_35 @cite_41 @cite_21 to name just a few. Recently, researchers have also integrated saliency with the latest deep learning techniques in salient object detection and category-specific object detection @cite_33 . As is summarized in @cite_43 , most visual saliency models are implemented in three stages: