Humans process visual scenes selectively and sequentially using attention. Central to models of human visual attention is the saliency map. We propose a hierarchical visual architecture that operates on a saliency map and uses a novel attention mechanism to sequentially focus on salient regions and take additional glimpses within those regions. The architecture is motivated by human visual attention, and is used for multi-label image classification on a novel multiset task, demonstrating that it achieves high precision and recall while localizing objects with its attention. Unlike conventional multi-label image classification models, the model supports multiset prediction due to a reinforcement-learning based training process that allows for arbitrary label permutation and multiple instances per label.
@cite_5: Inherent in visual scene analysis is a bottleneck associated with the need to sequentially sample locations with foveating eye movements. The concept of a ‘saliency map’ topographically encoding st...
@cite_9: Abstract This article reviews the past 25 years of research on eye movements (1986–2011). Emphasis is on three oculomotor behaviors: gaze control, smooth pursuit and saccades, and on their interactions with vision. Focus over the past 25 years has remained on the fundamental and classical questions: What are the mechanisms that keep gaze stable with either stationary or moving targets? How does the motion of the image on the retina affect vision? Where do we look – and why – when performing a complex task? How can the world appear clear and stable despite continual movements of the eyes? The past 25 years of investigation of these questions has seen progress and transformations at all levels due to new approaches (behavioral, neural and theoretical) aimed at studying how eye movements cope with real-world visual and cognitive demands. The work has led to a better understanding of how prediction, learning and attention work with sensory signals to contribute to the effective operation of eye movements in visually rich environments.
Overt attention is associated with an eye movement, so that the attentional focus coincides with the fovea's line of sight. The planning of eye movements is thought to be influenced by bottom-up (scene dependent) saliency as well as top-down (goal relevant) factors @cite_9 . In particular, one major view is that two types of maps, the saliency map and the priority map, encode measures used to determine the target of attention @cite_5 . Under this view, visual input is processed into a feature-agnostic saliency map that quantifies distinctiveness of a location relative to other locations in the scene based on bottom-up properties. The saliency map is then integrated to include top-down information, resulting in a priority map.