The purpose of this paper is the detection of salient areas in natural video by using the new deep learning techniques. Salient patches in video frames are predicted first. Then the predicted visual fixation maps are built upon them. We design the deep architecture on the basis of CaffeNet implemented with Caffe toolkit. We show that changing the way of data selection for optimisation of network parameters, we can save computation cost up to @math times. We extend deep learning approaches for saliency prediction in still images with RGB values to specificity of video using the sensitivity of the human visual system to residual motion. Furthermore, we complete primary colour pixel values by contrast features proposed in classical visual attention prediction models. The experiments are conducted on two publicly available datasets. The first is IRCCYN video database containing @math videos with an overall amount of @math frames and eye fixations of @math subjects. The second one is HOLLYWOOD2 provided @math movie clips with the eye fixations of @math subjects. On IRCYYN dataset, the accuracy obtained is of @math . On HOLLYWOOD2 dataset, results in prediction of saliency of patches show the improvement up to @math with regard to RGB use only. The resulting accuracy of @math is obtained. The AUC metric in comparison of predicted saliency maps with visual fixation maps shows the increase up to @math on a sample of video clips from this dataset.
@cite_36: In this paper, we present a novel model for saliency prediction under a unified framework of feature integration. The model distinguishes itself by directly learning from natural images and automatically incorporating higher-level semantic information in a scalable manner for gaze prediction. Unlike most existing saliency models that rely on specific features or object detectors, our model learns multiple stages of features that mimic the hierarchical organization of the ventral stream in the visual cortex and integrate them by adapting their weights based on the ground-truth fixation data. To accomplish this, we utilize a multi-layer sparse network to learn low-, mid- and high-level features from natural images and train a linear support vector machine (SVM) for weight adaption and feature integration. Experimental results show that our model could learn high-level semantic features like faces and texts and can perform competitively among existing approaches in predicting eye fixations.
Deep learning architectures which have been recently proposed for the prediction of salient areas in images differ essentially by the quantity of convolution and pooling layers, by the input data, by pooling strategies, by the nature of the final classifiers and the loss functions to optimize, but also by the formulation of the problem. The attempt to predict visual attention reveals the binary classification problem of areas in images as "salient" and "non-salient". It corresponds to the visual experiment with free instructions, when the subjects are simply asked to look at the content. Shen @cite_36 proposed a deep learning model to extract salient areas in images. It allows firstly to learn the relevant characteristics of the saliency of natural images, and secondly to predict the eye fixations on objects with semantic content. The proposed model is formed by three layer sequences of "filtering" and "pooling", followed by a layer of linear SVM classifier providing ranked "salient" or "non-salient" regions of the input image. With the filtering by sparse coding and the max pooling, this model approximates human gaze fixations.