Scene classification is a fundamental perception task for environmental understanding in today's robotics. In this paper, we have attempted to exploit the use of popular machine learning technique of deep learning to enhance scene understanding, particularly in robotics applications. As scene images have larger diversity than the iconic object images, it is more challenging for deep learning methods to automatically learn features from scene images with less samples. Inspired by human scene understanding based on object knowledge, we address the problem of scene classification by encouraging deep neural networks to incorporate object-level information. This is implemented with a regularization of semantic segmentation. With only 5 thousand training images, as opposed to 2.5 million images, we show the proposed deep architecture achieves superior scene classification results to the state-of-the-art on a publicly available SUN RGB-D dataset. In addition, performance of semantic segmentation, the regularizer, also reaches a new record with refinement derived from predicted scene labels. Finally, we apply our SUN RGB-D dataset trained model to a mobile robot captured images to classify scenes in our university demonstrating the generalization ability of the proposed algorithm.
@cite_8: Place categorization and object recognition are competencies needed by robots to perform a variety of service tasks in the home, such as fetch-and-carry, retrieval, cleaning, meal preparation, and companionship. Context is a powerful cue for place categorization and object recognition; rooms are laid out in a specific fashion to enable comfortable and efficient living, and objects are used within rooms for tasks specific to that room. This paper will present a technique which leverages contextual cues for joint reasoning about object and room classification via a conditional random field model.
@cite_9: We present conditional random fields , a framework for building probabilistic models to segment and label sequence data. Conditional random fields offer several advantages over hidden Markov models and stochastic grammars for such tasks, including the ability to relax strong independence assumptions made in those models. Conditional random fields also avoid a fundamental limitation of maximum entropy Markov models (MEMMs) and other discriminative Markov models based on directed graphical models, which can be biased towards states with few successor states. We present iterative parameter estimation algorithms for conditional random fields and compare the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
@cite_1: In this paper we propose an approach to holistic scene understanding that reasons jointly about regions, location, class and spatial extent of objects, presence of a class in the image, as well as the scene type. Learning and inference in our model are efficient as we reason at the segment level, and introduce auxiliary variables that allow us to decompose the inherent high-order potentials into pairwise potentials between a few variables with small number of states (at most the number of classes). Inference is done via a convergent message-passing algorithm, which, unlike graph-cuts inference, has no submodularity restrictions and does not require potential specific moves. We believe this is very important, as it allows us to encode our ideas and prior knowledge about the problem without the need to change the inference engine every time we introduce a new potential. Our approach outperforms the state-of-the-art on the MSRC-21 benchmark, while being much faster. Importantly, our holistic model is able to improve performance in all tasks.
@cite_16: In this paper, we tackle the problem of indoor scene understanding using RGBD data. Towards this goal, we propose a holistic approach that exploits 2D segmentation, 3D geometry, as well as contextual relations between scenes and objects. Specifically, we extend the CPMC [3] framework to 3D in order to generate candidate cuboids, and develop a conditional random field to integrate information from different sources to classify the cuboids. With this formulation, scene classification and 3D object recognition are coupled and can be jointly solved through probabilistic inference. We test the effectiveness of our approach on the challenging NYU v2 dataset. The experimental results demonstrate that through effective evidence integration and holistic reasoning, our approach achieves substantial improvement over the state-of-the-art.
@cite_10: Place and object recognition are two fundamental problems for mobile robot to understand its surroundings. In the field of computer vision it has been acknowledged that context plays an important role in image parsing, but in most of the researches contextual information is only used in one direction and little attention is paid to the relative pose context between objects and local features. We observe, however, place and object can serve as context to each other, that is the recognition of one facilitates the recognition of the other. In this paper, a new hierarchical random field which can encode multiple kinds of context including co-occurrence context, temporal context and relative pose context is proposed for simultaneous place and object recognition with a mobile platform. And a new kind of relative pose context, which is scale and rotation invariant, is defined to improve the stability of pose-encoded context. Experimental results with a mobile robot prove that the proposed method significantly improve the precision of the place and object recognition in familiar and unfamiliar environments.
@cite_12: Robust low-level image features have been proven to be effective representations for a variety of visual recognition tasks such as object recognition and scene classification; but pixels, or even local image patches, carry little semantic meanings. For high level visual tasks, such low-level image representations are potentially not enough. In this paper, we propose a high-level image representation, called the Object Bank, where an image is represented as a scale-invariant response map of a large number of pre-trained generic object detectors, blind to the testing dataset or visual task. Leveraging on the Object Bank representation, superior performances on high level visual recognition tasks can be achieved with simple off-the-shelf classifiers such as logistic regression and linear SVM. Sparsity algorithms make our representation more efficient and scalable for large scene datasets, and reveal semantically meaningful feature patterns.
Some previous works have demonstrated that interaction between scene and objects have the capability to promote each other @cite_1 @cite_16 @cite_10 @cite_8 . The typical idea is to build the relationship between scenes and objects using a graphical model such as Markov Random Field or Conditional Random Field @cite_9 . Though these works have achieved superior results, they are based on hand-crafted features, which means the feature extraction and classification in these works are not in a unified optimization framework. Compared to these works focusing on simultaneously labeling, our work is more close to Object Bank @cite_12 since we focus on regularizing scene classification with semantic segmentation. Object Bank proposed a high-level representation for scene classification by encoding the images with combination of a large amount of object detectors. However, the feature extraction and scene classification in Object Bank are still optimized separately, and it requires pre-training a large number of object detectors. Recently, the superior results achieved with deep learning methods suggest that learning features with a fully trainable architecture may be a better choice. In this paper, we implemented the scene classifier using a fully trainable deep architecture with a single semantic segmentation branch encoding all object-level information.