Learning Deep NBNN Representations for Robust Place Categorization
Massimiliano Mancini Samuel Rota Bulò Elisa Ricci Barbara Caputo




Disclaimer:

This work has been accepted for publication in the IEEE Robotics and Automation Letters. 
Copyright: 
 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses,  in  any  current  or  future  media,  including  reprinting/  republishing  this  material  for  advertising  or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.




Urban Vibrancy and Safety in Philadelphia
    Colman Humphrey1 and Shane T. Jensen1 and Dylan S. Small1 and Rachel Thurston2
    July 10, 2016; Revised January 10, 2017
==================================================================================

empty
empty




This paper presents an approach for semantic place categorization using data obtained from RGB cameras. Previous studies on visual place recognition and classification have shown that, by considering features derived from pre-trained Convolutional Neural Networks (CNNs) in combination with part-based classification models, high recognition accuracy can be achieved, even in presence of occlusions and severe viewpoint changes. Inspired by these works,  we propose to exploit local deep representations, representing images as set of regions applying a Naïve Bayes Nearest Neighbor (NBNN) model for image classification. As opposed to previous methods where CNNs are merely used as feature extractors, our approach seamlessly integrates the NBNN model into a fully-convolutional neural network.
Experimental results show that the proposed algorithm outperforms previous methods based on pre-trained CNN models and that, when employed in challenging robot place recognition tasks, it is robust to occlusions, environmental and sensor changes. 







§ INTRODUCTION


Recent years have seen the breakthrough
of mobile robotics into the consumer market. Domestic robots have become increasingly
common, as well as vehicles making use
of cameras, radar and other sensors to assist the driver.


An important aspect of human-robot interaction, is the ability of artificial agents to understand the way humans think and talk about abstract spatial concepts. For example,
a domestic robot may be asked to “clean the bathroom”, while a car may be asked to “stop at
the parking area”. Hence, a robot's definition of “bathroom”, or
“parking area” should point to the same set of places that a human would recognize as such.

The problem of assigning a semantic spatial label to an image has been extensively studied in the computer and robot vision 
literature <cit.>. 


The most important challenges in identifying places 
come from the complexity of the concepts
to be recognized and from the variability of the conditions in which the images are captured. Scenes
from the same category may differ significantly, 
while images corresponding to different places
may look similar. 
The historical take on these issues has been to model the visual appearance of scenes considering a large variety of both global and local descriptors <cit.> and several (shallow) learning models (e.g. SVMs, Random Forests).  



Since the (re-)emergence of Convolutional Neural Networks (CNNs), approaches based on learning deep representations have become  mainstream. 

Several works exploited deep models for visual-based scene classification and place recognition tasks, showing improved accuracy over traditional methods based on hand-crafted descriptors <cit.>. Some of these studies <cit.> demonstrated the benefit of adopting a region-based approach (i.e. considering only specific image parts) in combination with descriptors derived from CNNs, such as to obtain models which are robust to viewpoint changes and occlusions. With a similar motivation, lately several works in computer vision have attempted to bring back the notion of localities into deep networks, e.g. by designing appropriate pooling strategies <cit.> or by casting the problem within the Image-2-Class (I2C) recognition framework <cit.>, with a high degree of success.
All these works decouple the choice of the significant localities from the learning of deep representations,
as the CNN feature extraction and the classifier learning are implemented as two separate modules.
This leads to two drawbacks: first, choosing heuristically the relevant localities means concretely cropping parts of the images before feeding them to the chosen features extractor.
This is clearly sub-optimal, and might turn out to be computationally expensive. Second, it would be desirable to fully exploit the power of deep networks by directly learning the best representations for the task at hand, rather than re-use architectures trained on general-purpose databases like ImageNet and passively processing patches from the input images without adapting its weights. Ideally, 
a fully-unified approach would guarantee more discriminative representations, resulting in higher recognition accuracy.

This paper contributes to this last research thread by addressing these two issues. We propose an approach for semantic place categorization which exploits local representations within a deep learning framework. Our method is inspired by the recent work <cit.>, which demonstrates that, by dividing images into regions and representing them with CNN-based features, state-of-the-art scene recognition accuracy can be achieved by exploiting an I2C approach,  

namely a parametric extension of the Naïve Bayes Nearest Neighbor (NBNN) model. 




Following this intuition, we propose a  deep architecture for semantic scene classification which seamlessly integrates the NBNN and CNN frameworks (Fig. <ref>). We automatize the multi-scale patch extraction process by adopting a fully-convolutional network <cit.>, guaranteeing a significant advantage in terms of computational cost over two-steps methods. Furthermore, a differentiable counterpart of the traditional NBNN loss is considered to obtain an error that can be back-propagated to the underlying CNN layers, thus enabling end-to-end training.  
To the best of our knowledge, this is the first attempt to fully unify NBNN and CNN, building a deep version of Naïve Bayes Nearest Neighbor. 


We extensively evaluate our approach on several publicly-available benchmarks. Our results demonstrate the advantage of the proposed end-to-end learning scheme over previous works based on a two-step pipeline and the effectiveness of our deep network over state-of-the-art methods on challenging robot place categorization tasks.
































§ RELATED WORK


In this section we review previous works on (i) visual-based place recognition and categorization and (ii) Naïve Bayes Nearest Neighbor classification.



 §.§ Visual-based Place Recognition and Categorization

In the last decade several works in the robotic community addressed the problem of developing robust place recognition <cit.> and semantic classification <cit.> approaches using visual data.
In particular, focusing on place categorization from monocular images, earlier works adopted a two-step pipeline: first, hand-crafted features, such as GIST <cit.>, CENTRIST <cit.>, CRFH <cit.> or HOUP <cit.>, are extracted from the query image, and then the image is classified into one of the predefined categories using a previously-trained discriminative model (e.g., Support Vector Machines). 
Similarly, earlier studies on visual-based place recognition and loop closing also considered hand-crafted feature representations <cit.>.

More recently, motivated by the success of deep learning models in addressing visual recognition tasks <cit.>, robotic researchers have started to exploit feature representations derived from CNNs for both place recognition <cit.> and semantic scene categorization <cit.> tasks. Sunderhau ̈et al. <cit.> analyzed the performance of CNN-based descriptors with respect to viewpoint changes and time variations, presenting the first real-time place recognition system based on convolutional networks.
Arroyo et al. <cit.> addressed the problem of topological localization across different seasons and proposed an approach which fuses information derived from multiple convolutional layers of a deep architecture. Gout et al. <cit.> evaluated the representational power of deep features for analyzing images collected by an autonomous surface vessel, studying the effectiveness of CNN descriptors in case of large seasonal and illumination changes. Uršič et al. <cit.> proposed an approach for semantic room categorization: first, images are decomposed in regions and CNN-based descriptors are extracted for each region; then, a part-based classification model is derived for place categorization. Interestingly, they showed that their method outperforms traditional CNN architectures based on global representations <cit.>, as the part-based model guarantees robustness to occlusions and image scaling. Our work develops from a similar idea, but differently from <cit.> the deep network is not merely used as feature extractor and a novel CNN architecture, suitable to end-to-end training, is proposed. 



 §.§ Naïve Bayes Nearest Neighbor Classification

The NBNN approach has been widely adopted in the computer and robot vision community, as an effective method to overcome the limitations of local descriptor quantization and Image-2-Image recognition <cit.>. Several previous studies have demonstrated that the I2C paradigm implemented by NBNN models is especially beneficial for generalization and domain adaptation <cit.> and that, by adding a learning component to the non-parametric NBNN, performance can be further boosted <cit.>. 

Recent works have also shown that the NBNN can be successfully employed for place recognition and categorization tasks <cit.>. Kanji <cit.> introduced a NBNN scene descriptor for cross-seasonal place recognition. In a later work <cit.>, Kanji extended this approach by integrating CNN-based features and PCA, deriving a PCA-NBNN model for addressing the problem of self-localization in case of images with small view overlap. Kuzborskij et al. <cit.> proposed 
a multi-scale parametric version of the NBNN classifier and demonstrated its effectiveness in combination with precomputed CNN descriptors for scene recognition. Our work is inspired by <cit.>. However, the proposed learning model is based on a fully-convolutional network which can be trained in an end-to-end manner. Therefore, it is significantly faster and more accurate than  <cit.>.



§ FULLY-CONVOLUTIONAL CNN-NBNL

In this section we describe the proposed approach for semantic place categorization. 
As illustrated in Fig. <ref>, our method develops from the same idea of previous models based on local representations and CNN descriptors <cit.>: images are decomposed into multiple regions (represented with CNN features) and a part-based classifier is used to infer the labels associated to places. However, differently from previous works, our approach unifies the feature extraction and the classifier learning phases, and we propose a novel CNN architecture which implements a part-based classification strategy. As demonstrated in our experiments (Sect. <ref>), our deep network guarantees a significant boost in performance, both in term of accuracy and computational cost.
Since our framework is derived from previous works on NBNN-based methods <cit.>, 
we first provide a brief description of these approaches (Sect. <ref>-<ref>) and then we introduce the proposed fully-convolutional NBNN-based network (Sect. <ref>).





 §.§ Naïve Bayes Non-Linear Learning


Let X denote the set of possible images and let Y be a finite set of class labels, indicating the different scene categories. The goal is to estimate a classifier f: X→ Y from a training set T⊂ X× Y sampled from the underlying, unknown data distribution.
The NBNN method <cit.> works under the assumption that there is a an intermediate Euclidean space Z and a set-valued function ϕ that abstracts an input image x∈ X into a set of descriptors in Z, ϕ(x)⊂ Z. For instance, the image could be broken into patches and a descriptor in Z could be computed for each patch. Given a training set T, let Φ_y( T) be the set of descriptors computed from images in T having labels y∈ Y, Φ_y( T)={ϕ(x) : x∈ X,(x,y)∈ T }.
The NBNN classifier f_𝙽𝙱𝙽𝙽 is given as follows:

    f_𝙽𝙱𝙽𝙽(x; T)=_y∈ Y∑_z∈ϕ(x)d(z,Φ_y( T))^2 ,

where d(x, S)=inf{‖ z-s‖_2 : s∈ S} denotes the smallest Euclidean distance between z and an element of S⊂ Z, or in other terms it is the distance between z and its nearest neighbor in S.








Despite its effectiveness in terms of classification performance <cit.>, f_𝙽𝙱𝙽𝙽 has the drawback of being expensive at test time, due to the nearest-neighbor search. 
A possible way to reduce the complexity of this step consists in learning a small, finite set W_y⊂ Z of representative prototypes for each class y∈ Y to replace Φ_y( T). This idea was pursued by Fornoni et al. <cit.> with a method named Naïve Bayes Non-Linear Learning (NBNL). 

NBNL is developed from Eq. (<ref>) by replacing Φ_y( T) with the set of prototypes W_y and by assuming Z to be restricted to the unit ball. Under the latter assumption 
the bound d(z, S)^2≥ 2-ω(z, S) can be derived <cit.>, where:

    ω(z, S)= (∑_s∈ S|⟨ z,s⟩|_+^q)^1/q .

Here, ⟨·⟩ denotes the dot product, q∈[1,+∞] and |x|_+=max(0,x). The NBNL classifier is finally obtained in the form given below by using the bound as a replacement of d()^2 in Eq.(<ref>) (and after simple algebraic manipulations):

    f_𝙽𝙱𝙽𝙻(x; W)=_y∈ Y∑_z∈ϕ(x)ω(z, W_y) ,

where W={ W_y}_y∈ Y encompasses all the prototypes.

In order to learn the prototypes W_y for each y∈ Y, Fornoni et al. did not consider f_𝙽𝙱𝙽𝙻 as classifier and T as training set, but they considered (only at training time) a classifier having the form f(x)=_y∈ Yω(z, W_y) and an extended training set {(z,y) :z∈Φ_y( T), y∈ Y}, where each descriptor extracted from an image is promoted to a training sample. In this way they derived the equivalent of a Multiclass Latent Locally Linear SVM (ML3) that is trained using the algorithm in <cit.>. 

































 §.§ CNN-NBNL


Motivated by the robustness of NBNN/NBNL models and by the recent success of deep architectures in addressing challenging visual tasks, Kuzborskij et al. <cit.> introduced an approach, named CNN-NBNL, which combines the NBNL and CNN frameworks. Their method is an implementation of NBNL, where ϕ(x) is obtained by dividing an image x∈ X into patches at different scales and by employing a pre-trained CNN-based feature extractor <cit.> to compute a descriptor for each patch. In formal terms, if g_𝙲𝙽𝙽: X→ Z is the CNN-based feature extractor that takes an input image/patch and returns a single descriptor, then ϕ(x) (see, Sect. <ref>) is given by 

    ϕ_𝙲𝙽𝙽(x)={g_𝙲𝙽𝙽(x̂) : x̂∈patches(x)} ,

where patches(x)⊂ X returns a set of patches extracted from x at multiple scales and reshaped to be compatible in terms of resolution with the input dimensionality required by the implementation of g_𝙲𝙽𝙽 (CaffeNet <cit.> requires 227× 227).
To learn the prototypes W_y in <cit.> a training objective similar to <cit.> is adopted, but the optimization is performed using a stochastic version of ML3 (STOML3) that better scales to larger datasets. At test time, f_𝙽𝙱𝙽𝙻 defined as in Eq. <ref> is used with ϕ replaced by ϕ_𝙲𝙽𝙽.









 









 By moving from hand-crafted features to CNN-based features, the performance of the NBNL classifier improves considerably. Nonetheless, the approach proposed in <cit.> has two limitations: 1) it requires the extraction of patches for each image as a pre-processing step, and CNN-features are extracted sequentially from each patch; 2) the CNN architecture is used as a mere feature extractor and the method lacks the advantage of an end-to-end trainable system. The first limitation has a negative impact on the computation time of the method, while the latter makes way for further performance boosts.
 


 §.§ Fully-Convolutional CNN-NBNL









To overcome the two limitations of CNN-NBNL mentioned above, in this work we introduce a fully-convolutional version of CNN-NBNL that is end-to-end trainable (Fig. <ref>).


Fully-convolutional extension.
Extracting patches at multiple scales and extracting CNN features independently for each of them is a very costly operation, which severely impacts training and test time. In order to perform a similar operation but with a limited impact on computation time, we propose to employ a Fully-Convolutional CNN (FC-CNN) <cit.> to simulate the extraction of descriptors from multiple patches over the entire image. A FC-CNN can be derived from a standard CNN by replacing fully-connected layers with convolutional layers. By doing so, the network is able to map an input image of arbitrary size into a set of spatially-arranged output values (descriptors). 
To cover multiple scales, we simply aggregate descriptors that are extracted with the FC-CNN from images at different resolutions. In this way, 






as the receptive fields of the FC-CNN remain the same, changing the scale of the input image induces an implicit change in the scale of the descriptors. 


The number of obtained descriptors per image depends on the image resolution and can in general be controlled by properly shaping the convolutional layers: for instance, by increasing the stride of the last convolutional layer it is possible to reduce the number of descriptors that the FC-CNN returns.

In the following, we denote by g_𝙵𝙲𝙽(x;θ)⊂ Z the output of a FC-CNN parametrized by θ applied to an input image x∈ X. As opposed to g_𝙲𝙽𝙽 defined in Sect. <ref>, which returns a single descriptor, g_𝙵𝙲𝙽(x;θ) outputs a set of descriptors, one for each spatial location in the final convolutional layer of the FC-CNN. Each descriptor has a dimensionality that equals the number of output convolutional filters. We will also denote by η(x) the number of descriptors that the FC-CNN generates for an input image x. Note that this number does not depend on the actual parametrization of the network, but only on its topology, which is assumed to be fixed, and on the resolution of the input image.














End-to-end architecture. 

The NBNL classifier that we propose and detail below can be implemented using layers that are commonly found in deep learning frameworks and can thus be easily stacked on top of a FC-CNN (see, Fig. <ref>). By doing so, we obtain an architecture that can be trained end-to-end.


Given an input image x∈ X, we create a set of m scaled versions of x, which we denote by scale(x)⊂ X.
Each scaled image x̂∈scale(x) is fed to the FC-CNN described before, yielding a set of descriptors g_𝙵𝙲𝙽(x̂;θ). Instead of aggregating the descriptors from each scale, as done in Eq. (<ref>), we keep them separated because they undergo a normalization step which avoids biasing the classifier towards scales that have a larger number of descriptors. 
The final form of our NBNL classifier is given by:

    f_𝙵𝙲𝙽 𝙽𝙱𝙽𝙻(x; W,θ)=_y∈ Yh(x; W_y,θ) ,

where h defined below measures the likelihood of x given prototypes in W_y:

    h(x; W_y,θ)=1/m∑_x̂∈scale(x)ω̅(x̂; W_y,θ)

and ω̅ is the scale-specific normalized score:

    ω̅(x̂; W_y,θ)=1/η(x̂)∑_z∈ g_𝙵𝙲𝙽(x̂;θ)ω(z; W_y) .

This normalization step is necessary to prevent scales that generate many descriptors to bias the final likelihood.


To train our network we define the following regularized empirical risk with respect to both the classifiers' parameters W and the FC-CNN's parameters θ:

    R( W,θ; T)=1/ T∑_(x,y)∈ Tℓ(h(x; W,θ), y) + λΩ( W,θ) .

Here, h(x; W,θ)={h(x; W_y,θ)}_y∈ Y, Ω is a  ℓ_2-regularizer acting on all the networks' parameters, and ℓ(u,y) with u={u_y}_y∈ Y, u_y∈ℝ, is the following loss function:

    ℓ(u,y)=-u_y+log∑_y∈ Ye^u_y ,

obtained from the composition of the log-loss with the soft-max operator.

Following <cit.> we actually do not minimize directly R( W,θ; T) as defined above, but replace the loss terms with the following upper-bound, which is obtained by application of Jensen's inequality:

    ℓ(h(x; W,θ), y)≤1/m∑_x̂∈scale(x)1/η(x̂)∑_z∈ g_𝙵𝙲𝙽(x̂;θ)ℓ(ω(z, W),y) ,

with ω(z, W)={ω(z, W_y)}_y∈ Y. This is equivalent to promoting descriptors to training samples, as in <cit.>.


















 











§ EXPERIMENTAL RESULTS
 


In this section, we evaluate the performance of our approach. In Sect. <ref> we compare against the method in <cit.>, demonstrating the advantages of our end-to-end learning framework. 
In Sect. <ref> we assess the effectiveness of the proposed approach for  the place categorization task, considering images acquired from different robotic platforms in various indoor environments, comparing with state-of-the-art 

approaches. Finally, we demonstrate the robustness of our model to different environmental conditions and sensors (Sect. <ref>) and to occlusions and image perturbations (Sect. <ref>). Our evaluation has been performed using NVIDIA GeForce 1070 GTX GPU, implementing our approach with the popular Caffe framework <cit.>.



 §.§ Comparison with Holistic and Part-based CNN models



In a first series of experiments 

we demonstrate the advantages of the proposed part-based model and compare it with (i) its non end-to-end counterpart (i.e, the CNN-NBNL method in <cit.>) and (ii) traditional CNN-based approaches not accounting for local representations. 



To implement <cit.>

following the original paper, we split the input image into multiple patches, extracting features from the last fully-connected layer of a pre-trained CNN. The patches were extracted at three different scales (32,64,128 pixels) after the original image was rescaled (longest side 200 pixels). We adopted the sparse protocol in <cit.>, based on which features from 100 random patches are extracted. The features are equally distributed between the three scales and an additional descriptor representing the full image is considered.
As representative for deep models based on holistic representations, we chose the 


successful approach of Zhou et al. <cit.>: they pre-train a CNN on huge datasets (i.e., ImageNet <cit.>, Places <cit.> or both in the hybrid configuration) and used it as features extractor for learning a linear SVM model. Note that this is a strong baseline, widely used in the computer vision community for scene recognition tasks. 



To demonstrate the generality of our contribution, we tested all models considering three different base networks: 
the Caffe <cit.> version of AlexNet <cit.>, VGG-16 <cit.> and GoogLeNet <cit.>. For AlexNet and VGG-16 we considered the networks pre-trained on both Places <cit.> and ImageNet <cit.> datasets (i.e, the hybrid configuration). For GoogLeNet no pre-trained hybrid network was available, thus we took the model pre-trained on Places365. 
In order to fairly compare our model with the baseline method in <cit.>, our fully-convolutional network was designed to match the resolution of local patches adopted in <cit.>.

To accomplish this, since a 128x128 patch covers 64% of a 200x200 image, we rescaled the input image such that the receptive fields 
correspond to approximately 64% of the input (355 pxls for CaffeNet and 350 pxls for VGG and GoogLeNet). The other scale features were obtained by upsampling the image twice with a deconvolutional layer. We extracted 25 local features for the larger scale (128x128 pxls), 36 for the medium and 49 for the smallest, for a total of 110 local descriptors. These number of features were obtained by regulating the stride of the last layers of the network. 
As in <cit.>, we extracted features at the last fully-connected layer level, applying batch normalization <cit.> before the classifier. Since the datasets considered in our evaluation have small/medium dimensions, fine-tuning was performed only in the last two layers of the network.  
The networks were trained with a fixed learning rate which was decreased twice of a factor 0.1. 
To decide the proper learning rate schedule and number of epochs, we performed parameters tuning on a separate validation set.  
As parameters of the NBNL classifier, we chose k=10 and p=2, applying a weight decay of 10^-5 on the prototypes. 

Notice that in our model we considered 110 descriptors, while 100 
were used for the baseline method in <cit.>. However, we experimentally verified that a difference of 10 descriptors does not influence performance. This confirms previous findings in <cit.>, where Kuzborskij also tested their approach with a dense configuration employing 400 
patches without significant improvements in accuracy over the sampling protocol. 

We performed experiments on three different datasets, previously used in <cit.>: 
Sports8 <cit.>, Scene15 <cit.> and MIT67 <cit.>. The Sports8 dataset <cit.> contains 8 different indoor and outdoor sport scenes (rowing, badminton, polo, bocce, snowboarding, croquet, sailing and rock climbing). The number of images per category ranges from 137 to 200. We followed the common experimental setting, taking 70 images per class for training and 60 for testing. The Scene15 dataset <cit.> is composed by different categories of outdoor and indoor scenes. It contains a maximum of 400 gray scale images per category. We considered the standard protocol, taking 100 images for training and 100 for testing for each class. 
The MIT67 <cit.> is a common benchmark for indoor scene recognition. It contains images of 67 indoor scenes, with at least 100 images per class. We adopted the common experimental setting, using 80 images per class for training and 20 for testing.
For each dataset we took 5 random splits reporting the results as mean and standard deviation. 

Tab. <ref> shows the results of our evaluation. Mean and standard deviation are provided for our approach and <cit.>, while for the CNN models in <cit.> we report results from the original papers. 
From the table it is clear that, for all base networks and datasets, our method outperforms the baselines. These results confirm the significant advantage of the proposed part-based approach over traditional CNN architectures which do not consider local representations. Moreover, our results show that our end-to-end training model guarantees an improvement in performance compared to its non end-to-end counterpart CNN-NBNL. This improvement is mostly due to the proposed end-to-end training strategy. A pre-trained network is able to extract powerful features, but they are not always discriminative when applied to specific tasks. On the other hand, end-to-end training allows to overcome this limitation by adapting the pre-trained features to a new target task, producing class discriminative representations. This is shown in Fig. <ref> where we plot the fc7 features extracted at scale 64x64 pixels (t-SNE visualizations <cit.>) with CNN-NBNL (Fig. <ref>.a) and with our approach (Fig. <ref>.b): while a pre-trained network fails at creating discriminative local features, our model is able to learn representations that cluster accordingly to class labels.   





To further compare our approach and CNN-NBNL <cit.> we also analyzed the computational time required during the test phase to process an increasing number of patches. Fig. <ref> report the results of our analysis: as expected, our fully-convolutional architecture is greatly advantageous over the CNN-NBNL model which extract local features independently patch-by-patch. 


We 
remark that reduced classification time is a fundamental for the adoption of the proposed model in robotic platforms operating in real environments. 










 §.§ Robot Place Categorization

In this section we show the results of our evaluation when testing the proposed approach on publicly available robot vision datasets. These experiments aim at 
verifying the effectiveness of our fully-convolutional network and its robustness to varying environmental conditions and occlusions. 




  §.§.§ COLD dataset




We first tested our method on the COsy Localization Database (COLD) database <cit.>. This database contains three datasets of indoor scenes acquired in three different laboratories from different robots. The COLD-Freiburg contains 26 image sequences collected in the Autonomous Intelligent Systems Laboratory at the University of Freiburg with a camera mounted on an ActivMedia Pioneer-3 robot. COLD-Ljubljana contains 18 image sequences acquired from the camera of an iRobot ATRV-Mini platform at the Visual Cognitive Systems Laboratory of University of Ljubljana. In the COLD-Saarbrücken an ActivMedia PeopleBot has been employed to gather 29 image sequences inside the Language Technology Laboratory at the German Research Center for Artificial Intelligence in Saarbrücken. 

In our experiments we followed the protocol described in Rubio et al. <cit.>, considering images of path 2 of each laboratory. These data depict significant changes with respect to illumination conditions and time of data acquisition. Using path 2, there are 9 categories for COLD-Freiburg, 6 for COLD-Ljubljana and 8 for COLD-Saarbrücken. We trained and tested on data collected on the same laboratory, considering 5 random splits and reporting the average values. We compared our model with the methods proposed in <cit.>, since this work is one of the most recent studies adopting this dataset. In <cit.>, Rubio et al. proposed to extract HOG features and to apply a dimensionality reduction technique 
before providing the features as input to different classifiers.  
As classifiers they considered linear SVM, 
Naïve Bayes (NB), Bayesian Network (BN) and the Tree Augmented Naïve Bayes (TAN). 
In our experiments, to train our model we adopted the same setting

described in Sect. <ref>, fine-tuning the last two layers of the network.

The results are shown in Tab. <ref>. Our model outperforms all the baselines in <cit.>, confirming the advantage of CNN-based approaches over traditional classifiers and hand crafted features. The high accuracy of our method also demonstrates that the proposed fully-convolutional network 
is highly effective at discerning among different rooms, even with significant lighting and environmental changes.






  §.§.§ KTH-IDOL dataset





To further assess the ability of the proposed method to 
generalize across different robotics platforms and illumination conditions, 

we performed experiments on the KTH Image Database for rObot Localization (KTH-IDOL) <cit.>. This dataset contains 12 image sequences collected by two robots (Dumbo and Minnie) on 5 different rooms. The image sequences were collected along several days on three different illumination conditions: sunny, cloudy and night. Following <cit.> we considered the first two sequences for each robot and weather condition, performing three different type of tests. First, we trained and tested using the same robot and same weather condition with one sequence used for training and the other for testing and vice-versa. As a second experiment, we used the same robot for training and testing, varying the weather conditions of the two sets. In the last experiment we trained the classifier with the same weather condition but testing it on a different robot. Notice that, differently from Sect. <ref>, in this case the illumination changes are not present in the training set. Our model is trained with the same setting of Sect. <ref>. In this case, to reduce overfitting and improve the capability of our network, we apply data augmentation to the RGB channels, following the standard procedure introduced in <cit.>.

We compared our method with three state-of-the-art approaches: (i) <cit.> which used high dimensional histogram global features as input for a χ^2 kernel SVM; (ii) <cit.> which proposed the CENTRIST descriptor and performed nearest neighbor classification and (iii) <cit.> which used again the nearest neighbor classifier but with Histogram of Oriented Uniform Patterns (HOUP) as features. 

Tab. <ref> shows the results of our evaluation. Our method outperforms all the baselines in the first and third series of experiments (same lighting). In particular, the large improvements in performance in the third experiment clearly demonstrates its ability to generalize over different input representations of the same scene, independently of the camera mounted on the robot. These results suggest that it should be possible to train offline our model and apply it on arbitrary robotic platforms. On the second experiment,
while the high classification accuracy demonstrates a significant robustness to lighting variations, our model achieves comparable performance with previous works, showing a small advantage of CNN representations over traditional methods in case of illumination changes. 







  §.§.§ Household room dataset



In the last series of experiments we tested the robustness of our model with respect to occlusions. We evaluate the performance of our approach on the recently introduced household room (or MIT8) dataset <cit.>. This dataset is a subset of MIT67 which contains 8 room categories: bathroom, bedroom, children room, closet, corridor, dining room, kitchen and living room. We used the setting provided in <cit.>, with 641 images for training and 155 for testing. The challenge proposed by Uršič et al. <cit.> is to train the model on the original images and test its performances on various noisy conditions. The conditions are: occlusion in the center of the image, occlusion on the right border, occlusions from a person, addition of an outside border, upside down rotation and cuts on the top or right part of the image (inducing aspect ratio changes). All the test sets were produced following the protocol in <cit.>, apart from the person occlusions set provided directly by the authors.

We compare our approach with the part-based model developed by Uršič et al. <cit.> and the global CNN-based model in <cit.>. In <cit.> selective search is used to extract informative regions inside the image, which are then provided as input to a pre-trained CNN. From these features, exemplar parts are learned for each category and used by a part-based mixture model for the final classification. The standard hybrid CaffeNet <cit.> is employed as CNN architecture. For a fair comparison we adopted the same base architecture, extracting features at the last fully-connected layer before the classifier. In this case we used images rescaled to 256x256 as input, upsampling them twice to obtain descriptors at multiple scales. We extracted 45 descriptors, 4 for the smallest scale (256x256), 16 for the medium and 25 for the largest. The training procedure is the one described in Sect. <ref> and the same parameters are used for the NBNL classifier, with batch normalization applied to the last layer. We trained our model 10 times, computing the average accuracy. 

The results of the evaluation are reported in Tab. <ref>. As shown in the table, both our approach and the method in <cit.> achieve higher classification accuracy than the CNN model in <cit.>, confirming the benefit of part-based modeling. It is interesting to compare our approach with <cit.>: while our framework guarantees better performances in certain conditions (e.g. original frames, person occlusion), the method in <cit.> is more robust to changes of the aspect ratio (e.g. cuts in the image) and scale (e.g. outside border addition). Interestingly, when the occlusion is not created artificially obscuring patches (person occluder), our model achieves higher performance than <cit.>. Oppositely, in the case of the outside border experiments, almost half of the image is black and the real content reduces to a very small scale. In this (artificial) setting, <cit.> outperforms our model.


 


For sake of completeness, we also report the confusion matrix associated with our results on the original frames (Fig. <ref>).




	




§ CONCLUSIONS

We presented a novel deep learning architecture for addressing the semantic place categorization task. By seamlessly integrating the CNN and NBNN frameworks, our approach permits to learn local deep representations, enabling robust scene recognition. 
The effectiveness of the proposed method is demonstrated on various benchmarks. We show that our approach outperforms traditional CNN baselines and previous part-based models which use CNNs purely as features extractors. In robotics scenarios, our deep network achieves state-of-the-art results on three different benchmarks, demonstrating its robustness to occlusions, environmental changes and different sensors. As future work, we plan to extend this model in order to handle multimodal inputs (e.g. considering range sensors in addition to RGB cameras). 




                                  
                                  
                                  
                                  
                                  














IEEEtran



