abstract: This paper focuses on semantic scene completion, a task for producing a complete 3D voxel representation of volumetric occupancy and semantic labels for a scene from a single-view depth map observation. Previous work has considered scene completion and semantic labeling of depth maps separately. However, we observe that these two problems are tightly intertwined. To leverage the coupled nature of these two tasks, we introduce the semantic scene completion network (**SSCNet**), an end-to-end 3D convolutional network that takes a single depth image as input and simultaneously outputs occupancy and semantic labels for all voxels in the camera view frustum. Our network uses a dilation-based 3D context module to efficiently expand the receptive field and enable 3D context learning. To train our network, we construct SUNCG - a manually created large-scale dataset of synthetic 3D scenes with dense volumetric annotations. Our experiments demonstrate that the joint model outperforms methods addressing each task in isolation and outperforms alternative approaches on the semantic scene completion task. The dataset, code and pretrained model will be available online upon acceptance.
author: Shuran Song Fisher Yu Andy Zeng Angel X. Chang Manolis Savva Thomas Funkhouser
Princeton University
bibliography: RGBDdetect.bib; pvg.bib; ivs-eccv16.bib
title: SSCNet for Simultaneous Scene Completion and Semantic Labeling; Semantic Scene Completion from a Single Depth Image

# Introduction

We live in a 3D world where empty and occupied space is determined by the physical presence of objects. To successfully navigate within and interact with the world, we rely on an understanding of both the 3D geometry and the semantics of the environment. Similarly, for a robot, the ability to infer complete 3D shape from partial observations is necessary for low-level tasks such as grasping and obstacle avoidance , while the ability to infer the semantic meaning of objects in the scene enables high-level tasks such as retrieval of objects.

With this motivation, our goal is to have a model that predicts both volumetric occupancy (i.e., scene completion) and object category (i.e., scene labeling) from a single depth image of a 3D scene — in this paper we refer to this task as **semantic scene completion** (). Prior work is limited to address only part of this problem as shown in : RGB-D segmentation approaches consider only visible surfaces without the full 3D shape , while shape completion approaches consider only geometry without semantics or a single object out of context .

Our key observation is that the occupancy patterns of the environment and the semantic labels of the objects are tightly intertwined. Therefore, the two problems of predicting voxel occupancy and identifying object semantics are strongly coupled. In other words, knowing the identity of an object helps us predict what areas of the scene it is likely to occupy without direct observation (e.g., seeing the top of a chair behind a table and inferring the presence of a seat and legs). Likewise, having an accurate occupancy pattern for an object helps us recognize its semantic class.

To leverage the coupled nature of the two tasks we jointly train a deep neural network using supervision targeted at both tasks. Given a single-view depth map as input, our semantic scene completion network (**SSCNet**) produces one of N+1 labels for all voxels in the view frustum. Each voxel is labeled as occupied by one of N object categories or free space. Most critically, this prediction extends beyond the projected surface implied by the depth map, thus providing occupancy information for the entire scene.

To achieve this goal there are several issues that must be addressed. First, how do we effectively capture contextual information from 3D volumetric data, where the signal is sparse and lacks high frequency detail? Second, since existing RGB-D datasets only provide annotations on visible surfaces, how do we obtain training data with complete volumetric annotations at scene level?

To address the first issue, we design a 3D dilation-based context module that efficiently expands our network's receptive field to model the contextual information. We find that a big receptive field is crucial for the task. As demonstrated in , looking at the small region of a chair in isolation, it is hard to recognize and complete the chair. However, if we consider the context due to surrounding objects, such as the table and floor, the problem is much easier.

To address the second issue, we construct SUNCG, a large-scale synthetic 3D scene dataset with more than 45622 indoor environments designed by people. All the 3D scenes are composed of individually labeled 3D object meshes, from which we can compute 3D scene volumes with dense object labels though voxelization.

Our experiments with these solutions demonstrate that a method that jointly predicts volumetric occupancy and object semantic can outperform methods addressing each task in isolation. Both the 3D context model learned by our network and the large-scale synthetic training data help to improve performance significantly.

Our main contribution is to formulate an end-to-end 3D ConvNet model (SSCNet) for the joint task of volumetric scene completion and semantic labeling. In support of that goal, we design a dilation-based 3D context module that enables efficient context learning with large receptive fields. To provide the training data for our network, we introduce SUNCG, a manually created large-scale dataset of synthetic 3D scenes with dense occupancy and semantic annotations.

# Related work

We review related work on RGB-D segmentation, 3D shape completion, and voxel space semantic labeling.

#### RGB-D semantic segmentation.

Many prior works focus on RGB-D image segmentation . However, those methods focus on obtaining semantic labels for only the observed pixels without considering the full shape of the object, and hence cannot directly perform scene completion or predict labels beyond the visible surface.

#### Shape completion.

Other prior works focus on single object shape completion . To apply those methods to scenes, additional segmentation or object masks would be required. For scene completion, when the missing regions are relatively small, methods using plane fitting or object symmetry can be applied to fill in holes. However, these methods heavily rely on the regularity of the geometry and often fail when the missing regions are big. Firman show promising completing results on scenes. However, their approach is based purely on geometry without semantics, and thus it produces less accurate results when the scene structure becomes complex.

#### 3D model fitting.

One possible approach to obtain the complete geometry and semantic labels for a scene is to retrieve and fit instance-level 3D mesh models to the observed depth map . However, the prediction quality of this type of approach is limited by the quality and variety of 3D models available for retrieval. Naturally, observed objects that cannot be explained by the available models tend to be missed. Or, if the 3D model library is large enough to include all observations, then a difficult retrieval and alignment problem must be solved. Alternatively, it is possible to use 3D primitives such as bounding boxes to approximate the 3D geometry of objects . However, the bounding box approximation limits the geometric detail of the output predictions.

#### Voxel space reasoning.

Another line of work completes and labels 3D scenes, but with separate modules for feature extraction and context modeling. Zheng   predict the unobserved voxels by physical reasoning. Kim   train a Voxel-CRF model from labeled floor plans to optimize the semantic labeling and reconstruction for indoor scenes. Hane   and Blaha use joint optimization for multi-view reconstruction and segmentation for outdoor scenes. However, this line of work uses predefined features, and separates the feature learning from the context modeling, and it is expensive for CRF-based models to encode long-range contextual information. In contrast, our model is able to jointly learn the low-level feature representation and high-level contextual information end-to-end from large-scale 3D scene data, directly modeling long-range contextual cues though big receptive field.

#### Learning from synthetic scene data.

Our paper leverages data generated from a large-scale synthetic 3D scene dataset. Although recent works have been focusing on generating segmentation labels for 2D image through rendering synthetic scenes , the 3D aspect of such data has not been fully utilized. Existing datasets focus either on objects or a small number of rooms (57 rooms in ). In contrast, our dataset is several orders of magnitude larger than existing 3D scene datasets (45,622 houses with 775,574 rooms) providing a diverse set of furniture arrangements manually created by people.

# Semantic scene completion network

Given a single-view depth map observation of a 3D scene, the goal of our semantic scene completion network is to map the voxels in the view frustum to one of the class labels $C = \{c_{0}, ... c_{N+1}\}$, where N is number of object classes and $c_{0}$ represents empty voxels. During training, we render depth maps from virtual viewpoints of our synthetic 3D scenes and voxelize the full 3D scenes with object labels as ground truth. During testing, the observation depth images come from a RGB-D camera.

shows an overview of our processing pipeline. We take a single depth map as input and encode it as a 3D volume. This 3D volume is then fed into a 3D convolutional network, which extracts and aggregates both local geometric and contextual information. The network produces the probability distribution of voxel occupancy and object categories for all voxels inside the camera view frustum.

The following subsections describe the core issues addressed in the design of the system: the data encoding (), network architecture () and training data generation ().

## Volumetric data encoding

The first issue we need to address is how to encode the observed depth as input to the network. For the semantic scene completion task, the ideal encoding should directly represent the 2D observation into the same 3D physical space as the output in a way that is invariant to the viewpoint projection, and provide a meaningful signal for the network to learn geometry and scene representation. To this end, we adopt Truncated Signed Distance Function (TSDF) to encode the 3D space, where every voxel stores the distance value $d$ to its closest surface, and the sign of the value indicates whether the voxel is in free space or in occluded space. To better suit our task, we make the following modifications to the standard TSDF.

#### Eliminate view dependency.

Most RGB-D reconstruction pipelines speed up the TSDF computation by using the projective TSDF which finds the closest surface points only in the line of sight of the camera . This projective TSDF is fast to compute, but is inherently view-dependent. Instead, we choose to compute the distance to the closest point anywhere on the full observed surface.

#### Eliminate strong gradients in empty space.

Another issue with TSDF is that strong gradients occur in the empty space along the occlusion boundary between $\pm d_\mathrm{max}$. It is possible to eliminate this gradient by removing the sign, however, the sign is important for completion task since it indicates the occluded regions of the scene that need to be completed. To solve this problem we flip the TSDF value $d$ as follows: $d_\mathrm{flipped} = \mathop{\mathrm{sign}}(d)(d_\mathrm{max}-d)$. This flipped TSDF has the strong gradient on surface, providing a more meaningful signal for the network to learn better geometric features. The different encoding is visualized in , and shows its impact on performance.

## Network architecture

The network architecture of SSCNet is shown in . Taking a high-resolution 3D volume as input, the network first uses several 3D convolution layers to learn a local geometry representation. We use convolution layers with stride and pooling layers to reduce the resolution to one fourth of original input. Then, we use a dilation-based 3D context module to capture higher-level inter-object contextual information. After that, the network responses from different scales are concatenated and fed into two more convolution layers to aggregate information from multiple scales. At the end, a voxel-wise softmax layer is used to predict the final voxel label. Several shortcut connections are added for better gradient propagation. In implementing this architecture, we made the following design decisions:

#### Input volume generation.

Given a 3D scene, we rotate it to align with gravity and room orientation based on Manhattan assumption. The dimensions of the 3D space we consider are $4.8\ensuremath{\,\mathrm{m}}$ horizontally, $2.88\ensuremath{\,\mathrm{m}}$ vertically, and $4.8\ensuremath{\,\mathrm{m}}$ in depth. We encode the 3D scene into a flipped TSDF with grid size $0.02\ensuremath{\,\mathrm{m}}$, truncation value $0.24\ensuremath{\,\mathrm{m}}$, resulting in a $240 \times 144 \times 240$ volume as the network input.

#### Capturing 3D context with big receptive field.

Context can provide valuable information for understanding the scene, as demonstrated by much prior work in image segmentation . In the 3D domain, context is more useful due to a lack of high frequency signals compared to image textures. For example, tabletops, beds, and floors are all geometrically similar to flat horizontal surfaces, so it is hard to distinguish them given only local geometry. However, the relative positions of objects in the scene are a powerful discriminatory signal. To learn this contextual information, we need to make sure our network has a big enough receptive field. To this end, we extend the dilated convolution presented by Yu and Koltun  to 3D. Dilated convolution extends normal convolution by adding a step size when the convolution extracts values from the input before convolving with the kernel. Thus we can exponentially expand the receptive field without a loss of resolution or coverage, while still using the same number of parameters. compares the receptive field size of SSCNet with 3D ConvNet architectures from prior work.

#### Multi-scale context aggregation.

Different object categories have very different physical 3D sizes. This implies that the network will need to capture information at different scales in order to recognize objects reliably. For example, we need more local information to recognize smaller objects like TVs, while we need more global information to recognize bigger objects like beds. In order to aggregate information at different scales we add a layer that concatenates the network responses with different receptive field. We then feed this combined feature map into two $1\times1\times1$ convolution layers, which allows us to propagate information across responses from different scales.

#### Data balancing.

Due to the sparsity of 3D data, the ratio of empty vs. occupied voxels is around 9:1. To deal with this imbalanced data distribution, we sample the training so that each mini-batch has a balanced set of empty and occupied examples. For each training volume containing $N$ occupied voxels, we randomly sample $2N$ empty voxels from occluded regions for training. Voxels in free space, outside the field of view, or outside the room are ignored.

#### Loss: voxel-wise softmax.

The loss function of the network is the sum of voxel-wise softmax loss $L(p,y) = \sum\limits_{i,j,k}{w_{ijk}L_\mathrm{sm}(p_{ijk},y_{ijk})}$, where $L_\mathrm{sm}$ is softmax loss, $y_{ijk}$ is the ground truth label, $p_{ijk}$ is the predicted probability of the voxel at coordinates $(i,j,k)$ over the $N+1$ classes, where $N$ is the number of object categories and empty voxels are labeled as class $0$. The weight $w_{ijk}$ is equal to zero or one based on the sampling algorithm described above.

#### Training protocol.

We implement our network architecture in Caffe . Pre-training SSCNet on the SUNCG training set takes around a week on a Tesla K40 GPU, and fine-tuning on the NYU dataset takes 30 hours. During training, each mini-batch contains one 3D view volume, requiring $11\ensuremath{\,\mathrm{GB}}$ of GPU memory. To obtain more stable gradient estimates, we accumulate gradients over four iterations and update the weights once afterwards.

# Synthesizing training data

One of the main obstacles of training deep networks for scene-level dense 3D predictions is the lack of large annotated datasets with dense object semantic annotations at the voxel level. Existing RGB-D datasets with surface reconstructions are subject to occlusions or partial observations, and cannot provide the volumetric occupancy and semantic labels for the entire space at the voxel level. To obtain volumetric occupancy ground truth Firman   collect a tabletop dataset with reconstructed RGB-D video using KinectFusion . However, this data does not provide semantic labels, and only contains simple tabletop scenarios. In this paper, we present a new large-scale synthetic 3D scene dataset, from which we obtain a large amount of training data with synthetically rendered depth images and volumetric ground truth.

## SUNCG: a large-scale synthetic scene dataset

Our SUNCG dataset contains $45,622$ different scenes with realistic room and furniture layouts that are manually created though the Planner5D platform . Planner5D is an online interior design interface that allows users to create multi-floor room layouts, add furniture from a object library, and arrange them in the rooms. After removing duplicated and empty scenes, we ensured the quality of the data with a simple Mechanical Turk cleaning task. During the task, we show a set of top view renderings of each floor and ask turkers to vote whether this is a valid apartment floor. We collect three votes for each floor, and consider a floor valid when it has at least two positive votes. In the end, we have $49,884$ valid floors, with contain $404,058$ rooms and $5,697,217$ object instances from $2644$ unique object meshes covering $84$ categories. We manually labeled the all objects in the library to assign category labels. shows example scenes from the resulting SUNCG dataset. More information can be found in the appendix.

```latex
\begin{table*}[t]\vspace{-3mm}
    \setlength{\tabcolsep}{4.3 pt}
    \centering\small
    \begin{tabular}{l|ccc|cccccccccccc}
    \toprule
     & \multicolumn{3}{c|}{scene completion}  & \multicolumn{12}{c}{semantic scene completion } \tabularnewline
    %\cmidrule(lr){3-5}
    %\cmidrule(lr){6-17}
    \hline
    method (train) & prec. & recall & IoU & ceil.  & floor & wall & win. & chair & bed & sofa & table & tvs & furn. & objs. & avg. \tabularnewline
    \hline
    Lin \etal (NYU) \cite{lin2013holistic} & 58.5 &49.9 &36.4&0&11.7&13.3&{\bf 14.1}&9.4&29&24&6.0&7.0&16.2&1.1&12.0\tabularnewline
    Geiger and Wang (NYU) \cite{Geiger2015GCPR} & 65.7 &58&44.4&10.2&62.5&19.1&5.8&8.5&40.6&27.7&7.0&6.0&22.6&5.9&19.6\tabularnewline \hline
    SSCNet(NYU) & 57.0&{\bf 94.5}&55.1&15.1&{\bf 94.7}&24.4&0&12.6&32.1&35&13&7.8&27.1&10.1&24.7\tabularnewline
    SSCNet(SUNCG) & 55.6&91.9&53.2&5.8&81.8&19.6&5.4&12.9&34.4&26&13.6&6.1&9.4&7.4&20.2\tabularnewline
    SSCNet(SUNCG+NYU) & {\bf 59.3}&92.9&{\bf 56.6}&{\bf 15.1} & 94.6 &{\bf 24.7} &10.8&{\bf 17.3} &{\bf 53.2} &{\bf 45.9}&{\bf 15.9}&{\bf 13.9}&{\bf 31.1} &{\bf 12.6} &{\bf 30.5}\tabularnewline
    %\hline  \OURS  (SUNCG+ NYU)&render&74.2&97.0&72.5&38.7&92.9&47.7&9.1&31.1&56.5&57.8&32.4&11&43.2&26.3&40.6\tabularnewline
    %\hline
    \bottomrule
    \end{tabular}
    \caption{{\bf Semantic scene completion results on the NYU test set with kinect depth map.}}
    \label{table:full3dnyu}
\end{table*}
```

## Synthetic depth map generation

To generate synthetic depth maps that mimic a typical image capturing process, we use a set of simple heuristics to pick camera viewpoints. Given a 3D scene, we start with a uniform grid of locations spaced at $1\ensuremath{\,\mathrm{m}}$ intervals on the floor and not occupied by objects. We then choose camera poses based on the distribution of the NYU-Depth v2 dataset.[^1] Then, we render the depth map using the intrinsics and resolution of the Kinect. After that we use a set of simple heuristics to exclude bad viewpoints. Specifically, a rendered view is considered valid if it satisfies the following three criteria: a) valid depth area (depth values in range of $1\ensuremath{\,\mathrm{m}}$ to $8\ensuremath{\,\mathrm{m}}$) larger than $70\%$ of image area, b) there are more than two object categories apart from wall, ceiling, floor, and c) object area apart from wall, ceiling, floor is larger than $30\%$ of image area. To reduce data redundancy, we pick at most five images from each room. In total we generate $130,269$ valid views for training our SSCNet.

## Volumetric ground truth generation

Since the 3D scenes in the SUNCG dataset consist of a limited number of object instances, we speed up the voxelization process by first voxelizing each individual object in the library and then transforming the labels based on each scene configuration and view point. Specifically, we first voxelize each object to a ${128}\times{128}\times{128}$ voxel grid. We set the voxel size $s$ so that the largest dimension of the object is a tight fit to the object bounding box. Thus, $s$ varies between objects due to the difference in object dimensions. We use the binvox  voxelizer which accounts for both surface and interior voxels by using a space carving approach.

Given a camera view, we define a ${240}\times{144}\times{240}$ voxel grid in world coordinates, with scene voxel size equals to $2\ensuremath{\,\mathrm{cm}}$. Then for each object in the scene, we transform the object voxel grid by translating, rotating and scaling by the object's transformation. We then iterate over each voxel in the scene voxel grid that is inside the transformed object bounding box, and calculate the distance to the nearest neighbor object voxel. If the distance is smaller than the object voxel size $s$, this scene voxel will be labeled with this object category. Similarly, we label all voxels in the scene that belong to walls, floors, and ceilings by treating them as planes with thickness equal to one scene voxel size. All remaining voxels are marked as empty space, therefore providing a fully labeled voxel grid for the camera view.

# Evaluation

In this section, we evaluate our proposed methods with a comparison to alternative approaches and an ablation study to better understand the proposed model. We evaluate our algorithm on both real and synthetic datasets.

For the real data, we use the NYU dataset , which contains 1449 depth maps captured from Kinect. We obtain the ground truth by voxelizing the 3D mesh annotations from Guo  , mapping object categories based on Handa  . The annotations consist of 33 object meshes in 7 categories, other categories approximated using 3D boxes or planes. In some cases, the mesh annotation is not perfectly aligned with depth due to labeling error and the limited set of meshes. To deal with this misalignment, Firman at el.  propose to use rendered depth map from the annotation for testing. However, by rendering the overly simplified meshes, geometric detail is lost especially in cases where objects are represented as a box. Therefore, we test with both rendered depth maps and the originals.

For synthetic data, we created a test set from SUNCG which has objects with detailed geometry, and for which the depth map and ground truth volumes are perfectly aligned. The SUNCG test set consists of 500 depth images rendered from 184 scenes that are not in the training set.

#### Evaluation metric.

As our evaluation metric, we use the voxel-level intersection over union (IoU) of predicted voxel labels compared to ground truth labels. For the semantic scene completion task, we evaluate the IoU of each object classes on both the observed and occluded voxels. For the scene completion task, we treat all non-empty object class as one category and evaluate IoU of the binary predictions on occluded voxels. Following Firman  , we do not evaluate on voxels outside the view or the room.

| method            | training  |  prec.   |  recall  |   IoU    |
|:------------------|:----------|:--------:|:--------:|:--------:|
| Zheng             | NYU       |   60.1   |   46.7   |   34.6   |
| Firman            | NYU       |   66.5   |   69.7   |   50.8   |
| SSCNet completion | NYU       |   66.3   |   96.9   |   64.8   |
| SSCNet joint      | NYU       |   75.0   |   92.3   |   70.3   |
| SSCNet joint      | SUNCG+NYU | **75.0** | **96.0** | **73.0** |

**Scene completion on the rendered NYU test set as **

## Experimental results

summarize the quantitative results and shows qualitative comparisons. More qualitative results can be found in the appendix .

#### Comparison to alternative approaches.

In we compare on the semantic scene completion task with approaches from Lin   and Geiger and Wang . Both these algorithms take an RGB-D frame as input and produce object labels in the 3D scene. Lin use 3D bounding boxes and planes to approximate all objects. Geiger and Wang retrieve and fit 3D mesh models to the observed depth map at test time. The mesh model library used for retrieval is a superset of the models used for ground truth annotations. Therefore, they can achieve perfect alignments by finding the exact mesh model in a small database. In contrast, our algorithm is based on only depth and does not use additional mesh model at test time. Despite this data disparity, our network produces more accurate voxel-level predictions (30.5% vs. 19.6%). An example of the difference is shown in the third row of : both Lin and Geiger and Wang's approaches confuse the sofa as a bed while our network correctly recognizes the sofa. Moreover, since our method does not require the model fitting step it is much faster at 7s compared to 127s per image .

#### Does recognizing objects help scene completion?

Previous work has shown scene completion is possible without semantic understanding. We examine to what extent the supervision of object semantics benefits the scene completion task. To do this, we trained a model predicting the occupancy of each voxel by doing binary classification on each voxel ("empty" vs. "occupied"). We compare the performance of models trained with occupancy and multi-class labeling (see \[completion\] vs. \[joint\]). We also compare with Firmal   and Zheng   which both predict binary voxel occupancy based on a single depth map without semantic understanding of the scene. We use the re-implementation of Zheng 's approach from Firman , which only provides the completion result. We evaluate on the rendered NYU benchmark with the same test images used by Firman at al. (randomly picked 200 images from the full test set). While Firman produces good results for many cases, their approach fails when the scene becomes complex. For instance, their algorithm fails to complete half of the bed in the first row of , and also fails to complete the chairs in the fifth row. In contrast, SSCNet is able to better complete the geometry by leveraging the semantics of the 3D context. This result validates the idea that it is beneficial to understand object semantics in order to achieve better scene completion.

```latex
\begin{table*}[t]\vspace{-6mm}
    \setlength{\tabcolsep}{3.6pt}
    \centering\small
    \begin{tabular}{lc|c|ccc|cccccccccccc}
    \toprule
     &  &  & \multicolumn{3}{c|}{scene completion} & \multicolumn{12}{c}{semantic scene completion}\tabularnewline
    \hline
    method  & encoding &  eval & prec. & recall & IoU & ceil. & floor & wall & win. & chair & bed & sofa & table & tvs & furn. & objs. & avg.\tabularnewline
    \hline
    no completion&flipped&observed&-&-&-&97.2&95.5&61.9&24.6&30.1&55.3&58.9&48.7&14.8&42.1&34.5&51.2\tabularnewline
    joint&flipped&surface&-&-&-&97.7&94.5&66.4&30.0&36.9&60.2&62.5&56.3&12.1&46.7&33.0&54.2\tabularnewline\hline
    no balancing&flipped&&73.1&95.8&70.8&{\bf 96.4}&85.3&52.1&25.8&16.5&47.1&45.7&28.1&15.3&37.1&19.8&42.7\tabularnewline
    Basic&flipped&observed&73.4&95.0&70.7&94.6&83.8&47.0&24.0&15.1&38.2&37.2&26.0&0.0&34.8&17.3&38.0\tabularnewline
    Basic+D&flipped&and&72.2&96.2&70.4&94.7&{\bf 85.9}&47.5&{\bf 29.2}&21.1&50.9&50.7&29.0&{\bf 21.3} &37.2&20.1&44.3\tabularnewline
    Basic+D+M&proj&occluded&72.0&92.3&67.9&91.6&80.9&45.1&14.6&10.2&39.4&29.8&19.8&0.0&27.4&14.3&33.9\tabularnewline
    Basic+D+M&tsdf&voxels&74.8&94.0&71.4&95.8&84.4&45.1&17.5&15.2&28.2&37.2&25.6&0.0&28.2&21.9&36.3\tabularnewline
    Basic+D+M&flipped&&{\bf 76.3}&{\bf 95.2}&{\bf73.5}&96.3&84.9&{\bf 56.8}&28.2&{\bf 21.3}&{\bf 56.0} &{\bf 52.7} &{\bf 33.7} &10.9&{\bf 44.3} &{\bf 25.4}&{\bf 46.4}\tabularnewline
    \bottomrule
    \end{tabular}
    \caption{{\bf Ablation study on SUNCG testset.} First two row shows the evaluation on surface segmentation with and without joint training.
    The following rows show the evaluation on semantic scene completion task. D: 3D dilated convolution. M: multi-scale aggregation.
    }\label{table:control}
\end{table*}
```

#### Does scene completion help in recognizing objects?

To answer this question, we trained a model with a loss only accounting for semantic labels evaluated on the visible surface and compared with the model trained jointly with labeling and completion (see \[no completion\] vs. \[joint\]). Even when only evaluating on the visible surface, the model trained with the added supervision of the scene completion task outperforms the model trained only on surface labeling ($54.2\%$ vs. $51.2\%$). This demonstrates that understanding complete object geometry and the 3D context is beneficial for recognizing objects.

#### Does synthetic data help?

To investigate the effect of using synthetic training data, we compared models trained only with NYU and models pre-trained on SUNCG and then fine-tuned on NYU (see NYU vs. NYU+SUNCG). We see a performance gain by using additional synthetic data especially for the semantic scene completion task having an $10.3\%$ improvement in IoU.

#### Does a bigger receptive field help?

In , the networks labeled \[Basic\] and \[Basic+D\] have the same number of parameter, while in \[Basic+D\] three convolution layers are replaced by dilated convolution, increasing the receptive field from $1.16\ensuremath{\,\mathrm{m}}$ to $2.26\ensuremath{\,\mathrm{m}}$. Increasing the receptive field gives the network a opportunity to capture richer contextual information and significantly improve the network performance from $38.0\%$ to $44.3\%$. To visualize the contextual information learned by the network, we perform the following experiment: given a depth map of a single object we predict labels for all unobserved voxels. shows the input depth and the predictions. Even without observing depth information for other objects SSCNet hallucinates plausible contextual object based on the observed object.

#### Does multi-scale aggregation help?

Comparing the network performance with and without the aggregation layer (see \[Basic+D\] vs. \[Basic+D+M\]), we observe that the model with aggregation yields higher IoU for both the scene completion and semantic scene completion tasks by $3.1\%$ and $2.1\%$ respectively.

#### Do different encodings matter?

The last three rows in compare different volumetric encodings: projective TSDF \[proj.\], accurate TSDF \[tsdf\], and flipped TSDF \[flipped\]. We observe that removing the view dependency by using the accurate TSDF gives a $2.4\%$ improvement in IoU. Making the gradients concentrated on the surface with the flipped TSDF leads to a $10.1\%$ improvement.

#### Is data balancing necessary?

To balance the empty and occupied voxel examples, we proposed to sample the empty voxels during training. In , \[no balancing\] shows the performance when we remove the sampling process during training, where we see a drop in IoU from $46.4\%$ to $42.7\%$.

#### Limitations.

Firstly, we do not use any color information, so objects missing depth such as "windows" are hard to handle. This also leads to confusion between objects with similar geometry or functionality. For example, in the second row of the network predicts the desk as the broader furniture category. Secondly, due to the GPU memory constraints, our network output resolution is lower than that of input volume. This results in less detailed geometry and missing small objects, such as the missed objects on the desk of the second row in .

# Conclusion

In this paper, we introduced SSCNet, a 3D ConvNet for the semantic scene completion task of jointly predicting volumetric occupancy and semantic labels for full 3D scenes. We trained this network on a new large-scale synthetic 3D scene dataset. Experiment results demonstrate that our joint model outperforms methods addressing either component task in isolation, and that by leveraging the 3D contextual information and the synthetic training data, we significantly outperform alternative approaches on the semantic scene completion task.

#### Acknowledgment

This work is supported by Intel, Adobe, and NSF (IIS-1251217 and VEC 1539014/ 1539099). It makes use of data from Planner5D and hardware donated by NVIDIA and Intel. Shuran Song is supported by a Facebook Fellowship. Helpful advice was provided by Michael Firman, Yinda Zhang, Matthias Niessner and Angela Dai.

# SUNCG Dataset Statistics

In this section, we present several statistics related to our SUNCG dataset. We start by providing the basic statistics of scene structure and physical size for 3D scenes in our dataset, and then move on to talk about higher-level statistics regarding object categories, room types, and object-room relationships.

#### Scene Structure Statistics

Figure illustrates the distribution of number of rooms and number of floors per scene in the SUNCG dataset. The 3D scenes in our dataset are range from single room studio to multi-floor houses. The average and median number of rooms per-house are 8.9 and 7 respectively. The average and median number of floors per-house are 1.3 and 1 respectively.

#### Physical Size Statistics

All object meshes and 3D scenes in the SUNCG dataset are measured in real-world spatial dimensions (units are in meters). Figure shows statistics related to physical size over three levels: rooms, floors and houses.

#### Room Type Statistics

Figure shows the room type distribution and several example rooms per type from our dataset. In total, we have 24 room types that are labeled by the user during creation. These labels include: living room, kitchen, bedroom, child room, dining room, bathroom, toilet, hall, hallway, office, guest room, wardrobe, room, lobby, storage, boiler room, balcony, loggia, terrace, entryway, passenger elevator, freight elevator, aeration, and garage. The four most common room types in our dataset are bedroom, living room, kitchen and toilet, which agrees with the distribution in real-world living spaces.

#### Object Category Statistics

Figure shows overall object category occurrence in the SUNCG dataset. Figure shows examples of object models from the object library, which contains a diverse set of common furniture and objects for common living spaces. Furthermore, during the creation of the 3D scenes, users have the flexibility to reshape, resize, and re-apply texture to objects to better fit the room style, which further improves the dataset diversity.

#### Object-Room Relationships

With complete object and room type annotations, we can further study the object-room relationships in our dataset. Figure (a) shows the distribution of number of objects per room. Figure (b) shows the distribution of object categories conditioned on different room types. On average there are more than 14 objects in each room. The occurrence and arrangements of these objects in rooms provide rich contextual information that we can learn from.

[^1]: The camera height is sampled from a Gaussian distribution with $\mu = 1.5\ensuremath{\,\mathrm{m}}$ and $\sigma = 0.1\ensuremath{\,\mathrm{m}}$. The camera tilt angle is sampled from a Gaussian distribution with $\mu = -10\degree$ and $\sigma = 5\degree$.


abstract: This paper presents an approach for automation of interpretable feature selection for Internet Of Things Analytics (IoTA) using machine learning (ML) techniques. Authors have conducted a survey over different people involved in different IoTA based application development tasks. The survey reveals that feature selection is the most time consuming and niche skill demanding part of the entire workflow. This paper shows how feature selection is successfully automated without sacrificing the decision making accuracy and thereby reducing the project completion time and cost of hiring expensive resources. Several pattern recognition principles and state of art (SoA) ML techniques are followed to design the overall approach for the proposed automation. Three data sets are considered to establish the proof-of-concept. Experimental results show that the proposed automation is able to reduce the time for feature selection to $2$ days instead of $4-6$ months which would have been required in absence of the automation. This reduction in time is achieved without any sacrifice in the accuracy of the decision making process. Proposed method is also compared against Multi Layer Perceptron (MLP) model as most of the state of the art works on IoTA uses MLP based Deep Learning. Moreover the feature selection method is compared against SoA feature reduction technique namely Principal Component Analysis (PCA) and its variants. The results obtained show that the proposed method is effective.
author: **Snehasis Banerjee**; **Tanushyam Chattopadhyay**; **Arpan Pal**; **Utpal Garain**
title: Automation of Feature Engineering for IoT Analytics

*Copyright retained by the authors.*

# Introduction

Internet of Things analytics (IoTA) involves a lot of applications in health, wellness, sustainability, transportation, smart city administration, and urban health while deploying any industrial sensor systems. Some of such applications in health care like blood pressure classification, Coronary Artery Disease (CAD) classification or machine health classification (into good condition or bad condition) involves classification as the machine learning (ML) task . It is required to identify some characters, known as feature, of the input signal in order to classify it using popular ML methods like Support Vector Machine (SVM) or Random Forest (RF). But the method of identifying a set of suitable features for the classifier is a time and cost absorbing task as it needs mostly domain expertise of the signal processing and IoT expert. Hiring people with such niche skill set is costly. Another problem is that some times the features listed by the domain expert are not relevant and if the data set includes less number of instances with a lot of features, the curse of dimensionality comes into play. This results into the need for feature reduction which can be accomplished into two different ways namely feature dimension reduction or reducing the number of features by feature selection , . These two steps of listing features and reducing the number of features is known as Feature Engineering (FE) as a whole. Another approach to get rid of this issue of FE is to use some Deep Learning (DL) techniques like Multi Layer Perceptron (MLP) with fully connected layers or Convolutional Neural Network (CNN) which can automatically identify suitable feature set for the classification. But DL methods makes an assumption of having a huge number of annotated data-set to train the system which is mostly not available for IoT tasks. DL remains as a very popular technique for computer vision and natural language processing (NLP) and speech recognition. Another popular method is using Principal Component Analysis (PCA) to derive features that have high correlation to the class. However, in both cases interpretation of the features is an issue, and the domain expert cannot take advantage to carry out causal analysis for a given problem, once suitable features are identified. For example, a cardiologist can justify that the PPG (photoplethysmogram) signals should contain a signal at a particular frequency range to classify a CAD signal. But neither DL methods nor the feature dimension reduction techniques can interpret the features as the feature space do not directly relate to physical world. So in this paper authors have proposed a method of FE for classification using ML techniques where a Feature Listing Database is maintained that becomes useful once good features are identified. The contributions of the proposed work is enlisted:
1. A survey has been carried out to understand the workflow of a typical IoT System development and identify the stages that demand maximum effort and niche skills, thereby costing more money and time.
2. The work proposes a method for automation of FE so that the suitable features for classification can be obtained in less time, less cost and the recommended features are physically interpretable to assist the domain expert in making casual analysis.
3. A WIDE architecture for feature listing is proposed that starts to use basic features in the basic layer and extracts more derived features on the higher layer. This layered architecture also helps to reduce the complexity as the features for different layers are computed iteratively. Features of higher layer are not computed once the desired performance is obtained at a lower layer. The performance is measured using different metrics like sensitivity, specificity, F-score.
The proposed method is tested against annotated data sets on man and machine predictive analytics, the use cases of which demand interpretation of features. The used data sets contains (i) NASA's bearing data set that includes good and bad bearing data, (ii) Emotion dataset to classify emotion of users, and (iii) MIMIC-II data set to classify high and low blood pressure from PPG signal. The accuracy obtained by applying some classifier on the recommended feature set are compared against the performance obtained by applying the same classifier on the SoA features reported in literature to solve these point problems. Some of these point problems were also tried in our research lab and hence we were able to compare the development time. We have also compared the proposed method against DL techniques like many layered Multi Layer Perceptron (MLP) method as most of the literatures have used MLP for sensor signal processing . The proposed feature selection technique is compared against a popular feature dimension reduction technique namely Principal Component Analysis (PCA). Finally, through an example we have also given a physical interpretation of the recommended features so that the recommended features can be validated against the physical phenomenon.

Section 2 discusses about the survey conducted to analyse typical IoTA tasks. Section 3 describes the process and corresponding system architecture. Section 4 discusses the various experiments carried out. Finally section 5 concludes the paper.

# Background

```latex
\begin{table*}[!t]\small
\centering
\begin{tabular}{|l|c|c|c|}
\hline

Sensor &Signal Type &Goal &Steps \\
\hline
Camera, mobile, Kinect, &Stochastic  &Object and Scene &Sampling$\rightarrow$Calibration$\rightarrow$Object  \\
 &  & Recognition & Classification \\
\hline
Hyper spectral sensors, &NA  &Object recognition &Image acquisition$\rightarrow$Pre-Processing $\rightarrow$\\
 &  & &Calibration$\rightarrow$ Feature listing$\rightarrow$\\
 &  & &Supervised or Unsupervised learning \\
\hline
Biosensors like ECG, PPG, &NA  &Object recognition &Pre-Processing $\rightarrow$Feature Listing\\
EEG, EMG, Camera, &  & &$\rightarrow$Classification \\
\hline
Camera &NA &Object Recognition &Pre-Processing $\rightarrow$Feature Listing\\
&  & &$\rightarrow$Classification \\
\hline
Microphone, Camera, &Periodic, Stationary,  &Physiological &Pre-processing$\rightarrow$Signal Quality Checker \\
IR Camera  &Non-stationary &condition estimation  &$\rightarrow$Feature Listing$\rightarrow$Feature \\
           &  & &Selection$\rightarrow$Computation   \\
\hline
Thermal camera,  &Multiple frequencies &Classification, Estimation &Filtering$\rightarrow$ Denoising $\rightarrow$ \\
LED/PhotoDiode & &calculation, imaging &features detection $\rightarrow$classification \\
& &beamforming, denoising & \\
\hline
EEG, GSR, SpO2, &Aperiodic, Non-stationary & Classification and &Pre-processing$\rightarrow$Noisy window removal  \\
Camera, & &clustering, estimation &$\rightarrow$Feature Listing$\rightarrow$Feature Selection \\
Eye tracker & & &$\rightarrow$Computation \\
\hline
\end{tabular}
\caption{Steps involved in IoT sensor signal processing and analytics for different applications}
\label{tab:steps}
\vspace{-1.5em}
\end{table*}
```

We made a survey over the associates within our organization involved in the IoTA to identify the pain areas of the application developers. In this survey, we have surveyed seven ($7$) projects which include ninety-five ($95$) employees (female: $35$ and male: $60$). The associates under each project form a team. Each team consists of members among whom 30% having more than 10 years of experience and Master's or higher academic qualification and each team is mostly led by a Ph.D. person. Sixty per cent ($60$%) associates in each team have 5-10 years of experience and expertise in signal processing and the remaining $10$% are developers mostly equipped with good coding skill in C/Java/Python but don't have in depth signal processing, IoT or domain knowledge.

Seven projects involved in this survey are (i) Motion capture, object recognition and rendering of articulated objects from Kinect based skeleton data, (ii) Object classification and recognition from Hyper-spectral sensor, multi-spectral sensors (Landsat, digital globe WorldView etc.) (iii) Bio sensors based object recognition, (iv) Outdoor camera based object recognition. (v) Detecting heart rate and blood pressure from PPG, (vi) Thermal imaging based object classification, and (vii) EEG (electroencephalogram) based cognitive load classification. For each project, the project goal, sensors used, and the steps followed to achieve the goal are shown in Table . These steps can be combined to construct a superset of steps followed in any sensor signal processing based IoTA as shown in Figure 1. The steps used in this figure are briefly described here:

-   Pre-Processing: This module aims to filter out the noises in a signal. It also helps to remove the outlier in the signals. Different pre-processing techniques are available in the literature.

-   Feature Listing: Any signal is represented by some features. For example, time domain or frequency domain analysis normally give many features for an input signal.

-   Feature Reduction: The possible number of features for a sensor signal is very huge and so it is required to reduce the number of features. Feature reduction techniques should be selected with the following points kept in mind: (i) Interaction among the features, (ii) Interaction among the features with machine learning tool, (iii) Goal to achieve which can be obtained from an annotated data set, (iv) Interpretability of the reduced feature.

-   Classification: The technique takes the experience which is a representation of the feature set and the label as input and optimizes some parameters like accuracy, sensitivity to accomplish a goal like classification.

The analysis as shown in Figure 2 clearly reveals that most of the associates under survey express that feature listing and feature selection or feature reduction requires the maximum domain knowledge and technical knowledge. The other steps like pre-processing (including noise cleaning and outlier removal) takes the highest time to design. As the pre-processing algorithms are well established and even source code for them are available in the web, we propose to provide the developer with a list of source codes for all those different state of the art preprocessing techniques so that the developer can try any of the methods on their data set and select the best one among them. One such method is described in . But feature selection is a difficult process. The input to our proposed automated system is the labeled data set of the problem to solve. Another aspect of the study was to list down the features, commonly reported in the related literature, that are used in IoTA applications and finally recommend the features to be used for the given use case.

# Proposed Method

In this section, we are going to discuss how to automate the feature selection step which is the most time consuming and domain specific expertise dependent step. We plan to use state of the art pattern recognition techniques to achieve this. The key idea behind this is as follows.

*Exploration of all possible features*: Finding the right set of features for any pattern classification task is still an unsolved problem. Great amount of research is still concentrating on this problem . In our approach, we, at first, have studied the existing literature  - to have a more or less exhaustive list of features which have been used by different researchers for different classification tasks on different sensor data sets.

We have organized these reported features in an hierarchical manner as shown in Figure  . The features reported in literature can be mainly classified in three types: (i) time domain features (TD) (ii) Fourier transformation based features (FD) (iii) Discrete Wavelet transformation based features (DWT). A challenge of using features from Wavelet Transform is the appropriate selection of a suitable mother wavelet. This is because more than 100 different types of mother wavelets are reported in literature, and a ready made automatic wavelet selection method or tool do not exist. Hence, considering the the different feature types, it is possible to get a large number (say, $N$) of features (including coefficients of the transform domain) from the sensor signals. This results in $2^N-1$ possible combinations of features.

*Feature Selection*: The value of $N$ could be as high as $6,000,000$ as explained later in this section. Therefore, in order to find the optimal feature set, exploitation of all such $2^N-1$ combinations is practically infeasible as this would require testing of machine performance for each combination. This can be achieved mostly by the domain experts from their experience and understanding about the physical phenomena. Typically feature selection method takes 4-6 months for such problems. In pattern recognition literature, there are several feature reduction techniques available which can give a reduced set of features giving optimum performance , , . Principle Component Analysis (PCA) is one such commonly used technique for feature reduction. Actually, what PCA does is feature extraction, i.e. the resultant features are not interpretable. However, we need interpretable features for our tasks. For instance, in health analytics, if we know that certain features are very strong in predicting CAD (coronary artery disease) patients, the doctors need to know what these features are. Therefore, in order to reduce the number of feature to make the feature recommendation task tractable we follow feature selection instead of feature extraction as feature selection methods do not alter the original features.

In our method, we followed an iterative feature selection where $k$-features are selected at each iteration and system performance (e.g. classification accuracy) is checked for this feature set. If the selected feature set results in *expected* performance, that feature set is recommended and process halts. Else, another set of $k$-features is selected in the succeeding iteration and similar steps are carried out. For checking the classification accuracy, we choose SVM-(support vector machines) based classification with different kernels. Different values of $k$ are fitted in progressive increments to get a good result. For a given value of $k$, features are selected using two techniques namely, MRMS and mRMR . The reason for choosing these two techniques is their impressive efficiency in feature selection as demonstrated in . Details on mRMR and MRMS used for this framework can be found in .

## The proposed Architecture for Feature Recommendation

The proposed method is applicable in extracting and recommending features from 1D sensor signals. The sensor signal is subjected to time domain, frequency domain and time frequency domain analysis to extract features which for example counts to almost 12 million features for input data of size 20,000. Let us assume a 1D signal of length n sampled at fs frequency. So after subtracting the mean from the signal we get the same n number of TD features. Now the entire signal is splitted into multiple overlapping windows to extract the FD features. So if we consider a 1sec window with 50% overlap then we get (n\*2)/fs number of windows. Considering the STFT window size to be 256, number of FD features can be obtained is (256\*n\*2)/fs. Similarly by applying DWT we obtain another n features. So extracting first level features we get n+n+(512\*n)/fs $\approx$ 3n number of features. The level two features are derived from each window of the level one features. Thus in level 2 around 20\*(3n/fs)\*2$\approx$`<!-- -->`{=html}120n features are extracted. Therefore around 3n+120n=123n features are extracted at the end of layer2. Similarly in 3rd layer features are derived from each window of 2nd level features which is around 2\*(120n/fs)\*2 $\approx$ 480n. So after 3rd layer the number of features extracted is of the order 123n+480n $\approx$ 600n. Now in dataset 1, value of n is nearly 20,000. So number of features extracted after 3rd layer is nearly 600X20,000$\approx$ 12,000,000. Level two features are applied on top of each of the level one features. Thus after level 2 this number of features extracted is becoming nearly of the order of 4n\*4=16\*n. Similarly after 3rd layer this number becomes of the order or 30\*n. Now in data set 1 this n is nearly 20,000. So after 3rd layer of feature extraction total number of feature extracted from the signal is nearly $30\times20,000 \approx$ 6,00,000. So, it is not possible to apply any exhaustive search on top of this 12,000,000 features so that one can optimize the performance metrics say sensitivity or specificity. The feature selection architecture used in this paper is shown in Figure . The proposed method is as described below:
Input: Time domain signal
Output: Recommended feature list

-   Compute the mean subtracted time domain signal by subtracting the mean from the original signal. This entire signal is considered as the time domain (TD) feature.

-   Compute the short term Fourier transform(STFT) for each window where window size is provided as an user input.

-   The given signal is subjected to four level Discrete Wavelet Transform (DWT) using the most optimal mother wavelet. The optimal mother wavelet is selected using the property of maximum energy to entropy ratio .

-   Take the union of the all three TD, FD, and DWT features. Let this set be defined as $FL_{1}$

-   Apply minimum redundancy maximum relevance (mRMR) and Maximum Relevance Maximum Significance (MRMS) for $FL_{1}$. Each of these methods recommends a different feature set say x and y.

-   Iteratively select the number of features to be recommended by each of these methods so that the recommended features applied on a classifier can exceed the performance metric score defined by the user, say $\tau$.

-   Take union of the features recommended by both mRMR and MRMS methods. Let the recommended feature set be z where $z = x \cup y$.

-   Features from level 2 ($FL_{2}$) and level 3 ($FL_{3}$) are extracted using the algorithms shown in Figure .

-   In level 2 we have extracted spectral features, statistical features, and peak-trough features. Spectral features used in the proposed method are Spectral Centroid, Spectral Crest Factor, Spectral Decrease, Spectral Flatness, Spectral Flux, Spectral Kurtosis, Spectral Rolloff, Spectral Skewness, Spectral Slope and Spectral Spread which are computed for each window of mean subtracted signal. Statistical Features used here are mean, variance, standard deviation, root mean square, skewness, kurtosis. Average peak amplitude, average trough amplitude, average peak to peak distance and average trough to trough distance are computed as the peak-trough features. $FL_{3}$ includes different ratios and derivatives of the $FL_{2}$ features.

-   These $FL_{2}$ and $FL_{3}$ features are also reduced using mRMR and MRMS similarly.

-   Once the feature space is reduced, exhaustively generate all possible combinations of features and apply classifier with different parameters.

-   Apply Support Vector Machines (SVM) classifiers with different kernels namely (i) linear, (ii) radial basis function (RBF), (iii) sigmoid, and (iv) polynomial and each of the kernels is tested with different parameter values.

-   Thus construct the model for classification by selecting the feature list and the SVM kernel for which the recognition accuracy maximizes. SVM is selected as a classifier as the problems at hand are binary classification where SVM is known to excel and converge quickly.

# Experiments

```latex
\begin{table*}
\caption{Comparison of proposed method against state of the art for different data sets}
\label{Table:comparison}
\centering
\begin{tabular}{|l|c|c|c|c|c|c|}
\hline
Data set &SOA no of  &Recommended no  &SOA  &Recommended  & SOA  &Recommended \\
         &features   &of features     &accuracy &accuracy &effort &effort \\
\hline
NASA data set &15 &10 &99.38 &100 &6 months &2 days \\
\hline
Emotion data set &16 &11 &82.3 &90.91 &4 months &2 days \\
\hline
BP data set &23 &15 &79.5 &87.8 &6 months &2 days \\
\hline

\end{tabular}
\end{table*}
```
```latex
\begin{table*}
\caption{Comparison of proposed method against 5-layered MLP}
\label{Table:comparison}
\centering
\begin{tabular}{|l|c|c|c|c|c|c|}
\hline
Data set &Activation  &No. of Epoch &MLP Accuracy  &SoA Accuracy &Proposed\\
         &Function    &             &             &      & Method Accuracy       \\
\hline
NASA data set &softmax   &5         &   71.00             &99.38 &100  \\
\hline
Emotion data set &relu   &15         &   50.0             &82.3 &90.91  \\
\hline
BP data set &softmax   &10         &   50.0             &79.5 &87.8  \\
\hline

\end{tabular}
\end{table*}
```
```latex
\begin{table*}
\caption{Comparison of proposed method against PCA}
\label{Table:comparison}
\centering
\begin{tabular}{|l|c|c|c|c|c|c|}
\hline
Data set &Dimensionality           &No. of Principal &PCA Accuracy  &SoA Accuracy &Proposed\\
         &Reduction Algorithm  &Components              &             &      & Method Accuracy       \\
\hline
NASA data set &svd   &5         &   94.00          &99.38 &100  \\
\hline
Emotion data set &svd   &10        &   50.0             &82.3 &90.91  \\
\hline
BP data set &eig   &5         &   62.50             &79.5 &87.8  \\
\hline

\end{tabular}
\end{table*}
```

## Dataset

The experiment is performed on 3 data sets out of which two data set are publicly available i.e. the Bearing dataset from the Prognostics Data Repository and second one is the MIMIC data-set for classifying blood pressure. The third data-set is internal to our organization and this aims to classify the emotion of a person into three classes namely happy, sad, and neutral. As there is the rest two data sets are two class classification problem, we have used our own data set to prove that our proposed system works on multi-class use-cases also. The first data set used as a machine automation case study. The rest two data sets are used in health care and emotion detection. 2nd and 3rd data sets are based on photoplethysmogram (PPG) sensor signal. The bearing data set (named IMS-Rexnord) consists of three datasets describing a test to failure experiment. Each dataset comprises of several files each of which has a record of 1 sec vibration signal snapshot which are recorded at specific intervals. The sampling rate is 20KHz, recording interval is 10 min and there are 20480 data points in each of the file. In our experiment we have used the second dataset. It has 984 files. Each individual file holds the record of 4 channels representing 4 bearings where the first bearing eventually turns faulty due to outer race failure.

The experiment is performed in two ways. In the first case only the bearing 1 data is considered. State of the art shows that the bearing 1 starts degrading after the 700th point where each file of 1 minute readings is denoted as a point. So here we have formulated a two class classification problem where the first 700 files are considered to be healthy and the rest 282 files are considered to be faulty (the last 2 files are discarded due to presence of many noisy signal values).

In the second case we have considered all the bearings to formulate the two class classification problem. The bearing dataset has 984 files. Each file has record of 4 bearings. Therefore the total number of good bearing samples is 3652 (1st 700 samples of 1st bearing and 984 samples of each of the three other bearings) and the number of faulty bearing samples is 282 (last 282 files of 1st bearing). To avoid any biasness we have segregated the data into 5 folds. Each of the dataset consisting of 282 faulty bearing samples and 730 good bearing samples.

The second dataset which is used to classify the blood pressure into high and low, records the PPG signal of 118 subjects from Bangalore and Gujarat. Among the 118 subjects, 15 subjects have high systolic blood pressure and 103 have low systolic blood pressure. To avoid any biasness due to imbalance in the dataset, the dataset is segregated into three datasets. Each of the dataset consists of the PPG signal record of 49 subjects, 15 subjects having high systolic blood pressure and 34 subjects having low systolic blood pressure. The sampling rate is 60 Hz.

The third dataset (used to classify the emotion into happy and sad) records the fingertip pulse oximeter data of 33 healthy subjects (13F and 20M) with average age 27. No two emotion elicitation video was shown to a subject in one single day. The Pulse Oximeter is used to detect and record the PPG signal. We used standard video stimuli which itself served as ground-truth and the rigorous experimentation procedure ensured that the time synchronization error between the stimuli and recorded physiological data is always less than 1sec.

Table 2 shows the performance of proposed method against state of the art results, proving the efficacy of the method. It is to be noted that time and cost is a huge factor when comparing the performance. In many cases, a trade-off fits the solution well when performance gain is not much when automotion is compared to the manual effort.

## Comparison against MLP

Learning representations from raw data is an emerging field. Theano has been used as the software platform to carry out experiments.

Different number of layers (based on standard thumb rules of input size) for Multi-Layer Perceptron (MLP) with Dropout feature has been tried out so that automatic feature learning can take place. Different activation functions like tan hyperbolic, softmax, sigmoid, rectified linear unit (relu) etc. has been investigated at different layer levels to get a suitable architecture for classification task for the given problems.

Table 3 lists the configurations obtained for a 5 layered MLP for which the best performance was obtained. It is seen that MLP techniques fail in comparison to proposed method as well as state of the art. Also, MLP has some drawbacks when put into the IoT spectrum - a) need for a lot of data for training and b) non-interpretable feature extraction.

## Comparison against PCA

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to derive principal components representative of the features under consideration. This has two outcomes: a) dimension of feature space can be reduced by selecting most prominent principal components b) derived features is supposed to represent the feature space better. Gaussian kernel is used for SVM based classification on the principal component features extracted. Various dimension reduction techniques has been used like Alternating Least Squares (als), Eigen Value Decomposition (eig) and the traditional Singular Value Decomposition (svd). Table 4 lists the configurations leading to the best results. It is apparent from the table that PCA based methods do not perform well in comparison to proposed method. Another drawback of PCA is the derived features are not interpretable.

# Conclusion

This paper has carried out a survey to understand the workflow time and cost needs of a typical IoTA task. In the survey, Feature Engineering came out as the most taxing task among all endeavors that comprise IotA. To tackle the challenge, a system was built to automate this sub-task of IoTA. The system has been tested on three datasets and has been found to give good results when compared to state of the art. The proposed method is compared with PCA and MLP, which are two divergent paths of feature engineering. Feature interpretation is another notable aspect of the system. Future work will look into automation of parameter tuning and selection of machine learning models. Automatic window selection for a given dataset is also planned. Integration of knowledge bases and use of reasoning for ease of interpreting features and causality analysis is also planned.


abstract: Neutrino astronomy was initiated primarily to search for TeV to PeV neutrinos from Active Galactic Nuclei, and the optical Cherenkov technique is well suited for this energy range. Interest has grown recently in detecting EeV neutrinos, particularly the "cosmogenic" neutrinos produced during propagation of ultra-high-energy cosmic rays (UHECR) through the microwave background radiation. These neutrinos could be a powerful tool both to resolve the mystery of the UHECR sources and to test fundamental physics at the $\sim$`<!-- -->`{=html}100 TeV scale. The optical technique is not cost effective at these energies and newer techniques such as radio and acoustic detection are necessary. Accelerator experiments have confirmed the production of both types of signals from high-energy showers in various media, and quantitative measurements have confirmed theoretical descriptions of the signal strength, frequency content and pulse shape. While radio experiments have set the strongest limits so far, the acoustic method could contribute with an entirely independent signal production and detection mechanism and may be more effective at the highest energies. Efforts are underway to develop the acoustic method in various media around the world, with arrays operating in ocean water at the Bahamas, the UK, and the Mediterranean Sea; detectors prepared for deployment in the South Pole ice in the next year; and ideas for future acoustic detectors in salt domes and on Antarctica's Ross Ice Shelf. Regardless of which method is individually most sensitive, the best configuration may be to co-deploy arrays to combine the techniques and seek coincident detection of individual neutrino events.
address: Dept. of Physics, University of California, Berkeley, CA 94720, USA
author: J. A. Vandenbroucke
bibliography: references.bib
title: Summary: Acoustic Detection of EHE Neutrinos

# Acoustic Neutrino Detection in Water, Ice, and Salt

Ocean water is the most abundantly available and best understood acoustic medium for neutrino detection studies. The acoustic properties of the oceans have been mapped out in detail by military and marine science researchers. The first detailed search for neutrino-like signals from a large undersea array was completed by the SAUND (Study of Acoustic Ultra-high-energy Neutrino Detection) project using a military array in the Bahamas. Lessons learned in signal simulation, online triggering, and background rejection are now being applied to a larger array at the same location in the SAUND-II project.

Acoustic neutrino arrays in water currently have an energy threshold well above the EeV scale. Although site selection, signal processing and background rejection advances, and optimization of an array specifically for neutrinos could all lower the threshold, other media likely feature an inherently higher signal to noise ratio resulting in a lower neutrino energy threshold. Ocean-based arrays may be capable of probing topological defect fluxes and other super-EeV models, but seem not to have sufficient sensitivity at the Greisen-Zatsepin-Kuzmin (GZK) energy range.

Underground salt domes extending several km on each side could have long absorption and scattering lengths for both radio and acoustic waves. The cost of drilling into these domes, however, is prohibitive. Drilling into ice is less expensive and may be better suited for radio and acoustic as well as optical neutrino detection. Greenland ice and ice in temperate glaciers is dirty, ruling out the optical Cherenkov technique, and warm, causing it to absorb radio and acoustic waves well. South Pole ice, on the other hand, is clean and cold.

To compare media as targets for acoustic neutrino detection, several quantities should be considered: background noise, absorption, scattering, and signal strength (Table ). In the ocean, background noise is site-dependent but is dominated by weather patterns (wind and rain on the surface). Transient events that constitute a neutrino background include the sounds of crustacean, cetacean, and human activities. Noise levels and transient backgrounds in salt and ice are currently unknown in the relevant ($\sim$`<!-- -->`{=html}1-100 kHz) band. Possible noise sources include seasonal human activity at the station, seasonal wind (note that surface sources are somewhat shielded by the firn, which refracts waves back to the surface), and cracking in the ice bulk or at the bedrock interface due to glacial movement. Seismically (at $\sim$`<!-- -->`{=html}100 Hz), South Pole is measured to be the quietest place on Earth.

Acoustic properties of several media. The neutrino-induced pressure signal strength scales with the Gruneisen parameter.

Ocean
South Pole Ice
Salt

Temperature

T (∘C)
15
-51
30

Sound speed
vL (m/s)
1530
3920
4560

Volume expansivity
β (10−5 K−1)
25.5
12.5
11.6

Heat capacity
CP (J/kg/K)
3900
1720
839

Peak frequency
fp (kHz)
7.7
20
42

Gruneisen parameter
γ ≡ vL2β/CP
0.153
1.12
2.87

In the ocean, absorption is dominated by dissolved salts. The absorption length depends on the concentration of these salts locally but is typically several km. In ice, absorption occurs by reorienting the hydrogen atoms in the crystal lattice . This is a temperature-dependent relaxation process, and thus is depth-dependent at South Pole. Scattering is insignificant in both water and ice. In salt, the situation is reversed: The large grain size causes relatively large Rayleigh scattering (scattering length from one to several km). Absorption in salt (due to anharmonic interactions with thermal phonons) is insignificant; the absorption length is 10's of thousands of km. However, this assumes pure NaCl. Inclusions and sediment layers, likely present in salt domes, would significantly increase both scattering and absorption.

Absorption in ocean water scales with frequency-squared and thus high frequency components are preferentially attenuated. In ice, absorption increases with frequency-squared up to a temperature-dependent turnover frequency and is then flat. The absorption is frequency-independent above 1 kHz in ice below -24$^\circ$ C, so pulse shape is preserved during propagation.

# The Scottish Isles

The ACORNE (Acoustic COsmic Ray Neutrino Experiment) collaboration has been using a military array on the island of Rona, Scotland . They read out 8 hydrophones continuously at 140 kHz for two weeks in December 2005. They are now studying signal processing and event reconstruction techniques with this data. Two significant challenges to acoustic neutrino arrays are to calibrate the phase response of the detector chain and also to develop an in situ calibration device that can mimic the cascade pulse shape and radiation pattern. They have developed a design that can meet both with a string of several transmitters pulsed in phase. A clean bipolar acoustic pulse can be generated by inputting a more complicated electrical pulse shape determined using an equivalent LRC circuit model. The group has also improved simulations of Extremely High Energy (EHE) hadronic cascades .

# The Mediterranean Sea

In addition to the optical Cherenkov telescopes currently under construction, activities are underway to determine the feasibility of acoustic detection of EHE neutrinos in the Mediterranean Sea. In contrast with the SAUND and ACORNE projects, these efforts benefit from synergy with the optical telescopes. They can share deployment procedures, mechanical support, and electrical readout systems with the optical telescopes. If successful they may be able to search for coincident detection of the same neutrino events with both optical and acoustic arrays, although the energy scales may be too different for sufficient overlap.

Combined event reconstruction could be particularly powerful because optical arrays in the Mediterranean are expected to have excellent pointing resolution in both the muon and the cascade channel (scattering limits the resolution at South Pole, dramatically so for cascades). Depending on array geometry, three-dimensional acoustic arrays may have $\sim$`<!-- -->`{=html}1$^\circ$ pointing resolution. In addition to superior combined optical-acoustic reconstruction, a Mediterranean site would be able to view the opposite hemisphere with respect to the South Pole. Full sky coverage is as important for EeV neutrino arrays as for TeV-PeV neutrino arrays, because GZK neutrinos point back to UHECR sources on cosmological scales.

First, however, sites in the Mediterranean must be evaluated, and signal detection and background reduction techniques developed, for the acoustic technique as well as for the optical. This is being done as part of both the ANTARES and the NEMO optical neutrino experiments.

In 2005 the ANTARES "MILOM" line (mainly a calibration line) was installed with the SPY hydrophone to measure the acoustic background. In 2007 AMADEUS (ANTARES Modules for Acoustic Detection Under the Sea) , a dedicated array of sensors for acoustic studies, will be installed. AMADEUS (Fig. ) will comprise three acoustic storeys on the Instrumentation Line, with three more storeys planned for another line. An acoustic storey is an analog of the optical storey: a $\sim$`<!-- -->`{=html}1-2 m cluster of sensors attached to a line. On each AMADEUS line, the three acoustic storeys will be spaced vertically 10 m and 100 m apart. This will allow the study of correlated continuous and transient noise on three distance scales (1, 10, and 100 m). Two options are being pursued to convert the optical storey design to an acoustic storey design: either six hydrophones will be arranged with three in an upper plane and three in a lower plane within the storey, or the three optical modules of an optical storey will be replaced with three acoustic modules. Each acoustic module would feature two piezoelectric sensors glued to the inside of a pressure sphere. The sensors have a self noise level below the sea state zero level in the 1-100 kHz band.

Associated with the NEMO (NEutrino Mediterranean Observatory) project off the coast of Sicily, the Ocean Noise Detection Experiment (O$\nu$DE) has been deployed to collect data for 20 months . It consists of a cluster of four hydrophones with a 1-50 kHz band, spaced $\sim$`<!-- -->`{=html}1 m apart, $\sim$`<!-- -->`{=html}2 m off the sea floor at a depth of 2500 m. Digitized data are routed to shore over optical cable. It has been running since January 2005 and is characterizing the acoustic environment of the site, which has been shown to have the best optical qualities among investigated sites.

# The South Pole

Given the considerations of background noise, signal strength, absorption, and scattering discussed above, the South Pole is perhaps the most promising location for acoustic neutrino detection. However, these considerations are based on extrapolations from other measurements. It remains to determine these quantities in situ.

The South Pole Acoustic Test Setup (SPATS) is designed to do this . It will consist of three 400 m long strings to be deployed in IceCube holes in January 2007 (Fig. ). Each SPATS string will be deployed in a hole immediately after the corresponding IceCube string and will be mechanically and electrically independent. Each string will feature 7 acoustic "stages," with each stage consisting of a transmitter module and a sensor module. The transmitter module has an epoxy-cast piezoelectric ring that is in direct contact with the ice. The sensor module has a cylindrical pressure vessel holding three piezoelectric transducers, each pointing in a different azimuthal direction separated by 120$^\circ$.

The absolute uncertainty of both the transmitter amplitude and the sensor sensitivity is $\sim$`<!-- -->`{=html}40%, and the angular variation in the response of each sensor module is similar. However, the large channel multiplicity will provide many independent acoustic ray paths. Combining them will allow a measurement of the absorption length (or a lower limit on it if it is much larger than the SPATS baseline) despite the systematic uncertainty in individual modules .

# Antarctica's Ross Ice Shelf

Antarctica's Ross Ice Shelf is a body of ice the size of France, several hundred meters thick and floating on sea water. This unique configuration causes acoustic waves to reflect both off the ice-air interface and (with lower reflectance) off the ice-water interface, effectively trapping some fraction of energy generated by a source in the ice. Hydrophones have been used to determine that the noise in the water below the ice shelf is much quieter than sea state zero. Radio waves are reflected entirely off the ice-saltwater interface, making it an interesting site for a radio array as well .

While the acoustic peak frequency emitted by cascades is $\sim$`<!-- -->`{=html}10 kHz, the pulse is broadband. At frequencies below the peak, the signal spectrum scales with frequency squared. At these frequencies, the radiation pattern is more isotropic than the flat disk at peak frequency. For a cascade produced in the upper part of the Ross ice, where the sound speed is increasing with depth as the ice compactifies, some sound rays are refracted up to the surface, reflected, refracted up to the surface again, and so on, effectively propagating as a surface wave (Fig. ).

There is also the possibility of exciting Rayleigh surface waves. However, these waves only penetrate a skin depth of order their wavelength, arguing in favor of lower frequency to achieve a greater range of depths in which a cascade could excite such a wave. Another argument pushing the optimal frequency for a Ross experiment lower is the absorption length: For shallow South Pole ice (at -51$^\circ$ C), absorptivity scales with frequency-squared up to 30 Hz and is then flat . But for Ross ice (at -28$^\circ$ C), the absorptivity in the flat regime is higher, but the turnover frequency is also higher: 600 Hz. An experiment optimized below 600 Hz would benefit from the decreased absorption at lower frequencies.

# Toward a Hybrid EeV Neutrino Observatory

There may be a substantial advantage to combining two or three sub-arrays employing different techniques into a single hybrid neutrino detector . This is done routinely at particle accelerators, where sub-detectors are optimized for different signals emitted from the same interaction region, and the approach is now the key strategy of the Pierre Auger Observatory in its mission to resolve the UHECR discrepancy.

While the acoustic and radio techniques are promising in the EHE range, they are not well calibrated in comparison with the optical Cherenkov method, which is calibrated using a high rate of atmospheric neutrino events and an even higher rate of atmospheric muons. The radio technique in particular has been tested with detailed quantitative comparisons between theory and accelerator tests, but except for the small possibility of detecting air shower cores with small optimized arrays, neither will benefit from the in situ calibration sources the optical method enjoys.

However, it may be possible to build a large hybrid array in which both radio and acoustic sub-arrays detect a large GZK rate ($>$`<!-- -->`{=html}10 events per year), with a large fraction ($\sim$half) of events detected in coincidence between the two methods and a small fraction (on the order of 1 event per year) detected in coincidence with an optical array. Water is good for the optical method, satisfactory for the acoustic method, and impossible for the radio method; salt is good for the radio method, unknown for the acoustic method, and poor for the optical method. Only ice has the possibility to be well suited for all three techniques. While the new radio and acoustic ideas for the Ross Ice Shelf are interesting, the prospect of coincident detection with IceCube, the excellent energy and direction reconstruction capabilities of volume arrays compared to surface arrays for all three methods, and the logistical advantages of co-deployment argue strongly in favor of the South Pole.

There are technical questions that must be answered to determine whether this is a viable idea. Urgent for the acoustic method are the basic material and environmental properties that will be measured by SPATS. For the radio method, the RICE experiment has already determined that South Pole ice has good radio transparency, but there is still work to be done to optimize the antenna configuration and signal digitization and transmission technology.

Along with recent growth in interest in neutrinos in the EeV range has come a set of new projects aiming to detect these neutrinos acoustically. Among the three techniques of high energy neutrino detection, the acoustic method involves frequencies five orders of magnitude smaller than the other two, which allows simpler readout technology and makes it the most inexpensive method per channel. The acoustic method may therefore be the most cost-effective way to study the highest energy range of neutrino astronomy. Projects now underway are characterizing various media and sites for this purpose as well as developing general techniques that will apply in any medium. Their results, together with the first astrophysical neutrinos from IceCube and ANITA expected in the near future, will allow us to determine the best way to access the unique physics and astronomy potential of Extremely High Energy neutrinos.


author: **Baihan Lin and Nikolaus Kriegeskorte$^*$**
{Baihan.Lin, email@example.com
Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA 10027
$^*$corresponding author
email@example.com
The Brain and Mind Institute, University of Western Ontario, London, ON, Canada N6A 5B7

firstname.lastname@example.com
MRC Cognition and Brain Sciences Unit, Cambridge University, Cambridge, United Kingdom CB2 7EF
bibliography: main.bib
title: Visualizing Representational Dynamics with Multidimensional Scaling Alignment

# Abstract

Representational similarity analysis (RSA) has been shown to be an effective framework to characterize brain-activity profiles and deep neural network activations as representational geometry by computing the pairwise distances of the response patterns as a representational dissimilarity matrix (RDM). However, how to properly analyze and visualize the representational geometry as dynamics over the time course from stimulus onset to offset is not well understood. In this work, we formulated the pipeline to understand representational dynamics with RDM movies and Procrustes-aligned Multidimensional Scaling (pMDS), and applied it to neural recording of monkey IT cortex. Our results suggest that the the multidimensional scaling alignment can genuinely capture the dynamics of the category-specific representation spaces with multiple visualization possibilities, and that object categorization may be hierarchical, multi-staged, and oscillatory (or recurrent).

>
>
> **Keywords:** MDS, RSA, GPA, Neuroimaging

# Introduction

In recent years, technological innovations in computer vision have produced biologically-plausible models for human visual information processing. Among these models are goal-driven deep feedforward hierarchical neural networks, which have been proposed to model the ventral stream of visual cortex, the "what pathway" in the brain thought to underlie object recognition . However, there is a discrepancy in the hierarchical depth between the primate ventral visual stream ($<$ 10 stages of representation) and state-of-the-art computer-vision models ($>$ 100 layers). The primate visual system might make up for its limited hierarchical depth by recycling its resources through time, via recurrent connections and attention mechanisms, all of which require the analysis of the entire time-series of the dynamics in the brain. Few studies have investigated the dynamics of visual perception (for instance, object identity and categorization) and their representational changes .

The scarcity of methods to characterize the representational dynamics creates a major barrier to answer interesting questions such as: how are objects represented in the brain over the time course from early perception to categorical decision making, does the object identification or visual categorization follows a hierarchical classification paradigm; do different classes of objects merge and branch at different time points based on different tasks or recurrence paradigm; are these representational dynamics oscillatory or recurrent?

A central challenge is exactly to test computational theories implemented in deep neural network models with exactly this type of time-stamped brain-activity data. Analyses of representational geometry can help us to compare representations between biological brains and computational models, and to understand brain computation as the transformation of representational similarity structure across stages of processing . Representational Similarity Analysis (RSA) uses a region's representational dissimilarity matrix (RDM) as a multivariate summary statistic that characterizes the representational geometry using metric distances in representational space. This is useful to obviate the need for a detailed point-to-point mapping between neurons in the brain and units of a model.

Traditional RSA usually considers the entire time series of the neural measurement as the response pattern. In that sense, the dynamics is collapsed into one data point to characterize the brain-activity corresponding to a specific stimulus. However, it is unclear how to properly capture the representational dynamics, i.e., how the representations evolve over time. In this study, we propose a framework to first extract the snapshots of representational spaces with sliding-window RSA, and then align each frame of this RDM "movie" (as snapshots of the representational space) with Generalized Procrustes Analysis (GPA). We presented the visualizations on the data of monkey's stimulus-driven single-electrode recording, and demonstrated several neuroscience insights on visual object categorization revealed from the proposed method.

# Method

## Representational Similarity Analysis (RSA)

Representational Similarity Analysis (RSA) characterizes internal representations of a brain network by estimating all pairwise distances (as the representational geometry) across a large set of input conditions. These representational geometries are invariant to rotations of the underlying high-dimensional activation space. RSA involves two steps: (1) compute the dissimilarity for each pair of stimuli; (2) correlate RDMs to assess to what extent the brain representation reflects stimulus properties, can be accounted for by different computational models, and is reflected in behaviours.

## Multidimensional Scaling (MDS)

As a popular non-linear dimensionality reduction method, Multidimensional Scaling (MDS) rearrange the location of a set of data points from a set, given a distance matrix characterizing the distances between each pair of objects in a set (for instance, the RDM computed from the response patterns of different stimuli), into an N-dimensional space such that the between-object distances are preserved .

## Generalized Procrustes Analysis (GPA)

In computer vision and signal processing, Procrustes analysis is usually used to analyse the statistical distribution of a set of shapes. The Generalized Procrustes analysis (GPA) compares three or more shapes to an optimally determined "mean shape" and can align all the shapes according to this mean shape as a reference frame . GPA solves the mean shape iteratively by optimizing against the Procrustes distance, a metric to minimize in order to superimpose any pair of shape or time frame instances annotated by landmark points. The analysis starts from choosing an arbitrary reference frame, and then superimpose all instances to current reference shape. If Procrustes distance between the reference shape and the computed mean shape of the current set of superimposed shapes is above a certain threshold, the reference shape is set to the mean shape to continue above steps iteratively, until the Procrustes distance between the two is small enough within the trivial threshold.

## Procrustes-aligned MDS (pMDS)

In our specific problem, the RDM movie consists of the representational shapes of each time point without any intertemporal information. We apply GPA to the MDS embeddings computed from RDMs at each time point, such that each frame are optimally aligned to all other time frames. The Procrustes analysis has the option to constrain rotation, scaling and reflection, but in our case, we only allow the rotation and reflection, because the scaling contains information about the how the representations diverge and converge over time. Because Procrustes alignment doesn't distort the geometrical information between each stimuli (constrained by the individual RDM at each time point), the Procrustes-aligned MDS (pMDS) can offer us a genuine and illustrative visualization of the representational dynamics over time.

# Results

## Data of neural population code

The data used to demonstrate the proposed method are the monkey single-electrode recordings from the inferior bank of the ST segment . Two adult male rhesus monkeys were shown 100 grayscale object images from five different categories each with 20 instances (faces, fruits, places, body parts and objects) in a serial visual presentation. RSA was further applied to select visually-responsive neurons and extract single-trial response patterns from spike-density function. The recordings were truncated into sections of 821 ms (starting from 100ms before stimulus onset). RDM movies were generated using a sliding window of 21 ms with cross-validated spike rate distance (SRD) as the reponse-pattern dissimilarity measure.

## MDS alignment reveals smooth transition over time

We performed GPA on the MDS embeddings computed from each time frame of RDM movies MDS based on the stimulus label (not the category label). As shown in Figure , the average trajectory of all the data points of the same category are plotted in the MDS space over time. The dynamics over time can be reasonably visualized while the separation of each categories (as between-category distances) is well preserved. From the 3D plot of the representational space, the separation of each categories happens around 80ms and reached a maximum distinction at around 150ms after stimulus onset, then the trajectories gradually converges over time in an oscillatory fashion after the stimulus offset (at 300ms). A movie of Procrustes-aligned MDS plots is also generated and can be accessed at <https://youtu.be/WQbgDCq7Dhg>, where each data point (a stimulus instance) can be distinctively tracked as moving seamlessly across each time frame.

## Hierarchical visual categorization with major stages

Given the intuition from the 3D visualization, we further explored segments of the representational dynamics. Figure demonstrated the average representational trajectories of each categories in the first 100ms after stimlus onset (with the end marked as square and dot size indicating the standard deviation across different stimulus instances). We see that the categories faces and fruits diverges become discriminable rapidly in the IT population code due to their distinct visual dissimilarity, while places, objects and body parts diverge much later in time. Among the three late classes, the separation of the objects and body parts happens even later in time, suggesting a hierarchical process of categorization. Later during the stimulus is on, the representations of each category seems to be dwelling around their own cluster in a slowly drifting fashion, as shown in Figure (the convex hulls are plotted for all stimuli within the time range in the selected category). After the stimulus offset, the average trajectory of each category gradually converge into proximity, as shown in Figure , where the convex hulls of each categories gradually merge into one. The representational space for each category (as indicated by the areas of the convex hull covering all stimulus instances within the category) also follows several major segments (Figure ): a peak around 100 to 200 ms after onset and another bump after the stimlus offset around 300 to 400 ms. Further investigations can potentially illuminate the role of working memory and other factors that might contribute to the second rise of the representational areas.

## Temporal analysis of aligned representations

With the aligned representations, other temporal analyses can be applied to compare between time points. Figure offers a subset of such inquires that can potentially offer neuroscience insights. For instance, the MDS displacement away from the origin (the third plot) indicates a similar dual bump feature during stimulus onset and offset, suggesting the drifting of the centroids of each categorical representation follows a unique dynamics that was not previously widely understood. In the fourth plot, we aligned the signal for each category onto the category-specific 1D plane where the signal amplitudes are maximized. Initial analysis reveals that there seems to be interesting oscillations worth further investigation. Due to GPA's preservation of geometrical information of the pairwise stimulus-specific response pattern, these temporal analyses offer none artificial insights on the variance or noise (due to the scaling restriction) and directionality (due to the minimized Procrustes distance).

# Conclusion and Future Work

We here proposed a representation alignment method to extend the RSA framework to analyze time-stamped brain-activity profiles as representational dynamics. From the neural data, RDM movies are computed with as sliding-window snapshots of representational geometry, and then aligned across all time points with generalized Procrustes analysis. We applied the proposed method to the single-electrode recording of monkey's IT cortex viewing 100 images of 5 categories. The results demonstrated that the alignment can reasonably capture the temporal dynamics of the representation space for each category, and reveal insights on the hierarchical separations of classes and possibly connection with other mechanisms such as working memory and oscillatory behaviors.

Other than working with RDM movies, there are several alternative methods to study representational dynamics. One such approach under ongoing investigation is to work with raw data of the time-series measurements, by directly extracting the pairwise intertemporal distances into a full RDM of dimension $(N\times T)\times (N\times T)$ with N as number of stimuli and T as number of time points. However, there are clear advantages of our currently proposed multidimensional scaling alignment of RDM movie over this alternative approach of RSA of full RDM: (1) the algorithmic complexity of working with the full RDM is so expensive that it's computationally prohibitory, while Procrustes-aligned MDS is very scalable and light-weighted; (2) the pairwise distances of time series segment is an ongoing challenge and topic of interests in text mining and bioinformatics, that requires deeper understanding to apply in a logical way. Another alternative is to simply use the snapshots RDMs themselves. Figure compares these two methods, where RDM's grid-like and MDS's patch-like visualizations each offer a unique insight of the pattern. Future and ongoing work includes the application of this method to the neuroimaging data of different brain regions and time scales to explore whether the representations are also recurrent as the neural recordings , as well as visualizing the representational dynamics of deep neural networks to understand their behaviors.