Question:
represent the article for finding a single-sentence tldr
Fewshots:
'Classification systems typically act in isolation, meaning they are required to implicitly memorize the characteristics of all candidate classes in order to classify. The cost of this is increased memory usage and poor sample efficiency. We propose a model which instead verifies using reference images during the classification process, reducing the burden of memorization. The model uses iterative non-differentiable queries in order to classify an image. We demonstrate that such a model is feasible to train and can match baseline accuracy while being more parameter efficient. However, we show that finding the correct balance between image recognition and verification is essential to pushing the model towards desired behavior, suggesting that a pipeline of recognition followed by verification is a more promising approach towards designing more powerful networks with simpler architectures. FIG3: Overview of our hybrid model in contrast with two opposing approaches. (a) Recognition network directly predicts the class given the input. (b) Verification network predicts binary output indicating the amount of similarity or likelihood that they belong in the same class. The verification network can be used to compare to all reference images from each class to produce the final class prediction. (c) Our approach, RVNN, queries for reference image from a particular class at each time step, and makes a class prediction at the last time step. BID9. Then to classify, the image is pair-wise compared with a support image from every class and 37 image with the maximum similarity score is chosen. Matching networks extends the verification-38 based approach by outputting a prediction based on a weighted-sum of similarity across classes. Additionally the work introduces an episodic-training regime which encourages the model to better 40 learn for the one-shot learning scenario. Prototypical Networks uses Euclidean Distance in embedding 41 space as a verification metric rather than a learned metric, while maintaining the same training regime 42 as Matching Networks to encourage different classes to have distant means in embedding space. One recent work outside of few-shot learning domain is the Retrieval-Augmented Convolutional Neural Networks (RaCNN), which combines CNN recognition network with a retrieval engine 45 for support images to help increase adversarial robustness. For all the above few shot learning approaches, verification with support images from all classes are required before a classification decision is made. Hence the classification decision is solely derived 48 from verifications. RaCNN is closer to our approach, which uses a hybrid between verification 49 and recognition. However, RaCNN simply retrieves the K closest support image neighbours in the 50 embedding space, whereas our model is required to form a hypothesis as to which class to compare 51 with. In cases in which there are a large number of classes, we expect our approach to excel. As well, this introduces a non-differentiable component in our model not present in previous work. Prior work has also looked at the concept of incorporating external knowledge into the decision 55 making process of neural networks, often with non-differentiable components. Buck order to produce a prediction C for the class. FIG2 illustrates the full process. The subsequent sections detail the implementations of the three components as well as training 84 considerations. The recurrent querying model f rnn was implemented using a Gated Recurrent Unit (GRU). We 105 also considered passing in additional information to f rnn, such as the query that was used in the 106 previous time steps. We implemented f q as sampling a class based on the categorical probability given by the softmax of 109 the logits from the f rnn. Therefore, f q can be written as: DISPLAYFORM0 DISPLAYFORM1 To sample S n+1, we can use the Gumbel-Max trick: DISPLAYFORM2 where g i...g k are i.i.d samples drawn from Gumbel distribution. However, the arg max operator 112 is not differentiable, so instead we explored two approaches during training. The first is to use the Gumbel-Softmax trick, also known as the Concrete estimator. We 114 relaxed the arg max operator to a differentiable softmax function with temperature parameter τ: DISPLAYFORM0 The τ parameter is annealed exponentially from τ = 1 to τ = 0.5 as the training iterations progresses. The second approach is to use simple Straight-Through estimator BID0. In the forward pass, we apply 117 the Gumbel-Max trick to take discrete query choices. Then on the backward pass, we set the derivative 118 of the query with respect to the softmax probabilities to be identity so that the out-going gradient 119 from the arg max operator is equal to the incoming gradient during backpropagation: DISPLAYFORM0 We highlight that our model differ from several existing networks in various aspects. The performance of the model is assessed by both reduced parameter usage and sample efficiency. Reduced parameter usage is measured relative to a baseline model, in this case the CNN architecture The performance of the model alone does not indicate whether our approach is functioning as intended. We also experiment with several small modifications to our architecture as well as a few hyper- parameters that are unique to our model. We list them here below. • Architectural Considerations -Query Memory (QM): The query from the past time step is passed to the RNN. the RNN learns to adapt its policy to varying levels of accuracy, and see whether a better can 184 be achieved when they are used in tandem. We also experiment with fixing query policy to assess 185 whether the RNN actually learns intelligent query behavior. From our overall performance metrics we observe that at both smaller and larger sizes of model, our usage. This is in agreement with our hypotheses that our model would be more parameter efficient. RNN's learned policy is compared against a random query policy and the optimal query policy (never 218 repeating a query). From the figure we see that the model is able to conduct a better than random 219 query policy, but is not able to achieve optimal performance. We also observed that performance 220 increases with a higher RNN size up to 200. This suggests that the model is in some part able to track 221 previously unsuccessful queries and remember if there was a match. However, its memory is not 222 perfect and it cannot achieve optimal performance. and a real (neural network) comparator. Of the query-based models, informed queries performed the best over random and no query models. appropriate pipeline for our model is to perform a recognition operation which is then followed by 241 verification, rather than perform them simultaneously. This new pipeline would also imply that our model may be less well suited for the one-shot learning 243 task than initially believed, as a reasonably-well trained recognition module is required as the first tasks of this form are pre-requisite if neural-network models are considered to be as such BID5.' == 'Image classification via iteratively querying for reference image from a candidate class with a RNN and use CNN to compare to the input image' != 'theoretically understand the regularization effect of distillation. We show that early stopping is essential in this process. From this perspective, we developed a distillation method for learning with corrupted Label with theoretical guarantees.'

Question: Modern applications from Autonomous Vehicles to Video Surveillance generate massive amounts of image data. In this work we propose a novel image outlier detection approach (IOD for short) that leverages the cutting-edge image classifier to discover outliers without using any labeled outlier. We observe that although intuitively the confidence that a convolutional neural network (CNN) has that an image belongs to a particular class could serve as outlierness measure to each image, directly applying this confidence to detect outlier does not work well. This is because CNN often has high confidence on an outlier image that does not belong to any target class due to its generalization ability that ensures the high accuracy in classification. To solve this issue, we propose a Deep Neural Forest-based approach that harmonizes the contradictory requirements of accurately classifying images and correctly detecting the outlier images. Our experiments using several benchmark image datasets including MNIST, CIFAR-10, CIFAR-100, and SVHN demonstrate the effectiveness of our IOD approach for outlier detection, capturing more than 90% of outliers generated by injecting one image dataset into another, while still preserving the classification accuracy of the multi-class classification problem. Motivation. As modern applications such as autonomous vehicles and video surveillance generate larger amount of image data, the discovery of outliers from such image data is becoming increasingly critical. Examples of such image outliers include unauthorized personnel observed in a secret military base or unexpected objects encountered by self-driving cars on the road. Capturing these outliers can prevent intelligence leaks or save human lives. State-of-the-Art. Due to the exceptional success of deep learning over classical methods in computer vision, in recent years a number of works BID17 BID27 BID5 BID23 leverage the representation learning ability of a deep autoencoder or GAN BID7 for outlier detection. Outliers are either detected by plugging in the learned representation into classical outlier detection methods or directly reported by employing the reconstruction error as the outlier score BID36 BID4. However, these approaches use a generic network that is not trained specifically for outlier detection. Although the produced representation is perhaps effective in representing the common features of the "normal" data, it is not necessarily effective in distinguishing "outliers" from "inliers". Recently, some works BID26 BID21 were proposed to solve this issue by incorporating the outlier detection objective actively into the learning process. However, these approaches are all based on the one-class technique BID28 BID18 BID33 that learns a single boundary between outliers and inliers. Although they perform relatively well when handling simplistic data sets such as MNIST, they perform poorly at supporting complex data sets with multiple "normal" classes such as CIFAR-10 (BID11). This is due to the difficulty in finding a separator that encompasses all normal classes yet none of the outliers. Proposed Approach and Contributions. In this work we propose a novel image outlier detection (IOD) strategy that successfully detects image outliers from complex real data sets with multiple normal classes. IOD unifies the core principles of cutting edge deep learning image classifiers BID7 and classical outlier detection within one framework. Classical outlier detection techniques BID3 BID9 BID1 consider an object as an outlier if its outlierness score is above a certain cutoff threshold ct. Intuitively given a Convolutional Neural Network (CNN) BID12 ) trained using normal training data (namely, data without labeled outliers), the confidence that the CNN has that an image belongs to a particular class could be leveraged to measure the outlierness of the image. This is based on the intuition that we expect a CNN to be less confident about an outlier compared to inlier objects, since outliers by definition are dissimilar from any normal class. By using the confidence as an outlier score, IOD could separate outliers from all normal classes. However, our experiments (Sec. 2) show that directly using the confidence produced by CNN to identify outliers in fact is not particularly effective. This is because the requirements of accurately classifying images and correctly detecting the outlier images conflict with each other. CNN achieves high accuracy in image classification because of its excellent generalization capability that enables a CNN to overcome the gap between the training and testing images. However, the generalization capability jeopardizes the detection of outliers, because it increases the chance of erroneously assigning an outlier image to some class with high confidence to which actually it does not fit. We solve this problem by proposing a deep neural decision forest-based (DNDF) approach equipped with an information theory-based regularization function that leverages the strong bias of the classification decisions made within each single decision tree and the ensemble nature of the overall decision forest. Further, we introduce a new architecture of the DNDF that ensures independence amongst the trees and in turn improves the classification accuracy. Finally, we use a joint optimization strategy to train both the spit and leaf nodes of each tree. This speeds up the convergence. We demonstrate the effectiveness of our outlierness measure, the deep neural forest-based approach, the regularization function, and the new architecture using benchmark datasets including MNIST, CIFAR-10, CIFAR-100, and SVHN -with the accuracy higher than 0.9 at detecting outliers, while preserving the accuracy of multi-class classification. In convolutional neural networks (CNN) BID13, the final Fully Connected (FC) layer contains a single node n i for each target class C i in the model. Here i ∈ {1, 2, ..., m}, where m is the number of target classes. Given an input image, the final FC layer computes a weighted sum score, s i w.r.t. class C i as s i = n j =1 F j w ji, where F j is a feature produced by the convolutional model and w ji is the learned weight of the FC layer that connects F j and node n i. The maximum weighted sum is the largest weighted sum score among all classes, defined as max (s 1, s 2, ..., s m).Using the weighted sum scores as input, a softmax activation function is then applied to generate a class probability p i between 0-1 for each node n i (m i=1 p i = 1). This can be interpreted as relative measurement of how likely it is that one image falls into target class C i. A testing image t will be assigned to class C i if p i = max (p 1, p 2, . . ., p m). p i is called the maximum probability of image t. Intuitively, if an image does not have a good match with any known class, its maximum probability will be small. Since outliers typically do not exhibit many features of a known class as compared to inliers, we postulate the maximum probability of an outlier tends to be small relative to inliers. Therefore, it is natural to use the maximum probability or the corresponding maximum weighted sum score as a confidence measure for images. Confidence-based Outlier Detection. Next, we propose a method that uses this confidence to detect outliers. An outlierness threshold ct is established in the training process using the training image x k which has the kth smallest confidence among all objects in the training set. Then given a testing image, it will be considered as an outlier if its confidence is (b) Our improved score using max route probability distribution Figure 1: Max weighted sum VS max route probability (tested on CIFAR-10 model).smaller than the confidence of this selected x k. One advantage of this approach is that unlike the state-of-the-art image outlier detection work BID26 BID27 we do not have to explicitly set the outlierness cutoff threshold ct at the cost of introducing a different hyperparameter k. However, k is a more intuitive parameter to set than ct, because k can be set by simply assuming a percentage of the outliers in the training data. Our experiments show that this straightforward approach alone does not work very well, because many of the outlier images do have a large maximum probability (maximum weighted sum). We illustrate this with an example. First, we trained a CIFAR-10 model using the CIFAR-10 dataset BID11. We then used the model to classify the CIFAR-10 and CIFAR-100 testing data. Each dataset contains 60,000 images. Ideally, all CIFAR-100 images should be detected as outliers, because the CIFAR-100 and CIFAR-10 images contain disjoint classes. Fig. 1(a) shows the maximum weighted sum w.r.t. the CIFAR-10 and CIFAR-100 datasets when tested on the CIFAR-10 model. The CNN network is identical to the network we described in our experiment section (Sec. 5.2). Note we demonstrate the of using maximum weighted sum instead of maximum probability, because our experimental evaluation shows that maximum weighed sum performs bettern than maximum probability in detecting outliers. The black vertical line represents the cutoff threshold ct. In this case, k is set as 5000. We expect nearly 100% of CIFAR-100 images to fall to the left of the black dashed line and hence be captured as outliers. However, more than 30% CIFAR-100 images have large maximum weighted sum scores and are not correctly classified as outliers. Thus, the accuracy of this outlier detection scheme is less than 0.7.Analysis. The low accuracy of relying on the maximum probability or maximum weighted sum to detect outliers is caused by the contradictory requirements of accurately classifying images and effectively detecting outliers. It is well known that to accurately classify images, the image classifier has to have excellent generalization capability such that the gap between the training and testing images can be overcome -in other words avoiding overfitting. For this, regularization functions such as data augmentation BID30, random dropout BID31, or weight decay BID14 are commonly used to improve the generalization of CNNs. However, these generalization methods inevitably jeopardize the outier detection capability of the model. This is because some images will be classified to one class even if they are quite different from the common features of that class extracted during training. Yet instead such images might be outliers -generalization tends to blur the boundary between normal images and outliers. Therefore, high confidence may be assigned to image outliers as shown in above use case. To address this shortcoming of simply using the maximum class probability as the outlier score, we propose a deep neural decision forest-based approach that harmonizes the contradictory requirements of accurate image classification and effective outlier detection in one network structure. It does this by taking advantage of the strong bias of the classification decisions made within each tree and the ensemble nature of the decision forest. In this section, we first introduce the deep neural forest model BID10. We then show how to use the maximum route of each tree to distinguish outliers from inliers. Fully Connected LayersFinal Fully Connected Layer DISPLAYFORM0 The deep neural forest BID10 combines the representation learning of deep convolutional networks and the divide-and-conquer principle of decision trees. The key idea is to introduce a back-propagation compatible version of stochastic decision trees. As depicted in FIG1, the deep neural forest is composed of three key components: a deep CNN, decision nodes (split nodes), and prediction nodes (leaf nodes). The deep CNN component holds parameters from all convolutional modules and the FC layers except for the final FC layer. Decision nodes indexed by N are the internal nodes of the tree. Each decision node is connected to one output node of the final FC layer. Prediction nodes indexed by L are the leaf nodes of the tree. Each decision node n ∈ N is assigned a decision function d n (.; Θ): X → parametrized by Θ. Each decision node n classifies a sample x based on its features and routes it left or right down the tree. Note that the routing decision d n is probabilistic; formally d n (x ; Θ) is defined as: DISPLAYFORM0 where σ(x) = (1 + e −x) −1 is the sigmoid function and function f n (.; Θ) is a linear output unit provided by the FC layer of a deep network parametrized by Θ. Therefore, it is f n (.; Θ) that connects the neural network and the decision tree. The function is turned into a probabilistic routing decision in the range by applying the sigmoid activation. Each prediction node l ∈ L holds a probability distribution π l over the output space Y. Once a sample ends at a leaf node l, a prediction is given by π l. In the case of stochastic routing, the leaf predictions will be averaged by the probability of reaching the leaf. The final prediction for sample x from tree T with decision nodes parametrized by Θ is given by DISPLAYFORM1 where π = (π l) l∈L and π ly denotes the probability that a sample reaching leaf l belongs to class y. µ l (x |Θ) is considered as the routing function. It produces the probability that sample x will reach leaf l. l µ l (x |Θ) = 1 for all x ∈ X. Since π l is not influenced by the the input x, essentially the routing probability (µ l) determines the class of x. Based on the above description, given a sample image x, each decision tree in the forest produces a probability distribution µ(x |Θ) over each route, or path from root-to-leaf, where µ l (x |Θ) represents the probability that sample x reaches leaf l. More specifically, µ l (x |Θ) is expressed as follows: DISPLAYFORM0 whered n (x ; Θ) = 1 − d n (x ; Θ), and 1 l n is 1 if leaf l belongs to the left subtree of node n and 0 otherwise. Similarly, 1 l n is 1 if leaf l belongs to the right subtree of node n and 0 otherwise. As shown in Eq. 3, the probability of a route is computed as the product of the probabilities w.r.t. all split nodes on that route. Therefore, given an image x, the probability distribution of the routes is expected to be very skewed and biased to one particular route. A given route will have an extremely small probability if x does not fit the features represented by the split nodes on this route learned through the deep CNN, because the product of multiple small probabilities will diminish quickly. Typically just one route will stand out if all its split nodes match the features of x well. We call this route the max route, because it has the maximum probability. It determines the class of x. The probability of the max route (or max route probability) can be used to measure how confident the classifier is about its classification decision of image x. The larger the max route probability is, the more confident the classifier is about the image. More specially, given an image x and a decision tree T, the confidence is measured as: DISPLAYFORM1 where max {µ l (x |Θ)|l ∈ L} denotes the max route probability of decision tree T for the given image x. Since a forest is an ensemble of decision trees, the final confidence of image x is measured as: DISPLAYFORM2 Discussion of the Effectiveness in Outlier Detection. Intuitively this max route probability can be expected to be more effective than the maximum weighted sum of the classes of the CNN at separating outliers, as an outlier image will not have a good match with every split node on the max route. So its max route probability is limited by the product operation in the computation. In contrast, the maximum weighted sum in CNN tends to fall off much more slowly because the score is computed based on the linear combination of multiple features, such that a single matching feature can make the score high. This is also confirmed in our experimental studies. To further improve the effectiveness of using the max route probability to detect outliers, we introduce a regularization to prevent the generation of routing decision whose probability distribution is close to uniform. We use a information theory-based approach. That is, we penalize the routing decision whose probability distribution has a large entropy, because a uniform distribution has a large entropy. This ensures that the max route probability of each routing decision always stands out, making it more effective at detecting outliers. More specifically, given a decision tree T, |L| routing options exist for each sample x ∈ X reaching different prediction nodes. The entropy of the route probability distribution of sample x is given by: DISPLAYFORM0 where µ i (x |Θ) denotes the probability that sample x reaching leaf l i. However, the learning process does not converge during runtime if we directly apply this penalty function on the routing probability distribution due to numerical instability. We solve this problem by applying a softmax function on the probability distribution as a normalization. The revised entropy of the routing probability distribution is then given by: DISPLAYFORM1 where DISPLAYFORM2 To penalize the routing whose probability distribution has a large entropy, we add the entropy w.r.t. each training sample to the log-loss term. Given a sample x ∈ X and an output distribution y ∈ Y, the log-loss term of one decision tree T is represented as: DISPLAYFORM3 where β controls the strength of the penalty and DISPLAYFORM4 Therefore, the total log-loss for the random forest composed of |T | trees is defined as: Fig. 1(b) shows the max route probability distributions w.r.t. CIFAR-10 and CIFAR-100 tested on the CIFAR-10 model. By setting k as 5000, we get a max route probability cutoff threshold ct = 0.99864. As confirmed, 95% of CIFAR-100 images have their max route probabilities smaller than this large ct. Therefore, it achieves an outlier detection accuracy around 0.95, making it much more effective than using a threshold on CNN class weights. DISPLAYFORM5 Although skewness in the classification probability distribution (for example, the output of softmax in CNN) can benefit the detection of outlier images, it is known to hurt the generalization of image classifiers due to the high risk of overfitting BID24. The situation seems worse for our deep neural decision forest-based approach, because the product operation in the routing probability computation makes the probability distribution of the routing more skewed compared to the output of softmax in CNN.We overcome this in our decision forest-based approach as a of the ensemble nature of the decision forest, allowing us to maintain good generalization performance. This is because, although one tree may overfit some aspects of the images, the ensemble provides a tool to make a composite prediction so that the overfitting in each individual tree is overcome. More specifically, as an ensemble of decision trees F = {T 1, . . ., T k}, a forest F delivers a prediction for a sample x by averaging on the output of each tree, i.e., DISPLAYFORM0 Enhancement of Generalization Capability. Next, we introduce a new architecture of the deep neural decision forest to further enhance its generalization capability. As shown in FIG1, although in BID10, different trees are connected to different sets of nodes of the final FC layer, the trees still share the other FC layers of the CNNs. This violates the principle of random decision forest where the trees in a forest should be as independent as possible so that that they can compensate each other. That is why decision forests tend to show excellent generalization capability. Therefore, we design a new architecture FIG2, Appendix A). It divides all FC layers into k independent components, each of which is connected to one individual tree. This ensures independence amongst the trees and in turn improves the classification accuracy. In summary, regularization and our new architecture together enable our deep neural decision forest-based approach to provide high accuracy at image classification while also assuring effective outlier detection, as we demonstrate in our experiments (Sec. 5).In Appendix B, we show how to train the deep neural decision tree. The training requires estimating both the decision node parametrizations θ and the leaf predictions π. In BID10 the minimum empirical risk principle under log-loss is adopted for estimation by using a two-step optimization strategy, where θ and π are updated alternatively to minimize the log-loss. In this work we use a new form of prediction nodes that makes the deep neural forest fully differentiable. This enables us to abandon the two-step optimization strategy and instead optimize the Θ and Π parameters jointly in one step through back-propagation. This back-propagation based learning process is also described in Appendix B. Datasets. We empirically demonstrate the effectiveness of our proposed image outlier detection (IOD) strategy on several benchmark image datasets. Specifically, we train models on CIFAR-10 (BID11) and MNIST datasets. Given a trained model on one data set, we consider examples from other datasets as outliers when testing the model. For example given a CIFAR-10 model, images in CIFAR-100, SVHN BID20, and MNIST datasets are outliers, because the images in these datasets have features distinct from those common to the CIFAR-10 images captured by the classification model. We evaluate: IOD-IOD-Weighted-Sum described in Sec. 2. The weighted sum score in the final FC layer of the traditional CNN is utilized as the confidence measure; IOD-IOD-Max-Route-Shared-FC method based on the original deep neural forest architecture proposed in BID10 in which each tree shares the FC layers as described in Sec. 3.1. Here, the max route is used as the confidence measure of IOD; IODMax-Route-Different-FC: it uses our new deep neural forest architecture. Different trees are connected to isolated FC layers. Similarly the max route is used as the confidence measure; IOD-Max-Route-Penalty: it uses our new deep neural forest architecture with a regularization term applied to the loss function (Eq. 8); Deep SVDD BID26: the state-of-art fully deep once-class method described in Sec. 6; AnoGAN BID27: the state-of-art generative approach based on GAN. We compare against Deep SVDD and AnoGAN because they focus on image datasets. Experimental Setup. We ran experiments on a GPU. All IOD models are implemented based on Pytorch BID22. For Deep SVDD and AnoGAN we reuse the codes provided by their authors. The code and models will be made public available via Github. Metrics. We measure the accuracy of outlier detection and the classifation accuracy of our IOD-based approaches. Parameter Settings. We set learning rates manually. The detailed settings of the parameters are provided in each subsection. All networks are trained using mini-batches of size 128. The momentum is set to 0.9 for all models. The weighted decays are set to 0.0001. We do not use data augmentation. We use simple global contrast normalization to pre-process all images. When testing CIFAR-10, CIFAR-100, and SVHN datasets on the model trained for MNIST, we change the image to gray scale and take central crops of the images. On the other hand, when testing MNIST on models trained for other datasets, we increase its color channel from 1 to 3 by copying the original gray image 3 times. The CIFAR-10 dataset BID11 ) is composed of 10 classes of natural images with 50,000 training images and 10,000 testing images. Each image is an RGB image of size 32 × 32.We use a deep NN composed of 10 convolutional layers and 2 FC layers with 1024 and 384 hidden units correspondingly. For the IOD-Weighted-Sum method, a 10-way linear layer is used for the final prediction. For the other three max route-based approaches, the output is connected to a deep neural forest containing 20 depth-3 trees. Specifically, for the IOD-Max-Route-Shared-FC method, the output of the first FC layer using 384 hidden units is connected to the second FC layer with 300 (= 20 × (2 (3 +1) − 1 )) hidden nodes. It is then connected to 20 depth-3 trees. For the IOD-Max-Route-Different-FC and the IOD-Max-Route-Penalty methods, the output of the convolutional layer is broadcast to 20 different sets of FC layers. Each set contains 2 FC layers using 1024 and 384 hidden units. Each connects to the final FC layer with 15 (= 2 (3 +1) − 1 ) hidden nodes. This final FC layer is then connected to a tree. The learning rate is initialized as 0.1 and eventually decays to 0.001. For DEEP SVDD and AnoGAN we use the settings recommended by the authors. The parameter k is set as 5000. As shown in Sec. 2 k is used to establish an outlierness cut off threshold ct. For our own IOD approach, a testing image is an outlier if its maximum weighted sum score or max route probability is smaller than ct. For DEEP SVDD and AnoGAN an image is an outlier if its outlierness score is larger than the corresponding ct. In comparison to the state-of-the-art TAB0, our IOD-Max-Route-Penalty method significantly outperforms Deep SVDD and AnoGAN in all cases. Deep SVDD performs poorly, because as a one-class method it is not good at separating outliers from inliers that belong to multiple normal classes. AnoGAN only works well in detecting the MNIST images as outliers which are significantly different from CIFAR-10, while fails in all other cases when the outliers share more features with the inliers. As for our own IOD-based methods, our max route-based methods significantly outperform IOD-Weighted-Sum that uses weighted-sum as confidence measure to detect outliers, without giving up our ability to correctly classify the CIFAR-10 images. The performance gain from the deep neural forest architecture, in which the confidence w.r.t. each image is computed as the multiplication of the routing probabilities produced at the decision nodes on the route. Compared to the weighted sum approach, this multiplication of probabilities enlarges the confidence gap. Thus it is able to better detect outliers than the weighted sum approach. In addition, two of our max route-based methods (IOD-Max-Route-Shared-FC and IOD-Max-Route-Different-FC) achieve even better classification accuracy compared to the weighted sum-based method that uses the classical CNN architecture. The IOD-Max-Route-Penalty method outperforms the other two IOD-Max-Route based methods in detecting outliers in almost all cases, especially in detecting MNIST outliers. This is because by introducing regularization to penalize routing decisions that have large entropy, IOD-Max-Route-Penalty avoids uniform route probability distributions and thus leads to larger maximum route probabilities for inliers. At the same time, its classification accuracy decreases a little bit, because the regularization might introduce overfitting BID32. However, it is effectively alleviated because of the ensemble of the decision trees. Therefore, the drop on the classification accuracy is almost negligible. Moreover, IOD-Max-Route-Different-FC consistently outperforms IOD-Max-Route-Shared-FC in both outlier detection and classification. This demonstrates the effectiveness of our new proposed deep neural forest architecture as compared to the original version. This new architecture has a large capacity in detecting outliers and classifying images because of the fully isolated decision trees in the forest. Due to space limitation, please refer to Appendix C for the on MNIST. Classical Outlier Detection. Outlier detection has been extensively studied in the literature BID3 BID9 BID25 BID1. These outlier detection techniques in general share one common principle. Namely, an object is an outlier if its outlierness score is above a certain cutoff threshold. This principle is also leveraged in our context. However, unlike these works BID3 BID9 BID25 BID1, the outlierness score is not measured in the original feature space. This avoids the similarity search problem in the high dimensional image data. One-Class Classification. One-class Support Vector Machine (OC-SVM) and Support Vector Data Description (SVDD) methods BID28 BID18 BID33 use only normal training data to separate outliers from normal data based on kernel-based transformations. In particular, OC-SVM uses a hyperplane BID28 BID18 to separate the target objects from the origin with maximal margin. It is based on the assumption that the origin of a kernel-based transformed representation belongs to the outlier class. SVDD is a different kernel SVM method in which data is enclosed in a hypersphere of radius R in the transformed feature space. The squared radius is minimized together with penalties for margin violation. However, these approaches are shown to be extremely sensitive to specific choices of the representation, kernel, and the hyper-parameters. The difference in performance can be dramatic based on these choices BID19. Therefore, these methods are not robust. In addition, they assumed that all normal data belongs to one single class. This does not fit our scenario in which the applications generate rich classes of images. Shallow Image Outlier Solutions. In BID29, the authors proposed an unsupervised image outlier method. It first extracts a set of image features from the raw pixels and several combinations of image transforms. Then, the mean and the variance values of each feature are computed over all images in the dataset. An image is considered as an outlier if the values of its features significantly deviate from their mean values. This approach performs poorly even on the simplistic MNIST due to its limited feature extraction ability. In BID8, the authors proposed a probabilistic PCA model focusing on the extraction of the features that in accurate image reconstruction. During this process, the outliers are identified and removed to improve the performance of the PCA model. Therefore, the target of this work is on extracting the most typical features instead of outlier detection. Deep One-class methods. Research efforts have been put on enhancing one-class classification with the representation learning ability of deep neural network. In BID23; BID5, "mix" methods have been proposed that detect outliers by first learning a representation using deep learning and then feed that into a classical shallow outlier detection method. Recently, "fully deep" methods BID26 BID21 were proposed that produce representations and the boundary separating outliers from inliers (hyperplane in OC-SVM and hypersphere in SVDD) within one learning process. These methods outperform the mix methods according to BID26. Unfortunately, these methods still suffer from the fundamental problem of one-class classification, namely they only find a single boundary between outliers and inliers. However, real datasets tend to have multiple classes of normal images. For multi-class datasets, it is difficult to find a separator that encompasses all normal classes and none of the outliers. As confirmed in our experiments, our IOD method significantly outperforms the one-class approaches when handling complex image datasets such as CIFAR-10.Deep Autoencoder-based Methods. Deep autoencoder has been broadly used to detect outliers. It is based on the observation that these networks are able to extract the common factors of variation from normal samples and reconstruct them accurately, while anomalous samples do not contain these common factors of variation and thus cannot be reconstructed accurately. The deep autoencoders can be used to detect outliers in a mixed approach BID5, namely by plugging in the learned embeddings into classical outlier detection methods. It can also be directly used to detect outliers by employing the reconstruction error as the outlier score BID36 BID4. However, the objective of autoencoders is dimension reduction. It does not target on outlier detection directly. Therefore, the learned low dimensional representation is not necessarily effective in distinguishing outliers from inliers. Furthermore, choosing the right degree of compression is also difficult when applying autoencoders for outlier detection. GAN-based Methods. The GAN-based method BID27 ) (AnoGAN) first trains a GAN to generate samples according to the training data. A test object is an outlier if it cannot find a close point in the generator's latent space. However, similar to autoencoders, generative approaches have difficulty in regularizing the generator for compactness. Deep Statistical Methods. The deep statistical methods detect outliers by directly modeling the data distribution with the help of deep structures such as autoencoder. In particular, in BID37 the Deep Autoencoding Gaussian Mixture Model (DAGMM) method uses a deep autoencoder to generate a low-dimensional representation and reconstruction error which are then used to build a Gaussian Mixture Model (GMM). The parameters of the deep autoencoder and the mixture model are jointly optimized in an end-to-end fashion. Similarly, in BID35 the deep structured energy based models (DSEBMs) detects outliers by connecting the autoencoder and the energy based model (EBM). However, the statistical methods detect outliers purely based the density of the objects. The model might assign high density to the outliers if there are many proximate outliers, hence ing in false negative. Open Set Deep Network. a method was introduced to adapt Convolutional Neural Networks to discover a new class by adding one additional class at the softmax layer. It leverages the weighted sum score to determine whether one image belongs to the new class. However, its techniques are tightly coupled with one specific network architecture, namely AlexNet BID12. Our IOD framework instead can accommodate any type of image classifiers. Learning from Noisy Labeled Image Data. In BID34, the authors studied the problem of training CNNs that are robust to a large number of noisy labels. The idea is to extend CNNs with a probabilistic model, which infers the true labels from the noisy labels and then uses them as clean labels in the training of the network. This assumes that the noisy labels or outliers are already known. Therefore, this is not only totally different from our outlier detection problem, but also not practical in most applications where outliers are unknown and unexpected phenomenon. In this work we propose a novel approach that effectively detects outliers from image data. The key novelties include a general image outlier detection framework and effective outlierness measure that leverages the deep neural decision forest. Optimizations such as new architecture that connects deep neural network and decision tree and regularization to penalize the large entropy routing decisions are also proposed to further enhance the outlier detection capacity of IOD. In the future we plan to investigate how to make our approach work in multi-label classification setting. Convolutional Modules DISPLAYFORM0 DISPLAYFORM1 Fully Connected Layers − (Learning a deep neural decision tree requires estimating both the decision node parametrizations θ and the leaf predictions π. In BID10 the minimum empirical risk principle under log-loss is adopted for their estimation by using a two-step optimization strategy, where θ and π are updated alternatively to minimize the log-loss. In this section we first introduce a new form of prediction nodes that makes the deep neural forest fully differentiable. This enables us to abandon the two-step optimization strategy and instead optimize the Θ and Π parameters jointly in one step through back-propagation. Next, we show the back-propagation based learning process. The prediction node used in BID10 holds a probability distribution π l over Y and are optimized alternately with the decision nodes d. Specifically, the optimal predictions for all prediction nodes can be obtained by minimizing a convex objective given fixed decision nodes, while the parameters in the decision nodes are trained through the back-propagation process in each training epoch. Obviously, this two-step strategy is not efficient. Here, we use new forms of prediction nodes that makes the decision forest fully differentiable and therefore can be optimized jointly with the decision nodes as shown in Sec. B.2.Each prediction node is parametrized using a k-dimensional parametric probability distribution w l, denoted as: DISPLAYFORM0 where k is the number of classes. The softmax function takes a vector of real-valued scores and converts it to a vector of values between 0 and 1 that sum to one. In Sec. B.2, we will show that using the new prediction nodes, the final loss function is still convex. Learning the random forest model requires us to find a set of parameters Θ and π that can minimize the total log loss defined in Eq. 9. To minimize Eq. 9, in fact we only need to independently minimize the penalized loss (Eq. 8) of each individual tree. Next, we show that the loss function is fully differentiable. Therefore, we are able to employ a Stochastic Gradient Descent (SGD) approach to minimize the loss w.r.t. Θ and π, following the common practice of back-propagation in deep neural networks. Learning Decision Nodes by Back-Propagation. Given a decision tree, the gradient of the loss L with respect to Θ can be decomposed by the chain rule as follows: DISPLAYFORM0 Here, the derivative of the second part ∂fn (x ;Θ) ∂Θ is identical to the back-propagation process of traditional CNN modules and thus is omit here. Now let's show how to compute the first part: DISPLAYFORM1 where DISPLAYFORM2 and DISPLAYFORM3 By using the chain rule DISPLAYFORM4 where DISPLAYFORM5 and DISPLAYFORM6 where DISPLAYFORM7 Learning Prediction Nodes by Back-Propagation. Given a decision tree, the gradient of the Loss L w.r.t. the weights w of the prediction nodes (defined in Eq. 11) can be decomposed by the chain rule as follows: DISPLAYFORM8 where DISPLAYFORM9 and DISPLAYFORM10 DISPLAYFORM11 C Experiment Results on MNIST MNIST consists of 28 × 28 pixel grayscale images of handwritten digits from 0 to 9. There are 60,000 training images and 10,000 testing images in the dataset. In this case we use a neural network composed of 3 convolutional layers and one FC layer with 625 hidden units. Similar to the CIFAR-10 model, the weighted-sum method uses a 10-way softmax layer for the final prediction, while the max route-based methods connect the neural network to a decision forest containing 5 depth-8 trees. The initial learning rate is set as 0.0001 for the weighted-sum method and 0.001 for other max route-based methods. A multi-step decay is then applied in 50 epochs. The number of outliers in MNIST is set to 200 (k = 200) to get the confidence cutoff threshold (ct) (Sec. 2). BID26 94.73% 94.565% 94.49% XX% XX% AnoGAN BID27 96.50% 95.10% 88.67% XX% XX%As shown in TAB2, similar to the CIFAR-10 experiments, our max route-based IOD methods significantly outperforms the weighted-sum based IOD method in both outlier detection and classification due to the same reason discussed in the CIFAR-10 experiments. As for our three max route-based IOD approaches, the IOD-Max-Route-Penalty approach again outperforms the other two approaches in outlier detection in all cases, while only negligible classification accuracy is sacrificed. Moreover, IOD-Max-Route-Different-FC consistently outperforms IOD-Max-Route-Shared-FC in both outlier detection and classification because of the new deep neural forest architecture. Deep SVDD and AnoGAN perform well on this simplistic MNIST dataset, because the testing datasets (CIFAR-10, CIFAR-100, and SVHN) which are treated as outliers are significantly different from MNIST as normal data. In BID6, a new theoretical framework was proposed that casts dropout used in training deep neural networks as approximate Bayesian inference in deep Gaussian processes. A direct of this theory gives us a tool to model uncertainty with dropout NNs. As discussed in BID6, dropout is applied during inference. Given an image, we can get a range of softmax input values for each class by measuring 100 stochastic forward passes of the softmax input. If the range of the predicted class intersects that of other classes, then even though the softmax output is arbitrarily high (as far as 1 if the mean is far from the means of the other classes), the uncertainty of the softmax output can be as large as the entire space. In other words, the size of intersection signifies the model's uncertainty in its softmax output value -i.e., in the prediction. The larger the intersection is, the more uncertain the model is in its prediction. Therefore, this uncertainty could be used as a score to measure the outlierness of the image. Therefore, we evaluates this dropout method as an additional baseline method. Evaluation on CIFAR-10 Data. We use the identical network architecture as suggested in the author's github repository (https://github.com/yaringal/ DropoutUncertaintyCaffeModels/tree/master/cifar10_uncertainty).When the method is applied on the CIFAR-10 training data, for each CIFAR-10 image, we forward it in the model 100 times and record the softmax input values for each class. The "outlierness" (uncertainty) of the image is defined as the intersection between predict class and all other classes. We use the 5000-th largest value in the training as the cutoff threshold. Specifically, if an image has an uncertainty larger than the threshold, it is considered to be as an outlier. Then we forward each CIFAR-100 image 100 times in the model and record the outlier score for each CIFAR-100 images. Unfortunately, the accuracy of this outlier detection scheme is only 53%. That is, this is even worse than our maximum weighted sum baseline which has an accuracy of 70%.Evaluation on MNIST Data. We also applied the Dropout method on MNIST training images. Again we use the network architecture suggested by the authors in their repository. We forward each MNIST image 100 times in the model and compute the outlierness score. However, when we use the 200-th largest outlierness score in MNIST training as the outlierness cutoff threshold (the same parameter setting to our MNIST experiment in Appendix C), the accuracy in detecting CIFAR-10 images (also forward 100 times in the model) as outliers is lower than 10%. When we increase the parameter from 200 to 5000, its accuracy in detecting outliers increases to 48.13%, which is still much much lower than our proposed method (above 90%), although clearly in this case the parameter setting biases to Dropout method. We evaluate how the input parameter k influences the accuracy of outlier detection. As discussed in Sec. 2, in our IOD-based method, k is used to establish a confidence cut off threshold that corresponds to the kth smallest maximum route probability or maximum weighted sum among the training objects. While in Deep SVDD and AnoGAN, it corresponds to the kth largest outlierness score among the normal objects. In our IOD-based method, given a testing image, if its maximum weighted sum or maximum route probability is smaller than the cutoff threshold, it is considered as an outlier. In Deep SVDD and AnoGAN, a testing image is considered as an outlier if its outierness score produced by Deep SVDD or AnoGAN is larger than the corresponding cutoff threshold. FIG3 shows the on the CIFAR-10 model. Our IOD-based IOD-Max-RoutePenalty (Max-Penalty) method significantly outperforms the state-of-art SVDD and AnoGAN methods in all cases. When k decreases, all methods have a lower accuracy of outlier detection. This is expected, because a smaller k produces a smaller cut off threshold in our IOD-based method and a larger cut off in Deep SVDD and AnoGAN, both leading to the detection of smaller number of outliers. Similarly, as shown in FIG4, our IOD-Max-Route-Penalty (Max-Penalty) consistently outperforms all other methods in almost all cases when detecting outiers out of the simpler MNIST model. The only exception is that AnoGAN is slightly better than Max-Penalty when detecting CIFAR-10 outliers out of MNIST model. Isolation Forest Liu et al. FORMULA2 is a widely used unsupervised outlier detection method. It builds an ensemble of isolation trees for a given data set. Then the average path lengths on the isolation trees is used as the outlierness measurement. The shorter the average path length is, the more likely the instance is to be an outlier. Evaluation on CIFAR-10 Data. First, we used a CNN to extract features from the raw image data and then build an Isolation Forest on these extracted features to detect outliers. As one input parameter of Isolation Forest, the number of outliers in CIFAR-10 is set as 5000 -identical to the parameter k used in our IOD-based methods. More specifically, similar to our IOD-based approach, we use the 5,000th smallest average path length in CIFAR-10 as the cutoff threshold. If the average path length of a testing image is smaller than the cutoff threshold, it is considered an outlier. When the model is trained on CIFAR-10, we expect all images in CIFAR-100, MNIST and SVHN to be detected as outliers. However, in fact the outlier detection accuracy is poor -less than 2% in all cases, although we have carefully tuned the size of the network from producing 2042 dimensional feature vector to 512 dimensional feature vector and tuned other parameters in Isolation Forest such as the number of the trees and max_sample. We also tried applying dimensionality reduction techniques to reduce the dimension of the extracted feature vectors and then applying Isolation Forests on the lower dimension space. The were slightly better, although the detection rate for outliers is still lower than 2%.Finally, we directly applied Isolation Forests on the raw image. The outlier detection accuracy with respect to CIFAR-100, SVHN, and MNIST was 9.73%, 13.14% and 8.3% respectively.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A novel approach that detects outliers from image data,  while preserving the classification accuracy of image classification

Represent the short summary for finding the corresponding scientific article (long)

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources. We design a Dynamic Point-cloud Convolution (D-Conv) operator as the core component of CloudLSTMs, which performs convolution directly over point-clouds and extracts local spatial features from sets of neighboring points that surround different elements of the input. This operator maintains the permutation invariance of sequence-to-sequence learning frameworks, while representing neighboring correlations at each time step -- an important aspect in spatiotemporal predictive learning. The D-Conv operator resolves the grid-structural data requirements of existing spatiotemporal forecasting models and can be easily plugged into traditional LSTM architectures with sequence-to-sequence learning and attention mechanisms. We apply our proposed architecture to two representative, practical use cases that involve point-cloud streams, i.e. mobile service traffic forecasting and air quality indicator forecasting. Our , obtained with real-world datasets collected in diverse scenarios for each use case, show that CloudLSTM delivers accurate long-term predictions, outperforming a variety of neural network models. Point-cloud stream forecasting aims at predicting the future values and/or locations of data streams generated by a geospatial point-cloud S, given sequences of historical observations . Example data sources include mobile network antennas that serve the traffic generated by ubiquitous mobile services at city scale (b), sensors that monitor the air quality of a target region , or moving crowds that produce individual trajectories. Unlike traditional spatiotemporal forecasting on grid-structural data, like precipitation nowcasting or video frame prediction , point-cloud stream forecasting needs to operate on geometrically scattered sets of points, which are irregular and unordered, and encapsulate complex spatial correlations. While vanilla Long Short-term Memories (LSTMs) have modest abilities to exploit spatial features , convolution-based recurrent neural network (RNN) models, such as ConvLSTM and PredRNN++ , are limited to modeling grid-structural data, and are therefore inappropriate for handling scattered point-clouds.: Different approaches to geospatial data stream forecasting: predicting over input data streams that are inherently grid-structured, e.g., video frames using ConvLSTMs (top); mapping of pointcloud input to a grid, e.g., mobile network traffic collected at different antennas in a city, to enable forecasting using existing neural network structures (middle); forecasting directly over point-cloud data streams using historical information (as above, but without pre-processing), as proposed in this paper (bottom). permutations for the features . Through this, the proposed PointCNN leverages spatial-local correlations of point clouds, irrespective of the order of the input. Notably, although these architectures can learn spatial features of point-clouds, they are designed to work with static data, thus have limited ability to discover temporal dependencies. Next, we describe in detail the concept and properties of forecasting over point cloud-streams. We then introduce the DConv operator, which is at the core of our proposed CloudLSTM architecture. Finally, we present CloudLSTM and its variants, and explain how to combine CloudLSTM with Seq2seq learning and attention mechanisms, to achieve precise forecasting over point-cloud streams. We formally define a point-cloud containing a set of N points, as S = {p 1, p 2, · · ·, p N}. Each point p n ∈ S contains two sets of features, i.e., p n = {ν n, ς n}, where ν n = {v 1 n, · · ·, v H n} are value features (e.g., mobile traffic measurements, air quality indexes, etc.) of p n, and ς n = {c 1 n, · · ·, c L n} are its L-dimensional coordinates. At each time step t, we may obtain U different channels of S by conducting different measurements denoted by S Note that, in some cases, each point's coordinates may be unchanged, since the data sources are deployed at fixed locations. An ideal point-cloud stream forecasting model should embrace five key properties, similar to other point-cloud applications and spatiotemporal forecasting problems (a;): (i) Order invariance: A point cloud is usually arranged without a specific order. Permutations of the input points should not affect the output of the forecasting (a). (ii) Information intactness: The output of the model should have exactly the same number of points as the input, without losing any information, i.e., N out = N in. (iii) Interaction among points: Points in S are not isolated, thus the model should be able to capture local dependencies among neighboring points and allow interactions (a). (iv) Robustness to transformations: The model should be robust to correlation-preserving transformation operations on point-clouds, e.g., scaling and shifting (a). (v) Location variance: The spatial correlations among points may change over time. Such dynamic correlations should be revised and learnable during training . In what follows, we introduce the Dynamic Point Cloud Convolution (DConv) operator as the core module of the CloudLSTM, and explain how DConv satisfies the aforementioned properties. The Dynamic Point Cloud Convolution operator (DConv) generalizes the ordinary convolution on grids. Instead of computing the weighted summation over a small receptive field for each anchor point, DConv does so on point-clouds, while inheriting desirable properties of the ordinary convolution operation. The vanilla convolution takes U in channels of 2D tensors as input, and outputs U out channels of 2D tensors of smaller size (if without padding). Similarly, the DConv takes U in channels of a point-cloud S, and outputs U out channels of a point-cloud, but with the same number of elements as the input, to ensure the information intactness property (ii) discussed previously. For simplicity, we denote the i th channel of the input set as S and S j out are 3D tensors, of shape (N, (H +L), U in ) and (N, (H +L), U out ) respectively. We also define Q K n as a subset of points in S i in, which includes the K nearest points with respect to p n in the Euclidean space, i.e., Q n is the k-th nearest point to p n in the set S i in. Note that p n itself is included in Q K n as an anchor point, i.e., p n ≡ p 1 n. Recall that each p n ∈ S contains H value features and L coordinate features, i.e., p n = {ν n, ς n}, where Here, each w k is a set of weights w with index k (i.e., k-th nearest neighbor) in Eq. 2, shared across different p. in S i in, the DConv sums the element-wise product over all features and points in Q K n, to obtain the values and coordinates of a point p n in S j out. Note that we assume the value features are related to their positions at the previous layer/state, to better exploit the dynamic spatial correlations. Therefore, we aggregate coordinate features c(p In the above, we define learnable weights W as 5D tensors with shape . The weights are shared across different anchor points in the input map. Each element w m,m,k i,j ∈ W is a scalar weight for the i-th input channel, j-th output channel, k-th nearest neighbor of each point corresponding to the m-th value and coordinate features for each input point, and m -th value and coordinate features for output points. Similar to the convolution operator, we define b j as a bias for the j-th output map. In the above, h and h are the h -th value features of the input/output point set. Likewise, l and l are the l -th coordinate features of the input/output. σ(·) is the sigmoid function, which limits the range of predicted coordinates to, to avoid outliers. Before feeding them to the model, the coordinates of raw point-clouds are normalized to by ς = (ς − ς min)/(ς max − ς min), on each dimension. This improves the transformation robustness of the operator. The K nearest points can vary for each channel at each location, because the channels in the pointcloud dataset may represent different types of measurements. For example, channels in the mobile traffic dataset are related to the traffic consumption of different mobile apps, while those in the air quality dataset are different air quality indicators (SO 2, CO, etc.). The spatial correlations will vary between different measurements (channels), due to human mobility. For instance, more people may use Facebook at a social event, but YouTube traffic may be less significant in this case. This will be reflected by the data consumption of each app. The same applies to air quality indicators affected by vehicle movement and factory working times. We want these spatial correlations to be learnable, so we do not fix the K nearest neighbors across channels, but encourage each channel to find the best neighbor set. This is also a contribution of the CloudLSTM, which helps improve the forecasting performance. We provide a graphical illustration of DConv in Fig. 2. For each point p n, the DConv operator weights its K nearest neighbors across all features, to produce the values and coordinates in the next layer. Since the permutation of the input neither affects the neighboring information nor the ranking of their distances for any Q K n, DConv is a symmetric function whose output does not depend on the input order. This means that the property (i) discussed in Sec. 3.1 is satisfied. Further, DConv is performed on every point in set S i in and produces exactly the same number of features and points for its output; property (ii) is therefore naturally fulfilled. In addition, operating over a neighboring point set, irrespective of its layout, allows to capture local dependencies and improve the robustness to global transformations (e.g., shifting and scaling). The normalization over the coordinate features further improves the robustness to those transformations, as shown by the proof in Appendix C. This enables to meet the desired properties (iii) and (iv). More importantly, DConv learns the layout and topology of the cloud-point for the next layer, which changes the neighboring set Q K n for each point at output S j out. This enables the "location-variance" (property (v)), allowing the model to perform dynamic positioning tailored to each channel and time step. This is essential in spatiotemporal forecasting neural models, as spatial correlations change over time . DConv can be efficiently implemented using simple 2D convolution, by reshaping the input map and weight tensor, which can be parallelized easily in existing deep learning frameworks. We detail this in Appendix A and provide a complexity analysis of the DConv operator in Appendix B. Relations with PointCNN and Deformable Convolution . The DConv operator builds upon the PointCNN and deformable convolution neural network (DefCNN) on grids , but introduces several variations tailored to pointcloud structural data. PointCNN employs the X -transformation over point clouds, to learn the weight and permutation on a local point set using multilayer perceptrons (MLPs), which introduces extra complexity. This operator guarantees the order invariance property, but leads to information loss, since it performs aggregation over points. In our DConv operator, the permutation is maintained by aligning the weight of the ranking of distances between point p n and Q K n. Since the distance ranking is unrelated to the order of the inputs, the order invariance is ensured in a parameter-free manner without extra complexity and loss of information. Further, the DConv operator can be viewed as the DefCNN over point-clouds, with the differences that (i) DefCNN deforms weighted filters, while DConv deforms the input maps; and (ii) DefCNN employs bilinear interpolation over input maps with a set of continuous offsets, while DConv instead selects K neighboring points for its operations. Both DefCNN and DConv have transformation modeling flexibility, allowing adaptive receptive fields on convolution. The DConv operator can be plugged straightforwardly into LSTMs, to learn both spatial and temporal correlations over point-clouds. We formulate the Convolutional Point-cloud LSTM (CloudLSTM) as: Similar to ConvLSTM , i t, f t, and o t, are input, forget, and output gates respectively. C t denotes the memory cell and H t is the hidden states. Note that i t, f t, o t, C t, and H t are all point cloud representations. W and b represent learnable weight and bias tensors. In Eq. 3,'' denotes the element-wise product,'' is the DConv operator formalized in Eq. 2, and' *' a simplified DConv that removes the sigmoid function in Eq. 2. The latter only operates over the gates computation, as the sigmoid functions are already involved in outer calculations (first, second, and fourth expressions in Eq. 3). We show the structure of a basic CloudLSTM cell in the left part of Fig. 3. We combine our CloudLSTM with Seq2seq learning and the soft attention mechanism , to perform forecasting, given that these neural models have been proven to be effective in spatiotemporal modelling on grid-structural data (e.g., . We show the overall Seq2seq CloudLSTM in the right part of Fig. 3. The architecture incorporates an encoder and a decoder, which are different stacks of CloudLSTMs. The encoder encodes the historical information into a tensor, while the decoder decodes the tensor into predictions. The states of the encoder and decoder are connected using the soft attention mechanism via a context vector . Before feeding the point-cloud to the model and generating the final forecasting, the data is processed by Point Cloud Convolutional (CloudCNN) layers, which perform the DConv operations. Their function is similar to the word embedding layer in natural language processing tasks , which helps translate the raw point-cloud into tensors and vice versa. In this study, we employ a two-stack encoder-decoder architecture, and configure 36 channels for each CloudLSTM cell, as we found that further increasing the number of stacks and channels does not improve the performance significantly. Beyond CloudLSTM, we also explore plugging the DConv into vanilla RNN and Convolutional GRU, which leads to a new Convolutional Point-cloud RNN (CloudRNN) and Convolutional Point-cloud GRU (CloudGRU), as formulated in the Appendix E. The CloudRNN and CloudGRU share a similar Seq2seq architecture with CloudLSTM, except that they do not employ the attention mechanism. We compare their performance in the following section. To evaluate the performance of our architectures, we employ measurement datasets of traffic generated by 38 mobile services and recorded at individual network antennas, and of 6 air quality indicators collected at monitoring stations. We use the proposed CloudLSTM to forecast future mobile service demands and air quality indicators in the target regions. We provide a comprehensive comparison with 12 baseline deep learning models, over four performance metrics. All models considered in this study are implemented using the open-source Python libraries TensorFlow and TensorLayer . We train all architectures with a computing cluster with two NVIDIA Tesla K40M GPUs. We optimize all models by minimizing the mean square error (MSE) between predictions and ground truth, using the Adam optimizer . Next, we first introduce the datasets employed in this study, then discuss the baseline models used for comparison. Finally, we report on the experimental obtained. We conduct experiments on two typical spatiotemporal point-cloud stream forecasting tasks over 2D geospatial environments, with measurements collected in two different scenarios for each use case. Note that, as the data sources have fixed locations in these applications, the coordinate features will be omitted in the final output. However, in different use cases, such as crowd mobility forecasting, the coordinate features would be necessarily included. Mobile Traffic Forecasting. We experiment with large-scale multi-service datasets collected by a major operator in two large European metropolitan areas with diverse topology and size during 85 consecutive days. The data consists of the volume of traffic generated by devices associated to each of the 792 and 260 antennas in the two target cities, respectively. The antennas are non-uniformly distributed over the urban regions, thus they can be viewed as 2D point clouds over space. At each antenna, the traffic volume is expressed in Megabytes and aggregated over 5-minute intervals, which leads to 24,482 traffic snapshots. These snapshots are gathered independently for each of 38 different mobile services, selected among the most popular apps for video streaming, gaming, messaging, cloud services, and social networking. Further details about the dataset can be found in Appendix G. Air Quality Forecasting. Air quality forecasting performance is investigated using a public dataset , which comprises six air quality indicators (i.e., PM2.5, PM10, NO 2, CO, O 3 and SO 2) collected by 437 air quality monitoring stations in China, over a span of one year. The monitoring stations are partitioned into two city clusters, based on their geographic locations, and measure data on an hourly basis. The dataset includes 8,760 snapshots in total for each cluster. We conduct experiments on both clusters individually and fill missing data using linear interpolation. The reader is referred to Appendix G for details. Before feeding to the models, the measurements associated to each mobile service and air quality indicator are transformed into different input channels of the point-cloud S. All coordinate features ς are normalized to the range. In addition, for the baseline models that require grid-structural input (i.e., CNN, 3D-CNN, ConvLSTM and PredRNN++), the point-clouds are transformed into grids (a) using the Hungarian algorithm . The ratio of training plus validation, and test sets is 8:2. We compare the performance of our proposed CloudLSTM with a set of baseline models, as follows. PointCNN performs convolution over point-clouds and has been employed for point-cloud classification and segmentation. CloudCNN is an original benchmark we introduce, which stacks the proposed DConv operator over multiple layers for feature extraction from pointclouds. PointLSTM is another original benchmark, obtained by replacing the cells in ConvLSTM with the X -Conv operator employed by PointCNN, which provides a fair term of comparison for other Seq2seq architectures. Beyond these models, we also compare the CloudLSTM with two of its variations, i.e., CloudRNN and CloudGRU, which were introduced in Sec. 3.3. Other baseline models, including MLP , CNN , 3D-CNN , LSTM , ConvLSTM PredRNN++ , along with the detailed configuration of all models are discussed in Appendix E. We quantify the accuracy of the proposed CloudLSTM in terms of Mean Absolute Error (MAE) and Root Mean Square Error (RMSE). Since the mobile traffic snapshots can be viewed as "urban images" , we also select Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) to quantify the fidelity of the forecasts and their similarity with the ground truth, as suggested by relevant recent work. Details about the metrics are discussed in Appendix F. For the mobile traffic prediction task, we employ all neural networks to forecast city-scale mobile traffic consumption over a time horizon of J = 6 sampling steps, i.e., 30 minutes, given M = 6 consecutive past measurements. For RNN-based models, i.e., LSTM, ConvLSTM, PredRNN++, CloudLSTM, CloudRNN, and CloudGRU, we then extend the number of prediction steps to J = 36, i.e., 3 hours, to evaluate their long-term performance. In the air quality forecasting use case, all models receive a half day of measurements, i.e., M = 12, as input, and forecast indicators in the following 12 hours, i.e., J = 12. As for the previous use case, the number of prediction steps are then extended to J = 72, or 3 days, for all RNN-based models. We perform 6-step forecasting for 4,888 instances across the test set, and report in Table 1 the mean and standard deviation (std) of each metric. We also investigate the effect of a different number of neighboring points (i.e., K = 3, 6, 9), as well as the influence of the attention mechanism. Observe that RNN-based architectures in general obtain superior performance, compared to CNNbased models and the MLP. In particular, our proposed CloudLSTM, and its CloudRNN, and CloudGRU variants outperform all other banchmark architectures, achieving lower MAE/RMSE and higher PSNR/SSIM on both urban scenarios. This suggests that the DConv operator learns features over geospatial point-clouds more effectively than vanilla convolution and PointCNN. Among our approaches, CloudLSTM performs better than CloudGRU, which in turn outperforms CloudRNN. Interestingly, the forecasting performance of the CloudLSTM seems fairly insensitive to the number of neighbors (K); it is therefore worth using a small K in practice, to reduce model complexity. Further, we observe that the attention mechanism improves the forecasting performance, as it helps capturing better dependencies between input sequences and vectors in decoders. This effect has also been confirmed by other NLP tasks. We provide a complete service-wise evaluation in Appendix H. Long-term Forecasting Performance. We extend the prediction horizon to up to J = 36 time steps (i.e., 3 hours) for all RNN-based architectures, and show their MAE evolution with respect to this horizon in Fig. 4. Note that the input length remains unchanged, i.e., 6 time steps. In city 1, observe that the MAE does not grow significantly with the prediction step for most models, as the curves flatten. This means that these models are reliable in terms of long-term forecasting. As for city 2, we note that low K may lead to poorer long term performance for CloudLSTM, though not significant before step 20. This provides a guideline on choosing K for different forecast lengths required. We employ all models to deliver 12-step air quality forecasting on six indicators, given 12 snapshots as input. Results are in Table 2. Also in this use case, the proposed CloudLSTMs attain the best performance across all 4 metrics, outperforming state-of-the-art methods (ConvLSTM) by up to 12.2% and 8.8% in terms of MAE and RMSE, respectively. Unlike in the mobile traffic forecasting , a lower K yields better prediction performance, though the difference appears subtle. Again, the CloudCNN always proves superior to the PointCNN, indicating that CloudCNNs are better feature extractors over point-clouds. Overall, these demonstrate the effectiveness of the CloudLSTM models for modeling spatiotemporal point-cloud stream data, regardless of the tasks to which they are applied. Performance evaluations of long-term forecasting, i.e., up to 72 future time steps, are conducted on RNN-based models and are presented in Appendix I. Note that we conduct our experiments using strict variable-controlling methodology, i.e., only changing one factor while keep the remaining the same. Therefore, it is easy to study the effect of each factor. For example, taking a look at the performance of LSTM, ConvLSTM, PredRNN++, PointLSTM and CloudLSTM, which employ dense layers, and CNN, PointCNN and D-Conv as core operators but using LSTM as the RNN structure, it is clear that the D-Conv contributes significantly to the performance improvements. Further, by comparing CloudRNN, CloudGRU and CloudLSTM, it appears that CloudRNN CloudGRU < CloudLSTM. Similarly, by comparing the CloudLSTM and Attention CloudLSTM, we see that the effects of the attention mechanism are not very significant. Therefore, we believe the core operator > RNN structure > attention, ranked by their contribution. We introduce CloudLSTM, a dedicated neural model for spatiotemporal forecasting tailored to pointcloud data streams. The CloudLSTM builds upon the DConv operator, which performs convolution over point-clouds to learn spatial features while maintaining permutation invariance. The DConv simultaneously predicts the values and coordinates of each point, thereby adapting to changing spatial correlations of the data at each time step. DConv is flexible, as it can be easily combined with various RNN models (i.e., RNN, GRU, and LSTM), Seq2seq learning, and attention mechanisms. The DConv can be efficiently implemented using a standard 2D convolution operator, by data shape transformation. We assume a batch size of 1 for simplicity. Recall that the input and output of DConv, S in and S out, are 3D tensors with shape (N, (H + L), U in ) and (N, (H + L), U out ), respectively. Note that for each p n in S i in, we find the set of top K nearest neighbors Q K n. Combining these, we transform the input into a 4D tensor S i in, with shape (N, K, (H + L), U in ). To perform DConv over S i in, we split the operator into the following steps: Algorithm 1 Efficient algorithm for DConv implementation using the 2D convolution operator 1: Inputs: The weight tensor W. in, W) with step 1 without padding. S out becomes a 3D tensor with shape (N, 1, U out × (H + L)) 6: Reshape the output map S out to (N, (H + L), U out ) 7: Apply the sigmoid function σ(·) to the coordinates feature in S out This enables to translate the DConv into a standard convolution operation, which is highly optimized by existing deep learning frameworks. We study the complexity of DConv by separating the operation into two steps: (i) finding the neighboring set Q K n for each point p n ∈ S, and (ii) performing the weighting computation in Eq. 2. We discuss the complexity of each step separately. For simplicity and without loss of generality, we assume the number of input and output channels are both 1. For step (i), the complexity of finding K nearest neighbors for one point is close to O(K · L log N), 1 if using KD trees . For step (ii), it is easy to see from Eq. 2 that the complexity of computing one feature of the output p n is O((H + L) · K). Since each point has (H + L) features and the output point set S j out has N points, the overall complexity of step 2 ). This is equivalent to the complexity of a vanilla convolution operator, where both the input and output have (H + L) channels, and the input map and kernel have N and K elements, respectively. This implies that, compared to the convolution operator whose inputs, outputs, and filters have the same size, DConv introduces extra complexity by searching the K nearest neighbors for each point O(K · L log N). Such complexity does not increase much even with higher dimensional point clouds. We show that the normalization of the coordinates features enables transformation invariance with shifting and scaling. The shifting and scaling of a point can be represented as: where A and B are a positive scaling coefficient and respectively an offset. By normalizing the coordinates, we have: This implies that, by using normalization, the model is invariant to shifting and scaling transformations. We combine our proposed CloudLSTM with the attention mechanism introduced in . We denote the j-th and i-th states of the encoder and decoder as H j en and H i de. The context tensor for state i at the encoder can be represented as: where e i,j is a score function, which can be selected among many alternatives. In this paper, we choose e i,j = v We compared our proposal against a set of baselines models. MLP , CNN , and 3D-CNN are frequently used as benchmarks in mobile traffic forecasting . DefCNN learns the shape of the convolutional filters and has similarities with the DConv operator proposed in this study . LSTM is an advanced RNN frequently employed for time series forecasting . While ConvLSTM can be viewed as a baseline model for spatiotemporal predictive learning, the PredRNN++ is the state-of-the-art architecture for spatiotemporal forecasting on grid-structural data and achieves the best performance in many applications . The CloudRNN and CloudGRU have can be formulated as: CloudGRU: The CloudRNN and CloudGRU share a similar Seq2seq architecture with CloudLSTM, except that they do not employ the attention mechanism. We show in Table 3 the detailed configuration along with the number of parameters for each model considered in this study. Note that we used 2 layers (same as our CloudLSTMs) for ConvLSTM, PredRNN++ and PointLSTM, since we found that increasing the number of layers did not improve their performance. 3 × 3 filters are commonly used in image applications, where they have been proven effective. This yields a receptive field of 9 (3 × 3), which is equivalent to K = 9 in our CloudLSTMs. Thus this supports a fair comparison. In addition, the PredRNN++ follows a slightly different structure than that of other Seq2seq models, as specified in the original paper. 2-stack Seq2seq CloudLSTM, with 36 channels and K = 3 CloudLSTM (K = 6) 2-stack Seq2seq CloudLSTM, with 36 channels and K = 6 CloudLSTM (K = 9) 2-stack Seq2seq CloudLSTM, with 36 channels and K = 9 Attention CloudLSTM 2-stack Seq2seq CloudLSTM, with 36 channels, K = 9 and soft attention mechanism We optimize all architectures using the MSE loss function: Here v h n is the mobile traffic volume forecast for the h-th service, and respectively the forecast value of the h-th air quality indicator, at antenna/monitoring station n, at time t, while v h n is the corresponding ground truth. We employ MAE, RMSE, PSNR and SSIM to evaluate the performance of our models. These are defined as: PSNR(t) = 20 log v max (t) − 10 log 1 where µ v (t) and v max (t) are the average and maximum traffic recorded for all services/quality indicators, at all antennas/monitoring stations and time instants of the test set. VAR(·) and COV(·) denote the variance and covariance, respectively. Coefficients c 1 and c 2 are employed to stabilize the fraction in the presence of weak denominators. Following standard practice, we set 2, where L = 2 is the dynamic range of float type data, and k 1 = 0.1, k 2 = 0.3. City 2 Figure 5: The anonymized locations of the antenna set in both cities. The measurement data is collected via traditional flow-level deep packet inspection at the packet gateway (P-GW). Proprietary traffic classifiers are used to associate flows to specific services. Due to data protection and confidentiality constraints, we do not disclose the name of the operator, the target metropolitan regions, or the detailed operation of the classifiers. For similar reasons, we cannot name the exact mobile services studied. We show the anonymized locations of the antennas sets in both cities in Fig. 5 As a final remark on data collection, we stress that all measurements were carried out under the supervision of the competent national privacy agency and in compliance with applicable regulations. In addition, the dataset we employ for our study only provides mobile service traffic information accumulated at the antenna level, and does not contain personal information about individual subscribers. This implies that the dataset is fully anonymized and its use for our purposes does not raise privacy concerns. Due to a confidentiality agreement with the mobile traffic data owner, the raw data cannot be made public. As already mentioned, the set of services S considered in our analysis comprises 38 different services. An overview of the fraction of the total traffic consumed by each service and each category in both cities throughout the duration of the measurement campaign is in Fig. 6. The left plot confirms the power law previously observed in the demands generated by individual mobile services. Also, streaming is the dominant type of traffic, with five services ranking among the top ten. This is confirmed in the right plot, where streaming accounts for almost half of the total traffic consumption. Web, cloud, social media, and chat services also consume large fractions of the total mobile traffic, between 8% and 17%, whereas gaming only accounts for 0.5% of the demand. The air quality dataset comprises air quality information from 43 cities in China, collected by the Urban Computing Team at Microsoft Research. In total, there are 2,891,393 air quality records from 437 air quality monitoring stations, gathered over a period of one year. The stations are partitioned into two clusters, based on their geographic locations, as shown in Fig. 7. Cluster A has 274 stations, while Cluster B includes 163. Note that missing data exists in the records and gaps have been filled through linear interpolation. The dataset is available at https://www.microsoft.com/ en-us/research/project/urban-air/. We dive deeper into the performance of the proposed Attention CloudLSTMs, by evaluating the forecasting accuracy for each individual mobile service, averaged over 36 steps. To this end, we present the MAE evaluation on a service basis (left) and category basis (right) in Fig. 8. Observe that the attention CloudLSTMs obtain similar performance over both cities at the service and category level. Jointly analyzing with Fig. 6, we see that services with higher traffic volume on average (e.g., streaming and cloud) also yield higher prediction errors. This is because their traffic evolution exhibits more frequent fluctuations, which introduces higher uncertainty, making the traffic series more difficult to predict. Figure 9: MAE evolution wrt. prediction horizon achieved by RNN-based models on both city clusters for the air quality forecasting. We show the MAE for long-term forecasting (72 steps) of air quality on both city clusters in Fig. 9. Generally, the error grows with time for all models, as expected. Turning attention to the CloudLSTM with different K, though the performance of different settings appears similar at the beginning, larger K can significantly improve the robustness of the CloudLSTM, as the MAE grows much slower with time when K = 9. This is consistent with the made in the mobile traffic forecasting task. We complete the evaluation of the mobile traffic forecasting task by visualizing the hidden features of the CloudLSTM, which provide insights into the knowledge learned by the model. In Fig. 10, we show an example of the scatter distributions of the hidden state in H t of CloudLSTM and Attention CloudLSTM at both stacks, along with the first input snapshots. The first 6 columns show the H t for encoders, while the rest are for decoders. The input data snapshots are samples selected from City 2 (260 antennas/points). Recall that each H t has 1 value features and 2 coordinate features for each point, therefore each scatter subplot in Fig. 10 Step 1 Attention Cloud-LSTM Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Step 12 Figure 11: NO 2 forecasting examples in City Cluster A generated by all RNN-based models. stack 2, as features are extracted at a higher level, exhibiting more direct spatial correlations with respect to the output. Lastly, in Fig. 11 and 12 and we show a set of NO 2 forecasting examples in both city cluster A and B considered for air quality prediction, generated by all RNN-based models, offering a performance comparison from a purely visual perspective. Point-clouds are converted into heat maps using 2D linear interpolation. The better prediction offered by (Attention) CloudLSTMs is apparent, as our proposed architectures capture trends in the point-cloud streams and deliver high long-term visual fidelity, whereas the performance of other architectures degrade rapidly in time. Step 1 Attention Cloud-LSTM Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Step 12 Figure 12: NO 2 forecasting examples in Cluster B generated by the RNN-based models. The DConv uses Sigmoid functions to regularize the coordinate features of each point, such that those points which are far from others will move closer to each other and be more involved in the computation. Further, by stacking multiple DConv via dedicated structure (LSTM), the CloudLSTM has much stronger representability and therefore allows to refine the positions of each input point at each time step and each stack. Eventually, each point can learn to move to the position where it is best to be and therefore, our model continues to work well while forecasting with outlier points. For demonstration, we use the density-based spatial clustering of applications with noise (DBSCAN) to find those outliers (red points) in both clusters in the air quality dataset, as shown in Fig. 13. For each city cluster, the DBSCAN algorithm finds 16 outlier points, which are relatively isolated and far from the point-cloud center. We recompute the MAE and RMSE performance especially for these outlier points, as shown in Table 4. Observe that our CloudLSTM still obtains the lowest prediction error as compared to the other models considered. Taking a closer look at the CNN-based models, the CloudCNN, which employs the DConv operator, obtains the best forecasting performance relative to CNN, 3D-CNN, DefCNN and PointCNN. We dive deeper into the robustness of our CloudLSTM to outliers by conducting experiments under more controlled scenarios. To this end, we randomly selected 50 weather stations in each city cluster and construct a toy dataset. Among these weather stations, we randomly pick 10 as outliers, and move their positions away from the center by d = {0, 0.5, 1, 5} on both x and y axes. The direction of movement depends on the quadrant of each outlier. Note that the original position of each weather station is normalized to, so d = 5 means the point is moved at a distance 5 times the maximum range of its original position. The positions of the remaining 40 weather stations remain unchanged. We show the positions of each weather stations after moving by different d for both city clusters in Fig. 14 and 15. We retrain the CloudLSTM and PointLSTM under the same settings, and show the MAE and RMSE performance of each in Table. 5. Observe that the proposed CloudLSTM performs almost equally well when forecasting over inliers and outliers, irrespective of the distance to outliers. Importantly, CloudLSTM achieves significantly better performance over its counterpart PointLSTM. This further demonstrates that our model is robust to outliers, whose locations appear "lone". We further compare our proposal with simple baselines, which also perform forecasting based on knearest neighbors of each target point. To this end, we construct MLPs and LSTMs with the structures specified in Table 3, but with different input form. Specifically, for each point, the models perform prediction using only the K nearest neighbors' data, with K from {1, 3, 6, 9, 25, 50, 100}. We show their performance along with that of our CloudLSTM on the air quality dataset in Table 6. Observe that our CloudLSTM significantly outperforms MLPs and LSTMs, which conduct forecasting only relying on k-nearest neighbors. The number of neighbors K affects the receptive field of each model. A small K means the model only relies on limited local spatial dependencies, while global spatial correlations between points are neglected. In contrast, a large K enables looking around larger location spaces, while this might lead to overfitting. The in the table suggest that the K does not affect the performance of each baseline significantly. Meanwhile our proposed CloudLSTM, which extracts local spatial dependencies through DConv kernels and merges global spatial dependency via stacks of time steps and layers, is superior to these simple baselines. Finally, we notice that seasonal information exists in the mobile traffic series, which can be further exploited to improve the forecasting performance. However, directly feeding the model with data spanning multiple days is infeasible, since, e.g., a 7-day window corresponds to a 2016-long sequence as input (given that data is sampled every 5 minutes) and it is very difficult for RNN-based models to handle such long sequences. In addition, by considering the number of mobile services and antennas, the input for 7 days would have 60,673,536 data points. This would make any forecasting model extremely large and therefore impractical for real deployment. To capture seasonal information more efficiently, we concatenate the 30 minute-long sequences (sampled every 5 minutes) with a sub-sampled 7-day window (sampled every 2h). This forms an input with length 90 (6 + 84). We conduct experiments on a randomly selected subset (100 antennas) of the mobile traffic dataset (City 1), and show the forecasting performance without and with seasonal information (7-day window) in Table 7. By incorporating the seasonal information, the performance of most forecasting models is boosted. This indicates that the periodic information is learnt by the model, which helps reduce the prediction errors. However, the concatenation increases the length of the input, which also increases the model complexity. Future work will focus on a more efficient way to fuse the seasonal information, with marginal increase in complexity.

Represent the short summary for finding the corresponding scientific article (long)

This paper introduces CloudLSTM, a new branch of recurrent neural models tailored to forecasting over data streams generated by geospatial point-cloud sources.

Represent the TLDR for finding the corresponding scientific article

A control variate based stochastic training algorithm for graph convolutional networks that the receptive field can be only two neighbors per node.