Question: represent the article for finding a single-sentence tldr

Question: High intra-class diversity and inter-class similarity is a characteristic of remote sensing scene image data sets currently posing significant difficulty for deep learning algorithms on classification tasks. To improve accuracy, post-classification methods have been proposed for smoothing of model predictions. However, those approaches require an additional neural network to perform the smoothing operation, which adds overhead to the task. We propose an approach that involves learning deep features directly over neighboring scene images without requiring use of a cleanup model. Our approach utilizes a siamese network to improve the discriminative power of convolutional neural networks on a pair of neighboring scene images. It then exploits semantic coherence between this pair to enrich the feature vector of the image for which we want to predict a label. Empirical show that this approach provides a viable alternative to existing methods. For example, our model improved prediction accuracy by 1 percentage point and dropped the mean squared error value by 0.02 over the baseline, on a disease density estimation task. These performance gains are comparable with from existing post-classification methods, moreover without implementation overheads. Remote sensing scene image analysis is emerging as an important area of research for application of deep learning algorithms. Application areas include land-use land-cover analysis, urban planning, and natural disaster detection. A deep learning task for labeling a scene image is typically formulated as conditional probability of the form in Eq. 1 , , , , , , where l i is label for image patch s i. This formulation is sufficient for problems where spatial situatedness of a scene, which embodies knowledge of semantic likeness between neighborhoods in the geophysical world, is not important. However, for problems which require knowledge of neighborhood the formulation in Eq. 1 becomes inadequate. An example of such a problem would be estimating disease density for a small geographical region of interest, in which case the probability of label l is likely to depend on the labels for neighboring regions due to semantic coherence among them. The problem of how to improve model prediction by leveraging semantic coherence among neighboring scene images has previously been considered in the literature. Previous studies consider the problem as a post-classification task. For example, used a second classifier to do pixel smoothing to refine predictions made by another classifier. Based on a 5x5 window, a filter assigns pixels to the majority class if it had been assigned a different class. , a post-processing architecture is suggested for incorporating structure into image patch prediction. It involves stacking neural networks (NN) such that the output from a previous one becomes input for the next. Idea is for each network to clean up predictions of previous one in order to progressively improve overall accuracy. While improved model performance was achieved by these methods, they have overhead of performing same classification task in at least two stages. In other words, you need a minimum of two NN to perform the same classification task. Unlike post-classification methods, this work considers the problem of improving model accuracy on scene images by exploiting knowledge of neighboring scenes as part of the model training process. We make the assumption that l is conditionally co-dependent on information embedded in scene image i and in other similar, neighboring image j such that the problem is formulated as probability of the form in Eq. 2, where s j is image for a neighboring tile that is most similar to index tile i and P (l i |S i, S j) is observed probability distribution. We used Convolutional Neural Networks (CNN) for modeling the observed probability distribution in Eq. 2. A network architecture is proposed for training our model consisting of four components: a siamese sub-network, a similarity metric learning component, a convolutional network, and a decision layer. The siamese sub-network takes two neighboring scene images as input and extracts features from each. The similarity learning component evaluates how similar the input images are, using the extracted features. If the two input images are found to be similar the convolutional network learns additional features based on the merged feature vector, otherwise those from the index tile are used alone. We implemented the decision layer to perform classification or regression. A baseline model was implemented that takes a single image, the index tile i, as input. Empirical show the proposed model consistently outperforms the baseline. In addition to improving predictive performance with a relatively small training set, our model is fast to train since it uses a pre-trained model for the siamese sub-network. Furthermore, it does not require another NN to smooth out its predictions as is the case with post-classification approaches, while achieving comparable performance gain. In summary,our contributions include the following. 1. We propose an approach for training a probabilistic deep learning model to improve prediction accuracy by exploiting semantic coherence between neighboring tiles in aerial scene images. A CNN architecture is suggested for this purpose. 2. We provide empirical evidence that demonstrates the viability of this approach on a disease density estimation task. 3. Lastly, we discovered an important limitation of the synthetic minority over-sampling technique (SMOTE). This method fails when used for oversampling an under-represented class whereby knowledge of spatial proximity between scene image data points must be preserved, an important requirement under the framework of learning deep features over neighboring scene images introduced in this work. Remote sensing scene images represent a difficult problem for deep learning algorithms due to high intra-class diversity and inter-class similarity . Previous efforts to address this problem is presented first, followed by recent use of siamese networks to improve discriminative ability of CNN. One of the earliest works aimed at improving model accuracy by exploiting semantic coherence between neighboring images is by. The problem addressed there is'salt and pepper' noise in classified satellite images. The authors employ a multi-layer perceptron (MLP) consisting of five hidden units to classify low resolution (30m) satellite scene images. Their proposed network has four output units corresponding to four land-use classes. The sigmoid activation function was used with both hidden and output units. The network had seven input units, each corresponding to one of seven Landsat TM spectral bands. It generates two separate channels when classifying an input image: one for class prediction and the other representing how confident the classification is. A post-classification smoothing operation is then performed using a two-layer NN whose input is the classified image within a 5 x 5 window. For each pixel, the predicted class and confidence information are used as input to decide its new class belonging by applying a majority filter. Gain in classification accuracy was 2.9 percent for the network whose input data was enhanced with texture information (and 4.9 percent for the one without enhancement). introduce indirect dependencies between outputs of a model by using knowledge of structure (e.g. shape) to resolve the issue of disconnected blobs and holes or gaps in predicted building and road network maps. The architecture used consists of stacking NN, each using as input the outputs of a previous network. Precisely, letM 0 be the map predicted by a model. The ith level cleanup NN (f i) takes a w s x w s patch ofM i−1 and outputs a w m x w m patch ofM i. The model f i is trained by minimizing the negative log likelihood on patches of the observed mapM, just like other NN in the architecture. Thus, the ith cleanup NN improves predictions of the previous NN f i−1. The authors extended this idea to Conditional Random Fields (CRF) i.e. pairwise lattice CRF to introduce explicit dependencies between pairs of neighboring pixels. For both NN and CRF, an average gain of 0.0195 was achieved in precision and recall break-even points. The above works demonstrate that post-classification smoothing operations can improve performance of deep learning models. However, a draw back of those approaches is the necessary requirement to implement additional models to perform smoothing operations. Therefore, approaches that sidestep this requirement would be a welcome alternative. Another innovation that inspired the current work is siamese neural networks (SNN) introduced by and. A SNN architecture consists of two identical (shared weight) sub-networks merged by the same function at their outputs. During training each sub-network extracts features from one of two concurrent inputs, after which the joining function evaluates whether the input pair is similar based on a distance metric, such as cosine of angle between the pair of feature vectors. Output from the SNN is a decision score e.g., 1 for similar image pair, 0 otherwise. SNN have recently been used for remote sensing scene image understanding. used a siamese architecture to improve discriminative ability of CNN for satellite scene image classification. The architecture consists of two identical CNN models, 3 additional convolutional layers, and one square non-parametric layer. Each branch of the siamese network is responsible for extracting features from each dual input image, giving feature vectors f 1 and f 2. Two of the convolutional layers take these feature vectors as input and perform additional learning before a softmax function predicts the label for each input image. On the other hand, the square layer takes f 1 and f 2 as input and outputs the tensor f s = (f 1 − f 2) 2 which is used to measure similarity between the input pair. The model thus produces three outputs: predicted label for each input image and a score for their similarity. More precisely, feature discrimination enhancement is achieved in the square layer by imposing a metric learning regularization term that minimizes Euclidean distance between a similar image pair and maximizes that for a dissimilar pair (Eq. 3), where D(x i, x j) is distance between a pair of input images. A margin τ is set to separate as far apart as possible, the similar image pairs from dissimilar ones in feature space. If (x i, x j) is from the same scene class the distance between them is less than τ, otherwise it is greater than τ. Let label for training pair (x i, x j) be (y i, y j), respectively. The formulation for Euclidean distance between them is given by (Eq. 4), This model only minimizes feature distance between similar input pairs through the margin τ to achieve feature discrimination. Therefore, the model in optimizes for two objective functions: a distance loss function and a regularization term as shown in Eq. 5, Of the three pre-trained models experimented with (AlexNet, VGG-16, and ResNet-50) highest feature discrimination accuracy gain (1.14 percentage point) was achieved with a Siamese AlexNet model on the challenging NWPU-RESIC45 benchmark data set. reported a 1.53 percentage point gain in accuracy on the same data set. An end-to-end pipeline is proposed consisting of four components: a siamese sub-network, similarity metric learning function, convolutional network, and decision layer. The siamese sub-network takes a pair of scene images s i (tile we want to predict a label for) and s j (other tile neighboring s i) as input and extracts features from each, outputting feature vectors f i and f j, respectively. The similarity function takes as input the two feature vectors and learns a similarity metric between them. If the feature vector pair (f i, f j) is found to be similar they are merged and the ing feature vector used to train the convolutional network, otherwise training is done using (f i) alone. Output from the convolutional network is passed as input to the decision layer to predict a label (or to regress a numerical value) for image patch s i. The proposed and baseline architectures are shown in Figures 1a and 1b, respectively. To implement the siamese sub-network we explored two state-of-the-art pre-trained CNN models i.e. Xception net and ResNet-50 in a transfer learning (fine-tuning) strategy since we had a small training set. The architecture of Xception model is based on separating learning of space-wise and channel-wise feature representations. The is better performance over architectures such as Inception V3, ResNet-152, and VGG-16 in terms of computational speed and classification accuracy. On the other hand, ResNet architectures address the problem of vanishing/exploding gradients in deep neural network architectures by inserting shortcut connections between convolutional blocks. The ResNet model has benefit of enabling deeper architectures to be utilized in CNN models since these provide higher accuracy rates than shallow architectures. The similarity function takes as input feature vector (f i, f j) to evaluate how similar the input image pair is. This unit outputs one of a binary decision score i.e neighboring input pair (s i, s j) is similar or dissimilar. The optimization function used here works by minimizing feature distance between similar images. Following work in , given a pair of input image s i and s j the distance between them is calculated using Eq. 3. A margin τ separates similar image features from dissimilar ones such that D(f i, f j) < τ for a similar pair, D(f i, f j) > τ otherwise. We also optimized for two objective functions as defined in Eq. 5. An important question to consider is how to identify a neighboring image that is semantically closest in likeness to index tile i, over which to co-learn features. For any image patch in the middle region of a w x h scene image S where w≥3 and h≥3, there will be eight such tiles to choose from, considering four adjacent and diagonal neighbors. For edge tiles there will be five while for corner tiles there will be three neighbors, Figure 2 in Appendix. We can determine neighbors of the tile at location i by specifying the maximum Harversine distance within which the centroid coordinates of all neighbors will lie. Take as an example tiles of size 250m 2 in a 3 x 3 scene image. The centroid of all eight neighbors of the center tile would fall within a maximum distance of 350m from its own centroid. We can evaluate how similar tile at i is to its neighbor at location j using Euclidean distance. If at least one is similar, we select the neighbor with least distance to be most similar. Given that D(s i, s j) < τ for images from same scene class, a neighboring image is most similar to s i if it has the smallest distance Eq. 6, whereŝ i is neighboring image most similar to index tile at location i. In the current analysis however, we consider only the preceding neighbor to tile at location i (image s i−1) for similarity analysis. The loss function used is binary cross-entropy, defined in Eq. 7. This network is trained using features ing from merging feature vector pair (f i, f j) if respective input image pair (s i, s j) is found to be similar and it outputs feature vector f m (f i, f j). If the input pair is dissimilar, only the feature vector f i is fed to the convolutional network, which outputs f i (f i). Our implementation used three dense layers in the convolutional network. Merge operation used is feature averaging. For a classification task we learn the parameters (weights) of the CNN by minimizing the negative log likelihood of the training data under our model. For the multi-class problem considered in this work the negative log likelihood under the model in Eq. 2 assumes the form of a cross-entropy between the probability distribution for observed labels l and the predicted label probabilitiesŷ (Eq. 8), where M, the number of classes, is 3. The outer sum of objective function L is over all training samples. Stochastic gradient descent with mini-batches is used for optimizing L. Our implementation for the current problem uses a fully connected network (FCN) with a 3-way softmax classifier. Taken as a regression task, we optimized for the mean absolute error (MAE) loss and mean squared error (MSE). The form of MAE we used is given by Eq. 9, where M, the number of features to be predicted, is 2. For MSE the loss function is given by Eq. 10, We applied our method to the task of estimating disease density from satellite scene imagery. Below we describe data sets and methods used. A number of data sets were used in our experiments. The epidemic data consists of monthly disease case counts aggregated by sub-county for year 2015. We dis-aggregated the data to 250m 2 grids, our geospatial unit of analysis, using population data from. The latter data gives estimate of people living in a 30m 2 grid based on recent census data and high resolution (0.5m 2) satellite imagery. The data was up-sampled to 250m 2 grid to be consistent with our spatial unit of analysis. The population data was used to create a weighting scheme for dis-aggregating epidemic data. The building (housing) concentration data consists of satellite imagery extracted from Google Static Maps API using method in. It consists of land-use type buildings i.e. objects of interest in a satellite scene image is buildings or houses. The housing data was used as input to train the model, as proxy for indoor overcrowding -a known risk factor for our case study disease. All input data was pre-processed into feature vectors and normalized to value range by min-max scaling or (-1,1) by standardizing (zero mean, unit variance). We created disease density classes out of the epidemic data to make it a classification task. We did this by binning the normalized data such that each 250m 2 grid assumed a class value depending on which bin its disease density lies in. Following the procedure in for binning population data, we created a matrix C where an entry C i = 0 if 5.74e − 05 ≤ d i ≤ 1.52e − 4, 1 if 1.52e − 4 < d i ≤ 2.46e − 4, and 2 if 2.46e − 4 < d i ≤ 3.4e − 4, where d i is normalized disease density. The classes correspond to the semantic labels 0, 1, 2 for high, low, and moderate disease density with 3,628, 6,472, and 1,970 membership count, respectively. Let D be a grid of disease case counts for a region of interest at time t, C grid of disease density class label values, and S grid of housing concentration data. For every disease case count value d i and density class label value c i there is an associated housing concentration image s i. We formulated the learning task as estimating unknown function f based on the idea of learning deep features over neighboring scene images (one neighbor in this case) using Eq. 11 (Figure 1a), where c i is estimated disease density for a 250m 2 patch of land on earth's surface (approx. 224m 2 pixels for an image), represented by image s i. s j is image of a neighboring scene that is most similar to image of tile at location i, identified using method in Eq. 6. We used CNN to estimate function f because the mapping from input data to disease incidence estimate is non-linear, noisy, and dependent on semantic content of input data. To train our model we used transfer learning by fine-tuning two pre-trained models on our data set i.e., Xception net and ResNet-50. Transfer learning eliminates network architecture design time while often achieving higher accuracy rates than training a model from scratch. We applied the same regularization methods and other hyperparameter values (Table 4 in Appendix) to ensure uniformity across the proposed and baseline network architectures. Once a satisfactory model was achieved, it was trained on a combined data set (12,070 images) of training and validation before evaluating on the test set. Evaluation for similarity metric learning for our proposed model built on top of ResNet-50 and Xception net are shown in Table 1. ResNet-50 performed better than Xception net on precision (0.39 vs. 0.70) and recall (0.10 vs. 0.60) on the task of detecting whether or not a pair of satellite scene images is similar, even though the overall performance is low for both pre-trained models (maximum OA score of 83 percent got with Xception net). Performance for the baseline and proposed model are shown in Table 2. Overall, the Xception based model performed better than the ResNet-50 based one on the task of estimating disease density from housing satellite scene data when performance is measured using precision, recall, and F1-Score metrics. However, both models generally performed poorly, achieving a maximum overall accuracy score of 44 percent for ResNet-50. Our proposed model however, consistently out performed the baseline across both pre-trained models (scores marked with bold). For example, using Xception net as the Siamese network our model achieved precision score of 0.15 vs. 0.13, recall of 0.39 vs. 0.26, and f1-score of 0.22 vs. 0.18 for the baseline model, respectively. A similar performance trend is observed for the model based on ResNet-50. Confusion matrix plots for baseline and our model built using Xception net are shown in Figure 3a and 3b, respectively (Appendix). On the other hand Figure 4 shows AUROC plots. Both show that our model performed better than the baseline on medium and low disease density classes with 58 vs. 48 percent and 47 vs. 37 percent, respectively. However, the model performed worse than the baseline on high disease density class (0 vs. 13 percent, respectively). Generally, the from both models are as good as chance. Table 3 gives MAE and MSE scores for baseline and proposed model. Overall, the model built using ResNet-50 performed better than the one built with Xception net when used to estimate disease density from housing concentration satellite scene image data. Again, our proposed model performed consistently (even though marginally) better than the baseline (scores marked with bold). It achieved MAE score of 0.35 vs. 0.38 and MSE score of 0.18 vs. 0.20 for the baseline, respectively. A similar trend is observed for the model built with Xception net as base. Our model performed better than the baseline in both the classification and regression tasks for disease density estimation. For example, our model achieved 1 percentage point gain in accuracy over the baseline model. While a gain as of deploying a siamese network to boost discriminative power of CNN for aerial scene image classification is consistent with findings in previous studies overall from our model are poor. For instance, our model was only able to attain a maximum overall accuracy of 34 percent on the classification task. We would like to pin these poor to a combination of three factors. First, the small data set used to train our model (12,070 images) could have impacted accuracy negatively, despite the use of regularization methods. It is therefore, possible that our model suffered the problem of overfitting. Secondly, our data set was unbalanced. It is likely that the extra parameter we introduced in the loss function to weight classes by giving higher importance to under-represented classes did not work as expected. The could have been that our model did not learn all the necessary features required to make a prediction but rather could have resorted to guessing the output and hence, failing to generalize well over the test set. Class imbalance in our data set could also have negatively affected feature correlation which in turn could have reduced model performance. Besides, well-known methods for mitigating sample size bias in imbalanced data sets, for example by over-sampling under-represented classes could not be applied directly to our data set without modifying the algorithm. That is because it was not immediately clear how to preserve spatial proximity between neighboring tiles, an idea that is central to learning deep features over neighboring scene images. However, despite the low overall performance by our model we have been able to demonstrate that it is possible to improve model accuracy by learning deep features over neighboring scene images in a disease density estimation task. Figure 2: Finding a neighboring image j that is semantically most similar to image i.

represent this text to find a much much longer article about it with say hundreds-thousands of words!

Approach for improving prediction accuracy by learning deep features over neighboring scene images in satellite scene image analysis.


Question: represent the article for finding a single-sentence tldr

Question: Sparsely available data points cause a numerical error on finite differences which hinder to modeling the dynamics of physical systems. The discretization error becomes even larger when the sparse data are irregularly distributed so that the data defined on an unstructured grid, making it hard to build deep learning models to handle physics-governing observations on the unstructured grid. In this paper, we propose a novel architecture named Physics-aware Difference Graph Networks (PA-DGN) that exploits neighboring information to learn finite differences inspired by physics equations. PA-DGN further leverages data-driven end-to-end learning to discover underlying dynamical relations between the spatial and temporal differences in given observations. We demonstrate the superiority of PA-DGN in the approximation of directional derivatives and the prediction of graph signals on the synthetic data and the real-world climate observations from weather stations. Modeling real world phenomena, such as climate observations, traffic flow, physics and chemistry simulation (; ; ; de ; ;), is important but extremely challenging. While deep learning has achieved remarkable successes in prediction tasks by learning latent representations from data-rich applications such as image recognition , text understanding , and speech recognition, we are confronted with many challenging scenarios in modeling natural phenomena by deep neural networks when a limited number of observations are only available. Particularly, the sparsely available data points cause substantial numerical error and the limitation requires a more principled way to redesign deep learning models. Although many works have been proposed to model physics-simulated observations using deep learning, many of them are designed under the assumption that input is on a continuous domain. For example, Raissi et al. (2017a; proposed Physics-informed neural networks (PINNs) to learn nonlinear relations between input (spatial-and temporal-coordinates (x, t)) and output simulated by a given PDE. Since Raissi et al. (2017a; use the coordinates as input and compute derivatives based on the coordinates to represent a given PDE, the setting is only valid when the data are continuously observed over spatial and temporal space. Under the similar direction of proposed a method to leverage the nonlinear diffusion process for image restoration. de incorporated the transport physics (advection-diffusion equation) with deep neural networks for forecasting sea surface temperature by extracting the motion field. introduced Deep Lagrangian Networks specialized to learn Lagrangian mechanics with learnable parameters. proposed a physicsinformed regularizer to impose data-specific physics equations. In common, the methods in; de; are not efficiently applicable to sparsely discretized input as only a small number of data points are available and continuous properties on given space are not easily recovered. It is inappropriate to directly use continuous differential operators to provide local behaviors because it is hard to approximate the continuous derivatives precisely with the sparse points (; ;). Furthermore, they are only applicable when the specific physics equations are explicitly given and still hard to be generalized to incorporate other types of equations. As another direction to modeling physics-simulated data, proposed PDE-Net which uncovers the underlying hidden PDEs and predicts the dynamics of complex systems. derived new CNNs: parabolic and hyperbolic CNNs based on ResNet architecture motivated by PDE theory.; are flexible to uncover hidden physics from the constrained kernels, it is still restrictive to a regular grid where the proposed constraints on the learnable filters are easily defined. Reasoning physical dynamics of discrete objects has been actively studied; ) as the appearance of graph-based neural networks (; ;). Although these models can handle sparsely located data points without explicitly given physics equations, they are purely data-driven so that the physics-inspired inductive bias, exploiting finite differences, is not considered at all. In contrast, our method consists of physics-aware modules allowing efficiently leveraging the inductive bias to learn spatiotemporal data from the physics system. In this paper, we propose Physics-aware Difference Graph Networks (PA-DGN) whose architecture is motivated to leverage differences of sparsely available data from the physical systems. The differences are particularly important since most of the physics-related dynamic equations (e.g., Navier-Stokes equations) handle differences of physical quantities in spatial and temporal space instead of using the quantities directly. Inspired by the property, we first propose Spatial Difference Layer (SDL) to efficiently learn the local representations by aggregating neighboring information in the sparse data points. The layer is based on Graph Networks (GN) as it easily leverages structural features to learn the localized representations and the parameters for computing the localized features are shared. Then, the layer is connected with Recurrent Graph Networks (RGN) to be combined with temporal difference which is another core component of physics-related dynamic equations. PA-DGN is applicable to various tasks and we provide two representative tasks; the approximation of directional derivatives and the prediction of graph signals. • We tackle a limitation of the sparsely discretized data which cause numerical error to model the physical system by proposing Spatial Difference Layer (SDL) for efficiently exploiting neighboring information under the limitation of sparsely observable points. • We combine SDL with Recurrent Graph Networks to build PA-DGN which automatically learns the underlying spatiotemporal dynamics in graph signals. • We verify that PA-DGN is effective in approximating directional derivatives and predicting graph signals in synthetic data. Then, we conduct exhaustive experiments to predict climate observations from land-based weather stations and demonstrate that PA-DGN outperforms other baselines. In this section, we introduce the building module used to learn spatial differences of graph signals and describe how the module is used to predict signals in the physics system. As approximations of derivatives in continuous domain, difference operators have been used as a core role to compute numerical solutions of (continuous) differential equations. Since it is hard to derive closed-form expressions of derivatives in real-world data, the difference operators have been considered as alternative tools to describe and solve PDEs in practice. The operators are especially important for physics-related data (e.g., meteorological observations) because the governing rules behind the observations are mostly differential equations. Graph signal Given a graph G = (V, E) where V is a set of vertices V = {1, . . ., N v} and E a set of edges E ⊆ {(i, j)|i, j ∈ V} (|E| = N e), graph signals on all nodes at time t are f (t) ∈ R Nv where f: V → R. In addition, graph signals on edges can be defined similarly, F (t) ∈ R Ne where F: E → R. Note that both signals can be multidimensional. Gradient on graph The gradient (∇) of a function on nodes of a graph is represented by finite difference where L 2 (V) and L 2 (E) denote vector spaces of node/edge functions, respectively. The gradients on a graph provide finite differences of graph signals and they become corresponding edge (i, j) features. Laplace-Beltrami operator Laplace-Beltrami operator (or Laplacian, ∆) in graph domain is defined as This operator is usually regarded as a matrix form in other literature, L = D − A where A is an adjacency matrix and D = diag(j:j =i A ij) is a degree matrix. According to , the gradient and Laplacian operator on the triangulated mesh can be discretized by incorporating the coordinates of nodes. To obtain the gradient operator, the per-face gradient of each triangular face is calculated first. Then, the gradient on each node is the area-weighted average of all its neighboring faces, and the gradient on edge (i, j) is defined as the dot product between the per-node gradient value and the direction vector e ij. The Laplacian operator can be discretized with Finite Element Method (FEM): where node j belongs to node i's immediate neighbors (j ∈ N i) and (α j, β j) are two opposing angles of the edge (i, j). While the difference operators are generalized in Riemannian manifolds , there are numerical error compared to those in continuous space and it can be worse when the nodes are spatially far from neighboring nodes because the connected nodes (j ∈ N i) of i's node fail to represent local features around the i-th node. Furthermore, the error is even larger if available data points are sparsely distributed (e.g., sensor-based observations). In other words, since the difference operators are highly limited to immediate neighboring information only, they are unlikely to discover meaningful spatial variations behind the sparse observations. To mitigate the limitation, we propose Spatial Difference Layer (SDL) which consists of a set of parameters to define learnable difference operators as a form of gradient and Laplacian to fully utilize neighboring information: where w ij are the parameters tuning the difference operators along with the corresponding edge direction e ij. Note that the two forms (Eq 1) are associated with edge and node features, respectively. The subscript in ∇ w and ∆ w denotes that the difference operators are functions of the learnable parameters w. w (g) ij and w (l) ij are obtained by integrating local information as follow: While the standard difference operators consider two connected nodes only (i and j) for each edge (i, j), Eq 2 uses a larger view (h-hop) to represent the differences between i and j nodes. Since Graph Networks (GN) are efficient networks to aggregate neighboring information, we use GN for g(·) function and w ij are edge features from the output of GN. Note that Eq 2 can be viewed as a higher-order difference equation because nodes/edges which are multi-hop apart are considered. w ij has a similar role of parameters in convolution kernels of CNNs. For example, while the standard gradient operator can be regarded as an example of simple edge-detecting filters, the operator can be a sharpening filter if w (g1) ij = 1 and w (g2) ij = |Ni|+1 |Ni| for i node and the operators over each edge are summed. In other words, by modulating w ij, it is readily extended to conventional kernels including edge detection or sharpening filters and even further complicated kernels. On top of w ij, the difference forms in Eq 1 make an optimizing process for learnable parameters based on the differences instead of values themselves intentionally. Thus, Eq 1 naturally provides the physics-inspired inductive bias which is particularly effective for modeling physics-related observations. Furthermore, it is possible to increase the number of channels for w (g) ij and w (l) ij to be more expressive. Figure 1 illustrates how the exemplary filters convolve the given graph signals. Difference graph Once the modulated spatial differences (∇ w f (t), ∆ w f (t)) are obtained, they will be concatenated with the current signals f (t) to construct node-wise (z i) and edge-wise (z ij) features and the graph is called a difference graph. Note that the difference graph includes all information to describe spatial variations. Recurrent graph networks Given a snapshot (f (t), F (t)) of a sequence of graph signals, one difference graph is obtained and it is used to predict next graph signals. While a non-linear layer can be used to combine the learned spatial differences to predict the next signals, it is limited to discover spatial relations only among the features in the difference graph. Since many equations describing physics-related phenomena are non-static (e.g., Navier-Stokes equations), we adopt Recurrent Graph Networks (RGN) with a graph state G h as input to combine the spatial differences with temporal variations. RGN returns a graph state (G * h = (h * (v), h * (e) )) and next graph signal z * i and z * ij. The update rule is described as follow: ) for all i ∈ V. z i is an aggregated edge attribute related to the node i. where φ e, φ v are edge and node update functions, respectively, and they can be a recurrent unit (e.g., GRU cell). Finally, the prediction is made through a decoder by feeding the graph signal, z * i and z * ij. Figure 4: Gradients and graph structure of sampled points. Left: the synthetic function is f 1 (x, y) = 0.1x 2 + 0.5y 2. Right: the synthetic function is f 2 (x, y) = sin(x) + cos(y). Learning objective Letf andF denote predictions of the target node/edge signals. PA-DGN is trained by minimizing the following objective: For multistep predictions, L is summed as many as the number of predicting steps. If only one type (node or edge) of signal is given, the corresponding term in Eq 3 is used to optimize the parameters in SDL and RGN simultaneously. To investigate if the proposed spatial difference forms (Eq 1) can be beneficial to learning physicsrelated patterns, we use SDL to two different tasks: approximate directional derivatives and predict synthetic graph signals. Figure 3: Directional derivative on graph As we claimed in Section 2.3, the standard difference forms (gradient and Laplacian) on a graph can become easily inaccurate because they are susceptible to a distance of two points and variations of a given function. To evaluate the applicability of the proposed SDL, we train SDL to approximate directional derivatives on a graph. First, we define a synthetic function and its gradients on 2D space and sample 200 points (x i, y i). Then, we construct a graph on the sampled points by using k-NN algorithm (k = 4). With the known gradient ∇f = (∂f ∂x, ∂f ∂y) at each point (a node in the graph), we can compute directional derivatives by projecting ∇f to a connected edge e ij (See Figure 3). We compare against four baselines: the finite gradient (FinGrad) Multilayer Perceptron Layer (MLP) Graph Networks (GN) a different form of Eq 1 (One-w). For the finite gradient ((f j − f i)/||x j − x i ||), there is no learnable parameter and it only uses two points. For MLP, we feed (f i, f j, x i, x j) as input to see whether learnable parameters can benefit the approximation or not. For GN, we use distances of two connected points as edge features and function values on the points as node features. The edge feature output of GN is used as a prediction for the directional derivative on the edge. Finally, we modify the proposed form as (∇ w f) ij = w ij * f j − f i. GN and the modified form are used to verify the effectiveness of Eq 1. Note that we define two synthetic functions (Figure 4) which have different property; monotonically increasing from a center and periodically varying. Approximation accuracy As shown in Table 1, the proposed spatial difference layer outperforms others by a large margin. As expected, FinGrad provides the largest error since it only considers two points without learnable parameters. It is found that the learnable parameters can significantly benefit to approximate the directional derivatives even if the input is the same (FinGrad vs. MLP). Note that utilizing neighboring information is generally helpful to learn spatial variations properly. However, simply training parameters in GN is not sufficient and explicitly defining difference, which is important to understand spatial variations, provides more robust inductive bias. One important thing we found is that One-w is not effective as much as GN and it can be even worse than FinGrad. It is because of its limited degree of freedom. As implied in the form (∇ w f) ij = w ij * f j − f i, only one w ij adjusts the relative difference between f i and f j, and this is not enough to learn whole possible linear combinations of f i and f j. The unstable performance supports that the form of SDL is not ad-hoc but more effectively designed. We evaluate PA-DGN on the synthetic data sampled from the simulation of specific convectiondiffusion equations, to provide if the proposed model can predict next signals of the simulated dynamics from observations on discrete nodes only. For the simulated dynamics, we use an equation similar to the one in. where the index i is for pointing the i-th node whose coordinate is Then, we uniformly sample 250 points in the above 2D space. The task is to predict signal values of all points in the future M steps given observed values of first N steps. For our experiments, we choose N = 5 and M = 15. Since there is no a priori graph structure on sampled points, we construct a graph with k-NN algorithm (k = 4) using the Euclidean distance. Figure 5 shows the dynamics and the graph structure of sampled points. To evaluate the effect of the proposed SDL on the above prediction task, we cascade SDL and a linear regression model as our prediction model since the dynamics follows a linear partial differential equation. We compare its performance with four baselines: Vector Auto-Regressor (VAR); Multi-Layer Perceptron (MLP); StandardOP: the standard approximation of differential operators in Section 2.1 followed by a linear regressor; MeshOP: similar to StandardOP but use the discretization on triangulated mesh in Section 2.2 for differential operators. Prediction Performance Table 2 shows the prediction performance of different models measured with mean absolute error. The prediction model with our proposed spatial differential layer outperforms other baselines. All models incorporating any form of spatial differential operators (StandardOP, MeshOP and SDL) outperform those without spatial differential operators (VAR and MLP), showing that introducing spatial differences information inspired by the intrinsic dynamics helps prediction. However, in cases where points with observable signal are sparse in the space, spatial differential operators approximated with fixed rules can be inaccurate and sub-optimal for prediction since the locally linear assumption which they are based on no longer holds. Our proposed spatial differential layer, to the contrary, is capable of bridging the gap between approximated difference operators and accurate ones by introducing learnable coefficients utilizing neighboring information, and thus improves the prediction performance of the model. We evaluate the proposed model on the task of predicting climate observations (Temperature) from the land-based weather stations located in the United States. Data and task We sample the weather stations located in the United States from the Online Climate Data Directory of the National Oceanic and Atmospheric Administration (NOAA) and choose the stations which have actively measured meteorological observations during 2015. We choose two geographically close but meteorologically diverse groups of stations: the Western and Southeastern states. We use k-Nearest Neighbor (NN) algorithm (k = 4) to generate graph structures and the final adjacency matrix is A = (A k + A k)/2 to make it symmetric where A k is the output adjacency matrix from k-NN algorithm. Our main task is to predict the next graph signals based on the current and past graph signals. All methods we evaluate are trained through the objective (Eq 3) with the Adam optimizer and we use scheduled sampling for the models with recurrent modules. We evaluate PA-DGN and other baselines on two prediction tasks, 1-step and multistep-ahead predictions. Furthermore, we demonstrate the ablation study that provides how much the spatial derivatives are important signals to predict the graph dynamics. We compare against the widely used baselines (VAR, MLP, and GRU) for 1-step and multistep prediction. Then, we use Recurrent Graph Neural Networks (RGN) to examine how much the graph structure is beneficial. Finally, we evaluate PA-DGN to verify if the proposed architecture (Eq 1) is able to improve the prediction quality. Experiment for the prediction task are summarized in Table 3. Overall, RGN and PA-DGN are better than other baselines and it implies that the graph structure provides useful inductive bias for the task. It is intuitive as the meteorological observations are continuously changing over the space and time and thus, the observations of the closer stations from the i-th station are strongly related to observations at the i-th station. PA-DGN outperforms RGN and the discrepancy comes from the fact that the spatial derivatives (Eq 1) we feed in PA-DGN are beneficial and this finding is expected because the meteorological signals at a certain point are a function of not only its previous signal but also the relative differences between neighbor signals and itself. Knowing the relative differences among local observations is particularly essential to understand physics-related dynamics. For example, Diffusion equation, which describes how physical quantities (e.g., heat) are transported through space over time, is also a function of relative differences of the quantities (df dt = D∆f) rather than values of the neighbor signals. In other words, spatial differences are physics-aware features and it is desired to leverage the features as input to learn dynamics related to physical phenomena. We further investigate if the modulated spatial derivatives (Eq 1) are effectively advantageous compared to the spatial derivatives defined in Riemannian manifolds. First, RGN without any spatial derivatives is assessed for the prediction tasks on Western and Southeastern states graph signals. Note that this model does not use any extra features but the graph signal, f (t). Secondly, we add StandardOP, the discrete spatial differences (Gradient and Laplacian) in Section 2.1 and MeshOP, the triangular mesh approximation of differential operators in Section 2.2 separately as additional signals to RGN. Finally, we incorporate with RGN our proposed Spatial Difference Layer. Table 3 shows the contribution of each component. As expected, PA-DGN provides much higher drops in MAE (3.56%,5.50%,8.51% and 8.73%,8.32%,5.49% on two datasets, respectively) compared to RGN without derivatives and the demonstrate that the derivatives, namely, relative differences from neighbor signals are effectively useful. However, neither RGN with StandardOP nor with MeshOP can consistently outperform RGN. We also found that PA-DGN consistently shows positive effects on the prediction error compared to the fixed derivatives. This finding is a piece of evidence to support that the parameters modulating spatial derivatives in our proposed Spacial Difference Layer are properly inferred to optimize the networks. In this paper, we introduce a novel architecture (PA-DGN) that approximates spatial derivatives to use them to represent PDEs which have a prominent role for physics-aware modeling. PA-DGN effectively learns the modulated derivatives for predictions and the derivatives can be used to discover hidden physics describing interactions between temporal and spatial derivatives. A.1 SIMULATED DATA For the simulated dynamics, we discretize the following partial differential equation similar to the one in to simulate the corresponding linear variable-coefficient convection-diffusion equation on graphs. In a continuous space, we define the linear variable-coefficient convection-diffusion equation as:, with We follow the setting of initialization in:, where N = 9, λ k,l, γ k,l ∼ N 0, 1 50, and k and l are chosen randomly. We use spatial difference operators to approximate spatial derivatives:, where s is the spatial grid size for discretization. Then we rewrite with difference operators defined on graphs:, where Then we replace the gradient w.r.t time in with temporal discretization:, where ∆t is the time step in temporal discretization. Equation is used for simulating the dynamics described by the equation. Then, we uniformly sample 250 points in the above 2D space and choose their corresponding time series of u as the dataset used in our synthetic experiments. We generate 1000 sessions on a 50 × 50 regular mesh with time step size ∆t = 0.01. 700 sessions are used for training, 150 for validation and 150 for test. Here we provide additional details for models we used in this work, including model architecture settings and hyper-parameter settings. Unless mentioned otherwise, all models use a hidden dimension of size 64. • VAR: A vector autoregression model with 2 lags. Input is the concatenated features of previous 2 frames. The weights are shared among all nodes in the graph. • MLP: A multilayer perceptron model with 2 hidden layers. Input is the concatenated features of previous 2 frames. The weights are shared among all nodes in the graph. • GRU: A Gated Recurrent Unit network with 2 hidden layers. Input is the concatenated features of previous 2 frames. The weights are shared among all nodes in the graph. • RGN: A recurrent graph neural network model with 2 GN blocks. Each GN block has an edge update block and a node update block, both of which use a 2-layer GRU cell as the update function. We set its hidden dimension to 73 so that it has the same number of learnable parameters as our proposed model PA-DGN. • RGN(StandardOP): Similar to RGN, but use the output of difference operators in Section 2.1 as extra input features. We set its hidden dimension to 73. • RGN(MeshOP): Similar to RGN(StandardOP), but the extra input features are calculated using opeartors in Section 2.2. We set its hidden dimension to 73. • PA-DGN: Our proposed model. The spatial derivative layer uses a message passing neural network (MPNN) with 2 GN blocks using 2-layer MLPs as update functions. The forward network part uses a recurrent graph neural network with 2 recurrent GN blocks using 2-layer GRU cells as update functions. The numbers of learnable parameters of all models are listed as follows: The number of evaluation runs We performed 3 times for every experiment in this paper to report the mean and standard deviations. Length of prediction For experiments on synthetic data, all models take first 5 frames as input and predict the following 15 frames. For experiments on NOAA datasets, all models take first 12 frames as input and predict the following 12 frames. Training hyper-parameters We use Adam optimizer with learning rate 1e-3, batch size 8, and weight decay of 5e-4. All experiments are trained for a maximum of 2000 epochs with early stopping. All experiments are trained using inverse sigmoid scheduled sampling with the coefficient k = 107. Environments All experiments are implemented with Python3.6 and PyTorch 1.1.0, and are run with NVIDIA GTX 1080 Ti GPUs. In this section, we evaluate the effect of 2 different graph structures on baselines and our models: k-NN: a graph constructed with k-NN algorithm (k = 4); TriMesh: a graph generated with Delaunay Triangulation. All graphs use the Euclidean distance. Table 5 and Table 6 show the effect of different graph structures on the synthetic dataset used in Section 3.2 and the real-world dataset in Section 4.2 separately. We find that for different models the effect of graph structures is not homogeneous. For RGN and PA-DGN, k-NN graph is more beneficial to the prediction performance than TriMesh graph, because these two models rely more on neighboring information and a k-NN graph incorporates it better than a Delaunay Triangulation graph. However, switching from TriMesh graph to k-NN graph is harmful to the prediction accuracy of RGN(MeshOP) since Delaunay Triangulation is a well-defined method for generating triangulated mesh in contrast to k-NN graphs. Given the various effect of graph structures on different models, our proposed PA-DGN under k-NN graphs always outperforms other baselines using any graph structure. Figure 7 provides the distribution of MAEs across the nodes of PA-DGN applied to the graph signal prediction task of the west coast region of the real-world dataset in Section 4.2. As shown in the figure, nodes with the highest prediction error for short-term prediction are gathered in the inner part where the observable nodes are sparse, while for long-term prediction nodes in the area with a limited number of observable points no longer have the largest MAE. This implies that PA-DGN can utilize neighboring information efficiently even under the limitation of sparsely observable points.

We propose physics-aware difference graph networks designed to effectively learn spatial differences to modeling sparsely-observed dynamics.

We show shortcut connections should be placed in patterns that minimize between-layer distances during backpropagation, and design networks that achieve log L distances using L log(L) connections.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Solving long-horizon sequential decision making tasks in environments with sparse rewards is a longstanding problem in reinforcement learning (RL) research. Hierarchical Reinforcement Learning (HRL) has held the promise to enhance the capabilities of RL agents via operation on different levels of temporal abstraction. Despite the success of recent works in dealing with inherent nonstationarity and sample complexity, it remains difficult to generalize to unseen environments and to transfer different layers of the policy to other agents. In this paper, we propose a novel HRL architecture, Hierarchical Decompositional Reinforcement Learning (HiDe), which allows decomposition of the hierarchical layers into independent subtasks, yet allows for joint training of all layers in end-to-end manner. The main insight is to combine a control policy on a lower level with an image-based planning policy on a higher level. We evaluate our method on various complex continuous control tasks for navigation, demonstrating that generalization across environments and transfer of higher level policies can be achieved. See videos https://sites.google.com/view/hide-rl Reinforcement learning (RL) has been succesfully applied to sequential-decision making tasks, such as learning how to play video games in Atari , mastering the game of Go or continuous control in robotics (; ; . However, despite the success of RL agents in learning control policies for myopic tasks, such as reaching a nearby target, they lack the ability to effectively reason over extended horizons. In this paper, we consider continuous control tasks that require planning over long horizons in navigation environments with sparse rewards. The task becomes particularly challenging with sparse and delayed rewards since an agent needs to infer which actions caused the reward in a domain where most samples give no signal at all. Common techniques to mitigate the issue of sparse rewards include learning from demonstrations or using enhanced exploration strategies (; ;). Hierarchical Reinforcement Learning (HRL) has been proposed in part to solve such tasks. Typically, a sequential decision making task is split into several simpler subtasks of different temporal and functional abstraction levels . Although the hierarchies would ideally be learned in parallel, most methods resort to curriculum learning (; ; ;). Recent goal-conditioned hierarchical architectures have successfully trained policies jointly via off-policy learning (; ; . However, these methods often do not generalize to unseen environments as we show in Section 5.1. We argue that this is due to a lack of true separation of planning and low-level control across the hierarchy. In this paper, we consider two main problems, namely functional decomposition of HRL architectures in navigation-based domains and generalization of RL agents to unseen environments (figure 1). To address these issues, we propose a novel multi-level HRL architecture that enables both functional decomposition and temporal abstraction. We introduce a 3-level hierarchy that decouples the major roles in a complex navigation task, namely planning and low-level control. The benefit of a modular design is twofold. First, layers have access to only task-relevant information for a predefined task, which significantly improves the generalization ability of the overall policy. Hence, this enables policies learned on a single task to solve randomly configured environments. Second, Figure 1: Navigation environments. The red sphere indicates the goal an agent needs to reach, with the starting point at the opposite end of the maze. The agent is trained on environment a). To test generalization, we use the environments with b) reversed starting and goal positions, c) mirrored maze with reversed starting and goal positions and d) randomly generated mazes. the planning and control layers are modular and thus allow for composition of cross-agent architectures. We empirically show that the planning layer of the hierarchy can be transferred successfully to new agents. During training we provide global environment information only to the planning layer, whereas the full internal state of the agent is only accessible by the control layer. The actions of the top and middle layers are in the form of displacement in space. Similarly, the goals of the middle and lowest layers are relative to the current position. This prevents the policies from overfitting to the global position in an environment and hence encourages generalization to new environments. In our framework (see figure 2), the planner (i.e., the highest level policy π 2) learns to find a trajectory leading the agent to the goal. Specifically, we learn a value map of the environment by means of a value propagation network . To prevent the policy from issuing too ambitious subgoals, an attention network estimates the range of the lower level policy π 0 (i.e., the agent). This attention mask also ensures that the planning considers the agent performance. The action of π 2 is the position which maximizes the masked value map, which serves as goal input to the policy π 1. The middle layer implements an interface between the upper planner and lower control layer, which refines the coarser subgoals into shorter and reachable targets for the agent. The middle layer is crucial in functionally decoupling the abstract task of planning (π 2) from agent specific continuous control. The lowest layer learns a control policy π 0 to steer the agent to intermediate goals. While the policies are functionally decoupled, they are trained together and must learn to cooperate. In this work, we focus on solving long-horizon tasks with sparse rewards in complex continuous navigation domains. We first show in a maze environment that generalization causes challenges for state-of-the-art approaches. We then demonstrate that training with the same environment configuration (i.e., fixed start and goal positions) can generalize to randomly configured environments. Lastly, we show the benefits of functional decomposition via transfer of individual layers between different agents. In particular, we train our method with a simple 2DoF ball agent in a maze environment to learn the planning layer which is later used to steer a more complex agent. The indicate that the proposed decomposition of policy layers is effective and can generalize to unseen environments. In summary our main contributions include: • A novel multi-layer HRL architecture that allows functional decomposition and temporal abstraction for navigation tasks. • This architecture enables generalization beyond training conditions and environments. • Functional decomposition that allows transfer of individual layers across different agents. Learning hierarchical policies has seen lasting interest (; ; ; ; ;), but many approaches are limited to discrete domains or induce priors. More recent works (; ; ; ;) have demonstrated HRL architectures in continuous domains. introduce FeUdal Networks (FUN), which was inspired by feudal reinforcement learn- Figure 2: Our 3-layer HRL architecture. The planning layer π 2 receives a birds eye view of the environment and the agent's position s xy and sets an intermediate target position g 2. The interface layer π 2 splits this subgoal into reachable targets g 1. A goal-conditioned control policy π 0 learns the required motor skills to reach the target g 1 given the agent's joint information s joints. ing . In FUN, a hierarchic decomposition is achieved via a learned state representation in latent space. While being able to operate in continuous state space, the approach is limited to discrete action spaces. introduce hierarchical structure into KLdivergence regularized RL using latent variables and induces semantically meaningful representations. The separation of concerns between high-level and low-level policy is guided by information asymmetry theory. Transfer of ing structure can solve or speed up training of new tasks or different agents. present HIRO, an off-policy HRL method with two levels of hierarchy. The non-stationary signal of the upper policy is mitigated via off-policy corrections, while the lower control policy benefits from densely shaped rewards. propose an extension of HIRO, which we call HIRO-LR, by learning a representation space from environment images, replacing the state and subgoal space with neural representations. While HIRO-LR can generalize to a flipped environment, it needs to be retrained, as only the learned space representation generalizes. Contrarily, HiDe generalizes without retraining. introduce Hierarchical Actor-Critic (HAC), an approach that can jointly learn multiple policies in parallel. The policies are trained in sparse reward environments via different hindsight techniques. HAC, HIRO and HIRO-LR consist of a set of nested policies where the goal of a policy is provided by the top layer. In this setting the goal and state space of the lower policy is identical to the action space of the upper policy. This necessitates sharing of the state space across layers. Overcoming this limitation, we introduce a modular design to decouple the functionality of individual layers. This allows us to define different state, action and goal spaces for each layer. Global information about the environment is only available to the planning layer, while lower levels only receive information that is specific to the respective layer. Furthermore, HAC and HIRO have a state space that includes the agent's position and the goal position, while and our method both have access to global information in the form of a top-down view image. In model-based reinforcement learning much attention has been given to learning of a dynamics model of the environment and subsequent planning (; ;). propose a planning method that performs a graph search over the replay buffer. However, they require to spawn the agent at different locations in the environment and let it learn a distance function in order to build the search graph. Unlike model-based RL, we do not learn state transitions explicitly. Instead, we learn a spatial value map from collected rewards. Recently, differentiable planning modules that can be trained via model-free reinforcement learning have been proposed (; ; ;). establish a connection between convolutional neural networks and Value Iteration . They propose Value Iteration Networks (VIN), an approach where modelfree RL policies are additionally conditioned on a fully differrentiable planning module. MVProp extends this work by making it more parameter-efficient and generalizable. The planning layer in our approach is based on MVProp, however contrary to prior work we do not rely on a fixed neighborhood mask to sequentially provide actions in its vicinity in order to reach a goal. Instead we propose to learn an attention mask which is used to generate intermediate goals for the underlying layers. learn a map of indoor spaces and planning on it using a multi-scale VIN. In their setting, the policy is learned from expert actions using supervised learning. Moreover, the robot operates on discrete set of high level macro actions. propose Universal Planning Networks (UPN), which learn how to plan an optimal action trajectory via a latent space representation. In contrast to our approach, the method relies on expert demonstrations and transfer to harder tasks can only be achieved after retraining. We model a Markov Decision Process (MDP) augmented with a set of goals G. We define the MDP as a tuple M = {S, A, G, R, T, ρ 0, γ}, where S and A are set of states and actions, respectively, R t = r(s t, a t, g t) a reward function, γ a discount factor ∈, T = p(s t+1 |s t, a t) the transition dynamics of the environment and ρ 0 = p(s 1) the initial state distribution, with s t ∈ S and a t ∈ A. Each episode is initialized with a goal g ∈ G and an initial state is sampled from ρ 0. We aim to find a policy π: S × G → A, which maximizes the expected return. We train our policies by using an actor-critic framework where the goal augmented action-value function is defined as: The Q-function (critic) and the policy π (actor) are approximated by using neural networks with parameters θ Q and θ π. The objective for θ Q minimizes the loss:, where The policy parameters θ π are trained to maximize the Q-value: To address the issue of sparse rewards, we utilize Hindsight Experience Replay (HER) , a technique to improve sample-efficiency in training goal-conditioned environments. The insight is that the desired goals of transitions stored in the replay buffer can be relabeled as states that were achieved in hindsight. Such data augmentation allows learning from failed episodes, which may generalize enough to solve the intended goal. apply two hindsight techniques to address the challenges introduced by the non-stationary nature of hierarchical policies and the environments with sparse rewards. In order to train a policy π i, optimal behavior of the lower-level policy is simulated by hindsight action transitions. More specifically, the action a i is replaced with a state s i−1 that is actually achieved by the lower-level policy π i−1. Identically to HER, hindsight goal transitions replace the subgoal g i−1 with an achieved state s i−1, which consequently assigns a reward to the lower-level policy π i−1 for achieving the virtual subgoal. Additionally, a third technique called subgoal testing is proposed. The incentive of subgoal testing is to help a higher-level policy understand the current capability of a lower-level policy and to learn Q-values for subgoal actions that are out of reach. We find both techniques effective and apply them to our model during training. propose differentiable value iteration networks (VIN) for path planning and navigation problems. propose value propagation networks (MVProp) with better sample efficiency and generalization behavior. MVProp creates reward-and propagation maps covering the environment. The reward map highlights the goal location and the propagation map determines the propagation factor of values through a particular location. The reward map is an imager i,j of the same size as the environment image I, wherer i,j = 0 if the pixel (i, j) overlaps with the goal position and −1 otherwise. The value map V is calculated by unrolling max-pooling operations in a neighborhood N for k steps as follows: Figure 3: Planner layer π 2 (s xy, G, I). Given the top-view environment image I and goal G on the map, the maximum value propagation network (MVProp) calculates a value map V. By using the agent's current position s xy, we estimate an attention mask M restricting the global value map V to a local and reachable subgoal mapV. The policy π 2 selects the coordinates with maximum value and assigns the lower policy π 1 with a sugboal that is relative to the agent's current position. The action (i.e., the target position) is selected to be the pixels (i, j) maximizing the value in a predefined 3x3 neighborhood N (i 0, j 0) of the agent's current position (i 0, j 0): Note that the window N (i 0, j 0) is determined by the discrete, pixel-wise actions. We introduce a novel hierarchical architecture, HiDe, allowing for an explicit functional decomposition across layers. Similar to HAC , our method achieves temporal abstractions via nested policies. Moreover, our architecture enables functional decomposition explicitly. This is achieved by nesting i) an abstract planning layer, followed ii) by a local planer to iii) guide a control component. Crucially, only the top layer receives global information and is responsible for planning a trajectory towards a goal. The lowest layer learns a control policy for agent locomotion. The middle layer converts the planning layer output into subgoals for the control layer. Achieving functional decoupling across layers crucially depends on reducing the state in each layer to the information that is relevant to its specific task. This design significantly improves generalization (see Section 5). The highest layer of a hierarchical architecture is expected to learn high-level actions over a longer horizon, which define a coarse trajectory in navigation-based tasks. In the related work (; ;, the planning layer, learning an implicit value function, shares the same architecture as lower layers. Since the task is learned for a specific environment, limits to generalization are inherent to this design choice. In contrast, we introduce a planning specific layer consisting of several components to learn the map and to find a feasible path to the goal. The planning layer is illustrated in figure 3 . We utilize a value propagation network (MVProp) to learn an explicit value map which projects the collected rewards onto the environment image. Given a top-down image of the environment, a convolutional network determines the per pixel flow probability p i,j. For example, the probability value of a pixel corresponding to a wall should be 0 and that for free passages 1 respectively. use a predefined 3 × 3 neighborhood of the agent's current position and pass the location of the maximum value in this neighbourhood as goal position to the agent (equation 5). We augment a MVProp network with an attention model which learns to define the neighborhood dynamically and adaptively. Given the value map V and the agent's current position s xy, we estimate how far the agent can go, modeled by a 2D Gaussian. More specifically, we predict a full covariance matrix Σ with the agent's global position s xy as mean. We later build a 2D mask M of the same size as the environment image I by using the likelihood function: Figure 4: A visual comparison of (left) our dynamic attention window with a (right) fixed neighborhood. The green dot corresponds to the selected subgoal in this case. Notice how our window is shaped so that it avoids the wall and induces a further subgoal. Intuitively, the mask defines the density for the agent's success rate. Our planner policy selects an action (i.e., subgoal) that maximizes the masked value map as follows: wherev i,j corresponds to the value at pixel (i, j) on the masked value mapV. Note that the subgoal selected by the planning layer g 2 is relative to the agent's current position s xy, which improves generalization performance of our model. The benefits of having an attention model are twofold. First, the planning layer considers the agent dynamics in assigning subgoals which may lead to fine-or coarse-grained subgoals depending on the underlying agent's performance. Second, the Gaussian window allows us to define a dynamic set of actions for the planner policy π 2, which is essential to find a trajectory of subgoals on the map. While the action space includes all pixels of the value map V, it is limited to the subset of only reachable pixels by the Gaussian mask M. Qualitatively we find this leads to better obstacle avoidance behaviour such as the corners and walls shown in figure 4. Since our planner layer operates in a discrete action space (i.e., pixels), the resolution of the projected maze image defines the minimum amount of displacement for the agent, affecting maneuverability. This could be tackled by using a soft-argmax to select the subgoal pixel, allowing to choose real-valued actions and providing in-variance to image resolution. In our experiments we see no difference in terms of the final performance. However, since the former setting allows for the use of DQN instead of DDPG , we prefer the discrete action space for simplicity and faster convergence. The middle layer in our hierarchy interfaces the high-level planning with low-level control by introducing an additional level of temporal abstraction. The planner's longer-term goals are further split into a number of shorter-term targets. Such refinement policy provides the lower-level control layer with reachable targets, which in return yields easier rewards and hence accelerated learning. The interface layer policy is the only layer that is not directly interacting with the environment. More specifically, the policy π 1 only receives the subgoal g 2 from the upper layer π 2 and chooses an action (i.e. subgoal g 1) for the lower-level locomotion layer π 0. Note that all the goal, state and action spaces of the policy π 1 are in 2D space. Contrary to , we use subgoals that are relative to the agent's position s xy. This helps to generalize and learn better. The lowest layer learns a goal-conditioned control policy. Due to our explicit functional decomposition, it is the only layer with access to the agent's internal state s joints including joint positions and velocities. Whereas the higher layers only have access to the agent's position. In a navigation task, the agent has to learn locomotion to reach the goal position. Similar to HAC, we use hindsight goal transition techniques so that the control policy receives rewards even in failure cases. All policies in our hierarchy are jointly-trained. We use the DDPG algorithm with the goal-augmented actor-critic framework (equation 2-3) for the control and interface layers, and DQN for the planning layer (see section 4.1). We evaluate our method on a series of simulated continuous control tasks in navigation-based environments 1. All environments are simulated in the MuJoCo physics engine . Experiment and implementation details are provided in the Appendix B. First, in section 5.1, we compare to various baseline methods. In section 5.2, we move to a new maze with a more complex design in order to show our model's generalization capabilities. Section 5.3 demonstrates that our approach indeed leads to functional decomposition by composing new agents via combining the planning layer of one agent with the locomotion layer of another. Finally, in section 5.4 we provide an ablation study for our design choices. We introduce the following task configurations: Maze Forward: the training environment in all experiments. The task is to reach a goal from a fixed pre-determined start position. Maze Backward: the training maze layout with swapped start and goal positions. Maze Flipped: a mirrored version of the training environment. Maze Random: a set of randomly generated mazes with random start and goal positions. In our experiments, we always train in the Maze Forward environment. The reward signal during training is constantly -1, unless the agent reaches the given goal (except for HIRO and HIRO-LR, see section 5.1). We test the agents on the above tasks with fixed starting and fixed goal position. For more details about the environments, we refer to Appendix A. We intend to answer the following two questions: 1) Can our method generalize to unseen test environments? 2) Is it possible to transfer the planning layer policies between agents? We compare our method to state-of-the-art approaches including HIRO , HIRO-LR , HAC and a modified version of HAC called Rel-HAC in a simple Maze Forward environment as shown in figure 6. For a fair comparison, we made a number of improvements to the HAC and HIRO implementations. For HAC, we introduced target networks and used the hindsight experience replay technique with the future strategy . In our experiments we observed that oscillations around the goal kept HIRO agents from finishing the task, which was solved via doubling the distance-threshold of success. HIRO-LR is the closest to our method, as it also receives a top-down view image of the environment. Note that both HIRO and HIRO-LR have access to dense negative distance reward, which is an advantage over HAC and HiDe that only receive a reward when finishing the task. We train a modified HAC model, dubbed RelHAC, to asses our planning layer. RelHAC has the same lowest and middle layers as HiDe, whereas the top layer has the same structure as the middle layer, therefore missing an effective planner. Preliminary experiments using fixed start and fixed goal positions during training for HAC, HIRO and HIRO-LR yielded 0 success rates in all cases. Therefore, the baseline models are trained by using fixed start and random goal positions, allowing it to receive a reward signal without having to reach the intended goal at the other end of the maze. Contrarily, HiDe is trained with fixed start and fixed goal positions, whereas HiDe-R represents HiDe under the same conditions as the baseline methods. All models learned this task successfully as shown in figure 5 and table 1 (Forward column). HIRO demonstrates slightly better convergence and final performance, which can be attributed to the fact that it is trained with dense rewards. RelHAC performs worse than HAC due to the pruned state space of each layer and due to the lack of an effective planner. HIRO-LR takes longer to converge because it has to learn a latent goal space representation. Table 1 summarizes the models' generalization abilities to the unseen Maze Backward and Maze Flipped environments (see figure 6). While HIRO, HIRO-LR and HAC manage to solve the training environment (Maze Forward) with success rates between 99% and 82%, they suffer from overfiting to the training environment, indicated by the 0% success rates in the unseen test environments. Contrarily, our method is able to achieve 54% and 69% success rates in this generalization task. As expected, training our model with random goal positions (i.e., HiDe-R) yields a more robust model outperforming vanilla HiDe. In subsequent experiments, we only report the for our method, as our experiments have shown that the baseline methods cannot solve the training task for more complex environments. In this experiment, we train an ant and a ball agent (see Appendix A.1) in the Maze Forward task with a more complex environment layout (cf. figure 1), while we keep both the start and goal positions intact. We then evaluate this model in 4 different tasks (see section 5). Table 2 reports success rates of both agents in this complex task. Our model successfully transfers its navigation skills to unseen environments. The performance for the Maze Backward and Maze Flipped tasks is similar to the shown in section 5.1 despite the increased difficulty. Since the randomly generated mazes are typically easier, our model shows similar or better performance. To demonstrate that the layers in our architecture indeed learn separate sub-tasks we transfer individual layers across different agents. We first train an agent without our planning layer, i.e., with RelHAC. We then replace the top layer of this agent with the planning layer from the models trained in section 5.2. Additionally, we train a humanoid agent and show as a proof of concept that transfer to a very complex agent can be achieved. We carry out two sets of experiments. First, we transfer the ant model's planning layer to the simpler 2 DoF ball agent. As indicated in Table 3, the performance of the ball with the ant's planning layer matches the in Table 2. The ball agent's success rate increases for random (from 96% to 100%) and forward (96% to 97%) maze tasks whereas it decreases slightly in the backward (from 100% to 90%) and flipped (from 99% to 88%) configurations. HiDe-A 0 ± 0 0 ± 0 0 ± 0 HiDe-AR 95 ± 1 52 ± 33 34 ± 45 Table 4: Success rates in the simple maze. HiDe-A is our method with absolute subgoals. HiDe-AR has absolute goals and samples random goals during training. HiDe-A 0 ± 0 0 ± 0 0 ± 0 0 ± 0 HiDe-AR 0 ± 0 0 ± 0 0 ± 0 0 ± 0 HiDe-NI 10 ± 5 46 ± 16 0 ± 0 3 ± 4 Table 5: Success rates of achieving a goal in the complex maze environment. HiDe-A and HiDe-AR as in Table 4. HiDe-NI is our method without the inferface layer. Second, we attach the ball agent's planning layer to the more complex ant agent. Our new compositional agent performs marginally better or worse in the Flipped, Random and Backward tasks. Please note that this experiment is an example of a case where the environment is first learned with a fast and easy-to-train agent (i.e., ball) and then utilized by a more complex agent. We hereby show that transfer of layers between agents is possible and therefore find our hypothesis to be valid. Moreover, an estimate indicates that the training is roughly 3 -4 times faster, since the complex agent does not have to learn the planning layer. To demonstrate our method's transfer capabilities, we train a humanoid agent (17 DoF) in an empty environment with shaped rewards. We then use the planning and interface layer from a ball agent and connect it as is with the locomotion layer of the trained humanoid 2. To support the claim that our architectural design choices lead to better generalization and functional decomposition, we compare empirical for different variants of our method with an ant agent. First, we compare the performance of relative and the absolute positions for both experiment 1 and experiment 2. For this reason, we train HiDe-A and HiDe-AR, the corresponding variants of HiDe and HiDe-R that use absolute positions. Unlike for relative positions, the policy needs to learn all values within the range of the environment dimensions. Second, we compare HiDe against a variant of HiDe without the interface layer called HiDe-NI. The for experiment 1 are in Table 4. HiDe-A does not manage to solve the task at all, similar to HAC and HIRO without random goal sampling. HiDe-AR succeeds in solving the Forward task. However, it generalizes worse than both Hide and HiDe-R in the Backward and Flipped task. Both HiDe-A and HiDe-AR fail to solve the complex maze for experiment 2 as shown in the Table 5. These indicate that 1) relative positions improve performance and are an important aspect of our method to achieve generalization to other environments and 2) random goal position sampling can help agents, but may not be available depending on the environment. As seen in Table 5, the variant of HiDe without interface layer (HiDe-NI) performs worse than HiDe (cf. Table 2) in all experiments. Thus, the interface layer is an important part of our architecture. We also run an ablation study for HiDe with a fixed window size. More specifically, we train and evaluate an ant agent on window sizes 3×3, 5×5, and 9×9. The are included in Tables 12,13, and 14. The learned attention window (HiDe) achieves better or comparable performance. In all cases, HiDe generalizes better in the Backward complex maze. Moreover, the learned attention eliminates the need for tuning the window size hyperparameter per agent and environment. In this paper, we introduce a novel HRL architecture that can solve complex navigation tasks in 3D-based maze environments. The architecture consists of a planning layer which learns an explicit value map and is connected with a subgoal refinement layer and a low-level control layer. The framework can be trained end-to-end. While training with a fixed starting and goal position, our method is able to generalize to previously unseen settings and environments. Furthermore, we demonstrate that transfer of planners between different agents can be achieved, enabling us to transfer a planner trained with a simplistic agent such as a ball to a more complex agent such as an ant or humanoid. In future work, we want to consider integration of a more general planner that is not restricted to navigation-based environments. We build on the Mujoco environments used in. All environments use dt = 0.02. Each episode in experiment 1 is terminated after 500 steps and after 800 steps in the rest of the experiments or after the goal in reached. All rewards are sparse as in , i.e., 0 for reaching the goal and −1 otherwise. We consider goal reached if |s − g| max < 1. Since HIRO sets the goals in the far distance to encourage the lower layer to move to the goal faster, it can't stay exactly at the target position. Moreover, they do not terminate the episode after the goal is reached. Thus for HIRO, we consider a goal reached if |s − g| 2 < 2.5. Our Ant agent is equivalent to the one in. In other words, the Ant from Rllab with gear power of 16 instead of 150 and 10 frame skip instead of 5. Our Ball agent is the PointMass agent from DM Control Suite . We made the change the joints so that the ball rolls instead of sliding. Furthermore, we resize the motor gear and the ball itself to match the maze size. All mazes are modelled by immovable blocks of size 4 × 4 × 4. uses blocks of 8 × 8 × 8. The environment shapes are clearly depicted in figure 1. For the randomly generated maze, we sample each block with probability being empty p = 0.8,start and goal positions are also sampled randomly at uniform. Mazes where start and goal positions are adjacent or where goal is not reachable are discarded. For the evaluation, we generated 500 of such environments and reused them (one per episode) for all experiments. Our PyTorch implementation will be available at the project website. For both HIRO and HAC we used the original authors implementation 45. In HIRO, we set the goal success radius for evaluation as described above. We ran the hiro xy variant, which uses only position coordinates for subgoal instead of all joint positions, to have a fair comparison with our method. To improve the performance of HAC in experiment one, we modified their Hindsight Experience Replay implementation so that they use FUTURE strategy. More importantly, we also added target networks to both the actor and critic to improve the performance. For evaluation, we trained 5 seeds each for 2.5M steps on the Forward environment with continuous evaluation (every 100 episodes for 100 episodes). After training, we selected the best checkpoint based on the continuous evaluation of each seed. Then, we tested the learned policies for 500 episodes and reported the average success rate. Although the agent and goal positions are fixed, the initial joint positions and velocities are sampled from uniform distribution as standard in OpenAI Gym environments. Therefore, the tables in the paper contain means and standard deviation across 5 seeds. B.3 NETWORK STRUCTURE B.3.1 PLANNING LAYER Input images for the planning layer were binnarized in the following way: each pixel corresponds to one block (0 if it was a wall or 1 if it was a corridor). In our planning layer, we process the input image of size 32x32 (20x20 for experiment 1) via two convolutional layers with 3 × 3 kernels. Both layers have only 1 input and output channel and are padded so that the output size is the same as the input size. We propagate the value through the value map as in K = 35 times using a 3 × 3 max pooling layer. Finally, the value map and agent position image (a black image with a dot at agent position) is processed by 3 convolutions with 32 output channels and 3 × 3 filter window interleaved by 2 × 2 max pool with ReLU activation functions and zero padding. The final is flatten and processed by two fully connected layers with 64 neurons each producing three outputs: σ 1, σ 2, ρ with softplus, softplus and tanh activation functions respectively. The final covariance matrix Σ is given by, so that the matrix is always symmetric and positive definite. For numerical reasons, we multiply by binnarized kernel mask instead of the actual Gaussian densities. We set values higher than mean to 1 and others to zeros. In practice, we use this line: kernel = t.where(kernel >= kernel.mean(dim=, keepdim=True), t.ones_like(kernel), t.zeros_like(kernel)) We use the same network architecture for the middle and lower layer as proposed by , i.e. we use 3 times fully connected layer with ReLU activation function. The locomation layer is activated with tanh, which is then scaled to the action range. • Discount γ = 0.98 for all agents. • Adam optimizer. Learning rate 0.001 for all actors and critics. • Soft updates using moving average; τ = 0.05 for all controllers. • Replay buffer size was designed to store 500 episodes, similarly as in • We performed 40 actor and critic learning updates after each epoch on each layer, after the replay buffer contained at least 256 transitions. • Batch size 1024. • No gradient clipping • Rewards 0 and -1 without any normalization. • Subgoal testing only for the middle layer. • Maximum subgoal horizon H = 10 for all 3 layers algorithms and H = 25 for ablations without the inferace layer. See psuedocode 1. • Observations also were not normalized. • 2 HER transitions per transition using the FUTURE strategy . • Exploration noise: 0.05, 0.01 and 0.1 for the planning, middle and locomotion layer respectively. In this section, we present all collected for this paper including individual runs.

Learning Functionally Decomposed Hierarchies for Continuous Navigation Tasks

Represent this text to find a much much longer article about it with say hundreds-thousands of words

Adaptation of an RL agent in a target environment with unknown dynamics is fast and safe when we transfer prior experience in a variety of environments and then select risk-averse actions during adaptation.