Question: A plethora of methods attempting to explain predictions of black-box models have been proposed by the Explainable Artificial Intelligence (XAI) community. Yet, measuring the quality of the generated explanations is largely unexplored, making quantitative comparisons non-trivial. In this work, we propose a suite of multifaceted metrics that enables us to objectively compare explainers based on the correctness, consistency, as well as the confidence of the generated explanations. These metrics are computationally inexpensive, do not require model-retraining and can be used across different data modalities. We evaluate them on common explainers such as Grad-CAM, SmoothGrad, LIME and Integrated Gradients. Our experiments show that the proposed metrics reflect qualitative observations reported in earlier works. Over the past few years, deep learning has made significant progress, outperforming the state-ofthe-art in many tasks like image classification , semantic segmentation , machine translation and even surpassing humans in the games of Chess and Go . As these models are deployed in more mission-critical systems, we notice that despite their incredible performance on standard metrics, they are fragile and can be easily fooled by small perturbations to the inputs . Further research has also exposed that these models are biased in undesirable ways exacerbating gender and racial biases (; Escudé Font & Costa-Jussà, 2019). These issues have amplified the need for making these black-box models interpretable. Consequently, the XAI community has proposed a variety of algorithms that aim to explain predictions of these models (; ; ; ; ;). With such an explosion of interpretability methods (hereon referred to as explainers), evaluating them has become non-trivial. This is due to the lack of a widely accepted metric to quantitatively compare them. There have been several attempts to propose such metrics. Unfortunately, they tend to suffer from major drawbacks like computational cost , inability to be extended to non-image domains (a), or simply focusing only one desirable attribute of a good explainer. . In this paper, we propose a suite of metrics that attempt to alleviate these drawbacks and can be applied across multiple data modalities. Unlike the vast majority of prior work, we not only consider the correctness of an explainer, but also the consistency and confidence of the generated explanations. We use these metrics to evaluate and compare widely used explainers such as LIME , Grad-CAM , SmoothGrad and Integrated Gradients on an Inception-V3 model pretrained on the ImageNet dataset (ILSVRC2012) , in an objective manner (i.e., without the need of a human-in-the-loop). Moreover, our proposed metrics are general and computationally inexpensive. Our main contributions are: 1. Identifying and formulating the properties of a good explainer. 2. Proposing a generic, computationally inexpensive suite of metrics to evaluate explainers. 3. Comparing common explainers and discussing pros and cons of each. We find that while Grad-CAM seems to perform best overall, it does suffer from drawbacks not reported in prior works. On the other hand, LIME consistently underperforms in comparison to the other models. The field of XAI has become an active area of research with significant efforts being made to explain AI models, either by generating local (; ; ; ;) or global explanations. Simultaneously, there are growing research efforts into methods to formally evaluate and compare explainers (; ; ;). introduced a framework with three desiderata for evaluation, viz. predictive accuracy, descriptive accuracy and relevancy, with relevancy judged relative to a human. In contrast, compiled a set of desired characteristics around effectiveness, versatility, constraints (i.e., privacy, computation cost, information collection effort) and the type of generated explanations, which do not need human evaluation, and therefore are objective. However, they focus very little on aspects such as correctness. Recently, DeConvNet , Guided BackProp and LRP have been shown to not produce theoretically correct explanations of linear models (b). As a , two explanation techniques, PatternNet and PatternAttribution, that are theoretically sound for linear models were proposed. Other efforts focus on evaluating saliency methods (a;) and show that they are unreliable for tasks that are sensitive to either data or model. and its variations (; ;) infer whether a feature attribution is correct by measuring performance degradation when highly attributed features are removed. For instance, shows that commonly used interpretability methods are less accurate or are on-par with a random designation of feature importance, whereas ensemble approaches such as SmoothGrad are superior. proposed three complementary metrics to evaluate explainers: model contrast score -comparing two models trained to consider opposite concepts as important, input dependence score -comparing one model with two inputs of different concepts, and input dependence ratecomparing one model with two functionally identical inputs. These metrics aim to specifically cover aspects of false-positives. define an alternative set of metrics, around explicitness -intelligibility of explanations, faithfulness -feature relevance, and stabilityconsistency of explanations for similar or neighboring samples. define and evaluate fidelity of explanations, namely quantifying the degree to which an explanation captures how the underlying model itself changes in response to significant perturbations. Similar to previous work, we focus on objective metrics to evaluate and compare explainers. However, we not only consider correctness, but also consistency and confidence (as defined next). 3.1 PRELIMINARIES In the following discussions, let x ∈ R n be an arbitrary data point and y be the corresponding ground truth label from the dataset D = {(x i, y i), 1 ≤ i ≤ M }. Let f be the classifier realized as a neural network parameterized by weights θ. Let T be the set of transformations under which semantic information in the input remains unchanged. If t is an arbitrary transform from this set, let t −1 be its inverse transform. For example, if t = Rot−90 Let E f be any explainer that generates explanations for predictions made by classifier f 1. Finally, let d(,) be a function that computes the difference between the generated explanations 2. For example, if the explainer E generates saliency maps (e.g. GradCAM and SmoothGrad), d could be a simple p norm. Additionally, in order to ensure that we minimize the impact that pathologies of the underlying classifier have on the properties we are interested in, we assume that the classifier has acceptable test-set performance. Furthermore, we also assume that the classifier performance does not degrade significantly under the transformations we perform (described in Sec. 3.2.2). If the classifier does not satisfy these conditions, it is prudent to improve its performance to acceptable levels prior to attempting to explain its outputs. One cannot extract reliable explanations from underperforming underlying models . Inspired by earlier works on important aspects of an explainer's quality (; ;), our proposed evaluation framework consists of the following components: We elaborate on these components as well as methods to compute them in the image classification scenario. Even though these are evaluated independently, they can be combined together to give a single scalar value to compare explainers in a straightforward way. However, the weight for each component depends heavily on the use case and end-user preference. This is beyond the scope of the current work and thus is not discussed further. Further, since we elaborate on the image classification scenario, we use inputs and images interchangeably with the understanding that the described methods or equivalents can be trivially adapted in other modalities. Correctness (sensitivity or fidelity in literature) refers to the ability of an explainer to correctly identify components of the input that contribute most to the prediction of the classifier. Most metrics proposed so far focus solely on correctness and attempt to compute it in different ways, often requiring retraining of the underlying classifier. Moreover, they do not capture all aspects of correctness nor do they generalize to other data modalities. We propose a novel computationally-inexpensive method that addresses these drawbacks. It takes into consideration both that the explainer identifies most of the relevant components and does not incorrectly select non-important components as important. If the explainers are performing as expected, a simple masking of the input image with the associated explanation should provide better accuracy as the network is unlikely to be confused by the nonimportant pixels. However, we do not observe this in practice, as we show empirically that vanilla masking in severe performance deterioration (see Table 9 and 10 for ). We hypothesize that this is because of the following reasons: • The masked image has a large proportion of empty pixels 3 and thus does not belong to the data distribution (p data) • Extracted pixels are important in the context of the pixels, and as such removing context makes the masking meaningless. Additionally, Convolutions have the inductive bias that the neighbouring pixels are highly correlated that helps perform well on visual tasks . Simple masking breaks this correlation. Based on the above observations, we conclude that it is crucial to have a realistic for the extracted patches to properly evaluate them. We propose the following procedure to provide a such that the ing image is closer the data distribution 2 We do not require d(,) to be a distance metric in the strictest sense. 3 using the first convolution layer bias as a values for the blank pixels does not help either For each class in the dataset, we select the top k and bottom k images, sorted in decreasing order based on the probability assigned by the classifier to the ground-truth class. We then randomly pair each of the top images with one of the bottom images. For each pair, we extract important regions identified by the explainer from the top image (i.e high confidence images) and overlap them over the corresponding bottom image (i.e low confidence images). We use the bottom k images for this task as we know that they are uninformative for the classifier as evidenced by the assigned probability. We thus obtain a dataset of masked images with important regions from the most important images along with relevant yet non-informative s for each class (see Fig. 1 for an example). Formally, the masking operation can be represented as: Where M is the new masked image, a threshold function, H the high confidence image, L the low confidence image and ⊗theelement − wisemultiplicationoperator. We then measure the accuracy on this masked dataset and compare it with the accuracy on the bottom k images subset. Note that the above mentioned process only evaluates if the explainer is capturing important pixels. In order to verify that the explainer does not select non-important pixels, we repeat the same process but instead use the inverted saliency map 4 and recompute accuracy on this dataset. In this scenario, we expect the accuracy to deteriorate. Formally, the inverse masking process can be defined as follow: Figure 1: Examples of the proposed algorithm for correctness Interestingly, these masked accuracies are similar to the precision and recall metrics used in information retrieval . This provides motivation to combine these differences into a Pseudo-F1 score by computing the harmonic mean of accuracy on normal masked images and 1 -accuracy on inverse masked images. Formally this can be computed as: We define consistency as the ability of the explainer to capture the same relevant components under various transformations to the input. More specifically, if the classifier predicts the same class for both the original and transformed inputs. Then, consistency measures whether the generated explanation for the transformed input (after applying an inverse transform) is similar to the one generated for the original input. For example, if we apply vertical flip as a semantically invariant transformation, we flip the generated heatmap from the transformed image before comparing with the heatmap generated for the original image. Formally, this can be represented as Semantically Invariant Transforms We focus on a subset of all potential transformations which does not change the semantic information contained in the input. We call this subset Semantically Invariant Transforms. Most work so far has considered only noising as a method of transforming the input. By constraining the magnitude of the added noise, we can control the size of the neighbourhood in which we perturb the images. In this work, we consider not only simple transformations that perturb the image in a small neighbourhood but also those that move the data point to vastly different regions in the input space while still retaining the semantic information contained within. This allows us to verify whether the explainer works as expected across larger regions of the input space. For example, in the case of images, the family of transformations T include affine transformations (translations and rotations), horizontal and vertical flips, noising (white, blue etc.), scaling etc. In the image domain, d could be realized as the 2 (Euclidean) distance between explanations of the ground truth and inverted the transformed images (according to Eq. 4). and have shown that 2 is not robust for images and may in larger distances between the pairs of mostly similar explanations. This is attributed to the fact that 2 is only a summation of the pixel-wise intensity differences and, as a , small deformations may in large distances. Even when the images are normalized before hand, 2 is still not a suitable distance for images. Therefore, we instead use Dynamic Time Warping (DTW) which allows for computing distances between two time series, even when misaligned or out of phase. has shown that DTW is effective for images as well, not only for temporal data as originally intended. Due to DTW's high computational cost (quadratic time and space complexity), we use FastDTW , an approximation of DTW that has linear complexity in order to compute the distance between pairs of explanations. Finally, confidence is concerned with whether the generated explanation and the masked input in high confidence predictions. This is a desirable property to enable explanations to be useful for downstream processes including human inspection . So far, our method for computing correctness sheds light only on the average case and is not particularly useful for individual explanations. Generating high-confidence predictions is related to the well researched field of max-margin classifiers . A large margin in classifiers is widely accepted as a desirable property. Here, we extend this notion to explainers and propose that explainers generating explanations that in high confidence predictions are desirable to those that do not. In addition to the desirable statistical properties that this enforces, high confidence predictions are also vital for building trust with human users of the explainer as they are more interested in the per-instance performance than the average . Concretely, we use the same procedure as in Sec. 3.2.1. Instead of computing the increase in accuracy, we compute instead the difference in probability assigned to the ground-truth class, as well as the difference in entropy of the softmax distributions of the original and masked images. We report this for both normal and inverted saliency maps. We expect to observe a positive probability difference and negative entropy difference under normal masking and an inverted behavior under inverse masking owing to similar reasons discussed in Sec. 3.2.1. However, explainers that generate coarse explanations can easily fool this metric. An extreme case is when the explainer considers the entire input as useful. Such an explainer is useless but will have the theoretically highest change in confidence and entropy. To combat this and to establish how sparse the generated explanations are, we also report the average number of pixels in the explanations, normalized by the total number of pixels in the image. We do not combine these numbers into one as different situations have different preferences. For example, in computational biology domains, sparsity is not as important as increase in confidence. The right weighting again depends on the use case and user preference. We use an Inception v3 architecture pretrained on ImageNet (ISLVC-2012) 5. We compare LIME, Grad-CAM, SmoothGrad and Integrated Gradients and measure how they perform on the metrics described previously. All experiments and explainers (except LIME) were implemented in PyTorch . Wherever possible, we reused the official implementation or kept our re-implementation as close to the official codebase as possible. The correctness and confidence metrics for every explainer are computed over 5 runs and mean values are reported. We use a fixed constant threshold to binarize explanation masks. We conducted further experiments by thresholding on the percentiles 6 instead(as done in ). These have been reported in tables 6 and 7. We found that the choice did not affect the relative trends observed. We consider the following semantically invariant transforms: translations (x = ±0.2, y = ±0.2), rotations (−15 •, −10 •), flips (horizontal and vertical). To establish that these do not produce too many out-of-distribution samples (causing a decrease in classifier performance), we compute the accuracy of the underlying classifier under these transformations. Table 1 shows that, indeed, drops in accuracy are not significant. Even though noising is semantically invariant in the image domain, we do not consider it in our experiments as some explainers like Smoothgrad would be unfairly favoured. Table 2. The baseline acc@1 and acc@5 were 11.42% and 53.8% respectively. We hypothesized that a good explainer's accuracy increases with the normal masking and decreases with inverse masking. We see the expected increases in accuracies across all the explainers with Grad-CAM obtaining the highest increase at 97.44%. However, for the inverse masking, we see that both LIME and Grad-CAM show contrary to our hypothesis. This can be explained by observing examples of maps generated in Figs. 1 and 4. We see that, on average, Grad-CAM generates much larger explanations than all other explainers (can be seen in Table 3 as well). This implies that Grad-CAM misidentifies several non-important pixels as important and thus when we compute the inverse masks, we remove non-important pixels that could confuse the classifier. In the case of LIME, we again see from Table 3 and Figs. 1 and 4 that LIME generates the smallest explanations. We further see from Table 2 that LIME has the smallest accuracy gain (in both acc@1 and acc@5). These indicate that LIME fails to select important pixels that were selected by all other explainers. Thus, we can conclude that the inverse masks in case of LIME would contain important pixels as well and thus would cause increase in accuracy as observed. As detailed previously, our methodology for computing correctness involves choosing a number k of top and bottom images to be used for masking. We evaluate how sensitive the measured correctness of explainers are to the value of k. We report the changes in accuracy with respect to the unmasked bottom images for k={5,10,15,20,25} in Fig. 2. The actual accuracy numbers are also reported in Tables 4 and 5. We see that for both acc@1 and acc@5, the change in accuracy for normal masking decreases as we increase k. This is as expected since the average confidence gap between the top-k and the bottom-k images decreases as k increases. This means that the important pixels in the images are masked with non-important pixels from the foreground images. On the contrary, LIME shows a smaller decrease in accuracy (both acc@1 and acc@5). This can be explained by the fact that LIME does not capture all important pixels, and therefore all important pixels from the are not replaced by less-informative pixels. Similarly, for acc@1 and acc@5 for inverse masking, we see that LIME, Smoothgrad and Integrated Gradients behave as expected, i.e., the drop in accuracy is diminished with k is increased as we are retaining the informative parts from the new images. Interestingly, the drop in accuracy for Grad-CAM is stable and close to zero. To understand this, we refer again to Table 3 and note that Grad-CAM produces the smallest inverse maps on average. This implies that when we perform the inverse masking, we retain much of the informative pixels of the image and thus do not see significant drops in accuracy relative to the unmasked bottom-k image dataset. Next, we evaluate consistency by computing the distance with FastDTW between the saliency maps generated on the original images and those generated when transformations are applied to the input image (Following Eq. 4). Fig. 3 and Table 8 report the normalized distances relative to each transformation (i.e., heatmaps sum to 1). First, as transformations become more drastic relative to the original saliency maps, the distances also increase. This is the desired behavior one would expect, thus motivating our choice for using FastDTW. Second, Grad-CAM outperforms all other explainers, as reflected by the fact that its corresponding distances are always smallest. It is followed by Smoothgrad, Integrated Gradients and LIME. This is expected given the grainy saliency maps obtained with Integrated Gradients and Smoothgrad, as well as the patchy heatmaps generated with LIME. Measuring confidence quantifies the performance of the explainers on a per-instance case and not only in the average. As described in Sec. 3.2.3, we compute the change in probability assigned to the ground-truth class (∆ conf) as well as the change in entropy of the softmax distribution (∆ entropy) as proxies for estimating the confidence of explanations. Additionally, we report the proportions of pixels in the heatmaps to the total number of pixels 7, averaged across the top-k dataset. We see that for confidence, the trends mimic the ones observed in Table 2. This implies that masking with extracted heatmaps not only increases accuracy but also in high-confidence predictions across explainers. More specifically, we see that Grad-CAM again outperforms the other explainers (both ∆ conf and ∆ entropy) in the normal heatmaps by large margins. In the case of inverse masking, confidence and entropy for LIME show behaviours contrary to our expectations. This can be attributed to the "patchiness" of the explanations generated by LIME which was discussed in the previous sections. 7 89401 in the standard ImageNet preprocessing pipeline for Inception-v3 In this paper, we formulated desired properties of a good explainer and proposed a generic, computationally inexpensive suite of metrics -correctness, consistency and confidence -to objectively evaluate and compare explainers. We compared well-known explainers, such as LIME, Grad-CAM, Integrated Gradients and SmoothGrad, on a pretrained Inception-V3 model on the ImageNet dataset. Our experiments show that the metrics proposed capture various pros and cons of each explainer allowing users to make an informed choice about which explainer to use for their use case. Specifically, we observe that Grad-CAM often performs better than the other explainers but suffers from drawbacks when inverse masking situations are considered. On the other hand, LIME performs poorly in all situations we consider. Moreover, we also point out the pitfalls of trying to combine from multiple metrics as they tend to hide anomalous behaviours of the underlying metrics (as seen from Pseudo-F1 from Table 2). We recommend that users sanity-check explainers by looking at individual metrics before making a decision based on the combined metric. Furthermore, we urge the XAI community to resist the temptation to propose an one-size-fits-all metrics as we have shown that such metrics tend to hide nuanced trade-offs that practitioners need to be aware of. Going forward, we invite the research community to test our metrics on other explainers, datasets, underlying classifiers and data modalities. Additionally, since the metrics proposed are differentiable, we believe exciting new liens of research would be to develop explainers that directly optimize for these metrics, as well as self-explaining models that incorporate such metrics into their learning regiment. A ADDITIONAL EXPERIMENTAL

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a suite of metrics that capture desired properties of explainability algorithms and use it to objectively compare and evaluate such methods

represent the tldr for finding the corresponding scientific article

We find that deep networks which generalize poorly are more reliant on single directions than those that generalize well, and evaluate the impact of dropout and batch normalization, as well as class selectivity on single direction reliance.


Question: Represent the scientific article to find a short TLDR

Question: Neural networks are known to produce unexpected on inputs that are far from the training distribution. One approach to tackle this problem is to detect the samples on which the trained network can not answer reliably. ODIN is a recently proposed method for out-of-distribution detection that does not modify the trained network and achieves good performance for various image classification tasks. In this paper we adapt ODIN for sentence classification and word tagging tasks. We show that the scores produced by ODIN can be used as a confidence measure for the predictions on both in-distribution and out-of-distribution datasets. The majority of the sentences in English-LinES treebank are from literature. English-EWT dataset is 89 larger and is more diverse. The datasets are described in Let s be a sentence from one of the datasets, and w 1,..., w M be the words. The embedding of 97 the m-th word of the sentence will be x m = W e hash(w m). We apply bidirectional LSTM on the DISPLAYFORM0. For sentiment analysis we apply a dense layer on the 99 concatenation of the last states of the two LSTMs: DISPLAYFORM1 is a cross-entropy: loss(s) = ce(S(f sc (s), 1)), where DISPLAYFORM2 is the modified 101 softmax function, T is the temperature scaling parameter, and C is the number of classes. For POS tagging we apply a dense layer on every hidden state: and ODIN(s) = max S(f sc (x), T ), wherex = x + sign(∇ x Sŷ(x))), whereŷ = argmax S(x, 1). DISPLAYFORM3 Here (perturbation magnitude) and T (temperature) are hyperparameters, which are chosen based 109 on the OOD detection performance on the development sets. For POS tagging, the gradient in the ODIN score formula is applied to the mean of word-level probability maximums. TAB3 shows the for OOD detection and Table 4 shows the rank correlation coefficients for PbThreshold and ODIN methods. The role of the temperature scaling and input perturbations All our experiments confirm the 129 observation from BID3 ] that temperature scaling improves out-of-distribution detection. The effect of higher temperatures saturates when T reaches thousands (Figure 1). The positive effect 131 of the perturbations on the inputs is visible for sentiment analysis, but not for POS tagging. We Ranking of the sentences ODIN is clearly better than PbThreshold according to Spearman's rank 141 correlation coefficient for POS tagging tasks (Table 4). For a neural network trained on en-LinES, ODIN scores are a good indicator how the network will perform on OOD samples. It is a much ODIN). and T for ODIN are determined based on the development sets of en-LinES (ID) and en-EWT (OOD). The size of a circle is proportional to the number of samples that fall into that bucket. Ideally, accuracy scores for the i-th bucket should be higher than for the (i − 1)-th bucket, and y coordinates of the three circles for each bucket should be the same. In this work we have adapted ODIN out-of-distribution detection method on sentence classification 163 and sequence tagging tasks. We showed that as an OOD detector it performs consistently better than 164 for the PbThreshold baseline. Additionally, we attempted to quantify how well the scores produced 165 by these methods can be used as confidence scores for the predictions of neural models. There are many other OOD detection methods that have yet to be tested on NLP tasks. On the other 167 hand, our analysis notably doesn't cover sequence-to-sequence tasks. We have shown that the usage

A recent out-of-distribution detection method helps to measure the confidence of RNN predictions for some NLP tasks

Represent the short summary for finding the corresponding scientific article (long)

The Frechet Distance between train and test distribution correlates with the change in performance for functions that are not invariant to the shift.


Question: Represent the scientific article to find a short TLDR.

Question: Some recent work has shown separation between the expressive power of depth-2 and depth-3 neural networks. These separation are shown by constructing functions and input distributions, so that the function is well-approximable by a depth-3 neural network of polynomial size but it cannot be well-approximated under the chosen input distribution by any depth-2 neural network of polynomial size. These are not robust and require carefully chosen functions as well as input distributions. We show a similar separation between the expressive power of depth-2 and depth-3 sigmoidal neural networks over a large class of input distributions, as long as the weights are polynomially bounded. While doing so, we also show that depth-2 sigmoidal neural networks with small width and small weights can be well-approximated by low-degree multivariate polynomials. Understanding the remarkable success of deep neural networks in many domains is an important problem at present (e.g., BID10). This problem has many facets such as understanding generalization, expressive power, optimization algorithms in deep learning. In this paper, we focus on the question of understanding the expressive power of neural networks. In other words, we study what functions can and cannot be represented and approximated by neural networks of bounded size, depth, width and weights. The early on the expressive power of neural networks showed that the depth-2 neural networks are universal approximators; that is to say, with only mild restrictions on the activation functions or neurons, the depth-2 neural networks are powerful enough to uniformly approximate arbitrary continuous functions on bounded domains in R d, e.g., BID2; BID9; BID0. However, the bounds that they provide on the size or width of these neural networks are quite general, and therefore, weak. Understanding what functions can be represented or wellapproximated by neural networks with bounded parameters is a general direction in the study of expressive power of neural networks. Here the parameters could mean the number of neurons, the width of hidden layers, the depth, and the magnitude of its weights etc. Natural signals (images, speech etc.) tend to be representable as compositional hierarchies BID10, and deeper networks can be thought of as representing deeper hierarchies. The power of depth has been a subject of investigation in deep learning, e.g., BID8. We are interested in understanding the effect of depth on the expressive power. In particular, one may ask whether having more depth allows representation of more functions if the size bound remains the same. BID5 show a separation between depth-2 and depth-3 neural networks. More precisely, they exhibit a function g: R d → R and a probability distribution µ on R d such that g is bounded and supported on a ball of radius O(√ d) and expressible by a depth-3 network of size polynomially bounded in d. But any depth-2 network approximating g in L 2 -norm (or squared error) within a small constant under the distribution µ must be of size exponentially large in d. Their separation works for all reasonable activation functions including ReLUs (Rectified Linear Units) and sigmoids. The function and the input distribution in BID5 are carefully constructed and their proof techniques seem to crucially rely on the specifics of these constructions. Building upon this , BID14 show that while the indicator function of the L 2 -ball can be well-approximated by depth-3 networks of polynomial size, any good approximation to it by depth-2 networks must require exponential size. Here, the notion of approximation in the lower bound is the same as in BID5 and a carefully constructed distribution that is arguably not quite natural. (see also BID12) also gave a separation between depth-2 and depth-3 networks by exhibiting a function g: S d−1 × S d−1 → R which can be well-approximated by a depth-3 ReLU neural network of polynomially bounded size and weights but cannot be approximated by any depth-2 (sigmoid, ReLU or more general) neural network of polynomial size with (exponentially) bounded weights. This separation holds under uniform distribution on S d−1 × S d−1, which is more natural than the previous distributions. However, the proof technique crucially uses harmonic analysis on the unit sphere, and does not seems robust or applicable to other distributions. shows a separation between depth-2k 3 + 8 and depth-k ReLU neural networks, for any positive integer k, when the input is uniformly distributed over [−1, 1] d. BID11 (see also BID14 BID21) show that there are univariate functions on a bounded interval such that neural networks of constant depth require size at least Ω (poly(1/)) for a uniform -approximation over the interval, whereas deep networks (the depth can depend on) can have size O (polylog(1/)).The above separation all fit the following template: certain carefully constructed functions can be well approximated by deep networks, but are hard to approximate by shallow networks using a notion of error that uses a carefully defined distribution. (is distribution-independent as it deals with uniform approximation everywhere in the domain). Thus these do not tell us the extent to which deeper networks are more expressive than the shallow ones. We would like to understand whether there are large classes of functions and distributions that witness the separation between deep and shallow networks. An answer to this question is also more likely to shed light on practical applications of neural networks. BID17; BID16; BID18 show that even functions computed by a depth-2 neural network of polynomial size can be hard to learn using gradient descent type of algorithms for a wide class of distributions. These address questions about learnability rather than the expressive power of deep neural networks. BID7 shows that piecewise affine functions on d with N pieces can be exactly represented by a width (d + 3) network of depth at most N. Lower bound of Ω((N + d − 1)/(d + 1)) on the depth is proven for functions of the above type when the network has width at most (d + 1) and very closely approximates the function. Our depth separation apply to neural networks with bounds on the magnitudes of the weights. While we would prefer to prove our without any weight restrictions, we now argue that small weights are natural. In training neural networks, often weights are not allowed to be too large to avoid overfitting. Weight decay is a commonly used regularization heuristic in deep learning to control the weights. Early stopping can also achieve this effect. Another motivation to keep the weights low is to keep the Lipschitz constant of the function computed by the network (w.r.t. changes in the input, while keeping the network parameters fixed) small. BID6 contains many of these references. One of the surprising discoveries about neural networks has been the existence of adversarial examples BID19 ). These are examples obtained by adding a tiny perturbation to input from class so that the ing input is misclassified by the network. The perturbations are imperceptible to humans. Existence of such examples for a network suggests that the Lipschitz constant of the network is high as noted in BID19. This lead them to suggest regularizing training of neural nets by penalizing high Lipschitz constant to improve the generalization error and, in particular, eliminate adversarial examples. This is carried out in BID1, who find a way to control the Lipschitz constant by enforcing an orthonormality constraint on the weight matrices along with other tricks. They report better resilience to adversarial examples. On the other hand, BID13 suggest that Lipschitz constant cannot tell the full story about generalization. We exhibit a simple function (derived from BID3) over the unit ball B d in d-dimensions can be well-approximated by a depth-3 sigmoidal neural network with size and weights polynomially bounded in d. However, its any reasonable approximation using a depth-2 sigmoidal neural network with polynomially bounded weights must have size exponentially large in d. Our separation is robust and works for a general class of input distributions, as long as their density is at least 1/poly(d) on some small ball of radius 1/poly(d) in Bd. The function we use can also be replaced by many other functions that are polynomially-Lipschitz but not close to any low-degree polynomial. As a by-product of our argument, we also show that constant-depth sigmoidal neural networks are well-approximated by low-degree multivariate polynomials (with a degree bound that allows the depth separation mentioned above). In this section, we show that a sigmoid neuron can be well-approximated by a low-degree polynomial. As a corollary, we show that depth-2 (and in genenral, small-depth) sigmoidal neural networks can be well-approximated by low-degree multivariate polynomials. The main idea is to use Chebyshev polynomial approximation as in , which closely approximates the minimax polynomial (or the polynomial that has the smallest maximum deviation) to a given function. For the simplicity of presentation and arguments, we drop the bias term b in the activation function σ(w, x + b). This is without loss of generality, as explained at the end of the last section. The activation function of a sigmoid neuron σ: R → R is defined as DISPLAYFORM0.Chebyshev polynomials of the first kind {T j (t)} j≥0 are defined recursively as T 0 (t) = 1, T 1 (t) = t, and T j+1 (t) = 2t · T j (t) − T j−1 (t). They form an orthonormal basis of polynomials over [−1, 1] with respect to the density 1/ DISPLAYFORM1 Proposition 1 (see Lemma B.1 in) bounds the magnitude of coefficients c j in the Chebyshev expansion of σ(wt) = ∞ j=0 c j T j (t). Proposition 1. For any j > 1, the coefficient c j in the Chebyshev expansion of a sigmoid neuron σ(wt) is bounded by DISPLAYFORM2 Proposition 1 implies low-degree polynomial approximation to sigmoid neurons as follows. This observation appeared in (see equation (B.7) in their paper). For completeness, we give the proof in Appendix A. Proposition 2. Given any w ∈ R with |w| ≤ B, there exists a polynomial p of degree DISPLAYFORM3 We use this O (log(1/)) dependence in the above bound crucially in some of our , e.g., a weaker version of Daniely's separation for depth-2 and depth-3 neural networks. Notice that this logarithmic dependence does not hold for a ReLU neuron; it is O(1/) instead. A depth-2 sigmoidal neural network on input t ∈ [−1, 1] computes a linear combination of sigmoidal neurons σ(w 1 t), σ(w 2 t),..., σ(w n t), for w 1, w 2,..., w n ∈ R, and computes a function DISPLAYFORM0 Here are a few propositions on polynomial approximations to small-depth neural networks. For completeness, their proofs are included in Appendix A.Proposition 3 shows that a depth-2 sigmoidal neural network of bounded weights and width is close to a low-degree polynomial. Proposition 3. Let f: [−1, 1] → R be a function computed by a depth-2 sigmoidal neural network of width n and weights bounded by DISPLAYFORM1 Now consider a depth-2 sigmoidal neural network on input x ∈ B d, where DISPLAYFORM2 It is given by a linear combination of sigmoidal activations applied to linear functions w 1, x, w 2, x,..., w n, x (or affine functions when we have biases), for w 1, w 2,..., w n ∈ R d and it computes a function F: DISPLAYFORM3 Proposition 4 below is a multivariate version of Proposition 3.Proposition 4. Let F: B d → R be a function computed by a depth-2 sigmoidal neural network with width n and bounded weights, that is, |a i | ≤ B and DISPLAYFORM4 Note that its proof crucially uses the fact that Proposition 2 guarantees a low-degree polynomial that approximates a sigmoid neuron everywhere in [−1, 1].A depth-k sigmoidal neural network can be thought of as a composition -a depth-2 sigmoidal neural network on top, whose each input variable is a sigmoid applied to a depth-(k − 2) sigmoidal neural network. In other words, it computes a function F: DISPLAYFORM5 where y = (y 1, y 2, . . ., y m) has each coordinate y j = σ(F j (x)), for 1 ≤ j ≤ m, such that each DISPLAYFORM6 Now we show an interesting consequence, namely, any constant-depth sigmoidal neural network with polynomial width and polynomially bounded weights can be well-approximated by a lowdegree multivariate polynomial. The bounds presented in Proposition 5 are not optimal but the qualitative statement is interesting in contrast with the depth separation . The growth of the degree of polynomial approximation is dependent on the widths of hidden layers and it is also the subtle reason why a depth separation is still possible (when the weights are bounded).Proposition 5. Let F: B d → R be a function computed by a depth-k sigmoidal neural network of width at most n in each layer and weights bounded by B, then DISPLAYFORM7 Note that when n and B are polynomial in d and the depth k is constant, then this low-degree polynomial approximation also has degree polynomial in d. DISPLAYFORM8 y ) cannot be approximated by any depth-2 neural network of polynomial size and (exponentially) bounded weights. Daniely shows this lower bound for a general neuron or activation function that includes sigmoids and ReLUs. Daniely then uses G(x, y) = g(x, y) = sin(πd 3 x, y) which, on the other hand, is approximable by a depth-3 ReLU neural network with polynomial size and polynomially bounded weights. This gives a separation between depth-2 and depth-3 ReLU neural networks w.r.t. uniform distribution over DISPLAYFORM9 Daniely's proof uses harmonic analysis on the unit sphere, and requires the uniform distribution on DISPLAYFORM10 We show a simple proof of separation between depth-2 and depth-3 sigmoidal neural networks that compute functions F: B d → R. Our proof works for a large class of distributions on B d but requires the weights to be polynomially bounded. The following lemma appears in BID4. Assumption 1 in BID5 and their version of this lemma for ReLU networks was used by BID3 in the proof of separation between the expressive power of depth-2 and depth-3 ReLU networks. Lemma 6. Let f: [−1, 1] → R be any L-Lipschitz function. Then there exists a function g: [−1, 1] → R computed by a depth-2 sigmoidal neural network such that DISPLAYFORM11 the width n as well as the weights are bounded by poly(L, 1/), and |f (t) − g(t)| ≤, for all t ∈ [−1, 1]. Now we are ready to show the separation between depth-2 and depth-3 sigmoidal neural networks. The main idea, similar to BID3, is to exhibit a function that is Lipschitz but far from any low-degree polynomial. The Lipschitz property helps in showing that our function can be wellapproximated by a depth-3 neural network of small size and small weights. However, being far from any low-degree polynomial, it cannot be approximated by any depth-2 neural network. By modifying the function to G(x) = sin(πN x 2), this lower bound with L ∞ -norm holds for any distribution over B d whose support contains a radial line segment of length at least 1/poly(d), by making N = poly(d), for a large enough polynomial. Remark: Given any distribution µ over B d whose probability density is at least 1/poly(d) on some small ball of radius 1/poly(d), the lower bound or inapproximability by any depth-2 sigmoidal neural network can be made to work with L 2 -norm (squared error), for a large enough N = poly(d).Proof. First, we will show that G(x) can be well-approximated by a depth-3 sigmoidal neural network of polynomial size and weights. The idea is similar to Daniely's construction for ReLU networks in BID3. By Lemma 6, there exists a function f: [−1, 1] → R computed by a depth-2 sigmoidal neural network of size and weights bounded by poly(d, 1/) such that DISPLAYFORM0 6, for all t ∈ [−1, 1]. Thus, we can compute x 2 i for each coordinate of x and add them up to get an -approximation to x 2 over B d. That is, there exists a function S: B d → R computed by a depth-2 sigmoidal neural network of size and weights bounded by poly(d, 1/) such that S(x) − x 2 ≤ /10d 5, for all x ∈ B d. Again, by Lemma 6, we can approximate sin(πd 3 t) over using f: [−1, 1] → R computed by another depth-2 sigmoidal neural network with size and weights bounded by poly(d, 1/) such that sin(πd 3 t) − f (t) ≤ /2, for all t ∈. Note that the composition of these two depth-2 neural networks f (N (x)) gives a depth-3 neural network as the output of the hidden layer of the bottom network can be fed into the top network as inputs. DISPLAYFORM1 using that f that approximates sin(πd 5 t) closely must also be 4d 5 -Lipschitz DISPLAYFORM2 Now we will show the lower bound. Consider any function F: B d → R computed by a depth-2 sigmoidal neural network whose weights are bounded by B = O(d 2) and width is n. Proposition 4 shows that there exists a d-variate polynomial P (x) of degree O B log(nB . Consider S = {t ∈ [a, a + l]: t = −1 + (i + 1/2)/N, for some integer i}. Then S contains at least N l − 2 points where sin(πN t) alternates as ±1. Any polynomial p of degree D cannot match the sign of sin(πN t) on all the points in S. Otherwise, by intermediate value theorem, p must have at least N l − 3 roots between the points of S, which means D ≥ N l − 3, a contradiction. Thus, there exists t 0 ∈ S such that p(t 0) and sin(πN t 0) have opposite signs. Since sin(πN t) = ±1, for any t ∈ S, the sign mismatch implies |sin(πN DISPLAYFORM3 DISPLAYFORM4 DISPLAYFORM5 An important remark on biases: Even though we handled the case of sigmoid neurons without biases, the proof technique carries over to the sigmoid neurons with biases σ( w, x + b). The idea is to consider a new DISPLAYFORM6 ) with x d+1 = 1, and consider the new weight vector w new = (w, b). Thus, w new, x new = w, x + b. The new input lies on a d-dimensional hyperplane slice of B d+1, so we need to look at the restriction of the input distribution µ to this slice. Most of the ideas in our proofs generalize without any technical modifications. We defer the details to the full version. In this section we show lower bounds under the L 2 -norm. The theorem below gives a technical condition on the class of densities µ on B d for which our lower bound holds. Let's give an example to illustrate that the condition on density is reasonable: Let K ⊂ B d be a convex set such that every point in K is at least r away from the boundary of B d (where r = 1/poly(d) is a parameter). Further assume that the probability mass of K is at least a constant and for every point in K the probability density is within a constant factor of the uniform density on K. Then our lower bound applies to µ.Theorem 9. Consider the function G: B d → R given by G(x) = sin(πN x 2). Let µ be any probability density over B d such that there exists a subset C ⊆ B d satisfying the following two conditions:• The r-interior of C defined as C = {x ∈ C : B(x, r) ⊆ C} contains at least γ fraction of the total probability mass for some γ > 0, i.e., C µ(x)dx ≥ γ.• For any affine line, the induced probability density on every segment of length at least r in the intersection ∩ C is (α, β)-uniform, i.e., it is at least α times and at most β times the uniform density on that segment. Let F: B d → R be any function computed by a depth-2 sigmoidal neural network with weights bounded by B and width n. Then for any 0 < δ αγ/3β and N (B/r 2) log(nB 2 /δ), the function F cannot δ-approximate G on B d under L 2 -norm (squared error) under the probability density µ.In particular, if α, β, γ are constants, B = poly(d), n = 2 d, and r = 1/poly(d), then it suffices to choose N = poly(d) for a sufficiently large degree polynomial. Proof. We show a lower bound on L 2 -error of approximating G(x) with any multivariate polynomial P: B d → R of degree D under the distribution given by µ on B d. For any fixed unit vector v, consider u ∈ B d−1 orthogonal to v and let u be the affine line going through u and parallel to the direction v given by u = {x = u + tv : t ∈ R}. DISPLAYFORM0 where IC(u, t) = 1, if u ∈ B d−1 and u + tv ∈ {u + t v ∈ C : t ∈ I} ⊆ C for some interval I of length at least r, and IC(u, t) = 0, otherwise DISPLAYFORM1 because for any x = u + tv ∈ C we have B(x, r) ⊆ C, therefore IC(u, t) = 1 DISPLAYFORM2 because for any line, the distribution induced by µ(x) along any line segment of length at least r in the intersection ∩ C is (α, β)-uniform, for any line DISPLAYFORM3 The last inequality is using the condition C µ(x)dx ≥ γ given in Theorem 9 and an adaptation of the following idea from Lemma 5 of BID3. For any fixed u and v, G(u + tv) = sin(πN ( u 2 + t 2)) and P (u + tv) is a polynomial of degree at most D in t. The function sin(πN ( u 2 +t 2)) alternates its sign as u 2 +t 2 takes values that are successive integer multiples of 1/N. Consider s = t 2 ∈ and divide into N disjoint segments using integer grid of step size 1/N. For any polynomial p(s) of degree at most D and any interval I ⊆ of length r D/N, there exists at least N r − D − 2 segments of length 1/N each on which sin(πN s) and p(s) do not change signs and have opposite signs. Now using (sin(πN s) − p(s)) 2 ≥ sin 2 (πN s), integrating we get that I (sin(πN s) − p(s)) 2 ds ≥ r/2. Extending this proof to t instead of s = t 2, using sin 2 (πN t 2)t ≤ sin 2 (πN t) for all t ∈, and incorporating the shift πN u 2, we can similarly show that I sin 2 (πN ( u 2 + t 2)) − P (u + tv)) 2 dt ≥ r/3. Summing up over multiple such intervals gives the final inequality. The L 2 separation between depth-2 and depth-3 neural networks under probability density µ now follows by taking a small enough δ, and combining the following ingredients (i) Proposition 4 says that any depth-2 sigmoid neural networks of width n = 2 d and weights bounded by B = poly(d) can be δ-approximated in L ∞ (and hence, also L 2) by a multivariate polynomials of degree DISPLAYFORM4 (ii) proof of Theorem 7 (initial part) says that G(x) can be δ-approximated in L ∞ (and hence, also L 2) by a depth-3 sigmoid neural network of width and size poly(d), but (iii) Theorem 9 says that, for N = poly(d) of large enough degree, G(x) cannot be 3δ-approximated in L 2 by any multivariate polynomial of degree D, and (iv) triangle inequality. Proof. Consider the degree-D approximation to σ(wt) given by the first D terms in its Chebyshev expansion. The error of this approximation for any t ∈ [−1, 1] is bounded by DISPLAYFORM0 using Proposition 1, |w| ≤ B, and D = O (B log (B/)). Proof. Let f be computed by a depth-2 sigmoidal neural network given by f (t) = n i=1 a i σ(w i t). Define a parameter = δ/nB. Proposition 2 guarantees polynomial p 1, p 2,..., p n of degree O (B log(B/)) such that |σ(w i t) − p i (t)| ≤, for all t ∈ [−1, 1]. Thus, the polynomial p(t) = n i=1 a i p i (t) has degree O (B log(B/)) = O B log(nB 2 /δ), and for any t ∈ [−1, 1], DISPLAYFORM0 Proof. Let F be computed by a depth-2 neural network given by DISPLAYFORM0 Define a parameter = δ/nB. Proposition 2 guarantees polynomial p 1, p 2,..., p n of degree O (B log(B/)) such that |σ(w i t) − p i (t)| ≤, for all t ∈ [−1, 1]. Consider the following polynomial P (x) = P (x 1, x 2, . . ., x d) = n i=1 a i p i (w i / w i, x). P (x) is a d-variate polynomial of degree O (B log(B/)) = O B log(nB 2 /δ) in each variable x 1, x 2,..., x d. For any DISPLAYFORM1 |σ(w i t i) − p i (t i)| using t i = w i / w i, x and |a i | ≤ B ≤ nB using |σ(w i t) − p i (t)| ≤, for all t ∈ [−1, 1] = δ. Proof. We prove this by induction on the depth k. By induction hypothesis each F j (x) can be 1 -approximated (in L ∞ -norm) by a d-variate polynomial Q j (x) of degree O (nB) k−2 log (k−2) (nB/ 1) in each variable. Thus, |F j (x) − Q j (x)| = 1, for any x ∈ B d and 1 ≤ j ≤ m. Because a sigmoid neuron is Lipschitz, DISPLAYFORM0 for any x ∈ B d and 1 ≤ j ≤ m. Since F j (x) is the output of a depth-(k−2) sigmoidal neural network of width at most n and weights at most B, we must have |F j (x)| ≤ nB, for all x ∈ B d. Thus, |Q j (x)| ≤ nB + 1 ≤ 2nB. By Proposition 2, there exists a polynomial q(t) of degree at most O (nB log(nB/ 2)) such that |σ(Q j (x)) − q(Q j (x))| ≤ 2, for all x ∈ B d and 1 ≤ j ≤ m. Consider q ∈ R m as q = (q(Q 1 (x)), q(Q 2 (x)),..., q(Q m (x))). Then, for any x ∈ B d, we have DISPLAYFORM1 Again by Proposition 2, there is a polynomial p of degree at most O (nB log(nB/)) such that |σ(w i, q) − p(w i, q)| ≤, for all x ∈ B d and 1 ≤ i ≤ n. This is because | w i, q | = O(nB).Let's define P (x) = n i=1 a i p(w i, q). Therefore, for any x ∈ B d, DISPLAYFORM2 if we use 1 = 2 = δ/3n 3/2 B 2 and = δ/3nB.P (x) is a d-variate polynomial of degree DISPLAYFORM3 in each variable.

depth-2-vs-3 separation for sigmoidal neural networks over general distributions

Represent the TLDR for finding the corresponding scientific article

To solve the gradient vanishing/exploding problems, we proprose an efficient parametrization of the transition matrix of RNN that loses no expressive power, converges faster and has good generalization.


Question: Represent the scientific article to find a short TLDR

Question: The smallest eigenvectors of the graph Laplacian are well-known to provide a succinct representation of the geometry of a weighted graph. In reinforcement learning (RL), where the weighted graph may be interpreted as the state transition process induced by a behavior policy acting on the environment, approximating the eigenvectors of the Laplacian provides a promising approach to state representation learning. However, existing methods for performing this approximation are ill-suited in general RL settings for two main reasons: First, they are computationally expensive, often requiring operations on large matrices. Second, these methods lack adequate justification beyond simple, tabular, finite-state settings. In this paper, we present a fully general and scalable method for approximating the eigenvectors of the Laplacian in a model-free RL context. We systematically evaluate our approach and empirically show that it generalizes beyond the tabular, finite-state setting. Even in tabular, finite-state settings, its ability to approximate the eigenvectors outperforms previous proposals. Finally, we show the potential benefits of using a Laplacian representation learned using our method in goal-achieving RL tasks, providing evidence that our technique can be used to significantly improve the performance of an RL agent. The performance of machine learning methods generally depends on the choice of data representation BID2. In reinforcement learning (RL), the choice of state representation may affect generalization , exploration , and speed of learning BID7. As a motivating example, consider goal-achieving tasks, a class of RL tasks which has recently received significant attention BID1 ). In such tasks, the agent's task is to achieve a certain configuration in state space; e.g. in FIG0 the environment is a two-room gridworld and the agent's task is to reach the red cell. A natural reward choice is the negative Euclidean (L2) distance from the goal (e.g., as used in Nachum et al. FORMULA4). The ability of an RL agent to quickly and successfully solve the task is thus heavily dependent on the representation of the states used to compute the L2 distance. Computing the distance on one-hot (i.e. tabular) representations of the states (equivalent to a sparse reward) is most closely aligned with the task's directive. However, such a representation can be disadvantageous for learning speed, as the agent receives the same reward signal for all non-goal cells. One may instead choose to compute the L2 distance on (x, y) representations of the grid cells. This allows the agent to receive a clear signal which encourages it to move to cells closer to the goal. Unfortunately, this representation is agnostic to the environment dynamics, and in cases where the agent's movement is obstructed (e.g. by a wall as in FIG0), this choice of reward is likely to cause premature convergence to sub-optimal policies unless sophisticated exploration strategies are used. The ideal reward structure would be defined on state representations whose distances roughly correspond to the ability of the agent to reach one state from another. Although there are many suitable such representations, in this paper, we focus on a specific approach based on the graph Laplacian, which is notable for this and several other desirable properties. For a symmetric weighted graph, the Laplacian is a symmetric matrix with a row and column for each vertex. The d smallest eigenvectors of the Laplacian provide an embedding of each vertex in R d which has been found to be especially useful in a variety of applications, such as graph visualization BID9, clustering , and more BID6.Naturally, the use of the Laplacian in RL has also attracted attention. In an RL setting, the vertices of the graph are given by the states of the environment. For a specific behavior policy, edges between states are weighted by the probability of transitioning from one state to the other (and vice-versa). Several previous works have proposed that approximating the eigenvectors of the graph Laplacian can be useful in RL. For example, shows that using the eigenvectors as basis functions can accelerate learning with policy iteration. Machado et al. (2017a; b) show that the eigenvectors can be used to construct options with exploratory behavior. The Laplacian eigenvectors are also a natural solution to the aforementioned reward-shaping problem. If we use a uniformly random behavior policy, the Laplacian state representations will be appropriately aware of the walls present in the gridworld and will induce an L2 distance as shown in FIG0 (right). This choice of representation accurately reflects the geometry of the problem, not only providing a strong learning signal at every state, but also avoiding spurious local optima. While the potential benefits of using Laplacian-based representations in RL are clear, current techniques for approximating or learning the representations are ill-suited for model-free RL. For one, current methods mostly require an eigendecomposition of a matrix. When this matrix is the actual Laplacian , the eigendecomposition can easily become prohibitively expensive. Even for methods which perform the eigendecomposition on a reduced matrix (a; b), the eigendecomposition step may be computationally expensive, and furthermore precludes the applicability of the method to stochastic or online settings, which are common in RL. Perhaps more crucially, the justification for many of these methods is made in the tabular setting. The applicability of these methods to more general settings is unclear. To resolve these limitations, we propose a computationally efficient approach to approximate the eigenvectors of the Laplacian with function approximation based on the spectral graph drawing objective, an objective whose optimum yields the desired eigenvector representations. We present the objective in a fully general RL setting and show how it may be stochastically optimized over minibatches of sampled experience. We empirically show that our method provides a better approximation to the Laplacian eigenvectors than previous proposals, especially when the raw representation is not tabular. We then apply our representation learning procedure to reward shaping in goal-achieving tasks, and show that our approach outperforms both sparse rewards and rewards based on L2 distance in the raw feature space. Results are shown under a set of gridworld maze environments and difficult continuous control navigation environments. We present the eigendecomposition framework in terms of general Hilbert spaces. By working with Hilbert spaces, we provide a unified treatment of the Laplacian and our method for approximating its eigenvectors BID5 -eigenfunctions in Hilbert spaces -regardless of the underlying space (discrete or continuous). To simplify the exposition, the reader may substitute the following simplified definitions:• The state space S is a finite enumerated set {1, . . ., |S|}.• The probability measure ρ is a probability distribution over S.• The Hilbert space H is R |S|, for which elements f ∈ H are |S| dimensional vectors representing functions f: S → R.• The inner product f, g H of two elements f, g ∈ H is a weighted dot product of the corresponding vectors, with weighting given by ρ; i.e. f, DISPLAYFORM0 • A linear operator is a mapping A: H → H corresponding to a weighted matrix multiplication; i.e. DISPLAYFORM1 • A self-adjoint linear operator A is one for which f, Ag H = Af, g H for all f, g ∈ H.This corresponds to A being a symmetric matrix. We now present the more general form of these definitions. Let S be a set, Σ be a σ-algebra, and ρ be a measure such that (S, Σ, ρ) constitutes a measure space. Consider the set of square-integrable real-valued functions L 2 (S, Σ, ρ) = {f : S → R s.t. S |f (u)| 2 dρ(u) < ∞}. When associated with the inner-product, DISPLAYFORM0 this set of functions forms a complete inner product Hilbert space BID8 ). The inner product gives rise to a notion of orthogonality: Functions f, g are orthogonal if f, g H = 0. It also induces a norm on the space: ||f || 2 = f, f H. We denote H = L 2 (S, Σ, ρ) and additionally restrict ρ to be a probability measure, i.e. S 1 dρ(u) = 1. To construct the graph Laplacian in this general setting, we consider linear operators D which are Hilbert-Schmidt integral operators BID3, expressable as, DISPLAYFORM0 where with a slight abuse of notation we also use D: S × S → R + to denote the kernel function. We assume that (i) the kernel function D satisfies D(u, v) = D(v, u) for all u, v ∈ S so that the operator D is self-adjoint; (ii) for each u ∈ S, D(u, v) is the Radon-Nikodym derivative (density function) from some probability measure to ρ, i.e. S D(u, v) dρ(v) = 1 for all u. With these assumptions, D is a compact, self-adjoint linear operator, and hence many of the spectral properties associated with standard symmetric matrices extend to D.The Laplacian L of D is defined as the linear operator on H given by, DISPLAYFORM1 The Laplacian may also be written as the linear operator I − D, where I is the identity operator. Any eigenfunction with associated eigenvalue λ of the Laplacian is an eigenfunction with eigenvalue 1 − λ for D, and vice-versa. Our goal is to find the first d eigenfunctions f 1,..., f d associated with the smallest d eigenvalues of L (subject to rotation of the basis). 1 The mapping φ: DISPLAYFORM2 ] then defines an embedding or representation of the space S. Spectral graph drawing BID9 provides an optimization perspective on finding the eigenvectors of the Laplacian. Suppose we have a large graph, composed of (possibly infinitely many) vertices with weighted edges representing pairwise (non-negative) affinities (denoted by D(u, v) ≥ 0 for vertices u and v). To visualize the graph, we would like to embed each vertex in a low dimensional space (e.g., R d in this work) so that pairwise distances in the low dimensional space are small for vertices with high affinity. Using our notation, the graph drawing objective is to find a set of orthonormal functions f 1,..., f d defined on the space S which minimize DISPLAYFORM0 The orthonormal constraints can be written as DISPLAYFORM1 The graph drawing objective may be expressed more succinctly in terms of the Laplacian: DISPLAYFORM2 The minimum value of FORMULA8 is the sum of the d smallest eigenvalues of L. Accordingly, the minimum is achieved when f 1,..., f d span the same subspace as the corresponding d eigenfunctions. In the next section, we will show that the graph drawing objective is amenable to stochastic optimization, thus providing a general, scalable approach to approximating the eigenfunctions of the Laplacian. In this section, we specify the meaning of the Laplacian in the RL setting (i.e., how to set ρ, D appropriately). We then elaborate on how to approximate the eigenfunctions of the Laplacian by optimizing the graph drawing objective via stochastic gradient descent on sampled states and pairs of states. In RL, an agent interacts with an environment by observing states and acting on the environment. We consider the standard MDP setting . Briefly, at time t the environment produces an observation s t ∈ S, which at time t = 0 is determined by a random sample from an environmentspecific initial distribution P 0. The agent's policy produces a probability distribution over possible actions π(a|s t) from which it samples a specific action a t ∈ A to act on the environment. The environment then yields a reward r t sampled from an environment-specific reward distribution function R(s t, a t), and transitions to a subsequent state s t+1 sampled from an environment-specific transition distribution function P (s t, a t). We consider defining the Laplacian with respect to a fixed behavior policy π. Then, the transition distributions P π (s t+1 |s t) form a Markov chain. We assume this Markov chain has a unique stationary distribution. We now introduce a choice of ρ and D for the Laplacian in the RL setting. We define ρ to be the stationary distribution of the Markov chain P π such that for any measurable U ⊂ S we have DISPLAYFORM0 As D(u, v) represents the pairwise affinity between two vertices u and v on the graph, it is natural to define D(u, v) in terms of the transition distribution. DISPLAYFORM1 is the density function from a probability measure to ρ for all u. We define DISPLAYFORM2 which satisfies these conditions 3. In other words, the affinity between states u and v is the average of the two-way transition probabilities: If S is finite then the first term in FORMULA11 is P π (s t+1 = v|s t = u)/ρ(v) and the second term is P π (s t+1 = u|s t = v)/ρ(u). Given this definition of the Laplacian, we now aim to learn the eigen-decomposition embedding φ. In the model-free RL context, we have access to states and pairs of states (or sequences of states) only via sampling; i.e. we may sample states u from ρ(u) and pairs of u, v from ρ(u)P π (v|u). This imposes several challenges on computing the eigendecomposition:• Enumerating the state space S may be intractable due to the large cardinality or continuity.• For arbitrary pairs of states (u, v), we do not have explicit access to D(u, v).• Enforcing exact orthonormality of f 1,..., f d may be intractable in innumerable state spaces. With our choices for ρ and D, the graph drawing objective (Eq. 2) is a good start for resolving these challenges because it can be expressed as an expectation (see Appendix C for the derivation): DISPLAYFORM0 Minimizing the objective with stochastic gradient descent is straightforward by sampling transition pairs (s t, s t+1) as (u, v) from the replay buffer. The difficult part is ensuring orthonormality of the functions. To tackle this issue, we first relax the orthonormality constraint to a soft constraint DISPLAYFORM1 Using standard properties of expectations, we rewrite the inequality as follows: DISPLAYFORM2 In practice, we transform this constraint into a penalty and solve the unconstrained minimization problem. The ing penalized graph drawing objective is DISPLAYFORM3 where β is the penalty weight (KKT multiplier). DISPLAYFORM4 may be learned using a neural network function approximator. We note thatG has a form which appears in many other representation learning objectives, being comprised of an attractive and a repulsive term. The attractive term minimizes the squared distance of embeddings of randomly sampled transitions experienced by the policy π, while the repulsive term repels the embeddings of states independently sampled from ρ. The repulsive term is especially interesting and we are unaware of anything similar to it in other representation learning objectives: It may be interpreted as orthogonalizing the embeddings of two randomly sampled states while regularizing their norm away from zero by noticing DISPLAYFORM5 4 RELATED WORK One of the main contributions of our work is a principled treatment of the Laplacian in a general RL setting. While several previous works have proposed the use of the Laplacian in RL (; a), they have focused on the simple, tabular setting. In contrast, we provide a framework for Laplacian representation learning that applies generally (i.e., when the state space is innumerable and may only be accessed via sampling).Our main is showing that the graph drawing objective may be used to stochastically optimize a representation module which approximates the Laplacian eigenfunctions. Although a large body of work exists regarding stochastic approximation of an eigendecomposition BID4 ), many of these approaches require storage of the entire eigendecomposition. This scales poorly and fails to satisfy the desiderata for model-free RL -a function approximator which yields arbitrary rows of the eigendecomposition. Some works have proposed extensions that avoid this requirement by use of Oja's rule . Originally defined within the Hebbian framework, recent work has applied the rule to kernelized PCA , and extending it to settings similar to ours is a potential avenue for future work. Other approaches to eigendecomposition that may eventually prove fruitful in the RL setting include , which proposes to scale to large datasets by subsampling representative subgraphs, and BID0, which provides some techniques to extend spectral clustering to out-of-sample points. In RL, Machado et al. (2017b) propose a method to approximate the Laplacian eigenvectors with functions approximators via an equivalence between proto-value functions and spectral decomposition of the successor representation . Importantly, they propose an approach for stochastically approximating the eigendecomposition when the state space is large. Unfortunately, their approach is only justified in the tabular setting and, as we show in our below, does not generalize beyond. Moreover, their eigenvectors are based on an explicit eigendecomposition of a constructed reduced matrix, and thus are not appropriate for online settings. Approaches more similar to ours optimize objectives similar to Eq. 2, but handle the orthonormality constraint differently. Shaham et al. FORMULA4 introduce a special-purpose orthonormalizing layer, which ensures orthonormality at the mini-batch level. Unfortunately, this does not ensure orthonormality over the entire dataset and requires large minibatches for stability. Furthermore, the orthonormalization process can be numerically unstable, and in our preliminary experiments we found that TensorFlow frequently crashed due to numerical errors from this sort of orthonormalization. Pfau et al. FORMULA4 turn the problem into an unconstrained optimization objective. However, in their chosen form, one cannot compute unbiased stochastic gradients. Moreover, their approach scales quadratically in the number of embedding dimensions. Our approach does not suffer from these issues. Finally, we note that our work provides a convincing application of Laplacian representations on difficult RL tasks, namely reward-shaping in continuous-control environments. Although previous works have presented interesting preliminary , their applications were either restricted to small discrete state spaces or focused on qualitative assessments of the learned options (a; b). We first evaluate the learned representations by how well they approximate the subspace spanned by the smallest eigenfunctions of the Laplacian. We use the following evaluation protocol: (i) Given an embedding φ: S → R d, we first find its principal d-dimensional orthonormal basis h 1,..., h d, onto which we project all embeddings in order to satisfy the orthonormality constraint of the graph drawing objective; (ii) the evaluation metric is then computed as the value of the graph drawing objective using the projected embeddings. In this subsection, we use finite state spaces, so step (i) can be performed by SVD. We used a FourRoom gridworld environment FIG1. We generate a dataset of experience by randomly sampling n transitions using a uniformly random policy with random initial state. We compare the embedding learned by our approximate graph drawing objective against methods proposed by Machado et al. (2017a; b). Machado et al. (2017a) find the first d eigenvectors of the Laplacian by eigen-decomposing a matrix formed by stacked transitions, while Machado et al. (2017b) eigen-decompose a matrix formed by stacked learned successor representations. We evaluate the methods with three different raw state representations of the gridworld: (i) one-hot vectors ("index"), (ii) (x, y) coordinates ("position") and (iii) top-down pixel representation ("image").We present the of our evaluations in FIG2. Our method outperforms the previous methods with all three raw representations. Both of the previous methods were justified in the tabular setting, however, surprisingly, they underperform our method even with the tabular representation. Moreover, our method performs well even when the number of training samples is small. We now move on to demonstrating the power of our learned representations to improve the performance of an RL agent. We focus on a family of tasks -goal-achieving tasks -in which the agent is rewarded for reaching a certain state. We show that in such settings our learned representations are well-suited for reward shaping. Goal-achieving tasks and reward shaping. A goal-achieving task is defined by an environment with transition dynamics but no reward, together with a goal vector z g ∈ Z, where Z is the goal space. We assume that there is a known predefined function h: S → Z that maps any state s ∈ S to a goal vector h(s) ∈ Z. The learning objective is to train a policy that controls the agent to get to some state s such that h(s) − z g ≤. For example the goal space may be the same as the state space with Z = S and h(s) = s being the identity mapping, in which case the target is a state vector. More generally the goal space can be a subspace of the state space. For example, in control tasks a state vector may contain both position and velocity information while a goal vector may just be a specific position. See Plappert et al. FORMULA4 for an extensive discussion and additional examples. A reward function needs to be defined in order to apply reinforcement learning to train an agent that can perform a goal achieving task. Two typical ways of defining a reward function for this family of tasks are (i) the sparse reward: Reward shaping with learned representations. We expect that distance based reward shaping with our learned representations can speed up learning compared to sparse reward while avoiding the bias in the raw feature space. More specifically, we define the reward based on distance in a learned latent space. If the goal space is the same as the state space, i.e. S = Z, the reward function can be defined as r t = − φ(s t+1) − φ(z g). If S = Z we propose two options: (i) The first is to learn an embedding φ: Z → R d of the goal space and define r t = − φ(h(s t+1)) − φ(z g). (ii) The second options is to learn an an embedding φ: S → R d of the state space and define DISPLAYFORM0 DISPLAYFORM1, where h −1 (z) is defined as picking arbitrary state s (may not be unique) that achieves h(s) = z. We experiment with both options when S = Z. We experiment with the gridworld environments with (x, y) coordinates as the observation. We evaluate on three different mazes: OneRoom, TwoRooms and HardMaze, as shown in the top row of FIG4. The red grids are the goals and the heatmap shows the distances from each grid to the goal in the learned Laplacian embedding space. We can qualitatively see that the learned rewards are well-suited to the task and appropriately reflect the environment dynamics, especially in TwoRoom and HardMaze where the raw feature space is very ill-suited. These representations are learned according to our method using a uniformly random behavior policy. Then we define the shaped reward as a half-half mix of the L2 distance in the learned latent space and the sparse reward. We found this mix to be advantageous, as the L2 distance on its own does not provide enough difference between the reward of the goal state and rewards of the states near the goal. When the L2 distance between the representations of the goal state and adjacent states is small the Q-function can fail to provide a significant signal to actually reach the goal state (rather than a state that is just close to the goal). Thus, to better align the shaped reward with the task directive, we use a half-half mix, which clearly draws a boundary between the goal state and its adjacent states (as the sparse reward does) while retaining the structure of the distance-shaped reward. We plot the learning performance of an agent trained according to this learned reward in FIG4. All plots are based on 5 different random seeds. We compare against (i) sparse: the sparse reward, (ii) l2: the shaped reward based on the L2 distance in the raw (x, y) feature space, (iii) rawmix: the mixture of (i) and (ii). Our mixture of shaped reward based on learning representations and the sparse reward is labelled as "mix" in the plots. We observe that in the OneRoom environment all shaped reward functions significantly outperform the sparse reward, which indicates that in goal-achieving tasks properly shaped reward can accelerate learning of the policy, justifying our motivation of applying learned representations for reward shaping. In TwoRoom and HardMaze environments when the raw feature space cannot reflect an accurate distance, our Laplacian-based shaped reward learned using the graph drawing objective ("mix") significantly outperforms all other reward settings. To further verify the benefit of our learned representations in reward shaping, we also experiment with continuous control navigation tasks. These tasks are much harder to solve than the gridworld tasks because the agent must simultaneously learn to control itself and navigate to the goal. We use Mujoco to create 3D mazes and learn to control two types of agents, PointMass and Ant, to navigate to a certain area in the maze, as shown in FIG5. Unlike the gridworld environments the (x, y) goal space is distinct from the state space, so we apply our two introduced methods to align the spaces: (i) learning φ to only embed the (x, y) coordinates of the state (mix) or (ii) learning φ to embed the full state (fullmix). We run experiments with both methods. As shown in FIG5 both "mix" and "fullmix" outperform all other methods, which further justifies the benefits of using our learned representations for reward shaping. It is interesting to see that both embedding the goal space and embedding the state space still provide a significant advantage even if neither of them is a perfect solution. For goal space embedding, part of the state vector (e.g. velocities) is ignored so the learned embedding may not be able to capture the full structure of the environment dynamics. For state space embedding, constructing the state vector from the goal vector makes achieving the goal more challenging since there is a larger set of states (e.g. with different velocities) that achieve the goal but the shaped reward encourage the policy to reach only one of them. Having a better way to align the two spaces would be an interesting future direction. We have presented an approach to learning a Laplacian-based state representation in RL settings. Our approach is both general -being applicable to any state space regardless of cardinality -and scalable -relying only on the ability to sample mini-batches of states and pairs of states. We have further provided an application of our method to reward shaping in both discrete spaces and continuous-control settings. With our scalable and general approach, many more potential applications of Laplacian-based representations are now within reach, and we encourage future work to continue investigating this promising direction. A EXISTENCE OF SMALLEST EIGENVALUES OF THE LAPLACIAN.Since the Hilbert space H may have infinitely many dimensions we need to make sure that the smallest d eigenvalues of the Laplacian operator is well defined. Since L = I − D if λ is an eigenvalue of D then 1 − λ is an eigenvalue of L. So we turn to discuss the existence of the largest d eigenvalues of D. According to our definition D is a compact self-adjoint linear operator on H. So it has the following properties according to the spectral theorem:• D has either (i) a finite set of eigenvalues or (ii) countably many eigenvalues {λ 1, λ 2, ...} and λ n → 0 if there are infinitely many. All eigenvalues are real.• Any eigenvalue λ satisfies − D ≤ λ ≤ D where · is the operator norm. If the operator D has a finite set of n eigenvalues its largest d eigenvalues exist when d is smaller than n. If D has a infinite but countable set of eigenvalues we first characterize what the eigenvalues look like: DISPLAYFORM0 Recall that the operator norm is defined as DISPLAYFORM1 Define q u be the probability measure such that DISPLAYFORM2 and DISPLAYFORM3 which hold for any f ∈ H. Hence D ≤ 1.So the absolute values of the eigenvalues of D can be written as a non-increasing sequence which converges to 0 with the largest eigenvalue to be 1. If d is smaller than the number of positive eigenvalues of D then the largest d eigenvalues are guaranteed to exist. Note that this condition for d is stricter than the condition when D has finitely many eigenvalues. We conjecture that this restriction is due to an artifact of the analysis and in practice using any value of d would be valid when H has infinite dimensions. To introduce a more general definition of D, we first introduce a generalized discounted transition distribution P π λ defined by DISPLAYFORM0 where λ ∈ is a discount factor, with λ = 0 corresponding to the one-step transition distribution P π 0 = P π. Notice that P π λ (v|u) can be also written as P π λ (v|u) = E τ ∼q λ [P π (s t+τ = v|s t = u)] where q λ (τ) = λ τ −1 − λ τ. So sampling from P π λ (v|u) can be done by first sampling τ ∼ q λ then rolling out the Markov chain for τ steps starting from u. Note that for readability we state the definition of P π λ in terms of discrete probability distributions but in general P π λ (·|u) are defined as a probability measure by stating the discounted sum for any measurable set of states U ∈ Σ, U ⊂ S instead of a single state v. Also notice that when λ > 0 sampling v from P π λ (v|u) required rolling out more than one steps from u (and can be arbitrarily long). Given that the replay buffer contains finite length (say T) trajectories sampling exactly from the defined distribution is impossible. In practice, after sampling u = s t in a trajectory and τ from q λ (τ) we discard this sample if t + τ > T.With the discounted transition, distributions now the generalized D is defined as DISPLAYFORM1 We assume that P π λ (·|u) is absolutely continuous to ρ for any u so that the Radon Nikodym derivatives are well defined. This assumption is mild since it is saying that for any state v that is reachable from some state u under P π we have a positive probability to sample it from ρ, i.e. the behavior policy π is able to explore the whole state space (not necessarily efficiently).Proof of D(u, ·) being a density of some probability measure with respect to ρ. We need to show that S D(u, v) dρ(v) = 1. Let f (·|u) be the density function of P First notice that if ρ is the stationary distribution of P π it is also the stationary distribution of DISPLAYFORM2 (Property of the stationary distribution.)which means that g is the density function of ρ with respect to ρ. So g(u) = 1 holds for all u. (For simplicity we ignore the statement of "almost surely" throughout the paper.)Discussion of finite time horizon. Because proving D(u, ·) to be a density requires the fact that ρ is the stationary distribution of P π, the astute reader may suspect that sampling from the replay buffer will differ from the stationary distribution when the initial state distribution is highly concentrated, the mixing rate is slow, and the time horizon is short. In this case, one can adjust the definition of the transition probabilities to better reflect what is happening in practice: Define a new transition distribution by adding a small probability to "reset":P π (·|u) = (1 − δ)P π (·|u) + δP 0. This introduces a randomized termination to approximate termination of trajectories (e.g,. due to time limit) without adding dependencies on t (to retain the Markov property). Then, ρ and D can be defined in the same way with respect toP π. Now the replay buffer can be viewed as rolling out a single long trajectory withP π so that sampling from the replay buffer approximates sampling from the stationary distribution. Note that under the new definition of D, minimizing the graph drawing objective requires sampling state pairs that may span over the "reset" transition. In practice, we ignore these pairs as we do not want to view "resets" as edges in RL. When δ (e.g. 1/T) is small, the chance of sampling these "reset" pairs is very small, so our adjusted definition still approximately reflects what is being done in practice. Hyperparameters For representation learning we use d = 20. In the definition of D we use the discounted multi-step transitions with λ = 0.9. For the approximate graph drawing objective we use β = 5.0 and δ jk = 0.05 (instead of 1) if j = k otherwise 0 to control the scale of L2 distances. We pretrain the representations for 30000 steps (This number of steps is not optimized and we observe that the training converges much earlier) by Adam with batch size 128 and learning rate 0.001. For policy training, we use the vanilla DQN with a online network and a target network both representing the Q-function. The policy used for testing is to select the action with the highest Q-value according to the online network at each state. The online network is trained to minimize the Bellman error by sampling transitions from the replay buffer. The target network is updated every 50 steps with a mixing rate of 0.05 (of the current online network with 0.95 of the previous target network). Epsilon greedy with = 0.2 is used for exploration. Reward discount is 0.98. The policy is trained with Adam optimizer with learning rate 0.001. For both representation mapping and Q-functions we use a fully connected network (parameter not shared) with 3 hidden layers and 256 units in each layer. All activation functions are relu. The PointMass agent has a 6 dimensional state space a 2 dimensional action space. The Ant agent has a 29 dimensional state space and a 8 dimensional action space. The success criteria is set as reaching an L2 ball centered around a specific (x, y) position with the radius as 10% of the total size of the maze, as shown in FIG5. Each episode has a length of 300.Hyperparameters For representation learning we use d = 20. In the definition of D we use the discounted multi-step transitions with λ = 0.99 for PointMass and λ = 0.999 for Ant. For the approximate graph drawing objective we use β = 2.0 and δ jk = 0.1 (instead of 1) if j = k otherwise 0 to control the scale of L2 distances. We pretrain the representations for 50000 steps by Adam with batch size 128 and learning rate 0.001 for PointMass and 0.0001 for Ant. For policy training, we use the vanilla DDPG BID10 with a online network and a target network. Each network contains two sub-networks: an actor network representing the policy and a critic network representing the Q-function. The online critic network is trained to minimize the Bellman error and the online actor network is trained to maximize the Q-value achieved by the policy. The target network is updated every 1 step with a mixing rate of 0.001. For exploration we follow the Ornstein-Uhlenbeck process as described in the original DDPG paper BID10. Reward discount is 0.995. The policy is trained with Adam optimizer with batch size 100, actor learning rate 0.0001 and critic learning rate 0.001 for PointMass and 0.0001 for Ant. For representation mapping we use a fully connected network (parameter not shared) with 3 hidden layers and 256 units in each layer. Both actor network and critic network have 2 hidden layers with units. All activation functions are relu. Online learning of representations We also present of learning the representations online instead of pretraining-and-fix and observe equivalent performance, as shown in FIG8, suggesting that our method may be successfully used in online settings. For online training the agent moves faster in the maze during policy learning so we anneal the λ in D from its inital value to 0.95 towards the end of training with linear decay. The reason that online training provides no benefit is that our randomized starting position setting enables efficient exploration even with just random walk policies. Investigating the benefit of online training in exploration-hard tasks would be an interesting future direction.

Represent the short summary for finding the corresponding scientific article (long).

We propose a scalable method to approximate the eigenvectors of the Laplacian in the reinforcement learning context and we show that the learned representations can improve the performance of an RL agent.

We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.