Question: Represent the article for finding a single-sentence TLDR

Question: We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform. Different from graph Fourier transform, graph wavelet transform can be obtained via a fast algorithm without requiring matrix eigendecomposition with high computational cost. Moreover, graph wavelets are sparse and localized in vertex domain, offering high efficiency and good interpretability for graph convolution. The proposed GWNN significantly outperforms previous spectral graph CNNs in the task of graph-based semi-supervised classification on three benchmark datasets: Cora, Citeseer and Pubmed. Convolutional neural networks (CNNs) BID15 have been successfully used in many machine learning problems, such as image classification BID10 and speech recognition BID11, where there is an underlying Euclidean structure. The success of CNNs lies in their ability to leverage the statistical properties of Euclidean data, e.g., translation invariance. However, in many research areas, data are naturally located in a non-Euclidean space, with graph or network being one typical case. The non-Euclidean nature of graph is the main obstacle or challenge when we attempt to generalize CNNs to graph. For example, convolution is not well defined in graph, due to that the size of neighborhood for each node varies dramatically.Existing methods attempting to generalize CNNs to graph data fall into two categories, spatial methods and spectral methods, according to the way that convolution is defined. Spatial methods define convolution directly on the vertex domain, following the practice of the conventional CNN. For each vertex, convolution is defined as a weighted average function over all vertices located in its neighborhood, with the weighting function characterizing the influence exerting to the target vertex by its neighbors. The main challenge is to define a convolution operator that can handle neighborhood with different sizes and maintain the weight sharing property of CNN. Although spatial methods gain some initial success and offer us a flexible framework to generalize CNNs to graph, it is still elusive to determine appropriate neighborhood. Spectral methods define convolution via graph Fourier transform and convolution theorem. Spectral methods leverage graph Fourier transform to convert signals defined in vertex domain into spectral domain, e.g., the space spanned by the eigenvectors of the graph Laplacian matrix, and then filter is defined in spectral domain, maintaining the weight sharing property of CNN. As the pioneering work of spectral methods, spectral CNN BID3 exploited graph data with the graph Fourier transform to implement convolution operator using convolution theorem. Some subsequent works make spectral methods spectrum-free BID4 BID14 BID12, achieving locality in spatial domain and avoiding high computational cost of the eigendecomposition of Laplacian matrix. In this paper, we present graph wavelet neural network to implement efficient convolution on graph data. We take graph wavelets instead of the eigenvectors of graph Laplacian as a set of bases, and define the convolution operator via wavelet transform and convolution theorem. Graph wavelet neural network distinguishes itself from spectral CNN by its three desirable properties: Graph wavelets can be obtained via a fast algorithm without requiring the eigendecomposition of Laplacian matrix, and thus is efficient; Graph wavelets are sparse, while eigenvectors of Laplacian matrix are dense. As a , graph wavelet transform is much more efficient than graph Fourier transform; Graph wavelets are localized in vertex domain, reflecting the information diffusion centered at each node BID27. This property eases the understanding of graph convolution defined by graph wavelets. We develop an efficient implementation of the proposed graph wavelet neural network. Convolution in conventional CNN learns an individual convolution kernel for each pair of input feature and output feature, causing a huge number of parameters especially when the number of features is high. We detach the feature transformation from convolution and learn a sole convolution kernel among all features, substantially reducing the number of parameters. Finally, we validate the effectiveness of the proposed graph wavelet neural network by applying it to graph-based semi-supervised classification. Experimental demonstrate that our method consistently outperforms previous spectral CNNs on three benchmark datasets, i.e., Cora, Citeseer, and Pubmed.2 OUR METHOD 2.1 PRELIMINARY Let G = {V, E, A} be an undirected graph, where V is the set of nodes with |V| = n, E is the set of edges, and A is adjacency matrix with A i,j = A j,i to define the connection between node i and node j. The graph Laplacian matrix L is defined as L = D −A where D is a diagonal degree matrix with D i,i = j A i,j, and the normalized Laplacian matrix is L = I n − D −1/2 AD −1/2 where I n is the identity matrix. Since L is a real symmetric matrix, it has a complete set of orthonormal eigenvectors U = (u 1, u 2, ..., u n), known as Laplacian eigenvectors. These eigenvectors have associated real, non-negative eigenvalues {λ l} n l=1, identified as the frequencies of graph. Eigenvectors associated with smaller eigenvalues carry slow varying signals, indicating that connected nodes share similar values. In contrast, eigenvectors associated with larger eigenvalues carry faster varying signals across connected nodes. Taking the eigenvectors of normalized Laplacian matrix as a set of bases, graph Fourier transform of a signal x ∈ R n on graph G is defined asx = U x, and the inverse graph Fourier transform is x = Ux BID24. Graph Fourier transform, according to convolution theorem, offers us a way to define the graph convolution operator, denoted as * G. Denoting with y the convolution kernel, * G is defined as DISPLAYFORM0 where is the element-wise Hadamard product. Replacing the vector U y by a diagonal matrix g θ, then Hadamard product can be written in the form of matrix multiplication. Filtering the signal x by the filter g θ, we can write Equation as U g θ U x. However, there are some limitations when using Fourier transform to implement graph convolution: Eigendecomposition of Laplacian matrix to obtain Fourier basis U is of high computational cost with O(n 3); Graph Fourier transform is inefficient, since it involves the multiplication between a dense matrix U and the signal x; Graph convolution defined through Fourier transform is not localized in vertex domain, i.e., the influence to the signal on one node is not localized in its neighborhood. To address these limitations, ChebyNet BID4 restricts convolution kernel g θ to a polynomial expansion DISPLAYFORM1 where K is a hyper-parameter to determine the range of node neighborhoods via the shortest path distance, θ ∈ R K is a vector of polynomial coefficients, and Λ =diag({λ l} n l=1 ). However, such a polynomial approximation limits the flexibility to define appropriate convolution on graph, i.e., with a smaller K, it's hard to approximate the diagonal matrix g θ with n free parameters. While with a larger K, locality is no longer guaranteed. Different from ChebyNet, we address the aforementioned three limitations through replacing graph Fourier transform with graph wavelet transform. Similar to graph Fourier transform, graph wavelet transform projects graph signal from vertex domain into spectral domain. Graph wavelet transform employs a set of wavelets as bases, defined as ψ s = (ψ s1, ψ s2, ..., ψ sn), where each wavelet ψ si corresponds to a signal on graph diffused away from node i and s is a scaling parameter. Mathematically, ψ si can be written as DISPLAYFORM0 where U is Laplacian eigenvectors, G s =diag g(sλ 1),..., g(sλ n) is a scaling matrix and g(sλ i) = e λis.Using graph wavelets as bases, graph wavelet transform of a signal x on graph is defined asx = ψ −1 s x and the inverse graph wavelet transform is x = ψ sx. Note that ψ −1 s can be obtained by simply replacing the g(sλ i) in ψ s with g(−sλ i) corresponding to a heat kernel BID6. Replacing the graph Fourier transform in Equation FORMULA0 with graph wavelet transform, we obtain the graph convolution as DISPLAYFORM1 Compared to graph Fourier transform, graph wavelet transform has the following benefits when being used to define graph convolution: 2. High spareness: the matrix ψ s and ψ −1 s are both sparse for real world networks, given that these networks are usually sparse. Therefore, graph wavelet transform is much more computationally efficient than graph Fourier transform. For example, in the Cora dataset, more than 97% elements in ψ −1 s are zero while only less than 1% elements in U are zero TAB3. 3. Localized convolution: each wavelet corresponds to a signal on graph diffused away from a centered node, highly localized in vertex domain. As a , the graph convolution defined in Equation FORMULA3 is localized in vertex domain. We show the localization property of graph convolution in Appendix A. It is the localization property that explains why graph wavelet transform outperforms Fourier transform in defining graph convolution and the associated tasks like graph-based semisupervised learning. 4. Flexible neighborhood: graph wavelets are more flexible to adjust node's neighborhoods. Different from previous methods which constrain neighborhoods by the discrete shortest path distance, our method leverages a continuous manner, i.e., varying the scaling parameter s. A small value of s generally corresponds to a smaller neighborhood. FIG0 shows two wavelet bases at different scale on an example network, depicted using GSP toolbox BID22. Replacing Fourier transform with wavelet transform, graph wavelet neural network (GWNN) is a multi-layer convolutional neural network. The structure of the m-th layer is DISPLAYFORM0 where ψ s is wavelet bases, ψ −1 s is the graph wavelet transform matrix at scale s which projects signal in vertex domain into spectral domain, X m [:,i] with dimensions n × 1 is the i-th column of X m, F m i,j is a diagonal filter matrix learned in spectral domain, and h is a non-linear activation function. This layer transforms an input tensor X m with dimensions n × p into an output tensor X m+1 with dimensions n × q. In this paper, we consider a two-layer GWNN for semi-supervised node classification on graph. The formulation of our model is DISPLAYFORM1 second layer: DISPLAYFORM2 where c is the number of classes in node classification, Z of dimensions n × c is the prediction . The loss function is the cross-entropy error over all labeled examples: DISPLAYFORM3 where y L is the labeled node set, Y li = 1 if the label of node l is i, and Y li = 0 otherwise. The weights F are trained using gradient descent. In Equation FORMULA4, the parameter complexity of each layer is O(n × p × q), where n is the number of nodes, p is the number of features of each vertex in current layer, and q is the number of features of each vertex in next layer. Conventional CNN methods learn convolution kernel for each pair of input feature and output feature. This in a huge number of parameters and generally requires huge training data for parameter learning. This is prohibited for graph-based semi-supervised learning. To combat this issue, we detach the feature transformation from graph convolution. Each layer in GWNN is divided into two components: feature transformation and graph convolution. Spectially, we have feature transformation: DISPLAYFORM0 graph convolution: DISPLAYFORM1 where W ∈ R p×q is the parameter matrix for feature transformation, X m with dimensions n × q is the feature matrix after feature transformation, F m is the diagonal matrix for graph convolution kernel, and h is a non-linear activation function. After detaching feature transformation from graph convolution, the parameter complexity is reduced from O(n × p × q) to O(n + p × q). The reduction of parameters is particularly valuable fro graphbased semi-supervised learning where labels are quite limited. Graph convolutional neural networks on graphs. The success of CNNs when dealing with images, videos, and speeches motivates researchers to design graph convolutional neural network on graphs. The key of generalizing CNNs to graphs is defining convolution operator on graphs. Existing methods are classified into two categories, i.e., spectral methods and spatial methods. Spectral methods define convolution via convolution theorem. Spectral CNN BID3 is the first attempt at implementing CNNs on graphs, leveraging graph Fourier transform and defining convolution kernel in spectral domain. BID1 developed a local spectral CNN approach based on the graph Windowed Fourier Transform. BID4 introduced a Chebyshev polynomial parametrization for spectral filter, offering us a fast localized spectral filtering method. BID14 provided a simplified version of ChebyNet, gaining success in graph-based semi-supervised learning task. BID12 represented images as signals on graph and learned their transformation invariant representations. They used Chebyshev approximations to implement graph convolution, avoiding matrix eigendecomposition. BID16 used rational functions instead of polynomials and created anisotropic spectral filters on manifolds. Spatial methods define convolution as a weighted average function over neighborhood of target vertex. GraphSAGE takes one-hop neighbors as neighborhoods and defines the weighting function as various aggregators over neighborhood BID8. Graph attention network (GAT) proposes to learn the weighting function via self-attention mechanism BID28. MoNet offers us a general framework for design spatial methods, taking convolution as the weighted average of multiple weighting functions defined over neighborhood. Some works devote to making graph convolutional networks more powerful. BID20 alternated convolutions on vertices and edges, generalizing GAT and leading to better performance. GraphsGAN BID5 generalizes GANs to graph, and generates fake samples in low-density areas between subgraphs to improve the performance on graph-based semi-supervised learning. Graph wavelets. presented a lifting scheme, a simple construction of wavelets that can be adapted to graphs without learning process. BID9 proposed a method to construct wavelet transform on graphs. Moreover, they designed an efficient way to bypass the eigendecomposition of the Laplacian and approximated wavelets with Chebyshev polynomials. BID27 leveraged graph wavelets for multi-scale community mining by modulating a scaling parameter. Owing to the property of describing information diffusion, BID6 learned structural node embeddings via wavelets. All these works prove that graph wavelets are not only local and sparse but also valuable for signal processiong on graph. To evaluate the proposed GWNN, we apply GWNN on semi-supervised node classification, and conduct experiments on three benchmark datasets, namely, Cora, Citeseer and Pubmed BID23. In the three citation network datasets, nodes represent documents and edges are citation links. Details of these datasets are demonstrated in TAB0. Here, the label rate denotes the proportion of labeled nodes used for training. Following the experimental setup of GCN , we fetch 20 labeled nodes per class in each dataset to train the model. We compare with several traditional semi-supervised learning methods, including label propagation (LP) BID31, semi-supervised embedding (SemiEmb) BID29, manifold regularization (ManiReg) BID0, graph embeddings (DeepWalk) BID21, iterative classification algorithm (ICA) BID17 and Planetoid BID30.Furthermore, along with the development of deep learning on graph, graph convolutional networks are proved to be effective in semi-supervised learning. Since our method is a spectral method based on convolution theorem, we compare it with the Spectral CNN BID3. ChebyNet BID4 and GCN , two variants of the Spectral CNN, are also included as our baselines. Considering spatial methods, we take MoNet as our baseline, which also depends on Laplacian matrix. We train a two-layer graph wavelet neural network with 16 hidden units, and prediction accuracy is evaluated on a test set of 1000 labeled samples. The partition of datasets is the same as GCN BID14 with an additional validation set of 500 labeled samples to determine hyper-parameters. Weights are initialized following BID7. We adopt the Adam optimizer for parameter optimization with an initial learning rate lr = 0.01. For computational efficiency, we set the elements of ψ s and ψ −1 s smaller than a threshold t to 0. We find the optimal hyper-parameters s and t through grid search, and the detailed discussion about the two hyperparameters is introduced in Appendix B. For Cora, s = 1.0 and t = 1e − 4. For Citeseer, s = 0.7 and t = 1e − 5. For Pubmed, s = 0.5 and t = 1e − 7. To avoid overfitting, dropout BID25 is applied. Meanwhile, we terminate the training if the validation loss does not decrease for 100 consecutive epochs. Since the number of parameters for the undetached version of GWNN is O(n × p × q), we can hardly implement this version in the case of networks with a large number n of nodes and a huge number p of input features. Here, we validate the effectiveness of detaching feature transformation form convolution on ChebyNet (introduced in Section 2.2), whose parameter complexity is O(K × p × q). For ChebyNet of detaching feature transformation from graph convolution, the number of parameters is reduced to O(K + p × q). TAB1 shows the performance and the number of parameters on three datasets. Here, the reported performance is the optimal performance varying the order K = 2, 3, 4. As demonstrated in TAB1, with fewer parameters, we improve the accuracy on Pubmed by a large margin. This is due to that the label rate of Pubmed is only 0.003. By detaching feature transformation from convolution, the parameter complexity is significantly reduced, alleviating overfitting in semi-supervised learning and thus remarkably improving prediction accuracy. On Citeseer, there is a little drop on the accuracy. One possible explanation is that reducing the number of parameters may restrict the modeling capacity to some degree. We now validate the effectiveness of GWNN with detaching technique on node classification. Experimental are reported in TAB2. GWNN improves the classification accuracy on all the three datasets. In particular, replacing Fourier transform with wavelet transform, the proposed GWNN is comfortably ahead of Spectral CNN, achieving 10% improvement on Cora and Citeseer, and 5% improvement on Pubmed. The large improvement could be explained from two perspectives: Convolution in Spectral CNN is non-local in vertex domain, and thus the range of feature diffusion is not restricted to neighboring nodes; The scaling parameter s of wavelet transform is flexible to adjust the diffusion range to suit different applications and different networks. GWNN consistently outperforms ChebyNet, since it has enough degree of freedom to learn the convolution kernel, while ChebyNet is a kind of approximation with limited degree of freedom. Furthermore, our GWNN also performs better than GCN and MoNet, reflecting that it is promising to design appropriate bases for spectral methods to achieve good performance. Besides the improvement on prediction accuracy, wavelet transform with localized and sparse transform matrix holds sparsity in both spatial domain and spectral domain. Here, we take Cora as an example to illustrate the sparsity of graph wavelet transform. The sparsity of transform matrix. There are 2,708 nodes in Cora. Thus, the wavelet transform matrix ψ −1 s and the Fourier transform matrix U both belong to R 2,708×2,708. The first two rows in TAB3 demonstrate that ψ −1 s is much sparser than U. Sparse wavelets not only accelerate the computation, but also well capture the neighboring topology centered at each node. The sparsity of projected signal. As mentioned above, each node in Cora represents a document and has a sparse bag-of-words feature. The input feature X ∈ R n×p is a binary matrix, and X [i,j] = 1 when the i-th document contains the j-th word in the bag of words, it equals 0 otherwise. Here, X [:,j] denotes the j-th column of X, and each column represents the feature vector of a word. Considering a specific signal X [:,984], we project the spatial signal into spectral domain, and get its projected vector. Here, p = ψ −1 s X [:,984] denotes the projected vector via wavelet transform, q = U X [:,984] denotes the projected vector via Fourier transform, and p, q ∈ R 2,708. The last row in TAB3 lists the numbers of non-zero elements in p and q. As shown in TAB3, with wavelet transform, the projected signal is much sparser. Compare with graph convolution network using Fourier transform, GWNN provides good interpretability. Here, we show the interpretability with specific examples in Cora. Each feature, i.e. word in the bag of words, has a projected vector, and each element in this vector is associated with a spectral wavelet basis. Here, each basis is centered at a node, corresponding to a document. The value can be regarded as the relation between the word and the document. Thus, each value in p can be interpreted as the relation between W ord 984 and a document. In order to elaborate the interpretability of wavelet transform, we analyze the projected values of different feature as following. Considering two features W ord 984 and W ord 1177, we select the top-10 active bases, which have the 10 largest projected values of each feature. As illustrated in Figure 2, for clarity, we magnify the local structure of corresponding nodes and marked them with bold rims. The central network in each subgraph denotes the dataset Cora, each node represents a document, and 7 different colors represent 7 classes. These nodes are clustered by OpenOrd BID18 based on the adjacency matrix. Figure 2a shows the top-10 active bases of W ord 984. In Cora, this word only appears 8 times, and all the documents containing W ord 984 belong to the class " Case-Based ". Consistently, all top-10 nodes activated by W ord 984 are concentrated and belong to the class " Case-Based ". And, the frequencies of W ord 1177 appearing in different classes are similar, indicating that W ord 1177 is a universal word. In concordance with our expectation, the top-10 active bases of W ord 1177 are discrete and belong to different classes in Figure 2b. DISPLAYFORM0 Figure 2: Top-10 active bases of two words in Cora. The central network of each subgraph represents the dataset Cora, which is split into 7 classes. Each node represents a document, and its color indicates its label. The nodes that represent the top-10 active bases are marked with bold rims. (a) W ord 984 only appears in documents of the class " Case-Based " in Cora. Consistently, all its 10 active bases also belong to the class " Case-Based ". (b) The frequencies of W ord 1177 appearing in different classes are similar in Cora. As expected, the top-10 active bases of W ord 1177 also belong to different classes. Owing to the properties of graph wavelets, which describe the neighboring topology centered at each node, the projected values of wavelet transform can be explained as the correlation between features and nodes. These properties provide an interpretable domain transformation and ease the understanding of graph convolution. Replacing graph Fourier transform with graph wavelet transform, we proposed GWNN. Graph wavelet transform has three desirable properties: Graph wavelets are local and sparse; Graph wavelet transform is computationally efficient; Convolution is localized in vertex domain. These advantages make the whole learning process interpretable and efficient. Moreover, to reduce the number of parameters and the dependence on huge training data, we detached the feature transformation from convolution. This practice makes GWNN applicable to large graphs, with remarkable performance improvement on graph-based semi-supervised learning. We use a diagonal matrix Θ to represent the learned kernel transformed by wavelets ψ −1 s y, and replace the Hadamard product with matrix muplication. Then Equation FORMULA3 is: DISPLAYFORM0 We set ψ s = (ψ s1, ψ s2, ..., ψ sn), ψ DISPLAYFORM1 ). Equation FORMULA0 becomes: DISPLAYFORM2 As proved Since each M k is local, for any convolution kernel Θ, ψ s Θψ −1 s is local, and it means that convolution is localized in vertex domain. By replacing Θ with an identity matrix in Equation FORMULA0, we get x * G y = n k=1 M k x. We define H = n k=1 M k, and Figure 4 shows H [1,:] in different scaling, i.e., correlation between the first node and other nodes during convolution. The locality of H suggests that graph convolution is localized in vertex domain. Moreover, as the scaling parameter s becomes larger, the range of feature diffusion becomes larger. GWNN leverages graph wavelets to implement graph convolution, where s is used to modulate the range of neighborhoods. From Figure 5, as s becomes larger starting from 0, the range of neighboring nodes becomes large, ing the increase of accuracy on Cora. However when s becomes too large, some irrelevant nodes are included, leading to decreasing of accuracy. The hyperparameter t only used for computational efficiency, has any slight influence on its performance. For experiments on specific dataset, s and t are choosen via grid search using validation. Generally, a appropriate s is in the range of [0.5, 1], which can not only capture the graph structure but also guarantee the locality of convolution, and t is less insensive to dataset. We show the parameter complexity of node classification in TAB4. The high parameter complexity O(n * p * q) of Spectral CNN makes it difficult to generalize to real world networks. ChebyNet approximates the convolution kernel via polynomial function of the diagonal matrix of Laplacian eigenvalues, reducing parameter complexity to O(K * p * q) with K being the order of polynomial function. GCN simplifies ChebyNet via setting K=1. We detach feature transformation from graph convolution to implement GWNN and Spectral CNN in our experiments, which can reduce parameter to O(n + p * q). In Cora and Citeseer, with smaller parameter complexity, GWNN achieves better performance than ChebyNet, reflecting that it is promising to implement convolution via graph wavelet transform. As Pubmed has a large number of nodes, the parameter complexity of GWNN is larger than ChebyNet. As future work, it is an interesting attempt to select wavelets associated with a subset of nodes, further reducing parameter complexity with potential loss of performance. With the stable recurrence relation T k (y) = 2yT k−1 (y) − T k−2 (y), we can generate the Chebyshev polynomials T k (y). Here T 0 = 1 and T 1 = y. For y sampled between -1 and 1, the trigonometric expression T k (y) = cos(k arccos (y) cos(kθ)g(s(a(cos(θ) + 1)))dθ. we truncate the Chebyshev expansion to m terms and achieve Polynomial approximation. Here we give the example of the ψ −1 s and g(sx) = e −sx, the graph signal is f ∈ R n. Then we can give the fast approximation wavelets by ψ The sparsity of the graph wavelets depends on the sparsity of the Laplacian matrix and the hyperparameter s, We show the sparsity of spectral transform matrix and Laplacian matrix in TAB5. The sparsity of Laplacian matrix is sparser than graph wavelets, and this property limits our method, i.e., the higher time complexity than some methods depending on Laplacian matrix and identity matrix, e.g., GCN. Specifically, both our method and GCN aim to improve Spectral CNN via designing localized graph convolution. GCN, as a simplified version of ChebyNet, leverages Laplacian matrix as weighted matrix and expresses the spectral graph convolution in spatial domain, acting as spatial-like method. However, our method resorts to using graph wavelets as a new set of bases, directly designing localized spectral graph convolution. GWNN offers a localized graph convolution via replacing graph Fourier transform with graph wavelet transform, finding good spectral basis with localization property and good interpretability. This distinguishes GWNN from ChebyNet and GCN, which express the graph convolution defined via graph Fourier transform in vertex domain.

represent the tldr for finding the corresponding scientific article

We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcoming of previous spectral graph CNN methods that depend on graph Fourier transform.

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.


Question: Represent the scientific article to find a short TLDR:

Question: Learning rate decay (lrDecay) is a \emph{de facto} technique for training modern neural networks. It starts with a large learning rate and then decays it multiple times. It is empirically observed to help both optimization and generalization. Common beliefs in how lrDecay works come from the optimization analysis of (Stochastic) Gradient Descent: 1) an initially large learning rate accelerates training or helps the network escape spurious local minima; 2) decaying the learning rate helps the network converge to a local minimum and avoid oscillation. Despite the popularity of these common beliefs, experiments suggest that they are insufficient in explaining the general effectiveness of lrDecay in training modern neural networks that are deep, wide, and nonconvex. We provide another novel explanation: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns. The proposed explanation is validated on a carefully-constructed dataset with tractable pattern complexity. And its implication, that additional patterns learned in later stages of lrDecay are more complex and thus less transferable, is justified in real-world datasets. We believe that this alternative explanation will shed light into the design of better training strategies for modern neural networks. Modern neural networks are deep, wide, and nonconvex. They are powerful tools for representation learning and serve as core components of deep learning systems. They are top-performing models in language translation , visual recognition (, and decision making . However, the understanding of modern neural networks is way behind their broad applications. A series of pioneering works (; ;) reveal the difficulty of applying conventional machine learning wisdom to deep learning. A better understanding of deep learning is a major mission in the AI field. One obstacle in the way of understanding deep learning is the existence of magic modules in modern neural networks and magic tricks to train them. Take batch normalization module for example, its pervasiveness in both academia and industry is undoubted. The exact reason why it expedites training and helps generalization, however, remains mysterious and is actively studied in recent years (; ;). Only when we clearly understand these magical practices can we promote the theoretical understanding of modern neural networks. Learning rate is "the single most important hyper-parameter" in training neural networks. Learning rate decay (lrDecay) is a de facto technique for training modern neural networks, where we adopt an initially large learning rate and then decay it by a certain factor after pre-defined epochs. Popular deep networks such as ResNet , DenseNet (b) are all trained by Stochastic Gradient Descent (SGD) with lrDecay. Figure 1 (a) is an example of lrDecay, with the learning rate decayed by 10 every 30 epochs. The training is divided into several stages by the moments of decay. These stages can be easily identified in learning curves (such as Figure 1(b) ), where the performance boosts sharply shortly after the learning rate is decayed. The lrDecay enjoys great popularity due to its simplicity and general effectiveness. Common beliefs in how lrDecay works are derived from the optimization analysis in (Stochastic) Gradient Descent . They attribute the effect of an initially optimization escapes bad local minima converges to local minimum Proposed pattern complexity avoids fitting noisy data learns more complex patterns Table 1: Comparison of explanations on why lrDecay helps training neural networks. The column "supported" means whether the explanation is supported by the empirical experiments in this paper. large learning rate to escaping spurious local minima or accelerating training and attribute the effect of decaying the learning rate to avoiding oscillation around local minima. However, these common beliefs are insufficient to explain our empirical observations from a series of carefully-designed experiments in Section 4. In this paper, we provide an alternative view: the magnitude of the learning rate is closely related to the complexity of learnable patterns. From this perspective, we propose a novel explanation for the efficacy of lrDecay: an initially large learning rate suppresses the memorization of noisy data while decaying the learning rate improves the learning of complex patterns. This is validated on a carefully-constructed dataset with tractable pattern complexity. The pattern complexity in realworld datasets is often intractable. We thus validate the explanation by testing its implication on real-world datasets. The implication that additional patterns learned in later stages of lrDecay are more complex and thus less transferable across different datasets, is also justified empirically. A comparison between the proposed explanation and the common beliefs is summarized in Table 1. Our explanation is supported by carefully-designed experiments and provides a new perspective on analyzing learning rate decay. The contribution of this paper is two-fold: • We demonstrate by experiments that existing explanations of how lrDecay works are insufficient in explaining the training behaviors in modern neural networks. • We propose a novel explanation based on pattern complexity, which is validated on a dataset with tractable pattern complexity, and its implication is validated on real-world datasets. The explanation also suggests that complex patterns are only learnable after learning rate decay. Thus, when the model learns all simple patterns, but the epoch to decay has not reached, immediately decaying the learning rate will not hurt the performance. This implication is validated in Section A.1. Recently, researchers reveal the behavior of SGD from multiple perspectives (; ;). They respect the difference among data items rather than treat them as identical samples from a distribution. They study the behavior of SGD in a given dataset. , they show that deep models first learn easy examples classifiable by shallow methods. The mutual information between deep models and linear models is measured in , which suggests deep models first learn data explainable by linear models. Note that they are not relevant to learning rates. analyze a toy problem to uncover the regularization effect of an initially large learning rate. Their theoretical explanation is, however, based on a specific two-layer neural network they design. Different from these works, Section 5 studies the behavior of SGD induced by lrDecay in a modern WideResNet , finding that learning rate decay improves learning of complex patterns. We formally define pattern complexity by expected class conditional entropy, while the measure of pattern complexity in; relies on an auxiliary model. Adaptive learning rate methods such as AdaGrad , AdaDelta , and ADAM are sophisticated optimization algorithms for training modern neural networks. It remains an active research field to study their behaviors and underlying mechanisms . However, we focus on learning rate decay in SGD rather than on the adaptive learning rate methods. On the one hand, SGD is the de facto training algorithm for popular models (; b) while lrDecay is not common in the adaptive methods; On the other hand, many adaptive methods are not as simple as SGD and even degenerate in convergence in some scenarios . We choose to study SGD with lrDecay, without introducing adaptive learning rate to keep away from its confounding factors. Besides the commonly used lrDecay, there are other learning rate strategies. proposes a cyclic strategy, claiming to dismiss the need for tuning learning rates. Warm restart of learning rate is explored in. They achieve better when combined with Snapshot Ensemble (a). These learning rate strategies often yield better at the cost of additional hyperparameters that are not intuitive. Consequently, it is still the de facto to decay the learning rate after pre-defined epochs as in Figure 1 (a). We stick our analysis to lrDecay rather than to other fancy ones because of its simplicity and general effectiveness. Training a model on one dataset that can be transferred to other datasets has long been a goal of AI researches. The exploration of model transferability has attracted extensive attention. , deep features trained for classification are transferred to improve object detection successfully. study the transferability of different modules in pre-trained networks, indicating that higher layers are more task-specific and less transferable across datasets. By varying network architectures, show architectures with a better ImageNet accuracy generally transfer better. explore transfer learning in the field of medical imaging to address domain-specific difficulties. Different from these works that only consider the transferability of models after training, we investigate another dimension of model transferability in Section 6: the evolution of transferability during training with lrDecay. The practice of lrDecay in training neural networks dates back to. The most popular belief in the effect of lrDecay comes from the optimization analysis of Gradient Descent (GD) . Although SGD is more practical in deep learning, researchers are usually satisfied with the analysis of GD considering that SGD is a stochastic variant of GD. From left to right: 1) learning rate is small enough to converge around a minimum, 2) moderate so that it bounces among minima, 3) too large to converge. analyze the property of a quadratic loss surface which can be seen as a second-order approximation around a local minimum in nonconvex optimization. Learning rates are characterized by the relationship with eigenvalues of the Hessian at a local minimum. Denote η the learning rate, H the Hessian, λ an eigenvalue of H, and v an eigenvector of λ. The behavior of the network along the direction v can be characterized as (1 − ηλ) k v, with k the iteration number. Convergence in the direction of v requires 0 < η < 2/λ, while η > 2/λ leads to divergence in the direction of v. If 0 < η < 2/λ holds for every eigenvalue of the Hessian, the network will converge quickly (Figure 2 left). If it holds for some directions but not for all directions, the network will diverge in some directions and thus jump into the neighborhood of another local minimum (Figure 2 middle). If the learning rate is too large, the network will not converge (Figure 2 right). In particular, when oscillation happens, it means the learning rate is too large and should be decayed. The effect of lrDecay hence is to avoid oscillation and to obtain faster convergence. only analyze a simple one-layer network. It may not hold for modern neural networks (see Section 4.1). Another common belief is the Stochastic Gradient Descent explanation, arguing that "with a high learning rate, the system is unable to settle down into deeper, but narrower parts of the loss function." 1 Although it is common, this argument has not been formally analyzed until very recently.). The first plot: an initially large learning rate helps escape spurious local minima. From the second to the fourth plots: after more rounds of learning rate decay, the probability of reaching the minimum becomes larger. Under some assumptions, prove SGD is equivalent to the convolution of loss surface, with the learning rate serving as the conceptual kernel size of the convolution. With an appropriate learning rate, spurious local minima can be smoothed out, thus helping neural networks escape bad local minima. The decay of learning rate later helps the network converge around the minimum. Figure 3 is an intuitive one-dimensional example. The first plot shows that a large learning rate helps escape bad local minima in both sides. The lrDecay in subsequent plots increases the probability of reaching the global minimum. Although intuitive, the explanation requires some assumptions that may not hold for modern neural networks (see Section 4.2). Although the (Stochastic) Gradient Descent explanations in Section 3 account for the effect of lrDecay to some extent, in this section, we show by carefully-designed experiments that they are insufficient to explain the efficacy of lrDecay in modern neural networks. In all the experiments except for Section 6, we use a modern neural network named WideResNet . It is deep, wide, nonconvex, and suitable for datasets like CIFAR10 . We train a WideResNet on CIFAR10 dataset with GD, decay the learning rate at different epochs, and report the training loss (optimization) as well as the test accuracy (generalization) in Figure 4. WideResNet and CIFAR10 are commonly used for studying deep learning . CIFAR10 is not too large so that we can feed the whole dataset as a single batch using distributed training, computing the exact gradient rather than estimating it in mini-batches. Experiments show that lrDecay brings negligible benefit to either optimization or generalization. No matter when the learning rate is decayed, the final performances are almost the same. The instability in the beginning is related to the high loss wall described in , which is not the focus of this paper. The GD explanation in Section 3.1 attributes the effect of lrDecay to avoiding oscillation. Oscillation means there is a small divergence in some directions of the landscape so that the network bounces among nearby minima. However, the divergence factor 1 − ηλ for the largest eigenvalue (≈ 200) is too large even for a small growth of learning rate. Thus, the learning rate is either small enough to converge in a local minimum or large enough to diverge. It is hardly possible to observe the oscillation in learning curves (Figure 2 middle), and diverging learning curves (Figure 2 right) can be discarded during hyperparameter tuning. Therefore, only stable solutions are observable where η is small enough (Figure 2 left), leaving no necessity for learning rate decay. Indeed, when the learning rate is increased mildly, we immediately observe diverging learning curves (Section A.2). In short, the GD explanation cannot explain the effect of lrDecay in training modern neural networks. We follow the experiment setups in Section 4.1, but replace GD with SGD in Figure 7. According to the SGD explanation in Section 3.2, the effect of learning rate decay is to increase the probability of reaching a good minimum. If it is true, the model trained before decay can also reach minima, only by a smaller probability compared to the model after decay. In other words, the SGD explanation indicates the best performances before and after decay are the same. It predicts learning curves like Figure 6. However, Figure 7 does not comply with the SGD explanation: the best performances before and after lrDecay are different by a noticeable margin. Without lrDecay (the rightmost column in Figure 7), the performance plateaus and oscillates, with no chance reaching the performance of the other columns after decay. The performance boost after learning rate decay is widely observed (Figure 1(b) for example). However, possibly due to the violation of its assumptions , the SGD explanation cannot explain the underlying effect of lrDecay. Section 4 uncovers the insufficiency of common beliefs in explaining lrDecay. We thus set off to find a better explanation.; reveal that SGD (without learning rate decay) learns from easy to complex. As learning rates often change from large to small in typical learning rate strategies, we hypothesize that the complexity of learned patterns is related to the magnitude of learning rates. Based on this, we provide a novel explanation from the view of pattern complexity: the effect of learning rate decay is to improve the learning of complex patterns while the effect of an initially large learning rate is to avoid memorization of noisy data. To justify this explanation, we carefully construct a dataset with tractable pattern complexity, and record model accuracies in simple and complex patterns separately with and without lrDecay. The explanation we propose involves pattern complexity, which is generally conceptual and sometimes measured with the help of a simple auxiliary model as in;. Here we try to formalize the idea of pattern complexity: the complexity of a dataset is defined as the expected class conditional entropy: C({(x i, y i)} n i=1 ) = E y H(P (x|y)), where H denotes the entropy functional. The complexity of patterns depends on the complexity of the dataset they belong to. Higher C means larger complexity because there are averagely more patterns in each class to be recognized (consider an animal dataset with 10 subspecies in each species vs. an animal dataset with 100 subspecies in each species). Equipped with the formal definition of complexity, we construct a Pattern Separation 10 (PS10) dataset with ten categories and explicitly separated simple patterns and complex patterns. We first generate a simple sub-dataset together with a complex sub-dataset in R 3. As shown in Figure 8 (a) and Figure 8 (b), patterns are visualized as colors because they lie in R 3. The category label can be identified by either simple patterns or complex patterns. We then merge the two sub-datasets into one dataset. The merging method in Figure 8 (c) is specifically designed such that the simple subset and complex subset are fed into different channels of the WideResNet. This mimics the intuition of patterns as the eye pattern and the nose pattern have different locations in an image of human face. To be compatible with the sliding window fashion of convolutional computation, we make patterns the same across spatial dimensions of height and weight to have the same image size as CIFAR10. To reveal the effect of decaying the learning rate, we compare experiments with and without lrDecay. For those without lrDecay, we set the learning rates equal to the learning rate of each stage in lrDecay. We measure not only the total accuracy but also the accuracies on simple and complex patterns separately. These accuracies are plotted in Figure 9. The first plot in Figure 9 clearly shows the model first learns simple patterns quickly. The boost in total accuracy mainly comes from the accuracy gain on complex patterns when the learning rate is decayed. Plots 2, 3, and 4 show the network learns more complex patterns with a smaller learning rate, leading to the that learning rate decay helps the network learn complex patterns. Figure 9 seems to indicate that an initially large learning rate does nothing more than accelerating training: in Plot 4, a small constant learning rate can achieve roughly the same accuracy compared with lrDecay. However, by adding 10% noisy data to mimic real-world datasets, we observe something interesting. Figure 10 shows the accuracies on simple patterns, complex patterns, and noise data when we add noise into the dataset. Plot 2 in Figure 10 shows an initially large learning rate helps the accuracy on complex patterns. Plot 3 in Figure 10 further shows the accuracy gain on complex patterns comes from the suppression of fitting noisy data. (Note that a larger accuracy on noisy data implies overfitting the noisy data, which is undesirable.) In short, the memorizing noisy data hurts the learning of complex patterns but can be suppressed by an initially large learning rate. report that an initially large learning rate with decay outperforms a small and constant learning rate. They suspect that the network starting with an initially small learning rate will be stuck at some spurious local minima. Our experiments provide an alternative view that spurious local minima may stem from noisy data. And the regularization effect of an initially large learning rate is to suppress the memorization of noisy data. Section 5 examines the proposed explanation on the PS10 dataset. Now we further validate the explanation on real-world datasets. Because there are no clearly separated simple and complex patterns in real-world datasets, it is difficult to directly validate the explanation. The proposed explanation suggests that SGD with lrDecay learns patterns of increasing complexity. Intuitively, more complex patterns are less transferable, harder to generalize across datasets. Thus an immediate implication is that SGD with lrDecay learns patterns of decreasing transferability. We validate it by transfer-learning experiments on real-world datasets, to implicitly support the proposed explanation. The transferability is measured by transferring a model from ImageNet to different target datasets. To get models in different training stages, we train a ResNet-50 on ImageNet from scratch and save checkpoints of models in different stages. The learning rate is decayed twice, leading to three stages. Target datasets for transferring are: Caltech256 with 256 general object classes; CUB-200 with 200 bird classes; MITIndoors with 67 indoor scenes; Sketch250 with sketch painting in 250 general classes. Sketch250 is the most dissimilar to ImageNet because it contains sketch paintings. We study two widely-used strategies of transfer learning: "fix" (ImageNet snapshot models are only used as fixed feature extractors) and "finetune" (feature extractors are jointly trained together with task-specific layers). Let acc i denotes the accuracy of stage i snapshot model on ImageNet and tacc i denotes the accuracy of transferring the snapshot to the target dataset, then the transferability of additional patterns learned in stage i is defined as tacci−tacci−1 acci−acci−1, i = 2, 3. By definition, the transferability of patterns from ImageNet to ImageNet is 1.0, complying with common sense. The transferability is plotted in Figure 11. Table 2 contains the accuracies used to compute it. In all experiments, we find that the transferability of additional patterns learned in stage 3 is less than that in stage 2. Besides, in Sketch250 dataset, the transferability of additional patterns learned in stage 3 is negative. These findings support our claim that additional patterns learned in later stages of lrDecay are more complex and thus less transferable. They also suggest deep model-zoo developer provide pre-trained model snapshots in different stages so that downstream users can select the most transferable snapshot model according to their tasks. In this paper, we dive into how learning rate decay (lrDecay) helps modern neural networks. We uncover the insufficiency of common beliefs and propose a novel explanation: the effect of decaying learning rate is to improve the learning of complex patterns, and the effect of an initially large learning rate is to avoid memorization of noisy data. It is supported by experiments on a dataset with tractable pattern complexity as well as on real-world datasets. It would be interesting to further bridge the proposed explanation and the formal analysis of optimization procedure. A.1 AUTODECAY Experiments in Section 5.2 implies that not all complex patterns are learnable under a constant learning rate. The training under a certain learning rate has no effect when the loss plateaus. This indicates we can expedite the training process by killing the over-training of each stage (decay the learning rate when the loss plateaus) with little influence on the performance. To validate the implication, we propose AutoDecay to shorten the useless training and check if the performance of the model can be untouched. In Figure 7, it appears obvious to decide the optimal moment to decay when we have a big picture of the training process. The problem is, however, how can we make a decision to decay depending on the current and past observations. It is a non-trivial problem given that the statistics exhibit noticeable noise. We formalize the observed training loss into two parts:ˆ (t) = (t) + (t), with (t) the ground truth loss (unobservable) and (t) the noise introduced by SGD. Here t indicates the training process (typically the epoch number) and takes value in N = {1, 2, 3, . . .}. To simplify the problem, we assume (t) is independent with t and (t) is independent of (t)(t = t) in SGD. The nature of noise gives rise to the zero-expectation property E (t) = 0. Denote σ 2 = Var (t) the variance of the noise. Due to the noise of SGD, the observed training loss usually vibrates in a short time window but decreases in a long time window. Our task is to find out whether the loss value is stable in the presence of noise. Exponential Decay Moving Average (EDMA) with Bias Correction. Observations with lower variance are more trustworthy. However, there is nothing we can do about the variance ofˆ (t). We consider computing a low-variance statistic aboutˆ (t). We adopt moving average with bias correction . Let g(t) be the moving average of (t) andĝ(t) be the moving average ofˆ (t). The explicit form is in Equation 1, where β ∈ is the decay factor in EDMA. The recursive (and thus implicit) form is in Equation 2. It enables us to compute the statisticĝ online (without storing all the previous {ˆ (i)|i < t}) at the cost of maintainingf (t). Asĝ(t) is a linear combination of {ˆ (i)|i ≤ t}, it is easy to showĝ(t) is unbiased: The variance ofĝ(t) is The fact that β ∈ indicates Varĝ(t) is monotonically decreasing. Typically β = 0.9 (Figure 13), and the variance can rapidly converge to 0.05σ 2, much smaller than the variance of the noise.ĝ(t) well represents the unobservable g(t). If (t) gets stable, we shall observe thatĝ(t) is stable, too. Criterion of Being Stable. We only want to decay the learning rate when the loss plateaus, i.e. when the loss is stable. For observed values of G = {ĝ(i)|i − W + 1 ≤ i ≤ t} within the window size of W, we call them stable if where is a small constant that prevents zero-division error, and η indicates the tolerance of variation. Criterion of Significant Drop. When we keep decaying the learning rate, there comes a time when the learning rate is too small and the network cannot make any progress. When it happens, we should terminate the training. Termination is adopted when their is no significant drop between the stable value and the original valueĝ. To be specific, the criterion of significant drop isĝ where is a small constant that prevents zero-division error, and ζ indicates the degree of drop. The entire procedure of AutoDecay is described in Figure 12. We try AutoDecay on ImageNet to test whether it can expedite the training without hurting the performance. We are not trying to set up a new state-of-the-art record. We train a ResNet-50 model on ImageNet following the official code of PyTorch. The only change is we replace the StepDecay strategy with the proposed AutoDecay strategy. Each experiment costs roughly two days with 8 TITAN X GPUs. The in Table 14 show that AutoDecay can shorten the training time by 10% without hurting the performance (even bringing a slight improvement), successfully vaidates the proposed explanation in this paper. When we increase the learning rate mildly for Gradient Descent, we immediately observe diverging learning curves (Figure 15), which echos with the reason mentioned in Section 4.1 why the Gradient Descent explanation fails to work in modern neural networks: modern neural networks have a very large spectrum norm at a local minimum, and even a small growth of learning rate can lead to divergence. In other words, training modern neural networks with GD must use a small enough learning rate, dismissing the value of learning rate decay.

Represent the TLDR for finding the corresponding scientific article

We provide another novel explanation of learning rate decay: an initially large learning rate suppresses the network from memorizing noisy data while decaying the learning rate improves the learning of complex patterns.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose a extension of the batch normalization, show a first-of-its-kind convergence analysis for this extension and show in numerical experiments that it has better performance than the original batch normalizatin.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: We show how an ensemble of $Q^*$-functions can be leveraged for more effective exploration in deep reinforcement learning. We build on well established algorithms from the bandit setting, and adapt them to the $Q$-learning setting. We propose an exploration strategy based on upper-confidence bounds (UCB). Our experiments show significant gains on the Atari benchmark. Deep reinforcement learning seeks to learn mappings from high-dimensional observations to actions. Deep Q-learning BID15 ) is a leading technique that has been used successfully, especially for video game benchmarks. However, fundamental challenges remain, for example, improving sample efficiency and ensuring convergence to high quality solutions. Provably optimal solutions exist in the bandit setting and for small MDPs, and at the core of these solutions are exploration schemes. However these provably optimal exploration techniques do not extend to deep RL in a straightforward way. Bootstrapped DQN BID16 ) is a previous attempt at adapting a theoretically verified approach to deep RL. In particular, it draws inspiration from posterior sampling for reinforcement learning (PSRL, BID17 ; BID16), which has near-optimal regret bounds. PSRL samples an MDP from its posterior each episode and exactly solves Q *, its optimal Q-function. However, in high-dimensional settings, both approximating the posterior over MDPs and solving the sampled MDP are intractable. Bootstrapped DQN avoids having to establish and sample from the posterior over MDPs by instead approximating the posterior over Q *. In addition, bootstrapped DQN uses a multi-headed neural network to represent the Q-ensemble. While the authors proposed bootstrapping to estimate the posterior distribution, their empirical findings show best performance is attained by simply relying on different initializations for the different heads, not requiring the sampling-with-replacement process that is prescribed by bootstrapping. In this paper, we design new algorithms that build on the Q-ensemble approach from BID16. However, instead of using posterior sampling for exploration, we construct uncertainty estimates from the Q-ensemble. Specifically, we first propose the Ensemble Voting algorithm where the agent takes action by a majority vote from the Q-ensemble. Next, we propose the UCB exploration strategy. This strategy is inspired by established UCB algorithms in the bandit setting and constructs uncertainty estimates of the Q-values. In this strategy, agents are optimistic and take actions with the highest UCB. We demonstrate that our algorithms significantly improve performance on the Atari benchmark. We model reinforcement learning as a Markov decision process (MDP). We define an MDP as (S, A, T, R, p 0, γ), in which both the state space S and action space A are discrete, T: S × A × S → R + is the transition distribution, R: S × A → R is the reward function, assumed deterministic given the state and action, and γ ∈ is a discount factor, and p 0 is the initial state distribution. We denote a transition experience as τ = (s, a, r, s) where s ∼ T (s |s, a) and r = R(s, a). A policy π: S → A specifies the action taken after observing a state. We denote the Q-function for policy π as Q π (s, a):= E π ∞ t=0 γ t r t |s 0 = s, a 0 = a where r t = R(s t, a t). The optimal Q * -function corresponds to taking the optimal policy Q * (s, a):= sup π Q π (s, a)and satisfies the Bellman equation Q * (s, a) = E s ∼T (·|s,a) r + γ · max a Q * (s, a). A notable early optimality in reinforcement learning was the proof by; that an online Q-learning algorithm is guaranteed to converge to the optimal policy, provided that every state is visited an infinite number of times. However, the convergence of Watkins' Q-learning can be prohibitively slow in MDPs wheregreedy action selection explores state space randomly. Later work developed reinforcement learning algorithms with provably fast (polynomial-time) convergence BID10 BID4 ). At the core of these provably-optimal learning methods is some exploration strategy, which actively encourages the agent to visit novel state-action pairs. For example, R-MAX optimistically assumes that infrequently-visited states provide maximal reward, and delayed Q-learning initializes the Q-function with high values to ensure that each state-action is chosen enough times to drive the value down. Since the theoretically sound RL algorithms are not computationally practical in the deep RL setting, deep RL implementations often use simple exploration methods such as -greedy and Boltzmann exploration, which are often sample-inefficient and fail to find good policies. One common approach of exploration in deep RL is to construct an exploration bonus, which adds a reward for visiting state-action pairs that are deemed to be novel or informative. In particular, several prior methods define an exploration bonus based on a density model or dynamics model. Examples include VIME by BID9, which uses variational inference on the forward-dynamics model, and Tang et al. FORMULA8, BID2, , BID8. While these methods yield successful exploration in some problems, a major drawback is that this exploration bonus does not depend on the rewards, so the exploration may focus on irrelevant aspects of the environment, which are unrelated to reward. Earlier works on Bayesian reinforcement learning include BID6 BID7. BID6 studied Bayesian Q-learning in the model-free setting and learned the distribution of Q * -values through Bayesian updates. The prior and posterior specification relied on several simplifying assumptions, some of which are not compatible with the MDP setting. BID7 took a model-based approach that updates the posterior distribution of the MDP. The algorithm samples from the MDP posterior multiple times and solving the Q * values at every step. This approach is only feasible for RL problems with very small state space and action space. proposed posterior sampling for reinforcement learning (PSRL). PSRL instead takes a single sample of the MDP from the posterior in each episode and solves the Q * values. Recent works including BID17 and BID16 established near-optimal Bayesian regret bounds for episodic RL. models the environment and constructs exploration bonus from variance of model parameters. These methods are experimented on low dimensional problems only, because the computational cost of these methods is intractable for high dimensional RL. Inspired by PSRL, but wanting to reduce computational cost, prior work developed approximate methods. proposed randomized least-square value iteration for linearly-parameterized value functions. applies to Q-functions parameterized by deep neural networks. Bootstrapped DQN BID16 ) maintains a Q-ensemble, represented by a multi-head neural net structure to parameterize K ∈ N + Q-functions. This multi-head structure shares the convolution layers but includes multiple "heads", each of which defines a Q-function Q k.Bootstrapped DQN diversifies the Q-ensemble through two mechanisms. The first mechanism is independent initialization. The second mechanism applies different samples to train each Q-function. These Q-functions can be trained simultaneously by combining their loss functions with the help of a random mask DISPLAYFORM0 where y Q k τ is the target of the kth Q-function. Thus, the transition τ updates Q k only if m k τ is nonzero. To avoid the overestimation issue in DQN, bootstrapped DQN calculates the target value y Q k τ using the approach of Double DQN , such that the current Q k (·; θ t) network determines the optimal action and the target network Q k (·; θ −) estimates the value DISPLAYFORM1 In their experiments on Atari games, BID16 set the mask m τ = (1, . . ., 1) such that all {Q k} are trained with the same samples and their only difference is initialization. Bootstrapped DQN picks one Q k uniformly at random at the start of an episode and follows the greedy action a t = argmax a Q k (s t, a) for the whole episode. Ignoring computational costs, the ideal Bayesian approach to reinforcement learning is to maintain a posterior over the MDP. However, with limited computation and model capacity, it is more tractable to maintain a posterior of the Q * -function. This motivates using a Q-ensemble as a particle filter-based approach to approximate the posterior over Q * -function and we display our first proposed method, Ensemble Voting, in Algorithm 1. DISPLAYFORM0 is parametrized with a deep neural network whose parameters are initialized independently at the start of training. Each Q k proposes an action that maximizes the Q-value according to Q k at every time step and the agent chooses the action by a majority vote DISPLAYFORM1 At each learning interval, a minibatch of transitions is sampled from the replay buffer and each Q k takes a Bellman update based on this minibatch. For stability, Algorithm 1 also uses a target network for each Q k as in Double DQN in the batched update. We point out that the difference among the parameters of the Q-ensemble {Q k} comes only from the independent random initialization. The deep neural network parametrization of the Q-ensemble introduces nonconvexity into the objective function of Bellman update, so the Q-ensemble {Q k} do not converge to the same Q-function during training even though they are trained with the same minibatches at every update. We also experimented with bagging by updating each Q k using an independently drawn minibatch. However, bagging led to inferior learning performance. This phenomenon that that bagging deteriorates the performance of deep ensembles is also observed in supervised learning settings. BID13 observed that supervised learning trained with deep ensembles with random initializations perform better than bagging for deep ensembles. BID12 used deep ensembles for uncertainty estimates and also observed that bagging deteriorated performance in their experiments. develop ensemble sampling for bandit problems with deep neural network parametrized policies and the theoretical justification. We derive a posterior update rule for the Q * function and approximations to the posterior update using ensembles in Appendix C. We note that in bootstrapped DQN, ensemble voting is applied for evaluation while Algorithm 1 uses ensemble voting during learning. In the experiments (Sec. 5), we demonstrate that Algorithm 1 is superior to bootstrapped DQN. The action choice of Algorithm 1 is exploitation only. In the next section, we propose our UCB exploration strategy. Pick an action according to DISPLAYFORM2 Execute a t. Receive state s t+1 and reward r t from the environment 8:Add (s t, a t, r t, s t+1) to replay buffer B At learning interval, sample random minibatch and update {Q k} 10:end for 11: end for In this section, we propose optimism-based exploration by adapting the UCB algorithms BID1; BID0 ) from the bandit setting. The UCB algorithms maintain an upper-confidence bound for each arm, such that the expected reward from pulling each arm is smaller than this bound with high probability. At every time step, the agent optimistically chooses the arm with the highest UCB. BID1 constructed the UCB based on empirical reward and the number of times each arm is chosen. BID0 incorporated the empirical variance of each arm's reward into the UCB, such that at time step t, an arm A t is pulled according to DISPLAYFORM0 wherer i,t andV i,t are the empirical reward and variance of arm i at time t, n i,t is the number of times arm i has been pulled up to time t, and c 1, c 2 are positive constants. We extend the intuition of UCB algorithms to the RL setting. Using the outputs of the {Q k} functions, we construct a UCB by adding the empirical standard deviationσ(DISPLAYFORM1 . The agent chooses the action that maximizes this UCB a t ∈ argmax a μ(s t, a) + λ ·σ(s t, a), where λ ∈ R + is a hyperparameter. We present Algorithm 2, which incorporates the UCB exploration. The hyperparemeter λ controls the degrees of exploration. In Section 5, we compare the performance of our algorithms on Atari games using a consistent set of parameters. Pick an action according to a t ∈ argmax a μ(s t, a) + λ ·σ(s t, a) Receive state s t+1 and reward r t from environment, having taken action a t 8:Add (s t, a t, r t, s t+1) to replay buffer B 9:At learning interval, sample random minibatch and update {Q k} We evaluate the algorithms on each Atari game of the Arcade Learning Environment BID3 ). We use the multi-head neural net architecture of BID16. We fix the common hyperparameters of all algorithms based on a well-tuned double DQN implementation, which uses the Adam optimizer BID11 ), different learning rate and exploration schedules compared to BID15. Appendix A tabulates the hyperparameters. The number of {Q k} functions is K = 10. Experiments are conducted on the OpenAI Gym platform BID5 ) and trained with 40 million frames and 2 trials on each game. We take the following directions to evaluate the performance of our algorithms:1. we compare Algorithm 1 against Double DQN and bootstrapped DQN, 2. we isolate the impact of UCB exploration by comparing Algorithm 2 with λ = 0.1, denoted as ucb exploration, against Algorithm 1, Double DQN, and bootstrapped DQN.3. we compare Algorithm 1 and Algorithm 2 with the count-based exploration method of BID2.4. we aggregate the comparison according to different categories of games, to understand when our methods are suprior. In Appendix B, we tabulate detailed that compare our algorithms, Ensemble Voting and ucb exploration, against prior methods. In TAB5, we tabulate the maximal mean reward in 100 consecutive episodes for Ensemble Voting, ucb exploration, bootstrapped DQN and Double DQN. Without exploration, Ensemble Voting already achieves higher maximal mean reward than both Double DQN and bootstrapped DQN in a majority of Atari games. Ensemble Voting performs better than Double DQN in 37 games out of the total 49 games evaluated, better than bootstrapped DQN in 41 games. ucb exploration achieves the highest maximal mean reward among these four algorithms in 30 games out of the total 49 games evaluated. Specifically, ucb exploration performs better than Double DQN in 38 out of 49 games evaluated, better than bootstrapped DQN in 45 games, and better than Ensemble Voting in 35 games. Figure 2 displays the learning curves of these five algorithms on a set of six Atari games. Ensemble Voting outperforms Double DQN and bootstrapped DQN. ucb exploration outperforms Ensemble Voting. In TAB6, we compare our proposed methods with the count-based exploration method A3C+ of BID2 based on their published of A3C+ trained with 200 million frames. We point out that even though our methods were trained with only 40 million frames, much less than A3C+'s 200 million frames, UCB exploration achieves the highest average reward in 28 games, Ensemble Voting in 10 games, and A3C+ in 10 games. Our approach outperforms A3C+.Finally to understand why and when the proposed methods are superior, we aggregate the comparison according to four categories: Human Optimal, Score Explicit, Dense Reward, and Sparse Reward. These categories follow the taxonomy in Probability of random action ingreedy exploration, as a function of the iteration t.replay start size 50000 Number of uniform random actions taken before learning starts. Table 4: Comparison of each method across different game categories. The Atari games are separated into four categories: human optimal, score explicit, dense reward, and sparse reward. In each row, we present the number of games in this category, the total number of games where each algorithm achieves the optimal performance according to TAB5. The game categories follow the taxonomy in TAB0 of C APPROXIMATING BAYESIAN Q-LEARNING WITH Q-ENSEMBLESIn this section, we first derive a posterior update formula for the Q * -function under full exploration assumption and this formula turns out to depend on the transition Markov chain. Next, we approximate the posterior update with Q-ensembles {Q k} and demonstrate that the Bellman equation emerges as the approximate update rule for each Q k. * -FUNCTION An MDP is specified by the transition probability T and the reward function R. Unlike prior works outlined in Section 2.3 which learned the posterior of the MDP, we will consider the joint distribution over (Q *, T). Note that R can be recovered from Q * given T. So (Q *, T) determines a unique MDP. In this section, we assume that the agent samples (s, a) according to a fixed distribution. The corresponding reward r and next state s given by the MDP append to (s, a) to form a transition τ = (s, a, r, s), for updating the posterior of (Q *, T). Recall that the Q * -function satisfies the Bellman equation DISPLAYFORM0 Denote the joint prior distribution as p(Q *, T) and the posterior asp. We apply Bayes' formula to expand the posterior: DISPLAYFORM1 where Z(τ) is a normalizing constant and the second equality is because s and a are sampled randomly from S and A. Next, we calculate the two conditional probabilities in. First, DISPLAYFORM2 where the first equality is because given T, Q * does not influence the transition. Second, DISPLAYFORM3 where 1 {·} is the indicator function and in the last equation we abbreviate it as 1(Q *, T). Substituting FORMULA9 and FORMULA10 into FORMULA8, we obtain the joint posterior of Q * and T after observing an additional randomly sampled transition τ DISPLAYFORM4 The exact Q * -posterior update is intractable in high-dimensional RL due to the large space of (Q *, T). Thus, we make several approximations to the Q * -posterior update. First, we approximate the prior of Q * by sampling K ∈ N + independently initialized Q * -functions {Q k} K k=1. Next, we update them as more transitions are sampled. The ing {Q k} approximate samples drawn from the posterior. The agent chooses the action by taking a majority vote from the actions determined by each Q k.We derive the update rule for {Q k} after observing a new transition τ = (s, a, r, s). At iteration i, given Q * = Q k,i (·; θ k) parametrized by θ k the joint probability of (Q *, T) factors into DISPLAYFORM0 Substitute FORMULA12 into FORMULA11 and we obtain the corresponding posterior for each Q k,i+1 at iteration i + 1 as DISPLAYFORM1 We update Q k,i to Q k,i+1 according to DISPLAYFORM2 We first derive a lower bound of the the posteriorp(Q k,i+1 |τ): DISPLAYFORM3 where we apply a limit representation of the indicator function in the third equation. The fourth equation is due to the bounded convergence theorem. The inequality is Jensen's inequality. The last equation replaces the limit with an indicator function. A sufficient condition for FORMULA14 is to maximize the lower-bound of the posterior distribution in by ensuring the indicator function in to hold. We can replace with the following update DISPLAYFORM4 However, FORMULA8 is not tractable because the expectation in is taken with respect to the posterior p(T |Q k,i, τ) of the transition T. To overcome this challenge, we approximate the posterior update by reusing the one-sample next state s from τ. Solving the exact minimal for each Q k,i+1 is impractical, thus we take a gradient step on Q k,i+1 according to the following gradient DISPLAYFORM5 where η is the step size. Instead of updating Q k after each transition, we use an experience replay buffer B to store observed transitions and sample a minibatch B mini of transitions (s, a, r, s) for each update. In this case, the batched update of each Q k,i to Q k,i+1 becomes a standard Bellman update DISPLAYFORM6 In this section, we also studied an "InfoGain" exploration bonus, which encourages agents to gain information about the Q * -function and examine its effectiveness. We found it had some benefits on top of Ensemble Voting, but no uniform additional benefits once already using Q-ensembles on top of Double DQN. We describe the approach and our experimental findings. Similar to , we define the information gain from observing an additional transition τ n as DISPLAYFORM0 wherep(Q * |τ 1, . . ., τ n) is the posterior distribution of Q * after observing a sequence of transitions (τ 1, . . ., τ n). The total information gain is DISPLAYFORM1 Our Ensemble Voting, Algorithm 1, does not maintain the posteriorp, thus we cannot calculate explicitly. Instead, inspired by BID12, we define an InfoGain exploration bonus that measures the disagreement among {Q k}. Note that DISPLAYFORM2 where H(·) is the entropy. If H τ1,...,τ N is small, then the posterior distribution has high entropy and high residual information. Since {Q k} are approximate samples from the posterior, high entropy of the posterior leads to large discrepancy among {Q k}. Thus, the exploration bonus is monotonous with respect to the residual information in the posterior H(p(Q * |τ 1, . . ., τ N)). We first compute the Boltzmann distribution for each Q k DISPLAYFORM3 where T > 0 is a temperature parameter. Next, calculate the average Boltzmann distribution DISPLAYFORM4 The InfoGain exploration bonus is the average KL-divergence from DISPLAYFORM5 The modified reward isr (s, a, s) = r(s, a) DISPLAYFORM6 where ρ ∈ R + is a hyperparameter that controls the degree of exploration. The exploration bonus b T (s t) encourages the agent to explore where {Q k} disagree. The temperature parameter T controls the sensitivity to discrepancies among {Q k}. When T → +∞, {P T,k} converge to the uniform distribution on the action space and b T (s) → 0. When T is small, the differences among {Q k} are magnified and b T (s) is large. We display Algorithrim 3, which incorporates our InfoGain exploration bonus into Algorithm 2. The hyperparameters λ, T and ρ vary for each game. We demonstrate the performance of the combined UCB+InfoGain exploration in FIG4 and FIG4. We augment the previous figures in Section 5 with the performance of ucb+infogain exploration, where we set λ = 0.1, ρ = 1, and T = 1 in Algorithm 3. FIG4 shows that combining UCB and InfoGain exploration does not lead to uniform improvement in the normalized learning curve. At the individual game level, FIG4 shows that the impact of InfoGain exploration varies. UCB exploration achieves sufficient exploration in games including Demon Attack and Kangaroo and Riverraid, while InfoGain exploration further improves learning on Enduro, Seaquest, and Up N Down. The effect of InfoGain exploration depends on the choice of the temperature T. The optimal temperature parameter varies across games. In FIG5, we display the behavior of ucb+infogain exploration with different temperature values. Thus, we see the InfoGain exploration bonus, tuned with the appropriate temperature parameter, can lead to improved learning for games that require extra exploration, such as ChopperCommand, KungFuMaster, Seaquest, UpNDown.

Represent the short summary for finding the corresponding scientific article (long):

Adapting UCB exploration to ensemble Q-learning improves over prior methods such as Double DQN, A3C+ on Atari benchmark

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.