Question: Represent the scientific article to find a short TLDR

Question: We propose a novel hierarchical generative model with a simple Markovian structure and a corresponding inference model. Both the generative and inference model are trained using the adversarial learning paradigm. We demonstrate that the hierarchical structure supports the learning of progressively more abstract representations as well as providing semantically meaningful reconstructions with different levels of fidelity. Furthermore, we show that minimizing the Jensen-Shanon divergence between the generative and inference network is enough to minimize the reconstruction error. The ing semantically meaningful hierarchical latent structure discovery is exemplified on the CelebA dataset. There, we show that the features learned by our model in an unsupervised way outperform the best handcrafted features. Furthermore, the extracted features remain competitive when compared to several recent deep supervised approaches on an attribute prediction task on CelebA. Finally, we leverage the model's inference network to achieve state-of-the-art performance on a semi-supervised variant of the MNIST digit classification task. Deep generative models represent powerful approaches to modeling highly complex high-dimensional data. There has been a lot of recent research geared towards the advancement of deep generative modeling strategies, including Variational Autoencoders (VAE) BID16, autoregressive models BID32 b) and hybrid models BID9 BID31. However, Generative Adversarial Networks (GANs) BID8 have emerged as the learning paradigm of choice across a varied range of tasks, especially in computer vision BID47, simulation and robotics BID7 BID41. GANs cast the learning of a generative network in the form of a game between the generative and discriminator networks. While the discriminator is trained to distinguish between the true and generated examples, the generative model is trained to fool the discriminator. Using a discriminator network in GANs avoids the need for an explicit reconstruction-based loss function. This allows this model class to generate visually sharper images than VAEs while simultaneously enjoying faster sampling than autoregressive models. Recent work, known as either ALI or BiGAN, has shown that the adversarial learning paradigm can be extended to incorporate the learning of an inference network. While the inference network, or encoder, maps training examples x to a latent space variable z, the decoder plays the role of the standard GAN generator mapping from space of the latent variables (that is typically sampled from some factorial distribution) into the data space. In ALI, the discriminator is trained to distinguish between the encoder and the decoder, while the encoder and decoder are trained to conspire together to fool the discriminator. Unlike some approaches that hybridize VAE-style inference with GAN-style generative learning (e.g. BID20,), the encoder and decoder in ALI use a purely adversarial approach. One big advantage of adopting an adversarial-only formalism is demonstrated by the high-quality of the generated samples. Additionally, we are given a mechanism to infer the latent code associated with a true data example. One interesting feature highlighted in the original ALI work is that even though the encoder and decoder models are never explicitly trained to perform reconstruction, this can nevertheless be easily done by projecting data samples via the encoder into the latent space, copying these values across to the latent variable layer of the decoder and projecting them back to the data space. Doing this yields reconstructions that often preserve some semantic features of the original input data, but are perceptually relatively different from the original samples. These observations naturally lead to the question of the source of the discrepancy between the data samples and their ALI reconstructions. Is the discrepancy due to a failure of the adversarial training paradigm, or is it due to the more standard challenge of compressing the information from the data into a rather restrictive latent feature vector? BID44 show that an improvement in reconstructions is achievable when additional terms which explicitly minimize reconstruction error in the data space are added to the training objective. BID23 palliates to the non-identifiability issues pertaining to bidirectional adversarial training by augmenting the generator's loss with an adversarial cycle consistency loss. In this paper we explore issues surrounding the representation of complex, richly-structured data, such as natural images, in the context of a novel, hierarchical generative model, Hierarchical Adversarially Learned Inference (HALI), which represents a hierarchical extension of ALI. We show that within a purely adversarial training paradigm, and by exploiting the model's hierarchical structure, we can modulate the perceptual fidelity of the reconstructions. We provide theoretical arguments for why HALI's adversarial game should be sufficient to minimize the reconstruction cost and show empirical evidence supporting this perspective. Finally, we evaluate the usefulness of the learned representations on a semi-supervised task on MNIST and an attribution prediction task on the CelebA dataset. Our work fits into the general trend of hybrid approaches to generative modeling that combine aspects of VAEs and GANs. For example, Adversarial Autoencoders BID28 replace the Kullback-Leibler divergence that appears in the training objective for VAEs with an adversarial discriminator that learns to distinguish between samples from the approximate posterior and the prior. A second line of research has been directed towards replacing the reconstruction penalty from the VAE objective with GANs or other kinds of auxiliary losses. Examples of this include BID20 that combines the GAN generator and the VAE decoder into one network and that uses the loss of a pre-trained classifier as an additional reconstruction loss in the VAE objective. Another research direction has been focused on augmenting GANs with inference machinery. One particular approach is given by;, where, like in our approach, there is a separate inference network that is jointly trained with the usual GAN discriminator and generator. BID15 presents a theoretical framework to jointly train inference networks and generators defined on directed acyclic graphs by leverage multiple discriminators defined nodes and their parents. Another related work is that of BID12 which takes advantage of the representational information coming from a pre-trained discriminator. Their model decomposes the data generating task into multiple subtasks, where each level outputs an intermediate representation conditioned on the representations from higher level. A stack of discriminators is employed to provide signals for these intermediate representations. The idea of stacking discriminator can be traced back to BID4 which used used a succession of convolutional networks within a Laplacian pyramid framework to progressively increase the resolution of the generated images. The goal of generative modeling is to capture the data-generating process with a probabilistic model. Most real-world data is highly complex and thus, the exact modeling of the underlying probability density function is usually computationally intractable. Motivated by this fact, GANs BID8 model the data-generating distribution as a transformation of some fixed distribution over latent variables. In particular, the adversarial loss, through a discriminator network, forces the generator network to produce samples that are close to those of the data-generating distribution. While GANs are flexible and provide good approximations to the true data-generating mechanism, their original formulation does not permit inference on the latent variables. In order to mitigate this, Adversarially Learned Inference (ALI) extends the GAN framework to include an inference network that encodes the data into the latent space. The discriminator is then trained to discriminate between the joint distribution of the data and latent causes coming from the generator and inference network. Thus, the ALI objective encourages a matching of the two joint distributions, which also in all the marginals and conditional distributions being matched. This enables inference on the latent variables. We endeavor to improve on ALI in two aspects. First, as reconstructions from ALI only loosely match the input on a perceptual level, we want to achieve better perceptual matching in the reconstructions. Second, we wish to be able to compress the observables, x, using a sequence of composed features maps, leading to a distilled hierarchy of stochastic latent representations, denoted by z 1 to z L. Note that, as a consequence of the data processing inequality BID2, latent representations higher up in the hierarchy cannot contain more information than those situated lower in the hierarchy. In information-theoretic terms, the conditional entropy of the observables given a latent variable is non-increasing as we ascend the hierarchy. This loss of information can be seen as responsible for the perceptual discrepancy observed in ALI's reconstructions. Thus, the question we seek to answer becomes: How can we achieve high perceptual fidelity of the data reconstructions while also having a compressed latent space that is strongly coupled with the observables? In this paper, we propose to answer this using a novel model, Hierarchical Adversarially Learned Inference (HALI), that uses a simple hierarchical Markovian inference network that is matched through adversarial training to a similarly constructed generator network. Furthermore, we discuss the hierarchy of reconstructions induced by the HALI's hierarchical inference network and show that the ing reconstruction errors are implicitly minimized during adversarial training. Also, we leverage HALI's hierarchial inference network to offer a novel approach to semi-supervised learning in generative adversarial models. Denote by P(S) the set of all probability measures on some set S. Let T Z|X be a Markov kernel associating to each element x ∈ X a probability measure P Z|X=x ∈ P(Z). Given two Markov kernels T W |V and T V |U, a further Markov kernel can be defined by composing these two and then marginalizing over V, i.e. T W |V • T V |U: U → P(W). Consider a set of random variables x, z 1,..., z L. Using the composition operation, we can construct a hierarchy of Markov kernels or feature transitions as DISPLAYFORM0 A desirable property for these feature transitions is to have some form of inverses. Motivated by this, we define the adjoint feature transition as DISPLAYFORM1 This can be interpreted as the generative mechanism of the latent variables given the data being the "inverse" of the data-generating mechanism given the latent variables. Let q(x) denote the distribution of the data and p(z L) be the prior on the latent variables. Typically the prior will be a simple distribution, e.g. DISPLAYFORM2 The composition of Markov kernels in Eq. 1, mapping data samples x to samples of the latent variables z L using z 1,..., z L−1 constitutes the encoder. Similarly, the composition of kernels in Eq. 2 mapping prior samples of z L to data samples x through z L−1,..., z 1 constitutes the decoder. Thus, the joint distribution of the encoder can be written as DISPLAYFORM3 while the joint distribution of the decoder is given by DISPLAYFORM4 Algorithm 1 HALI training procedure. DISPLAYFORM5 Sample from the prior for l ∈ {1, . . ., L} dô z DISPLAYFORM6 Sample from each level in the encoder's hierarchy end for for l ∈ {L . . . 1} do z DISPLAYFORM7 Sample from each level in the decoder's hierarchy end for ρ DISPLAYFORM8 Get discriminator predictions on decoder's distribution end for DISPLAYFORM9 Compute discriminator loss DISPLAYFORM10 Compute generator loss DISPLAYFORM11 Gradient update on generator networks until convergenceThe encoder and decoder distributions can be visualized graphically as DISPLAYFORM12 Having constructed the joint distributions of the encoder and decoder, we can now match these distributions through adversarial training. It can be shown that, under an ideal (non-parametric) discriminator, this is equivalent to minimizing the Jensen-Shanon divergence between the joint Eq. 3 and Eq. 4, see. Algorithm 1 details the training procedure. The Markovian character of both the encoder and decoder implies a hierarchy of reconstructions in the decoder. In particular, for a given observation x ∼ p(x), the model yields L different reconstructionŝ x l ∼ T x|z l • T z l |x for l ∈ {1, . . ., L} withx l the reconstruction of the x at the l-th level of the hierarchy. Here, we can think of T z l |x as projecting x to the l-th intermediate representation and T x|z l as projecting it back to the input space. Then, the reconstruction error for a given input x at the l-th hierarchical level is given by DISPLAYFORM0 Contrary to models that try to merge autoencoders and adversarial models, e.g. BID36 BID20, HALI does not require any additional terms in its loss function in order to minimize the above reconstruction error. Indeed, the reconstruction errors at the different levels of HALI are minimized down to the amount of information about x that a given level of the hierarchy is able to encode as training proceeds. Furthermore, under an optimal discriminator, training in HALI minimizes the Jensen-Shanon divergence between q(x, z 1, . . ., z L) and p(x, z 1, . . ., z L) as formalized in Proposition 1 below. Furthermore, the interaction between the reconstruction error and training dynamics is captured in Proposition 1. Proposition 1. Assuming q(x, z l) is bounded away for zero for all l ∈ {1, . . ., L}, we have that DISPLAYFORM1 where H(x | z l) is computed under the encoder's distribution and K is as defined in Lemma 2 in the appendix. On the other hand, proposition 2 below relates the intermediate representations in the hierarchy to the corresponding induced reconstruction error. Proposition 2. For any given latent variable z l, DISPLAYFORM2 i.e. the reconstruction error is an upper bound on H(x | z l).In summary, Propositions 1 and 2 establish the dynamics between the hierarchical representation learned by the inference network, the reconstruction errors and the adversarial matching of the joint distributions Eq. 3 and Eq. 4. The proofs on the two propositions above are deferred to the appendix. Having theoretically established the interplay between layer-wise reconstructions and the training mechanics, we now move to the empirical evaluation of HALI. We designed our experiments with the objective of addressing the following questions: Is HALI successful in improving the fidelity perceptual reconstructions? Does HALI induces a semantically meaningful representation of the observed data? Are the learned representations useful for downstream classification tasks? All of these questions are considered in turn in the following sections. We evaluated HALI on four datasets, CIFAR10 BID18, SVHN BID30, ImageNet 128x128 BID37 and CelebA BID25. We used two conditional hierarchies in all experiments with the Markov kernels parametrized by conditional isotropic Gaussians. For SVHN, CIFAR10 and CelebA the resolutions of two level latent variables are z 1 ∈ R 64×16×16 and z 2 ∈ R 256. For ImageNet, the resolutions is z 1 ∈ R 64×32×32 and z 2 ∈ R 256.For both the encoder and decoder, we use residual blocks BID10 with skip connections between the blocks in conjunction with batch normalization BID13. We use convolution with stride 2 for downsampling in the encoder and bilinear upsampling in the decoder. In the discriminator, we use consecutive stride 1 and stride 2 convolutions and weight normalization BID38. To regularize the discriminator, we apply dropout every 3 layers with a probability of retention of 0.2. We also add Gaussian noise with standard deviation of 0.2 at the inputs of the discriminator and the encoder. One of the desired objectives of a generative model is to reconstruct the input images from the latent representation. We show that HALI offers improved perceptual reconstructions relative to the (non-hierarchical) ALI model. First, we present reconstructions obtained on ImageNet. Reconstructions from SVHN and CIFAR10 can be seen in FIG7 in the appendix. Fig. 1 highlights HALI's ability to reconstruct the input samples with high fidelity. We observe that reconstructions from the first level of the hierarchy exhibit local differences in the natural images, while reconstructions from the second level of the hierarchy displays global change. Higher conditional reconstructions are more often than not reconstructed as a different member of the same class. Moreover, we show in Fig. 2 that this increase in reconstruction fidelity does not impact the quality of the generative samples from HALI's decoder. We further investigate the quality of the reconstructions with a quantitative assessment of the preservation of perceptual features in the input sample. For this evaluation task, we use the CelebA dataset where each image comes with a 40 dimensional binary attributes vector. A VGG-16 classifier BID42 was trained on the CelebA training set to classify the individual attributes. This trained model is then used to classify the attributes of the reconstructions from the validation set. We consider a reconstruction as being good if it preserves -as measured by the trained classifier -the attributes possessed by the original sample. We report a summary of the statistics of the classifier's accuracies in Table 1. We do this for three different models, VAE, ALI and HALI. An inspection of the table reveals that the proportion of attributes where HALI's reconstructions outperforms the other models is clearly dominant. Therefore, the encoder-decoder relationship of HALI better preserves the identifiable attributes compared to other models leveraging such relationships. Please refer to Table 5 in the appendix for the full table of attributes score. In the same spirit as Larsen et al. FORMULA0, we construct a metric by computing the Euclidean distance between the input images and their various reconstructions in the discriminator's feature space. More precisely, let · →D(·) be the embedding of the input to the pen-ultimate layer of the discriminator. We compute the discriminator embedded distance DISPLAYFORM0 where · → · 2 is the Euclidean norm. We then compute the average distances d c (x,x 1) and d c (x,x 2) over the ImageNet validation set. Fig. 3a shows that under d c, the average reconstruction errors for bothx 1 andx 2 decrease steadily as training advances. Furthermore, the reconstruction error under d c of the reconstructions from the first level of the hierarchy are uniformly bounded by above by those of the second. We note that while the VAEGAN model of BID20 explicitly minimizes the perceptual reconstruction error by adding this term to their loss function, HALI implicitly minimizes it during adversarial training, as shown in subsection 3.2.(a) (b) Figure 3: Comparison of average reconstruction error over the validation set for each level of reconstructions using the Euclidean (a) and discriminator embedded (b) distances. Using both distances, reconstructions errors for x ∼ T x|z 1 are uniformly below those for x ∼ T x|z 2. The reconstruction error using the Euclidean distance eventually stalls showing that the Euclidean metric poorly approximates the manifold of natural images. We now move on to assessing the quality of our learned representation through inpainting, visualizing the hierarchy and innovation vectors. Inpainting is the task of reconstructing the missing or lost parts of an image. It is a challenging task since sufficient prior information is needed to meaningfully replace the missing parts of an image. While it is common to incorporate inpainting-specific training BID45; BID35; BID34, in our case we simply use the standard HALI adversarial loss during training and reconstruct incomplete images during inference time. We first predict the missing portions from the higher level reconstructions followed by iteratively using the lower level reconstructions that are pixel-wise closer to the original image. FIG2 shows the inpaintings on center-cropped SVHN, CelebA and MS-COCO BID24 datasets without any blending post-processing or explicit supervision. The effectiveness of our model at this task is due the hierarchy -we can extract semantically consistent reconstructions from the higher levels of the hierarchy, then leverage pixel-wise reconstructions from the lower levels. To qualitatively show that higher levels of the hierarchy encode increasingly abstract representation of the data, we individually vary the latent variables and observe the effect. The process is as follow: we sample a latent code from the prior distribution z 2. We then multiply individual components of the vector by scalars ranging from −3 to 3. For z 1, we fix z 2 and multiply each feature map independently by scalars ranging from −3 to 3. In all cases these modified latent vectors are then decoded back to input data space. FIG4 (a) and (b) exhibit some of those decodings for z 2, while (c) and (d) do the same for the lower conditional z 1. The last column contain the decodings obtained from the originally sampled latent codes. We see that the representations learned in the z 2 conditional are responsible for high level variations like gender, while z 1 codes imply local/pixel-wise changes such as saturation or lip color. We sample a set of z2 vectors from the prior. We repeatedly replace a single relevant entry in each vector by a scalar ranging from −3 to 3 and decode. (c) and (d) follows the same process using the z1 latent space. With HALI, we can exploit the jointly learned hierarchical inference mechanism to modify actual data samples by manipulating their latent codes. We refer to these sorts of manipulations as latent semantic innovations. Consider a given instance from a dataset x ∼ q(x). Encoding x yieldsẑ 1 andẑ 2. We modifyẑ 2 by multiplying a specific entry by a scalar α. We denote the ing vector byẑ α 2. We decode the latter and getz α 1 ∼ T z1|z2. We decode the unmodified encoding vector and getz 1 ∼ T z1|ẑ2. We then form the innovation tensor η α =z 1 −z α 1. Finally, we subtract the innovation vector from the initial encoding, thus gettingẑ α 1 =ẑ 1 − η α, and samplex α ∼ T x|ẑ α. This method provides explicit control and allows us to carry out these variations on real samples in a completely unsupervised way. The are shown in FIG3. These were done on the CelebA validation set and were not used for training. We evaluate the usefulness of our learned representation for downstream tasks by quantifying the performance of HALI on attribute classification in CelebA and on a semi-supervised variant of the MNIST digit classification task. Following the protocol established by BID0; BID25, we train 40 linear SVMs on HALI encoder representations (i.e. we utilize the inference network) on the CelebA validation set and subsequently measure performance on the test set. As in BID0 BID11; BID14, we report the balanced accuracy in order to evaluate the attribute prediction performance. We emphasize that, for this experiment, the HALI encoder and decoder were trained in on entirely unsupervised data. Attribute labels were only used to train the linear SVM classifiers. A summary of the are reported in TAB2. HALI's unsupervised features surpass those of VAE and ALI, but more remarkably, they outperform the best handcrafted features by a wide margin BID46. Furthermore, our approach outperforms a number of supervised BID11 and deeply supervised BID25 features. Table 6 in the appendix arrays the per attribute. Std # Best Triplet-kNN BID40 71.55 12.61 0 PANDA BID46 76.95 13.33 0 Anet BID25 79.56 12.17 0 LMLE-kNN BID11 MNIST (# errors) VAE (M1+M2) BID17 233 ± 14 VAT BID29 136 CatGAN BID43 191 ± 10 Adversarial Autoencoder BID28 190 ± 10 PixelGAN BID27 108 ± 15 ADGM BID26 96 ± 2 Feature-Matching GAN 93 ± 6.5 Triple GAN BID22 91 ± 58 GSSLTRABG BID3 79.5 ± 9.8 HALI (ours) 73 Table 3: Comparison on semi-supervised learning with state-of-the-art methods on MNIST with 100 labels instance per class. Only methods without data augmentation are included. The HALI hierarchy can also be used in a more integrated semi-supervised setting, where the encoder also receives a training signal from the supervised objective. The currently most successful approach to semi-supervised in adversarially trained generative models are built on the approach introduced by. This formalism relies on exploiting the discriminator's feature to differentiate between the individual classes present in the labeled data as well as the generated samples. Taking inspiration from BID28 BID27, we adopt a different approach that leverages the Markovian hierarchical inference network made available by HALI, DISPLAYFORM0 Where z = enc(x + σ), with ∼ N (0, I), and y is a categorical random variable. In practice, we characterize the conditional distribution of y given z by a softmax. The cost of the generator is then augmented by a supervised cost. Let us write D sup as the set of pairs all labeled instance along with their label, the supervised cost reads DISPLAYFORM1 We showcased this approach on a semi-supervised variant of MNIST digit classification task with 100 labeled examples evenly distributed across classes. Table 3 shows that HALI achieves a new state-of-the-art for this setting. Note that unlike BID3, HALI uses no additional regularization. In this paper, we introduced HALI, a novel adversarially trained generative model. HALI learns a hierarchy of latent variables with a simple Markovian structure in both the generator and inference networks. We have shown both theoretically and empirically the advantages gained by extending the ALI framework to a hierarchy. While there are many potential applications of HALI, one important future direction of research is to explore ways to render the training process more stable and straightforward. GANs are well-known to be challenging to train and the introduction of a hierarchy of latent variables only adds to this. Operation Kernel Strides Feature maps BN/WN? Dropout Nonlinearity DISPLAYFORM0 DISPLAYFORM1 Concatenate D(x, z 1) and z 2 along the channel axis B PROOFS Lemma 1. Let f be a valid f-divergence generator. Let p and q be joint distributions over a random vector x. Let x A be any strict subset of x and x −A its complement, then DISPLAYFORM2 DISPLAYFORM3 Proof. By definition, we have DISPLAYFORM4 Using that f is convex, Jensen's inequality yields DISPLAYFORM5 Simplifying the inner expectation on the right hand side, we conclude that DISPLAYFORM6 Lemma 2 (Kullback-Leibler's upper bound by Jensen-Shannon). Assume that p and q are two probability distribution absolutely continuous with respect to each other. Moreover, assume that q is bounded away from zero. Then, there exist a positive scalar K such that DISPLAYFORM7 Proof. We start by bounding the Kullblack-Leibler divergence by the χ 2 -distance. We have DISPLAYFORM8 The first inequality follows by Jensen's inequality. The third inequality follows by the Taylor expansion. Recall that both the χ 2 -distance and the Jensen-Shanon divergences are f-divergences with generators given by f χ 2 (t) = (t − 1) 2 and f JS (t) = u log(2t t+1) + log(2t t+1), respectively. We form the function t → h(t) = f χ 2 (t) f JS (t). h is strictly increasing on [0, ∞). Since we are assuming q to be bounded away from zero, we know that there is a constant c 1 such that q(x) > c 1 for all x. Subsequently for all x, we have that q(x) ). Intergrating with respect to q, we conclude DISPLAYFORM9 Proposition 3. Assuming q(x, z l) and p(x, z l) are positive for any l ∈ {1, . . ., L}. We have DISPLAYFORM10 Where H(x | z l) is computed under the encoder's distribution q(x, z l)Proof. By elementary manipulations we have. DISPLAYFORM11 Where the conditional entropy H(x l | z l) is computed q(x, z l). By the non-negativity of the KL-divergence we obtain DISPLAYFORM12 Using lemma 2, we have DISPLAYFORM13 The Jensen-Shanon divergence being f-divergence, using Lemma 1, we conclude DISPLAYFORM14 Proposition 4. For any given latent variable z l, the reconstruction likelihood E x∼qx [E z∼T z l |x [− log p(x | z l)]] is an upper bound on H(x | z l).Proof. By the non-negativity of the Kullback-Leibler divergence, we have that DISPLAYFORM15. Integrating over the marginal and applying Fubini's theorem yields DISPLAYFORM16 where the conditional entropy H(x | z l) is computed under the encoder distribution.

Represent the short summary for finding the corresponding scientific article (long)!

Adversarially trained hierarchical generative model with robust and semantically learned latent representation.

Represent the TLDR for finding the corresponding scientific article

We apply the copula transformation to the Deep Information Bottleneck which leads to restored invariance properties and a disentangled latent space with superior predictive capabilities.


Question: Represent the text to find a very short summary (1-2 sentences max).

Question: Conservation laws are considered to be fundamental laws of nature. It has broad application in many fields including physics, chemistry, biology, geology, and engineering. Solving the differential equations associated with conservation laws is a major branch in computational mathematics. Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods. In this paper, we are the first to explore the possibility and benefit of solving nonlinear conservation laws using deep reinforcement learning. As a proof of concept, we focus on 1-dimensional scalar conservation laws. We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner. We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision making process and the numerical schemes learned in such a way can easily enforce long-term accuracy. Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach. In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts. Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and how well it generalizes. Our code is released anomynously at \url{https://github.com/qwerlanksdf/L2D}. Conservation laws are considered to be one of the fundamental laws of nature, and has broad applications in multiple fields such as physics, chemistry, biology, geology, and engineering. For example, Burger's equation, a very classic partial differential equation (PDE) in conservation laws, has important applications in fluid mechanics, nonlinear acoustics, gas dynamics, and traffic flow. Solving the differential equations associated with conservation laws has been a major branch of computational mathematics (; 2002), and a lot of effective methods have been proposed, from classic methods such as the upwind scheme, the Lax-Friedrichs scheme, to the advanced ones such as the ENO/WENO schemes , the flux-limiter methods , and etc. In the past few decades, these traditional methods have been proven successful in solving conservation laws. Nonetheless, the design of some of the high-end methods heavily relies on expert knowledge and the coding of these methods can be a laborious process. To ease the usage and potentially improve these traditional algorithms, machine learning, especially deep learning, has been recently incorporated into this field. For example, the ENO scheme requires lots of'if/else' logical judgments when used to solve complicated system of equations or high-dimensional equations. This very much resembles the old-fashioned expert systems. The recent trend in artificial intelligence (AI) is to replace the expert systems by the so-called'connectionism', e.g., deep neural networks, which leads to the recent bloom of AI. Therefore, it is natural and potentially beneficial to introduce deep learning in traditional numerical solvers of conservation laws. In the last few years, neural networks (NNs) have been applied to solving ODEs/PDEs or the associated inverse problems. These works can be roughly classified into three categories according to the way that the NN is used. The first type of works propose to harness the representation power of NNs, and are irrelevant to the numerical discretization based methods. For example, Raissi et al. (2017a; b); treated the NNs as new ansatz to approximate solutions of PDEs. It was later generalized by to allow randomness in the solution which is trained using policy gradient. More recent works along this line include (; ;). Besides, several works have focused on using NNs to establish direct mappings between the parameters of the PDEs (e.g. the coefficient field or the ground state energy) and their associated solutions (; ; b).; proposed a method to solve very high-dimensional PDEs by converting the PDE to a stochastic control problem and use NNs to approximate the gradient of the solution. The second type of works focus on the connection between deep neural networks (DNNs) and dynamic systems (; ; ; b;). These works observed that there are connections between DNNs and dynamic systems (e.g. differential equations or unrolled optimization algorithms) so that we can combine deep learning with traditional tools from applied and computational mathematics to handle challenging tasks in inverse problems (b; a;).The main focus of these works, however, is to solve inverse problems, instead of learning numerical discretizations of differential equations. Nonetheless, these methods are closely related to numerical differential equations since learning a proper discretization is often an important auxiliary task for these methods to accurately recover the form of the differential equations. The third type of works, which target at using NNs to learn new numerical schemes, are closely related to our work. However, we note that these works mainly fall in the setting of supervised learning (SL). For example, proposed to integrate NNs into high-order numerical solvers to predict artificial viscosity; trained a multilayer perceptron to replace traditional indicators for identifying troubled-cells in high-resolution schemes for conservation laws. These works greatly advanced the development in machine learning based design of numerical schemes for conservation laws. Note that in , the authors only utilized the one-step error to train the artificial viscosity networks without taking into account the longterm accuracy of the learned numerical scheme. first constructed several functions with known regularities and then used them to train a neural network to predict the location of discontinuity, which was later used to choose a proper slope limiter. Therefore, the training of the NNs is separated from the numerical scheme. Then, a natural question is whether we can learn discretization of differential equations in an end-to-end fashion and the learned discrete scheme also takes long-term accuracy into account. This motivates us to employ reinforcement learning to learn good solvers for conservation laws. The main objective of this paper is to design new numerical schemes in an autonomous way. We propose to use reinforcement learning (RL) to aid the process of solving the conservation laws. To our best knowledge, we are the first to regard numerical PDE solvers as a MDP and to use (deep) RL to learn new solvers. We carefully design the proposed RL-based method so that the learned policy can generate high accuracy numerical schemes and can well generalize in varied situations. Details will be given in section 3. Here, we first provide a brief discussion on the benefits of using RL to solve conservation laws (the arguments apply to general evolution PDEs as well): • Most of the numerical solvers of conservation law can be interpreted naturally as a sequential decision making process (e.g., the approximated grid values at the current time instance definitely affects all the future approximations). Thus, it can be easily formulated as a Markov Decision Process (MDP) and solved by RL. • In almost all the RL algorithms, the policy π (which is the AI agent who decides on how the solution should be approximated locally) is optimized with regards to the values Q π (s 0, a 0) = r(s 0, a 0) + ∞ t=1 γ t r(s t, a t), which by definition considers the long-term accumulated reward (or, error of the learned numerical scheme), thus could naturally guarantee the long-term accuracy of the learned schemes, instead of greedily deciding the local approximation which is the case for most numerical PDEs solvers. Furthermore, it can gracefully handle the cases when the action space is discrete, which is in fact one of the major strength of RL. • By optimizing towards long-term accuracy and effective exploration, we believe that RL has a good potential in improving traditional numerical schemes, especially in parts where no clear design principles exist. For example, although the WENO-5 scheme achieves optimal order of accuracy at smooth regions of the solution , the best way of choosing templates near singularities remains unknown. Our belief that RL could shed lights on such parts is later verified in the experiments: the trained RL policy demonstrated new behaviours and is able to select better templates than WENO and hence approximate the solution better than WENO near singularities. • Non-smooth norms such as the infinity norm of the error is often used to evaluate the performance of the learned numerical schemes. As the norm of the error serves as the loss function for the learning algorithms, computing the gradient of the infinity norm can be problematic for supervised learning, while RL does not have such problem since it does not explicitly take gradients of the loss function (i.e. the reward function for RL). • Learning the policy π within the RL framework makes the algorithm meta-learning-like (; ; ; ;). The learned policy π can decide on which local numerical approximation to use by judging from the current state of the solution (e.g. local smoothness, oscillatory patterns, dissipation, etc). This is vastly different from regular (non-meta-) learning where the algorithms directly make inference on the numerical schemes without the aid of an additional network such as π. As subtle the difference as it may seem, meta-learning-like methods have been proven effective in various applications such as in image restoration (; a;). See for a comprehensive survey on meta-learning. • Another purpose of this paper is to raise an awareness of the connection between MDP and numerical PDE solvers, and the general idea of how to use RL to improve PDE solvers or even finding brand new ones. Furthermore, in computational mathematics, a lot of numerical algorithms are sequential, and the computation at each step is expert-designed and usually greedy, e.g., the conjugate gradient method, the fast sweeping method , matching pursuit , etc. We hope our work could motivate more researches in combining RL and computational mathematics, and stimulate more exploration on using RL as a tool to tackle the bottleneck problems in computational mathematics. Our paper is organized as follows. In section 2 we briefly review 1-dimensional conservation laws and the WENO schemes. In section 3, we discuss how to formulate the process of numerically solving conservation laws into a Markov Decision Process. Then, we present details on how to train a policy network to mimic human expert in choosing discrete schemes in a spatial-temporary adaptive manner by learning upon WENO. In section 4, we conduct numerical experiments on 1-D conservation laws to demonstrate the performance of our trained policy network. Our experimental show that the trained policy network indeed learned to adaptively choose good discrete schemes that offer better than the state-of-the-art WENO scheme which is 5th order accurate in space and 4th order accurate in time. This serves as an evidence that the proposed RL framework has the potential to design high-performance numerical schemes for conservation laws in a data-driven fashion. Furthermore, the learned policy network generalizes well to other situations such as different initial conditions, mesh sizes, temporal discrete schemes, etc. The paper ends with a in section 5, where possible future research directions are also discussed. In this paper, we consider solving the following 1-D conservation laws: For example, f = u 2 2 is the famous Burger's Equation. We discretize the (x, t)-plane by choosing a mesh with spatial size ∆x and temporal step size ∆t, and define the discrete mesh points (x j, t n) by We denote x j+ 2 )∆x. The finite difference methods will produce approximations U n j to the solution u(x j, t n) on the given discrete mesh points. We denote pointwise values of the true solution to be u n j = u(x j, t n), and the true point-wise flux values to be f WENO (Weighted Essentially Non-Oscillatory) is a family of high order accurate finite difference schemes for solving hyperbolic conservation laws, and has been successful for many practical problems. The key idea of WENO is a nonlinear adaptive procedure that automatically chooses the smoothest local stencil to reconstruct the numerical flux. Generally, a finite difference method solves Eq.1 by using a conservative approximation to the spatial derivative of the flux: where u j (t) is the numerical approximation to the point value u(x j, t) andf j+ 1 2 is the numerical flux generated by a numerical flux policŷ which is manually designed. Note that the term "numerical flux policy" is a new terminology that we introduce in this paper, which is exactly the policy we shall learn using RL. In WENO, π f works as follows. Using the physical flux values {f j−2, f j−1, f j}, we could obtain a 3 th order accurate polynomial interpolationf, where the indices {j − 2, j − 1, j} is called a'stencil'. We could also use the stencil {j−1, j, j+1}, {j, j+1, j+2} or {j+1, j+2, j+3} to obtain another three interpolantŝ f. The key idea of WENO is to average (with properly designed weights) all these interpolants to obtain the final reconstruction:f j+ The weight w i depends on the smoothness of the stencil. A general principal is: the smoother is the stencil, the more accurate is the interpolant and hence the larger is the weight. To ensure convergence, we need the numerical scheme to be consistent and stable . It is known that WENO schemes as described above are consistent. For stability, upwinding is required in constructing the flux. The most easy way is to use the sign of the Roe speedā j+ to determine the upwind direction: ifā j+ 1 2 ≥ 0, we only average among the three interpolantsf. Some further thoughts. WENO achieves optimal order of accuracy (up to 5) at the smooth region of the solutions , while lower order of accuracy at singularities. The key of the WENO method lies in how to compute the weight vector (w 1, w 2, w 3, w 4), which primarily depends on the smoothness of the solution at local stencils. In WENO, such smoothness is characterized by handcrafted formula, and was proven to be successful in many practical problems when coupled with high-order temporal discretization. However, it remains unknown whether there are better ways to combine the stencils so that optimal order of accuracy in smooth regions can be reserved while, at the same time, higher accuracy can be achieved near singularities. Furthermore, estimating the upwind directions is another key component of WENO, which can get quite complicated in high-dimensional situations and requires lots of logical judgments (i.e. "if/else"). Can we ease the (some time painful) coding and improve the estimation at the aid of machine learning? In this section we present how to employ reinforcement learning to solve the conservation laws given by Eq.1. To better illustrate our idea, we first show in general how to formulate the process of numerically solving a conservation law into an MDP. We then discuss how to incorporate a policy network with the WENO scheme. Our policy network targets at the following two key aspects of WENO: Can we learn to choose better weights to combine the constructed fluxes? Can we learn to automatically judge the upwind direction, without complicated logical judgments? Compute the numerical fluxf j+s ), e.g., using the WENO scheme ), e.g., using the Euler scheme As shown in Algorithm 1, the procedure of numerically solving a conservation law is naturally a sequential decision making problem. The key of the procedure is the numerical flux policy π f and the temporal scheme π t as shown in line 6 and 8 in Algorithm 1. Both policies could be learned using RL. However, in this paper, we mainly focus on using RL to learn the numerical flux policy π f, while leaving the temporal scheme π t with traditional numerical schemes such as the Euler scheme or the Runge-Kutta methods. A quick review of RL is given in the appendix. Now, we show how to formulate the above procedure as an MDP and the construction of the state S, action A, reward r and transition dynamics P. Algorithm 2 shows in general how RL is incorporated into the procedure. In Algorithm 2, we use a single RL agent. Specifically, when computing U n j: • The state for the RL agent is s j+s ), where g s is the state function. • In general, the action of the agent is used to determine how the numerical fluxesf n j+ 1 2 andf n j− 1 2 is computed. In the next subsection, we detail how we incorporate a n j to be the linear weights of the fluxes computed using different stencils in the WENO scheme. • The reward should encourage the agent to generate a scheme that minimizes the error between its approximated value and the true value. Therefore, we define the reward function as r n j = g r (U n j−r−1 − u n j−r−1, · · ·, U n j+s − u n j+s), e.g., a simplest choice is g r = −|| · || 2. • The transition dynamics P is fully deterministic, and depends on the choice of the temporal scheme at line 10 in Algorithm 2. Note that the next state can only be constructed when we have obtained all the point values in the next time step, i.e., s n+1 j = g s (U n j−r−1, ..., U n j+s) does not only depends on action a n j, but also on actions a n j−r−1,..., a n j+s (action a n j can only determine the value U n j). This subtlety can be resolved by viewing the process under the framework of multi-agent RL, in which at each mesh point j we use a distinct agent A RL j, and the next state s n+1 j = g s (U n j−r−1, ..., U n j+s) depends on these agents' joint action a n j = (a n j−r−1, ..., a n j+s). However, it is impractical to train J different agents as J is usually very large, therefore we enforce the agents at different mesh point j to share the same weight, which reduces to case of using just a single agent. The single agent can be viewed as a counterpart of a human designer who decides on the choice of a local scheme based on the current state in traditional numerical methods. Compute the action a ), e.g., the Euler scheme Compute the reward r. 14 Return the well-trained RL policy π RL. We now present how to transfer the actions of the RL policy to the weights of WENO fluxes. Instead of directly using π RL to generate the numerical flux, we use it to produce the weights of numerical fluxes computed using different stencils in WENO. Since the weights are part of the configurations of the WENO scheme, our design of action essentially makes the RL policy a meta-learner, and enables more stable learning and better generalization power than directly generating the fluxes. Specifically, at point x j (here we drop the time superscript n for simplicity), to compute the numerical fluxf j−. Note that the determination of upwind direction is automatically embedded in the RL policy since it generates four weights at once. For instance, when the roe speedā j+ ≈ 0. Note that the upwind direction can be very complicated in a system of equations or in the high-dimensional situations, and using the policy network to automatically embed such a process could save lots of efforts in algorithm design and implementation. Our numerical experiments show that π RL can indeed automatically determine upwind directions for 1D scalar cases. Although this does not mean that it works for systems and/or in high-dimensions, it shows the potential of the proposed framework and value for further studies. In this section, we describe training and testing of the proposed RL conservation law solver and compare it with WENO. More comparisons and discussions can be found in the appendix. In this subsection, we explain the general training setup. We train the RL policy network on the Burger's equation, whose flux is computed as f (u) = 1 2 u 2. In all the experiments, we set the left-shift r = 2 and the right shift s = 3. The state function g s (s j) = g s (U j−r−1, ..., U j+s) will generate two vectors: s l = (f j−r−1, ..., f j+s−1,ā j− respectively. s l and s r will be passed into the same policy neural network π RL θ to produce the desired actions, as described in section 3.2. The reward function g r simply computes the infinity norm, i.e., g r (U j−r−1 − u j−r−1, ..., U j+s − u j+s) = −||(U j−r−1 − u j−r−1, ..., U j+s − u j+s)|| ∞. The policy network π RL θ is a feed-forward Multi-layer Perceptron with 6 hidden layers, each has 64 neurons and use Relu as the activation function. We use the Deep Deterministic Policy Gradient Algorithm to train the RL policy. To guarantee the generalization power of the trained RL agent, we randomly sampled 20 initial conditions in the form u 0 (x) = a + b · func(cπx), where |a| + |b| ≤ 3.5, func ∈ {sin, cos} and c ∈ {2, 4, 6}. The goal of generating such kind of initial conditions is to ensure they have similar degree of smoothness and thus similar level of difficulty in learning. The computation domain is −1 ≤ x ≤ 1 and 0 ≤ t ≤ 0.8 with ∆x = 0.02, ∆t = 0.004, and evolve steps N = 200 (which ensures the appearance of shocks). When training the RL agent, we use the Euler scheme for temporal discretization. The true solution needed for reward computing is generated using WENO on the same computation domain with ∆x = 0.001, ∆t = 0.0002 and the 4th order Runge-Kutta (RK4). In the following, we denote the policy network that generates the weights of the WENO fluxes (as described in section 3.2) as RL-WENO. We randomly generated another different 10 initial conditions in the same form as training for testing. We compare the performance of RL-WENO and WENO. We also test whether the trained RL policy can generalize to different temporal discretization schemes, mesh sizes and flux functions that are not included in training. Table 1 and Table 2 with the 2-norm taking over all x) between the approximated solution U and the true solution u, averaged over 250 evolving steps (T = 1.0) and 10 random initial values. Numbers in the bracket shows the standard deviation over the 10 initial conditions. Several entries in the table are marked as'-' because the corresponding CFL number is not small enough to guarantee convergence. Recall that training of the RL-WENO was conducted with Euler time discretization, (∆x, ∆t) = (0.02, 0.004), T = 0.8 and f (u) = 1 2 u 2. Our experimental show that, compared with the high order accurate WENO (5th order accurate in space and 4th order accurate in time), the linear weights learned by RL not only achieves smaller errors, but also generalizes well to: 1) longer evolving time (T = 0.8 for training and T = 1.0 for testing); 2) new time discretization schemes (trained on Euler, tested on RK4); 3) new mesh sizes (see Table 1 and Table 2 for of varied ∆x and ∆t); and 4) a new flux function (trained on f (u) = Table 2 ). Figure 1 shows some examples of the solutions. As one can see, the solutions generated by RL-WENO not only achieve the same accuracy as WENO at smooth regions, but also have clear advantage over WENO near singularities which is particularly challenging for numerical PDE solvers and important in applications. Figure 2 shows that the learned numerical flux policy can indeed correctly determine upwind directions and generate local numerical schemes in an adaptive fashion. More interestingly, Figure 2 further shows that comparing to WENO, RL-WENO seems to be able to select stencils in a different way from it, and eventually leads to a more accurate solution. This shows that the proposed RL framework has the potential to surpass human experts in designing numerical schemes for conservation laws. In this paper, we proposed a general framework to learn how to solve 1-dimensional conservation laws via deep reinforcement learning. We first discussed how the procedure of numerically solving conservation laws can be naturally cast in the form of Markov Decision Process. We then elaborated how to relate notions in numerical schemes of PDEs with those of reinforcement learning. In particular, we introduced a numerical flux policy which was able to decide on how numerical flux should be designed locally based on the current state of the solution. We carefully design the action of our RL policy to make it a meta-learner. Our numerical experiments showed that the proposed RL based solver was able to outperform high order WENO and was well generalized in various cases. As part of the future works, we would like to consider using the numerical flux policy to inference more complicated numerical fluxes with guaranteed consistency and stability. Furthermore, we can use the proposed framework to learn a policy that can generate adaptive grids and the associated numerical schemes. Lastly, we would like consider system of conservation laws in 2nd and 3rd dimensional space. A COMPLEMENTARY EXPERIMENTS We first note that most of the neural network based numerical PDE solvers cited in the introduction requires retraining when the initialization, terminal time, or the form of the PDE is changed; while the proposed RL solver is much less restricted as shown in our numerical experiments. This makes proper comparisons between existing NN-based solvers and our proposed solver very difficult. Therefore, to demonstrate the advantage of our proposed RL PDE solver, we would like to propose a new SL method that does not require retraining when the test setting (e.g. initialization, flux function, etc.) is different from the training. However, as far as we are concerned, it is challenging to design such SL methods without formulating the problem into an MDP. One may think that we can use WENO to generate the weights for the stencil at a particular grid point on a dense grid, and use the weights of WENO generated from the dense grid as the label to train a neural network in the coarse grid. But such setting has a fatal flaw in that the stencils computed in the dense grids are very different from those in the coarse grids, especially near singularities. Therefore, good weights on dense grids might perform very poorly on coarse grids. In other words, simple imitation of WENO on dense grids is not a good idea. One might also argue that instead of learning the weights of the stencils, we could instead generate the discrete operators, such as the spatial discretization of (u), etc., on a dense grid, and then use them as labels to train a neural network in the supervised fashion on a coarse grid. However, the major problem with such design is that there is no guarantee that the learned discrete operators obey the conservation property of the equations, and thus they may also generalize very poorly. After formulating the problem into a MDP, there is indeed one way that we can use back-propagation (BP) instead of RL algorithms to optimize the policy network. Because all the computations on using the stencils to calculate the next-step approximations are differentiable, we can indeed use SL to train the weights. One possible way is to minimize the error (e.g. 2 norm) between the approximated and the true values, where the true value is pre-computed using a more accurate discretization on a fine mesh. The framework to train the SL network is described in Algorithm 3. Note that the framework to train the SL network is essentially the same as that of the proposed RL-WENO (Algorithm 2). The only difference is that we train the SL network using BP and the RL network using DDPG. Compute ), e.g., the Euler scheme U However, we argue that the main drawback of using SL (BP) to optimize the stencils in such a way is that it cannot enforce long-term accuracy and thus cannot outperform the proposed RL-WENO. To support such claims, we have added experiments using SL to train the weights of the stencils, and the are shown in table 3 and 4. The SL policy is trained till it achieves very low loss (i.e., converges) in the training setting. However, as shown in the table, the SL-trained policy does not perform well overall. To improve longer time stability, one may argue that we could design the loss of SL to be the accumulated loss over multiple prediction steps, but in practice as the dynamics of our problem (computations for obtaining multiple step approximations) is highly non-linear, thus the gradient flow through multiple steps can be highly numerically unstable, making it difficult to obtain a decent . As mentioned in section 2.2, WENO itself already achieves an optimal order of accuracy in the smooth regions. Since RL-WENO can further improve upon WENO, it must have obtained higher accuracy especially near singularities. Here we provide additional demonstrations on how RL-WENO performs in the smooth/singular regions. We run RL-WENO and WENO on a set of initial conditions, and record the approximation errors at every locations and then separate the errors in the smooth and singular regions for every time step. We then compute the distribution of the errors on the entire spatial-temporal grids with multiple initial conditions. The are shown in figure 3. In figure 3, the x-axis is the logarithmic (base 10) value of the error and the y-axis is the number of grid points whose error is less than the corresponding value on the x-axis, i.e., the accumulated distribution of the errors. The show that RL-WENO indeed performs better than WENO near singularities. RL-WENO even achieves better accuracy than WENO in the smooth region when the flux function is A.3 INFERENCE TIME OF RL-WENO AND WENO In this subsection we report the inference time of RL-WENO and WENO. Although the computation complexity of the trained RL policy (a MLP) is higher than that of WENO, we could parallel and accelerate the computations using GPU. Our test is conducted in the following way: for each grid size ∆x, we fix the initial condition as u 0 (x) = 1 + cos(6πx), the evolving time T = 0.8 and the flux function f = u 2. We then use RL-WENO and WENO to solve the problem 20 times, and report the average running time. For completeness, we also report the relative error of RL-WENO and WENO in each of these grid sizes in table 6. Note that the relative error is computed on average of several initial functions, and our RL-WENO policy is only trained on grid (∆x, ∆t) = (0.02, 0.004). For RL-WENO, we test it on both CPU and on GPU; For WENO, we test it purely on CPU, with a well-optimized version (e.g., good numpy vectorization in python), and a poor-implemented version (e.g., no vectorization, lots of loops). The CPU used for the tests is a custom Intel CORE i7, and the GPU is a custom NVIDIA GTX 1080. The are shown in From the table we can tell that as ∆x decreases, i.e., as the grid becomes denser, all methods, except for the RL-WENO (GPU), requires significant more time to finish the computation. The reason that the time cost of the GPU-version of RL-WENO does not grow is that on GPU, we can compute all approximations in the next step (i.e., to compute (U, which dominates the computation cost of the algorithm) together in parallel. Thus, the increase of grids does not affect much of the computation time. Therefore, for coarse grid, well-optimized WENO indeed has clear speed advantage over RL-WENO (even on GPU), but on a much denser grid, RL-WENO (GPU) can be faster than well-optimized WENO by leveraging the paralleling nature of the algorithm. B REVIEW OF REINFORCEMENT LEARNING B.1 REINFORCEMENT LEARNING Reinforcement Learning (RL) is a general framework for solving sequential decision making problems. Recently, combined with deep neural networks, RL has achieved great success in various tasks such as playing video games from raw screen inputs , playing Go , and robotics control . The sequential decision making problem RL tackles is usually formulated as a Markov Decision Process (MDP), which comprises five elements: the state space S, the action space A, the reward r: S × A → R, the transition probability of the environment P: S × A × S →, and the discounting factor γ. The interactions between an RL agent and the environment forms a trajectory τ = (s 0, a 0, r 0, ..., s T, a T, r T, ...). The return of τ is the discounted sum of all its future rewards: Similarly, the return of a state-action pair (s t, a t) is: A policy π in RL is a probability distribution on the action A given a state S: π: S × A →. We say a trajectory τ is generated under policy π if all the actions along the trajectory is chosen following π, i.e., τ ∼ π means a t ∼ π(·|s t) and s t+1 ∼ P (·|s t, a t). Given a policy π, the value of a state s is defined as the expected return of all the trajectories when the agent starts at s and then follows π: Similarly, the value of a state-action pair is defined as the expected return of all trajectories when the agent starts at s, takes action a, and then follows π: As aforementioned in introduction, in most RL algorithms the policy π is optimized with regards to the values Q π (s, a), thus naturally guarantees the long-term accumulated rewards (in our setting, the long-term accuracy of the learned schemes). Bellman Equation, one of the most important equations in RL, connects the value of a state and the value of its successor state: The goal of RL is to find a policy π to maximize the expected discounted sum of rewards starting from the initial state s 0, J(π) = E s0∼ρ [V π (s 0)], where ρ is the initial state distribution. If we parameterize π using θ, then we can optimize it using the famous policy gradient theorem: where ρ π θ is the state distribution deduced by the policy π θ. In this paper we focus on the case where the action space A is continuous, and a lot of mature algorithms has been proposed for such a case, e.g., the Deep Deterministic Policy Gradient (DDPG) , the Trust Region Policy Optimization algorithm , and etc.

Represent this text to find a much much longer article about it with say hundreds-thousands of words:

We observe that numerical PDE solvers can be regarded as Markov Desicion Processes, and propose to use Reinforcement Learning to solve 1D scalar Conservation Laws

Represent the TLDR for finding the corresponding scientific article.

We identify and formalize the memorization problem in meta-learning and solve this problem with novel meta-regularization method, which greatly expand the domain that meta-learning can be  applicable to and effective on.