Question: Represent the scientific article to find a short TLDR:

Question: Learning can be framed as trying to encode the mutual information between input and output while discarding other information in the input. Since the distribution between input and output is unknown, also the true mutual information is. To quantify how difficult it is to learn a task, we calculate a observed mutual information score by dividing the estimated mutual information by the entropy of the input. We substantiate this score analytically by showing that the estimated mutual information has an error that increases with the entropy of the data. Intriguingly depending on how the data is represented the observed entropy and mutual information can vary wildly. There needs to be a match between how data is represented and how a model encodes it. Experimentally we analyze image-based input data representations and demonstrate that performance outcomes of extensive network architectures searches are well aligned to the calculated score. Therefore to ensure better learning outcomes, representations may need to be tailored to both task and model to align with the implicit distribution of the model. Sometimes perspective is everything. While the information content of encoded data may not change when the way it is represented changes, its usefulness can vary dramatically (see Fig. 1). A "useful" representation then is one that makes it easy to extract information of interest. This in turn very much depends on who or which algorithm is extracting the information. Evidently the way data is encoded and how a model "decodes" the information needs to match. Historically, people have invented a large variety of "data representations" to convey information. An instance of this theme is the heliocentric vs. geocentric view of the solar system. Before the heliocentric viewpoint was widely accepted, scholars had already worked out the movements of the planets . The main contribution of the new perspective was that now the planetary trajectories were simple ellipses instead of more complicated movements involving loops 1. In a machine learning context, many have experimented with finding good data representations for specific tasks such as speech recognition , different color spaces for face recognition , for increased robustness in face detection , and many others. Yet no clear understanding has emerged of why a given representation is more suited to one task but less for another. We cast the problem of choosing the data representation for learning as one of determining the ease of encoding the relationship between input and output which depends both on how the data is represented and which model is supposed to encode it. Contribution: In this work, we argue that learning a task is about encoding the relationship between input and output. Each model implicitly has a way of encoding information, where some variations in the data are easier to encode than others. Armed with this insight, we empirically evaluate different data representations and record what impact data representations have on learning outcomes and types of networks found by automated network optimization. Most interestingly, we are able Figure 1: These images contain different representations of the same information. However, one of the two is much easier for us to understand. We posit that, for our nervous system, one of the two images has a higher observed mutual information for the task of recognizing the person. to show that relative learning outcomes can be predicted by an empirical mutual information score, which we coin Observed Mutual Information (OMI) score. This work aims to bring us a bit closer to understanding what makes a given learning task easier or harder. While there appears to be little work on this question, a fresh-eyed excursion into what makes an optimization easier or harder has been ventured before. Data representations: Data representations have been optimized for a long time. In fact there is a rich theory of linear invertible representations for both finite and infinite dimensional spaces called Frame Theory . Specific popular examples of frames are Wavelets and Curvelets . Empirically tested only on Imagenet, Uber research showed that using a data representation closer to how JPEG encodes information, may help to create faster residual network architecture with slightly better performance. In a similar spirit in a robotics context, Grassmann and Kahrs evaluated learning performance on approximating robot dynamics using various common robot dynamics data representations such as Euler angles. What is more common in deep learning is to adapt the network architecture to the task at hand. An intriguing recent example taking this idea a step further are Weight Agnostic Neural Networks which have been designed to already "function" on a task even when starting from randomly initialized weights. Measuring learning difficulty: There appears to be little newer literature on the question, yet already posed the question of how to measure how easy or difficult a learning task is in the nineties. Similar to our own findings, they related the difficulty to information theoretic measures called the information gain (mutual information) and the information gain ratio (very similar to our proposed OMI value) introduced in the context of decision trees by Quinlan (1986; 2014). Sadly, this interesting line of inquiry does not appear to have received much attention since. take a different road by comparing several possible scores to assess the difficulty of classification learning problems such as linear separability and feature efficiency. More commonly, instead of judging task difficulty, there is a vast literature on feature selection , e.g. judging how suitable a feature is for a given learning problem. Desirable features are reliably selected for a learning task (Meinshausen & Bühlmann, 2010) and ideally are highly predictive of the output variable. Relating the overall difficulty of selecting good features to how difficult a learning task is, has not been established to our understanding. The objective of learning can be phrased as finding a function that minimizes the uncertainty of the output given the input while discarding as much task irrelevant information as possible. In information theoretic language, this viewpoint was introduced by and extended by in the form of the objective of the Information Bottleneck (IB) Lagrangian. Given an input x, a model encoding z, an output y and mutual information I(·; ·), the aims to minimize the following cost function: L(p(z|x)) = I(x; z) − βI(y; z) The model is supposed to find an encoding z of data x that maximizes the mutual information to output y while also minimizing the mutual information with x. It becomes apparent that the "ideal" learning task is one where all of the information in x is highly relevant to output y. In this case minimizing I(z; x) becomes unnecessary and the learning algorithm can place all of its focus on simply associating x with y. A highly difficult task would be one where I(x; y) is very small in absolute terms or where estimating I(x; y) is difficult for the chosen model. An attractive option to evaluate how difficult a learning task is thus is measuring its mutual information I(x; y). Empirically estimating the mutual information however leads to errors that have a bias with a bound proportional to the entropy H(x). To adjust, we divide by the entropy term, leading to a score of mutual information which takes into account the uncertainty of the estimate. In the following we bound the deviation of the true mutual information from the estimated mutual information. This estimated mutual information we coin Observed Mutual Information since it is the information, dependent on representation and model, that we may ultimately be able to extract from the data. We begin by stating a of bounding the error in entropy estimation. Lemma 1 . For a discrete random variable x ∈ X, with the plug-in estimateĤ(·) on its entropy, based on an i.i.d sample of size m, we have that Using the above bound we are able to bound the error in mutual information estimation, in a particular regime which is relevant for several applications of interest, such as object detection. Definition 1 (Distillation regime). In the distillation regime we assume that: i The samples x have very high entropy H(x). ii The entropy of y is small with respect to the entropy of x. iii The number of samples m of x that we have is small compared to H(x). Example 1. Typical object detection tasks are in the distillation regime. The entropy of images is high (property 1), while labels are compactly represented (property 2). Furthermore, the number of image samples is small compared to all possible samples (property 3). Lemma 2 (Mutual information bias). For discrete random variables x ∈ X, y ∈ Y, with the plug-in estimatesĤ(·) on their entropy, based on an i.i.d sample of size m, we bound the deviation of the estimated mutual information to the true mutual information in the distillation regime and conclude Proof. We start with a similar left-hand side as Ineq. 1 and expand the mutual information with entropy terms: Using the stated assumptions and applying Ineq. 1 we arrive at the following bound using the asymptotic equipartition property (AEP) |X | ≤ 2 H(x)+ (Cover&T homas, 2012): A similar relationship can be extracted from the information bottleneck bound . Theorem 3 (Theorem 4,). For any probability distribution p(x, y), with a probability of at least 1 − δ over the draw of the sample of size m from p(x, y), we have that for all bottleneck variables z, Again we consider the case where |Y| is small, |X | is large, and where the learned bottleneck variable z captures the output variable y exactly (hence |Z| is small). Then I(x; z) = H(x) −. For large terms H(x) the above bound becomes dominated byĤ(x) s.t. (for fixed |Y|, m, δ) When it comes to mutual information, we care about a high mutual information between input and output, but also about a reliable estimate of this mutual information. From the above calculations, we distill that the estimated entropyĤ(x) has a decisive effect on the uncertainty of the achievable mutual information between bottleneck variable z and output y. As a metaphor imagine the task of predicting the birthrate in Mongolia from the movements of the stock market for a given month. Most certainly one will be able to correlate the two. This is a common occurrence called spurious correlation making us fools of randomness . Hence we arrive at a score for mutual information that captures both the magnitude of the estimated mutual information and an estimate of the uncertainty of this estimate. Definition 2 (OMI). Given random variables x and y, empirical mutual informationÎ(x; y), and empirical entropyĤ(x), then the observed mutual information score (OMI) is defined as A similar thought process of introducing the same mutual information score has long been used for selecting features in decision trees . While we base our score on a whole dataset of (x, y)-pairs, the information gain ratio (IGR) for decision trees are feature-level criteria. Having defined our focus on mutual information and its associated OMI values, we turn our attention to the effect data representations may have on the learning process. We begin by defining what we mean when we talk about a data representation. Definition 3. A data representation r ∈ R is the output of a left-invertible mapping m(·): X → R applied to the "original" data x ∈ X. Therefore, all data representations are in a sense "equivalent". Since they are invertible they share the same information content I(x; y) = I(m(x); y)). Yet this is only half true. Clearly, how data is represented does make a difference. As a "worst case", an encrypted version of a dataset is unlikely to be useful for a learning task. In a perfect setting, a dataset could be represented in a way that directly maps to the desired output while keeping around additional "bits" needed to reconstruct the input. In such a case, learning would only require disregarding the extra dimensions. To understand what impact a data representation may have we will employ the idea of expected coding length E[l(x)] and focus on what happens when we choose the "wrong code". From Information Theory , we learn that the most efficient encoding we can possibly find is lower bounded by the entropy of the distribution we are trying to compress. In this case, we assume that we have a candidate distribution q(x) that we are trying to fit to the true distribution p(x). The expected coding length of our candidate distribution can then never be smaller than the entropy of the true distribution : In the following we will assume that any function family F has an associated candidate distribution q(·) through which it measures how uncertain a variable is, e.g. linear regression uses a normal distribution and assesses uncertainty via the determinant of the covariance matrix. The difficulty with assuming a candidate distribution is that it data may not follow the same distribution. Given such a mismatch, the model will overestimate the entropy of the distribution as shown in theorem 2. Theorem 5 (Representation-Model-Alignment). Assuming a candidate distribution q(·) and representations r 1, r 2 with D(p(r 1)||q(r 1)) > D(p(r 2)||q(r 2)) + 1 we have that Proof. From theorem 2 we know that A consequence of the above theorem is that the representation of the data distribution influences the observed entropy by changing the alignment of the true distribution to the assumed candidate distribution of the model. Critically for real-world situations, the wrong code theorem invalidates the assumption that the estimated mutual information does not change when an invertible transformation is applied. The true mutual information does indeed not change, yet this would be purely anecdotal if one were to encrypt the data and tried to learn from it. Changing data representations (invertible) can better align true task and assumed model distribution in the sense of minimizing relative entropy between candidate q(·) and true distribution p(·). This then has direct influence on the observed mutual information and the OMI score. The closer representation r aligns the true distribution to the model candidate distribution, the smaller the data entropiesĤ(r) andĤ(y|r) will be. In this sense there are thus better and worse data representations given a model and learning task. The theoretical findings we presented in the previous sections have to be verified in the real world. We therefore conducted a wide ranging empirical study on their relevance over a number of different datasets and network architectures. In total we evaluated over 8000 networks of varying sizes. We provide the full code and data required to replicate our experiments at the following URL: https://drive.google.com/open?id=1D8wICzJVPJRUWB9y5WgceslXZfurY34g To have a chance that our findings are applicable beyond the scope of this publication we chose a diverse set of four datasets that capture different vision tasks. We chose two datasets for classification (KDEF and Groceries) and two for regression (Lane Following and Drone Racing). Sample images for each of the datasets can be found in the appendix. Lane Following (LF): We generated this dataset using the Duckietown Gym , which is a simulation environment used for the AI Driving Olympics placed in Duck-ietown 2. It is the only simulated dataset we used in our experiments. Data is generated using a pure pursuit lane following algorithm, that returns the desired angular velocity. Longitudinal velocity is assumed to be constant. The learned model has to predict the angular velocity returned by the expert based on the image of the lane. Both train and test sets include domain randomization. To reduce the cost of training we downsampled each of the images to a third of their original size. KDEF: This dataset is based on an emotion recognition dataset by. Each of the images shows male and female actors expressing one of seven emotions. Images are captured from a number of different fixed viewpoints and centered on the face. To add more diversity to the data we added small color and brightness perturbations and a random crop. Moreover, since the dataset provides few samples, we downsampled each of the images to a sixth of their original size. Drone Racing: This dataset is based on the Drone Racing dataset by. We use the mDAVIS data from subsets 3, 5, 6, 9, and 10. While the original dataset provides the full pose (all six DOF), we train our feedforward networks to recover only the rotational DOF (roll, pitch, and yaw) from grayscale images. We matched the IMU data, which is sampled at 1000Hz to the timestamp for each grayscale image captured at 50Hz using linear interpolation. Since the images do not have multiple color channels we did not investigate a separate YCbCr or PREC representation for this dataset. We use the Freiburg Groceries Dataset and their original "test0/train0" split. Our only modifications are that we reserve a random subset of the test data for the evaluation of our hyperparameter optimization and that we reduce the size of the images from 256 × 256 to 120 × 120 pixels. Each of the images has to be classified into one of 25 categories. To estimate the OMI values, we calculated the individual entropiesĤ(x),Ĥ(y), andĤ(x, y). To computeĤ(x) andĤ(x, y), we assume a multivariate Gaussian distribution on the variables. The entropy is then computed asĤ(x) = 1 2 log(det(2πeΣ)), where Σ denotes the covariance matrix of x. To calculate this we apply an SVD decomposition and use the sum of log singular values to estimate the entropy. To avoid severe underestimation (entropy estimation is biased to underestimate ), entropy values are lower bounded by 2. Entropy values for y are calculated the same way for the regression task. For the classification task, we assume a multinoulli distribution and estimate the entropy of y accordingly. Manually tuning the hyperparameters for neural networks is both time consuming and may introduce unwanted bias into experiments. There is a wide range of automated methods available to mitigate these flaws . We utilize Bayesian optimization to find the set of optimal hyperparameters for each network. Since the initial weights of our networks are sampled from a uniform random distribution we can expect the performance to fluctuate between runs. Due to its probabilistic approach Bayesian optimization can account for this uncertainty . Moreover, it can optimize categorical and continuous dimensions concurrently. The dimensions and their constraints were chosen to be identical for each representation but were adapted to each dataset. For an in-depth introduction to Bayesian optimization we refer to. More information on our particular implementation of Bayesian optimization can be found in the appendix. We investigated three basic architectures. Our optimization was constrained to the same domain for all representations of a dataset for each of the network architectures. The full list of constraints for each network and dataset can be found in the code accompanying this paper. The initial learning rate was optimized for all architectures. We use a variable number of convolutional layers , with or without maxpooling layers between them, followed by a variable number of fully connected layers. Moreover, kernel sizes and number of filters are also parametrized. Dense Neural Networks: These networks consist of blocks of variably sized fully connected layers. We optimize the activation function after each layer as a categorical variable. ResNets: The Residual Neural Networks or ResNets are made up of a variable number of residual layers. Each of the layers contains a variable number of convolutions which themselves are fully parametrized. While one could select an arbitrary number of representations for images, we limit ourselves to five which have previously been used in image processing. Our focus is not on findings the best representations but to show how sensitive learning processes are to the representation of input data. Sample images for each of the representations can be found in the appendix. RGB: RGB is likely the representation we use most in our everyday life. Almost all modern displays and cameras capture or display data as an overlay of red, green and blue color channels. For simplicity we refer to the grayscale images of the Drone Racing dataset as being "RGB". The YCbCr representation is used in a number of different image storage formats such as JPEG and MPEG (David S.). It represents the image as a combination of a luminance and two chrominance channels. It is useful for compression because the human eye is much less sensitive to changes in chrominance than it is to changes in luminance. PREC: This representation partially decorrelates the color channels of the image based on previous work on preconditioning convolutional neural networks. For an image x ∈ R m×n×c with c channels we first calculate the expected value of the covariance between the channels for each image in the dataset: c×c, where x ij ∈ R c is the channel vector at pixel (i, j) of image x ∈ D. We then solve the eigenvalue problem for Σ obtaining real eigenvalues Λ and V containing the eigenvectors. A small is added for numerical stability. u is stored in memory and consecutively applied to each image in the dataset. We get U = diag(Λ + I) − 1 2 V which yields x prec = x rgb * U. The 2D type II discrete cosine transform (DCT) is a frequency-based representation. Low frequency coefficients are located in the top left corner of the representation and horizontal/vertical frequencies increase towards the right or down, respectively. This representation applies the DCT transform to each of the channels separately. DCT has been used extensively for face detection and all its coefficients bar one are invariant to uniform changes in brightness . Block DCT: Unlike for the DCT representation we apply the discrete cosine transform not to the whole image but to 8×8 non-overlapping patches of each of the channels. This exact type of DCT is widely used in JPEG compression by quantizing the coefficients of the DCT and applying Huffman encoding (David S.). 3.6 TRAINING Each network was trained using an Adam optimizer . Training was terminated when there were more than 7 previous epochs without a decrease in loss on the validation set or after 30 epochs were reached. After evaluating a total of 5753 networks, 3702 of which finished training, we have verified our intuition that representations are important. We see a fairly consistent pattern over all datasets of RGB and YCbCr being the best representations, followed by PREC and blockwise DCT, while DCT falls short (see Fig. 3). Moreover, we observe the great importance of hyperparameters in the spread of for each network architecture. Had we chosen to hand-tune our parameters and accidentally picked a very poor performing network for the RGB representation we could have easily come to the that DCT achieves better on all datasets. As we predicted, the performance of a representations also depends greatly on the network architecture. This is especially visible for the lane following dataset. We can see that, surprisingly, dense networks are among the best for both RGB and YCbCr, while they fall far behind on all other representations. The OMI scores we proposed show strong correlation with the we obtained from architecture search. They can even predict the comparatively small differences between the other representations with reasonable accuracy (see Fig. 2). It is unclear why the prediction fails for some of the datasets, especially the linear score on the KDEF dataset. Overall, we observe the significant correlated effect representation has on estimated entropy, OMI scores as well as performance. This work started by trying to pave a way to the exciting task of determining the difficulty of a learning task. To achieve this we introduced OMI, as a score that takes both estimated mutual information and possible variance of such an estimate into account. If the score proves itself also in future works, it may serve as a useful tool for automatic representation or architecture search. As outlined, this will depend to a great deal on how well we understand the candidate distributions of network architectures and representations that are currently in use. Similarly, it will be beneficial to study further techniques on removing the bias in the estimation of mutual information and entropy. We have shown that in many problems of interests the naive computation of mutual information is biased and representation dependent; our OMI score partially remove this bias. This score also now provides a criterion to evaluate different representations in a principled way. To narrate Tab. 2 we note that that estimating small entropy values is very error prone. When assuming a normal distribution, the entropy is calculated via the sum of the logarithm of eigenvalues of the covariance matrix of the data. The conditioning of the logarithm however gets worse, the closer its argument is to zero. Eigenvalues close enough to zero are thus likely to carry a significant error when used for entropy computation. This all is to say that while purely mathematically it is impossible to have negative mutual information values, numerically such things are bound to happen when dealing with small eigenvalues as is prominent with the DCT representation. While there have been some theoretical proposals that would allow Bayesian optimization to be run in parallel asynchronously , we restrict ourselves to a simple form of batch parallelization evaluating n = 6 points in parallel. We acquire the points by using the minimum constant liar strategy . The base estimator is first used after 10 points have been evaluated. Our acquisition function is chosen at each iteration from a portfolio of acquisition functions using a GP-Hedge strategy, as proposed in . We optimize the acquisition function by sampling it at n points for categorical dimensions and 20 iterations of L-BFGS for continuous dimensions. Since the optimizer has no information about the geometric properties of the network or if the network can fit in the systems memory, some of the generated networks cannot be trained. Two common modes of failure were too many pooling layers (ing in a layer size smaller than the kernel of subsequent layers) and running out of memory, which was especially prevalent for dense networks. In our experiments we observed that roughly 35% of all networks did not complete training. To stop the Bayesian optimizer from evaluating these points again we reported a large artificially generated loss to the optimizer at the point where the network crashed. The magnitude of this loss was chosen manually for each dataset to be roughly one order of magnitude larger than the expected loss. The influence of this practice will have to be investigated in future research.

We take a step towards measuring learning task difficulty and demonstrate that in practice performance strongly depends on the match of the representation of the information and the model interpreting it.

Represent the TLDR for finding the corresponding scientific article

We introduce the NLC, a metric that is cheap to compute in the networks randomly initialized state and is highly predictive of generalization, at least in fully-connected networks.


Question: represent the article for finding a single-sentence tldr

Question: Sepsis is a life-threatening complication from infection and a leading cause of mortality in hospitals. While early detection of sepsis improves patient outcomes, there is little consensus on exact treatment guidelines, and treating septic patients remains an open problem. In this work we present a new deep reinforcement learning method that we use to learn optimal personalized treatment policies for septic patients. We model patient continuous-valued physiological time series using multi-output Gaussian processes, a probabilistic model that easily handles missing values and irregularly spaced observation times while maintaining estimates of uncertainty. The Gaussian process is directly tied to a deep recurrent Q-network that learns clinically interpretable treatment policies, and both models are learned together end-to-end. We evaluate our approach on a heterogeneous dataset of septic spanning 15 months from our university health system, and find that our learned policy could reduce patient mortality by as much as 8.2\% from an overall baseline mortality rate of 13.3\%. Our algorithm could be used to make treatment recommendations to physicians as part of a decision support tool, and the framework readily applies to other reinforcement learning problems that rely on sparsely sampled and frequently missing multivariate time series data. Sepsis is a poorly understood complication arising from infection, and is both a leading cause in patient mortality BID8 ) and in associated healthcare costs BID37 ). Early detection is imperative, as earlier treatment is associated with better outcomes BID33, BID21 ). However, even among patients with recognized sepsis, there is no standard consensus on the best treatment. There is a pressing need for personalized treatment strategies tailored to the unique physiology of individual patients. Guidelines on sepsis treatment previously centered on early goal directed therapy (EGDT) and more recently have focused on sepsis care bundles, but none of these approaches are individualized. Before the landmark publication on the use of early goal directed therapy BID31 ), there was no standard management for severe sepsis and septic shock. EGDT consists of early identification of high-risk patients, appropriate cultures, infection source control, antibiotics administration, and hemodynamic optimization. The study compared a 6-hour protocol of EGDT promoting use of central venous catheterization to guide administration of fluids, vasopressors, inotropes, and packed red-blood cell transfusions, and was found to significantly lower mortality. Following the initial trial, EGDT became the cornerstone of the sepsis resuscitation bundle for the Surviving Sepsis Campaign (SCC) and the Centers for Medicare and Medicaid Services (CMS) BID6 ).Despite the promising of EGDT, concerns arose. External validity outside the single center study was unclear, it required significant resources for implementation, and the elements needed to achieve pre-specified hemodynamic targets held potential risks. Between 2014-2017, a trio of trials reported an all-time low sepsis mortality, and questioned the continued need for all elements of EGDT for patients with severe and septic shock BID28, , BID27 ). The trial authors concluded EGDT did not improve patient survival compared to usual care but was associated with increased ICU admissions BID0 ). As a , they did not recommend it be included in the updated SCC guidelines BID30 ).Although the SSC guidelines provide an overarching framework for sepsis treatment, there is renewed interest in targeting treatment and disassembling the bundle BID22 ). A recent metaanalysis evaluated 12 randomized trials and 31 observational studies and found that time to first an-tibiotics explained 96-99% of the survival benefit BID17 ). Likewise, a study of 50,000 patients across the state of New York found mortality benefit for early antibiotic administration, but not intravenous fluids BID33 ). Beyond narrowing the bundle, there is emerging evidence that a patient's baseline risk plays an important role in response to treatment, as survival benefit was significantly reduced for patients with more severe disease BID17 ).Taken together, the poor performance of EGDT compared to standard-of-care and improved understanding of individual treatment effects calls for re-envisioning sepsis treatment recommendations. Though general consensus in critical care is that the individual elements of the sepsis bundle are typically useful, it is unclear exactly when each element should be administered and in what quantity. In this paper, we aim to directly address this problem using deep reinforcement learning. We develop a novel framework for applying deep reinforcement learning to clinical data, and use it to learn optimal treatments for sepsis. With the widespread adoption of Electronic Health Records, hospitals are already automatically collecting the relevant data required to learn such models. However, real-world operational healthcare data present many unique challenges and motivate the need for methodologies designed with their structure in mind. In particular, clinical time series are typically irregularly sampled and exhibit large degrees of missing values that are often informatively missing, necessitating careful modeling. The high degree of heterogeneity presents an additional difficulty, as patients with similar symptoms may respond very differently to treatments due to unmeasured sources of variation. Alignment of patient time series can also be a potential issue, as patients admitted to the hospital may have very different unknown clinical states and can develop sepsis at any time throughout their stay (with many already septic upon admission).Part of the novelty in our approach hinges on the use of a Multi-output Gaussian process (MGP) as a preprocessing step that is jointly learned with the reinforcement learning model. We use an MGP to interpolate and to impute missing physiological time series values used by the downstream reinforcement learning algorithm, while importantly maintaining uncertainty about the clinical state. The MGP hyperparameters are learned end-to-end during training of the reinforcement learning model by optimizing an expectation of the standard Q-learning loss. Additionally, the MGP allows for estimation of uncertainty in the learned Q-values. For the model architecture we use a deep recurrent Q-network, in order to account for the potential for non-Markovian dynamics and allow the model to have memory of past states and actions. In our experiments utilizing EHR data from septic patients spanning 15 months from our university health system, we found that both the use of the MGP and the deep recurrent Q-network offered improved performance over simpler approaches. In this section we outline important that motivates our improvements on prior work. Reinforcement learning (RL) considers learning policies for agents interacting with unknown environments, and are typically formulated as a Markov decision process (MDP) BID36 ). At each time t, an agent observes the state of the environment, s t ∈ S, takes an action a t ∈ A, and receives a reward r t ∈ R, at which time the environment transitions to a new state s t+1. The state space S and action space A may be continuous or discrete. The goal of an RL agent is to select actions in order to maximize its return, or expected discounted future reward, defined as R t = T t =t γ t −t r t, where γ captures tradeoff between immediate and future rewards. Q-Learning BID40 ) is a model-free off-policy algorithm for estimating the expected return from executing an action in a given state. The optimal action value function is the maximum discounted expected reward obtained by executing action a in state s and acting optimally afterwards, defined as Q * (s, a) = max π E[R t |s t = s, a t = a, π], where π is a policy that maps states to actions. Given Q *, an optimal policy is to act by selecting argmax a Q * (s, a). In Q-learning, the Bellman equation is used to iteratively update the current estimate of the optimal action value function according to Q(s, a). = Q(s, a) + α(r + γmax a Q(s, a) − Q(s, a)), adjusting towards the observed reward plus the maximal Q-value at the next state s. In Deep Q-learning a deep neural network is used to approximate Q-values BID24 ), overcoming the issue that there may be infinitely many states if the state space is continuous. Denoting the parameters of the neural network by θ, Q-values Q(s, a|θ) are now estimated by performing a forward pass through the network. Updates to the parameters are obtained by minimizing a differentiable loss function, DISPLAYFORM0 2, and training is usually accomplished with stochastic gradient descent. A fundamental limiting assumption of Markov decision processes is the Markov property, which is rarely satisfied in real-world problems. In medical applications such as our problem of learning optimal sepsis treatments, it is unlikely that a patient's full clinical state will be measured. A Partially Observable Markov Decision Process (POMDP) better captures the dynamics of these types of realworld environments. An extension of an MDP, a POMDP assumes that an agent does not receive the true state of the system, instead receiving only observations o ∈ Ω generated from the underlying system state according to some unknown observation model o ∼ O(s). Deep Q-learning has no reliable way to learn the underlying state of the POMDP, as in general Q(o, a|θ) = Q(s, a|θ), and will only perform well if the observations well reflect the underlying state. Returning to our medical application, the system state might be the patient's unknown clinical status or disease severity, and our observations in the form of vitals or laboratory measurements offer some insight into the state. The Deep Recurrent Q-Network (DRQN) BID13 ) extends vanilla Deep Qnetworks (DQN) by using recurrent LSTM layers BID15 ), which are well known to capture long-term dependencies. LSTM recurrent neural network (RNN) models have frequently been used in past applications to medical time series, such as BID23. In our experiments we investigate the effect of replacing fully connected neural network layers with LSTM layers in our Q-network architecture in order to test how realistic the Markov assumption is in our application. Multi-output Gaussian processes (MGPs) are commonly used probabilistic models for irregularly sampled multivariate time series, as they seamlessly handle variable spacing, differing numbers of observations per series, and missing values. In addition, they maintain estimates of uncertainty about the state of the series. MGPs have been frequently applied to model patient physiological time series, e.g. BID12, BID7,.Given M time series (physiological labs/vitals), an MGP is specified by a mean function for each series {µ m (t)} M m=1, commonly assumed to be zero, and a covariance function or kernel K. Letting f m (t) denote the latent function for series m at time t, then K(t, t, m, m) = cov(f m (t), f m (t)). Typically the actual observations are centered on the latent functions according to some distribution, e.g. DISPLAYFORM0 noise parameters. We use the linear model of coregionalization covariance function with an Ornstein-Uhlenbeck base kernels k(t, t) = e −|t−t |/l to flexibly model temporal correlations in time as well as covariance structure between different physiological variables. For each patient, letting t denote the complete set of measurement times across all observations, the full joint kernel is K(t, t) = P p=1 B p ⊗ k p (t *, t *), where P denotes the number of mixture kernel. t * denotes the time vector for each physiological sign, assumed here to be the same for notational convenience, but in practice the full kernel need only be computed at the observed variables. Each B p ∈ R M ×M encodes the scale covariance between different time series. We found that P = 2 works well in practice and allows learning of correlations on both short and long time scales. Given the MGP kernel hyperparameters shared across all patients, collectively referred to as η, imputation and interpolation at arbitrary times can be computed either using the posterior mean or the full posterior distribution over unknown function values. Multi-output Gaussian processes and recurrent neural networks can be combined and trained endto-end (MGP-RNNs), in order to solve supervised learning problems for sequential data BID10, BID11 ). This methodology was shown to exhibit superior predictive performance at early detection of sepsis from clinical time series data, when compared with vanilla RNNs with last-one-carried-forward imputation. In fitting the two models end-to-end, the MGP hyperparameters are learned discriminatively, in essence learning an imputation and interpolation mechanism tuned for the supervised task at hand. Learning an MGP-RNN consists of minimizing an expectation of some loss function, with respect to the posterior distribution of the MGP. Letting z denote a set of latent time series values distributed according to an MGP posterior, and g(z) denote the prediction(s) made by an RNN from this time series, then the goal is to minimize E z∼MGP [l(o, g(z) )], where l is some loss function (e.g. crossentropy for a classification task) and o is the true label(s). We can express the MGP distributed latent variable z as z = µ z + R z ξ, where µ z is the posterior mean and R z R z = Σ z with Σ z the posterior covariance, and ξ ∼ (0, I). This allows us to apply the reparameterization trick BID19 ) and use Monte Carlo sampling to compute approximate gradients of this expectation with respect to both MGP hyperparameters η and RNN parameters θ, so that the loss can be minimized via stochastic gradient descent. The stochasticity in this learning procedure introduced from the Monte Carlo sampling additionally acts as a form of regularization, and helps prevent the RNN from overfitting. In Section 3 we show how this can be applied to a reinforcement learning task. There has been substantial recent interest in development of machine learning methodologies motivated by healthcare data. However, most prior work in clinical machine learning focuses on supervised tasks, such as diagnosis BID9 ) or risk stratification BID10 ). Many recent papers have developed models for early detection of sepsis, a related problem to our task of learning treatments for sepsis, e.g. BID35, BID14, BID11. However, as supervised problems rely on ground truth they cannot be applied to treatment recommendation, unless the assumption is made that past training examples of treatments represent optimal behavior. Instead, it is preferable to frame the problem using reinforcement learning in order to learn optimal treatment actions from data collected from potentially suboptimal actions. While deep reinforcement learning has seen huge success over the past few years, only very recently have reinforcement learning methods been designed with healthcare applications in mind. Applying reinforcement learning methods to healthcare data is difficult, as it requires careful consideration to set up the problem, especially the rewards. Furthermore, it is typically not possible to collect additional data and so evaluating learned policies on retrospective data presents a challenge. Most related to this paper are BID29 and BID20, who also look at the problem of learning optimal sepsis treatments. We build off of their work by using a more sophisticated network architecture that takes into account both memory through the use of DRQNs and uncertainty in time series imputation and interpolation using MGPs. Other relevant work includes BID26, who use a simpler learning algorithms to learn optimal strategies for ventilator weaning, and BID25, who also use a deep RL approach for modeling ICU heparin dosing as a POMDP with discriminative hidden Markov models and Q-networks. There also exists a rich set of work from the statistics and causal inference literature on learning dynamic treatment regimes, e.g. BID3, BID34, although the models are typically fairly simple for ease of interpretability. We now introduce Multi-Output Gaussian Process Deep Recurrent Q-Networks, or MGP-DRQNs, a novel reinforcement learning algorithm for learning optimal treatment policies from noisy, sparsely sampled, and frequently missing clinical time series data. We assume a discrete action space, a ∈ A = {1, . . ., A}. Let x denote T regularly spaced grid times at which we would like to learn optimal treatment decisions. Given a set of clinical physiological time series y that we assume to be distributed according to an MGP, we can compute a posterior distribution for z t |y, the latent unobserved time series values at each grid time. The loss function we optimize is similar to in normal deep Q-learning, with the addition of the expectation due to the MGP and the fact that we compute the loss over full patient trajectories. In particular, we learn optimal DRQN parameters θ * and MGP hyperparameters η * via: DISPLAYFORM0 where the t'th target value is Q (t) target = r t + γmax a Q([z t+1, s t+1], a ), the outer expectation is over training samples, and the inner one is with respect to the MGP posterior for one patient. We concatenate the two separate types of model inputs at time t, with z t denoting latent variables distributed according to an MGP posterior from other relevant inputs to the model denoted s t, such as static baseline covariates. In Section 4.1 we go into detail on the particular variables included in s t.We use a Dueling Double-Deep Q-network architecture, similar to BID29. The Dueling Q-network architecture BID39 has separate value and advantage streams to separate the effect of a patient being in a good underlying state from a good action being taken. The Double-Deep Q-network architecture BID38 ) helps correct overestimation of Q-values by using a second target network to compute the Q-values in the target Q target. Finally, we use Prioritized Experience Replay in order to speed learning, so that patient encounters with higher training error will be resampled more frequently. We use 2 LSTM layers with 64 hidden units each that feed to a final fully connected layer with 64 hidden units, before splitting into equally sized value and advantage streams that are finally then projected onto the action space to obtain Q-value estimates. We implemented our methods in Tensorflow using the Adam optimizers BID18 ) with minibatches of 50 encounters sampled at a time, a learning rate of 0.001, and L 2 regularization on weights. We use 25 Monte Carlo samples from the MGP for each sampled encounter in order to approximate the expected loss and compute approximate gradients, and these samples and other inputs are fed in a forward pass through the DRQN to get predictions Q(s, a). We will release source code via Github after the review period. In this section we first describe the details of our dataset of septic patients before highlighting how the experiments were set up and how the algorithms were evaluated. Our dataset consists of information collected during 9,255 patient encounters ing in sepsis at our university hospital, spanning a period of 15 months. We define sepsis to be the first time at which a patient simultaneously had persistently abnormal vitals (as measured by a 2+ SIRS score, BID2), a suspicion of infection (as measured by an order for a blood culture), and an abnormal laboratory value indicative of organ damage. This differs from the new Sepsis-3 definition BID32 ), which has since been largely criticized for its detection of sepsis late in the clinical course . We break the full dataset into 7867 training patient encounters and reserve the remaining 1388 for testing. We discretize the data to learn actions in 4 hour windows. We emphasize that the raw data itself is not down-sampled; rather, we use the MGP to learn a posterior for the time series values every 4 hours. Actions for the RL setup consist of 3 treatments commonly given to septic patients: antibiotics, vasopressors, and IV fluids. Antibiotics and vasopressors are broken down into 3 categories, based on whether 0, 1, or 2+ were administered in each 4 hour window. For IV Fluids, we consider 5 discrete categories: either 0, or one of 4 aggregate doses based on empirical quartiles of total fluid volumes. This yields a discrete action space with 3 × 3 × 5 = 45 distinct actions. Our data consists of 36 longitudinal physiological variables (e.g. blood pressure, pulse, white blood cell count), 2 longitudinal categorical variables, and 38 variables available at baseline (e.g. age, previous medical conditions). 8 medications tangential to sepsis treatment are included as inputs to MGP-DRQN, as well as an indicator for which of the 45 actions was administered at the last time. Additionally, 36 indicator variables for whether or not each lab/vital was recently sampled allows the model to learn from informative sampling due to non-random missingness. In total, there are 165 input observation variables to each of the Q-network models at each time. Our outcome of interest is mortality within 30 days of onset of sepsis. We use a sparse reward function in this initial work, so that the reward at every non-terminal time point is 0, with a reward of ±10 at the end of a trajectory based on patient survival/death. Although this presents a challenging credit assignment problem, this allows for data to inform what actions should be taken to reduce chance of death without being overly prescriptive. We use SARSA, an on-policy algorithm, to estimate state-action values for the physician policy. We compare a number of different architectures for learning optimal sepsis treatments. In addition to our proposed MGP-DRQN, we compare against MGP-mean-DRQN, a variant where we move the posterior expectation inside the DRQN loss function, meaning we use the posterior mean of the MGP rather than use Monte Carlo samples from the MGP. We also compare against a DRQN with identical architecture, but replace the MGP with last-one-carried-forward imputation to fill in any missing values, and use the mean if there are multiple measurements. We also compare against a vanilla DQN, a MGP-DQN, and a MGP-mean-DQN, with an equivalent number of layers and parameters, to test the effect of the recurrence in the DRQN models. We use Doubly Robust Off-policy Value Evaluation BID16 ) to compute unbiased estimates of each learned optimal policy using our observed off-policy data. For each patient trajectory in the test set we estimate its value using this method, and the average . In order to apply this method we train an MGP-RNN to estimate the action probabilities of the physician policy. In FIG0 we show the of using SARSA to estimate expected returns for the physician policy on the test data. The Q-values appear to be well calibrated with mortality, as patients who were estimated to have higher expected returns tended to have lower mortality. Due to small sample sizes for very low expected returns, the mortality rate does not always monotonically decrease. We can estimate the potential reduction in mortality a learned policy might have by computing an unbiased estimate of the policy value, as described in Section 4.3, and then use the in FIG0. Table 1 contains the policy value estimates for each algorithm considered, along with estimated mortality rates. The physician policy has an estimated value of 5.52 and corresponding mortality of 13.3%, matching the observed mortality in the test set of 13.3%. Overall the MGP-DRQN performs and might reduce mortality by as much as 8%. The DRQN architectures tended to yield higher expected returns, probably because they are able to retain some memory of past clinical states and actions taken. The MGP consistently improved as well, and the additional uncertainty information contained in the full MGP posterior appeared to do better than the policies that only used the posterior mean. Expected Return Estimated Mortality Physician 5.52 13.3 ± 0.7% MGP-DRQN 7.51 5.1 ± 0.5% MGP-mean-DRQN 6.97 6.6 ± 0.4% DRQN 6.63 8.4 ± 0.4% MGP-DQN 7.05 6.6 ± 0.4% MGP-mean-DQN 6.73 7.5 ± 0.4% DQN 6.09 10.6 ± 0.5% Table 1: Expected returns for the various policies considered. For the 6 reinforcement learning algorithms considered, we estimate their expected returns using an off-policy value evaluation algorithm. Using the from FIG0, we estimate the potential expected mortality reduction associated with each policy. We also qualitatively evaluate the of the policy from our best performing learning algorithm, the MGP-DRQN. In FIG2 we compare the number of times each type of action was actually taken by physicians, and how many times the learned policy selected that action. The MGP-DRQN policy tended to recommend more use of antibiotics and more vasopressors than were actually used by physician, while strangely recommending somewhat less use of IV fluids. In FIG3, we show how mortality rates differ on the test set as a function of how different the observed physician action was from what the MGP-DRQN would have recommended. For all 3 types of treatments, there appears to be a local minimum at 0 and we observe a V shape, indicating that empirically, mortality tended to be lowest when the clinicians took the same actions that the MGP-DRQN would have. Uncertainty tends to be higher due to smaller sample sizes for situations where there is larger disparity. After the patient is transferred to the Intensive Care Unit, their white blood cell count continues to rise (a sign of worsening infection) and their blood pressure continues to fall (a sign of worsening shock). By hour 14, the RL model starts and continues to recommend use of vasopressors to attempt to increase blood pressure, but they are not actually administered for about another 16 hours at hour 30. Ultimately, by hour 45 care was withdrawn and the patient passed away at hour 50. Cases such as this one illustrate the potential benefits of using our learned treatment policy in a decision support tool to recommend treatments to providers. If such a tool were used in this situation, it is possible that earlier treatments and more aggressive interventions might have ed in a different outcome. In this paper we presented a new framework combining multi-output Gaussian processes and deep reinforcement learning for clinical problems, and found that our approach performed well in estimating optimal treatment strategies for septic patients. The use of recurrent structure in the Q-network architecture yielded higher expected returns than a standard Q-network, accounting for the nonMarkovian nature of real-world medical data. The multi-output Gaussian process also improved performance by offering a more principled method for interpolation and imputation, and use of the full MGP posterior improved upon the from just using the posterior mean. In the future, we could include treatment recommendations from our learned policies into our dashboard application we have developed for early detection of sepsis. The treatment recommendations might help providers better care for septic patients after sepsis has been properly identified, and start treatments faster. There are many potential avenues for future work. One promising direction is to investigate the use of more complex reward functions, rather than the sparse rewards used in this work. More sophisticated rewards might take into account clinical targets for maintaining hemodynamic stability, and penalize an overzealous model that recommends too many unnecessary actions. Our modeling framework is fairly generalizable, and can easily be applied to other medical applications where there is a need for data-driven decision support tools. In future work we plan to use similar methods to learn optimal treatment strategies for treating patients with cardiogenic shock, and to learn effective insulin dosing regimes for patients on high-dose steroids.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We combine Multi-output Gaussian processes with deep recurrent Q-networks to learn optimal treatments for sepsis and show improved performance over standard deep reinforcement learning methods,

The applicability of inverse reinforcement learning is often hampered by the expense of collecting expert demonstrations; this paper seeks to broaden its applicability by incorporating prior task information through meta-learning.


Question: Represent the article for finding a single-sentence TLDR!

Question: Unsupervised and semi-supervised learning are important problems that are especially challenging with complex data like natural images. Progress on these problems would accelerate if we had access to appropriate generative models under which to pose the associated inference tasks. Inspired by the success of Convolutional Neural Networks (CNNs) for supervised prediction in images, we design the Neural Rendering Model (NRM), a new hierarchical probabilistic generative model whose inference calculations correspond to those in a CNN. The NRM introduces a small set of latent variables at each level of the model and enforces dependencies among all the latent variables via a conjugate prior distribution. The conjugate prior yields a new regularizer for learning based on the paths rendered in the generative model for training CNNs–the Rendering Path Normalization (RPN). We demonstrate that this regularizer improves generalization both in theory and in practice. Likelihood estimation in the NRM yields the new Max-Min cross entropy training loss, which suggests a new deep network architecture–the Max- Min network–which exceeds or matches the state-of-art for semi-supervised and supervised learning on SVHN, CIFAR10, and CIFAR100.

Represent the short summary for finding the corresponding scientific article (long)

We develop a new deep generative model for semi-supervised learning and propose a new Max-Min cross-entropy for training CNNs.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We propose Janossy pooling, a method for learning deep permutation invariant functions designed to exploit relationships within the input sequence and tractable inference strategies such as a stochastic optimization procedure we call piSGD


Question: Represent the article for finding a single-sentence TLDR

Question: Deep reinforcement learning (RL) agents often fail to generalize to unseen environments (yet semantically similar to trained agents), particularly when they are trained on high-dimensional state spaces, such as images. In this paper, we propose a simple technique to improve a generalization ability of deep RL agents by introducing a randomized (convolutional) neural network that randomly perturbs input observations. It enables trained agents to adapt to new domains by learning robust features invariant across varied and randomized environments. Furthermore, we consider an inference method based on the Monte Carlo approximation to reduce the variance induced by this randomization. We demonstrate the superiority of our method across 2D CoinRun, 3D DeepMind Lab exploration and 3D robotics control tasks: it significantly outperforms various regularization and data augmentation methods for the same purpose. Deep reinforcement learning (RL) has been applied to various applications, including board games (e.g., Go and Chess ), video games (e.g., Atari games and StarCraft ), and complex robotics control tasks . However, it has been evidenced in recent years that deep RL agents often struggle to generalize to new environments, even when semantically similar to trained agents (; b; ;). For example, RL agents that learned a near-optimal policy for training levels in a video game fail to perform accurately in unseen levels , while a human can seamlessly generalize across similar tasks. Namely, RL agents often overfit to training environments, thus the lack of generalization ability makes them unreliable in several applications, such as health care and finance . The generalization of RL agents can be characterized by visual changes , different dynamics , and various structures . In this paper, we focus on the generalization across tasks where the trained agents take various unseen visual patterns at the test time, e.g., different styles of s, floors, and other objects (see Figure 1). We also found that RL agents completely fail due to small visual changes 1 because it is challenging to learn generalizable representations from high-dimensional input observations, such as images. To improve generalization, several strategies, such as regularization (; b;) and data augmentation , have been proposed in the literature (see Section 2 for further details). In particular, showed that training RL agents in various environments generated by randomizing rendering in a simulator improves the generalization performance, leading to a better performance in real environments. This implies that RL agents can learn invariant and robust representations if diverse input observations are provided during training. However, their method is limited by requiring a physics simulator, which may not always be available. This motivates our approach of developing a simple and plausible method applicable to training deep RL agents. The main contribution of this paper is to develop a simple randomization technique for improving the generalization ability across tasks with various unseen visual patterns. Our main idea is to utilize random (convolutional) networks to generate randomized inputs (see Figure 1 (a)), and train RL agents (or their policy) by feeding them into the networks. Specifically, by re-initializing the parameters of random networks at every iteration, the agents are encouraged to be trained under a broad range of perturbed low-level features, e.g., various textures, colors, or shapes. We discover that the proposed idea guides RL agents to learn generalizable features that are more invariant in unseen environments (see Figure 3) than conventional regularization and data augmentation techniques. Here, we also provide an inference technique based on the Monte Carlo approximation, which stabilizes the performance by reducing the variance incurred from our randomization method at test time. We demonstrate the effectiveness of the proposed method on the 2D CoinRun game, the 3D DeepMind Lab exploration task , and the 3D robotics control task . For evaluation, the performance of the trained agents is measured in unseen environments with various visual and geometrical patterns (e.g., different styles of s, objects, and floors), guaranteeing that the trained agents encounter unseen inputs at test time. Note that learning invariant and robust representations against such changes is essential to generalize to unseen environments. In our experiments, the proposed method significantly reduces the generalization gap in unseen environments unlike conventional regularization and data augmentation techniques. For example, compared to the agents learned with the cutout data augmentation methods proposed by , our method improves the success rates from 39.8% to 58.7% under 2D CoinRun, the total score from 55.4 to 358.2 for 3D DeepMind Lab, and the total score from 31.3 to 356.8 for the Surreal robotics control task. Our can be influential to study other generalization domains, such as tasks with different dynamics , as well as solving real-world problems, such as sim-to-real transfer . Generalization in deep RL. Recently, the generalization performance of RL agents has been investigated by splitting training and test environments using random seeds (a) and distinct sets of levels in video games . Regularization is one of the major directions to improve the generalization ability of deep RL algorithms. and showed that regularization methods can improve the generalization performance of RL agents using various game modes of Atari and procedurally generated arcade environments called CoinRun, respectively. On the other hand, data augmentation techniques have also been shown to improve generalization. proposed a domain randomization method to generate simulated inputs by randomizing rendering in the simulator. Motivated by this, proposed a data augmentation method by modifying the cutout method . Our method can be combined with the prior methods to further improve the generalization performance. Random networks for deep RL. Random networks have been utilized in several approaches for different purposes in deep RL. utilized a randomly initialized neural network to define an intrinsic reward for visiting unexplored states in challenging exploration problems. By learning to predict the reward from the random network, the agent can recognize unexplored states. studied a method to improve ensemble-based approaches by adding a randomized network to each ensemble member to improve the uncertainty estimation and efficient exploration in deep RL. Our method is different because we introduce a random network to improve the generalization ability of RL agents. Transfer learning. Generalization is also closely related to transfer learning (; a; b), which is used to improve the performance on a target task by transferring the knowledge from a source task. However, unlike supervised learning, it has been observed that finetuning a model pre-trained on the source task for adapting to the target task is not beneficial in deep RL. proposed a domain transfer method using generative adversarial networks and utilized regularization techniques to improve the performance of fine-tuning methods. proposed a multi-stage RL, which learns to extract disentangled representations from the input observation and then trains the agents on the representations. Alternatively, we focus on the zero-shot performance of each agent at test time without further fine-tuning of the agent's parameters. We consider a standard reinforcement learning (RL) framework where an agent interacts with an environment in discrete time. Formally, at each timestep t, the agent receives a state s t from the environment 2 and chooses an action a t based on its policy π. The environment returns a reward r t and the agent transitions to the next state s t+1. The return R t = ∞ k=0 γ k r t+k is the total accumulated rewards from timestep t with a discount factor γ ∈. RL then maximizes the expected return from each state s t. We introduce a random network f with its parameters φ initialized with a prior distribution, e.g., Xavier normal distribution . Instead of the original input s, we train an agent using a randomized input s = f (s; φ). For example, in the case of policy-based methods, 3 the parameters θ of the policy network π are optimized by minimizing the following policy gradient objective function: where D = {(s t, a t, R t)} is a set of past transitions with cumulative rewards. By re-initializing the parameters φ of the random network per iteration, the agents are trained using varied and randomized input observations (see Figure 1(a) ). Namely, environments are generated with various visual patterns, but with the same semantics by randomizing the networks. Our agents are expected to adapt to new environments by learning invariant representation (see Figure 3 for supporting experiments). To learn more invariant features, the following feature matching (FM) loss between hidden features from clean and randomized observations is also considered: where h(·) denotes the output of the penultimate layer of policy π. The hidden features from clean and randomized inputs are combined to learn more invariant features against the changes in the input observations. 4 Namely, the total loss is: where β > 0 is a hyper-parameter. The full procedure is summarized in Algorithm 1 in Appendix B. ResNet-18 + ours 95.9 ± 1.6 84.4 ± 4.5 Table 1: The classification accuracy (%) on dogs vs. cats dataset. The show the mean and standard deviation averaged over three runs and the best is indicated in bold. Figure 2: Samples of dogs vs. cats dataset. The training set consists of bright dogs and dark cats, whereas the test set consists of dark dogs and bright cats. Details of the random networks. We propose to utilize a single-layer convolutional neural network (CNN) as a random network, where its output has the same dimension with the input (see Appendix F for additional experimental on the various types of random networks). To reinitialize the parameters of the random network, we utilize the following mixture of distributions: 2 nin+nout, where I is an identity kernel, α ∈ is a positive constant, N denotes the normal distribution, and n in, n out are the number of input and output channels, respectively. Here, clean inputs are used with the probability α because training only randomized inputs can complicate training. The Xavier normal distribution is used for randomization because it maintains the variance of the input s and the randomized input s. We empirically observe that this distribution stabilizes training. Removing visual bias. To confirm the desired effects of our method, we conduct an image classification experiment on the dogs and cats database from Kaggle. 5 Following the same setup as , we construct datasets with an undesirable bias as follows: the training set consists of bright dogs and dark cats while the test set consists of dark dogs and bright cats (see Appendix J for further details). A classifier is expected to make a decision based on the undesirable bias, (e.g., brightness and color) since CNNs are biased towards texture or color, rather than shape . Table 1 shows that ResNet-18 does not generalize effectively due to overfitting to an undesirable bias in the training data. To address this issue, several image processing methods , such as grayout (GR), cutout (CO; DeVries & Taylor 2017), inversion (IV), and color jitter (CJ), can be applied (see Appendix E for further details). However, they are not effective in improving the generalization ability, compared to our method. This confirms that our approach makes DNNs capture more desired and meaningful information such as the shape by changing the visual appearance of attributes and entities in images while effectively keeping the semantic information. Prior sophisticated methods require additional information to eliminate such an undesired bias, while our method does not. 6 Although we mainly focus on RL applications, our idea can also be explorable in this direction. Since the parameter of random networks is drawn from a prior distribution P (φ), our policy is modeled by a stochastic neural network: π(a|s; θ) = E φ π (a|f (s; φ); θ). Based on this interpretation, our training procedure (i.e., randomizing the parameters) consists of training stochastic models using the Monte Carlo (MC) approximation (with one sample per iteration). Therefore, at the inference or test time, an action a is taken by approximating the expectations as follows: and M is the number of MC samples. In other words, we generate M random inputs for each observation and then aggregate their decisions. The show that this estimator improves the performance of the trained agents by approximating the posterior distribution more accurately (see Figure 3(d) ). 5 https://www.kaggle.com/c/dogs-vs-cats 6 Using the known bias information (i.e., {dark, bright}) and ImageNet pre-trained model, achieve 90.3%, while our method achieves 84.4% without using both inputs. In this section, we demonstrate the effectiveness of the proposed method on 2D CoinRun , 3D DeepMind Lab exploration , and 3D robotics control task . To evaluate the generalization ability, we measure the performance of trained agents in unseen environments which consist of different styles of s, objects, and floors. Due to the space limitation, we provide more detailed experimental setups and in the Appendix. For CoinRun and DeepMind Lab experiments, similar to , we take the CNN architecture used in IMPALA as the policy network, and the Proximal Policy Optimization (PPO) method to train the agents. 7 At each timestep, agents are given an observation frame of size 64 × 64 as input (resized from the raw observation of size 320 × 240 as in the DeepMind Lab), and the trajectories are collected with the 256-step rollout for training. For Surreal robotics experiments, similar to , the hybrid of CNN and long short-term memory (LSTM) architecture is taken as the policy network, and a distributed version of PPO (i.e., actors collect a massive amount of trajectories, and the centralized learner updates the model parameters using PPO) is used to train the agents. 8 We measure the performance in the unseen environment for every 10M timesteps and report the mean and standard deviation across three runs. Our proposed method, which augments PPO with random networks and feature matching (FM) loss (denoted PPO + ours), is compared with several regularization and data augmentation methods. As regularization methods, we compare dropout (DO; Srivastava et al. 2014), L2 regularization (L2), and batch normalization (BN; Ioffe & Szegedy 2015). For those methods, we use the hyperparameters suggested in , which are empirically shown to be effective: a dropout probability of 0.1 and a coefficient of 10 −4 for L2 regularization. We also consider various data augmentation methods: a variant of cutout (CO; DeVries & Taylor 2017) proposed in , grayout (GR), inversion (IV), and color jitter (CJ) by adjusting brightness, contrast, and saturation (see Appendix E for more details). As an upper bound, we report the performance of agents trained directly on unseen environments, dented PPO (oracle). For our method, we use β = 0.002 for the weight of the FM loss, α = 0.1 for the probability of skipping the random network, M = 10 for MC approximation, and a single-layer CNN with the kernel size of 3 as a random network. Task description. In this task, an agent is located at the leftmost side of the map and the goal is to collect the coin located at the rightmost side of the map within 1,000 timesteps. The agent observes its surrounding environment in the third-person point of view, where the agent is always located at the center of the observation. CoinRun contains an arbitrarily large number of levels which are generated deterministically from a given seed. In each level, the style of , floor, and obstacles is randomly selected from the available themes (34 s, 6 grounds, 5 agents, and 9 moving obstacles). Some obstacles and pitfalls are distributed between the agent and the coin, where a collision with them in the agent's immediate death. We measure the success rates, which correspond to the number of collected coins divided by the number of played levels. Ablation study on small-scale environments. First, we train agents on one level for 100M timesteps and measure the performance in unseen environments by only changing the style of the , as shown in Figure 3 (a). Note that these visual changes are not significant to the game's dynamics, but the agent should achieve a high success rate if it can generalize accurately. However, Table 2 shows that all baseline agents fail to generalize to unseen environments, while they achieve a near-optimal performance in the seen environment. This shows that regularization techniques have no significant impact on improving the generalization ability. Even though data augmentation techniques, such as cutout (CO) and color jitter (CJ), slightly improve the performance, our proposed method is most effective because it can produce a diverse novelty in attributes and entities. Training with randomized inputs can degrade the training performance, but the high expressive power Embedding analysis. We analyze whether the hidden representation of trained RL agents exhibits meaningful abstraction in the unseen environments. The features on the penultimate layer of trained agents are visualized and reduced to two dimensions using t-SNE . Figure 3 shows the projection of trajectories taken by human demonstrators in seen and unseen environments (see Figure 8 in Appendix C for further ). Here, trajectories from both seen and unseen environments are aligned on the hidden space of our agents, while the baselines yield scattered and disjointed trajectories. This implies that our method makes RL agents capable of learning the invariant and robust representation. To evaluate the quality of hidden representation quantitatively, the cycle-consistency proposed in is also measured. Given two trajectories V and U, v i ∈ V first locates its nearest neighbor in the other trajectory u j = arg min u∈U h(v i) − h(u) 2, where h(·) denotes the output of the penultimate layer of trained agents. Then, the nearest neighbor of u j in V is located, i.e., v k = arg min v∈V h(v) − h(u j) 2, and v i is defined as cycle-consistent if |i − k| ≤ 1, i.e., it can return to the original point. Note that this cycle-consistency implies that two trajectories are accurately aligned in the hidden space. Similar to , we also evaluate the three-way cycle-consistency by measuring whether v i remains cycle-consistent along both paths, V → U → J → V and V → J → U → V, where J is the third trajectory. Using the trajectories shown in Figure 3 (a), Table 2 reports the percentage of input observations in the seen environment (blue curve) that are cycle-consistent with unseen trajectories (red and green curves). Similar to the shown in Figure 3 Visual interpretation. To verify whether the trained agents can focus on meaningful and highlevel information, the activation maps are visualized using Grad-CAM by averaging activations channel-wise in the last convolutional layer, weighted by their gradients. As shown in Figure 4, both vanilla PPO and our agents make a decision by focusing on essential objects, such as obstacles and coins in the seen environment. However, in the unseen environment, the vanilla PPO agent displays a widely distributed activation map in some cases, while our agent does not. As a quantitative metric, we measure the entropy of normalized activation maps. Specifically, we first normalize activations σ t,h,w ∈, such that it represents a 2D discrete probability distribution at timestep t, i.e., H h=1 W w=1 σ t,h,w = 1. Then, we measure the entropy averaged over the timesteps as follows: H h=1 W w=1 σ t,h,w log σ t,h,w. Note that the entropy of the activation map quantitatively measures the frequency an agent focuses on salient components in its observation. Results show that our agent produces a low entropy on both seen and unseen environments (i.e., 2.28 and 2.44 for seen and unseen, respectively), whereas the vanilla PPO agent produces a low entropy only in the seen environment (2.77 and 3.54 for seen and unseen, respectively). Results on large-scale experiments. Similar to , the generalization ability by training agents is evaluated on a fixed set of 500 levels of CoinRun. To explicitly separate seen and unseen environments, half of the available themes are utilized (i.e., style of s, floors, agents, and moving obstacles) for training, and the performances on 1,000 different levels consisting of unseen themes are measured. 9 As shown in Figure 5 (a), our method outperforms all baseline methods by a large margin. In particular, the success rates are improved from 39.8% to 58.7% compared to the PPO with cutout (CO) augmentation proposed in , showing that our agent learns generalizable representations given a limited number of seen environments. Results on DeepMind Lab. We also demonstrate the effectiveness of our proposed method on DeepMind Lab , which is a 3D game environment in the first-person point of view with rich visual inputs. The task is designed based on the standard exploration task, where a goal object is placed in one of the rooms in a 3D maze. In this task, agents aim to collect as many goal objects as possible within 90 seconds to maximize their rewards. Once the agent collects the goal object, it receives ten points and is relocated to a random place. Similar to the small-scale CoinRun experiment, agents are trained to collect the goal object in a fixed map layout and tested in unseen environments with only changing the style of the walls and floors. We report the mean and standard deviation of the average scores across ten different map layouts, which are randomly selected. Additional details are provided in Appendix I. Note that a simple strategy of exploring the map actively and recognizing the goal object achieves high scores because the maze size is small in this experiment. Even though the baseline agents achieve high scores by learning this simple strategy in the seen environment (see Figure 6 (c) in Appendix A for learning curves), Figure 5 (b) shows that they fail to adapt to the unseen environments. However, the agent trained by our proposed method achieves high scores in both seen and unseen environments. These show that our method can learn generalizable representations from high-dimensional and complex input observations (i.e., 3D environment). Results on Surreal robotics control. We evaluate our method in the Block Lifting task using the Surreal distributed RL framework : the Sawyer robot receives a reward if it succeeds to lift a block randomly placed on a table. We train agents on a single environment and test on five unseen environments with various styles of tables and blocks (see Appendix K for further details). Figure 5 (c) shows that our method achieves a significant performance gain compared to all baselines in unseen environments while maintaining its performance in the seen environment (see Figure 14 in Appendix K), implying that our method can maintain essential properties, such as structural spatial features of the input observation. Comparison with domain randomization. To further verify the effectiveness of our method, the vanilla PPO agents are trained by increasing the number of seen environments generated by randomizing rendering in a simulator, while our agent is still trained in a single environment (see Appendices I and K for further details). Table 3 shows that the performance of baseline agents can be improved with domain randomization . However, our method still outperforms the baseline methods trained with more diverse environments than ours, implying that our method is more effective in learning generalizable representations than simply increasing the (finite) number of seen environments. In this paper, we explore generalization in RL where the agent is required to generalize to new environments in unseen visual patterns, but semantically similar. To improve the generalization ability, we propose to randomize the first layer of CNN to perturb low-level features, e.g., various textures, colors, or shapes. Our method encourages agents to learn invariant and robust representations by producing diverse visual input observations. Such invariant features could be useful for several other related topics, like an adversarial defense in RL (see Appendix D for further discussions), sim-toreal transfer , transfer learning (; a; b), and online adaptation . We provide the more detailed discussions on an extension to the dynamics generalization and failure cases of our method in Appendix L and M, respectively. The adversarial (visually imperceptible) perturbation to clean input observations can induce the DNN-based policies to generate an incorrect decision at test time . This undesirable property of DNNs has raised major security concerns. In this section, we evaluate if the proposed method can improve the robustness on adversarial Algorithm 1 PPO + random networks, Actor-Critic Style for iteration= 1, 2, · · · do Sample the parameter φ of random networks from prior distribution P (φ) for actor= 1, 2, · · ·, N do Run policy π (a|f (s; φ); θ) in the given environment for T timesteps Compute advantage estimates end for Optimize L random in equation with respect to θ end for attacks. Our method is expected to improve the robustness against such adversarial attacks because the agents are trained with randomly perturbed inputs. To verify that the proposed method can improve the robustness to adversarial attacks, the adversarial samples are generated using FGSM by perturbing inputs to the opposite direction to the most probable action initially predicted by the policy: where ε is the magnitude of noise and a * = arg max a π(a|s; θ) is the action from the policy. Table 4 shows that our proposed method can improve the robustness against FGSM attacks with ε = 0.01, which implies that hidden representations of trained agents are more robust., two boxes are painted in the upper-left corner, where their color represents the x-and y-axis velocity to help the agents quickly learn to act optimally. In this way, the agent does not need to memorize previous states, so a simple CNN-based policy without LSTM can effectively perform in our experimental settings. Data augmentation methods. In this paper, we compare a variant of cutout proposed in , grayout, inversion, and color jitter . Specifically, the cutout augmentation applies a random number of boxes in random size and color to the input, the grayout method averages all three channels of the input, the inversion method inverts pixel values by a 50% chance, and the color jitter changes the characteristics of images commonly used for data augmentation in computer vision tasks: brightness, contrast, and saturation. For every timestep in the cutout augmentation, we first randomly choose the number of boxes from zero to five, assign them a random color and size, and place them in the observation. For the color jitter, the parameters for brightness, contrast, and saturation are randomly chosen in [0.5,1.5]. 10 For each episode, the parameters of these methods are randomized and fixed such that the same image preprocessing is applied within an episode. In this section, we apply random networks to various locations in the network architecture (see Figure 10) and measure the performance in large-scale CoinRun without the feature matching loss. For all methods, a single-layer CNN is used with a kernel size of 3, and its output tensor is padded in order to be in the same dimension as the input tensor. As shown in Figure 9, the performance of unseen environments decreases as the random network is placed in higher layers. On the other hand, the random network in residual connections improves the generalization performance, but it does not outperform the case when a random network is placed at the beginning of the network, meaning that randomizing only the local features of inputs can be effective for a better generalization performance. For small-scale CoinRun environments, we consider a fixed map layout with two moving obstacles and measure the performance of the trained agents by changing the style of the s (see Figure 11). Below is the list of seen and unseen s in this experiment: In CoinRun, there are 34 themes for s, 6 for grounds, 5 for agents, and 9 for obstacles. For the large-scale CoinRun experiment, we train agents on a fixed set of 500 levels of CoinRun using half of the available themes and measure the performances on 1,000 different levels consisting of unseen themes. Specifically, the following is a list of seen and unseen themes used in this experiment: • Seen s: • Unseen s: Dataset. The original database is a set of 25,000 images of dogs and cats for training and 12,500 images for testing. Similar to , the data is manually categorized according to the color of the animal: bright or dark. Biased datasets are constructed such that the training set consists of bright dogs and dark cats, while the test and validation sets contain dark dogs and bright cats. Specifically, training, validation, and test sets consist of 10,047, 1,000, and 5,738 images, respectively. 11 ResNet-18 is trained with an initial learning rate chosen from {0.05, 0.1} and then dropped by 0.1 at 50 epochs with a total of 100 epochs. We use the Nesterov momentum of 0.9 for SGD, a mini-batch size chosen from {32, 64}, and the weight decay set to 0.0001. We report the training and test set accuracies with the hyperparameters chosen by validation. , we do not use ResNet-18 pre-trained with ImageNet in order to avoid inductive bias from the pre-trained dataset. Our method is evaluated in the Block Lifting task using the Surreal distributed RL framework . In this task, the Sawyer robot receives a reward if it successfully lifts a block randomly placed on a table. Following the experimental setups in , the hybrid CNN-LSTM architecture (see Figure 14 (a)) is chosen as the policy network and a distributed version of PPO (i.e., actors collect massive amount of trajectories and the centralized learner updates the model parameters using PPO) is used to train the agents. 12 Agents take 84 × 84 observation frames with proprioceptive features (e.g., robot joint positions and velocities) and output the mean and log of the standard deviation for each action dimension. The actions are then sampled from the Gaussian distribution parameterized by the output. Agents are trained on a single environment and tested on five unseen environments with various styles of table, floor, and block, as shown in Figure 15. For the Surreal robot manipulation experiment, the vanilla PPO agent is trained on 25 environments generated by changing the styles of tables and boxes. Specifically, we use {blue, gray, orange, white, purple} and {red, blue, green, yellow, cyan} for table and box, respectively. In this section, we consider an extension to the generalization on domains with different dynamics. Similar to dynamics randomization , one can expect that our idea can be useful for improving the dynamics generalization. To verify this, we conduct an experiment on CartPole and Hopper environments where an agent takes proprioceptive features (e.g., positions and velocities). The goal of CartPole is to prevent the pole from falling over, while that of Hopper is to make an onelegged robot hop forward as fast as possible, respectively. Similar to the randomization method we applied to visual inputs, we introduce a random layer between the input and the model. As a natural extension of the proposed method, we consider performing the convolution operation by multiplying a d × d diagonal matrix to d-dimensional input states. For every training iteration, the elements of the matrix are sampled from the standard uniform distribution U (0.8, 1.2). One can note that this method can randomize the amplitude of input states while maintaining the intrinsic information (e.g., sign of inputs). method is used to train the agents. The mass of the training environment is sampled from {1.0, 2.0, 3.0, 4.0, 5.0}, while it is sampled from {6.0, 7.0, 8.0} during testing. 14 Figure 16 reports the mean and standard deviation across 3 runs. Our simple randomization improves the performance of the agents in unseen environments, while achieving performance comparable to seen environments. We believe that this evidences a wide applicability of our idea beyond visual changes. In this section, we verify whether the proposed method can handle color (or texture)-conditioned RL tasks. One might expect that such RL tasks can be difficult for our methods to work because of the randomization. For example, our methods would fail if we consider an extreme seek-avoid object gathering setup, where the agent must learn to collect good objects and avoid bad objects which have the same shape but different color. However, we remark that our method would not always fail for such tasks if other environmental factors (e.g., the shape of objects in Collect Good Objects in DeepMind Lab ) are available to distinguish them. To verify this, we consider a modified CoinRun environment where the agent must learn to collect good objects (e.g., gold coin) and avoid bad objects (e.g., silver coin). Similar to the small-scale CoinRun experiment, agents are trained to collect the goal object in a fixed map layout (see Figure 17 (a)) and tested in unseen environments with only changing the style of the . Figure 17 (b) shows that our method can work well for such color-conditioned RL tasks because a trained agent can capture the other factors such as a location to perform this task. Besides, our method achieves a significant performance gain compared to vanilla PPO agent in unseen environments as shown in Figure 17 (c). As another example, in color-matching tasks such as the keys doors puzzle in DeepMind Lab , the agent must collect colored keys to open matching doors. Even though this task is color-conditioned, a policy trained with our method can perform well because the same colored objects will have the same color value even after randomization, i.e., our randomization method still maintains the structure of input observation. This evidences the wide applicability of our idea. We also remark that our method can handle more extreme corner cases by adjusting the fraction of clean samples during training. In summary, we believe that the proposed method covers a broad scope of generalization across low-level transformations in the observation space features. We investigate the effect of the fraction of clean samples during training. As shown in Figure 17 (d), the best unseen performance is achieved when the fraction of clean samples is 0.1 on large-scale CoinRun.

We propose a simple randomization technique for improving generalization in deep reinforcement learning across tasks with various unseen visual patterns.

We propose HiPPO, a stable Hierarchical Reinforcement Learning algorithm that can train several levels of the hierarchy simultaneously, giving good performance both in skill discovery and adaptation.