Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Parameters are one of the most critical components of machine learning models. As datasets and learning domains change, it is often necessary and time-consuming to re-learn entire models. Rather than re-learning the parameters from scratch, replacing learning with optimization, we propose a framework building upon the theory of \emph{optimal transport} to adapt model parameters by discovering correspondences between models and data, significantly amortizing the training cost. We demonstrate our idea on the challenging problem of creating probabilistic spatial representations for autonomous robots. Although recent mapping techniques have facilitated robust occupancy mapping, learning all spatially-diverse parameters in such approximate Bayesian models demand considerable computational time, discouraging them to be used in real-world robotic mapping. Considering the fact that the geometric features a robot would observe with its sensors are similar across various environments, in this paper, we demonstrate how to re-use parameters and hyperparameters learned in different domains. This adaptation is computationally more efficient than variational inference and Monte Carlo techniques. A series of experiments conducted on realistic settings verified the possibility of transferring thousands of such parameters with a negligible time and memory cost, enabling large-scale mapping in urban environments. The quintessential paradigm in the machine learning pipeline consists of the stages of data acquisition and inference of the given data. As data become plentiful, or as ones problem set become more diverse over time, it is common to learn new models tailored to the new data or problem. Contrasting this conventional modeling archetype, we argue that it is often redundant to perform inference and re-learn parameters from scratch. Such model adaptation procedures are indispensable in application domains such as robotics in which the operating environments change continuously. For instance, if the model is represented as a Bayesian model, its distribution should be redetermined regularly to adjust for changes in new data. In this paper, we focus on significantly improving the training time of building Bayesian occupancy maps such as automorphing Bayesian Hilbert maps (ABHMs) by transferring model parameters associated with a set of source datasets to a target dataset in a zero-shot fashion. Despite having attractive theoretical properties and being robust, the main reason that hinders models such as ABHM being used in real-world settings is the run-time cost of learning thousands of parameters (main parameters and hyperparameters). Moreover, these parameters not only vary across different places in the same environment, but also change over time. We demonstrate domain adaptation of "geometry-dependent spatial features" of the ABHM model from a pool of source domains to the current target domain. This is efficiently done using the theory of. Since the proposed approach completely bypasses explicitly learning parameters of the Bayesian model using domain adaptation, this process can be thought of as "replacing parameter learning with domain adapatation." The notation given in Table 1 will be used throughout the rest of the paper. An occupancy model is typically a parameterized function which gives the probability of a given point in the environment being occupied. For instance, having learned a function with parameters θ, it is possible to query y * = p(occupied|x *, θ) ∈ for anywhere in the space x * = (longitude, latitude) ∈ R 2. The parameters θ must be estimated from data gathered using a LIDAR sensor with labels y = {0, 1} = {free, hit}. The high level idea of ABHM is projecting LIDAR data into the reproducing kernel Hilbert space (RKHS)-a rich high dimensional feature space-and performing Bayesian logistic regression. The occupancy probability of a point x is given by p(y|x) = sigmoid) with weights w ∈ R, kernel hinged at spatial locations h ∈ R 2, and width of the squaredexponential (SE) kernel γ ∈ R +. As shown in Figure 1, here, M SE kernels positioned at M sparial locations {h m} M m=1 are used to project 2D data into a M dimensional vector such that each kernel has more effect from data in its locality. must be learned from LIDAR data. Slightly abusing standard notations, in this paper,¯ands ymbols are used to represent the mean and dispersion parameters, respectively. One of the most important parameters for later discussions is the location parameterh m ∈ R 2. Because of the intractable posterior, the parameters of the model are learned using variational inference through probabilistic programming. In this section, we propose a framework for swiftly adapting thousands of parameter and hyperparameters of the Bayesian mapping model. To adapt to domains, we require accurately pre-trained maps from which we can extract spatially relevant features. In the context of our problem we must extract LIDAR scans (hits and free) with their corresponding model parameters {(h, θ)}. To simplify further discussions, as in Figure 1, θ is defined as all parameters except the mean location parameterh. We define source LIDAR data with corresponding parameters learned from ABHM {θ as the source atom. The source is an environment small enough to be trainable with ABHM. Having determined the source atom, our objective is to determine the new set of parameters . As illustrated in Figure 7, we are looking for a nonlinear mapping technique to convert a source (S) to a target (T). We recognize this as an optimal transport (OT) problem. In occupancy mapping, the probability measures are from LIDAR data. For a new target dataset, we attempt to obtain the optimal coupling, for a given D ∈ R N (S) ×N (T) distance matrix (e.g. Euclidean distance between sourcetarget pairs) with the information entropy of P, r(P) = − ij P ij log P ij. This entropic regularization, commonly known as the Sinkhorn distance; , enables solving the otherwise hard integer programming problem using an efficient iterative algorithm. Here, λ controls the amount of regularization. Having obtained the optimal coupling between source and target LIDAR, as illustrated in Figures 7 (b) -(c), now it is possible to transport corresponding source parameters θ (S) to the target domain. This is done by associating the parameter positions with source samplesh (S) as a linear map x (S). Note that all other θ (S) parameters associated with h (S) will also be transported. This implicit transfer process is depicted in Figure 5. Since ABHM can only be executed in small areas due to the high computational cost, we learn individual ABHM maps for different areas and construct a dictionary of source atoms which we call a dictionary of atoms X (S). As a , as depicted in Figure 2, atoms from various domains will be transferred to the target. The entire algorithm is given in Algorithm 1. We used the Carla simulator and KITTI benchmark dataset for experiments. A summary of datasets is listed in Table 5. We compared against vanilla variational inference; and variational inference with reparameterization trick;. Intra-domain and inter-domain adaptation Here we consider two paradigms: intradomain and inter-domain transfer. In intra-domain transfer, the source atoms are generated from the first 10 frames of a particular dataset and parameters are transferred within the Figure 2: A high-level overview of our method: Parameter Optimal Transport. Training domains correspond to potentially independent, data-intensive, expensive, yet small-scale prelearned models. After storing in a dictionary of atoms, representative data-space and modelparameter tuples from the pre-learned set of models, we find data-space correspondences using optimal transport maps via the ranking procedure. These maps are then used to transport pre-learned parameters to out-of-sample test domains. Our method is largely insensitive to data-space invariances between source training domains and test domains reducing knowledge loss during the transfer process. same dataset. In inter-domain transfer they are transferred to a completely new town. Results are in Table 4 with 20% randomly sampled test LIDAR beams. We consider two paradigms: intra-domain and inter-domain transfer. In intra-domain transfer, the source atoms are generated from the first 10 frames of a particular dataset and parameters are transferred within the same dataset. In inter-domain transfer they are transferred to a new town. Results are in Table 4 Building instantaneous maps This experiment demonstrates performance of building instantaneous maps. For this purpose, we use the two dynamic datasets: SimCarla and RealKITTI. The source dictionary of atoms was prepared similar to the intra/inter-domain Table 2: Instantaneous map building in dynamic environments. Mean and SD are given. We evaluated the test performance of our model using accuracy (ACC), area under ROC curve (AUC), and negative log-likelihood (NLL). The higher the ACC and AUC or lower the NLL, the better. Figure 6. Table 2 shows the performance of transferring features extracted from each town to the dynamic datasets.

Represent the short summary for finding the corresponding scientific article (long)

We present a method of adapting hyperparameters of probabilistic models using optimal transport with applications in robotics

We learn a diversity sampling function with DPPs to obtain a diverse set of samples from a generative model.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: An algorithm is introduced for learning a predictive state representation with off-policy temporal difference (TD) learning that is then used to learn to steer a vehicle with reinforcement learning. There are three components being learned simultaneously: the off-policy predictions as a compact representation of state, the behavior policy distribution for estimating the off-policy predictions, and the deterministic policy gradient for learning to act. A behavior policy discriminator is learned and used for estimating the important sampling ratios needed to learn the predictive representation off-policy with general value functions (GVFs). A linear deterministic policy gradient method is used to train the agent with only the predictive representations while the predictions are being learned. All three components are combined, demonstrated and evaluated on the problem of steering the vehicle from images in the TORCS racing simulator environment. Steering from only images is a challenging problem where evaluation is completed on a held-out set of tracks that were never seen during training in order to measure the generalization of the predictions and controller. Experiments show the proposed method is able to steer smoothly and navigate many but not all of the tracks available in TORCS with performance that exceeds DDPG using only images as input and approaches the performance of an ideal non-vision based kinematics model. Predicting the future is an important topic in machine learning and is believed to be an important part of how humans process and interact with the world, cf. Study of the brain shows that it is highly predictive of future events and outcomes. Despite these advances, there is still much work needed to bridge the worlds of predictive learning and control. Most predictive control approaches learn either a forward model or a backward model however these next-step models suffer from compounding errors. This paper introduces a predictive control architecture using one kind of off-policy predictive learning, called general value functions (GVFs) Modayil et al. (2012 , that learns to predict the relevant aspects of the environment, decided by an expert, from raw sensor data such as pixel data captured from a camera. GVFs answer the predictive question, "if I follow policy τ, how much total cumulant will I see in the future?" The value of the GVF framework is not yet fully understood and realized despite the connections to neuroscience; but some early work has investigated its advantages for predictive representations and found that the representations are compact and general . An objective of this research is to better understand the value that GVFs have to offer in real-world applications. Our work is based on the hypothesis that predictive representations are good for generalization . We are motivated by the belief that GVFs, like RL, could allow for behavior that is anticipative of future consequences rather than reactive to the current state. General value functions (GVFs) are an understudied topic of interest in AI research fields and applications. There is a considerable focus on understanding how to learn these predictions but limited efforts on understanding how to use them in real applications. This is unfortunate, as todate, research into applications of GVFs suggest they have potential in real world robotics and its applications Günther et al. )White (2015 . However, several elements have been missing to apply these predictions to a larger scale problem such as autonomous driving: how to characterize the behavior policy to achieve off-policy learning when it is unknown, what predictions are useful, and how to use those predictions to control the vehicle. Our objective is two-fold: introduce a novel architecture combining elements of predictive learning, adversarial learning and reinforcement learning, and demonstrate how this architecture can be used to steer a vehicle in a racing simulator. Steering a vehicle is a challenging problem where the bicycle model is the classical approach. However, the bicycle model requires knowing the angle of the vehicle with respect to the road direction in order to compute the desired steering angle. Steering directly from images has been a long desired goal in autonomous driving where approaches like advocate for a two point model which inspired the multi-point predictive representation proposed in this paper. In comparison, learning to regress an image directly to the steering angle in an end-to-end manner has been a recent hot topic Chen & HuangSallab et al. (2017 . However, a serious challenge is ensuring robustness of the controller when learning end-to-end . In particular, the agent is not typically trained on recovery mode scenarios and so there are generalization and data coverage issues; for this reason, authors introduced augmented images in training by artificially shifting and rotating them to help the network learn to recover with some limited success. The approach in learns to predict the current road angle directly from images and then uses a classical steering controller to control the vehicle. The proposed approach is similar except we predict future road angles and lane centeredness at different temporal horizons which is then passed to a controller module to choose steering angles. Policy gradient with the predictive state representation is the approach used in this paper but this can also be replaced with other controllers. This architecture allows for a degree of interpretability in the controller that is not easily achieved with end-to-end approaches despite work on understanding and improving its robustness. We consider an environment described by a set of states S, a set of actions A, and Markov transition dynamics with probability P (s |s, a) of transitioning to next state s after taking action a from state s. This setting is nearly identical to a Markov Decision Process (MDP) where the only difference is the absence of a reward signal to maximize. The goal is to learn an estimator that predicts the return G t of a cumulant c t defined by where c t is a cumulant signal to be predicted, and 0 ≤ γ t < 1 is the continuation function. The general value function is defined as where τ (a|s), γ(s, a, s), and c(s, a, s) are the policy, continuation and cumulant functions, respectively, that make up the predictive question where V τ (s) represents the total discounted cumulant starting from state s and acting under policy τ. Unfortuantely, there are currently no algorithms to learn the predictive question through interaction with the environment; thus, τ, γ, and c are typically defined by an expert. Cumulants are commonly scaled by a factor of 1 − γ when γ is a constant in non-episodic predictions. A GVF can be approximated with a function approximator, such as a neural network, parameterized by θ to predict equation 1. The agent usually collects experience under a different behavior policy µ(a|s) where off-policy policy evaluation methods are needed to learn the GVF. The parameters θ are optimized with gradient descent minimizing the following loss function where δ = E[y −v τ (s; θ)|s, a] is the TD error and ρ = τ (a|s) µ(a|s) is the importance sampling ratio to correct for the difference between the target policy distribution τ and behavior distribution µ. Note that only the behavior policy distribution is corrected rather than the state distribution d µ. The target y is produced by bootstrapping a prediction of the value of the next state following target policy τ given by where y is a bootstrap prediction using recent parameters θ that are assumed constant in the gradient computation. Some approaches use older parameters θ of the network to make a bootstrapped prediction to improve stability in the learning. However, this was not found to be necessary when learning GVFs since the target policy is fixed and the learning is simply off-policy policy evaluation. d µ is the state distribution of the behavior policy µ and the time subscript on c and γ has been dropped to simplify notation. The gradient of the loss function equation 3 is given by An alternative approach to using importance sampling ratios ρ is to apply importance resampling. With importance resampling, a replay buffer D of size N is required and the gradient is multiplied with the average importance sampling ratio of the samples in the buffer. The importance resampling gradient is given by where the transitions in the replay buffer are sampled according to for transition with state s i and action a i in replay buffer D. This approach is proven to have lower variance than equation 5 with linear function approximation. An efficient data structure for the replay buffer is the SumTree used in prioritized experience replay. This is a natural approach to learning predictions with deep reinforcement learning since sampling a mini-batch from the replay buffer helps to decorrelate sample updates in deep function approximation. A behavior policy needs to be defined to adequately explore the environment when learning GVFs. This may be an evolving policy that is learned by RL, a random policy for exploring the environment, or a human driver collecting data safely. It is common, especially in the case of human drivers, for the behavior policy distribution µ(a|s) of the agent to be unknown. We propose an algorithm using the density ratio trick to learn the behavior policy distribution in an adversarial way. It is well suited for problems with low dimensional action spaces like autonomous driving. The ratio of two probability densities can be expressed as a ratio of discriminator class probabilities that distinguish samples from the two distributions. Let us define a probability density function η(a|s) for the distribution to compare to the behavior distribution µ(a|s) and class labels y = +1 and y = −1 that denote the class of the distribution that the state action pair was sampled from: µ(a|s) or η(a|s) respectively. A discriminator g(a, s) is learned that distinguishes state action pairs from these two distributions using the cross-entropy loss. The ratio of the densities can be computed using only the discriminator g(a, s). Here we assume that p(y = +1) = p(y = −1). From this , we can estimate µ(a|s) withμ(a|s) as followsμ where η(a|s) is a known distribution over action conditioned on state. The uniform distribution over the action is independent of state and has the advantage of being effective and easy to implement. The algorithm for training a GVF off-policy with an unknown behavior distribution is given by Algorithm 1 Off-policy GVF training algorithm with unknown µ(a|s) Compute cumulant c t+1 = c(s t, a t, s t+1) 7: Compute behavior density valueμ(a t |s t) according to equation 8 9: Compute importance sampling ratio Sample random minibatch A of transitions (s i, a i, c i+1, γ i+1, s i+1) from D according to probability Compute Perform gradient descent step on (y i −v τ (s i ; θ)) 2 according to equation 6 for minibatch A Sample random minibatch B of state action pairs (s i, a i) from D according to a uniform probability and assign label y = +1 to each pair 15: Randomly select half the samples in the minibatch B and temporarily replace the label with y = −1 and action with a t ∼ η(a|s) Update behavior discriminator g(a, s) with modified minibatch B Let us consider an MDP with a predictive representation φ(s) mapping state s to predictions φ(s). The reward for the problem is denoted as r. The problem is to find a policy π(a|φ(s)) that maximizes future return or accumulated discounted reward. We hypothesize that this approach should be easier to train than learning π(a|s) directly for the following reasons: • the target policy of the predictions τ is fixed making for faster learning • the compact abstraction φ(s) allows for simple (possibly even linear) policy and action-value functions • the cumulant signal c may only be available during training or is expensive to obtain The last advantage is particularly important in autonomous driving where localization techniques often require a collection of expensive sensors and high definition map data that is not always available or easily scalable to a fleet of autonomous vehicles. In this way, one can train a neural network to map images captured by inexpensive cameras to predictions of lane centeredness and road angle captured by any number of highly accurate but expensive localization approaches with the hopes of generalizing features for lane control. The agent learns to steer with deterministic policy gradient (DPG) using the predictions as the state of the agent. When linear policy function approximation is used, the controller learned is essentially equivalent to a prediction-based PID controller only where there is a deep mapping from images captured by a camera to predictions of the future used to control the vehicle. Using predictions for PID control is not new; this approach can be used to tackle problems with high temporal delay between the error signal and the corrective actions. One can also add integral and derivative terms of the predictions to the state space representation of the agent. Action-value Q π (φ(s), a) and policy π(φ(s)) networks are trained according to where the action value approximates the expected discounted return, i.e.. In addition, a policy network π(φ(s)) produces an action according to the current predictive state representation φ(s) that maximizes the expected discounted return. Because of the interesting connection between DPG and PID control when linear function approximation is used, the policy network is parameterized as where ψ is a matrix denoting parameters to be learned by DPG. They also represent the gain coefficients for a proportional controller which allows for interpretability of the learned parameters. The action-value network Q π (φ(s), a) is given by a small neural network that maps the predictive state representations φ(s) and action a to an estimate of the action-value in the current state s if a is taken and the optimal policy π followed thereafter. In autonomous driving, knowing the road curvature ahead can be informative for making decisions to maintain lane centeredness in a tight turn. In Figure 1, it is demonstrated how future off-policy predictions of lane centeredness can be used to predict the deviation (or error) between the true center of lane and the projected lane centeredness along the current direction of the vehicle. These predictions must be off-policy because if they were on-policy they would tell us no information to inform the agent how much adjustment is needed to make corrective actions to stay in the center of the lane. The lane centeredness α and road angle β are two kinds of predictions that are useful in steering the vehicle as depicted in Figure 2. We represent the road curvature as a set of predictions of future lane centeredness α and road angles β at different time horizons. The predictive state representation is given by the feature vector φ(s) where V τ α (s t ; γ = γ 0) is a GVF prediction under target policy τ, cumulant α (lane centeredness) and continuation function γ 0 while V τ β (s t ; γ = γ 0) is a GVF prediction of cumulant β (road angle). There are m α number of γ functions for predicting lane centeredness and m β number of γ functions for predicting road angle at different temporal horizons. Because the predictions represent deviations from the desired lane centeredness and road angle, the policy network of DPG can be linear. The predictive learning approach is applied to the challenging problem of learning to steer a vehicle in the racing environment. A kinematic-based steering approach is used as a baseline for all the experiments. The reward in the TORCS environment is given by r t = v t cos β t where v t is the speed of the vehicle in km/h, β t is the angle between the road direction and the vehicle direction. A simple scaling factor of 0.01 was applied to the reward in order to reduce the variance of the action-value. Notice that this reward doesn't force the agent to stay in the center of the lane; however, this strategy is likely a good idea to achieve high total reward on all the test tracks. The target speed of all the agents is 50 km/h where vehicle speed was controlled by a separate manually tuned PID controller. The agents were trained on 85% of the 40 tracks available in TORCS. The rest of the tracks were used for testing (6 in total); all of the in this section are on the testing tracks which were never presented to the agents during training. This was done to measure the generalization performance of the policies learned by the agents which can be a serious problem in RL, cf. Zhao et al.Farebrother et al. (2018 . Tracks where the road was not level were excluded since they proved to be challenging likely because a non-zero action was required to keep the vehicle centered. In all the figures, blue corresponds to DDPG-Image with only image and speed as input, green corresponds to DDPG-ImageLowDim with images, current speed, α and β provided as input, orange corresponds to the new GVF-DPG approach with image, current speed and last two actions since the target policy of the prediction depends on the last action taken, and red corresponds to the classical front wheel steering model. A history of two images were provided to each method. A supervised method of predicting the current lane centeredness and road angle directly from the image was attempted with negative : the controller learned was consistently unstable. Results were repeated over 5 runs. The total score achieved on the test tracks by the agents is plotted in Figure 3 over 1M training iterations. It is clear that DDPG-ImageLowDim performs best of the learned methods on all test tracks. This low dimensional information provides ground truth to the agent which may not always be available in all locations; however, it makes a good baseline target for what we hope our proposed method could achieve through generalization. With DDPG-Image, it is clear that it does not learn to steer from images very well. Figure 4 where DDPG-Image consistently converges to a solution that oscillates between the extreme left and right steering actions very rapidly. This oscillation is so extreme that the agent is unable to achieve the target speed of 50 km/h; instead it travels at 40 km/h on average for all the test tracks. The performance gap also suggests that DDPG-ImageLowDim may be relying more on the low dimensional lane information rather than the image. To highlight how uncomfortable the two DDPG agents drive compared to the GVF-DPG agent, Figure 4 shows the standard deviation of the change in action during 1M iterations of training. On most tracks, it is apparent that the GVF-DPG approach controls the vehicle more smoothly than the other learned methods. The performance of the individual test tracks is given in the following Figure 5 . The GVF-DPG approach does not steer successfully on all test tracks: it fails immediately on wheel-2, part-way through on a-wheelway, and drives well on most of alpine-2. The DDPG-Image fails to complete dirt-4, wheel-2, spring, and a-speedway. Finally, DDPG-ImageLowDim successfully completes all the test tracks; however, the agent has a strong bias to the left side of the track. The GVF-DPG agent often follows the classical controller relatively well except on the tracks where the agent fails. This suggests that using a predictive representation of lane centeredness α and road angle β achieves closer performance to a classical controller than an end-to-end learned approach. However, more work is needed to improve the generalization abilities of the approach. The learning curves for the predictors and the policy gradient agents is given in the following Figure 6 . It is interesting that the learning curves of the action-value function estimator of the GVF-DPG agent is much smaller than the other agents and quite smooth. The reason is believed to be because the predictive state representation is constrained to values between [−1, +1] acting as a sort of regularizer to the state representation of the agent. The learning curve of the DDPG-ImageLowDim however eventually approaches the low error of the GVF-DPG agent. The predictors converge relatively quickly as shown in Figure 6 (b). The behavior estimator in Figure 6 (c) stabilizes relatively quickly during learning as well; it is postulated that the error does not decrease further since the behavior policy is changing slowly over time and the behavior estimator must track this change. A method of learning a predictive representation off-policy is presented where the behavior policy distribution is estimated via an adversarial method employing the density ratio trick. It is demonstrated that deep off-policy predictions can be learned with a deep behavior policy estimation to predict future lane centeredness and road angles from images. The predictive representation is learned with linear deterministic policy gradient. All of these components are combined together in a framework called GVF-DPG and learned simultaneously on the challenging problem of steering a vehicle in TORCS from only images. The show that the GVF-DPG is able to steer smoothly with less change in action and achieve better performance than DDPG from only images and similar performance to the kinematics model in several but not all of the test tracks. This work is also a demonstration that we can learn off-policy predictions, characterize the behavior policy and learn the controller all at the same time despite the challenges of the behavior policy evolving with the agent and the predictive state representation changing over time. Our work demonstrates that a learned prediction-based vision-only steering controller could potentially be viable with more work on improving the generalizability of the off-policy predictions. This work supports the predictive state representation hypothesis in that deep predictions can improve the generalization of RL to new road environments when using only images as input. For future work, we hope to study how to learn the question for the predictive state representation: τ, γ, and c. Moreover, because the behavior policy is unknown and estimated, our suggest that collecting real-world human driving to train predictions off-policy without the need for a simulator could be a viable approach to steering a vehicle from images. This is potentially advantageous since the human driver can explore the road safely. The predictive learning approach presented in this paper is called GVF-DPG (general value function deterministic policy gradient). Exploration followed the same approach as where an Ornstein Uhlenbeck process is used to explore the track where the parameters of the process (θ = 0.01, σ = 0.01) were tuned to provide a gradual wandering behavior on the track without excessive oscillations in the action. The reason is to improve the learning of the off-policy predictions for GVF-DPG since the behavior policy µ(a|s) is closer to the target policy τ (a|s) of the predictions. The GVF-DPG approach learned 8 predictions: 4 predictions of lane centeredness α, and 4 predictions of road angle β. Each of the 4 predictions had different values of γ for different temporal horizons: 0.5, 0.9, 0.95, 0.97. This allowed the agent to make short-term, medium term and long term predictions of lane centeredness α and road angle β. The GVF predictors all share the same deep convolutional neural network where the convolutional layers are identical to the architecture in followed by three fully connected layers of 512, 384 and 8 outputs, respectively. The behavior estimator µ(a|s) also uses a very similar neural network with identical convolutional layers followed by three fully connected layers of 512, 256, and 1 output, respectively. The GVF predictors and behavior estimator are both given the current image, previous image, current speed and the last two actions taken. The policy network of the DPG agent is linear with respect to the predictions. The action-value network is a small 4 layer network of 64x64x32x1 and outputs the change in the action from the previous action. The predictions are supplied to the agent and the last action is needed to compute the next action. All networks use rectified linear unit activations except for the last layers which are linear. The learning rates for the linear policy network, action-value network, predictors and behavior policy network were 1e −4, 1e −8, 1e −4, and 1e −4 respectively. It was noted that the action-value learning rate needed to be small in order to faciliate two time-scale learning of the predictive representation and the action-values. The GVF-DPG approach is compared to two DDPG (deep deterministic policy gradient) baselines where the differences are given by only the information provided in the state. The first method is a vision-based approach where the image and current speed is provided to the agent; the only information available to the agent about the road is supplied via images. This proved challenging for the DDPG agent. Thus, a second agent was trained with the image and the lane centeredness α and road angle β that the GVF-DPG agent had access to during training. This is cheating in a sense because the agent must have the lane centeredness and road angle available at all times including at test time, whereas the GVF-DPG agent is hopefully learning features that are generalizable to unseen roads where this information may not be always available. However, it provides a good target for evaluating the GVF-DPG agent. An Ornstein Uhlenbeck process is used to explore the track (θ = 0.05, σ = 0.05). In previous works with DDPG, target networks were utilized for both the action-value and the policy networks. The target networks are copies of the action-value and policy networks that are updated more slowly in order make the bootstrapped prediction of the action-values more stable. However, in our experiments it was found that removing the target networks improved learning and so no target networks were deployed in any of the following experiments. Identical network architectures were used for the policy and action-value networks of the DDPG agents as was used for the behavior network for GVF-DPG. The low dimensional state information was feed through a separate branch of 12 neurons that were then merged with the fully connected layer of 512 neurons. This method of merging can present challenges due to mismatching statistical properties of the branches but that did not seem to present a significant challenge in learning. Future work would be to find better ways to bridge these two different pieces of information. The hyperbolic tangent activation was used for output layer of the policy network, the linear activation was used for the output layer of the action-value network and rectified linear activation was used everywhere else. During training, a track is selected randomly according to a priority sampling method, since the tracks were not balanced, and the agent is allowed to interact with the environment and learn until termination. Termination occurs when either the agent leaves the lane or the maximum number of steps has been reached (1200 steps = 120 seconds). A priority sampling method was chosen so that tracks that were more difficult were selected more often. The probability of sampling a track i is given by e − n i κ N j=1 e − n j κ where n i is the number of steps that the agent was able to achieve in the last episode for that track and κ controls the spread of the distribution. A value of κ = 1 N N j=1 n j was found to perform well. The initial probabilities are equal for all tracks. The TORCS environment was modified to provide higher resolution images in grayscale rather than RGB with most of the image above the horizon cropped out of the image. The grayscale images were 128 pixels wide by 64 pixels high. This allowed the agent to see more detail farther away which is very helpful in making long term predictions which is beneficial to both policy gradient methods and predictive learning.

Represent the TLDR for finding the corresponding scientific article

An algorithm to learn a predictive state representation with general value functions and off-policy learning is applied to the problem of vision-based steering in autonomous driving.

represent this text to find a much much longer article about it with say hundreds-thousands of words

We isolate one factor of RL generalization by analyzing the case when the agent only overfits to the observations. We show that architectural implicit regularizations occur in this regime.


Question: Represent the following document

Question: For numerous domains, including for instance earth observation, medical imaging, astrophysics,..., available image and signal datasets often irregular space-time sampling patterns and large missing data rates. These sampling properties is a critical issue to apply state-of-the-art learning-based (e.g., auto-encoders, CNNs,...) to fully benefit from the available large-scale observations and reach breakthroughs in the reconstruction and identification of processes of interest. In this paper, we address the end-to-end learning of representations of signals, images and image sequences from irregularly-sampled data, {\em i.e.} when the training data involved missing data. From an analogy to Bayesian formulation, we consider energy-based representations. Two energy forms are investigated: one derived from auto-encoders and one relating to Gibbs energies. The learning stage of these energy-based representations (or priors) involve a joint interpolation issue, which resorts to solving an energy minimization problem under observation constraints. Using a neural-network-based implementation of the considered energy forms, we can state an end-to-end learning scheme from irregularly-sampled data. We demonstrate the relevance of the proposed representations for different case-studies: namely, multivariate time series, 2{\sc} images and image sequences. In numerous application domains, the available observation datasets do not involve gap-free and regularly-gridded signals or images. The irregular-sampling may both from the characteristics of the sensors and sampling strategy, e.g. considered orbits and swaths in spacebone earth observation and astrophysics, sampling schemes in medical imaging, as well as environmental conditions which may affect the sensor, e.g. atmospheric conditions and clouds for earth observation. A rich literature exists on interpolation for irregularly-sampled signals and images (also referred to as inpainting in image processing). A classic framework states the interpolation issue as the miminisation of an energy, which may be interpreted in a Bayesian framework. A variety of energy forms, including Markovian priors, patch-based priors, gradient norms in variational and/or PDE-based formulations, Gaussian priors as well as dynamical priors in fluid dynamics. The later relates to optimal interpolation and kriging, which is among the state-of-the-art and operational schemes in geoscience. Optimal schemes classically involve the inference of the considered covariance-based priors from irregularly-sampled data. This may however be at the expense of Gaussianity and linearity assumptions, which do not often apply for real signals and images. For the other types of energy forms, their parameterization are generally set a priori and not learnt from the data. Regarding more particularly data-driven and learning-based approaches, most previous works (2; 11; 20) have addressed the learning of interpolation schemes under the assumption that a representative gap-free dataset is available. This gap-free dataset may be the image itself (9; 20; 18). For numerous application domains, as mentionned above, this assumption cannot be fulfilled. Regarding recent advances in learning-based schemes, a variety of deep learning models, e.g. (7; 16; 24; 23), have been proposed. Most of these works focus on learning an interpolator. One may however expect to learn not only an interpolator but also some representation of considered data, which may be of interest for other applications. In this respect, RBM models (Restricted Boltzmann In this section, we formally introduce the considered issue, namely the end-to-end learning of representations and interpolators from irregularly-sampled data. Within a classic Bayesian or energybased framework, interpolation issues may be stated as a minimization issue where X is the considered signal, image or image series (referred to hereafter as the hidden state), Y the observation data, only available on a subdomain Ω of the entire domain D, and U θ the considered energy prior parameterized by θ. As briefly introduced above, a variety of energy priors have been proposed in the literature, e.g. (4; 20; 5). We assume we are provided with a series of irregularly-sampled observations, that is to say a set is only defined on subdomain Ω (i). Assuming that all X (i) share some underlying energy representation U θ , we may define the following operator such that I(Here, we aim to learn the parameters θ of the energy U θ from the available observation dataset {Y (i), Ω (i) } i. Assuming operator I is known, this learning issue can be stated as the minimization of reconstruction error for the observed data where. 2 Ω refers to the L2 norm evaluated on subdomain. Learning energy U θ from observation dataset {Y (i), Ω (i) } i clearly involves a joint interpolation issue solved by operator I. Given this general formulation, the end-to-end learning issue comes to solve minimization according to some given parameterization of energy U θ . In, interpolation operator I is clearly critical. In Section 3, we investigate a neural-network implementation of this general framework, which embeds a neural-network formulations both for energy U θ and interpolation operator I. In this section, we detail the proposed neural-network-based implementation of the end-to-end formulation introduced in the previous section. We first present the considered paramaterizations for energy U θ in (Section 3.1). We derive associated NN-based interpolation operators I (Section 3.2) and describe our overall NN architectures for the end-to-end learning of representations and interpolators from irregularly-sampled datasets (Section 3.3). We first investigate NN-based energy representations based on auto-encoders. Let us denote by φ E and φ D the encoding and decoding operators of an auto-encoder (AE), which may comprise both dense auto-encoders (AEs), convolutional AEs as well as recurrent AEs when dealing with time-related processes. The key feature of AEs is that the encoding operator φ E maps the state X into a low-dimensional space. Auto-encoders are naturally associated with the following energy Minimizing according to this energy amounts to retrieving the hidden state whose lowdimensional representation in the encoding space matches the observed data in the original decoded space. Here, parameters θ refer to the parameters of the encoder φ E and decoder φ D, respectively θ E and θ D. The mapping to lower-dimensional space may be regarded as a potential loss in the representation potential of the representation. Gibbs models provide an appealing framework for an alternative energy-based representation, with no such dimensionality reduction constraint. Gibbs models introduced in statistical physics have also been widely explored in computer vision and pattern recognition from the 80s. Gibbs models relate to the decomposition of U θ as a sum of potentials U θ (X) = c∈C V c (X c) where C is a set of cliques, i.e. a set of interacting sites (typically, local neighbors), and V c the potential on clique c. In statistical physics, this formulation states the global energy of the system as the sum of local energies (the potential over locally-interacting sites). Here, we focus on the following parameterization of the potential function with N s the set of neighbors of site s for the entire domain D and ψ a potential function. Lowenergy state for this energy refers to state X which operator ψ provides a good prediction at any site s knowing the state in the neighborhood N s of s. This type of Gibbs energy relates to Gaussian Markov random fields, where the conditional likelihood at one site given its neighborhood follows a Gaussian distribution. We implement this type of Gibbs energy using the following NN-based parameterization of operator ψ: ψ(X) = ψ 2 (ψ 1 (X)) It involves the composition of a space and/or time convolutional operator ψ 1 and a coordinate-wise operator ψ 2. The convolutional kernel for operator ψ 1 is such that the coefficients for the center of convolutional window are set to zero. This property fulfills the constraint that X(s) is not involved in the computation of ψ (X Ns) at site s. As an example, for a univariate image, ψ 1 can be set as a convolutional layer with N F filters with kernels of size 3x3x1, such that for each kernel K f K f = 0 (the same applies to biases). In such a case, operator ψ 2 would be a convolution layer with one filter with a kernel of size 1x1xN F. Both ψ 1 and ψ 2 can also involve non-linear activations. Without loss of generality, given this parameterization for operator ψ, we may rewrite energy U θ as U θ (X) = X − ψ (X) 2 where ψ (X)) at site s is given by ψ (X Ns). Overall, we may use the following common formulation for the two types of energy-based representation They differ in the parameterization chosen for operator ψ. Besides the NN-based energy formulation, the general formulation stated in involves the definition of interpolation operator I, which refers to minimization. We here derive NN-based interpolation architectures from the considered NN-based energy parameterization. Given parameterization, a simple fixed-point algorithm may be considered to solve for. This algorithm at the basis of DINEOF algorithm and XXX for matrix completion under subspace constraints (2; 14) involves the following iterative update Interestingly, the algorithm is parameter-free and can be readily implemented in a NN architecture given the number of iterations to be considered. Given some initialisation, one may typically consider an iterative gradient-based descent which applies at each iteration k with J U θ the gradient of energy U θ w.r.t. state X, λ the gradient step and Ω the missing data area. Automatic differentiation tool embedded in neural network frameworks may provide the numerical computation for gradient J U θ given the NN-based parameterization for energy U θ. This proved numerically too expensive and was not further investigated in our experiments. Given the considered form for energy U θ, its gradient w.r.t. X decomposes as a product and X − ψ (X) may be regarded as a suboptimal gradient descent. Hence, rather than considering the true Jacobian J ψ for operator ψ, we may consider an approximation through a trainable CNN G such that the gradient descent becomes where andG is a CNN to be learnt jointly to ψ during the learning stage. Interestingly, this gradient descent embeds the fixed-point algorithm whenG is the identity. Let us denote respectively by I F P and I G the fixed-point and gradient-based NN-based interpolators, which implement N I iterations of the proposed interpolation updates. Below, I N N will denote both I F P and I G. Whereas I F P is parameter-free, I G involves the parameterization of operator G. We typically consider a CNN with ReLu activations with increasing numbers of filter through layers up to the final layer which applies a linear convolutional with a number of filters given by the dimension of the state. Figure 1: Sketch of the considered end-to-end architecture: we depict the considered N I -block architecture which implements a N I -step interpolation algorithm described in Section 3.2. Operator ψ is defined through energy representation and G refers to the NN-based approximation of the gradient-based update for minimization. This architecture uses as input a mask Ω corresponding to the missing-data-free domain and an initial gap-filling X for state X. We typically initially fill missing data with zeros for centered and normalized states. Given the parameterizations for energy U θ and the associated NN-based interpolators presented previously, we design an end-to-end learning for energy representation U θ and associated interpolator I N N, which uses as inputs an observed sample Y (i) and the associated missing-data-free domain Ω (i). Using a normalization preprocessing step, we initially fill missing data with zeros to provide an initial interpolated state to the architecture. We provide a sketch of the architecture in Fig.1. Regarding implementation details, beyond the design of the architectures, which may be applicationdependent for operators ψ andG (see Section 4), we consider an implementation under keras using tensorflow as backend. Regarding the training strategy, we use adam optimizer. We iteratively increase the number of blocks N I (number of gradient steps) to avoid the training to diverge. Similarly, we decrease the learning rate accross iterations, typically from 1e-3 to 1e-6. In our experiments, we typically consider from 5 to 15 blocks. All the experiments were run under workstations with a single GPU (Nvidia GTX 1080 and GTX 1080 Ti). In this section, we report numerical experiments on different datasets to evaluate and demonstrate the proposed scheme. We consider three different case-studies: an image dataset, namely MNIST; a multivariate time-series through an application to Lorenz-63 dynamics and an image sequence dataset through an application to ocean remote sensing data with real missing data patterns. In all experiments, we refer to the AE-based framework, respectively as FP(d)-ConvAE and G(d)-ConvAE using the fixed-point or gradient-based interpolator where the value of d refers to the number of interpolation steps. Similarly, we refer to the Gibbs-based frameworks respectively as FP(d)-GENN and G(d)-GENN. We evaluate the proposed framework on MNIST datasets for which we simulate missing data patterns. The dataset comprises 60000 28x28 grayscale images. For this dataset, we only evaluate the AE-based setting. We consider the following convolutional AE architecture with a 20-dimensional encoding space: • Encoder operator φ E: Conv2D+ ReLU + AvPooling + Conv2D + ReLU + AveragePooling + Dense + ReLU + Dense; • Decoder operator φ E: Conv2DTranspose + ResNet, ResNet: Conv2D+ReLU+Conv2D We generate random missing data patterns composed of N S squares of size W S xW S, the center of the square is randomly sampled uniformly over the image grid. As illustrated in Fig.3, we consider four missing data patterns: N S = 20 and W S = 5, N S = 30 and W S = 5, N S = 3 and W S = 9, N S = 6 and W S = 9. As performance measure, we evaluate an interpolation score (I-score), a global reconstruction score (R-score) for the interpolated images and an auto-encoding (AE-score) score of the trained auto-encoder applied to gap-free data, in terms of explained variance. We also evaluate a classification score (C-score), in terms of mean accurcay, using the 20-dimensional encoding space as feature space for classification with a 3-layer MLP. We report all performance measures for both the test dataset in Tab.1 for MNIST dataset. For benchmarking purposes, we also report the performance of DINEOF framework, which uses a 20-dimensional PCA trained on the gap-free dataset, the auto-encoder architecture trained on gap-free dataset as well as the considered convolutional auto-encoder trained using an initial zero-filling for missing data areas and a training loss computed only of observed data areas. The later can be regarded as a FP-ConvAE architecture using a single block in Fig.1. Overall, these illustrate that representations trained from gap-free data may not apply when considering significant missing data rates as illustrated by relatively poor performance of PCA-based and AE schemes, when trained from gap-free data. Similarly, training an AE representations using as input a zero-filling strategy lowers the auto-encoding power when applied to gap-free data. Overall, the proposed scheme guarantees a good representation in terms of AE score with an additional gain in terms of interpolation performance, typically between ≈ 15% and 30% depending of the missing data patterns, the gain being greater when considering larger missing data areas. Table 1: Performance of AE schemes in presence of missing data for Fashion MNIST dataset: for a given convolutional AE architecture (see main text for details), a PCA and ConvAE models trained on gap-free data with a 15-iteration projection-based interpolation (resp., DINEOF and ConvAE), a zero-filling stratefy with the same ConvAE architecture (Zero-ConvAE) and the fixedpoint and gradient-based versions of the proposed scheme. For each experiment, we evaluate four measures: the reconstruction performance for the known image areas (R-score), the interpolation performance for the missing data areas (I-score), the reconstruction performance of the trained AE when applied to gap-free images (AE-score), the classification score of a MLP classifier trained in the trained latent space for training images involving missing data. We present an application to the Lorenz-63 dynamics, which involve a 3-dimensional state governed by the following ordinary differential equation: Under parameterization σ = 10, ρ = 28 and β = 8/3 considered here, Lorenz-63 dynamics are chaotic dynamics, which make then challening in our context. They can be regarded as a reducedorder model of turbulence dynamics. We simulate Lorenz-63 time series of 200 time steps using a Runge-Kutta-4 ODE solver with an integration step of 0.01 from an initial condition in the attractor. For a given experiment, we first subsample the simulated series to a given time step dt and then generate using a uniform random samplong a missing data mask accounting for 75% of the data. Overall, training and test time series are formed by subsequences of 200 time steps. We report experiments with the GE-NN setting. The AE-based framework showed lower performance and is not included here. The considered GE-NN architecture is as follows: a 1D convolution layer with 120 filters with a kernel width of 3, zero-weight-constraints for the center of the convolution kernel and a Relu activation, a 1D convolution layer with 6 filters a kernel width of 1 and a Relu activation, a residual network with 4 residual units using 6 filters with a kernel width of 1 and a linear activation. The last layer is a convolutional layer with 3 filters, a kernel width of 1 and a linear activation. Figure 2: Example of missing data interpolation for Lorenz-63 dynamics: from left to right, the time series of each of the three components of Lorenz-63 states for dt = 0.02 and a 75% missing data rate. We depict the irregularly-sampled observed data (black dots), the true state (green,-), the interpolated states using DINEOF (blue, -) and the interpolated states using the proposed approach (G-NN-FP-OI) (red, -). Visually, the interpolated sequence using our approach can hardly be distinguished from the true states. For benchmarking purposes, we report the interpolation performance issued from an ensemble Kalman smoother (EnKS) knowing the true model, regarded as a lower-bound of the interpolation performance. The parameter setting of the EnKS is as follows: 200 members, noise-free dynamical model and spherical observation covariannce to 0.1 · I. We also compare the proposed approach to DINEOF (2; 21). Here, the learning of the PCA decomposition used in the DINEOF scheme relies on gap-free data. Fig.2 illustrates this comparison for one sequence of 200 time steps with dt = 0.02. In this example, one can hardly distinguish the interpolated sequence using the proposed approach (FP-GE-NN). By contrast, DINEOF scheme cannot retrieve some of the largest deviations. We report in Appendix Tab.3 the performance of the different interpolation schemes. The proposed approach clearly outperforms DINEOF by about one order of magnitude for the experiments with a time-step dt = 0.02. The interpolation error for observed states (first line in Tab.3) also stresses the improved prior issued from the proposed Gibbs-like energy setting. For chaotic dynamics, global PCA representation seems poorly adapted where local representations as embedded by the considered Gibbs energy setting appear more appealing. The third case-study addresses satellite-derived Sea Surface Temperature (SST) image time series. Due to their sensitivity to the cloud cover, such SST datasets issued from infrared sensors may involve large missing data rates (typically, between 70% and 90%, Fig.? ? for an illustration). For evaluation purposes, we build a groundtruthed dataset from high-resolution numerical simulations, namely NATL60 data, using real cloud masks from METOP AVHRR sensor (19 For this case-study, we consider the following four architectures for the AEs and the GE-NNs: • ConvAE 1 : the first convolutional auto-encoder involves the following encoder architecture: five consecutive blocks with a Conv2D layer, a ReLu layer and a 2x2 average pooling layer, the first one with 20 filters the following four ones with 40 filters, and a final linear convolutional layer with 20 filters. The output of the encoder is 4x16x20. The decoder involves a Conv2DTranspose layer with ReLu activation for an initial 16x16 upsampling stage a Conv2DTranspose layer with ReLu activation for an additional 2x2 upsampling stage, a Conv2D layer with 16 filters and a last Conv2D layer with 5 filters. All Conv2D layers use 3x3 kernels. Overall, this model involves ≈ 400,000 parameters. • ConvAE 2 : we consider a more complex auto-encoder with an architecture similar to ConvAE 1 where the number of filters is doubled (e.g., The output of the encoder is a 4x16x40 tensor). Overall, this model involves ≈ 900,000 parameters. • GE-NN 1,2: we consider two GE-NN architectures. They share the same global architecture with an initial 4x4 average pooling, a Conv2D layer with ReLu activation with a zeroweight constraint on the center of the convolution window, a 1x1 Conv2D layer with N filters, a ResNet with a bilinear residual unit, composed of an initial mapping to an initial 32x128x(5*N) space with a Conv2D+ReLu layer, a linear 1x1 Conv2D+ReLu layer with N filters and a final 4x4 Conv2DTranspose layer with a linear activation for an upsampling to the input shape. GE-NN 1 and GE-NN 2 differ in the convolutional parameters of the first Conv2D layers and in the number of residual units. GE-NN 1 involves 5x5 kernels, N = 20 and 3 residual units for a total of ≈ 30,000 parameters. For GE-NN 2, we consider 11x11 kernels, N = 100 and 10 residual units for a total of ≈ 570,000 parameters. These different parameterizations were selected so that ConvAE 1 and GE-NN 2 involve a modeling complexity in the same range. We may point out that the considered GE-NN architecture are not applied to the finest resolution but to downscaled grids by a factor of 4. The application of GENNs to the finest resolution showed poor performance. This is regarded as an illustration of the requirement for considering a scale-selection problem when applying a given prior. The upscaling involves the combination of a Conv2DTranspose layer with 11 filters, a Conv2D layer with a ReLu activation with 22 filters and a linear Conv2D layer with 11 filters. Similarly to MNIST dataset, we report the performance of the different models in terms of interpolation score (I-score), reconstruction score (R-score) and auto-encoding score (AE-score) both for the training and test dataset. We compare the performance of the four models using the fixed-point and gradient-based interpolation. Overall, we can draw conlusions similar to MNIST case-study. Representations trained from gap-free data lead to poor performance and the proposed scheme reaches the best performance (gain over 50% in terms of explained variance for the interpolation and reconstruction score). Here, models trained with a zero-filling strategy show good interpolation and reconstruction performance, but very poor AE score, stressing that cannot apply beyond the considered interpolation task. When comparing GE-NN and AE settings, GE-NNs show slightly better performance with a much lower complexity (e.g., 30,000 parameters for GE-NN 1 vs. 400,000 parameters for ConvAE 1). Regarding the comparison between the fixed-point and gradient-based interpolation strategies, the later reaches slightly better interpolation and reconstruction score. We may point out the significant gain w.r.t. OI, which is the current operational tool for ocean remote sensing data. We illustrate these in Appendix (Fig.6), which further stresses the gain w.r.t. OI for the reconstruction of finer-scale structures. Table 2: Performance on SST dataset: We evaluate for each model interpolation, reconstruction and auto-encoding scores, resp. I-score, R-score and AE-score, in terms of percentage of explained variance resp. for the interpolation of missing data areas, the reconstruction of the whole image with missing data and the reconstruction of gap-free images. For each model, we evaluate these score for the training data (first row) and the test dataset (second row in brackets). We consider four different auto-encoder models, namely 20 and 80-dimensional PCAs and ConvAE 1,2 models, and two GE-NN models, GE-NN 1,2, combined with three interpolation strategies: the classic zero-filling strategy (Zero) and proposed iterative fixed-point (FP) and gradient-based (G) schemes, the figure in brackets denoting the number of iterations. For instance, FP-GE-NN 1 refers to GE-NN 1 with a 10-step fixed-point interpolation scheme. The PCAs are trained from gap-free data. We also consider an Optimal Interpolation (OI) with a space-time Gaussian covariance with empiricallytuned parameters. We refer the reader to the main text for the detailed parameterization of the considered models. In this paper, we have addressed the learning of energy-based representations of signals and images from observation datasets involving missing data (with possibly very large missing data rates). Using the proposed architectures, we can jointly learn relevant representations of signals and images while jointly providing the associated interpolation schemes. Our experiments stress that learning representations from gap-free data may lead to representations poorly adapted to the analysis of data with large missing data areas. We have also introduced a Gibbs priors embedded in a neural network architecture. Relying on local characteristics rather than global ones as in AE schemes, these priors involve a much lower complexity. Our experiments support their relevance for addressing inverse problems in signal and image analysis. Future work may further explore multi-scale extensions of the proposed schemes along with couplings between global and local energy representations and hybrid minimization schemes combining both gradient-based and fixed-point strategies in the considered end-to-end formulation. A.1 SUPPLEMENTARY FOR MNIST DATASET We illustrate below both the considered masking patterns as well as reconstruction examples for the proposed framework applied to MNIST dataset. The first row depicts the reference image, the second row the missing data mask and the third one the interpolated image. The first two panels illustrate interpolation for training data and last two for test data. We depict grayscale mnist images using false colors to highmight differences. We report below a Table which details the interpolation performance of the proposed GE-NN representation applied to Lorenz-63 time series in comparison with a PCA-based scheme and a lowerbound provided by the interpolation assuming the ODE model is known. A.3 SUPPLEMENTARY FOR SST DATASET We report below reconstruction examples for the application of the proposed GE-NN approach to SST time series with real missing data masks, which involve very large missing data rates (typically above 80%). The consistency between the interpolation and the reconstruction of the gap-free image from the learnt energy-based representation further stresses the ability of the proposed approach to extract a generic representation from irregularly-sampled data. These reulsts is known, and a DINEOF scheme. We report interpolation for a 75% missing data rate with uniform random sampling for three different sampling time steps, dt = 0.01, dt = 0.02 and dt = 0.04. We report the mean square error of the interpolation for the observed data (first row) and masked ones (second row). also emphasize a much greater ability of the proposed learning-based scheme to reconstruct finescale structures, which can hardly be reconstructed by an OI scheme with a Gaussian space-time covariance model. We may recall that the later is the stae-of-the-art approach for the processing of satellite-derived earth observation data. Interpolation examples for SST data used during training: first row, reference SST images corresponding to the center of the considered 11-day time window; second row, associated SST observations with missing data, third row, interpolation issued from FP-GE-NN 2 model; third row, reconstruction of the gap-free image series issued from FP-GE-NN 2 model; interpolation issued from an optimal interpolation scheme using a Gaussian covariance model with empirically tuned parameters. Figure 6: Interpolation examples for SST data never seen during training: first row, reference SST images corresponding to the center of the considered 11-day time window; second row, associated SST observations with missing data, third row, interpolation issued from FP-GE-NN 2 model; third row, reconstruction of the gap-free image series issued from FP-GE-NN 2 model; interpolation issued from an optimal interpolation scheme using a Gaussian covariance model with empirically tuned parameters.

represent the tldr for finding the corresponding scientific article

We address the end-to-end learning of energy-based representations for signal and image observation dataset with irregular sampling patterns.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

This paper 1) characterizes functions representable by ReLU DNNs, 2) formally studies the benefit of depth in such architectures,  3) gives an algorithm to implement empirical risk minimization to global optimality for two layer ReLU nets.